From chris.hinsley at gmail.com Thu Jul 4 11:49:08 2013 From: chris.hinsley at gmail.com (Chris Hinsley) Date: Thu, 4 Jul 2013 16:49:08 +0100 Subject: First attempt at a Python prog (Chess) References: <2013021323250974803-chrishinsley@gmailcom> Message-ID: <2013070416490865880-chrishinsley@gmailcom> On 2013-02-13 23:25:09 +0000, Chris Hinsley said: > New to Python, which I really like BTW. > > First serious prog. Hope you like it. I know it needs a 'can't move if > your King would be put into check' test. But the weighted value of the > King piece does a surprising emergent job. New version with better search and hopefully a little better Python. I know it's still far from fully correct and it'll never best Gary Kasperov. :) With pypy on my Macbook I can go to 6 PLY with resonable performance now due to the alpha-beta pruneing. Regards #!/usr/bin/python -tt # -*- coding: utf-8 -*- # Copyright (C) 2013 Chris Hinsley, GPL V3 License import sys import random import os PLY = 5 EMPTY = 0 WHITE = 1 BLACK = -1 NO_CAPTURE = 2 MAY_CAPTURE = 3 MUST_CAPTURE = 4 piece_type = {' ' : EMPTY, 'K' : BLACK, 'Q' : BLACK, 'R' : BLACK, 'B' : BLACK, 'N' : BLACK, 'P' : BLACK, \ 'k' : WHITE, 'q' : WHITE, 'r' : WHITE, 'b' : WHITE, 'n' : WHITE, 'p' : WHITE} def display_board(board): print ' a b c d e f g h' print '+---+---+---+---+---+---+---+---+' for row in range(8): for col in range(8): print '| %c' % board[row * 8 + col], print '|', 8 - row print '+---+---+---+---+---+---+---+---+' def piece_moves(board, index, dx, dy, capture_flag, distance): piece = board[index] type = piece_type[piece] cy, cx = divmod(index, 8) for step in range(distance): nx = cx + (dx * (step + 1)) ny = cy + (dy * (step + 1)) if (0 <= nx < 8) and (0 <= ny < 8): newindex = ny * 8 + nx newpiece = board[newindex] newtype = piece_type[newpiece] if capture_flag == MUST_CAPTURE: if newtype != EMPTY and newtype != type: board[index] = ' ' if (ny == 0 or ny == 7) and piece in 'Pp': for promote in 'QRBN' if type == BLACK else 'qrbn': board[newindex] = promote yield board else: board[newindex] = piece yield board board[index], board[newindex] = piece, newpiece elif capture_flag == MAY_CAPTURE: if newtype == EMPTY: board[index], board[newindex] = ' ', piece yield board board[index], board[newindex] = piece, newpiece elif newtype != type: board[index], board[newindex] = ' ', piece yield board board[index], board[newindex] = piece, newpiece break else: break elif newtype == EMPTY: board[index] = ' ' if (ny == 0 or ny == 7) and piece in 'Pp': for promote in 'QRBN' if type == BLACK else 'qrbn': board[newindex] = promote yield board else: board[newindex] = piece yield board board[index], board[newindex] = piece, newpiece else: break else: break def pawn_moves(board, index, options): for x, y, flag, distance in options: for new_board in piece_moves(board, index, x, y, flag, distance): yield new_board def other_moves(board, index, options, distance): for x, y in options: for new_board in piece_moves(board, index, x, y, MAY_CAPTURE, distance): yield new_board def black_pawn_moves(board, index): distance = 2 if index in range(8, 16) else 1 for new_board in pawn_moves(board, index, [(0, 1, NO_CAPTURE, distance), (-1, 1, MUST_CAPTURE, 1), (1, 1, MUST_CAPTURE, 1)]): yield new_board def white_pawn_moves(board, index): distance = 2 if index in range(48, 56) else 1 for new_board in pawn_moves(board, index, [(0, -1, NO_CAPTURE, distance), (-1, -1, MUST_CAPTURE, 1), (1, -1, MUST_CAPTURE, 1)]): yield new_board def rook_moves(board, index): for new_board in other_moves(board, index, [(0, -1), (-1, 0), (0, 1), (1, 0)], 7): yield new_board def bishop_moves(board, index): for new_board in other_moves(board, index, [(-1, -1), (-1, 1), (1, 1), (1, -1)], 7): yield new_board def knight_moves(board, index): for new_board in other_moves(board, index, [(-2, 1), (2, -1), (2, 1), (-1, -2), (-1, 2), (1, -2), (1, 2)], 1): yield new_board def queen_moves(board, index): for new_board in bishop_moves(board, index): yield new_board for new_board in rook_moves(board, index): yield new_board def king_moves(board, index): for new_board in other_moves(board, index, [(0, -1), (-1, 0), (0, 1), (1, 0), (-1, -1), (-1, 1), (1, 1), (1, -1)], 1): yield new_board moves = {'P' : black_pawn_moves, 'p' : white_pawn_moves, \ 'R' : rook_moves, 'r' : rook_moves, \ 'B' : bishop_moves, 'b' : bishop_moves, \ 'N' : knight_moves, 'n' : knight_moves, \ 'Q' : queen_moves, 'q' : queen_moves, \ 'K' : king_moves, 'k' : king_moves} def all_moves(board, colour): for index, piece in enumerate(board): if piece_type[piece] == colour: for new_board in moves[piece](board, index): yield new_board piece_values = {'K' : (10000, 0), 'k' : (0, 10000), \ 'P' : (1, 0), 'p' : (0, 1), \ 'N' : (3, 0), 'n' : (0, 3), \ 'B' : (3, 0), 'b' : (0, 3), \ 'R' : (5, 0), 'r' : (0, 5), \ 'Q' : (9, 0), 'q' : (0, 9)} generic_position_values = [0, 0, 0, 0, 0, 0, 0, 0, \ 0, 1, 1, 1, 1, 1, 1, 0, \ 0, 1, 2, 2, 2, 2, 1, 0, \ 0, 1, 2, 3, 3, 2, 1, 0, \ 0, 1, 2, 3, 3, 2, 1, 0, \ 0, 1, 2, 2, 2, 2, 1, 0, \ 0, 1, 1, 1, 1, 1, 1, 0, \ 0, 0, 0, 0, 0, 0, 0, 0] white_king_position_values = [0, 0, 0, 0, 0, 0, 0, 0, \ 0, 0, 0, 0, 0, 0, 0, 0, \ 0, 0, 0, 0, 0, 0, 0, 0, \ 0, 0, 0, 0, 0, 0, 0, 0, \ 0, 0, 0, 0, 0, 0, 0, 0, \ 0, 0, 0, 0, 0, 0, 0, 0, \ 0, 0, 0, 0, 0, 0, 0, 0, \ 3, 3, 3, 3, 3, 3, 3, 3] black_king_position_values = [3, 3, 3, 3, 3, 3, 3, 3, \ 0, 0, 0, 0, 0, 0, 0, 0, \ 0, 0, 0, 0, 0, 0, 0, 0, \ 0, 0, 0, 0, 0, 0, 0, 0, \ 0, 0, 0, 0, 0, 0, 0, 0, \ 0, 0, 0, 0, 0, 0, 0, 0, \ 0, 0, 0, 0, 0, 0, 0, 0, \ 0, 0, 0, 0, 0, 0, 0, 0] piece_positions = {'K' : black_king_position_values, 'k' : white_king_position_values, \ 'P' : generic_position_values, 'p' : generic_position_values, \ 'N' : generic_position_values, 'n' : generic_position_values, \ 'B' : generic_position_values, 'b' : generic_position_values, \ 'R' : generic_position_values, 'r' : generic_position_values, \ 'Q' : generic_position_values, 'q' : generic_position_values} def evaluate(board): black_score, white_score = 0, 0 for index, piece in enumerate(board): type = piece_type[piece] if type != EMPTY: position_value = piece_positions[piece][index] if type == BLACK: black_score += position_value else: white_score += position_value black_value, white_value = piece_values[piece] black_score += black_value white_score += white_value return white_score - black_score def next_move(board, colour, alpha, beta, ply): if ply <= 0: return evaluate(board) * colour for new_board in all_moves(board[:], colour): score = -next_move(new_board, -colour, -beta, -alpha, ply - 1) if score >= beta: return score if score > alpha: alpha = score return alpha def best_move(board, colour, alpha, beta, ply): best_board = [] for new_board in all_moves(board, colour): score = -next_move(new_board, -colour, -beta, -alpha, ply - 1) if score > alpha: alpha = score best_board = new_board[:] return best_board def main(): board = list('RNBQKBNRPPPPPPPP pppppppprnbqkbnr') colour = WHITE while True: board = best_move(board, colour, -1000000, 1000000, PLY) colour = -colour os.system('clear') display_board(board) #raw_input() if __name__ == '__main__': main() From joshua.landau.ws at gmail.com Thu Jul 4 21:33:04 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Fri, 5 Jul 2013 02:33:04 +0100 Subject: First attempt at a Python prog (Chess) In-Reply-To: <2013070416490865880-chrishinsley@gmailcom> References: <2013021323250974803-chrishinsley@gmailcom> <2013070416490865880-chrishinsley@gmailcom> Message-ID: Just a minor suggestion: def display_board(board): print ' a b c d e f g h' print '+---+---+---+---+---+---+---+---+' for row in range(8): for col in range(8): piece = board[row * 8 + col] if piece_type[piece] == WHITE: print '| \x1b[31;01m%c\x1b[39;49;00m' % board[row * 8 + col], else: print '| \x1b[34;01m%c\x1b[39;49;00m' % board[row * 8 + col], print '|', 8 - row print '+---+---+---+---+---+---+---+---+' From aliencat777 at gmail.com Sat Jul 27 08:28:31 2013 From: aliencat777 at gmail.com (aliencat777 at gmail.com) Date: Sat, 27 Jul 2013 05:28:31 -0700 (PDT) Subject: programming course In-Reply-To: References: Message-ID: Hi, A good step by step easy book on Python is: "Start Here: Python 3x Programming Made Fun and Easier," at http://www.quantum-sight.com Aliencat From rosuav at gmail.com Sat Jul 27 08:40:57 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 27 Jul 2013 13:40:57 +0100 Subject: programming course In-Reply-To: References: Message-ID: On Sat, Jul 27, 2013 at 1:28 PM, wrote: > Hi, > A good step by step easy book on Python is: "Start Here: Python 3x Programming Made Fun and Easier," at http://www.quantum-sight.com This is a Usenet group and a mailing list, not a web forum. You do not need to dig up a dozen ancient threads in order to advertise your wares. And by the way, it's courteous to be up-front about (1) your connection with what you're recommending, and (2) the fact that it's a pay-for book. One post, in a thread of its own, announcing the release of the book, would be appropriate and on-topic. Just please don't reply to heaps of old threads with commercial proposals :) ChrisA From ptmcg at austin.rr.com Sat Jul 20 17:53:50 2013 From: ptmcg at austin.rr.com (Paul McGuire) Date: Sat, 20 Jul 2013 14:53:50 -0700 (PDT) Subject: Problem installing Pyparsing In-Reply-To: References: <009f01ce1e8a$03e89f60$0bb9de20$@com> Message-ID: <707c4d57-754d-47d4-8bae-fb4005056c38@googlegroups.com> Pyparsing 2.0.1 fixes this incompatibility, and should work with all versions of Python 2.6 and later. -- Paul From rurpy at yahoo.com Mon Jul 1 10:02:10 2013 From: rurpy at yahoo.com (rurpy at yahoo.com) Date: Mon, 1 Jul 2013 07:02:10 -0700 (PDT) Subject: Don't feed the troll... In-Reply-To: References: <7wfvwkcihf.fsf@benfinney.id.au> <51be1a42$0$29966$c3e8da3$5496439d@news.astraweb.com> <96ed14e3-79dd-40c2-8063-aa58a42c7b57@googlegroups.com> <744948f6-85dd-4de3-a886-8282ca903816@googlegroups.com> <3207a22f-846a-41a8-8165-79c25e3c4285@googlegroups.com> <51CB4513.6060201@rece.vub.ac.be> <51CC80B9.5090501@rece.vub.ac.be> Message-ID: <80e9fcec-ae50-4aae-b238-cb4f457efbbc@googlegroups.com> On 06/30/2013 11:25 AM, Antoon Pardon wrote: > Op 28-06-13 19:20, Ian Kelly schreef: >[...] >> Flaming a troll is not punishing to them. > > I see I didn't make my point clear. This was my response to > your remark about the collective experience going back decades. > The collective experience often enough doesn't carry over wisdom > but myth. To illustrate that, I gave the example of teachers whose > collective experience is contradicted by the research. So if the > only thing you can rely on is the collective experience of the > group your knowledge isn't very relyable. I don't have anything to add to the discussion beyond restating what I've already said (which I'm not interested in doing), except to address this point in light of recent posts on the list. You claim "collective experience" is not reliable and dismiss it in favor of your own theory, flaming trolls is a better response. And your evidence for that? Nothing (that I've read so far). Collective experience may not always be totally reliable but it seems to me it is better than the non-experience that you have offered. Ironically your flame war with Nikos in http://mail.python.org/pipermail/python-list/2013-June/650905.html provides evidence for the validity of the collective experience you dismiss, that engaging in flame wars with trolls simply produces more flames, hostility begets hostility. I, and I think the majority of people here, find that very unpleasant. You have become (as predicted by the collective experience you dismiss) as offensive as any troll. From rosuav at gmail.com Mon Jul 1 10:09:04 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 2 Jul 2013 00:09:04 +1000 Subject: Don't feed the troll... In-Reply-To: <80e9fcec-ae50-4aae-b238-cb4f457efbbc@googlegroups.com> References: <7wfvwkcihf.fsf@benfinney.id.au> <51be1a42$0$29966$c3e8da3$5496439d@news.astraweb.com> <96ed14e3-79dd-40c2-8063-aa58a42c7b57@googlegroups.com> <744948f6-85dd-4de3-a886-8282ca903816@googlegroups.com> <3207a22f-846a-41a8-8165-79c25e3c4285@googlegroups.com> <51CB4513.6060201@rece.vub.ac.be> <51CC80B9.5090501@rece.vub.ac.be> <80e9fcec-ae50-4aae-b238-cb4f457efbbc@googlegroups.com> Message-ID: On Tue, Jul 2, 2013 at 12:02 AM, wrote: > ... engaging in flame wars with trolls simply produces more > flames, hostility begets hostility ... It does. Please can these threads die quietly now? ChrisA From antoon.pardon at rece.vub.ac.be Mon Jul 1 15:38:28 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Mon, 01 Jul 2013 21:38:28 +0200 Subject: Don't feed the troll... In-Reply-To: <80e9fcec-ae50-4aae-b238-cb4f457efbbc@googlegroups.com> References: <7wfvwkcihf.fsf@benfinney.id.au> <96ed14e3-79dd-40c2-8063-aa58a42c7b57@googlegroups.com> <744948f6-85dd-4de3-a886-8282ca903816@googlegroups.com> <3207a22f-846a-41a8-8165-79c25e3c4285@googlegroups.com> <51CB4513.6060201@rece.vub.ac.be> <51CC80B9.5090501@rece.vub.ac.be> <80e9fcec-ae50-4aae-b238-cb4f457efbbc@googlegroups.com> Message-ID: <51D1DAB4.1010500@rece.vub.ac.be> Op 01-07-13 16:02, rurpy at yahoo.com schreef: > You claim "collective experience" is not reliable and dismiss it > in favor of your own theory, flaming trolls is a better response. > And your evidence for that? Nothing (that I've read so far). You can decide all for yourself on how you want to handle trolls. But consider the following possibilty. A couple of trolls that are good in getting each other riled up. The regular members who mostly have killfiled them. Then who will be burdened most by the trolls? The newcomers. The regulars may succeed in creating a coccoon with a welcoming and positive atmosphere, but they have by doing so blinded themselves to how the group looks like to outsiders and so have no idea how hostile the group may have become to newbees. > Ironically your flame war with Nikos in > http://mail.python.org/pipermail/python-list/2013-June/650905.html > provides evidence for the validity of the collective experience you > dismiss, that engaging in flame wars with trolls simply produces more > flames, hostility begets hostility. I, and I think the majority of > people here, find that very unpleasant. You have become (as predicted > by the collective experience you dismiss) as offensive as any troll. A lot of unfairness stays in the world because people find it unpleasant to fight it and even to observe fighting it. -- Antoon Pardon From rurpy at yahoo.com Tue Jul 2 19:41:36 2013 From: rurpy at yahoo.com (rurpy at yahoo.com) Date: Tue, 2 Jul 2013 16:41:36 -0700 (PDT) Subject: Don't feed the troll... In-Reply-To: References: <7wfvwkcihf.fsf@benfinney.id.au> <96ed14e3-79dd-40c2-8063-aa58a42c7b57@googlegroups.com> <744948f6-85dd-4de3-a886-8282ca903816@googlegroups.com> <3207a22f-846a-41a8-8165-79c25e3c4285@googlegroups.com> <51CB4513.6060201@rece.vub.ac.be> <51CC80B9.5090501@rece.vub.ac.be> <80e9fcec-ae50-4aae-b238-cb4f457efbbc@googlegroups.com> Message-ID: On 07/01/2013 01:38 PM, Antoon Pardon wrote: > > Op 01-07-13 16:02, rurpy at yahoo.com schreef: > > >> >> You claim "collective experience" is not reliable and dismiss it >> >> in favor of your own theory, flaming trolls is a better response. >> >> And your evidence for that? Nothing (that I've read so far). > > > > You can decide all for yourself on how you want to handle trolls. > > But consider the following possibilty. A couple of trolls that > > are good in getting each other riled up. The regular members > > who mostly have killfiled them. Then who will be burdened most > > by the trolls? The newcomers. The regulars may succeed in creating > > a coccoon with a welcoming and positive atmosphere, but they > > have by doing so blinded themselves to how the group looks like > > to outsiders and so have no idea how hostile the group may have > > become to newbees. That is a possibility. But your "cocoon" senario is highly improbable because, 1. There will never be 100% compliance with any consensus here, and 2. There will always be enough backscatter and other leakage that all cocoons will be will rather permeable. If you'll recall it was that permeability, and the cost it imposes on the large majority of people here who object to it, that was an argument against your proposal, back at the start of this thread. Even allowing the possibility of two trolls mixing it up [*1], how does your proposal, to amplify the volume of hostility several times, improve the situation for your newbee? Far better to simply remind your newbee that most here feel that not feeding trolls is the most appropriate response and then demonstrate that in practice. >> >> Ironically your flame war with Nikos in >> >> http://mail.python.org/pipermail/python-list/2013-June/650905.html >> >> provides evidence for the validity of the collective experience you >> >> dismiss, that engaging in flame wars with trolls simply produces more >> >> flames, hostility begets hostility. I, and I think the majority of >> >> people here, find that very unpleasant. You have become (as predicted >> >> by the collective experience you dismiss) as offensive as any troll. > > > > A lot of unfairness stays in the world because people find it unpleasant > > to fight it and even to observe fighting it. And a lot of misery is caused by people enforcing their idea of "right". ---- [*1] You and Nikos in the "python adds an extra half space..." thread might be an actual example of your hypothetical: two trolls intentionally provoking each other and in the process, successfully provoking a lot of emotional reaction from others. From aliencat777 at gmail.com Sat Jul 27 08:25:38 2013 From: aliencat777 at gmail.com (aliencat777 at gmail.com) Date: Sat, 27 Jul 2013 05:25:38 -0700 (PDT) Subject: My son wants me to teach him Python In-Reply-To: <575a3a4c-b99f-43b4-aa2b-84dfaf3e89b4@googlegroups.com> References: <575a3a4c-b99f-43b4-aa2b-84dfaf3e89b4@googlegroups.com> Message-ID: <1c057f03-e11e-476c-8aeb-6d05c569fa80@googlegroups.com> Hi, I am an IT and Learning Research professor. I wrote a set of lessons that became a beginning programming book for my two sons. They loved it because is it was simple, hands on, and funny. It covers the basics of programming, introducing; software design, planning a game, making/getting free assets, version systems, and packaging. This course takes the beginner from ground zero to making arcade style games complete with sound, music, graphics, and an installation package in 21 lessons. It also includes vast resources in the index. It is called; "Start Here: Python 3x Programming Made Fun and Easier," and can be found at http://www.quantumsight.mobi or http://www.toonzcat.com/book.html This site also introduces graphics for web sites, animation, and games. J.S.G, PhD From robert.kern at gmail.com Mon Jul 1 04:45:52 2013 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 01 Jul 2013 09:45:52 +0100 Subject: indexerror: list index out of range?? In-Reply-To: References: <8b12635e-c3e9-49b2-b3fa-792559088821@googlegroups.com> <51CEE4D7.2010709@davea.name> Message-ID: On 2013-06-29 16:52, Joshua Landau wrote: > On 29 June 2013 15:30, Mark Lawrence wrote: >> >> On 29/06/2013 14:44, Dave Angel wrote: >>> >>> Since you're using the arrogant and buggy GoogleGroups, this >>> http://wiki.python.org/moin/GoogleGroupsPython. >>> >> Please don't make comments like this, you'll upset the Python Mailing List >> Police. > > *doesn't understand* Mark frequently makes similar comments (in content and tone) to people who come here using Google Groups. Presumably, he has received criticism for this (mostly on tone grounds, I imagine), either on or off list. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From steve+comp.lang.python at pearwood.info Mon Jul 1 08:30:50 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 01 Jul 2013 12:30:50 GMT Subject: indexerror: list index out of range?? References: <8b12635e-c3e9-49b2-b3fa-792559088821@googlegroups.com> <51CEE4D7.2010709@davea.name> Message-ID: <51d1767a$0$29999$c3e8da3$5496439d@news.astraweb.com> On Mon, 01 Jul 2013 09:45:52 +0100, Robert Kern wrote: > On 2013-06-29 16:52, Joshua Landau wrote: >> On 29 June 2013 15:30, Mark Lawrence wrote: >>> >>> On 29/06/2013 14:44, Dave Angel wrote: >>>> >>>> Since you're using the arrogant and buggy GoogleGroups, this >>>> http://wiki.python.org/moin/GoogleGroupsPython. >>>> >>> Please don't make comments like this, you'll upset the Python Mailing >>> List Police. >> >> *doesn't understand* > > Mark frequently makes similar comments (in content and tone) to people > who come here using Google Groups. Presumably, he has received criticism > for this (mostly on tone grounds, I imagine), either on or off list. That would be me, and it wasn't about Google Groups. After Mark was repeatedly abusive towards somebody, including telling this person to kill themselves, and after he chose to ignore both polite requests to cease flaming, and a fair warning that he would be kill-filed if he persisted, he persisted and so I kill-filed him. Which is a pity, because Mark does know Python well and he can be very helpful (although sometimes abrasive). I think on balance, when he's not flaming, he makes a positive contribution to this community. The kill-filing is only temporary, and I look forward to seeing his posts again soon. Hopefully by that time, he'll stop feeling put-out and hurt that I've chosen to not read his flames, and cease misrepresenting my actions. -- Steven From antoon.pardon at rece.vub.ac.be Mon Jul 1 02:37:13 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Mon, 01 Jul 2013 08:37:13 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: Message-ID: <51D12399.3000004@rece.vub.ac.be> Op 30-06-13 23:57, Joshua Landau schreef: > On 30 June 2013 20:58, Robert Kern wrote: >> On 2013-06-30 18:24, ????? wrote: >>> >>> ???? 29/6/2013 8:00 ??, ?/? Mark Lawrence ??????: >>> >>>> Why this when the approach to Nick the Incompetant Greek has been to >>>> roll out the red carpet? >>> >>> >>> Your mother is incompetent who raised a brat like you. >> >> >> That is not acceptable behavior on this list. Please keep the gratuitous >> insults offlist. > > As much as you are right, this argument was started by Mark. If you > reprimand anyone (other threads being ignored) it should be him. > Reacting only to Nick, even though what Nick said was undue, implies > that you agree with Mark's actions. I don't agree Mark started this argument. That he gave a negative appraisal of Nikos is not starting an argument in itself. > Remember that Nick is as much a human as all of us, he is bound to > have his feelings hurt when so many people pick on him -- whether they > are justified or not. So? Should we particularly care about Nikos's feelings? Nikos is not the victim, he is the instigator. And should his feelings get hurt when it is pointed out what picture people got from him through his own behaviour, I say good. May that way he'll learn that if he doesn't want to be considered an incompetent inconsiderate jerk, he shouldn't behave like one. -- Antoon Pardon From nikos at superhost.gr Mon Jul 1 03:55:02 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Mon, 01 Jul 2013 10:55:02 +0300 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: Message-ID: ???? 1/7/2013 9:37 ??, ?/? Antoon Pardon ??????: >> Remember that Nick is as much a human as all of us, he is bound to >> have his feelings hurt when so many people pick on him -- whether they >> are justified or not. > > So? Should we particularly care about Nikos's feelings? Nikos is not > the victim, he is the instigator. And should his feelings get hurt > when it is pointed out what picture people got from him through his > own behaviour, I say good. May that way he'll learn that if he doesn't > want to be considered an incompetent inconsiderate jerk, he shouldn't > behave like one. Well, i will also follow your advice and not care at all about your feeling when you read the following. GO FUCK YOURSELF. -- What is now proved was at first only imagined! From antoon.pardon at rece.vub.ac.be Mon Jul 1 04:56:54 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Mon, 01 Jul 2013 10:56:54 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: Message-ID: <51D14456.8060904@rece.vub.ac.be> Op 01-07-13 09:55, ????? schreef: > ???? 1/7/2013 9:37 ??, ?/? Antoon Pardon ??????: >>> Remember that Nick is as much a human as all of us, he is bound to >>> have his feelings hurt when so many people pick on him -- whether they >>> are justified or not. >> >> So? Should we particularly care about Nikos's feelings? Nikos is not >> the victim, he is the instigator. And should his feelings get hurt >> when it is pointed out what picture people got from him through his >> own behaviour, I say good. May that way he'll learn that if he doesn't >> want to be considered an incompetent inconsiderate jerk, he shouldn't >> behave like one. > > Well, i will also follow your advice and not care at all about your > feeling when you read the following. This makes no sense. You are not following my advice, if you were already behaving like this. > > GO FUCK YOURSELF. > No thanks, I'm getting a little old for that. -- Antoon Pardon From walterhurry at lavabit.com Mon Jul 1 10:01:18 2013 From: walterhurry at lavabit.com (Walter Hurry) Date: Mon, 1 Jul 2013 14:01:18 +0000 (UTC) Subject: python adds an extra half space when reading from a string or list References: Message-ID: On Mon, 01 Jul 2013 10:56:54 +0200, Antoon Pardon wrote: > Op 01-07-13 09:55, ????? schreef: >> ???? 1/7/2013 9:37 ??, ?/? Antoon Pardon ??????: >>>> Remember that Nick is as much a human as all of us, he is bound to >>>> have his feelings hurt when so many people pick on him -- whether >>>> they are justified or not. >>> >>> So? Should we particularly care about Nikos's feelings? Nikos is not >>> the victim, he is the instigator. And should his feelings get hurt >>> when it is pointed out what picture people got from him through his >>> own behaviour, I say good. May that way he'll learn that if he doesn't >>> want to be considered an incompetent inconsiderate jerk, he shouldn't >>> behave like one. >> >> Well, i will also follow your advice and not care at all about your >> feeling when you read the following. > > This makes no sense. You are not following my advice, if you were > already behaving like this. > > >> GO FUCK YOURSELF. >> >> > No thanks, I'm getting a little old for that. Please...enough. Polite request: consider killfiling him and having done with it. It is irritating to see all the responses even though I killfiled him long ago. Whilst I realise, of course, that it is entirely your prerogative to choose what to do, I have no doubt that many others feel as I do. From rustompmody at gmail.com Mon Jul 1 11:00:49 2013 From: rustompmody at gmail.com (rusi) Date: Mon, 1 Jul 2013 08:00:49 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: Message-ID: <965c0343-de5e-420f-bfd3-cfa951f53556@googlegroups.com> On Monday, July 1, 2013 7:31:18 PM UTC+5:30, Walter Hurry wrote: > Please...enough. Polite request: consider killfiling him and having done > with it. > > > It is irritating to see all the responses even though I killfiled him > long ago. Whilst I realise, of course, that it is entirely your > prerogative to choose what to do, I have no doubt that many others feel > as I do. Walter I would humbly suggest that you consider that the real problem here is the "many others" not speaking up. And that is at least in part happening because of the wrong use of tools -- wrong uses that you are advocating above. To wit: 1. Kill-filing/spam-filtering are tools for spam. Nikos is certainly not spamming in the sense of automated sending out of cooked mail to zillions of recipients/lists. His posts are definite and intentional 2. Likewise Nikos is not trolling. A troll is someone who posts seemingly innocuous stuff to cause irritation and then enjoys the results. Here he clearly wants answers to the questions he is asking So I suggest that if we want to address this problem we need to sit back individually and collectively, put aside habitual labels like spam/troll, and likewise habits of mind, and ask what exactly is the problem we are dealing with. From neilc at norwich.edu Mon Jul 1 11:06:53 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 1 Jul 2013 15:06:53 GMT Subject: python adds an extra half space when reading from a string or list References: <965c0343-de5e-420f-bfd3-cfa951f53556@googlegroups.com> Message-ID: On 2013-07-01, rusi wrote: > To wit: > > 1. Kill-filing/spam-filtering are tools for spam. > Nikos is certainly not spamming in the sense of automated > sending out of cooked mail to zillions of recipients/lists. > His posts are definite and intentional I disagree. Kill-files are not only for spam. I filter out anything I believe I won't want to see. -- Neil Cerutti From rustompmody at gmail.com Tue Jul 2 00:34:42 2013 From: rustompmody at gmail.com (rusi) Date: Mon, 1 Jul 2013 21:34:42 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <965c0343-de5e-420f-bfd3-cfa951f53556@googlegroups.com> Message-ID: On Monday, July 1, 2013 8:36:53 PM UTC+5:30, Neil Cerutti wrote: > On 2013-07-01, rusi wrote: > > 1. Kill-filing/spam-filtering are tools for spam. > > Nikos is certainly not spamming in the sense of automated > > sending out of cooked mail to zillions of recipients/lists. > > His posts are definite and intentional > > I disagree. Kill-files are not only for spam. I filter out > anything I believe I won't want to see. I happen to be a champ in using a wrench to hammer screws into bricks. Others do a better job I am told -- and without being champs :-) Tools can be used right... and wrong. Sufficiently extreme wrong use adds to the original problem/mess. Kill-filing is a solution for one problem. In the current context it is much more the problem than the solution. Two cases: 1. My estimate is that about 30-40 *different* answers-ers in hundreds of different posts have expressed frustration with dealing with Nikos. The fact that they are now silent can mean one of two cases: a. Nikos has turned a new leaf and become a golden boy b. The protesters have opted out of the discussion/system (like so-called democracies where 20% of the populace vote and criminals come to power) You can take your pick on which case (Hint: Read this thread) 2. "I am killfiling you" is bullying behavior. It is worse than useless because a. The problem cases couldn't care a hoot b. Those who could contribute usefully are shut up c. The messengers are being shot as substitute for the culprits On the whole we techies have a penchant (may I say allure?) for solving problems technically rather than socially. It works well up to a point: "Thou shall indent thy programs" is a social requirement in C and technical in python. And so we consider -- rightly -- that python advances C. However this line cannot solve all problems. Our belief that technology can solve all problems is directly linked to the impending fact that the planet may go up in smoke. In short technology in its place is good but conscience is not substitutable: the use of (tools like) kill-filing analogizes to this piece of medical logic: Brain tumor causes headache Aspirin cures headache Therefore aspirin cures brain tumors tl;dr Nikos is not spamming/trolling/baiting/flaming. The Nikos-threads are a more serious disease than all these. The fact that this thread is not started by him suggests the disease is spreading. By not speaking up one is colluding. From joshua.landau.ws at gmail.com Tue Jul 2 01:32:24 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Tue, 2 Jul 2013 06:32:24 +0100 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <965c0343-de5e-420f-bfd3-cfa951f53556@googlegroups.com> Message-ID: On 2 July 2013 05:34, rusi wrote: > On Monday, July 1, 2013 8:36:53 PM UTC+5:30, Neil Cerutti wrote: >> On 2013-07-01, rusi wrote: >> > 1. Kill-filing/spam-filtering are tools for spam. >> > Nikos is certainly not spamming in the sense of automated >> > sending out of cooked mail to zillions of recipients/lists. >> > His posts are definite and intentional >> >> I disagree. Kill-files are not only for spam. I filter out >> anything I believe I won't want to see. > > I happen to be a champ in using a wrench to hammer screws into bricks. > Others do a better job I am told -- and without being champs :-) That's not really valid, your proposed technique is "use nothing", in which case using a wrench is a golden opportunity. And you've also implied that filtering out what you don't want is akin to hammering screws into bricks, in which case I've been programming wrong my whole programming life. > Tools can be used right... and wrong. Sufficiently extreme wrong use adds to the original problem/mess. > > Kill-filing is a solution for one problem. In the current context it is much more the problem than the solution. Two cases: True, kill-filing *is* a solution for one problem. Don't get ahead of yourself, though, by assuming that implies it's not a solution for anything else. > 1. My estimate is that about 30-40 *different* answers-ers in hundreds of different posts have expressed frustration with dealing with Nikos. The fact that they are now silent can mean one of two cases: I'm struggling to make those numbers add up. > a. Nikos has turned a new leaf and become a golden boy > b. The protesters have opted out of the discussion/system > (like so-called democracies where 20% of the populace vote and criminals come to power) First of all, the basis on which you claim there are only two cases is false. Hence this point is void. Secondly: wut? How is any of this a relevant point in the least? > You can take your pick on which case (Hint: Read this thread) > > 2. "I am killfiling you" is bullying behavior. It is worse than useless because No it is not. You can check the definition of "bully" if you want. Please try not to overuse emotionally rigged terms. > a. The problem cases couldn't care a hoot Nikos has shown that he does, in fact, "care a hoot". > b. Those who could contribute usefully are shut up You wouldn't killfire someone whose posts you want to read. > c. The messengers are being shot as substitute for the culprits If what the messengers are talking unacceptably for your tastes, then by all means you should have right not to listen. Is that not just basic reasoning? People aren't kill-firing because the "messengers" are telling truths we don't want to hear. People are kill-firing because they're doing it in ways that we don't find acceptable. > On the whole we techies have a penchant (may I say allure?) for solving problems technically rather than socially. > It works well up to a point: > "Thou shall indent thy programs" is a social requirement in C and technical in python. And so we consider -- rightly -- that python advances C. > > However this line cannot solve all problems. Our belief that technology can solve all problems is directly linked to the impending fact that the planet may go up in smoke. Wut? > In short technology in its place is good but conscience is not substitutable: > the use of (tools like) kill-filing analogizes to this piece of medical logic: > > Brain tumor causes headache > Aspirin cures headache > Therefore aspirin cures brain tumors I see no connection. No connection to what this is supposedly an "in short" of (despite being longer) and no connection to the actual point you are trying to make. Basically, I don't understand what you're trying to portray. > tl;dr > > Nikos is not spamming/trolling/baiting/flaming. The Nikos-threads are a more serious disease than all these. The fact that this thread is not started by him suggests the disease is spreading. By not speaking up one is colluding. You might want to look up colluding. Basically, rusi, I don't understand you. You seem to speak coherently, but I never know what point you are making, what side you are arguing. You seem to be strongly anti-Nikos, but further than that is opaque to me. What I'm not too fond of, though, is your seeming insistence that your methodology should be rigorously followed by every other party on this list -- we are not a dictatorship, least of all one of your control. As long as we all act civil, and occasionally on-topic, what you do is at your discretion. Plus it's hard to follow someone's command if you haven't the least idea what they're telling you to do. Actually, the one other thing I get from you is that you have the militant view of "we have to all act now in unison to do ". It might explain your odd choice of links. From steve+comp.lang.python at pearwood.info Tue Jul 2 03:14:42 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 02 Jul 2013 07:14:42 GMT Subject: python adds an extra half space when reading from a string or list References: <965c0343-de5e-420f-bfd3-cfa951f53556@googlegroups.com> Message-ID: <51d27de2$0$29973$c3e8da3$5496439d@news.astraweb.com> On Mon, 01 Jul 2013 21:34:42 -0700, rusi wrote: > 2. "I am killfiling you" is bullying behavior. It is worse than useless > because a. The problem cases couldn't care a hoot b. Those who could > contribute usefully are shut up c. The messengers are being shot as > substitute for the culprits I don't accept this analysis. Withholding my attention is not bullying. I have no responsibility towards people asking questions here, apart from the ethical responsibility to not maliciously give them bad advice. I can come or go as I see fit, I can ignore those whom I so choose. If I were to abuse this right, say by refusing to answer questions asked by women, that would make me a dirty sexist bigot, but it wouldn't make me a bully, any more than taking a week off and not responding during that time makes me a bully. If I choose to ignore those who (in my opinion) are not living up to the implied social contract ("don't be a dick, and I'll help you, if I can"), that's hardly bulling either. Kill-filing is just a version of shunning. Like shunning in Real Life, I can do it for good reasons or bad. If I kill-file people because they said they preferred Ruby to Python, that would make me a dick, but if I kill-file people who disrupt the community, and do so publicly, I'm sending a signal to them that "your behaviour is unacceptable to me". Provided enough people follow, shunning is an effective way to discourage disruptive behaviour. Trolls will get bored when they no longer get a response, and move on. Those actually wanting help will either get frustrated and move on, or mend their ways. Kill-filing is not perfect, of course, but until such time that we can deliver a swift kick to the behind over the internet, it is the best we can do. Oh, and one last point -- I have never kill-filed anyone merely for being the messenger that another person is causing trouble, as you suggest. I have kill-filed people for being abusive, for flaming, or for trolling. -- Steven From walterhurry at lavabit.com Tue Jul 2 15:06:01 2013 From: walterhurry at lavabit.com (Walter Hurry) Date: Tue, 2 Jul 2013 19:06:01 +0000 (UTC) Subject: python adds an extra half space when reading from a string or list References: < mailman.3999.1372525218.3114.python-list@python.org> < CAN1F8qUKFFVXy9EJwPD0kZt+XVJ4dNPn7xAUpqyr4dQDrBmPHA@mail.gmail.com> < mailman.4050.1372660636.3114.python-list@python.org> < kqs23e$j9g$1@news.albasani.net> <965c0343-de5e-420f-bfd3-cfa951f53556@ googlegroups.com> < ba6b6baf-f922-41f1-aa40-da2bdf5e8056@googlegroups.com> < 51d27de2$0$29973$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, 02 Jul 2013 07:14:42 +0000, Steven D'Aprano wrote: > On Mon, 01 Jul 2013 21:34:42 -0700, rusi wrote: > >> 2. "I am killfiling you" is bullying behavior. It is worse than >> useless because a. The problem cases couldn't care a hoot b. Those who >> could contribute usefully are shut up c. The messengers are being shot >> as substitute for the culprits > > I don't accept this analysis. Withholding my attention is not bullying. > I have no responsibility towards people asking questions here, apart > from the ethical responsibility to not maliciously give them bad advice. > I can come or go as I see fit, I can ignore those whom I so choose. If I > were to abuse this right, say by refusing to answer questions asked by > women, that would make me a dirty sexist bigot, but it wouldn't make me > a bully, > any more than taking a week off and not responding during that time > makes me a bully. > > If I choose to ignore those who (in my opinion) are not living up to the > implied social contract ("don't be a dick, and I'll help you, if I > can"), > that's hardly bulling either. > > Kill-filing is just a version of shunning. Like shunning in Real Life, I > can do it for good reasons or bad. If I kill-file people because they > said they preferred Ruby to Python, that would make me a dick, but if I > kill-file people who disrupt the community, and do so publicly, I'm > sending a signal to them that "your behaviour is unacceptable to me". > > Provided enough people follow, shunning is an effective way to > discourage disruptive behaviour. Trolls will get bored when they no > longer get a response, and move on. Those actually wanting help will > either get frustrated and move on, or mend their ways. > > Kill-filing is not perfect, of course, but until such time that we can > deliver a swift kick to the behind over the internet, it is the best we > can do. > > Oh, and one last point -- I have never kill-filed anyone merely for > being the messenger that another person is causing trouble, as you > suggest. I have kill-filed people for being abusive, for flaming, or for > trolling. For me it's even simpler. I killfile people because I don't want to read what they post. FWIW, there are only four entries in my killfile for comp.lang.python, and three of them are you-know-who. From nikos at superhost.gr Mon Jul 1 12:33:37 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Mon, 01 Jul 2013 19:33:37 +0300 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <965c0343-de5e-420f-bfd3-cfa951f53556@googlegroups.com> References: <965c0343-de5e-420f-bfd3-cfa951f53556@googlegroups.com> Message-ID: ???? 1/7/2013 6:00 ??, ?/? rusi ??????: > On Monday, July 1, 2013 7:31:18 PM UTC+5:30, Walter Hurry wrote: >> Please...enough. Polite request: consider killfiling him and having done >> with it. >> >> >> It is irritating to see all the responses even though I killfiled him >> long ago. Whilst I realise, of course, that it is entirely your >> prerogative to choose what to do, I have no doubt that many others feel >> as I do. > > Walter I would humbly suggest that you consider that the real problem here is the "many others" not speaking up. > > And that is at least in part happening because of the wrong use of tools -- wrong uses that you are advocating above. > > To wit: > > 1. Kill-filing/spam-filtering are tools for spam. > Nikos is certainly not spamming in the sense of automated sending out of cooked mail to zillions of recipients/lists. His posts are definite and intentional > > 2. Likewise Nikos is not trolling. > A troll is someone who posts seemingly innocuous stuff to cause irritation and then enjoys the results. Here he clearly wants answers to the questions he is asking > > So I suggest that if we want to address this problem we need to sit back individually and collectively, put aside habitual labels like spam/troll, and likewise habits of mind, and ask what exactly is the problem we are dealing with. It feels good to see that some people in this list understand that i all i wanted was to only receive helpful replies, because i was and still am a newbie at Python and it was not a trolling act as falsely considered.. -- What is now proved was at first only imagined! From antoon.pardon at rece.vub.ac.be Mon Jul 1 14:42:48 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Mon, 01 Jul 2013 20:42:48 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: Message-ID: <51D1CDA8.6050002@rece.vub.ac.be> Op 01-07-13 16:01, Walter Hurry schreef: > On Mon, 01 Jul 2013 10:56:54 +0200, Antoon Pardon wrote: > >> Op 01-07-13 09:55, ????? schreef: >>> ???? 1/7/2013 9:37 ??, ?/? Antoon Pardon ??????: >>>>> Remember that Nick is as much a human as all of us, he is bound to >>>>> have his feelings hurt when so many people pick on him -- whether >>>>> they are justified or not. >>>> >>>> So? Should we particularly care about Nikos's feelings? Nikos is not >>>> the victim, he is the instigator. And should his feelings get hurt >>>> when it is pointed out what picture people got from him through his >>>> own behaviour, I say good. May that way he'll learn that if he doesn't >>>> want to be considered an incompetent inconsiderate jerk, he shouldn't >>>> behave like one. >>> >>> Well, i will also follow your advice and not care at all about your >>> feeling when you read the following. >> >> This makes no sense. You are not following my advice, if you were >> already behaving like this. >> >> >>> GO FUCK YOURSELF. >>> >>> >> No thanks, I'm getting a little old for that. > > Please...enough. Polite request: consider killfiling him and having done > with it. > > It is irritating to see all the responses even though I killfiled him > long ago. Whilst I realise, of course, that it is entirely your > prerogative to choose what to do, I have no doubt that many others feel > as I do. > How about the following comprimise. I'll get myself a second identity. Every respons I make to Nikos will be done with the same identity. Normal python exchanges will be done with the other. You can then simply killfile my identity that engages with Nikos. Let me know what you think about this. -- Antoon Pardon From steve+comp.lang.python at pearwood.info Mon Jul 1 18:12:15 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 01 Jul 2013 22:12:15 GMT Subject: python adds an extra half space when reading from a string or list References: Message-ID: <51d1febf$0$29973$c3e8da3$5496439d@news.astraweb.com> On Mon, 01 Jul 2013 20:42:48 +0200, Antoon Pardon wrote: > How about the following comprimise. I'll get myself a second identity. > Every respons I make to Nikos will be done with the same identity. > Normal python exchanges will be done with the other. You can then simply > killfile my identity that engages with Nikos. "Please don't assault people in the street." "How about a compromise? I'll continue assaulting people in the street, but I'll wear a yellow hat when I do so. Then, if you don't want to see me assaulting people, whenever you see me wearing a yellow hat, just turn away." > Let me know what you think about this. Sure, no problem. *plonk* -- Steven From antoon.pardon at rece.vub.ac.be Tue Jul 2 03:55:53 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Tue, 02 Jul 2013 09:55:53 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <51d1febf$0$29973$c3e8da3$5496439d@news.astraweb.com> References: <51d1febf$0$29973$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51D28789.6080701@rece.vub.ac.be> Op 02-07-13 00:12, Steven D'Aprano schreef: > On Mon, 01 Jul 2013 20:42:48 +0200, Antoon Pardon wrote: > >> How about the following comprimise. I'll get myself a second identity. >> Every respons I make to Nikos will be done with the same identity. >> Normal python exchanges will be done with the other. You can then simply >> killfile my identity that engages with Nikos. > "Please don't assault people in the street." > > "How about a compromise? I'll continue assaulting people in the street, > but I'll wear a yellow hat when I do so. Then, if you don't want to see > me assaulting people, whenever you see me wearing a yellow hat, just turn > away." If you want to make an analogy you shouldn't cherry pick part of the situation and ignore how awful it would look in the complete picture. First of all these kind of requests were made to you to. They are requests from people who have killfiled Nikos but still are confronted with him because of those who reply to him. So if I am an assaulter then so are you in the context of this analogy. And what kind of solution is the killfile in this analogy? Perhaps this: ] Chris: Fred is really behaving very annoyingly ] Nils: You should wear protective gear so you can ignore him but don't bother the rest of us about it. >> Let me know what you think about this. > Sure, no problem. > > *plonk* Fine you give me an opportunity to point out the problems in your thinking without giving yourself the opportunity to respond. I can live with that. -- Antoon Pardon. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Jul 2 03:59:09 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 2 Jul 2013 17:59:09 +1000 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <51D28789.6080701@rece.vub.ac.be> References: <51d1febf$0$29973$c3e8da3$5496439d@news.astraweb.com> <51D28789.6080701@rece.vub.ac.be> Message-ID: On Tue, Jul 2, 2013 at 5:55 PM, Antoon Pardon wrote: > Fine you give me an opportunity to point out the problems in your thinking > without giving yourself the opportunity to respond. I can live with that. And by continuing to rant, you just make other people sick of reading your posts. Please, do us all a favour and let's get back to talking Python rather than people. ChrisA From denismfmcmahon at gmail.com Mon Jul 1 19:17:11 2013 From: denismfmcmahon at gmail.com (Denis McMahon) Date: Mon, 1 Jul 2013 23:17:11 +0000 (UTC) Subject: python adds an extra half space when reading from a string or list References: Message-ID: On Mon, 01 Jul 2013 20:42:48 +0200, Antoon Pardon wrote: > How about the following comprimise. .... > Let me know what you think about this. How about you just ignore him. -- Denis McMahon, denismfmcmahon at gmail.com From antoon.pardon at rece.vub.ac.be Tue Jul 2 02:54:18 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Tue, 02 Jul 2013 08:54:18 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: Message-ID: <51D2791A.3080709@rece.vub.ac.be> Op 02-07-13 01:17, Denis McMahon schreef: > On Mon, 01 Jul 2013 20:42:48 +0200, Antoon Pardon wrote: > >> How about the following comprimise. .... >> Let me know what you think about this. > How about you just ignore him. Why about you just ignore me? Serious, a lot of people here seem to think that if someone is bothered by someone else, the bothered person should just ignore the botherer. That is until something starts bothering these people themselves, then suddenly all kind of arguments are let loose in the hope of influencing the person that bothered them in changing his behaviour. -- Antoon Pardon From ian.g.kelly at gmail.com Tue Jul 2 03:43:09 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Tue, 2 Jul 2013 01:43:09 -0600 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <51D2791A.3080709@rece.vub.ac.be> References: <51D2791A.3080709@rece.vub.ac.be> Message-ID: On Tue, Jul 2, 2013 at 12:54 AM, Antoon Pardon wrote: > Why about you just ignore me? > > Serious, a lot of people here seem to think that if someone is bothered > by someone else, the bothered person should just ignore the botherer. > > That is until something starts bothering these people themselves, then > suddenly all kind of arguments are let loose in the hope of influencing > the person that bothered them in changing his behaviour. So what if they do? Failure to follow one's own advice does not make it bad advice. From antoon.pardon at rece.vub.ac.be Tue Jul 2 04:16:37 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Tue, 02 Jul 2013 10:16:37 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51D2791A.3080709@rece.vub.ac.be> Message-ID: <51D28C65.1070200@rece.vub.ac.be> Op 02-07-13 09:43, Ian Kelly schreef: > On Tue, Jul 2, 2013 at 12:54 AM, Antoon Pardon > wrote: >> Why about you just ignore me? >> >> Serious, a lot of people here seem to think that if someone is bothered >> by someone else, the bothered person should just ignore the botherer. >> >> That is until something starts bothering these people themselves, then >> suddenly all kind of arguments are let loose in the hope of influencing >> the person that bothered them in changing his behaviour. > So what if they do? Failure to follow one's own advice does not make > it bad advice. That is true. But it should give some appreciation on how difficult it can be to follow the advise. IMO "ignore the botherer" is bad advice in the same sense as telling people who are overweight they should just eat less and exercise more. Sure if overweight people would follow the advise they actually would loose weight, but the advise ignores how hard it can be for people to change their behaviour. People can't just decide they will behave a certain way in the future. There can be all kinds of hurdles that will tempt them to resume the original behaviour. I would like people to have that in mind when they try to tell others to just ignore the behaviour that bothers them. Maybe it would help them to be a bit more patient to those who have trouble following the advise. -- Antoon Pardon From denismfmcmahon at gmail.com Tue Jul 2 08:53:53 2013 From: denismfmcmahon at gmail.com (Denis McMahon) Date: Tue, 2 Jul 2013 12:53:53 +0000 (UTC) Subject: python adds an extra half space when reading from a string or list References: <51D2791A.3080709@rece.vub.ac.be> Message-ID: On Tue, 02 Jul 2013 10:16:37 +0200, Antoon Pardon wrote: > I would like people to have that in mind when they try to tell others to > just ignore the behaviour that bothers them. Maybe it would help them to > be a bit more patient to those who have trouble following the advise. It's quite clear that you're more interested in having arguments and slinging insults than you are in discussing python. I'm here to discuss python. -- Denis McMahon, denismfmcmahon at gmail.com From steve+comp.lang.python at pearwood.info Tue Jul 2 20:22:00 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 03 Jul 2013 00:22:00 GMT Subject: python adds an extra half space when reading from a string or list References: <51D2791A.3080709@rece.vub.ac.be> Message-ID: <51d36ea8$0$29999$c3e8da3$5496439d@news.astraweb.com> On Tue, 02 Jul 2013 12:53:53 +0000, Denis McMahon wrote: > On Tue, 02 Jul 2013 10:16:37 +0200, Antoon Pardon wrote: > >> I would like people to have that in mind when they try to tell others >> to just ignore the behaviour that bothers them. Maybe it would help >> them to be a bit more patient to those who have trouble following the >> advise. > > It's quite clear that you're more interested in having arguments and > slinging insults than you are in discussing python. I'm here to discuss > python. Well said. -- Steven From antoon.pardon at rece.vub.ac.be Mon Jul 1 02:23:43 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Mon, 01 Jul 2013 08:23:43 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: Message-ID: <51D1206F.4020706@rece.vub.ac.be> Op 30-06-13 22:14, ????? schreef: > ???? 30/6/2013 10:58 ??, ?/? Robert Kern ??????: >> On 2013-06-30 18:24, ????? wrote: >>> ???? 29/6/2013 8:00 ??, ?/? Mark Lawrence ??????: >>> >>>> Why this when the approach to Nick the Incompetant Greek has been to >>>> roll out the red carpet? >>> >>> Your mother is incompetent who raised a brat like you. >> >> That is not acceptable behavior on this list. Please keep the gratuitous >> insults offlist. >> > ?'m sorry but please put yourself into my shows, Why should we? You, yourself have shown no willingness to put yourself in the shoes of others. When it was pointed out your behaviour was unacceptable, you continued just the same with no consideration of how much you were antagonising the members of this group. > where in multiple > threads i'm still discussed and being insulted as a troll and as an > idiot and incompetent and stuff like that by lots of people who instead > of actually replying to my posts they deviate and have endless > conversation about me. What did you expect? That you could just keep asking to be helped in obnoxious ways without people commenting on that? > Enough is enough. Iam not a troll, neither incompetent. Period. No not period. You have by your behaviour made yourself a reputation of being an incompetent inconsiderate jerk. You don't lose such a repuation by simply claiming you are not. -- Antoon Pardon From nikos at superhost.gr Mon Jul 1 03:52:28 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Mon, 01 Jul 2013 10:52:28 +0300 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: Message-ID: ???? 1/7/2013 9:23 ??, ?/? Antoon Pardon ??????: >> Enough is enough. Iam not a troll, neither incompetent. Period. > > No not period. You have by your behaviour made yourself a reputation > of being an incompetent inconsiderate jerk. You don't lose such a > repuation by simply claiming you are not. Being a newbie doesn't mean that i'm incompetent jerk. All i did was asking for help for issues i couldn't solve. That what everybody would do if he couldn't help himself solve something. You on the other hand never care to reply to any of my questions(so it was in someway justified for you to say i didn't follow your advice), but instead you seem to be willing to judge endlessly for the help i receive from others. You are continuously spamming the list and my threads with arguments of trolling or not trolling instead of being helpful. I as well as others are fed up seeing you post about me. Its over 2 weeks now and you still fight of whether i'm trolling or not. So shut your piehole and start proving yourself useful in this list. Or sod off. Preferably do the latter. -- What is now proved was at first only imagined! From antoon.pardon at rece.vub.ac.be Mon Jul 1 04:54:21 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Mon, 01 Jul 2013 10:54:21 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: Message-ID: <51D143BD.8030408@rece.vub.ac.be> Op 01-07-13 09:52, ????? schreef: > ???? 1/7/2013 9:23 ??, ?/? Antoon Pardon ??????: >>> Enough is enough. Iam not a troll, neither incompetent. Period. >> >> No not period. You have by your behaviour made yourself a reputation >> of being an incompetent inconsiderate jerk. You don't lose such a >> repuation by simply claiming you are not. > > Being a newbie doesn't mean that i'm incompetent jerk. That is true. You'll notice I don't judge all newbies as incompetent jerks. Just you > > All i did was asking for help for issues i couldn't solve. > That what everybody would do if he couldn't help himself solve something. You can try to frame what you did as an innocent newbie asking questions but that is not what happened. What happened is that you started a project on which other people relied that was way over your head. That makes you more than just a newbie that makes you incompetent. When you then started to get into trouble you didn't just ask questions, you almost demanded that people would fix your problems for you, bumping a thread within a few hours if you didn't get an answer within that time. That and similar behaviour makes you a jerk. > > You on the other hand never care to reply to any of my questions(so it > was in someway justified for you to say i didn't follow your advice), > but instead you seem to be willing to judge endlessly for the help i > receive from others. I am not judging you for the you receive. I am judging you for behaving liken an asshole and trying to deny it. > > You are continuously spamming the list and my threads with arguments > of trolling or not trolling instead of being helpful. > > I as well as others are fed up seeing you post about me. Its over 2 > weeks now and you still fight of whether i'm trolling or not. And it probably becomes longer. If you don't want this to fester, you'd better stop denying that you were behaving like an incompetent inconsiderate jerk. People can forget and forgive jerky behaviour but they don't tend to when they see the other denying he was behaving like a jerk. Because they then expect more of that kind of behaviour. > So shut your piehole and start proving yourself useful in this list. > Or sod off. > Preferably do the latter. Oh we do have illusions of grandeur, don't we? You are in no position to judge who is useful on this list or not. -- Antoon Pardon From nikos at superhost.gr Mon Jul 1 05:05:46 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Mon, 01 Jul 2013 12:05:46 +0300 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: Message-ID: ???? 1/7/2013 11:54 ??, ?/? Antoon Pardon ??????: >> So shut your piehole and start proving yourself useful in this list. >> Or sod off. >> Preferably do the latter. > Oh we do have illusions of grandeur, don't we? You are in no position > to judge who is useful on this list or not. I'am not waste any more time explaining my self to you since you have already made up your mind and you refuse to believe that ni was actually trying to solve my problems first and then asked and then tried to apply the solutions, which in my my 2 last problems i was the one who actually solve the problem. Its easier to say to you "fuck off retard". -- What is now proved was at first only imagined! From antoon.pardon at rece.vub.ac.be Mon Jul 1 05:32:41 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Mon, 01 Jul 2013 11:32:41 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: Message-ID: <51D14CB9.1000709@rece.vub.ac.be> Op 01-07-13 11:05, ????? schreef: > ???? 1/7/2013 11:54 ??, ?/? Antoon Pardon ??????: >>> So shut your piehole and start proving yourself useful in this list. >>> Or sod off. >>> Preferably do the latter. >> Oh we do have illusions of grandeur, don't we? You are in no position >> to judge who is useful on this list or not. > > I'am not waste any more time explaining my self to you since you have > already made up your mind and you refuse to believe that ni was > actually trying to solve my problems first and then asked and then > tried to apply the solutions, which in my my 2 last problems i was the > one who actually solve the problem. The above is a first class illustration of why I think you are problematic. We don't want you to explain yourself, we want you to change your behaviour. Someone behaving like a jerk explaining himself is still behaving like a jerk and the explanation is just a method for avoiding changing the behaviour. It doesn't matter that you tried to solve your problem first. Accepting that you were, you were still acting like a jerk when you bumped the thread after only a few hours. > Its easier to say to you "fuck off retard". Indeed that is much easier than trying to change your behaviour in order to no longer behave like a jerk. -- Antoon Pardon From square.steve at gmail.com Mon Jul 1 05:31:31 2013 From: square.steve at gmail.com (Steve Simmons) Date: Mon, 01 Jul 2013 10:31:31 +0100 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: Message-ID: <216b6cf4-2cf8-4b8a-8692-83d70f072d54@email.android.com> "?????" wrote: >???? 1/7/2013 11:54 ??, ?/? Antoon Pardon ??????: >>> So shut your piehole and start proving yourself useful in this list. >>> Or sod off. >>> Preferably do the latter. >> Oh we do have illusions of grandeur, don't we? You are in no position >> to judge who is useful on this list or not. > >I'am not waste any more time explaining my self to you since you have >already made up your mind and you refuse to believe that ni was >actually >trying to solve my problems first and then asked and then tried to >apply >the solutions, which in my my 2 last problems i was the one who >actually >solve the problem. > >Its easier to say to you "fuck off retard". >-- >What is now proved was at first only imagined! >-- >http://mail.python.org/mailman/listinfo/python-list I don't know about the other members of this list but I am becoming increasingly disturbed by the rudeness and especially the foul language that is being perpetrated on this thread. Please, if you have any decency at all, continue the rest of this exchange by private email and allow python list members to continue discussing and helping each other with the python language and is adjuncts. Leave us in peace to enjoy our favoured language and let us see no more of your filth. Sent from a Galaxy far far away -------------- next part -------------- An HTML attachment was scrubbed... URL: From nikos at superhost.gr Mon Jul 1 05:37:56 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Mon, 01 Jul 2013 12:37:56 +0300 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: Message-ID: ???? 1/7/2013 12:32 ??, ?/? Antoon Pardon ??????: > Op 01-07-13 11:05, ????? schreef: >> ???? 1/7/2013 11:54 ??, ?/? Antoon Pardon ??????: >>>> So shut your piehole and start proving yourself useful in this list. >>>> Or sod off. >>>> Preferably do the latter. >>> Oh we do have illusions of grandeur, don't we? You are in no position >>> to judge who is useful on this list or not. >> >> I'am not waste any more time explaining my self to you since you have >> already made up your mind and you refuse to believe that ni was >> actually trying to solve my problems first and then asked and then >> tried to apply the solutions, which in my my 2 last problems i was the >> one who actually solve the problem. > > The above is a first class illustration of why I think you are > problematic. We don't want you to explain yourself, we want > you to change your behaviour. Someone behaving like a jerk > explaining himself is still behaving like a jerk and the > explanation is just a method for avoiding changing the > behaviour. Change what dickhead? What behavior? The behavior which you think its problematic? Maybe you should change your perception of understanding things. I dont expect anyone to solve my problems, iam happy if they point me to a direction which i can solve it myself, that what the list is about. > It doesn't matter that you tried to solve your problem first. > Accepting that you were, you were still acting like a jerk > when you bumped the thread after only a few hours. > >> Its easier to say to you "fuck off retard". > > Indeed that is much easier than trying to change your behaviour > in order to no longer behave like a jerk. Did i told you to fuck off? Well never mind. Fuck off! -- What is now proved was at first only imagined! From antoon.pardon at rece.vub.ac.be Mon Jul 1 08:28:02 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Mon, 01 Jul 2013 14:28:02 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: Message-ID: <51D175D2.3080401@rece.vub.ac.be> Op 01-07-13 11:37, ????? schreef: > ???? 1/7/2013 12:32 ??, ?/? Antoon Pardon ??????: >> >> The above is a first class illustration of why I think you are >> problematic. We don't want you to explain yourself, we want >> you to change your behaviour. Someone behaving like a jerk >> explaining himself is still behaving like a jerk and the >> explanation is just a method for avoiding changing the >> behaviour. > > Change what dickhead? What behavior? > The behavior which you think its problematic? You are goin to play the ignorance card now? You are going to pretend it is just me and ignore all those others who made remarks about particular behaviour of you being annoying and unacceptable. I gave a specific answer, you bumping threads are only a few hours. You could have acknowledge that indeed you have been doing that and that you understand that it is annoying to others and you could have made it clear you intended to be more patient in the future. That could have been the beginning of you gaining again some good will from members of the group. Instead you chose to be a jerk about it. Pretending that the multiple remarks you have received about this behaviour didn't happen. > Maybe you should change your perception of understanding things. Why should I. From the people with whom I discussed your behaviour, a lot disputed you were a troll, but none disputed you were behaving like an asshole. Yes even those who were willing to further help you agreed that your behaviour was problematic. > > I dont expect anyone to solve my problems, iam happy if they point me > to a direction which i can solve it myself, that what the list is about. Sure, you just play the impatient jerk, who bumps threads and asks the same question again after people have pointed you in a direction. > > Did i told you to fuck off? Well never mind. Fuck off! > I think this needs some work. You need more creativity. Try to think of something more forceful. My little niece of 10 can do better than that. -- Antoon Pardon From nikos at superhost.gr Mon Jul 1 09:22:08 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Mon, 01 Jul 2013 16:22:08 +0300 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: Message-ID: ???? 1/7/2013 3:28 ??, ?/? Antoon Pardon ??????: >> Did i told you to fuck off? Well never mind. Fuck off! > I think this needs some work. You need more creativity. > Try to think of something more forceful. My little niece > of 10 can do better than that. Well i could, but i don't want to spend my time and creativitism on assholes like yourself. So, here it is again, plain and simple. FUCK OFF! -- What is now proved was at first only imagined! From wuwei23 at gmail.com Mon Jul 1 22:44:53 2013 From: wuwei23 at gmail.com (alex23) Date: Tue, 02 Jul 2013 12:44:53 +1000 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: Message-ID: On 1/07/2013 7:37 PM, ????? wrote: > I dont expect anyone to solve my problems, This is not consistent with the number of "HELP ME I BROKE MY BUSINESS" posts you've made. > iam happy if they point me to > a direction which i can solve it myself This is not consistent with your repeated claim that you prefer people to explain things to you rather than read documentation. From nikos at superhost.gr Mon Jul 1 05:46:56 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Mon, 01 Jul 2013 12:46:56 +0300 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: Message-ID: ???? 1/7/2013 12:31 ??, ?/? Steve Simmons ??????: > I don't know about the other members of this list but I am becoming > increasingly disturbed by the rudeness and especially the foul language > that is being perpetrated on this thread. Please, if you have any > decency at all, continue the rest of this exchange by private email and > allow python list members to continue discussing and helping each other > with the python language and is adjuncts. Leave us in peace to enjoy our > favoured language and let us see no more of your filth. > > Sent from a Galaxy far far away I have kept in quite more than 2 weeks while kept reading any mail in this list especially the endless conversation and numerous name callings against me. You should know that i'm fed up reading this. If i'm provoked i will respond. it is not in my intentons to troll the list or talk dirty, but also i cannot withstand anymore people talk down to me like that, when the only thing i doid was to ask some questions, which in turn helped us all. Especially i cant stand people which never to spend 5 mins to reply, but they are devoted a couple of weeks to belittle me instead. -- What is now proved was at first only imagined! From square.steve at gmail.com Mon Jul 1 06:07:45 2013 From: square.steve at gmail.com (Steve Simmons) Date: Mon, 01 Jul 2013 11:07:45 +0100 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: Message-ID: <8e96b674-9bdc-4390-9ac5-2fdddb675ed9@email.android.com> "?????" wrote: >???? 1/7/2013 12:31 ??, ?/? Steve Simmons ??????: > >> I don't know about the other members of this list but I am becoming >> increasingly disturbed by the rudeness and especially the foul >language >> that is being perpetrated on this thread. Please, if you have any >> decency at all, continue the rest of this exchange by private email >and >> allow python list members to continue discussing and helping each >other >> with the python language and is adjuncts. Leave us in peace to enjoy >our >> favoured language and let us see no more of your filth. >> >> Sent from a Galaxy far far away > >I have kept in quite more than 2 weeks while kept reading any mail in >this list especially the endless conversation and numerous name >callings >against me. > >You should know that i'm fed up reading this. >If i'm provoked i will respond. >it is not in my intentons to troll the list or talk dirty, but also i >cannot withstand anymore people talk down to me like that, when the >only >thing i doid was to ask some questions, which in turn helped us all. > >Especially i cant stand people which never to spend 5 mins to reply, >but >they are devoted a couple of weeks to belittle me instead. > >-- >What is now proved was at first only imagined! >-- >http://mail.python.org/mailman/listinfo/python-list I understand some of your frustrations, just as I understand the frustrations of those who have tried to help in spite of your sometimes poor understanding of the etiquette expected on this list. On no account do I understand or endorse your continued loutish use of bad language so I repeat my request that you move the rest of this offensive conversation onto a private channel such as email. The majority of people on this list are probably as sick of this thread as I am but choose not to challenge you. Please leave. Sent from a Galaxy far far away -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoon.pardon at rece.vub.ac.be Mon Jul 1 08:44:40 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Mon, 01 Jul 2013 14:44:40 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: Message-ID: <51D179B8.8080309@rece.vub.ac.be> Op 01-07-13 11:46, ????? schreef: > ???? 1/7/2013 12:31 ??, ?/? Steve Simmons ??????: > >> I don't know about the other members of this list but I am becoming >> increasingly disturbed by the rudeness and especially the foul language >> that is being perpetrated on this thread. Please, if you have any >> decency at all, continue the rest of this exchange by private email and >> allow python list members to continue discussing and helping each other >> with the python language and is adjuncts. Leave us in peace to enjoy our >> favoured language and let us see no more of your filth. >> >> Sent from a Galaxy far far away > > I have kept in quite more than 2 weeks while kept reading any mail in > this list especially the endless conversation and numerous name > callings against me. > You poor thing. A horrible two weeks of people who couldn't see the pearls of consideration you had shown and who were seeing you as somekind of asshole. > You should know that i'm fed up reading this. > If i'm provoked i will respond. Yes, because if you are fed up with how others behave, it is the fault of these others in provoking you. And of course when the others are fed of with how you behave, it is again the fault of the others because the have a faulty perception of understanding things. > it is not in my intentons to troll the list or talk dirty, but also i > cannot withstand anymore people talk down to me like that, when the > only thing i doid was to ask some questions, which in turn helped us all. Indeed it is not your intention to troll, you only want to ask questions. And if it turns out that your behaviour during the asking of these questions is very annoying, we should just shoulder the burden of that annoyance since it wasn't intended. Asking you to change your behaviour? That wouldn't be fair because you never intended to annoy. > Especially i cant stand people which never to spend 5 mins to reply, > but they are devoted a couple of weeks to belittle me instead. I can't stand people who seem to think that only people who have helped them can give valid criticism. Yet here you are. It seems we'll both have to find a way in coping. -- Antoon Pardon From steve+comp.lang.python at pearwood.info Mon Jul 1 08:43:00 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 01 Jul 2013 12:43:00 GMT Subject: python adds an extra half space when reading from a string or list References: Message-ID: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> On Mon, 01 Jul 2013 10:52:28 +0300, ????? wrote: > All i did was asking for help for issues i couldn't solve. That what > everybody would do if he couldn't help himself solve something. [...] > So shut your piehole and start proving yourself useful in this list. Or > sod off. > Preferably do the latter. Enough ?????. If you can't be the "bigger man" and ignore other people's insults, take your flaming off-list. I don't care if you get into the mother of all flame-fests, so long as it is OFF LIST and doesn't upset this forum. If you continue to be abusive on this list, I will kill-file you and will not even see your posts, and no doubt others will do the same. It doesn't matter who started it. It doesn't matter if you're just trying to get some help. If you continue being abusive, you will lose the few allies you have left. The same goes to the Net Vigilantes trying to drive ????? away. I can't speak for others, but I can speak for myself, and abuse and flaming has no place in this forum, and if you continue to behave abusively, I will kill-file you too. ?????, I am not going to wade through this long, long thread to see what problem you are trying to solve today. I only have a limited number of hours available, and it is unfair for you to monopolise all my time. Also, sometimes I simply do not know how to solve your technical problems. I don't know everything. But I will give you a commitment: if you follow the advice here: http://sscce.org/ and come up with a short, self-contained, runnable example of the problem you are having in this thread, I commit to reading your post and making an honest effort to solve the problem by Wednesday evening (Melbourne, Australia time). The above website is written for Java programmers, but the advice holds for any language including Python. The above of course assumes that I have not kill-filed you for continuing to be abusive on-list. -- Steven From antoon.pardon at rece.vub.ac.be Mon Jul 1 09:08:18 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Mon, 01 Jul 2013 15:08:18 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51D17F42.6040702@rece.vub.ac.be> Op 01-07-13 14:43, Steven D'Aprano schreef: > ?????, I am not going to wade through this long, long thread to see what > problem you are trying to solve today. Nikos is not trying to solve a problem in this thread. What happened is that the original poster here got a rather short answer to his question. Mark Laurence reacted questioning why this person didn't get the same kind of treatment as "Nick the Incompetant Greek". Then Nikos came in with an insulting remark. The rest developed from their. So nobody was trying to drive Nikos away (for asking questions). -- Antoon Pardon. From nikos at superhost.gr Mon Jul 1 09:28:52 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Mon, 01 Jul 2013 16:28:52 +0300 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: ???? 1/7/2013 3:43 ??, ?/? Steven D'Aprano ??????: > On Mon, 01 Jul 2013 10:52:28 +0300, ????? wrote: > >> All i did was asking for help for issues i couldn't solve. That what >> everybody would do if he couldn't help himself solve something. > [...] >> So shut your piehole and start proving yourself useful in this list. Or >> sod off. >> Preferably do the latter. > > > Enough ?????. If you can't be the "bigger man" and ignore other people's > insults, take your flaming off-list. I don't care if you get into the > mother of all flame-fests, so long as it is OFF LIST and doesn't upset > this forum. If you continue to be abusive on this list, I will kill-file > you and will not even see your posts, and no doubt others will do the > same. > > It doesn't matter who started it. It doesn't matter if you're just trying > to get some help. If you continue being abusive, you will lose the few > allies you have left. > > The same goes to the Net Vigilantes trying to drive ????? away. I can't > speak for others, but I can speak for myself, and abuse and flaming has > no place in this forum, and if you continue to behave abusively, I will > kill-file you too. > > ?????, I am not going to wade through this long, long thread to see what > problem you are trying to solve today. I only have a limited number of > hours available, and it is unfair for you to monopolise all my time. > Also, sometimes I simply do not know how to solve your technical > problems. I don't know everything. But I will give you a commitment: if > you follow the advice here: > > http://sscce.org/ > > and come up with a short, self-contained, runnable example of the problem > you are having in this thread, I commit to reading your post and making > an honest effort to solve the problem by Wednesday evening (Melbourne, > Australia time). The above website is written for Java programmers, but > the advice holds for any language including Python. > > The above of course assumes that I have not kill-filed you for continuing > to be abusive on-list. So, Steven you want me to sit tight and read all the insults coming from this guy? If that happened to you, wouldn't you feel the need and urge to reply back and stand for yourself? Almost 2 weeks now that i solved my last website problems this person amongst others that especially him is keeping this list busy talking down to me. What am i supposed to do? Listening him for ever? I use Thunderbird and i'm currently looking for a way to kill-file him, because he is not likely to stop. And no, i do not want to piss off people like you, who have spend time helping me. But i cannot stand him either. -- What is now proved was at first only imagined! From robotsondrugs at gmail.com Mon Jul 1 09:45:19 2013 From: robotsondrugs at gmail.com (Andrew Berg) Date: Mon, 01 Jul 2013 08:45:19 -0500 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51D187EF.90405@gmail.com> On 2013.07.01 08:28, ????? wrote: > So, Steven you want me to sit tight and read all the insults coming from > this guy? > > If that happened to you, wouldn't you feel the need and urge to reply > back and stand for yourself? You can ignore it (this is the best solution) or you can take it off-list. Most on this list do not consider insults and flaming on the list acceptable. It doesn't matter who started it. > I use Thunderbird and i'm currently looking for a way to kill-file him, > because he is not likely to stop. Tools -> Message Filters... -- CPython 3.3.2 | Windows NT 6.2.9200 / FreeBSD 9.1 From steve+comp.lang.python at pearwood.info Mon Jul 1 11:34:11 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 01 Jul 2013 15:34:11 GMT Subject: python adds an extra half space when reading from a string or list References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> On Mon, 01 Jul 2013 16:28:52 +0300, ????? wrote: > ???? 1/7/2013 3:43 ??, ?/? Steven D'Aprano ??????: [...] >> The above of course assumes that I have not kill-filed you for >> continuing to be abusive on-list. > > So, Steven you want me to sit tight and read all the insults coming from > this guy? > > If that happened to you, wouldn't you feel the need and urge to reply > back and stand for yourself? "Stand up for yourself" and "be abusive" are two different things. I told you, I don't care what you do off-list, but if you continue flaming on- list, I will kill-file you. > And no, i do not want to piss off people like you, who have spend time > helping me. Too late. I asked you to stop flaming on-list, and you didn't. I am now kill-filing you for a month. Feel grateful that it is not permanent, and take this time to reflect that you are running out of people willing to help you. -- Steven From rustompmody at gmail.com Mon Jul 1 11:49:58 2013 From: rustompmody at gmail.com (rusi) Date: Mon, 1 Jul 2013 08:49:58 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Monday, July 1, 2013 9:04:11 PM UTC+5:30, Steven D'Aprano wrote: > > And no, i do not want to piss off people like you, who have spend time > > helping me. > Too late. I asked you to stop flaming on-list, and you didn't. I am now > kill-filing you for a month. Feel grateful that it is not permanent, and > take this time to reflect that you are running out of people willing to > help you. > -- > > Steven A suggestion (request??) to all list members in this respect: We strengthen Steven's decision as follows: For the next month (till 1st August) - No discussions about Nikos specifically or trolling/flaming in general - No answers to anything he asks From joshua.landau.ws at gmail.com Mon Jul 1 12:56:21 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Mon, 1 Jul 2013 17:56:21 +0100 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 1 July 2013 16:49, rusi wrote: > On Monday, July 1, 2013 9:04:11 PM UTC+5:30, Steven D'Aprano wrote: >> > And no, i do not want to piss off people like you, who have spend time >> > helping me. >> Too late. I asked you to stop flaming on-list, and you didn't. I am now >> kill-filing you for a month. Feel grateful that it is not permanent, and >> take this time to reflect that you are running out of people willing to >> help you. > > A suggestion (request??) to all list members in this respect: > > We strengthen Steven's decision as follows: > > For the next month (till 1st August) > - No discussions about Nikos specifically or trolling/flaming in general > - No answers to anything he asks (This does not target you, but is a response to your suggested response) No, it's: 1) Insults and off-topic derogatory are not allowed. That's it. If you're insulted, tough. Take it off-list. If you think it's totally and unabashedly Nikos's fault, tough. Take it off-list. If you think Nikos is completely innocent, tough. Take it off-list. Insults and derogatory are not cool. I thought we were adults. If you choose to ignore Nikos's posts, fine. If you choose like me to wait until he asks questions well, fine. Whatever. That's beside the point. As before. no insults, no derogatory. If you don't get why this is not allowed, you need to step back. To put things in perspective, these are the people who have been insulting on this post: Mark Lawrence (once, probably without meaning to be insulting) Nikos Antoon Pardon And here are the people who have reminded them to stop: Steve Simmons Steven D'Aprano Andrew Berg Walter Hurry rusi Joshua Landau (me) So yes, Antoon Pardon and Nikos, please stop. You are not representing the list. I haven't followed any of the other arguments, true, but you two in particular are causing a lot of trouble for the rest of us. It is not hard to avoid making your disagreements public. From antoon.pardon at rece.vub.ac.be Mon Jul 1 15:12:04 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Mon, 01 Jul 2013 21:12:04 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51D1D484.5070309@rece.vub.ac.be> Op 01-07-13 18:56, Joshua Landau schreef: > > To put things in perspective, these are the people who have been > insulting on this post: > > Mark Lawrence (once, probably without meaning to be insulting) > Nikos > Antoon Pardon I don't consider something insulting if it can be supported by argument and evidence. I stand by my statement that Nikos behaved like incompetent inconsidered jerk can be supported as such. So if this is insulting it only is in the same matter that the truth can be insulting. > And here are the people who have reminded them to stop: > > Steve Simmons > Steven D'Aprano > Andrew Berg > Walter Hurry > rusi > Joshua Landau (me) > > So yes, Antoon Pardon and Nikos, please stop. You are not representing > the list. I haven't followed any of the other arguments, true, but you > two in particular are causing a lot of trouble for the rest of us. It > is not hard to avoid making your disagreements public. Well nobody is representing the list. So where does that leave us? And perhaps I am causing a lot of trouble, that doesn't mean I am wrong. Maybe it is time to question the way this list in general tries to avoid trouble. -- Antoon Pardon. From joshua.landau.ws at gmail.com Mon Jul 1 15:28:58 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Mon, 1 Jul 2013 20:28:58 +0100 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <51D1D484.5070309@rece.vub.ac.be> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> Message-ID: On 1 July 2013 20:12, Antoon Pardon wrote: > Op 01-07-13 18:56, Joshua Landau schreef: > >> >> To put things in perspective, these are the people who have been >> insulting on this post: >> >> Mark Lawrence (once, probably without meaning to be insulting) >> Nikos >> Antoon Pardon > > > I don't consider something insulting if it can be supported > by argument and evidence. > > I stand by my statement that Nikos behaved like incompetent > inconsidered jerk can be supported as such. So if this > is insulting it only is in the same matter that the > truth can be insulting. Well then you are wrong. But fine, I'll use your definition incorrect as it may be (when talking to you, please don't misrepresent my other posts). Nevertheless, these statements that we are talking about, that I shall now term "pseudo-insults", are unwanted and not called for. Please keep them off-list. >> And here are the people who have reminded them to stop: >> >> Steve Simmons >> Steven D'Aprano >> Andrew Berg >> Walter Hurry >> rusi >> Joshua Landau (me) >> >> So yes, Antoon Pardon and Nikos, please stop. You are not representing >> the list. I haven't followed any of the other arguments, true, but you >> two in particular are causing a lot of trouble for the rest of us. It >> is not hard to avoid making your disagreements public. > > > Well nobody is representing the list. So where does that leave us? I am afraid you are wrong. The body of people of this list, as a whole, represent this list. As a whole, as I have shown, they resent your pseudo-insults. rusi is an exception, who only seems to resent Nikos's, but rusi has not fallen to pseudo-insults himself (herself?). > And perhaps I am causing a lot of trouble, that doesn't mean I am > wrong. Maybe it is time to question the way this list in general > tries to avoid trouble. Irrelevant. Your pseudo-insults are unwanted. Please stop. Argue all you want - I'm not going to judge who's right. But I'd rather you appreciate that no-one wants your pseudo-insults on this list. From antoon.pardon at rece.vub.ac.be Tue Jul 2 03:22:33 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Tue, 02 Jul 2013 09:22:33 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> Message-ID: <51D27FB9.7040406@rece.vub.ac.be> Op 01-07-13 21:28, Joshua Landau schreef: > Well then you are wrong. But fine, I'll use your definition incorrect > as it may be (when talking to you, please don't misrepresent my other > posts). > > Nevertheless, these statements that we are talking about, that I shall > now term "pseudo-insults", are unwanted and not called for. Please > keep them off-list. This doesn't make sense. This means that someone can behave like a jerk on this list, later tries to spin his behaviour as some innocent exchange of ideas and make it near impossible for others to make it clear what bullshit he is peddling because stating clearly how much like a jerk he behaved would be insulting. This is giving those who are behaving badly a huge advantage. >> Well nobody is representing the list. So where does that leave us? > I am afraid you are wrong. The body of people of this list, as a > whole, represent this list. As a whole, as I have shown, they resent > your pseudo-insults. rusi is an exception, who only seems to resent > Nikos's, but rusi has not fallen to pseudo-insults himself (herself?). You have shown no such thing. You have shown some people who resent my speaking clearly. That doesn't mean much, those that resent something are always making more noise than those who don't care or would even prefer it that way. >> And perhaps I am causing a lot of trouble, that doesn't mean I am >> wrong. Maybe it is time to question the way this list in general >> tries to avoid trouble. > Irrelevant. Your pseudo-insults are unwanted. Please stop. Argue all > you want - I'm not going to judge who's right. But I'd rather you > appreciate that no-one wants your pseudo-insults on this list. You have no authority to speak for everyone. You can't be even sure you speak for a majority, because once some small minority starts to speak up and pretend they speak for the whole group those who think differently often enough don't speak up. But for your ease of mind I'll make it clear I have no intention of haunting Nikos or of keeping him the subject of discussion. But should I stumble on a conversation in which his past behaviour is framed as him being innocentltly asking questions, I will point of what bullshit that is. -- Antoon Pardon From joshua.landau.ws at gmail.com Tue Jul 2 05:34:54 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Tue, 2 Jul 2013 10:34:54 +0100 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <51D27FB9.7040406@rece.vub.ac.be> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> Message-ID: On 2 July 2013 08:22, Antoon Pardon wrote: > Op 01-07-13 21:28, Joshua Landau schreef: > >> Well then you are wrong. But fine, I'll use your definition incorrect >> as it may be (when talking to you, please don't misrepresent my other >> posts). >> >> Nevertheless, these statements that we are talking about, that I shall >> now term "pseudo-insults", are unwanted and not called for. Please >> keep them off-list. > This doesn't make sense. This means that someone can behave like a jerk > on this list, later tries to spin his behaviour as some innocent exchange > of ideas and make it near impossible for others to make it clear what > bullshit he is peddling because stating clearly how much like a jerk he > > behaved would be insulting. This is giving those who are behaving badly > > a huge advantage. No it does not. I'd give you more of a counter but I actually have no idea how you came up with that. >>> Well nobody is representing the list. So where does that leave us? >> I am afraid you are wrong. The body of people of this list, as a >> whole, represent this list. As a whole, as I have shown, they resent >> your pseudo-insults. rusi is an exception, who only seems to resent >> Nikos's, but rusi has not fallen to pseudo-insults himself (herself?). > You have shown no such thing. You have shown some people who resent > my speaking clearly. I'd prefer you use my less ambiguous term "pseudo-insults". If you'd prefer, you can rename it. Using weasel-words helps nobody. I have no problem with you "speaking clearly" in a literal sense. I have a problem with your pseudo-insults. If you want to use "speaking clearly" to mean "pseudo-insult", you're going to have to point out that you're using words how they're not literally intended. > That doesn't mean much, those that resent > something are always making more noise than those who don't care or > would even prefer it that way. Only the people, no - person, making most noise is you. So you've sort-a countered your point. Sort of. But it's hard to counter a point that is basically just "well, despite the fact no-one supports my view there might be other silent people who do". > But for your ease of mind I'll make it clear I have no intention > of haunting Nikos or of keeping him the subject of discussion. > But should I stumble on a conversation in which his past behaviour > is framed as him being innocentltly asking questions, I will point > of what bullshit that is. Fair enough. If that's all you did in this thread, then I wouldn't care. But once again you seem to have missed the point that I and others keep reiterating: pseudo-insults have no place on this list. (If you need a reminder, pseudo-insults are just what other people term "insults". You can change the name if you see fit.) From antoon.pardon at rece.vub.ac.be Tue Jul 2 08:01:23 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Tue, 02 Jul 2013 14:01:23 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> Message-ID: <51D2C113.6070105@rece.vub.ac.be> Op 02-07-13 11:34, Joshua Landau schreef: > No it does not. I'd give you more of a counter but I actually have no > idea how you came up with that. Please answer the following question. If someone behaved incompetently, how can I clearly state that fact when "incompetently" is seen as an insult and insults don't belong on the list? >>>> Well nobody is representing the list. So where does that leave us? >>> I am afraid you are wrong. The body of people of this list, as a >>> whole, represent this list. As a whole, as I have shown, they resent >>> your pseudo-insults. rusi is an exception, who only seems to resent >>> Nikos's, but rusi has not fallen to pseudo-insults himself (herself?). >> You have shown no such thing. You have shown some people who resent >> my speaking clearly. > I'd prefer you use my less ambiguous term "pseudo-insults". If you'd > prefer, you can rename it. Using weasel-words helps nobody. > > I have no problem with you "speaking clearly" in a literal sense. I > have a problem with your pseudo-insults. If you want to use "speaking > clearly" to mean "pseudo-insult", you're going to have to point out > that you're using words how they're not literally intended. What if using pseudo-insults was the most clear way to describe the situation. Should I start looking for eufenisms? >> But for your ease of mind I'll make it clear I have no intention >> of haunting Nikos or of keeping him the subject of discussion. >> But should I stumble on a conversation in which his past behaviour >> is framed as him being innocentltly asking questions, I will point >> of what bullshit that is. > Fair enough. If that's all you did in this thread, then I wouldn't care. > > But once again you seem to have missed the point that I and others > keep reiterating: pseudo-insults have no place on this list. > > > (If you need a reminder, pseudo-insults are just what other people > term "insults". You can change the name if you see fit.) So what are the non-insulting terms for incompentent, (starting a webservice in a language you're a newby in, making changes on the life server so that any typo you make, can take your site out the air), inconsiderate (behave annoyingly in multiple ways and despite poeple pointing it out multiple times, mostly continue in the same manner, without taking their remarks into account) and jerk (trying to spin your inconsiderate behaviour as you being the victim, misrepresenting your behaviour when it is being discussed, always "explaining" your behaviour, as if an explanation would make a difference to the annoyance you caused to others...) -- Antoon Pardon From joshua.landau.ws at gmail.com Tue Jul 2 09:40:34 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Tue, 2 Jul 2013 14:40:34 +0100 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <51D2C113.6070105@rece.vub.ac.be> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> Message-ID: On 2 July 2013 13:01, Antoon Pardon wrote: > Op 02-07-13 11:34, Joshua Landau schreef: > >> No it does not. I'd give you more of a counter but I actually have no >> idea how you came up with that. > Please answer the following question. If someone behaved incompetently, > how can I clearly state that fact when "incompetently" is seen as an > insult and insults don't belong on the list? There is not ever a place on this list where you will need to call someone incompetent. You can explain to someone that they do not understand what they are doing, but when you attack the character of the person it is no longer acceptable. >>>>> Well nobody is representing the list. So where does that leave us? >>>> I am afraid you are wrong. The body of people of this list, as a >>>> whole, represent this list. As a whole, as I have shown, they resent >>>> your pseudo-insults. rusi is an exception, who only seems to resent >>>> Nikos's, but rusi has not fallen to pseudo-insults himself (herself?). >>> You have shown no such thing. You have shown some people who resent >>> my speaking clearly. >> I'd prefer you use my less ambiguous term "pseudo-insults". If you'd >> prefer, you can rename it. Using weasel-words helps nobody. >> >> I have no problem with you "speaking clearly" in a literal sense. I >> have a problem with your pseudo-insults. If you want to use "speaking >> clearly" to mean "pseudo-insult", you're going to have to point out >> that you're using words how they're not literally intended. > What if using pseudo-insults was the most clear way to describe the > situation. Should I start looking for eufenisms? No. In that case if you wish to describe the situation you would have to do with less clear means. I'm sure the people on this list would understand what you mean even if you left out the judgementalism. In this case in particular, many other people have "described the situation" perfectly adequately without resorting to blatant insults on Nikos. But how is a statement as blatantly inhumane as "Should we particularly care about Nikos's feelings?" a needed action -- it certainly isn't a good way of describing the situation. >>> But for your ease of mind I'll make it clear I have no intention >>> of haunting Nikos or of keeping him the subject of discussion. >>> But should I stumble on a conversation in which his past behaviour >>> is framed as him being innocentltly asking questions, I will point >>> of what bullshit that is. >> Fair enough. If that's all you did in this thread, then I wouldn't care. >> >> But once again you seem to have missed the point that I and others >> keep reiterating: pseudo-insults have no place on this list. >> >> >> (If you need a reminder, pseudo-insults are just what other people >> term "insults". You can change the name if you see fit.) > So what are the non-insulting terms for > > incompentent, (starting a webservice in a language you're a newby in, > making changes on the life server so that any typo you make, can take > your site out the air), You just did it. > inconsiderate (behave annoyingly in multiple ways and despite poeple pointing > it out multiple times, mostly continue in the same manner, without taking > their remarks into account) and I do not tend to consider "inconsiderate" inappropriate if said in earnest, as it is defensive. I'd still rather you talked about actions as inconsiderate rather than people, but baby steps. > jerk (trying to spin your inconsiderate behaviour as you being the victim, > misrepresenting your behaviour when it is being discussed, always "explaining" > your behaviour, as if an explanation would make a difference to the annoyance > you caused to others...) You came close. But, since Nikos has thankfully ceased, I'm saying here that unless you have good reasons otherwise I'd rather only continue this off-list. If you reply on-list without justification I will likely not reply. Apologies to everyone else for the clutter this has caused. From ben+python at benfinney.id.au Tue Jul 2 18:34:02 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 03 Jul 2013 08:34:02 +1000 Subject: python adds an extra half space when reading from a string or list References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> Message-ID: <7wsizwh9r9.fsf@benfinney.id.au> Joshua Landau writes: > On 2 July 2013 13:01, Antoon Pardon wrote: > > Please answer the following question. If someone behaved > > incompetently, how can I clearly state that fact when > > "incompetently" is seen as an insult and insults don't belong on the > > list? > > There is not ever a place on this list where you will need to call > someone incompetent. So even if that term describes their behaviour and manner, you think no-one should ever point it out? > > So what are the non-insulting terms for > > > > incompentent, (starting a webservice in a language you're a newby in, > > making changes on the life server so that any typo you make, can take > > your site out the air), > > You just did it. So you agree the correct term to use *is* ?incompetent?, as Anton said. Yet you don't want Anton to use the correct term. Needless to say, I disagree with your position. There is no place for baseless insults in this community; but when the behaviour of someone in this community is harmful, then it is entirely appropriate to use clear terms (e.g. ?incompetent?, ?inconsiderate?) to describe their behaviour. -- \ ?One bad programmer can easily create two new jobs a year. | `\ Hiring more bad programmers will just increase our perceived | _o__) need for them.? ?David Lorge Parnas, 1999-03 | Ben Finney From joshua.landau.ws at gmail.com Tue Jul 2 19:18:27 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Wed, 3 Jul 2013 00:18:27 +0100 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <7wsizwh9r9.fsf@benfinney.id.au> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <7wsizwh9r9.fsf@benfinney.id.au> Message-ID: On 2 July 2013 23:34, Ben Finney wrote: > > Joshua Landau writes: >> There is not ever a place on this list where you will need to call >> someone incompetent. > > So even if that term describes their behaviour and manner, you think > no-one should ever point it out? Yes. I see no instance where it would be needed or helpful. >> > So what are the non-insulting terms for >> > >> > incompentent, (starting a webservice in a language you're a newby in, >> > making changes on the life server so that any typo you make, can take >> > your site out the air), >> >> You just did it. > > So you agree the correct term to use *is* ?incompetent?, as Anton said. > Yet you don't want Anton to use the correct term. What? No, I think you misread what I intended in that post. I said that rather than attacking Niko's integrity by calling him incompetent, it is better to, for example, explain to him that he is "starting a webservice in a language you're a newby in, making changes on the life server so that any typo you make, can take your site out the air". Given, of course, that that is what prompted you to want to call him incompetent in the first place. Saying "you don't understand what you are doing" and "you are incompetent" are in this case interchangeable, except one involves making the other feel hurt. You don't need to do that, nor does it help anyone. > Needless to say, I disagree with your position. There is no place for > baseless insults in this community; but when the behaviour of someone in > this community is harmful, then it is entirely appropriate to use clear > terms (e.g. ?incompetent?, ?inconsiderate?) to describe their behaviour. Firstly, describing someone's behaviour and describing someone's character are two completely different things. The second is far more judgemental and unhelpful than the first. Antoon is describing people's character. You are talking about describing their actions. Secondly, as I said, I have few objections to the term "inconsiderate". Fewer still when describing a behaviour. As for calling someone incompetent, it would be wrong to associate incompetence with harm to this community. People were not upset -- at least I was not upset -- from Nikos being incompetent. Many people on this list lack skills. This is fine, everyone's unlearnt at some point. Some stay unlearnt forever. People were upset because of his manner, his demanding, his overbearing, him acting selfishly, etc. I will never complain to you calling someone rude, especially not calling someone's actions rude. I will never complain to you calling someone overbearing, especially not calling someone's actions overbearing. I will never complain to you calling someone's actions selfish. etc. I may disagree, and I may voice disagreements, but you have right to speak up if you feel mistreated. Please don't believe I think otherwise. Calling someone incompetent is an attack, and is not the same. If you cannot see the difference, I'm not sure what more I can say. I feel it is reasonable to continue my correspondence with you on-list, but if it spawns more than a few more posts I would rather take it off-list. I have no objections if you would like to Cc Antoon Pardon in the process, seeing as it's his post involved. From ben+python at benfinney.id.au Tue Jul 2 20:36:21 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 03 Jul 2013 10:36:21 +1000 Subject: python adds an extra half space when reading from a string or list References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <7wsizwh9r9.fsf@benfinney.id.au> Message-ID: <7w4nccxywq.fsf@benfinney.id.au> Joshua Landau writes: > Firstly, describing someone's behaviour and describing someone's > character are two completely different things. I agree with that. > Antoon is describing people's character. I disagree with that. To merely describe someone as ?incompetent? is not to describe their behaviour: they are not competent to do what they are attempting, which is not a judgement on their character. Just as to merely describe someone as ?ignorant? is to describe their lack of knowledge in some area, and is not a judgement on their character. Even character judgements can be appropriate, so long as they are salient to the acceptable behaviours in this forum; there are many character attributes that are incompatible with this community. So since both accurate description of actions and character judgement can be appropriate for discussing acceptable behaviour here, I'll thank you not to make absolute prohibitions against either. > You are talking about describing their actions. Yes. Since you seem to be imputing character judgements to descriptions which are accurate descriptions of behaviour, I think we've found the root of the disagreement. I've made my position clear and will let it rest there. > Calling someone incompetent is an attack It can be part of an attack, but in itself is merely a description of someone's behaviour. > If you cannot see the difference, I'm not sure what more I can say. Likewise. Thanks for caring enough about this community to act in the interest of keeping it open, considerate, and respectful. -- \ ?Beware of and eschew pompous prolixity.? ?Charles A. Beardsley | `\ | _o__) | Ben Finney From joshua.landau.ws at gmail.com Tue Jul 2 20:51:56 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Wed, 3 Jul 2013 01:51:56 +0100 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <7w4nccxywq.fsf@benfinney.id.au> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <7wsizwh9r9.fsf@benfinney.id.au> <7w4nccxywq.fsf@benfinney.id.au> Message-ID: On 3 July 2013 01:36, Ben Finney wrote: > I think we've found the root of > the disagreement. I've made my position clear and will let it rest there. Seconded. > Thanks for caring enough about this community to act in the > interest of keeping it open, considerate, and respectful. Thank you in turn. From rurpy at yahoo.com Tue Jul 2 21:21:11 2013 From: rurpy at yahoo.com (rurpy at yahoo.com) Date: Tue, 2 Jul 2013 18:21:11 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <7wsizwh9r9.fsf@benfinney.id.au> Message-ID: <72d35bd5-7048-4b8a-8a50-229a54703f05@googlegroups.com> On 07/02/2013 05:18 PM, Joshua Landau wrote: > On 2 July 2013 23:34, Ben Finney wrote: >[...] >> Needless to say, I disagree with your position. There is no place for >> baseless insults in this community; but when the behaviour of someone in >> this community is harmful, then it is entirely appropriate to use clear >> terms (e.g. ?incompetent?, ?inconsiderate?) to describe their behaviour. > > Firstly, describing someone's behaviour and describing someone's > character are two completely different things. The second is far more > judgemental and unhelpful than the first. Antoon is describing > people's character. You are talking about describing their actions. In practice there is often little difference. "You are acting like an asshole" could be defended as describing a behavior and "you are rude" as describing a character (rude by nature). The reality is that few of the people at whom such statements are aimed will make such fine distinction. If you use negatively judgemental statements against other posters, a significant number of them will respond to you in kind. From joshua.landau.ws at gmail.com Wed Jul 3 19:12:30 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Thu, 4 Jul 2013 00:12:30 +0100 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <72d35bd5-7048-4b8a-8a50-229a54703f05@googlegroups.com> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <7wsizwh9r9.fsf@benfinney.id.au> <72d35bd5-7048-4b8a-8a50-229a54703f05@googlegroups.com> Message-ID: On 3 July 2013 02:21, wrote: > On 07/02/2013 05:18 PM, Joshua Landau wrote: >> On 2 July 2013 23:34, Ben Finney wrote: >>[...] >>> Needless to say, I disagree with your position. There is no place for >>> baseless insults in this community; but when the behaviour of someone in >>> this community is harmful, then it is entirely appropriate to use clear >>> terms (e.g. ?incompetent?, ?inconsiderate?) to describe their behaviour. >> >> Firstly, describing someone's behaviour and describing someone's >> character are two completely different things. The second is far more >> judgemental and unhelpful than the first. Antoon is describing >> people's character. You are talking about describing their actions. > > In practice there is often little difference. "You are acting > like an asshole" could be defended as describing a behavior > and "you are rude" as describing a character (rude by nature). "You are acting like an asshole" *is* describing someone's behaviour. "You are an asshole" is the equivalent describing their character. In turn, "You are rude" *is* describing their character. However, I said the "second is far more judgemental and unhelpful than the first". I stand by that, but don't take that to mean that comments on people's behaviour cannot be insulting of derogatory, nor that comments on character have to be insulting. Calling someone rude may be worse than saying that they are acting rude, but it is not unjust if they have been rude, for example. > The reality is that few of the people at whom such statements > are aimed will make such fine distinction. If you use negatively > judgemental statements against other posters, a significant > number of them will respond to you in kind. I think most people will take "you are stupid" and "what you've done is stupid" to have very different levels of hostility, even though the same thing to be said has been expressed. That does not mean people will not take offense at the second, but they will take less. I believe that this does generalise. Note that I am not advocating calling people's actions stupid, as it is still just a harsher way of saying "you've made a mistake". From joshua.landau.ws at gmail.com Wed Jul 3 19:40:09 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Thu, 4 Jul 2013 00:40:09 +0100 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <7wsizwh9r9.fsf@benfinney.id.au> <72d35bd5-7048-4b8a-8a50-229a54703f05@googlegroups.com> Message-ID: On 3 July 2013 11:01, Antoon Pardon wrote: > Op 02-07-13 15:40, Joshua Landau schreef: >> On 2 July 2013 13:01, Antoon Pardon wrote: >>> >> >> There is not ever a place on this list where you will need to call >> someone incompetent. You can explain to someone that they do not >> understand what they are doing, but when you attack the character of >> the person it is no longer acceptable. > > This is not an attack of character. Level of skill/competence is > not in general seen as a character trait. I disagree. I'm not sure how to argue this, rather than point out that by "character trait" I'm not talking about "intrinsic trait" but rather "revolving around who you are". The alternative I posed is talking about things revolving around specific actions that you have taken. > It is something one can > generally increase if one is willing to work on it, and once > you acquired it, you don't have to keep your guard for situations > in which you might loose it. That's not relevant to whether it's a character trait, ? mon avis. >>> So what are the non-insulting terms for >>> >>> incompentent, (starting a webservice in a language you're a newby in, >>> making changes on the life server so that any typo you make, can take >>> your site out the air), >> You just did it. > > But how do I describe this in one word? What conclusion am I allowed > to make from all this? That it's worth taking a little more time doing things if it makes sure you aren't harming the person on the other end. > Can I say: > He displayed a pattern of incompetence? WIth trepidation. Saying what someone did is "incompetent" is quite judgemental. > He has been blundering about? With trepidation. This is just another way of saying the same thing. > His skill level was unimpressive? With trepidation. This refers to his character, but is still quite light and far lower than the level of aggression that spawned my speaking-up. > The skill level he displayed, left much to be desired? Probably. Except only if you remove the comma. Bear in mind that if the way you were acting was all in my "with trepidation" category, I would likely have not spoken up. I believe you crossed a lot further beyond that line. >>> inconsiderate (behave annoyingly in multiple ways and despite poeple pointing >>> it out multiple times, mostly continue in the same manner, without taking >>> their remarks into account) and >> I do not tend to consider "inconsiderate" inappropriate if said in >> earnest, as it is defensive. I'd still rather you talked about actions >> as inconsiderate rather than people, but baby steps. > > I don't understand this, since (in)consideration is in general seen as > a character trait. On that basis I think you have it exactly backwards > when you consider "incompetent" an attack of character yet seem to > have little problem with "inconsiderate". "You are inconsiderate" is describing someone's character. Hence I believe it would be better to say "what you did was inconsiderate". What I was saying, though, is that because "inconsiderate" is defensive and reactionary it is not an insult*. It refers to the fact that you believe the person you say it to should respect other people and what he is doing negatively impacts them. Another example would be: "you are awesome". Despite being a description of character, I'm not too idiotic to realise that it is not a mean thing to say. (Even here, "what you did is awesome" is a *milder* way of saying the same idea.) It's easy to misread what I've said, but I was never trying to imply that my problem was solely in this distinction -- it was just that it's important to realise that the divide exists and very explanatory to some aspects of how people take commentary. Both rurpy and you seem to have mistaken me to believe that describing character vs. describing actions is the be-all and end-all. Rather, I just believe it is a large factor in the harshness of what you say. *You seem to have moved back to the term "insult"; this seems especially appropriate since we are discussing its meaning. > Yet people who have a more > inconsiderate character can't work on acquiring consideration, as one > can on acquiring skill/competence. Sure one can work on it, but it is > something you have to keep watchful for. Otherwise you can easily slip > again into a pattern of being/behaving inconsiderate. Firstly, I disagree. Consideration is a learnt trait as most human interaction is. Secondly, I don't understand why you are bringing this up. >>> jerk (trying to spin your inconsiderate behaviour as you being the victim, >>> misrepresenting your behaviour when it is being discussed, always "explaining" >>> your behaviour, as if an explanation would make a difference to the annoyance >>> you caused to others...) >> You came close. > > Same question as two entries above. I shall leave this open to see if you think my answer above applies here. From antoon.pardon at rece.vub.ac.be Thu Jul 4 07:19:26 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Thu, 04 Jul 2013 13:19:26 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <7wsizwh9r9.fsf@benfinney.id.au> <72d35bd5-7048-4b8a-8a50-229a54703f05@googlegroups.com> Message-ID: <51D55A3E.1040609@rece.vub.ac.be> Op 04-07-13 01:40, Joshua Landau schreef: > On 3 July 2013 11:01, Antoon Pardon wrote: > >> This is not an attack of character. Level of skill/competence is >> not in general seen as a character trait. > I disagree. I'm not sure how to argue this, rather than point out that > by "character trait" I'm not talking about "intrinsic trait" but > rather "revolving around who you are". The alternative I posed is > talking about things revolving around specific actions that you have > taken. All right I can work with that. However I get the impression what you are talking about is then not limited to character traits but can be applied to other traits too. It is unlikely to come up in this group but I guess you would be opposed in a similar way to someone being called a weakling after it became clear he was not physically fit for a particular task. >> But how do I describe this in one word? What conclusion am I allowed >> to make from all this? > That it's worth taking a little more time doing things if it makes > sure you aren't harming the person on the other end. I agree to a certain extend. But sometimes I seem to come up (not only on usenet but in real life too) to people who seem to be oblivious to the message you are sending until you cast it into words that sting. I have been guily of such behaviour myself, not really reacting to the warnings from my partner until they were termed in a way that was hard to ignore. And sure, there is a serious risk of a hostile reaction, but getting a hostile reaction doesn't contradict that it was effective. People often do change their behaviour even if there first reaction was hostile and denial. So I'll agree to doing as little harm as possible and that there is a level of harm that shouldn't be crossed. But I won't go as far as making sure no harm is done. > >> [examples] > Bear in mind that if the way you were acting was all in my "with > trepidation" category, I would likely have not spoken up. I believe > you crossed a lot further beyond that line. I had to look up "trepidation" and the translation I got from my dictionary was not 100% helpful. Would "hesitation" or "wariness" be a good substitute. Can we go back to my first entry in this thread? I would like to know whether you think that already crossed the line or if that would fall under your "with trepidation" category and I crossed the line later. I think the relevant part is this: ] Maybe that way he'll learn that if he doesn't want to be considered ] an incompetent inconsiderate jerk, he shouldn't behave like one. -- Antoon Pardon From joshua.landau.ws at gmail.com Thu Jul 4 21:58:27 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Fri, 5 Jul 2013 02:58:27 +0100 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <51D55A3E.1040609@rece.vub.ac.be> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <7wsizwh9r9.fsf@benfinney.id.au> <72d35bd5-7048-4b8a-8a50-229a54703f05@googlegroups.com> <51D55A3E.1040609@rece.vub.ac.be> Message-ID: On 4 July 2013 12:19, Antoon Pardon wrote: > Op 04-07-13 01:40, Joshua Landau schreef: > >> Bear in mind that if the way you were acting was all in my "with >> trepidation" category, I would likely have not spoken up. I believe >> you crossed a lot further beyond that line. > > I had to look up "trepidation" and the translation I got from my > dictionary was not 100% helpful. Would "hesitation" or "wariness" > be a good substitute. Yes; it's along the lines of "hesitation /due/ to wariness". > Can we go back to my first entry in this thread? I would like to > know whether you think that already crossed the line or if that > would fall under your "with trepidation" category and I crossed > the line later. > > I think the relevant part is this: > > ] Maybe that way he'll learn that if he doesn't want to be considered > ] an incompetent inconsiderate jerk, he shouldn't behave like one. If my records are accurate*, by the time I responded (excluding that to Mark Lawrence), you had also said: > Should we particularly care about Nikos's feelings? > May that way he'll learn that if he doesn't > want to be considered an incompetent inconsiderate jerk, he shouldn't > behave like one. > You'll notice I don't judge all newbies as > incompetent jerks. Just you > I am judging you for > behaving liken an asshole and trying to deny it. > you were behaving like an asshole > You poor thing. (Obvious hostile sarcasm in context) And quite a few repeats of the same things. * I thought I stepped in earlier, but I'm not going to argue with GMail's history. This quote: > Maybe that way he'll learn that if he doesn't want to be considered > an incompetent inconsiderate jerk, he shouldn't behave like one. in solidarity -- although calling someone "incompetent" and, especially, a "jerk" is unwanted -- would probably not have made me respond. However, it is definitely past the "with trepidation" category. From rurpy at yahoo.com Wed Jul 3 21:56:46 2013 From: rurpy at yahoo.com (rurpy at yahoo.com) Date: Wed, 3 Jul 2013 18:56:46 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <7wsizwh9r9.fsf@benfinney.id.au> <72d35bd5-7048-4b8a-8a50-229a54703f05@googlegroups.com> Message-ID: On 07/03/2013 05:12 PM, Joshua Landau wrote: > On 3 July 2013 02:21, wrote: >[...] >> The reality is that few of the people at whom such statements >> are aimed will make such fine distinction. If you use negatively >> judgemental statements against other posters, a significant >> number of them will respond to you in kind. > > I think most people will take "you are stupid" and "what you've done > is stupid" to have very different levels of hostility, even though the > same thing to be said has been expressed. That does not mean people > will not take offense at the second, but they will take less. I > believe that this does generalise. I think the first is perceived as more hostile not because of any character vs behavior difference but simply because it is broader -- the implication is that most anything done by the person will be stupid rather than one specific action for the second. If one compares "you're stupid to have done that" (character) vs "what you've done is stupid" (behavior), I am not sure there is much difference; at least I don't think I'd make much distinction. IOW, I maintain that it is the "stupid" part that results in the major part of the recipient's negative reaction. Of course since neither of us are psychologists this is all pretty speculative. > Note that I am not advocating calling people's actions stupid, as it > is still just a harsher way of saying "you've made a mistake". Agreed, 100%. From rurpy at yahoo.com Tue Jul 2 20:30:32 2013 From: rurpy at yahoo.com (rurpy at yahoo.com) Date: Tue, 2 Jul 2013 17:30:32 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> Message-ID: <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> Since this thread too has gone into the trash hole thanks to our resident trolls I might as well comment... On 07/02/2013 04:34 PM, Ben Finney wrote:> Joshua Landau writes: > >> On 2 July 2013 13:01, Antoon Pardon wrote: >> > Please answer the following question. If someone behaved >> > incompetently, how can I clearly state that fact when >> > "incompetently" is seen as an insult and insults don't belong on the >> > list? >> >> There is not ever a place on this list where you will need to call >> someone incompetent. > > So even if that term describes their behaviour and manner, you think > no-one should ever point it out? If it is relevant and likely to be helpful to the asker, by all means please point it out. But if someone is asking a question about *python*, what relevance does their competence as an Apache administrator (or ability as a father for that matter) have to do with it? [*1] If your going to point out something negative about someone then do so politely. Ask yourself if you were pointing out incompetence to your boss (or anyone else where impoliteness could have real consequences for you) if you would say, "you're incompetent." In Nikos' case, unless you're an idiot or severely socially challenged, it should have been clear early on that pointing out his knowledge deficits was not going to be helpful to him. Telling him to "hire someone" without any knowledge of his goals or resources is as patently assine as trying to justify it as "telling the truth". >> > So what are the non-insulting terms for >> > >> > incompentent, (starting a webservice in a language you're a newby in, >> > making changes on the life server so that any typo you make, can take >> > your site out the air), >> >> You just did it. > > So you agree the correct term to use *is* ?incompetent?, as Anton said. > Yet you don't want Anton to use the correct term. > > Needless to say, I disagree with your position. There is no place for > baseless insults in this community; but when the behaviour of someone in > this community is harmful, then it is entirely appropriate to use clear > terms (e.g. ?incompetent?, ?inconsiderate?) to describe their behaviour. Those are anything but clear. They are subjective and highly judgmental and frequently applied one-sidedly. (Not "rude" when an alpha poster here says it but "rude" when someone else does.) Please use non-emotional, neutral, factual descriptions and only do so when it is actually relevant. IOW, please resist your desire to "tell off" the poster -- it usually just produces more responses in kind. ---- [*1] I realize in Nikos' case, there were questions about web site administration, but then the answer should be (if you don't want to deal with it and you can't can't control your need to reply with something), a polite, "that's off topic here but the apache list might be able to help you." From steve+comp.lang.python at pearwood.info Tue Jul 2 21:24:22 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 03 Jul 2013 01:24:22 GMT Subject: python adds an extra half space when reading from a string or list References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> Message-ID: <51d37d45$0$29999$c3e8da3$5496439d@news.astraweb.com> On Tue, 02 Jul 2013 17:30:32 -0700, rurpy wrote: >> Needless to say, I disagree with your position. There is no place for >> baseless insults in this community; but when the behaviour of someone >> in this community is harmful, then it is entirely appropriate to use >> clear terms (e.g. ?incompetent?, ?inconsiderate?) to describe their >> behaviour. > > Those are anything but clear. They are subjective and highly judgmental > and frequently applied one-sidedly. (Not "rude" when an alpha poster > here says it but "rude" when someone else does.) That is a very good point. One of the reasons I do not use the Python IRC channel is that in my experience anyone disagreeing with the privileged few's *opinions* gets banned without either warning or explanation. On the other hand, I've certainly learned a lot in my newbie days from being told off quite harshly by some of the Python community alphas, like the Effbot Fredrik Lundh, and Alex Martelli. It hurts to be told you're an idiot by one of the alphas, but sometimes you need to be told you're an idiot[1]. Even if you're not, realising that others think you are builds character[2]. The difference is that, as a rule, the alphas don't get suckered into interminable flame wars that suck the life out of a community. Short, harsh, surgical strikes is one thing. Four years of trench warfare is another thing altogether. [1] And for the record, I can think of two specific cases, although I may have forgotten others. In one case I think time has proven that I was right in my opinion, or at least the opinion of the Python devs shifted towards my viewpoint (introduction of tuple methods). In the other I have come to accept that Alex Martelli was right, although I still think that *failing an assignment* for a single unnecessary global declaration is unfairly harsh. [2] Sometimes it builds stronger, better character. Sometimes it builds weaker, broken character. It's all character. -- Steven From rurpy at yahoo.com Wed Jul 3 13:08:46 2013 From: rurpy at yahoo.com (rurpy at yahoo.com) Date: Wed, 3 Jul 2013 10:08:46 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: <51d37d45$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51d37d45$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1d3f4c08-8166-451e-b03d-105ae3ac7eb0@googlegroups.com> On 07/02/2013 07:24 PM, Steven D'Aprano wrote: >[...] > On the other hand, I've certainly learned a lot in my newbie days from > being told off quite harshly by some of the Python community alphas, like > the Effbot Fredrik Lundh, and Alex Martelli. It hurts to be told you're > an idiot by one of the alphas, but sometimes you need to be told you're > an idiot[1]. Even if you're not, realising that others think you are > builds character[2]. True, for some people, some times. As your [2] footnote admits it can be damaging as well. I have noticed that the ratio of female to mail posters here (at least as I can guess from names) seems far lower than the same ratio of programmers where I've worked. I asked a female programmer once why she didn't use groups like c.l.p. and she cited harshness and bickering as the main reason. Obviously a sample of one doesn't prove anything but... Effbot was around when I first started reading c.l.p and although I immediately recognized his contributions to and knowledge of python, I always questioned whether his positive contributions weren't negated by his negative ones. (Consider how many positive things he contributed that wouldn't have been mentioned by someone else had he not posted them, versus the disruption caused in reaction to his attitude.) >[...] > [2] Sometimes it builds stronger, better character. Sometimes it builds > weaker, broken character. It's all character. From wuwei23 at gmail.com Wed Jul 3 21:16:10 2013 From: wuwei23 at gmail.com (alex23) Date: Thu, 04 Jul 2013 11:16:10 +1000 Subject: Whatever happened to the Effbot? [was Re: python adds an etc] In-Reply-To: <1d3f4c08-8166-451e-b03d-105ae3ac7eb0@googlegroups.com> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51d37d45$0$29999$c3e8da3$5496439d@news.astraweb.com> <1d3f4c08-8166-451e-b03d-105ae3ac7eb0@googlegroups.com> Message-ID: On 4/07/2013 3:08 AM, rurpy at yahoo.com wrote: > Effbot was around when I first started reading c.l.p Does anyone know if Fredrik Lundh is still an active Python user? His site hasn't been updated for 3-4+ years now (there's an index error on the articles page of effbot.org). Has he pulled a Pilgrim? From skip at pobox.com Wed Jul 3 21:31:30 2013 From: skip at pobox.com (Skip Montanaro) Date: Wed, 3 Jul 2013 20:31:30 -0500 Subject: Whatever happened to the Effbot? [was Re: python adds an etc] In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51d37d45$0$29999$c3e8da3$5496439d@news.astraweb.com> <1d3f4c08-8166-451e-b03d-105ae3ac7eb0@googlegroups.com> Message-ID: Last I knew, Fredrik was working for Google. According to his LinkedIn profile he's a Google employee in Zurich, apparently doing YouTube stuff (assuming his profile is up-to-date). Skip From antoon.pardon at rece.vub.ac.be Wed Jul 3 05:21:40 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Wed, 03 Jul 2013 11:21:40 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> Message-ID: <51D3ED24.9030202@rece.vub.ac.be> Op 03-07-13 02:30, rurpy at yahoo.com schreef: > If your going to point out something negative about someone > then do so politely. Ask yourself if you were pointing out > incompetence to your boss (or anyone else where impoliteness > could have real consequences for you) if you would say, > "you're incompetent." And so we shift from no problem speaking bluntly or clearly to wording it in a way that wouldn't antagonize your boss too much. Off course that would also mean throwing out remarks like: ] You have betrayed the trust of all your customers. Which seemed to be accepted on this list without a problem. > Please use non-emotional, neutral, factual descriptions > and only do so when it is actually relevant. IOW, please > resist your desire to "tell off" the poster -- it usually > just produces more responses in kind. This is often not workable. Limiting to factual description means that you often can't summarize a list of such factual descriptions into a conclusion. You can list 8 examples of someone betraying the trust of his customers but you can't summarize it into: "is/behaves untrustworthy to his customers," even if all signs point to this person going to continue in the same vein. It is limiting yourself into pointing out all the trees without being allowed to call it a forest. -- Antoon Pardon From rosuav at gmail.com Wed Jul 3 05:45:15 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 3 Jul 2013 19:45:15 +1000 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <51D3ED24.9030202@rece.vub.ac.be> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> Message-ID: On Wed, Jul 3, 2013 at 7:21 PM, Antoon Pardon wrote: > Op 03-07-13 02:30, rurpy at yahoo.com schreef: >> If your going to point out something negative about someone >> then do so politely. Ask yourself if you were pointing out >> incompetence to your boss (or anyone else where impoliteness >> could have real consequences for you) if you would say, >> "you're incompetent." > > And so we shift from no problem speaking bluntly or clearly > to wording it in a way that wouldn't antagonize your boss > too much. > > Off course that would also mean throwing out remarks like: > > ] You have betrayed the trust of all your customers. > > Which seemed to be accepted on this list without a problem. If my boss gave a random stranger from a mailing list the root password to one of our servers, I would say to his face that he had betrayed his (our) customers' trust. I would say it with strong emphasis and a raised tone, too, and no small heat. The words you quote above are perfectly factual and, in my opinion, business-like language. ChrisA From antoon.pardon at rece.vub.ac.be Wed Jul 3 06:04:19 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Wed, 03 Jul 2013 12:04:19 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> Message-ID: <51D3F723.4010006@rece.vub.ac.be> Op 03-07-13 11:45, Chris Angelico schreef: > On Wed, Jul 3, 2013 at 7:21 PM, Antoon Pardon > wrote: >> Op 03-07-13 02:30, rurpy at yahoo.com schreef: >>> If your going to point out something negative about someone >>> then do so politely. Ask yourself if you were pointing out >>> incompetence to your boss (or anyone else where impoliteness >>> could have real consequences for you) if you would say, >>> "you're incompetent." >> And so we shift from no problem speaking bluntly or clearly >> to wording it in a way that wouldn't antagonize your boss >> too much. >> >> Off course that would also mean throwing out remarks like: >> >> ] You have betrayed the trust of all your customers. >> >> Which seemed to be accepted on this list without a problem. > If my boss gave a random stranger from a mailing list the root > password to one of our servers, I would say to his face that he had > betrayed his (our) customers' trust. I would say it with strong > emphasis and a raised tone, too, and no small heat. The words you > quote above are perfectly factual and, in my opinion, business-like > language. > > ChrisA But the real question of course is, would your boss see it that way? Maybe your boss would, but talking about a random boss, I wouldn't bet any money on it. -- Antoon Pardon From nikos at superhost.gr Wed Jul 3 06:00:14 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Wed, 03 Jul 2013 13:00:14 +0300 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> Message-ID: ???? 3/7/2013 12:45 ??, ?/? Chris Angelico ??????: >> ] You have betrayed the trust of all your customers. >> >> Which seemed to be accepted on this list without a problem. > > If my boss gave a random stranger from a mailing list the root > password to one of our servers, I would say to his face that he had > betrayed his (our) customers' trust. I would say it with strong > emphasis and a raised tone, too, and no small heat. The words you > quote above are perfectly factual and, in my opinion, business-like > language. I just received a call form on of my customers asking me to explain your mail and i did tell him the complete truth. He was surprised by what i did my hopefully he will not leave from my server. Also he asked me to redesign his website. Thank God my business left intact from what you did. Of course i should have give you the root pass(it was indeed stupid), but you violated my trust. You should have been clear that you didnt want to help and not asking me via private mail for the root pass. any way all is well now. -- What is now proved was at first only imagined! From feedthetroll at gmx.de Wed Jul 3 10:12:19 2013 From: feedthetroll at gmx.de (feedthetroll at gmx.de) Date: Wed, 3 Jul 2013 07:12:19 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> Message-ID: <448b55c5-f0c6-4e16-9a9f-1749d9208397@googlegroups.com> Am Mittwoch, 3. Juli 2013 12:00:14 UTC+2 schrieb ?????: > ???? 3/7/2013 12:45 ??, ?/? Chris Angelico ??????: > >> ] You have betrayed the trust of all your customers. > ... > I just received a call form on of my customers asking me to explain your > mail ... > Of course i should have give you the root pass(it was indeed stupid), > but you violated my trust. > You should have been clear that you didnt want to help and not asking me > via private mail for the root pass. May i cite: Am Dienstag, 4. Juni 2013 19:12:41 UTC+2 schrieb ???????? ??????: > ?? ?????, 4 ??????? 2013 8:09:18 ?.?. UTC+3, ? ??????? Chris Angelico ??????: >> On Wed, Jun 5, 2013 at 3:02 AM, ???????? ?????? >> wrote: >>> I'm willing to let someone with full root access to my webhost to see >>> thigns from the inside. >>> ... >> You need to read up on what happens when you enter Dummy Mode and give >> someone full root access to your web host. You really REALLY need to >> understand what that means before you offer random strangers that kind >> of access to someone else's data. >> ************************************************************************* ************************************************************************* >> I've half a mind to take you up on your offer, then go look for >> personal and private info from your clients, and email it to them >> (along with a link to this thread) to point out what's going on. ************************************************************************* ************************************************************************* >> ChrisA > > I know what full root access mean. > I also trust you. > ... Am Mittwoch, 5. Juni 2013 00:12:26 UTC+2 schrieb Chris Angelico: > The call is strong... I could rule the galaxy alongside my father... > I've searched my feelings, and I know this to be true! > ************************************************************************* ************************************************************************* > Okay. I accept. I'll **do as I promised.** Might be interesting, and > educative - for someone, at least. ************************************************************************* ************************************************************************* [emphasis added for those, who do not want to read the whole post] Any questions? From square.steve at gmail.com Wed Jul 3 11:21:26 2013 From: square.steve at gmail.com (Steve Simmons) Date: Wed, 03 Jul 2013 16:21:26 +0100 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <448b55c5-f0c6-4e16-9a9f-1749d9208397@googlegroups.com> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> <448b55c5-f0c6-4e16-9a9f-1749d9208397@googlegroups.com> Message-ID: <51D44176.7080302@gmail.com> On 03/07/2013 15:12, feedthetroll at gmx.de wrote: > Am Mittwoch, 3. Juli 2013 12:00:14 UTC+2 schrieb ?????: >> ???? 3/7/2013 12:45 ??, ?/? Chris Angelico ??????: >>>> ] You have betrayed the trust of all your customers. >> ... >> I just received a call form on of my customers asking me to explain your >> mail ... >> Of course i should have give you the root pass(it was indeed stupid), >> but you violated my trust. >> You should have been clear that you didnt want to help and not asking me >> via private mail for the root pass. > May i cite: > Am Dienstag, 4. Juni 2013 19:12:41 UTC+2 schrieb ???????? ??????: >> ?? ?????, 4 ??????? 2013 8:09:18 ?.?. UTC+3, ? ??????? Chris Angelico ??????: >>> On Wed, Jun 5, 2013 at 3:02 AM, ???????? ?????? >>> wrote: >>>> I'm willing to let someone with full root access to my webhost to see >>>> thigns from the inside. >>>> ... >>> You need to read up on what happens when you enter Dummy Mode and give >>> someone full root access to your web host. You really REALLY need to >>> understand what that means before you offer random strangers that kind >>> of access to someone else's data. >>> > ************************************************************************* > ************************************************************************* >>> I've half a mind to take you up on your offer, then go look for >>> personal and private info from your clients, and email it to them >>> (along with a link to this thread) to point out what's going on. > ************************************************************************* > ************************************************************************* >>> ChrisA >> I know what full root access mean. >> I also trust you. >> ... > Am Mittwoch, 5. Juni 2013 00:12:26 UTC+2 schrieb Chris Angelico: >> The call is strong... I could rule the galaxy alongside my father... >> I've searched my feelings, and I know this to be true! >> > ************************************************************************* > ************************************************************************* >> Okay. I accept. I'll **do as I promised.** Might be interesting, and >> educative - for someone, at least. > ************************************************************************* > ************************************************************************* > [emphasis added for those, who do not want to read the whole post] > > Any questions? > +1000 I missed a few key posts on this thread and was wondering what happened between Chris and Nicos - thanks for the summary :-) From rustompmody at gmail.com Wed Jul 3 13:00:50 2013 From: rustompmody at gmail.com (rusi) Date: Wed, 3 Jul 2013 10:00:50 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: <448b55c5-f0c6-4e16-9a9f-1749d9208397@googlegroups.com> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> <448b55c5-f0c6-4e16-9a9f-1749d9208397@googlegroups.com> Message-ID: On Wednesday, July 3, 2013 7:42:19 PM UTC+5:30, feedth... at gmx.de wrote: > Any questions? YES! Who is that hiding behind 'FeedTheTroll' ? Well thanks anyways :-) I was thinking of doing that but could not find my oxygen mask needed to wade into the steaming pile of ... From feedthetroll at gmx.de Thu Jul 4 04:24:19 2013 From: feedthetroll at gmx.de (feedthetroll at gmx.de) Date: Thu, 4 Jul 2013 01:24:19 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> <448b55c5-f0c6-4e16-9a9f-1749d9208397@googlegroups.com> Message-ID: <195cd600-8486-4d4d-9eda-54057eef4e5b@googlegroups.com> Am Mittwoch, 3. Juli 2013 19:00:50 UTC+2 schrieb rusi: > On Wednesday, July 3, 2013 7:42:19 PM UTC+5:30, feedth... at gmx.de wrote: > > Any questions? > YES! > Who is that hiding behind 'FeedTheTroll' ? Oh, it's just yattt (yet another troll trolling troll) lurching around to find amazing threads (Well, in fact these threads stopped being amazing weeks ago. I must be some sort of masochist :-) ) From rurpy at yahoo.com Wed Jul 3 13:01:23 2013 From: rurpy at yahoo.com (rurpy at yahoo.com) Date: Wed, 3 Jul 2013 10:01:23 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: <448b55c5-f0c6-4e16-9a9f-1749d9208397@googlegroups.com> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> <448b55c5-f0c6-4e16-9a9f-1749d9208397@googlegroups.com> Message-ID: <79dd34f7-f1df-4d4f-9f9b-60ddc28b6754@googlegroups.com> On 07/03/2013 08:12 AM, feedthetroll at gmx.de wrote: > > Am Mittwoch, 3. Juli 2013 12:00:14 UTC+2 schrieb ?????: >> >> ???? 3/7/2013 12:45 ??, ?/? Chris Angelico ??????: >>>> >> >> ] You have betrayed the trust of all your customers. >> >> ... >> >> I just received a call form on of my customers asking me to explain your >> >> mail ... >> >> Of course i should have give you the root pass(it was indeed stupid), >> >> but you violated my trust. >> >> You should have been clear that you didnt want to help and not asking me >> >> via private mail for the root pass. > > > > May i cite: > > Am Dienstag, 4. Juni 2013 19:12:41 UTC+2 schrieb ???????? ??????: >> >> ?? ?????, 4 ??????? 2013 8:09:18 ?.?. UTC+3, ? ??????? Chris Angelico ??????: >>> >>> On Wed, Jun 5, 2013 at 3:02 AM, ???????? ?????? >>> >>> wrote: >>>> >>>> I'm willing to let someone with full root access to my webhost to see >>>> >>>> thigns from the inside. >>>> >>>> ... >>> >>> You need to read up on what happens when you enter Dummy Mode and give >>> >>> someone full root access to your web host. You really REALLY need to >>> >>> understand what that means before you offer random strangers that kind >>> >>> of access to someone else's data. >>> >>> > > ************************************************************************* > > ************************************************************************* >>> >>> I've half a mind to take you up on your offer, then go look for >>> >>> personal and private info from your clients, and email it to them >>> >>> (along with a link to this thread) to point out what's going on. > > ************************************************************************* > > ************************************************************************* >>> >>> ChrisA >> >> >> >> I know what full root access mean. >> >> I also trust you. >> >> ... > > > > Am Mittwoch, 5. Juni 2013 00:12:26 UTC+2 schrieb Chris Angelico: >> >> The call is strong... I could rule the galaxy alongside my father... >> >> I've searched my feelings, and I know this to be true! >> >> > > ************************************************************************* > > ************************************************************************* >> >> Okay. I accept. I'll **do as I promised.** Might be interesting, and >> >> educative - for someone, at least. > > ************************************************************************* > > ************************************************************************* > > [emphasis added for those, who do not want to read the whole post] > > > > Any questions? Yes. If you tell someone you're going to beat them up but they don't take you seriously because they are naive, or they think you're kidding, or they don't understand you because you told them in hints rather than directly, is it ok to beat them up? Are the existence of laws against beating people up negated because you told them in advance? Or negated because they "deserve" the beating? Does it make any difference if they contribute to their own beating, say by showing up at a designated place and time? Does it make any difference if the beating is for their own good? (In the beater's opinion?) (In the majority opinion?) Does it make any difference if there are ways other than beating of communicating the message? How long and how hard should those alternate ways be tried before resorting to beating? I am sorry Mr/Ms. FeedTheTroll for being so dumb. I know the answers to all these questions are obvious to everyone else here but I am not sure about them. From rustompmody at gmail.com Wed Jul 3 14:08:11 2013 From: rustompmody at gmail.com (rusi) Date: Wed, 3 Jul 2013 11:08:11 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: <79dd34f7-f1df-4d4f-9f9b-60ddc28b6754@googlegroups.com> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> <448b55c5-f0c6-4e16-9a9f-1749d9208397@googlegroups.com> <79dd34f7-f1df-4d4f-9f9b-60ddc28b6754@googlegroups.com> Message-ID: <3f58609f-c249-4d2d-bfe4-70ed153f343c@googlegroups.com> On Wednesday, July 3, 2013 10:31:23 PM UTC+5:30, ru... at yahoo.com wrote: > > > Are the existence of laws against beating people up > negated because you told them in advance? Or negated > because they "deserve" the beating? One of the fundamental purpose of laws is to legalize what you call 'beating-up' Goes by the names like jail-sentences, fines and ultimately capital punishment. Civilization would not be possible without these. BTW the etymological roots of 'civilization' come from 16th century usages such as "I shall civilise thee with my rod" > Does it make any difference if the beating is for > their own good? (In the beater's opinion?) (In > the majority opinion?) As a good Christian I believe that Chris tried more than anyone else on this list to help Nikos before talking recourse to another gem of biblical wisdom: He that spareth his rod hateth his son: but he that loveth him chasteneth him betimes. And when Nikos moves up from petty criminal status to responsible citizen, I bet he will thank Chris more for all his real-help cum chastisement than your honeyed charity-at-others-expense. > Does it make any difference if there are ways other > than beating of communicating the message? How long > and how hard should those alternate ways be tried > before resorting to beating? Here are a couple of random Chris' posts picked up from the dung-heap of Nikos threads. Anyone will see that they represent a genuine and active context of trying-to-help. Yes, there is irritation, frustration, humor etc but always actively in the context of trying actively to help Nikos. http://mail.python.org/pipermail/python-list/2013-June/648158.html http://mail.python.org/pipermail/python-list/2013-June/648167.html http://mail.python.org/pipermail/python-list/2013-June/648200.html http://mail.python.org/pipermail/python-list/2013-June/648224.html Now multiply that by a couple of orders of magnitude and you get the extent Chris tried to help before deciding to wield the rod. BTW where were you then, when all this genuine but ultimately futile help-attempting was going on? And if you feel so strongly about this as you claim how come you did not protest when Chris clearly gave the warnings about his impending actions on this list that 'feedthetroll' has kindly reminded us about? From steve+comp.lang.python at pearwood.info Wed Jul 3 14:29:51 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 03 Jul 2013 18:29:51 GMT Subject: python adds an extra half space when reading from a string or list References: <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> <448b55c5-f0c6-4e16-9a9f-1749d9208397@googlegroups.com> <79dd34f7-f1df-4d4f-9f9b-60ddc28b6754@googlegroups.com> <3f58609f-c249-4d2d-bfe4-70ed153f343c@googlegroups.com> Message-ID: <51d46d9e$0$29999$c3e8da3$5496439d@news.astraweb.com> On Wed, 03 Jul 2013 11:08:11 -0700, rusi wrote: > And when Nikos moves up from petty criminal status to responsible > citizen, "Petty criminal status"? /headdesk -- Steven From rurpy at yahoo.com Wed Jul 3 15:18:58 2013 From: rurpy at yahoo.com (rurpy at yahoo.com) Date: Wed, 3 Jul 2013 12:18:58 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: <3f58609f-c249-4d2d-bfe4-70ed153f343c@googlegroups.com> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> <448b55c5-f0c6-4e16-9a9f-1749d9208397@googlegroups.com> <79dd34f7-f1df-4d4f-9f9b-60ddc28b6754@googlegroups.com> <3f58609f-c249-4d2d-bfe4-70ed153f343c@googlegroups.com> Message-ID: <0b7aa01c-ea0a-4eb2-8c61-ff94006e6c8c@googlegroups.com> On 07/03/2013 12:08 PM, rusi wrote: > On Wednesday, July 3, 2013 10:31:23 PM UTC+5:30, ru... at yahoo.com wrote: >> Are the existence of laws against beating people up negated because >> you told them in advance? Or negated because they "deserve" the >> beating? > > One of the fundamental purpose of laws is to legalize what you call > 'beating-up' Goes by the names like jail-sentences, fines and > ultimately capital punishment. Thanks, I hadn't realized there were laws requiring Nikos to be competent to run a web server business. Next time I interact with Nikos I'll inform him that he is acting illegally and C.L.P. has been empowered (I'm still a little fuzzy about by who) to punish him! > Civilization would not be possible without these. > > BTW the etymological roots of 'civilization' come from 16th century > usages such as "I shall civilise thee with my rod" Again thanks! That never occurred to me. Since "civilization" is a word in the *English* language and since we English speakers (especially us Christian ones!!) define morality and civilization for the world, then that word's root is really significant! >> Does it make any difference if the beating is for their own good? >> (In the beater's opinion?) (In the majority opinion?) > > As a good Christian I believe that Chris tried more than anyone else > on this list to help Nikos before talking recourse to another gem of > biblical wisdom: He that spareth his rod hateth his son: but he that > loveth him chasteneth him betimes. Wow! And yet again thanks! Now that I know I can just apply Christian ethics, my questions are answered. You know, those ethics that results in the crusades against the Islamic world, the ethics that thought it a great idea to torture heretics to death in the Inquisition, the ethics that resulted in the genocide of millions of South, Central and North American indigenous people who refused to accept Jesus and persisted in their heathen devil-worshiping ways? The ethics that even today deny abortions to women at risk of death carrying pregnancy to full term? I'll will go and reread my bible to figure out how to deal with Nikos. That's what I love about the Internet!!! Post a question and one gets an answer in minutes! No need to think or study all those dull old philosophy and sociology books. >[...snip...] From steve+comp.lang.python at pearwood.info Wed Jul 3 23:08:48 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 04 Jul 2013 03:08:48 GMT Subject: python adds an extra half space when reading from a string or list References: <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> <448b55c5-f0c6-4e16-9a9f-1749d9208397@googlegroups.com> <79dd34f7-f1df-4d4f-9f9b-60ddc28b6754@googlegroups.com> <3f58609f-c249-4d2d-bfe4-70ed153f343c@googlegroups.com> <0b7aa01c-ea0a-4eb2-8c61-ff94006e6c8c@googlegroups.com> Message-ID: <51d4e740$0$29999$c3e8da3$5496439d@news.astraweb.com> On Wed, 03 Jul 2013 12:18:58 -0700, rurpy wrote: > On 07/03/2013 12:08 PM, rusi wrote: >> On Wednesday, July 3, 2013 10:31:23 PM UTC+5:30, ru... at yahoo.com wrote: >>> Are the existence of laws against beating people up negated because >>> you told them in advance? Or negated because they "deserve" the >>> beating? >> >> One of the fundamental purpose of laws is to legalize what you call >> 'beating-up' Goes by the names like jail-sentences, fines and >> ultimately capital punishment. > > Thanks, I hadn't realized there were laws requiring Nikos to be > competent to run a web server business. Next time I interact with Nikos > I'll inform him that he is acting illegally and C.L.P. has been > empowered (I'm still a little fuzzy about by who) to punish him! Some weeks ago, some helpful (not) soul tried contacting the Greek government to report Nikos to the police. >> Civilization would not be possible without these. >> >> BTW the etymological roots of 'civilization' come from 16th century >> usages such as "I shall civilise thee with my rod" > > Again thanks! That never occurred to me. Since "civilization" is a > word in the *English* language and since we English speakers > (especially us Christian ones!!) define morality and civilization for > the world, then that word's root is really significant! For the record, the above is folk etymology, i.e. wrong. The English word "civilization" is ultimately derived from the Latin "civis", meaning "citizen". The suggestion that it is derived from a 16th century word meaning "to beat or to hit" is, well, bullshit. The first use of "civilization" in recorded English is from the 18th century, 1704, with the meaning of the legal process of turning a criminal case into a civil case. It didn't get it's modern meaning of the opposite of barbarism until the second half of the 18th century. The first recorded use of "civilize", or -ise, meaning to enlighten, refine, educate, bring into the standards of civilized society, occurred in the early 17th century, 1600-1629. -- Steven From ben+python at benfinney.id.au Wed Jul 3 20:53:44 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 04 Jul 2013 10:53:44 +1000 Subject: python adds an extra half space when reading from a string or list References: <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> <448b55c5-f0c6-4e16-9a9f-1749d9208397@googlegroups.com> <79dd34f7-f1df-4d4f-9f9b-60ddc28b6754@googlegroups.com> <3f58609f-c249-4d2d-bfe4-70ed153f343c@googlegroups.com> Message-ID: <7wli5n16xz.fsf@benfinney.id.au> rusi writes: > As a good Christian I believe that Chris tried more than anyone else > on this list to help Nikos before talking recourse to another gem of > biblical wisdom: > He that spareth his rod hateth his son: but he that loveth him > chasteneth him betimes. Good Christian morality entails biblical encouragement to beat one's child with a rod, I see. Please, may I be spared encounters with good Christians. Let's end right now the insidious doctrine that beating a person ? metaphorically or otherwise ? is ever acceptable in this forum. If that contradicts anyone's good Christian morality, then good Christian morality is dead wrong and needs to be rejected. -- \ ?Wrinkles should merely indicate where smiles have been.? ?Mark | `\ Twain, _Following the Equator_ | _o__) | Ben Finney From oscar.j.benjamin at gmail.com Wed Jul 3 21:08:31 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 4 Jul 2013 02:08:31 +0100 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <7wli5n16xz.fsf@benfinney.id.au> References: <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> <448b55c5-f0c6-4e16-9a9f-1749d9208397@googlegroups.com> <79dd34f7-f1df-4d4f-9f9b-60ddc28b6754@googlegroups.com> <3f58609f-c249-4d2d-bfe4-70ed153f343c@googlegroups.com> <7wli5n16xz.fsf@benfinney.id.au> Message-ID: On 4 July 2013 01:53, Ben Finney wrote: > rusi writes: > >> As a good Christian I believe that Chris tried more than anyone else >> on this list to help Nikos before talking recourse to another gem of >> biblical wisdom: > >> He that spareth his rod hateth his son: but he that loveth him >> chasteneth him betimes. > > Good Christian morality entails biblical encouragement to beat one's > child with a rod, I see. > > Please, may I be spared encounters with good Christians. > > Let's end right now the insidious doctrine that beating a person - > metaphorically or otherwise - is ever acceptable in this forum. If that > contradicts anyone's good Christian morality, then good Christian > morality is dead wrong and needs to be rejected. And also, let's end this and all the related discussions about trolling and how to deal with trolls. I can see how some are annoyed by ????? and his posts but I for one am *much more* concerned/bothered by the surrounding (often highly) unpleasant discussion by others. Oscar From rustompmody at gmail.com Wed Jul 3 23:07:12 2013 From: rustompmody at gmail.com (rusi) Date: Wed, 3 Jul 2013 20:07:12 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> <448b55c5-f0c6-4e16-9a9f-1749d9208397@googlegroups.com> <79dd34f7-f1df-4d4f-9f9b-60ddc28b6754@googlegroups.com> <3f58609f-c249-4d2d-bfe4-70ed153f343c@googlegroups.com> <7wli5n16xz.fsf@benfinney.id.au> Message-ID: <79d82a8c-6ee5-46b6-9b07-a2fd1213b87b@googlegroups.com> On Thursday, July 4, 2013 6:38:31 AM UTC+5:30, Oscar Benjamin wrote: > And also, let's end this and all the related discussions about > trolling and how to deal with trolls. I can see how some are annoyed > by ????? and his posts but I for one am *much more* concerned/bothered > by the surrounding (often highly) unpleasant discussion by others. Yes count me too as one of those concerned about other post(ers) more than Nikos. And I too am stepping out of this brawl. I got into it because I felt Chris had done more service to Nikos and the list than others and then was being misrepresented. Anyway there is a very good reason to stop these 'discussions' -- the problem being discussed is too much in-band with the discussions. From rurpy at yahoo.com Thu Jul 4 01:44:14 2013 From: rurpy at yahoo.com (rurpy at yahoo.com) Date: Wed, 3 Jul 2013 22:44:14 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: <79d82a8c-6ee5-46b6-9b07-a2fd1213b87b@googlegroups.com> References: <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> <448b55c5-f0c6-4e16-9a9f-1749d9208397@googlegroups.com> <79dd34f7-f1df-4d4f-9f9b-60ddc28b6754@googlegroups.com> <3f58609f-c249-4d2d-bfe4-70ed153f343c@googlegroups.com> <7wli5n16xz.fsf@benfinney.id.au> <79d82a8c-6ee5-46b6-9b07-a2fd1213b87b@googlegroups.com> Message-ID: <3eb56d2e-1569-4c0e-b2f6-cacc667c9230@googlegroups.com> On 07/03/2013 09:07 PM, rusi wrote: >[...] > I got into it because I felt Chris had done more service to Nikos and > the list than others and then was being misrepresented. I don't know why you think I "misrepresented" him. I questioned the morality of his accepting access to Nikos' server and then doing what he did. If you think it is a black-and-white question then you are welcome to your opinion but you are wrong. Somebody needed to point out the other side even if it happens to be a minority opinion here. Those actions of Chris and their morality are independent of any other help he did or did not provide Nikos which is why I clipped that part. (For the record I could post links to a number of distinctly unhelpful posts Chris also made to Nikos but I am not disputing in general Chris' efforts here.) From nikos at superhost.gr Thu Jul 4 01:57:42 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Thu, 04 Jul 2013 08:57:42 +0300 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <3eb56d2e-1569-4c0e-b2f6-cacc667c9230@googlegroups.com> References: <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> <448b55c5-f0c6-4e16-9a9f-1749d9208397@googlegroups.com> <79dd34f7-f1df-4d4f-9f9b-60ddc28b6754@googlegroups.com> <3f58609f-c249-4d2d-bfe4-70ed153f343c@googlegroups.com> <7wli5n16xz.fsf@benfinney.id.au> <79d82a8c-6ee5-46b6-9b07-a2fd1213b87b@googlegroups.com> <3eb56d2e-1569-4c0e-b2f6-cacc667c9230@googlegroups.com> Message-ID: ???? 4/7/2013 8:44 ??, ?/? rurpy at yahoo.com ??????: > On 07/03/2013 09:07 PM, rusi wrote: >> [...] >> I got into it because I felt Chris had done more service to Nikos and >> the list than others and then was being misrepresented. > > I don't know why you think I "misrepresented" him. I questioned > the morality of his accepting access to Nikos' server and then > doing what he did. If you think it is a black-and-white question > then you are welcome to your opinion but you are wrong. Somebody > needed to point out the other side even if it happens to be a > minority opinion here. > > Those actions of Chris and their morality are independent of > any other help he did or did not provide Nikos which is why I > clipped that part. (For the record I could post links to a > number of distinctly unhelpful posts Chris also made to Nikos > but I am not disputing in general Chris' efforts here.) Actually Chris's way of responding is by giving me general hints or urls to look into and not provide relevant code to show me my errors or specific instructions on what needs to be done to solve something. And yes, since he didn't wanted to help he should have stated clearly that helping me by looking my code wasn't into his intentions. Thus, he shouldn't have logged on to my server. -- What is now proved was at first only imagined! From feedthetroll at gmx.de Thu Jul 4 04:50:26 2013 From: feedthetroll at gmx.de (feedthetroll at gmx.de) Date: Thu, 4 Jul 2013 01:50:26 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: <79dd34f7-f1df-4d4f-9f9b-60ddc28b6754@googlegroups.com> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> <448b55c5-f0c6-4e16-9a9f-1749d9208397@googlegroups.com> <79dd34f7-f1df-4d4f-9f9b-60ddc28b6754@googlegroups.com> Message-ID: <77599c6c-5572-4001-9b5c-9e28e0d5da91@googlegroups.com> Am Mittwoch, 3. Juli 2013 19:01:23 UTC+2 schrieb ru... at yahoo.com: > On 07/03/2013 08:12 AM, feedthetroll at gmx.de wrote: >>> Am Mittwoch, 3. Juli 2013 12:00:14 UTC+2 schrieb ?????: >>>>> ???? 3/7/2013 12:45 ??, ?/? Chris Angelico ??????: >>>>>>>>> ] You have betrayed the trust of all your customers. >>>>> ... >>>>> I just received a call form on of my customers asking me to explain your >>>>> mail ... >>>>> Of course i should have give you the root pass(it was indeed stupid), >>>>> but you violated my trust. >>>>> You should have been clear that you didnt want to help and not asking me >>>>> via private mail for the root pass. >>> >>> May i cite: >>> Am Dienstag, 4. Juni 2013 19:12:41 UTC+2 schrieb ???????? ??????: >>>>> ?? ?????, 4 ??????? 2013 8:09:18 ?.?. UTC+3, ? ??????? Chris Angelico >>>>> ??????: >>>>>>> On Wed, Jun 5, 2013 at 3:02 AM, ???????? ?????? >>>>>>> wrote: >>>>>>>>> I'm willing to let someone with full root access to my webhost to see >>>>>>>>> thigns from the inside. >>>>>>>>> ... >>>>>>> You need to read up on what happens when you enter Dummy Mode and give >>>>>>> someone full root access to your web host. You really REALLY need to >>>>>>> understand what that means before you offer random strangers that kind >>>>>>> of access to someone else's data. >>>>>>> >>> ************************************************************************* >>> ************************************************************************* >>>>>>> I've half a mind to take you up on your offer, then go look for >>>>>>> personal and private info from your clients, and email it to them >>>>>>> (along with a link to this thread) to point out what's going on. >>> ************************************************************************* >>> ************************************************************************* >>>>>>> ChrisA >>>>> >>>>> I know what full root access mean. >>>>> I also trust you. >>>>> ... >>> >>> Am Mittwoch, 5. Juni 2013 00:12:26 UTC+2 schrieb Chris Angelico: >>>>> The call is strong... I could rule the galaxy alongside my father... >>>>> I've searched my feelings, and I know this to be true! >>>>> >>> ************************************************************************* >>> ************************************************************************* >>>>> Okay. I accept. I'll **do as I promised.** Might be interesting, and >>>>> educative - for someone, at least. >>> ************************************************************************* >>> ************************************************************************* >>> [emphasis added for those, who do not want to read the whole post] >>> >>> Any questions? > > Yes. Ahhh! Like you I *love* to answer rhetorical questions. ;-) > If you tell someone you're going to beat them up but > they don't take you seriously because they are naive, > or they think you're kidding, or they don't understand > you because you told them in hints rather than directly, > is it ok to beat them up? Yes, or the police would have to arrest every participant of a boxing, martial arts, ... championship. > Are the existence of laws against beating people up > negated because you told them in advance? Or negated > because they "deserve" the beating? Yes, or every M.D. would be arrested. You allow him to cut your body and therefore it's O.K. > Does it make any difference if they contribute to > their own beating, say by showing up at a designated > place and time? Yes. In case of a punchfest (is this correct in english, the german word is "Massenschl?gerei") *every* participant is convicted. > Does it make any difference if the beating is for > their own good? (In the beater's opinion?) (In > the majority opinion?) Yes. In case the one who confirms the "their own good" is an accredited specialist (psychiatrist, judge [yes there is corporal punishment in many, many legal systems, established and endorsed by majority opinion], ...) > I am sorry Mr/Ms. FeedTheTroll for being so dumb. I forgive, no problem ;-) And, if you have to be personal on a public list, I would prefer Mrs. > I know the answers to all these questions are obvious > to everyone else here but I am not sure about them. Then I hope, I was able to enlighten you. ;-) btw: Highly offtopic. Therefore I'm out. From rurpy at yahoo.com Fri Jul 5 18:42:44 2013 From: rurpy at yahoo.com (rurpy at yahoo.com) Date: Fri, 5 Jul 2013 15:42:44 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: <77599c6c-5572-4001-9b5c-9e28e0d5da91@googlegroups.com> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> <448b55c5-f0c6-4e16-9a9f-1749d9208397@googlegroups.com> <79dd34f7-f1df-4d4f-9f9b-60ddc28b6754@googlegroups.com> <77599c6c-5572-4001-9b5c-9e28e0d5da91@googlegroups.com> Message-ID: On 07/04/2013 02:50 AM, feedthetroll at gmx.de wrote: > Am Mittwoch, 3. Juli 2013 19:01:23 UTC+2 schrieb ru... at yahoo.com: >[...] >>>> Any questions? >> Yes. >[...] >> I know the answers to all these questions are obvious >> to everyone else here but I am not sure about them. > > Then I hope, I was able to enlighten you. ;-) as i said, i love the internet, it's so easy to get answers to all those questions i thought were difficult and complex. You cleared everything up in just a few sentences, thanks. i'm now going to go read the truth about the 9-11 attacks. From rosuav at gmail.com Wed Jul 3 10:44:35 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Jul 2013 00:44:35 +1000 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> Message-ID: On Wed, Jul 3, 2013 at 8:00 PM, ????? wrote: > ???? 3/7/2013 12:45 ??, ?/? Chris Angelico ??????: > >>> ] You have betrayed the trust of all your customers. >>> >>> Which seemed to be accepted on this list without a problem. >> >> >> If my boss gave a random stranger from a mailing list the root >> password to one of our servers, I would say to his face that he had >> betrayed his (our) customers' trust. I would say it with strong >> emphasis and a raised tone, too, and no small heat. The words you >> quote above are perfectly factual and, in my opinion, business-like >> language. > > > I just received a call form on of my customers asking me to explain your > mail and i did tell him the complete truth. > > He was surprised by what i did my hopefully he will not leave from my > server. I'd like to have been cc'd in on that conversation; if indeed you gave him the complete truth, then there's nothing to hide, right? > any way all is well now. Until you retract your statement that you would happily give out root access again to some other random person across the internet, all is NOT well. Calling you incompetent at managing a server is like calling a doctor incompetent for surgically removing your throat to cure a common cold. It's too late, at that point, to undo what's happened, but if that same doctor still says he'd do the same again to cure another cold in another patient, I think most of us would look for another doctor. People deciding to stay with that doctor is NOT "all is well". ChrisA From nikos at superhost.gr Wed Jul 3 11:36:27 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Wed, 03 Jul 2013 18:36:27 +0300 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> Message-ID: ???? 3/7/2013 5:44 ??, ?/? Chris Angelico ??????: > On Wed, Jul 3, 2013 at 8:00 PM, ????? wrote: >> ???? 3/7/2013 12:45 ??, ?/? Chris Angelico ??????: >> >>>> ] You have betrayed the trust of all your customers. >>>> >>>> Which seemed to be accepted on this list without a problem. >>> >>> >>> If my boss gave a random stranger from a mailing list the root >>> password to one of our servers, I would say to his face that he had >>> betrayed his (our) customers' trust. I would say it with strong >>> emphasis and a raised tone, too, and no small heat. The words you >>> quote above are perfectly factual and, in my opinion, business-like >>> language. >> >> >> I just received a call form on of my customers asking me to explain your >> mail and i did tell him the complete truth. >> >> He was surprised by what i did my hopefully he will not leave from my >> server. > > I'd like to have been cc'd in on that conversation; if indeed you gave > him the complete truth, then there's nothing to hide, right? > >> any way all is well now. > > Until you retract your statement that you would happily give out root > access again to some other random person across the internet, all is > NOT well. > > Calling you incompetent at managing a server is like calling a doctor > incompetent for surgically removing your throat to cure a common cold. > It's too late, at that point, to undo what's happened, but if that > same doctor still says he'd do the same again to cure another cold in > another patient, I think most of us would look for another doctor. > People deciding to stay with that doctor is NOT "all is well". > > ChrisA > I will *not* give away my root pass to anyone for any reason but i will open a norla user account for someone if i feel like trusting him and copy my python file to his homr dir to take alook from within. -- What is now proved was at first only imagined! From rosuav at gmail.com Wed Jul 3 11:44:19 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Jul 2013 01:44:19 +1000 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> Message-ID: On Thu, Jul 4, 2013 at 1:36 AM, ????? wrote: > I will *not* give away my root pass to anyone for any reason but i will open > a norla user account for someone if i feel like trusting him and copy my > python file to his homr dir to take alook from within. Well... well... baby steps. That's something at least. That's still a huge level of access, though; with a non-root account on your server, I would be able to - I think - read all your customers' code. You would have to chroot the user you give, and if you're going to do that, you may as well just give the code as a .py file. Really, you need to have a MUCH stronger respect for shell access, even non-root. ChrisA From square.steve at gmail.com Wed Jul 3 12:42:50 2013 From: square.steve at gmail.com (Steve Simmons) Date: Wed, 03 Jul 2013 17:42:50 +0100 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> Message-ID: <51D4548A.40607@gmail.com> On 03/07/2013 16:44, Chris Angelico wrote: > On Thu, Jul 4, 2013 at 1:36 AM, ????? wrote: >> I will *not* give away my root pass to anyone for any reason but i will open >> a norla user account for someone if i feel like trusting him and copy my >> python file to his homr dir to take alook from within. > Well... well... baby steps. That's something at least. That's still a > huge level of access, though; with a non-root account on your server, > I would be able to - I think - read all your customers' code. You > would have to chroot the user you give, and if you're going to do > that, you may as well just give the code as a .py file. Really, you > need to have a MUCH stronger respect for shell access, even non-root. > > ChrisA Nicos A hard lesson learnt, I think. I have read most of the responses to your posts but kept my contributions to a minimum. Here's my advice to you: 1. Don't trust ANYBODY on the internet unless you have thought carefully about what you are being offered. 2. Do seriously consider following advice from this list, especially the advice to read external references and documents - obviously subject to point 1 :-) 3. Don't EVER compromise security for some real or imagined deadline - your customers will probably grumble if you are late but they will likely sue you if you compromise their data. They'll definitely sue if you compromise their money in any way. Chris taught you a valuable lesson - hard but valuable. 4. Take a few hours out and re-read your recent threads. Pick out the constructive advice you have ignored and follow up on it. It may take days or even weeks to get your head around it but IMHO there is huge value to be gained from the exercise. You have taken some big strides over the past several weeks, supported by some *very* patient experts, but it is clear you still have plenty to learn - pause, read, digest, reflect and then move forward. Take care Steve From nikos at superhost.gr Wed Jul 3 13:10:40 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Wed, 03 Jul 2013 20:10:40 +0300 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> Message-ID: ???? 3/7/2013 7:42 ??, ?/? Steve Simmons ??????: > > On 03/07/2013 16:44, Chris Angelico wrote: >> On Thu, Jul 4, 2013 at 1:36 AM, ????? wrote: >>> I will *not* give away my root pass to anyone for any reason but i >>> will open >>> a norla user account for someone if i feel like trusting him and copy my >>> python file to his homr dir to take alook from within. >> Well... well... baby steps. That's something at least. That's still a >> huge level of access, though; with a non-root account on your server, >> I would be able to - I think - read all your customers' code. You >> would have to chroot the user you give, and if you're going to do >> that, you may as well just give the code as a .py file. Really, you >> need to have a MUCH stronger respect for shell access, even non-root. >> >> ChrisA > > Nicos > > A hard lesson learnt, I think. I have read most of the responses to > your posts but kept my contributions to a minimum. Here's my advice to > you: > > 1. Don't trust ANYBODY on the internet unless you have thought > carefully about what you are being offered. > 2. Do seriously consider following advice from this list, especially > the advice to read external references and documents - obviously subject > to point 1 :-) > 3. Don't EVER compromise security for some real or imagined deadline - > your customers will probably grumble if you are late but they will > likely sue you if you compromise their data. They'll definitely sue if > you compromise their money in any way. Chris taught you a valuable > lesson - hard but valuable. > 4. Take a few hours out and re-read your recent threads. Pick out the > constructive advice you have ignored and follow up on it. It may take > days or even weeks to get your head around it but IMHO there is huge > value to be gained from the exercise. > > You have taken some big strides over the past several weeks, supported > by some *very* patient experts, but it is clear you still have plenty to > learn - pause, read, digest, reflect and then move forward. > > Take care > > Steve Thanks Steven, i keep learning new thing every day that passes by. -- What is now proved was at first only imagined! From nikos at superhost.gr Wed Jul 3 12:47:24 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Wed, 03 Jul 2013 19:47:24 +0300 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> Message-ID: ???? 3/7/2013 6:44 ??, ?/? Chris Angelico ??????: > On Thu, Jul 4, 2013 at 1:36 AM, ????? wrote: >> I will *not* give away my root pass to anyone for any reason but i will open >> a norla user account for someone if i feel like trusting him and copy my >> python file to his homr dir to take alook from within. > > Well... well... baby steps. That's something at least. That's still a > huge level of access, though; with a non-root account on your server, > I would be able to - I think - read all your customers' code. You > would have to chroot the user you give, and if you're going to do > that, you may as well just give the code as a .py file. Really, you > need to have a MUCH stronger respect for shell access, even non-root. > > ChrisA > I did not understand you. How with a normal user account named "chris" how will you be able to ready my customers html files and even my python scripts? I feel the urge to open you one just to see if you can do it or not.....but i'm also scared.... -- What is now proved was at first only imagined! From rosuav at gmail.com Wed Jul 3 12:53:17 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Jul 2013 02:53:17 +1000 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> Message-ID: On Thu, Jul 4, 2013 at 2:47 AM, ????? wrote: > ???? 3/7/2013 6:44 ??, ?/? Chris Angelico ??????: >> >> On Thu, Jul 4, 2013 at 1:36 AM, ????? wrote: >>> >>> I will *not* give away my root pass to anyone for any reason but i will >>> open >>> a norla user account for someone if i feel like trusting him and copy my >>> python file to his homr dir to take alook from within. >> >> >> Well... well... baby steps. That's something at least. That's still a >> huge level of access, though; with a non-root account on your server, >> I would be able to - I think - read all your customers' code. You >> would have to chroot the user you give, and if you're going to do >> that, you may as well just give the code as a .py file. Really, you >> need to have a MUCH stronger respect for shell access, even non-root. >> >> ChrisA >> > I did not understand you. > > How with a normal user account named "chris" how will you be able to ready > my customers html files and even my python scripts? > > I feel the urge to open you one just to see if you can do it or not.....but > i'm also scared.... What are the file permissions (file modes) on all your home directories? Do you know what they mean? I'm happy to take you up on that offer if you need another lesson in not giving out shell access. And don't forget, privilege escalation attacks do exist. ChrisA From nikos at superhost.gr Wed Jul 3 13:07:13 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Wed, 03 Jul 2013 20:07:13 +0300 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> Message-ID: ???? 3/7/2013 7:53 ??, ?/? Chris Angelico ??????: > On Thu, Jul 4, 2013 at 2:47 AM, ????? wrote: >> ???? 3/7/2013 6:44 ??, ?/? Chris Angelico ??????: >>> >>> On Thu, Jul 4, 2013 at 1:36 AM, ????? wrote: >>>> >>>> I will *not* give away my root pass to anyone for any reason but i will >>>> open >>>> a norla user account for someone if i feel like trusting him and copy my >>>> python file to his homr dir to take alook from within. >>> >>> >>> Well... well... baby steps. That's something at least. That's still a >>> huge level of access, though; with a non-root account on your server, >>> I would be able to - I think - read all your customers' code. You >>> would have to chroot the user you give, and if you're going to do >>> that, you may as well just give the code as a .py file. Really, you >>> need to have a MUCH stronger respect for shell access, even non-root. >>> >>> ChrisA >>> >> I did not understand you. >> >> How with a normal user account named "chris" how will you be able to ready >> my customers html files and even my python scripts? >> >> I feel the urge to open you one just to see if you can do it or not.....but >> i'm also scared.... > > What are the file permissions (file modes) on all your home > directories? Do you know what they mean? root at nikos [~]# ls -al /home total 88 drwx--x--x 22 root root 4096 Jul 3 20:03 ./ drwxr-xr-x 22 root root 4096 Jun 12 01:21 ../ drwx--x--x 14 akis akis 4096 Apr 5 22:21 akis/ same with others just +x for group and others. Does that mean you can easily i.e. 'cd /home/akis/' accessing their home directories? Shall i 'chmod -x /home/dirs' ? > I'm happy to take you up on that offer if you need another lesson in > not giving out shell access. And don't forget, privilege escalation > attacks do exist. Yes they do, but cPanel offers some protection against these kind of methods called "CPHulk" so it wont be easy! -- What is now proved was at first only imagined! From rosuav at gmail.com Wed Jul 3 13:23:34 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Jul 2013 03:23:34 +1000 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> Message-ID: On Thu, Jul 4, 2013 at 3:07 AM, ????? wrote: > ???? 3/7/2013 7:53 ??, ?/? Chris Angelico ??????: >> What are the file permissions (file modes) on all your home >> directories? Do you know what they mean? > > > root at nikos [~]# ls -al /home > total 88 > drwx--x--x 22 root root 4096 Jul 3 20:03 ./ > drwxr-xr-x 22 root root 4096 Jun 12 01:21 ../ > drwx--x--x 14 akis akis 4096 Apr 5 22:21 akis/ > same with others just +x for group and others. > > Does that mean you can easily i.e. 'cd /home/akis/' accessing their home > directories? Yes. > Shall i 'chmod -x /home/dirs' ? Only if you know what it will do. Your solutions to problems always seem to be "If I do this, will the problem be fixed?" without demonstrating any understanding of what will be changed. Maybe you do know and aren't showing it, but I suspect that (in many cases at least) you simply do not understand what you are doing. >> I'm happy to take you up on that offer if you need another lesson in >> not giving out shell access. And don't forget, privilege escalation >> attacks do exist. > > > Yes they do, but cPanel offers some protection against these kind of methods > called "CPHulk" so it wont be easy! Neat. Now I know how to lock you out of your own account. Five seconds with Google brought this up: http://docs.cpanel.net/twiki/bin/view/11_30/WHMDocs/CPHulk Can you, by reading that page, tell me what I would have to do to stop you from accessing your login? Also, CPHulk does not appear to have _any_ protection against privilege escalation. It's a completely different thing. So once again, it appears - maybe that appearance is wrong - that you have done something that "ought to fix security" without knowing anything about what it actually does. ChrisA From nikos at superhost.gr Wed Jul 3 13:50:15 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Wed, 03 Jul 2013 20:50:15 +0300 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> Message-ID: ???? 3/7/2013 8:23 ??, ?/? Chris Angelico ??????: >>> What are the file permissions (file modes) on all your home >>> directories? Do you know what they mean? >> >> >> root at nikos [~]# ls -al /home >> total 88 >> drwx--x--x 22 root root 4096 Jul 3 20:03 ./ >> drwxr-xr-x 22 root root 4096 Jun 12 01:21 ../ >> drwx--x--x 14 akis akis 4096 Apr 5 22:21 akis/ >> same with others just +x for group and others. >> >> Does that mean you can easily i.e. 'cd /home/akis/' accessing their home >> directories? > > Yes. You can cd to the other users home directories but you wont be able to view their files because of the lack of +r attribute. But i'll remove it just in case by: chmod go-x -R /home/whatever_username >> Yes they do, but cPanel offers some protection against these kind of methods >> called "CPHulk" so it wont be easy! > > Neat. Now I know how to lock you out of your own account. Five seconds > with Google brought this up: > > http://docs.cpanel.net/twiki/bin/view/11_30/WHMDocs/CPHulk > > Can you, by reading that page, tell me what I would have to do to stop > you from accessing your login? yes constantly ping my server by faking you ip address as my ip address so to force CPHulk to refuse to let me login. Of course the same page provides a means of how to unlock myself in case that happens. > Also, CPHulk does not appear to have _any_ protection against > privilege escalation. It's a completely different thing. Yes, it does not. Its mostly a way to block nmap requests of pinging and identifying of services running on the server itself. From nikos at superhost.gr Thu Jul 4 06:06:20 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Thu, 04 Jul 2013 13:06:20 +0300 Subject: Fwd: Re: python adds an extra half space when reading from a string or list In-Reply-To: References: Message-ID: ???? 3/7/2013 8:23 ??, ?/? Chris Angelico ??????: >>> What are the file permissions (file modes) on all your home >>> directories? Do you know what they mean? >> >> >> root at nikos [~]# ls -al /home >> total 88 >> drwx--x--x 22 root root 4096 Jul 3 20:03 ./ >> drwxr-xr-x 22 root root 4096 Jun 12 01:21 ../ >> drwx--x--x 14 akis akis 4096 Apr 5 22:21 akis/ >> same with others just +x for group and others. >> >> Does that mean you can easily i.e. 'cd /home/akis/' accessing their home >> directories? > > Yes. You can cd to the other users home directories but you wont be able to view their files because of the lack of +r attribute. But i'll remove it just in case by: chmod go-x -R /home/whatever_username >> Yes they do, but cPanel offers some protection against these kind of methods >> called "CPHulk" so it wont be easy! > > Neat. Now I know how to lock you out of your own account. Five seconds > with Google brought this up: > > http://docs.cpanel.net/twiki/bin/view/11_30/WHMDocs/CPHulk > > Can you, by reading that page, tell me what I would have to do to stop > you from accessing your login? yes constantly ping my server by faking you ip address as my ip address so to force CPHulk to refuse to let me login. Of course the same page provides a means of how to unlock myself in case that happens. > Also, CPHulk does not appear to have _any_ protection against > privilege escalation. It's a completely different thing. Yes, it does not. Its mostly a way to block nmap requests of pinging and identifying of services running on the server itself. -- What is now proved was at first only imagined! From rustompmody at gmail.com Wed Jul 3 08:31:54 2013 From: rustompmody at gmail.com (rusi) Date: Wed, 3 Jul 2013 05:31:54 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> Message-ID: <01597ae1-cad3-4ed0-85ca-857b7eb2f3df@googlegroups.com> On Wednesday, July 3, 2013 3:15:15 PM UTC+5:30, Chris Angelico wrote: > If my boss gave a random stranger from a mailing list the root > password to one of our servers, I would say to his face that he had > betrayed his (our) customers' trust. I would say it with strong > emphasis and a raised tone, too, and no small heat. The words you > quote above are perfectly factual and, in my opinion, business-like > language. I am guessing Chris (do correct me if wrong) that you are making a theoretical/hypothetical statement. I guess your boss is not an asshole in matters of things like passwords so you are making a vacuous statement. [For those not familiar with the terminology: "A implies B is vacuously true, if A is 'grounded' to false. Then B can be anything wild] Speaking as one who has had to change jobs because of impossible bosses... From rosuav at gmail.com Wed Jul 3 10:37:00 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Jul 2013 00:37:00 +1000 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <01597ae1-cad3-4ed0-85ca-857b7eb2f3df@googlegroups.com> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> <51D3ED24.9030202@rece.vub.ac.be> <01597ae1-cad3-4ed0-85ca-857b7eb2f3df@googlegroups.com> Message-ID: On Wed, Jul 3, 2013 at 8:04 PM, Antoon Pardon wrote: >> If my boss gave a random stranger from a mailing list the root >> password to one of our servers, I would say to his face that he had >> betrayed his (our) customers' trust. I would say it with strong >> emphasis and a raised tone, too, and no small heat. The words you >> quote above are perfectly factual and, in my opinion, business-like >> language. >> >> ChrisA > > But the real question of course is, would your boss see it that way? > Maybe your boss would, but talking about a random boss, I wouldn't > bet any money on it. On Wed, Jul 3, 2013 at 10:31 PM, rusi wrote: > I am guessing Chris (do correct me if wrong) that you are making a theoretical/hypothetical statement. I guess your boss is not an asshole in matters of things like passwords so you are making a vacuous statement. > > [For those not familiar with the terminology: "A implies B is vacuously true, if A is 'grounded' to false. Then B can be anything wild] > > Speaking as one who has had to change jobs because of impossible bosses... Speaking about my current boss, rusi's right that the situation will never come up. He errs on the side of paranoia, not sloppiness. (And before I criticize him for the paranoia, I remind myself of this[1] PHDComics strip.) But if I had a boss who actually did that, then I would say my piece, and then my continued employment would depend on how he took it. If he accepts the criticism and moves on, then we look to damage control; if not, then I look to getting a new job. I don't think I would ever mind get fired for making that sort of statement - if it gets me fired, I'm probably happier out of there anyway. ChrisA [1] http://www.phdcomics.com/comics/archive.php?comicid=640 - start at http://www.phdcomics.com/comics/archive.php?comicid=638 for context From rurpy at yahoo.com Wed Jul 3 13:11:02 2013 From: rurpy at yahoo.com (rurpy at yahoo.com) Date: Wed, 3 Jul 2013 10:11:02 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> Message-ID: On 07/03/2013 03:21 AM, Antoon Pardon wrote: > Op 03-07-13 02:30, rurpy at yahoo.com schreef: >> If your going to point out something negative about someone >> then do so politely. Ask yourself if you were pointing out >> incompetence to your boss (or anyone else where impoliteness >> could have real consequences for you) if you would say, >> "you're incompetent." > > And so we shift from no problem speaking bluntly or clearly > to wording it in a way that wouldn't antagonize your boss > too much. As I pointed out, emotionally-loaded, judgmental language *is not* clear. And yes, I think "wouldn't antagonize your boss" is not a bad heuristic for judging the politeness of your response. You shouldn't be asking for respect from other posters here if you're not willing to provide the same respect IMO. But even if you reject that there is the fundamental nature of people: that if you disrespect others they are likely to respond in kind and (IMO) most people here find the resulting climate unpleasant. You can't do much about trolls who occasionally show up and disrespect you but you can promote a climate of respect by not responding in kind. > Off course that would also mean throwing out remarks like: > > ] You have betrayed the trust of all your customers. > > Which seemed to be accepted on this list without a problem. There are many things "accepted on this list" that are questionable. Again I ask, what does his web site admin skills or lack thereof have to do with python? If you want to decline providing Python help to someone because you don't like some real-world behavior of the person, fine. But when you attack him over it, and publicly engage in a long, noisy discussion here in which you trumpet your moral and technical superiority, then you should not be surprised when the target takes offense and responds in kind. If someone from Wikileaks posts here seeking Python help, should we engage in a long discussion about the morality of Wikileaks and how they aid US fugitives from justice? How about someone who lets slip he's just been released from prison for child sexual abuse? How about someone who's writing software for bulk mailing? How about someone who is writing membership management software for the American Nazi Party? >> Please use non-emotional, neutral, factual descriptions >> and only do so when it is actually relevant. IOW, please >> resist your desire to "tell off" the poster -- it usually >> just produces more responses in kind. > > This is often not workable. Limiting to factual description > means that you often can't summarize a list of such factual > descriptions into a conclusion. You can list 8 examples of > someone betraying the trust of his customers but you can't > summarize it into: "is/behaves untrustworthy to his customers," > even if all signs point to this person going to continue in the > same vein. > > It is limiting yourself into pointing out all the trees > without being allowed to call it a forest. You can summarize while being polite and non-judgmental. You do not have state your belief on every off-topic inflammatory subject that happens to come up. And if you don't want to help someone because of something you don't like about them, you don't have to. But you also don't have to tell all of us why we should agree with you or try to publicly bully into not helping those who don't share your opinion. From antoon.pardon at rece.vub.ac.be Thu Jul 4 08:09:29 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Thu, 04 Jul 2013 14:09:29 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> Message-ID: <51D565F9.4020900@rece.vub.ac.be> Op 03-07-13 19:11, rurpy at yahoo.com schreef: > On 07/03/2013 03:21 AM, Antoon Pardon wrote: >> Op 03-07-13 02:30, rurpy at yahoo.com schreef: >>> If your going to point out something negative about someone >>> then do so politely. Ask yourself if you were pointing out >>> incompetence to your boss (or anyone else where impoliteness >>> could have real consequences for you) if you would say, >>> "you're incompetent." >> And so we shift from no problem speaking bluntly or clearly >> to wording it in a way that wouldn't antagonize your boss >> too much. > As I pointed out, emotionally-loaded, judgmental language > *is not* clear. Well that is true, but mostly in the trivial sense that language is rarely clear even when you are talking facts. When I meet someone new and I talk about my love of spaghetti and the other inivites me to the spaghetti evening the next day, that can turn out to be a big disappointment because when I talk about spaghetti, I mean a carbonarra while I was invited to a bolognaise-evening. > And yes, I think "wouldn't antagonize your > boss" is not a bad heuristic for judging the politeness of > your response. That may be true for you personnally, but you are unsufficiently clear for anyone else to be of any help. The problem is that when you wrote this, you had a specific kind of boss in mind who would react in certain ways to certain kinds of treatment. However it would be extremely unlikely that other people would come up with the same idea of boss. And not everybody is in the same situation, some people can't afford to lose there job, others are in a less desperate situation, for some people their priority is their career, while for others it is the service to their clients. All these people are going to come up with wildly different answers. > Again I ask, what does his web site admin skills or lack > thereof have to do with python? > > If you want to decline providing Python help to someone > because you don't like some real-world behavior of the > person, fine. But when you attack him over it, and > publicly engage in a long, noisy discussion here in > which you trumpet your moral and technical superiority, > then you should not be surprised when the target takes > offense and responds in kind. > > If someone from Wikileaks posts here seeking Python help, > should we engage in a long discussion about the morality > of Wikileaks and how they aid US fugitives from justice? > > How about someone who lets slip he's just been released > from prison for child sexual abuse? > > How about someone who's writing software for bulk mailing? > > How about someone who is writing membership management > software for the American Nazi Party? Are you saying we should either help the person with his (python) problem or decline any help no matter how nefarious the goals he wants to accomplish or are you saying these examples are not serious enough so people should show some tolerance in these cases? >>> Please use non-emotional, neutral, factual descriptions >>> and only do so when it is actually relevant. IOW, please >>> resist your desire to "tell off" the poster -- it usually >>> just produces more responses in kind. >> This is often not workable. Limiting to factual description >> means that you often can't summarize a list of such factual >> descriptions into a conclusion. You can list 8 examples of >> someone betraying the trust of his customers but you can't >> summarize it into: "is/behaves untrustworthy to his customers," >> even if all signs point to this person going to continue in the >> same vein. >> >> It is limiting yourself into pointing out all the trees >> without being allowed to call it a forest. > You can summarize while being polite and non-judgmental. Somethings are not expressable in a way that is acceptable to who you are talking too, simply because they find the fact or opinion to be hurtful/insulting in itself. > You do not have state your belief on every off-topic > inflammatory subject that happens to come up. So what do you suggest? That we simply let those who bring up an off-topic inflammatory subject, go on about it whithout challenge? -- Antoon Pardon From rurpy at yahoo.com Fri Jul 5 18:40:38 2013 From: rurpy at yahoo.com (rurpy at yahoo.com) Date: Fri, 5 Jul 2013 15:40:38 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> Message-ID: On 07/04/2013 06:09 AM, Antoon Pardon wrote: > Op 03-07-13 19:11, rurpy at yahoo.com schreef: >> On 07/03/2013 03:21 AM, Antoon Pardon wrote: >>> Op 03-07-13 02:30, rurpy at yahoo.com schreef: >>>> If your going to point out something negative about someone >>>> then do so politely. Ask yourself if you were pointing out >>>> incompetence to your boss (or anyone else where impoliteness >>>> could have real consequences for you) if you would say, >>>> "you're incompetent." >>> And so we shift from no problem speaking bluntly or clearly >>> to wording it in a way that wouldn't antagonize your boss >>> too much. >> As I pointed out, emotionally-loaded, judgmental language >> *is not* clear. > > Well that is true, but mostly in the trivial sense that > language is rarely clear even when you are talking facts. Its true in more than a trivial sense. > When I meet someone new and I talk about my love of spaghetti > and the other inivites me to the spaghetti evening the next > day, that can turn out to be a big disappointment because > when I talk about spaghetti, I mean a carbonarra while I > was invited to a bolognaise-evening. Of course there is some degree of uncertainty. But that uncertainty is relatively small when compared with the range of possible food you might have been served had your evening's companion not specified "spaghetti". Had he or she invited you to a really "delicious" meal on the other hand, the uncertainty about the what "delicious" means would far greater. Unless you know the person, "delicious" is so subjective as to have very little meaning since its meaning varies between people much more than "spaghetti" does. Describing someone as "stupid", "incompetent", "a dick", etc has a similar high degree of subjectivity and its meaning depends more on the sayer than on any objective attribute of the subject. If you can tell me something objective about the subject, then that may be helpful to me in deciding how to respond to him or her. If you just spout subjective invective, then its just noise because I don't know enough about you to trust your judgment. (Advanced technical python knowledge does not qualify one in judging other human beings.) And since I'm reading the thread I have access to the same info you do, and can form my own subjective opinion. Judging other people is in my opinion a moral action that is too important to delegate or to do by just going along with the crowd. So such subjective, emotionally-loaded, judgmental responses provide little benefit to others, amplify whatever negative tone was created by the troll and stimulate the troll. The only benefits are to you who gets to vent and argue, and a (hopefully few) voyeurs and fellow vigilantes who enjoy watching and joining in on that kind of flamage. >> And yes, I think "wouldn't antagonize your >> boss" is not a bad heuristic for judging the politeness of >> your response. > > That may be true for you personnally, but you are unsufficiently > clear for anyone else to be of any help. The problem is that > when you wrote this, you had a specific kind of boss in mind [...] No. I originally wrote "your boss (or anyone else where impoliteness could have real consequences for you)." I believe most people would read that as intended, some sort of generalized authority figure, if not boss then maybe a police officer, or a powerful politician that could quaff your permit application, or the touchy father of a woman you want to date, and not necessarily "Mr. Joel Davis, my boss at this job at this moment and also my golfing buddy." > who would react in certain ways to certain kinds of treatment. > However it would be extremely unlikely that other people > would come up with the same idea of boss. And not everybody > is in the same situation, some people can't afford to lose > there job, others are in a less desperate situation, for > some people their priority is their career, while for others > it is the service to their clients. All these people are going > to come up with wildly different answers. The answers will presumably share a common characteristic: the need to address the person with politeness and respect, even when expressing disagreement with them. >> Again I ask, what does his web site admin skills or lack >> thereof have to do with python? >> >> If you want to decline providing Python help to someone >> because you don't like some real-world behavior of the >> person, fine. But when you attack him over it, and >> publicly engage in a long, noisy discussion here in >> which you trumpet your moral and technical superiority, >> then you should not be surprised when the target takes >> offense and responds in kind. >> >> If someone from Wikileaks posts here seeking Python help, >> should we engage in a long discussion about the morality >> of Wikileaks and how they aid US fugitives from justice? >> >> How about someone who lets slip he's just been released >> from prison for child sexual abuse? >> >> How about someone who's writing software for bulk mailing? >> >> How about someone who is writing membership management >> software for the American Nazi Party? > > Are you saying we should either help the person with his > (python) problem or decline any help no matter how nefarious > the goals he wants to accomplish or are you saying these > examples are not serious enough so people should show some > tolerance in these cases? The former. >>>> Please use non-emotional, neutral, factual descriptions >>>> and only do so when it is actually relevant. IOW, please >>>> resist your desire to "tell off" the poster -- it usually >>>> just produces more responses in kind. >>> This is often not workable. Limiting to factual description >>> means that you often can't summarize a list of such factual >>> descriptions into a conclusion. You can list 8 examples of >>> someone betraying the trust of his customers but you can't >>> summarize it into: "is/behaves untrustworthy to his customers," >>> even if all signs point to this person going to continue in the >>> same vein. >>> >>> It is limiting yourself into pointing out all the trees >>> without being allowed to call it a forest. >> You can summarize while being polite and non-judgmental. > > Somethings are not expressable in a way that is acceptable > to who you are talking too, simply because they find the > fact or opinion to be hurtful/insulting in itself. And so? That someone may be hurt or insulted by a polite reasoned response obviously does not mean that impolite emotional flamage is better. >> You do not have state your belief on every off-topic >> inflammatory subject that happens to come up. > > So what do you suggest? That we simply let those who > bring up an off-topic inflammatory subject, go on > about it whithout challenge? Yes, that's what "don't feed the troll" means. If you are going to come back with, that means we the group (or I the reader) acquiesce to whatever vile filth is posted here, no it doesn't mean that. It means that instead of highlighting it and encouraging more of it by responding (which increases its credibility), its impact is minimized by the lack of attention it receives, and that in turn generally results in the miscreant leaving sooner rather than later. If you are going to come back with, then the poster of vile filth will, unchallenged, just go on doing it. No, experience (although you dismiss it) shows that generally such people will get tired of being ignored and will move on. And keep in mind that even if "everyone" here adopted a "don't feed the troll" ethic, trolls would not be greeted with absolute silence -- there will always be enough non- compliance that someone will tell off the troll. The idea is that by promoting a "don't feed the troll" policy, there will only be one or two such responses followed by a couple of "don't feed the troll" posts followed by quiet (except for the troll, who, getting no further responses, to his trolling eventually gives up and seeks food elsewhere.) And in more complex cases like Nikos, whose posts are a mixture of trolling and requests for help, the trollish part will be ignored as above and the help will or will not eventually dry up as he wears out the patience of those willing to help or learns to frame requests in more helpable form. From antoon.pardon at rece.vub.ac.be Mon Jul 8 04:19:19 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Mon, 08 Jul 2013 10:19:19 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <1264cfb9-f451-40e3-9d59-0619547c8138@googlegroups.com> Message-ID: <51DA7607.20107@rece.vub.ac.be> Op 06-07-13 00:40, rurpy at yahoo.com schreef: > On 07/04/2013 06:09 AM, Antoon Pardon wrote: >> Op 03-07-13 19:11, rurpy at yahoo.com schreef: >>> On 07/03/2013 03:21 AM, Antoon Pardon wrote: >>>> Op 03-07-13 02:30, rurpy at yahoo.com schreef: >>>>> If your going to point out something negative about someone >>>>> then do so politely. Ask yourself if you were pointing out >>>>> incompetence to your boss (or anyone else where impoliteness >>>>> could have real consequences for you) if you would say, >>>>> "you're incompetent." >>>> And so we shift from no problem speaking bluntly or clearly >>>> to wording it in a way that wouldn't antagonize your boss >>>> too much. >>> As I pointed out, emotionally-loaded, judgmental language >>> *is not* clear. >> Well that is true, but mostly in the trivial sense that >> language is rarely clear even when you are talking facts. > Its true in more than a trivial sense. > >> When I meet someone new and I talk about my love of spaghetti >> and the other inivites me to the spaghetti evening the next >> day, that can turn out to be a big disappointment because >> when I talk about spaghetti, I mean a carbonarra while I >> was invited to a bolognaise-evening. > Of course there is some degree of uncertainty. But that > uncertainty is relatively small when compared with the > range of possible food you might have been served had > your evening's companion not specified "spaghetti". > > Had he or she invited you to a really "delicious" meal > on the other hand, the uncertainty about the what > "delicious" means would far greater. Unless you know > the person, "delicious" is so subjective as to have very > little meaning since its meaning varies between people > much more than "spaghetti" does. That is not true. Subjective and unclear are not the same. And that you don't know what kind of food someone find delicious, doesn't make it unclear what kind of expectations you (try to) evoke when you promisse him a delicious meal. When you go to someone's restaurant buddies and tell them you want to cook him a delicious meal, do they have suggestions? Then those buddies will find that question clear enough to come up with useful answers. And I doubt that you can get just as useful answers by asking about objective facts. > Describing someone as "stupid", "incompetent", "a > dick", etc has a similar high degree of subjectivity > and its meaning depends more on the sayer than on any > objective attribute of the subject. I doubt that. Subjective is not the same as arbitrary. We as humans react in large degree the same to the same kind of stimuli. If someone tells me something tastes sweet, then that tells me more about what he is tasting than about him. Especially if more than one person reports the same. > If you can tell me something objective about the subject, > then that may be helpful to me in deciding how to respond > to him or her. If you just spout subjective invective, > then its just noise because I don't know enough about > you to trust your judgment. And why should you trust my objective statement? The fact that the statement is objective, doesn't mean I am qualified in making it. > (Advanced technical python > knowledge does not qualify one in judging other human > beings.) And since I'm reading the thread I have access > to the same info you do, and can form my own subjective > opinion. Judging other people is in my opinion a moral > action that is too important to delegate or to do by just > going along with the crowd. So nobody is stopping you AFAICS. > So such subjective, emotionally-loaded, judgmental responses > provide little benefit to others, amplify whatever negative > tone was created by the troll and stimulate the troll. The > only benefits are to you who gets to vent and argue, and a > (hopefully few) voyeurs and fellow vigilantes who enjoy > watching and joining in on that kind of flamage. This is just your subjective emotional evaluation. By your own words why should I trust you? Why should we see the benefits to the vigilantes and the voyeurs as little? Why should we see the tone that is created by the troll as negative? Why should we hope that the voyeurs and vigilantes are few? Doesn't this all just say more about you than about the trolls, vigilantes and voyeurs? Why should I take your judgement over that of those you call trolls, vigilantes and voyeurs? Why do you use the words "troll", "vigilante" and "voyeur", which are emotionnaly laden, instead of stating objective facts? >> Are you saying we should either help the person with his >> (python) problem or decline any help no matter how nefarious >> the goals he wants to accomplish or are you saying these >> examples are not serious enough so people should show some >> tolerance in these cases? > The former. Then I strongly disagree. If at some point a pedophile would come to this group to ask for advice for his python program he's using to spy on the kids in the neighbourhood, then just declining to help is IMO a kind of moral negligence. >>> You can summarize while being polite and non-judgmental. >> Somethings are not expressable in a way that is acceptable >> to who you are talking too, simply because they find the >> fact or opinion to be hurtful/insulting in itself. > And so? That someone may be hurt or insulted by a polite > reasoned response obviously does not mean that impolite > emotional flamage is better. It doesn't mean it is worse either. Sometimes impolite emotional flammage gets the message intended better accross than a polite reasoned response. Polite reasoned responses are easier to ignore. -- Antoon Pardon From antoon.pardon at rece.vub.ac.be Wed Jul 3 06:01:43 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Wed, 03 Jul 2013 12:01:43 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> Message-ID: <51D3F687.8060901@rece.vub.ac.be> Op 02-07-13 15:40, Joshua Landau schreef: > On 2 July 2013 13:01, Antoon Pardon wrote: >> Op 02-07-13 11:34, Joshua Landau schreef: >> >>> No it does not. I'd give you more of a counter but I actually have no >>> idea how you came up with that. >> Please answer the following question. If someone behaved incompetently, >> how can I clearly state that fact when "incompetently" is seen as an >> insult and insults don't belong on the list? > There is not ever a place on this list where you will need to call > someone incompetent. You can explain to someone that they do not > understand what they are doing, but when you attack the character of > the person it is no longer acceptable. This is not an attack of character. Level of skill/competence is not in general seen as a character trait. It is something one can generally increase if one is willing to work on it, and once you acquired it, you don't have to keep your guard for situations in which you might loose it. >>>> But for your ease of mind I'll make it clear I have no intention >>>> of haunting Nikos or of keeping him the subject of discussion. >>>> But should I stumble on a conversation in which his past behaviour >>>> is framed as him being innocentltly asking questions, I will point >>>> of what bullshit that is. >>> Fair enough. If that's all you did in this thread, then I wouldn't care. >>> >>> But once again you seem to have missed the point that I and others >>> keep reiterating: pseudo-insults have no place on this list. >>> >>> >>> (If you need a reminder, pseudo-insults are just what other people >>> term "insults". You can change the name if you see fit.) >> So what are the non-insulting terms for >> >> incompentent, (starting a webservice in a language you're a newby in, >> making changes on the life server so that any typo you make, can take >> your site out the air), > You just did it. But how do I describe this in one word? What conclusion am I allowed to make from all this? Can I say: He displayed a pattern of incompetence? He has been blundering about? His skill level was unimpressive? The skill level he displayed, left much to be desired? >> inconsiderate (behave annoyingly in multiple ways and despite poeple pointing >> it out multiple times, mostly continue in the same manner, without taking >> their remarks into account) and > I do not tend to consider "inconsiderate" inappropriate if said in > earnest, as it is defensive. I'd still rather you talked about actions > as inconsiderate rather than people, but baby steps. I don't understand this, since (in)consideration is in general seen as a character trait. On that basis I think you have it exactly backwards when you consider "incompetent" an attack of character yet seem to have little problem with "inconsiderate". Yet people who have a more inconsiderate character can't work on acquiring consideration, as one can on acquiring skill/competence. Sure one can work on it, but it is something you have to keep watchful for. Otherwise you can easily slip again into a pattern of being/behaving inconsiderate. >> jerk (trying to spin your inconsiderate behaviour as you being the victim, >> misrepresenting your behaviour when it is being discussed, always "explaining" >> your behaviour, as if an explanation would make a difference to the annoyance >> you caused to others...) > You came close. Same question as two entries above. > > > But, since Nikos has thankfully ceased, I'm saying here that unless > you have good reasons otherwise I'd rather only continue this > off-list. If you reply on-list without justification I will likely not > reply. > Since others have in the mean time responded in this thread, I didn't think it a good idea, to have this part in private. -- Antoon Pardon From rustompmody at gmail.com Tue Jul 2 09:48:59 2013 From: rustompmody at gmail.com (rusi) Date: Tue, 2 Jul 2013 06:48:59 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> Message-ID: <9eca9613-7da0-4109-875c-5037beb39bf5@googlegroups.com> A plague is raging in the town A rat scampers into the room. People are harried --- A RAT! Rurpy: Rats are living beings dont you know?! Never kill a living being! Its not humanitarian, er rattatitarian. (200 more posts on humanitarianism, veganism, rattatitarianism etc) Alex: Hear Hear! But first lets rename our august assembly as rat support assembly Steven: I know how to solve this problem. Quarantining a rat for a month cures plague. Now I dont know how to do that... so I am quarantining myself. Somebody begins to protest: Steven: You dare protest?! I quarantine you... er myself. Joel: Go away Rat. Go far away. You are not a good person. You are not even a good rat. You are just rat the brat. You should take up something you can do better than this.. like maybe sleeping Chris1 (saunters into the room in shorts carrying a hockey stick): Here folks I am going to solve the problem once and for all. [Swish -- after flying across the room the rat lies still on its back] Everyone: Bravo Chris! [After a while the rat is scampering more energetically than ever before] Everyone: Aww Chris! Why did you not hit it harder?!?! Chris1: [Yawn] Im bored. Bye... Chris2: I have a solution. Here's a rat-trap. Can someone please buy me some bait? Mark: Hey rat! Ive a game for you. Willing to play? [Rat picks up its ears] Mark: Its called suicide-game. I pretend to commit suicide. You actually commit suicide. Do you like my game? [Rat ignores Mark and keeps scampering about] Antoon: Since we dont have a cure for plague (this is a couple of centuries before tetracycline) lets rename plague to troll... From steve+comp.lang.python at pearwood.info Tue Jul 2 22:37:35 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 03 Jul 2013 02:37:35 GMT Subject: OT Plague [was Re: python adds an extra half space when reading from a string or list] References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D484.5070309@rece.vub.ac.be> <51D27FB9.7040406@rece.vub.ac.be> <51D2C113.6070105@rece.vub.ac.be> <9eca9613-7da0-4109-875c-5037beb39bf5@googlegroups.com> Message-ID: <51d38e6e$0$29999$c3e8da3$5496439d@news.astraweb.com> On Tue, 02 Jul 2013 06:48:59 -0700, rusi wrote: > A plague is raging in the town > A rat scampers into the room. > People are harried --- A RAT! [...] Very imaginative, but your characterisation of people's responses to the plague rat appears to have very little in common with the actual responses you are attempting to mock. -- Steven From nikos at superhost.gr Mon Jul 1 13:15:59 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Mon, 01 Jul 2013 20:15:59 +0300 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: ???? 1/7/2013 7:56 ??, ?/? Joshua Landau ??????: > So yes, Antoon Pardon and Nikos, please stop. You are not representing > the list. I haven't followed any of the other arguments, true, but you > two in particular are causing a lot of trouble for the rest of us. It > is not hard to avoid making your disagreements public. I promise i will not reply to him any more even if still provoked. -- What is now proved was at first only imagined! From joshua.landau.ws at gmail.com Mon Jul 1 13:26:38 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Mon, 1 Jul 2013 18:26:38 +0100 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 1 July 2013 18:15, ????? wrote: > ???? 1/7/2013 7:56 ??, ?/? Joshua Landau ??????: > >> So yes, Antoon Pardon and Nikos, please stop. You are not representing >> the list. I haven't followed any of the other arguments, true, but you >> two in particular are causing a lot of trouble for the rest of us. It >> is not hard to avoid making your disagreements public. > > > I promise i will not reply to him any more even if still provoked. Thank you. From rustompmody at gmail.com Mon Jul 1 14:29:28 2013 From: rustompmody at gmail.com (rusi) Date: Mon, 1 Jul 2013 11:29:28 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Monday, July 1, 2013 10:26:21 PM UTC+5:30, Joshua Landau wrote: > So yes, Antoon Pardon and Nikos, please stop. You are not representing > the list. This 'and' is type-wrong. > I haven't followed any of the other arguments, true, but you > two in particular are causing a lot of trouble for the rest of us. This thread only makes sense in the context of the last year of threads culminating in the last month of bullshit that this list is drowning under. > It is not hard to avoid making your disagreements public. The reverse is also true in general and more appropriate in this case. Sometimes the consequences of not speaking up are more tragic than speaking too much, see http://en.wikipedia.org/wiki/First_they_came...#The_text Yeah sure, abusive posts, derogatory remarks, snide baiting etc is a problem. However when you take Alex sarcastic comment "rename comp.lang.python to comp.support.superhost" and see the seriousness behind the sarcasm, you would find a considerably bigger problem -- the python list is stopping to be a python list. In short if you agree and expand on only the first part of my request > - No discussions about Nikos specifically or trolling/flaming in general without the second > - No answers to anything he asks then we are not in agreement. Let me end with 2 points: 1. If this discussion is to go beyond name-calling and over-specific, under-effective bandaids to getting a real grasp of the problem and hopefully a real solution, it would be good if people could familiarize themselves with the paradigm called "tragedy of the commons" http://www.systems-thinking.org/arch/arch.htm#archtc 2. Steven's "I am kill-filing you for a month" will have a salutary effect on all only if we all do it. From joshua.landau.ws at gmail.com Mon Jul 1 15:16:40 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Mon, 1 Jul 2013 20:16:40 +0100 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 1 July 2013 19:29, rusi wrote: > On Monday, July 1, 2013 10:26:21 PM UTC+5:30, Joshua Landau wrote: >> So yes, Antoon Pardon and Nikos, please stop. You are not representing >> the list. > > This 'and' is type-wrong. I don't follow. >> I haven't followed any of the other arguments, true, but you >> two in particular are causing a lot of trouble for the rest of us. > > This thread only makes sense in the context of the last year of threads culminating in the last month of bullshit that this list is drowning under. Yes, but I followed that. What I haven't followed are the probably-100-post-long "wars" that are happening in other threads between specific individuals who I can't be bothered to look up. So I don't know who else has been slandering off of this thread. >> It is not hard to avoid making your disagreements public. > > The reverse is also true in general and more appropriate in this case. > Sometimes the consequences of not speaking up are more tragic than speaking too much, see > http://en.wikipedia.org/wiki/First_they_came...#The_text That's hardly relevant. Nikos isn't "targeting" any minority. > Yeah sure, abusive posts, derogatory remarks, snide baiting etc is a problem. > > However when you take Alex sarcastic comment > "rename comp.lang.python to comp.support.superhost" > and see the seriousness behind the sarcasm, you would find a considerably bigger problem -- the python list is stopping to be a python list. No it hasn't. Nikos is a fraction of this list, and over the past while has shrunk far below his peak. The people spewing insults (including Nikos, but he's said he'll stop and I respect him for that) are a much more voluminous sect of this list and a much bigger problem. > In short if you agree and expand on only the first part of my request > >> - No discussions about Nikos specifically or trolling/flaming in general I don't care whether people "discuss" Nikos. I care whether they insult people, flame people and whether they act derogatorily. That is uncalled for. Python-List is known for going off-topic, *but we should not be known for insults*. This is especially true as only two people have been insulting on this thread -- none of the others have endorsed it. If [http://en.wikipedia.org/wiki/First_they_came...#The_text] applies to any situation here, it's your lack of appreciation of the fact that these insults are harming people. > without the second > >> - No answers to anything he asks > > then we are not in agreement. And that is why I disagreed with you. I feel that your post implied either everyone "STFU" or I'm OK with insults. It's not that. It's "STFU", or don't, but insults are not OK. Ever. > Let me end with 2 points: > 1. If this discussion is to go beyond name-calling and over-specific, under-effective bandaids to getting a real grasp of the problem and hopefully a real solution, it would be good if people could familiarize themselves with the paradigm called "tragedy of the commons" > http://www.systems-thinking.org/arch/arch.htm#archtc That doesn't apply here. > 2. Steven's "I am kill-filing you for a month" will have a salutary effect on all only if we all do it. Good for him. I prefer not to kill-fire - I don't even know how yet - because it's easy enough to ignore posts. But that's fine - it's his choice and it's not a bad one. From rustompmody at gmail.com Mon Jul 1 15:29:31 2013 From: rustompmody at gmail.com (rusi) Date: Mon, 1 Jul 2013 12:29:31 -0700 (PDT) Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tuesday, July 2, 2013 12:46:40 AM UTC+5:30, Joshua Landau wrote: > On 1 July 2013 19:29, rusi wrote: > > > On Monday, July 1, 2013 10:26:21 PM UTC+5:30, Joshua Landau wrote: > > >> So yes, Antoon Pardon and Nikos, please stop. You are not representing > >> the list. > > This 'and' is type-wrong. > I don't follow. Yes that's evident. From nikos at superhost.gr Mon Jul 1 12:27:24 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Mon, 01 Jul 2013 19:27:24 +0300 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: ???? 1/7/2013 6:34 ??, ?/? Steven D'Aprano ??????: >> And no, i do not want to piss off people like you, who have spend time >> helping me. > > Too late. I asked you to stop flaming on-list, and you didn't. I am now > kill-filing you for a month. Feel grateful that it is not permanent, and > take this time to reflect that you are running out of people willing to > help you. First i saw his post and replied as i did and then i saw your reply. -- What is now proved was at first only imagined! From nikos at superhost.gr Mon Jul 1 12:36:23 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Mon, 01 Jul 2013 19:36:23 +0300 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a173$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: ???? 1/7/2013 6:34 ??, ?/? Steven D'Aprano ??????: >>> The above of course assumes that I have not kill-filed you for >>> continuing to be abusive on-list. >> So, Steven you want me to sit tight and read all the insults coming from >> this guy? >> >> If that happened to you, wouldn't you feel the need and urge to reply >> back and stand for yourself? > "Stand up for yourself" and "be abusive" are two different things. I told > you, I don't care what you do off-list, but if you continue flaming on- > list, I will kill-file you. I was name called directly, as "incompetent jerk", thats a straightforward insult, so i responded back to him in the same manner. I'm not saying it was a nice thing to do, but i just had enough. -- What is now proved was at first only imagined! From steve+comp.lang.python at pearwood.info Mon Jul 1 11:33:10 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 01 Jul 2013 15:33:10 GMT Subject: python adds an extra half space when reading from a string or list References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51d1a135$0$29999$c3e8da3$5496439d@news.astraweb.com> On Mon, 01 Jul 2013 15:08:18 +0200, Antoon Pardon wrote: > Op 01-07-13 14:43, Steven D'Aprano schreef: > >> ?????, I am not going to wade through this long, long thread to see >> what problem you are trying to solve today. > > Nikos is not trying to solve a problem in this thread. What happened is > that the original poster here got a rather short answer to his question. > Mark Laurence reacted questioning why this person didn't get the same > kind of treatment as "Nick the Incompetant Greek". Then Nikos came in > with an insulting remark. The rest developed from their. Okay, thank you for explaining the situation. Now, please stop baiting Nikos. -- Steven From antoon.pardon at rece.vub.ac.be Mon Jul 1 15:18:26 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Mon, 01 Jul 2013 21:18:26 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <51d1a135$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a135$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51D1D602.5010406@rece.vub.ac.be> Op 01-07-13 17:33, Steven D'Aprano schreef: > On Mon, 01 Jul 2013 15:08:18 +0200, Antoon Pardon wrote: > >> Op 01-07-13 14:43, Steven D'Aprano schreef: >> >>> ?????, I am not going to wade through this long, long thread to see >>> what problem you are trying to solve today. >> >> Nikos is not trying to solve a problem in this thread. What happened is >> that the original poster here got a rather short answer to his question. >> Mark Laurence reacted questioning why this person didn't get the same >> kind of treatment as "Nick the Incompetant Greek". Then Nikos came in >> with an insulting remark. The rest developed from their. > > Okay, thank you for explaining the situation. > > Now, please stop baiting Nikos. I am not baiting Nikos. I know he mentions this two week period but all I have done in the last two weeks that involves Nikos as a subject is the "Don't feed the troll thread" and today this thread. So if he feels baited for the last two weeks, it is either by that "Don't feed the troll" thread or I have very little to do with it. -- Antoon Pardon. From joshua.landau.ws at gmail.com Mon Jul 1 15:32:21 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Mon, 1 Jul 2013 20:32:21 +0100 Subject: python adds an extra half space when reading from a string or list In-Reply-To: <51D1D602.5010406@rece.vub.ac.be> References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a135$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D602.5010406@rece.vub.ac.be> Message-ID: On 1 July 2013 20:18, Antoon Pardon wrote: > Op 01-07-13 17:33, Steven D'Aprano schreef: > >> On Mon, 01 Jul 2013 15:08:18 +0200, Antoon Pardon wrote: >> >>> Op 01-07-13 14:43, Steven D'Aprano schreef: >>> >>>> ?????, I am not going to wade through this long, long thread to see >>>> what problem you are trying to solve today. >>> >>> >>> Nikos is not trying to solve a problem in this thread. What happened is >>> that the original poster here got a rather short answer to his question. >>> Mark Laurence reacted questioning why this person didn't get the same >>> kind of treatment as "Nick the Incompetant Greek". Then Nikos came in >>> with an insulting remark. The rest developed from their. >> >> >> Okay, thank you for explaining the situation. >> >> Now, please stop baiting Nikos. > > > I am not baiting Nikos. I know he mentions this two week period but all > I have done in the last two weeks that involves Nikos as a subject is > the "Don't feed the troll thread" and today this thread. > > So if he feels baited for the last two weeks, it is either by that > "Don't feed the troll" thread or I have very little to do with it. Obviously Steven, as I do, considers your actions in this thread as "baiting". Whether or not you agree with his definition of baiting, he wants you to stop. From antoon.pardon at rece.vub.ac.be Tue Jul 2 03:25:56 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Tue, 02 Jul 2013 09:25:56 +0200 Subject: python adds an extra half space when reading from a string or list In-Reply-To: References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a135$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D1D602.5010406@rece.vub.ac.be> Message-ID: <51D28084.8070704@rece.vub.ac.be> Op 01-07-13 21:32, Joshua Landau schreef: > On 1 July 2013 20:18, Antoon Pardon wrote: >> Op 01-07-13 17:33, Steven D'Aprano schreef: >> >>> Okay, thank you for explaining the situation. Now, please stop >>> baiting Nikos. >> I am not baiting Nikos. I know he mentions this two week period but all >> I have done in the last two weeks that involves Nikos as a subject is >> the "Don't feed the troll thread" and today this thread. >> >> So if he feels baited for the last two weeks, it is either by that >> "Don't feed the troll" thread or I have very little to do with it. > Obviously Steven, as I do, considers your actions in this thread as > "baiting". Whether or not you agree with his definition of baiting, he > wants you to stop. No that is not obvious at all. You see, Steven has made it clear he wasn't going to wade throught this list. So it is very well possible he missed most of my actions in this thread, which makes it far from obvious that his remark was based on actions in this thread. -- Antoon Pardon From denismfmcmahon at gmail.com Mon Jul 1 19:14:17 2013 From: denismfmcmahon at gmail.com (Denis McMahon) Date: Mon, 1 Jul 2013 23:14:17 +0000 (UTC) Subject: python adds an extra half space when reading from a string or list References: <51d17954$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d1a135$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Mon, 01 Jul 2013 21:18:26 +0200, Antoon Pardon wrote: > I am not baiting Nikos. Your opinion, mine differs. > all I have done in the last two weeks that involves Nikos as a > subject is the "Don't feed the troll thread" Calling him a troll is baiting. Please stop. -- Denis McMahon, denismfmcmahon at gmail.com From wuwei23 at gmail.com Mon Jul 1 00:57:20 2013 From: wuwei23 at gmail.com (alex23) Date: Mon, 01 Jul 2013 14:57:20 +1000 Subject: Closures in leu of pointers? In-Reply-To: References: <2a2072e3-4b12-4ada-872c-1240d2379928@googlegroups.com> <51cf2b49$0$29999$c3e8da3$5496439d@news.astraweb.com> <2f5e7801-a4cd-4d01-a1e7-7f67bca18199@googlegroups.com> <51cf3695$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 30/06/2013 3:46 PM, Ian Kelly wrote: > In general I agree, although when reading code I would definitely > prefer if the locals were declared. If you import the code into the interpreter as an adjunct to reading it you can see the locals with: >>> somefunc.func_code.co_varnames # 2.x >>> somefunc.__code__.co_varnames # 3.x > On a related note, I think that generator functions should in some way > be explicitly marked as such in the declaration, rather than needing > to scan the entire function body for a yield statement to determine > whether it's a generator or not. >>> inspect.isgenerator(somefunc) However, a dedicated code reader built around the inspect module could be a handy tool. From steve+comp.lang.python at pearwood.info Mon Jul 1 03:36:54 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 01 Jul 2013 07:36:54 GMT Subject: Closures in leu of pointers? References: <2a2072e3-4b12-4ada-872c-1240d2379928@googlegroups.com> <51cf2b49$0$29999$c3e8da3$5496439d@news.astraweb.com> <2f5e7801-a4cd-4d01-a1e7-7f67bca18199@googlegroups.com> <51cf3695$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51d13195$0$29973$c3e8da3$5496439d@news.astraweb.com> On Sat, 29 Jun 2013 23:46:12 -0600, Ian Kelly wrote: > On a related note, I think that generator functions should in some way > be explicitly marked as such in the declaration, rather than needing to > scan the entire function body for a yield statement to determine whether > it's a generator or not. That was considered when generators were introduced in Python 2.2. Guido's rationale for preferring to keep "def" for both generator functions and normal functions is given in the PEP: Issue: Introduce another new keyword (say, "gen" or "generator") in place of "def", or otherwise alter the syntax, to distinguish generator-functions from non-generator functions. Con: In practice (how you think about them), generators *are* functions, but with the twist that they're resumable. The mechanics of how they're set up is a comparatively minor technical issue, and introducing a new keyword would unhelpfully overemphasize the mechanics of how generators get started (a vital but tiny part of a generator's life). Pro: In reality (how you think about them), generator-functions are actually factory functions that produce generator-iterators as if by magic. In this respect they're radically different from non-generator functions, acting more like a constructor than a function, so reusing "def" is at best confusing. A "yield" statement buried in the body is not enough warning that the semantics are so different. BDFL: "def" it stays. No argument on either side is totally convincing, so I have consulted my language designer's intuition. It tells me that the syntax proposed in the PEP is exactly right - not too hot, not too cold. But, like the Oracle at Delphi in Greek mythology, it doesn't tell me why, so I don't have a rebuttal for the arguments against the PEP syntax. The best I can come up with (apart from agreeing with the rebuttals ... already made) is "FUD". If this had been part of the language from day one, I very much doubt it would have made Andrew Kuchling's "Python Warts" page. http://www.python.org/dev/peps/pep-0255/ 5+ versions later, I think that Guido has been shown to be correct. Even if you believe that generator functions would have been better with different syntax, there is no evidence that re-using def is actively harmful. -- Steven From neilc at norwich.edu Mon Jul 1 08:59:16 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 1 Jul 2013 12:59:16 GMT Subject: Stupid ways to spell simple code References: Message-ID: On 2013-06-30, Chris Angelico wrote: > So, here's a challenge: Come up with something really simple, > and write an insanely complicated - yet perfectly valid - way > to achieve the same thing. Bonus points for horribly abusing > Python's clean syntax in the process. > > Go on, do your worst! I've often thought it was redundant for Python to support 'if' when it has dictionaries, cf the rationale for having no 'switch'. valid_name = None while not valid_name: name = input("Enter your name: ") valid_name = { True: lambda: print("No name longer than 20 letters."), False: lambda: True, }[len(name) > 20]() Much better. -- Neil Cerutti From rosuav at gmail.com Mon Jul 1 09:14:15 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 1 Jul 2013 23:14:15 +1000 Subject: Stupid ways to spell simple code In-Reply-To: References: Message-ID: On Mon, Jul 1, 2013 at 10:59 PM, Neil Cerutti wrote: > On 2013-06-30, Chris Angelico wrote: >> So, here's a challenge: Come up with something really simple, >> and write an insanely complicated - yet perfectly valid - way >> to achieve the same thing. Bonus points for horribly abusing >> Python's clean syntax in the process. >> >> Go on, do your worst! > > I've often thought it was redundant for Python to support 'if' > when it has dictionaries, cf the rationale for having no > 'switch'. > > valid_name = None > while not valid_name: > name = input("Enter your name: ") > valid_name = { > True: lambda: print("No name longer than 20 letters."), > False: lambda: True, > }[len(name) > 20]() > > Much better. Good! Good! But, waaaah. Waaaah. def get_name(): while True: name = input("Enter your name: ") yield { True: lambda: print("No name longer than 20 letters."), False: lambda: name, }[len(name) > 20]() name = next(filter(None,get_name())) ChrisA From joshua.landau.ws at gmail.com Mon Jul 1 12:30:46 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Mon, 1 Jul 2013 17:30:46 +0100 Subject: Stupid ways to spell simple code In-Reply-To: References: Message-ID: On 1 July 2013 14:14, Chris Angelico wrote: > On Mon, Jul 1, 2013 at 10:59 PM, Neil Cerutti wrote: >> On 2013-06-30, Chris Angelico wrote: >>> So, here's a challenge: Come up with something really simple, >>> and write an insanely complicated - yet perfectly valid - way >>> to achieve the same thing. Bonus points for horribly abusing >>> Python's clean syntax in the process. >>> >>> Go on, do your worst! >> >> I've often thought it was redundant for Python to support 'if' >> when it has dictionaries, cf the rationale for having no >> 'switch'. >> >> valid_name = None >> while not valid_name: >> name = input("Enter your name: ") >> valid_name = { >> True: lambda: print("No name longer than 20 letters."), >> False: lambda: True, >> }[len(name) > 20]() >> >> Much better. > > Good! Good! But, waaaah. Waaaah. > > def get_name(): > while True: > name = input("Enter your name: ") > yield { > True: lambda: print("No name longer than 20 letters."), > False: lambda: name, > }[len(name) > 20]() > name = next(filter(None,get_name())) Oh, cruel. But you can do worse. Who needs "while" when you have filter(iter(FUNCTION, object()))? def get_name(): name = input("Enter your name: ") return [ lambda: name, lambda: print("No name longer than 20 letters."), ][len(name) > 20]() name = next(filter(None, iter(get_name, object()))) But who needs *any* of this! Defining functions is so old-hat. It's all already in the standard library (using only assignments and function-calls): from functools import partial from operator import getitem, ge, methodcaller from itertools import compress, tee apply = methodcaller("__call__") ret_true = partial(getitem, [True], 0) print_invalid = partial(print, "No name longer than 20 letters.") inputs = iter(partial(input, "Enter your name: "), ...) inputs, valid = tee(inputs) valid = map(len, valid) valid = map(partial(ge, 20), valid) side_effect_valid = map(partial(getitem, [print_invalid, ret_true]), valid) side_effect_valid = map(apply, side_effect_valid) valid_inputs = compress(inputs, side_effect_valid) name = next(valid_inputs) Which can be "neatly" expressed as two statements (I'm struggling to got it to one without those evil lambdas): from functools import partial from operator import getitem, ge, methodcaller from itertools import compress, tee inputs, valid = tee(iter(partial(input, "Enter your name: "), ...)) name = next( compress( inputs, map( methodcaller("__call__"), map( partial( getitem, [ partial(print, "No name longer than 20 letters."), partial(getitem, [True], 0) ] ), map( partial(ge, 20), map(len, valid) ) ) ) ) ) Beautiful, see? Of course, the most powerful function deals with this much more quickly: exec(""" while True: name = input("Enter your name: ") if len(name) <= 20: break else: print("No name longer than 20 letters.") """) From rosuav at gmail.com Mon Jul 1 12:32:18 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 2 Jul 2013 02:32:18 +1000 Subject: Stupid ways to spell simple code In-Reply-To: References: Message-ID: On Tue, Jul 2, 2013 at 2:30 AM, Joshua Landau wrote: > Beautiful, see? Truly a work of art! I am awed. ChrisA From mszamot at gmail.com Mon Jul 1 15:36:29 2013 From: mszamot at gmail.com (Marcin Szamotulski) Date: Mon, 1 Jul 2013 20:36:29 +0100 Subject: Stupid ways to spell simple code In-Reply-To: References: Message-ID: <20130701193629.GB28231@flying-circus> On 17:30 Mon 01 Jul , Joshua Landau wrote: > On 1 July 2013 14:14, Chris Angelico wrote: > > On Mon, Jul 1, 2013 at 10:59 PM, Neil Cerutti wrote: > >> On 2013-06-30, Chris Angelico wrote: > >>> So, here's a challenge: Come up with something really simple, > >>> and write an insanely complicated - yet perfectly valid - way > >>> to achieve the same thing. Bonus points for horribly abusing > >>> Python's clean syntax in the process. > >>> > >>> Go on, do your worst! > >> > >> I've often thought it was redundant for Python to support 'if' > >> when it has dictionaries, cf the rationale for having no > >> 'switch'. > >> > >> valid_name = None > >> while not valid_name: > >> name = input("Enter your name: ") > >> valid_name = { > >> True: lambda: print("No name longer than 20 letters."), > >> False: lambda: True, > >> }[len(name) > 20]() > >> > >> Much better. > > > > Good! Good! But, waaaah. Waaaah. > > > > def get_name(): > > while True: > > name = input("Enter your name: ") > > yield { > > True: lambda: print("No name longer than 20 letters."), > > False: lambda: name, > > }[len(name) > 20]() > > name = next(filter(None,get_name())) > > Oh, cruel. But you can do worse. Who needs "while" when you have > filter(iter(FUNCTION, object()))? > > def get_name(): > name = input("Enter your name: ") > return [ > lambda: name, > lambda: print("No name longer than 20 letters."), > ][len(name) > 20]() > > name = next(filter(None, iter(get_name, object()))) > > > But who needs *any* of this! Defining functions is so old-hat. It's > all already in the standard library (using only assignments and function-calls): > > from functools import partial > from operator import getitem, ge, methodcaller > from itertools import compress, tee > apply = methodcaller("__call__") > ret_true = partial(getitem, [True], 0) > print_invalid = partial(print, "No name longer than 20 letters.") > inputs = iter(partial(input, "Enter your name: "), ...) > inputs, valid = tee(inputs) > valid = map(len, valid) > valid = map(partial(ge, 20), valid) > side_effect_valid = map(partial(getitem, [print_invalid, ret_true]), valid) > side_effect_valid = map(apply, side_effect_valid) > valid_inputs = compress(inputs, side_effect_valid) > name = next(valid_inputs) > > > Which can be "neatly" expressed as two statements (I'm struggling to > got it to one without those evil lambdas): > > from functools import partial > from operator import getitem, ge, methodcaller > from itertools import compress, tee > > inputs, valid = tee(iter(partial(input, "Enter your name: "), ...)) > > name = next( > compress( > inputs, > map( > methodcaller("__call__"), > map( > partial( > getitem, > [ > partial(print, "No name longer than 20 letters."), > partial(getitem, [True], 0) > ] > ), > map( > partial(ge, 20), > map(len, valid) > ) > ) > ) > ) > ) > > > Beautiful, see? > > > Of course, the most powerful function deals with this much more quickly: > > exec(""" > while True: > name = input("Enter your name: ") > > if len(name) <= 20: > break > > else: > print("No name longer than 20 letters.") > """) > -- > http://mail.python.org/mailman/listinfo/python-list Here is another example which I came across when playing with generators, the first function is actually quite useful, the second generator is the whole fun: from functools import wraps def init(func): """decorator which initialises the generator """ @wraps(func) def inner(*args, **kwargs): g = func(*args, **kwargs) g.send(None) return g return inner @init def gen(func): x = (yield) while True: x = (yield func(x)) now if you have function f def f(arg): return arg**2 then calling f(5) is the same as g = gen(f) g.send(5) I wrote a blog post where I did include this as a `useless` example: http://pycorner.herokuapp.com/blog/5 Best regards, Marcin From steve+comp.lang.python at pearwood.info Mon Jul 1 18:09:18 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 01 Jul 2013 22:09:18 GMT Subject: Stupid ways to spell simple code References: Message-ID: <51d1fe0d$0$29973$c3e8da3$5496439d@news.astraweb.com> On Mon, 01 Jul 2013 20:36:29 +0100, Marcin Szamotulski wrote: > Here is another example which I came across when playing with > generators, the first function is actually quite useful, the second > generator is the whole fun: > > from functools import wraps > def init(func): > """decorator which initialises the generator """ > @wraps(func) > def inner(*args, **kwargs): > g = func(*args, **kwargs) > g.send(None) > return g > return inner > > @init > def gen(func): > x = (yield) > while True: > x = (yield func(x)) > > > now if you have function f > def f(arg): > return arg**2 > > then calling f(5) is the same as > > g = gen(f) > g.send(5) I think you must be missing an important part of the trick, because calling f(5) returns 25. It's not: @gen def f(arg): return arg**2 because that raises TypeError. -- Steven From dihedral88888 at gmail.com Tue Jul 2 00:16:39 2013 From: dihedral88888 at gmail.com (88888 Dihedral) Date: Mon, 1 Jul 2013 21:16:39 -0700 (PDT) Subject: Stupid ways to spell simple code In-Reply-To: <51d1fe0d$0$29973$c3e8da3$5496439d@news.astraweb.com> References: <51d1fe0d$0$29973$c3e8da3$5496439d@news.astraweb.com> Message-ID: <875111a8-4784-4590-aecd-3c505e26126b@googlegroups.com> Steven D'Aprano? 2013?7?2????UTC+8??6?09?18???? > On Mon, 01 Jul 2013 20:36:29 +0100, Marcin Szamotulski wrote: > > > > > Here is another example which I came across when playing with > > > generators, the first function is actually quite useful, the second > > > generator is the whole fun: > > > > > > from functools import wraps > > > def init(func): > > > """decorator which initialises the generator """ > > > @wraps(func) > > > def inner(*args, **kwargs): > > > g = func(*args, **kwargs) > > > g.send(None) > > > return g > > > return inner > > > > > > @init > > > def gen(func): > > > x = (yield) > > > while True: > > > x = (yield func(x)) > > > > > > > > > now if you have function f > > > def f(arg): > > > return arg**2 > > > > > > then calling f(5) is the same as > > > > > > g = gen(f) > > > g.send(5) > > > > > > > > I think you must be missing an important part of the trick, because > > calling f(5) returns 25. It's not: > > > > @gen > > def f(arg): > > return arg**2 > > > > > > because that raises TypeError. > > > > > > > > > > -- > > Steven Lets be serious about generators and iterators. A generator can be used only once in a program is different from a a generator method of a class that can produce several instances with generators of the same kind but operated in each instance of the class. From mszamot at gmail.com Tue Jul 2 02:22:10 2013 From: mszamot at gmail.com (Marcin Szamotulski) Date: Tue, 2 Jul 2013 07:22:10 +0100 Subject: Stupid ways to spell simple code In-Reply-To: <51d1fe0d$0$29973$c3e8da3$5496439d@news.astraweb.com> References: <51d1fe0d$0$29973$c3e8da3$5496439d@news.astraweb.com> Message-ID: <20130702062210.GB21317@flying-circus> On 22:09 Mon 01 Jul , Steven D'Aprano wrote: > On Mon, 01 Jul 2013 20:36:29 +0100, Marcin Szamotulski wrote: > > > Here is another example which I came across when playing with > > generators, the first function is actually quite useful, the second > > generator is the whole fun: > > > > from functools import wraps > > def init(func): > > """decorator which initialises the generator """ > > @wraps(func) > > def inner(*args, **kwargs): > > g = func(*args, **kwargs) > > g.send(None) > > return g > > return inner > > > > @init > > def gen(func): > > x = (yield) > > while True: > > x = (yield func(x)) > > > > > > now if you have function f > > def f(arg): > > return arg**2 > > > > then calling f(5) is the same as > > > > g = gen(f) > > g.send(5) > > > > I think you must be missing an important part of the trick, because > calling f(5) returns 25. It's not: > > @gen > def f(arg): > return arg**2 > > > because that raises TypeError. > -- > Steven > -- > http://mail.python.org/mailman/listinfo/python-list Sure it does, you're now supposed to use .send method instead of calling it but this is just different syntax. If you want to call it use this : def identity(func): @init def gen(func): x = (yield) while True: x = (yield func(x)) return gen(func).send Now you will get: >>> @identity >>> def f(a): a+1 ... >>> f(0) 1 Best regards, Marcin From russ.pobox at gmail.com Tue Jul 2 03:33:01 2013 From: russ.pobox at gmail.com (Russel Walker) Date: Tue, 2 Jul 2013 00:33:01 -0700 (PDT) Subject: Stupid ways to spell simple code In-Reply-To: References: Message-ID: On Sunday, June 30, 2013 8:06:35 AM UTC+2, Chris Angelico wrote: > There's a bit of a discussion on python-ideas that includes a function > > that raises StopIteration. It inspired me to do something stupid, just > > to see how easily I could do it... > > > > On Sun, Jun 30, 2013 at 3:45 PM, Nick Coghlan wrote: > > Re: [Python-ideas] "Iteration stopping" syntax > > >>>> def stop(): > > > ... raise StopIteration > > > > Here's a much more insane way to spell that: > > > > stop = (lambda: 0 and (yield 1))().__next__ > > > > So, here's a challenge: Come up with something really simple, and > > write an insanely complicated - yet perfectly valid - way to achieve > > the same thing. Bonus points for horribly abusing Python's clean > > syntax in the process. > > > > Go on, do your worst! > > > > ChrisA Here's a way to count items in a string. def count(string, x): return len(''.join(string)) - len(''.join(string).replace(x, '')) / len(x) From joshua at landau.ws Wed Jul 10 22:37:28 2013 From: joshua at landau.ws (Joshua Landau) Date: Thu, 11 Jul 2013 03:37:28 +0100 Subject: Stupid ways to spell simple code In-Reply-To: References: Message-ID: On 30 June 2013 07:06, Chris Angelico wrote: > So, here's a challenge: Come up with something really simple, and > write an insanely complicated - yet perfectly valid - way to achieve > the same thing. Bonus points for horribly abusing Python's clean > syntax in the process. This occurred to me out of the blue while working on something related. Here's a way to remove all instances of an element from an iterable. It's remarkably fast for it's course of action: from collections import deque from itertools import chain exhaust_iterable = deque(maxlen=0).extend def split_on(data, sentinel): chained = data = iter(data) while True: chunk = iter(chained.__next__, sentinel) yield chunk # Uses at least one item from chained, so "chained" and "data" are equivilant after this. # This is important as "data" is raw and thus will not get bogged down by pointless chain wrappings. # Yes, that is a premature optimisation. Go away. exhaust_iterable(chunk) # Throw StopIteration if not empty chained = chain([next(data)], data) def remove_all(iterable, victim): return list(chain.from_iterable(split_on(iterable, victim))) print(remove_all([1, 2, 1, 1, 0, 1, 2, 1, 1, 2, 2, 1, 0], 1)) and here it is again: from itertools import chain, repeat def remove_all(iterable, victim): iterable = iter(iterable) return list(chain.from_iterable(iter(chain([next(iterable)], iterable).__next__, victim) for _ in repeat(...))) print(remove_all([1, 2, 1, 1, 0, 1, 2, 1, 1, 2, 2, 1, 0], 1)) From akshay.ksth at gmail.com Mon Jul 1 15:55:39 2013 From: akshay.ksth at gmail.com (Akshay Kayastha) Date: Mon, 1 Jul 2013 12:55:39 -0700 (PDT) Subject: Issues compiling hunspell from source on windows In-Reply-To: <04acbd31-b7c5-4518-852a-0ed0deed44c5@googlegroups.com> References: <04acbd31-b7c5-4518-852a-0ed0deed44c5@googlegroups.com> Message-ID: <8f7919c2-9c1f-45f0-a876-9f608800d132@googlegroups.com> I installed a 32 bit python and it still gives me the same error. From skip at pobox.com Mon Jul 1 05:32:32 2013 From: skip at pobox.com (Skip Montanaro) Date: Mon, 1 Jul 2013 04:32:32 -0500 Subject: settrace doesn't trace builtin functions In-Reply-To: <694830e3-080c-4e66-90e2-6cbab53be071@googlegroups.com> References: <694830e3-080c-4e66-90e2-6cbab53be071@googlegroups.com> Message-ID: > I've been using the settrace function to write a tracer for my program, which is working great except that it doesn't seem to work for built-in functions, like open('filename.txt'). This doesn't seem to be documented, so I'm not sure if I'm doing something wrong or that's the expected behavior. I believe that's expected behavior. Stuff written in Python is traced. C functions are not. It's been a whole lot of years since I looked at that code though, so I might be misremembering. > The other option I considered was monkey-patching the open function through a wrapper ... but that seemed very brittle to me. For debugging purposes, practicality beats purity. I trust you won't be tracing in a production environment, so fragility shouldn't be a major concern. Monkey patching seems entirely appropriate to me. Python 2.x: >>> open >>> import __builtin__ >>> open is __builtin__.open True >>> _open = __builtin__.open >>> def open(name, mode=None, buffering=None): ... return _open(name, mode, buffering) ... >>> __builtin__.open = open >>> open is __builtin__.open True >>> open Python 3.x: >>> import builtins >>> open >>> open is builtins.open True >>> _open = builtins.open >>> def open(file, mode='r', buffering=-1, encoding=None, ... errors=None, newline=None, closefd=True, opener=None): ... return _open(file, mode, buffering, encoding, errors, newline, closefd, opener) ... >>> _open >>> builtins.open = open >>> open You can, of course, do this for more builtins. Presuming you can get a handle on a function's namespace and modify it, you should be able to perform the same trick for most functions or methods written in C. >>> sys._settrace = sys.settrace >>> def settrace(*args, **kwds): ... return sys._settrace(*args, **kwds) ... >>> sys.settrace = sys._settrace Totally untested. No warranties expressed or implied. YMMV... etc, etc Skip From prerit86 at gmail.com Mon Jul 1 05:13:50 2013 From: prerit86 at gmail.com (prerit86 at gmail.com) Date: Mon, 1 Jul 2013 02:13:50 -0700 (PDT) Subject: Need explanation of this error Message-ID: Hi, I'm new to Python and trying to run a already written code. Can someone please explain the error below? And if possible, how do I resolve this? Traceback (most recent call last): File "c:\Project_1\regression_1.py", line 7, in from sklearn import metrics, cross_validation, linear_model File "c:\Python27\lib\site-packages\sklearn\metrics\__init__.py", line 31, in from . import cluster File "c:\Python27\lib\site-packages\sklearn\metrics\cluster\__init__.py", line 8, in from .supervised import adjusted_mutual_info_score File "c:\Python27\lib\site-packages\sklearn\metrics\cluster\supervised.py", li ne 19, in from .expected_mutual_info_fast import expected_mutual_information File "expected_mutual_info_fast.pyx", line 10, in init sklearn.metrics.cluster .expected_mutual_info_fast (sklearn\metrics\cluster\expected_mutual_info_fast.c: 4886) File "c:\Python27\lib\site-packages\scipy\special\__init__.py", line 529, in < module> from ._ufuncs import * ImportError: DLL load failed: The specified module could not be found. From robert.kern at gmail.com Mon Jul 1 05:23:21 2013 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 01 Jul 2013 10:23:21 +0100 Subject: Need explanation of this error In-Reply-To: References: Message-ID: On 2013-07-01 10:13, prerit86 at gmail.com wrote: > Hi, > > I'm new to Python and trying to run a already written code. Can someone please explain the error below? And if possible, how do I resolve this? > > Traceback (most recent call last): > File "c:\Project_1\regression_1.py", line 7, in > from sklearn import metrics, cross_validation, linear_model > File "c:\Python27\lib\site-packages\sklearn\metrics\__init__.py", line 31, in > > from . import cluster > File "c:\Python27\lib\site-packages\sklearn\metrics\cluster\__init__.py", line > 8, in > from .supervised import adjusted_mutual_info_score > File "c:\Python27\lib\site-packages\sklearn\metrics\cluster\supervised.py", li > ne 19, in > from .expected_mutual_info_fast import expected_mutual_information > File "expected_mutual_info_fast.pyx", line 10, in init sklearn.metrics.cluster > .expected_mutual_info_fast (sklearn\metrics\cluster\expected_mutual_info_fast.c: > 4886) > File "c:\Python27\lib\site-packages\scipy\special\__init__.py", line 529, in < > module> > from ._ufuncs import * > ImportError: DLL load failed: The specified module could not be found. This particular module incorporates FORTRAN subroutines. My guess is that whoever compiled your scipy binary did it in such a way that the FORTRAN standard library was being linked in as a DLL instead of statically. This DLL is not present on your system. Windows is trying to find it, but failing. How did you install scipy? If you used a prebuilt binary installer, can you please link to the exact one that you used? Try using depends.exe to find out what DLL it is looking for. http://www.dependencywalker.com/ The file that you want to check in depends.exe: c:\Python27\lib\site-packages\scipy\special\_ufuncs.pyd -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From denismfmcmahon at gmail.com Mon Jul 1 05:33:36 2013 From: denismfmcmahon at gmail.com (Denis McMahon) Date: Mon, 1 Jul 2013 09:33:36 +0000 (UTC) Subject: Need explanation of this error References: Message-ID: On Mon, 01 Jul 2013 02:13:50 -0700, prerit86 wrote: > I'm new to Python and trying to run a already written code. Can someone > please explain the error below? And if possible, how do I resolve this? > ImportError: DLL load failed: The specified module could not be found. You're missing the dll that provides some function that you're trying to use. -- Denis McMahon, denismfmcmahon at gmail.com From prerit86 at gmail.com Mon Jul 1 06:44:55 2013 From: prerit86 at gmail.com (prerit86 at gmail.com) Date: Mon, 1 Jul 2013 03:44:55 -0700 (PDT) Subject: Need explanation of this error In-Reply-To: References: Message-ID: Thanks! Solved. I found the package that would resolve this dependency. It was numpy-MKL. Downloaded from http://www.lfd.uci.edu/~gohlke/pythonlibs/#pandas From prerit86 at gmail.com Mon Jul 1 06:47:14 2013 From: prerit86 at gmail.com (prerit86 at gmail.com) Date: Mon, 1 Jul 2013 03:47:14 -0700 (PDT) Subject: File exists but Python says 'not found'. Message-ID: <5c1d1e47-b67e-4bce-8601-b14c0215f450@googlegroups.com> I'm running this code that reads 2 csv files (one of them is train.csv). The code gives an error saying 'file not does not exist'. However, the file does exists in the same location as the .py file. Can someone please help me on this. Thanks! Code Output--> Reading dataset... Traceback (most recent call last): File "c:\Project_1\regression_2.py", line 163, in main(**args) File "c:\Project_1\regression_2.py", line 80, in main train_data = pd.read_csv(train) File "c:\Python27\lib\site-packages\pandas\io\parsers.py", line 401, in parser _f return _read(filepath_or_buffer, kwds) File "c:\Python27\lib\site-packages\pandas\io\parsers.py", line 209, in _read parser = TextFileReader(filepath_or_buffer, **kwds) File "c:\Python27\lib\site-packages\pandas\io\parsers.py", line 509, in __init __ self._make_engine(self.engine) File "c:\Python27\lib\site-packages\pandas\io\parsers.py", line 611, in _make_ engine self._engine = CParserWrapper(self.f, **self.options) File "c:\Python27\lib\site-packages\pandas\io\parsers.py", line 893, in __init __ self._reader = _parser.TextReader(src, **kwds) File "parser.pyx", line 312, in pandas._parser.TextReader.__cinit__ (pandas\sr c\parser.c:2846) File "parser.pyx", line 512, in pandas._parser.TextReader._setup_parser_source (pandas\src\parser.c:4893) IOError: File train.csv does not exist From robert.kern at gmail.com Mon Jul 1 06:56:38 2013 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 01 Jul 2013 11:56:38 +0100 Subject: File exists but Python says 'not found'. In-Reply-To: <5c1d1e47-b67e-4bce-8601-b14c0215f450@googlegroups.com> References: <5c1d1e47-b67e-4bce-8601-b14c0215f450@googlegroups.com> Message-ID: On 2013-07-01 11:47, prerit86 at gmail.com wrote: > I'm running this code that reads 2 csv files (one of them is train.csv). The code gives an error saying 'file not does not exist'. However, the file does exists in the same location as the .py file. Can someone please help me on this. Thanks! How are you running the code? If you are doing pandas.read_csv('train.csv'), the file must be in the current working directory of the running process. The location of the .py file contain the code is not relevant. For example, if you are using an IDE to run the script, you may need to configure how it runs the script to pick a particular directory for it to run in. > Code Output--> > > Reading dataset... > Traceback (most recent call last): > File "c:\Project_1\regression_2.py", line 163, in > main(**args) > File "c:\Project_1\regression_2.py", line 80, in main > train_data = pd.read_csv(train) Since the filename of regression_2.py in the traceback is fully-qualified, I expect that you are running the program from something other than the c:\Project_1\ directory. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From antoon.pardon at rece.vub.ac.be Mon Jul 1 06:59:53 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Mon, 01 Jul 2013 12:59:53 +0200 Subject: File exists but Python says 'not found'. In-Reply-To: <5c1d1e47-b67e-4bce-8601-b14c0215f450@googlegroups.com> References: <5c1d1e47-b67e-4bce-8601-b14c0215f450@googlegroups.com> Message-ID: <51D16129.3030808@rece.vub.ac.be> Op 01-07-13 12:47, prerit86 at gmail.com schreef: > I'm running this code that reads 2 csv files (one of them is train.csv). The code gives an error saying 'file not does not exist'. However, the file does exists in the same location as the .py file. Can someone please help me on this. Thanks! > > Code Output--> > > Reading dataset... > Traceback (most recent call last): > File "c:\Project_1\regression_2.py", line 163, in > main(**args) > File "c:\Project_1\regression_2.py", line 80, in main > train_data = pd.read_csv(train) > File "c:\Python27\lib\site-packages\pandas\io\parsers.py", line 401, in parser > _f > return _read(filepath_or_buffer, kwds) > File "c:\Python27\lib\site-packages\pandas\io\parsers.py", line 209, in _read > parser = TextFileReader(filepath_or_buffer, **kwds) > File "c:\Python27\lib\site-packages\pandas\io\parsers.py", line 509, in __init > __ > self._make_engine(self.engine) > File "c:\Python27\lib\site-packages\pandas\io\parsers.py", line 611, in _make_ > engine > self._engine = CParserWrapper(self.f, **self.options) > File "c:\Python27\lib\site-packages\pandas\io\parsers.py", line 893, in __init > __ > self._reader = _parser.TextReader(src, **kwds) > File "parser.pyx", line 312, in pandas._parser.TextReader.__cinit__ (pandas\sr > c\parser.c:2846) > File "parser.pyx", line 512, in pandas._parser.TextReader._setup_parser_source > (pandas\src\parser.c:4893) > IOError: File train.csv does not exist My guess is that train.csv is a symbolic link that points to a file that doesn't exist. So looking at your directory you can see an entry but actualy trying to open it will fail. -- Antoon Pardon From prerit86 at gmail.com Mon Jul 1 06:57:37 2013 From: prerit86 at gmail.com (prerit86 at gmail.com) Date: Mon, 1 Jul 2013 03:57:37 -0700 (PDT) Subject: File exists but Python says 'not found'. In-Reply-To: <5c1d1e47-b67e-4bce-8601-b14c0215f450@googlegroups.com> References: <5c1d1e47-b67e-4bce-8601-b14c0215f450@googlegroups.com> Message-ID: The variable 'train' is being called like this -> def main(train='train.csv', test='test.csv', submit='logistic_pred.csv'): print "Reading dataset..." train_data = pd.read_csv(train) test_data = pd.read_csv(test) Let me know if I need to post the full code. From antoon.pardon at rece.vub.ac.be Mon Jul 1 07:15:12 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Mon, 01 Jul 2013 13:15:12 +0200 Subject: File exists but Python says 'not found'. In-Reply-To: References: <5c1d1e47-b67e-4bce-8601-b14c0215f450@googlegroups.com> Message-ID: <51D164C0.7040504@rece.vub.ac.be> Op 01-07-13 12:57, prerit86 at gmail.com schreef: > The variable 'train' is being called like this -> > > def main(train='train.csv', test='test.csv', submit='logistic_pred.csv'): > print "Reading dataset..." > train_data = pd.read_csv(train) > test_data = pd.read_csv(test) > > Let me know if I need to post the full code. I think Robert wanted to know how you started the program. What instruction do you use to launch? In what directory are you launching your program? Is that the same directory where train.csv is present? From prerit86 at gmail.com Mon Jul 1 07:32:34 2013 From: prerit86 at gmail.com (prerit86 at gmail.com) Date: Mon, 1 Jul 2013 04:32:34 -0700 (PDT) Subject: File exists but Python says 'not found'. In-Reply-To: References: <5c1d1e47-b67e-4bce-8601-b14c0215f450@googlegroups.com> Message-ID: <95b8480f-5011-498e-bf36-278b89dced3a@googlegroups.com> My answers I think Robert wanted to know how you started the program. What instruction do you use to launch? - used command c:\python27\python.exe c:\project_1\code.py In what directory are you launching your program? - working directory was c: - python is in c:\python27 - code was in c:\project_1 - changed the working directory to c:\project_1 now and it worked Is that the same directory where train.csv is present? - train.csv is present in c:\project_1 From prerit86 at gmail.com Mon Jul 1 07:00:55 2013 From: prerit86 at gmail.com (prerit86 at gmail.com) Date: Mon, 1 Jul 2013 04:00:55 -0700 (PDT) Subject: File exists but Python says 'not found'. In-Reply-To: References: <5c1d1e47-b67e-4bce-8601-b14c0215f450@googlegroups.com> Message-ID: I got it. The working directory was different. Sorry, I'm new and didn't the working directory has to be the location of the data. I thought the location of .py file and data file should be same. Thanks! Es. Robert Kern. From davea at davea.name Mon Jul 1 07:38:12 2013 From: davea at davea.name (Dave Angel) Date: Mon, 01 Jul 2013 07:38:12 -0400 Subject: File exists but Python says 'not found'. In-Reply-To: References: <5c1d1e47-b67e-4bce-8601-b14c0215f450@googlegroups.com> Message-ID: <51D16A24.30309@davea.name> On 07/01/2013 07:00 AM, prerit86 at gmail.com wrote: > I got it. The working directory was different. Sorry, I'm new and didn't the working directory has to be the location of the data. I thought the location of .py file and data file should be same. Thanks! Es. Robert Kern. > Python didn't make that assumption, the author of the script did. By using a relative name like train.csv, he is implicitly forcing Python to use the current working directory. If he had wanted the file to be found in the same directory as one of the source files, he could have built an absolute path by using the __file__ attribute of the module. However, for non-const files, the source directory is generally a bad place to put them. Confounding things is the strange habit that Windows has of deciding that the script location *should* be your current directory. I believe Explorer does that if you right-click select a script. -- DaveA From prerit86 at gmail.com Mon Jul 1 08:17:45 2013 From: prerit86 at gmail.com (prerit86 at gmail.com) Date: Mon, 1 Jul 2013 05:17:45 -0700 (PDT) Subject: File exists but Python says 'not found'. In-Reply-To: References: <5c1d1e47-b67e-4bce-8601-b14c0215f450@googlegroups.com> Message-ID: <7fbe3171-417f-494e-a8b9-c02ffe2d5e14@googlegroups.com> I know. Had I written the code, I would have not done this. I just wanted to get some initial results by leveraging this code. I would now build on this to improve my work's accuracy. Thanks for the inputs! From tshepard at rcsreg.com Mon Jul 1 14:29:35 2013 From: tshepard at rcsreg.com (Tobiah) Date: Mon, 01 Jul 2013 11:29:35 -0700 Subject: PYTHONPATH and module names Message-ID: <51D1CA8F.4050708@tobiah.org> So today, I created a file called 'formatter.py', and my program broke. It turned out that I was also import 'gluon' from web2py, which in turn, somewhere, imported the regular python formatter.py with which I was not familiar. So the question is: Does one simply always have to be knowledgeable about existing python library names, or is having '.' in the python path just a bad idea? Is there a way, not having '.' in the path to explicitly specify the current directory? Something analogous to import ./foo ? Thanks, Tobiah From rustompmody at gmail.com Mon Jul 1 14:39:46 2013 From: rustompmody at gmail.com (rusi) Date: Mon, 1 Jul 2013 11:39:46 -0700 (PDT) Subject: PYTHONPATH and module names In-Reply-To: References: Message-ID: On Monday, July 1, 2013 11:59:35 PM UTC+5:30, Tobiah wrote: > So today, I created a file called 'formatter.py', > and my program broke. It turned out that I was > also import 'gluon' from web2py, which in turn, > somewhere, imported the regular python formatter.py > with which I was not familiar. > > So the question is: Does one simply always have > to be knowledgeable about existing python library > names, or is having '.' in the python path just > a bad idea? Is there a way, not having '.' in > the path to explicitly specify the current directory? > Something analogous to import ./foo ? Are you familiar with absolute and relative imports: http://docs.python.org/release/2.5/whatsnew/pep-328.html From toby at tobiah.org Mon Jul 1 15:54:30 2013 From: toby at tobiah.org (Tobiah) Date: Mon, 01 Jul 2013 12:54:30 -0700 Subject: PYTHONPATH and module names In-Reply-To: References: Message-ID: > Are you familiar with absolute and relative imports: > http://docs.python.org/release/2.5/whatsnew/pep-328.html Doesn't seem to work: Python 2.7.3 (default, May 10 2012, 13:31:18) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from __future__ import absolute_import >>> import .format File "", line 1 import .format ^ SyntaxError: invalid syntax >>> From spaghettitoastbook at gmail.com Mon Jul 1 16:40:03 2013 From: spaghettitoastbook at gmail.com (SpaghettiToastBook .) Date: Mon, 1 Jul 2013 16:40:03 -0400 Subject: PYTHONPATH and module names In-Reply-To: References: Message-ID: Relative imports only work with the "from ... import ..." form. ? SpaghettiToastBook On Mon, Jul 1, 2013 at 3:54 PM, Tobiah wrote: >> Are you familiar with absolute and relative imports: >> http://docs.python.org/release/2.5/whatsnew/pep-328.html > > > Doesn't seem to work: > > Python 2.7.3 (default, May 10 2012, 13:31:18) > [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> from __future__ import absolute_import >>>> import .format > File "", line 1 > import .format > ^ > SyntaxError: invalid syntax >>>> > > -- > http://mail.python.org/mailman/listinfo/python-list From rustompmody at gmail.com Mon Jul 1 17:38:50 2013 From: rustompmody at gmail.com (rusi) Date: Mon, 1 Jul 2013 14:38:50 -0700 (PDT) Subject: PYTHONPATH and module names In-Reply-To: References: Message-ID: On Tuesday, July 2, 2013 1:24:30 AM UTC+5:30, Tobiah wrote: > > Are you familiar with absolute and relative imports: > > http://docs.python.org/release/2.5/whatsnew/pep-328.html > > Doesn't seem to work: > Python 2.7.3 (default, May 10 2012, 13:31:18) > [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from __future__ import absolute_import > >>> import .format > File "", line 1 > import .format > ^ > SyntaxError: invalid syntax > >>> 1. My reading of http://www.python.org/dev/peps/pep-0328/ is that this only works for from statements not import statements. [See the section called Guido's decision] 2. The __future__ is not necessary in python 2.7 [Not necessary or not allowed I not know :-) ] From steve+comp.lang.python at pearwood.info Mon Jul 1 18:05:35 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 01 Jul 2013 22:05:35 GMT Subject: PYTHONPATH and module names References: Message-ID: <51d1fd2f$0$29973$c3e8da3$5496439d@news.astraweb.com> On Mon, 01 Jul 2013 14:38:50 -0700, rusi wrote: > On Tuesday, July 2, 2013 1:24:30 AM UTC+5:30, Tobiah wrote: >> > Are you familiar with absolute and relative imports: >> > http://docs.python.org/release/2.5/whatsnew/pep-328.html >> >> Doesn't seem to work: >> Python 2.7.3 (default, May 10 2012, 13:31:18) [GCC 4.2.4 (Ubuntu >> 4.2.4-1ubuntu4)] on linux2 Type "help", "copyright", "credits" or >> "license" for more information. >> >>> from __future__ import absolute_import import .format >> File "", line 1 >> import .format >> ^ >> SyntaxError: invalid syntax >> >>> >> >>> > 1. My reading of > http://www.python.org/dev/peps/pep-0328/ is that this only works for > from statements not import statements. [See the section called Guido's > decision] Correct. This would have to be written as: from . import format but note that this only work in a package, not from some arbitrary module inside a directory. > 2. The __future__ is not necessary in python 2.7 [Not necessary or not > allowed I not know :-) ] Not necessary. __future__ statements are guaranteed to "work" in all future versions, in the sense that once a __future__ feature is added, it will never be removed. So Python has had "nested scopes" since version 2.2 (by memory), but: from __future__ import nested_scopes still is allowed in Python 3.3, even though it has been a no-op since 2.2 or 2.3. -- Steven From lele at metapensiero.it Tue Jul 2 01:30:20 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Tue, 02 Jul 2013 07:30:20 +0200 Subject: PYTHONPATH and module names References: <51d1fd2f$0$29973$c3e8da3$5496439d@news.astraweb.com> Message-ID: <874ncdmsur.fsf@nautilus.nautilus> Steven D'Aprano writes: > On Mon, 01 Jul 2013 14:38:50 -0700, rusi wrote: >> 2. The __future__ is not necessary in python 2.7 [Not necessary or not >> allowed I not know :-) ] > > Not necessary. IIRC that it is needed, to solve the OP problem: one thing is the syntax, which under Python 2.7 is enabled by default, another thing is the behaviour, that is whether the interpreter will give priority to the sys.path. ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From fabiosantosart at gmail.com Mon Jul 1 18:08:48 2013 From: fabiosantosart at gmail.com (=?ISO-8859-1?Q?F=E1bio_Santos?=) Date: Mon, 1 Jul 2013 23:08:48 +0100 Subject: PYTHONPATH and module names In-Reply-To: References: Message-ID: On 1 Jul 2013 20:58, "Tobiah" wrote: >> >> Are you familiar with absolute and relative imports: >> http://docs.python.org/release/2.5/whatsnew/pep-328.html > > > Doesn't seem to work: > > Python 2.7.3 (default, May 10 2012, 13:31:18) > [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from __future__ import absolute_import > >>> import .format > File "", line 1 > import .format > ^ > SyntaxError: invalid syntax > >>> Have you tried from . import format ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From joel.goldstick at gmail.com Mon Jul 1 15:32:12 2013 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Mon, 1 Jul 2013 15:32:12 -0400 Subject: python adds an extra half space when reading froma string or list -- back to the question Message-ID: I copied the original question so that the rant on the other thread can continue. Let's keep this thread ontopic number_drawn=() def load(lot_number,number_drawn): first=input("enter first lot: ") last=input("enter last lot: ") for lot_number in range(first,last): line_out=str(lot_number) for count in range(1,5): number_drawn=raw_input("number: ") line_out=line_out+(number_drawn) print line_out finale_line.append(line_out) finale_line2=finale_line load(lot_number,number_drawn) print finale_line print(" "*4), for n in range(1,41): print n, #this is to produce a line of numbers to compare to output# for a in finale_line: print"\n", print a[0]," ", space_count=1 for b in range(1,5): if int(a[b])<10: print(" "*(int(a[b])-space_count)),int(a[b]), space_count=int(a[b]) else: print(" "*(a[b]-space_count)),a[b], space_count=a[b]+1 number_drawn=() def load(lot_number,number_drawn): first=input("enter first lot: ") last=input("enter last lot: ") for lot_number in range(first,last): line_out=str(lot_number) for count in range(1,5): number_drawn=raw_input("number: ") line_out=line_out+(number_drawn) print line_out finale_line.append(line_out) finale_line2=finale_line load(lot_number,number_drawn) print finale_line print(" "*4), for n in range(1,41): print n, #this is to produce a line of numbers to compare to output# for a in finale_line: print"\n", print a[0]," ", space_count=1 for b in range(1,5): if int(a[b])<10: print(" "*(int(a[b])-space_count)),int(a[b]), space_count=int(a[b]) else: print(" "*(a[b]-space_count)),a[b], space_count=a[b]+1 this generates enter first lot: 1 enter last lot: 4 number: 2 number: 3 number: 4 number: 5 12345 number: 1 number: 2 number: 3 number: 4 21234 number: 3 number: 4 number: 5 number: 6 33456 ['12345', '21234', '33456'] 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 1 2 3 4 5 2 1 2 3 4 3 3 4 5 6 >#as you can see many numbers are between the lines of a normal print# #I thought this was due to "white space" int he format .So I tried a list of strings and got the same results.# -- Joel Goldstick http://joelgoldstick.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshua.landau.ws at gmail.com Mon Jul 1 15:40:12 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Mon, 1 Jul 2013 20:40:12 +0100 Subject: python adds an extra half space when reading froma string or list -- back to the question In-Reply-To: References: Message-ID: On 1 July 2013 20:32, Joel Goldstick wrote: > I copied the original question so that the rant on the other thread can > continue. Let's keep this thread ontopic Thank you. I shall do the same below. Unfortunately I don't have high hopes that any progress will be made on this thread -- Charles Benoit hasn't made a single reply since his original post. On 29 June 2013 03:07, charles benoit wrote: 1) You haven't asked a question. 2) You posted your code twice. That makes it look a lot harder and longer than it really is. 3) Give us a *minimal* reproducible test case. I currently just get: %~> python2 /tmp/nd.py Traceback (most recent call last): File "/tmp/nd.py", line 12, in finale_line2=finale_line NameError: name 'finale_line' is not defined This isn't the only problem. In other words, you've given us an unsolvable problem. Try again. From davea at davea.name Mon Jul 1 16:02:44 2013 From: davea at davea.name (Dave Angel) Date: Mon, 01 Jul 2013 16:02:44 -0400 Subject: python adds an extra half space when reading froma string or list -- back to the question In-Reply-To: References: Message-ID: <51D1E064.7020108@davea.name> On 07/01/2013 03:32 PM, Joel Goldstick wrote: > I copied the original question so that the rant on the other thread > can continue. Let's keep this thread ontopic > > > number_drawn=() > def load(lot_number,number_drawn): > first=input("enter first lot: ") > last=input("enter last lot: ") > for lot_number in range(first,last): > line_out=str(lot_number) > for count in range(1,5): > number_drawn=raw_input("number: ") > line_out=line_out+(number_drawn) > print line_out > finale_line.append(line_out) > finale_line2=finale_line > > load(lot_number,number_drawn) > > > print finale_line > print(" "*4), > for n in range(1,41): > print n, #this is to produce a line of numbers to compare to > output# > for a in finale_line: > print"\n", > print a[0]," ", > space_count=1 > for b in range(1,5): > if int(a[b])<10: > print(" "*(int(a[b])-space_count)),int(a[b]), > space_count=int(a[b]) > else: > print(" "*(a[b]-space_count)),a[b], > space_count=a[b]+1 > > > > > > > > > > > > > this generates > > enter first lot: 1 > enter last lot: 4 > number: 2 > number: 3 > number: 4 > number: 5 > 12345 > number: 1 > number: 2 > number: 3 > number: 4 > 21234 > number: 3 > number: 4 > number: 5 > number: 6 > 33456 > ['12345', '21234', '33456'] > 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 > 27 28 29 30 31 32 33 34 35 36 37 38 39 40 > 1 2 3 4 5 > 2 1 2 3 4 > 3 3 4 5 6 >> #as you can see many numbers are between the lines of a normal print# > #I thought this was due to "white space" int he format .So I tried a list > of strings and got the same results.# > But what was the expected output? And who cares? The code made no sense, was incomplete, and the posted question was nonsensical. If the OP has abandoned it, so should we. -- DaveA From rustompmody at gmail.com Mon Jul 1 17:16:46 2013 From: rustompmody at gmail.com (rusi) Date: Mon, 1 Jul 2013 14:16:46 -0700 (PDT) Subject: python adds an extra half space when reading froma string or list -- back to the question In-Reply-To: References: Message-ID: On Tuesday, July 2, 2013 1:32:44 AM UTC+5:30, Dave Angel wrote: > But what was the expected output? And who cares? The code made no > sense, was incomplete, and the posted question was nonsensical. Yes in this specific instance all this is probably true. I believe however, that Joel's intent in reposting this is more global (and important) in its scope, viz: If this list persists in the current unhealthy state which it is experiencing, authentic noob questions will get buried in mountains of bullshit. Note: I find Joshua's answer fine given the question. > If the OP has abandoned it, so should we. If you were a noob-OP who asked that question and the result unfolded as it has, what would you do? From davea at davea.name Mon Jul 1 21:11:32 2013 From: davea at davea.name (Dave Angel) Date: Mon, 01 Jul 2013 21:11:32 -0400 Subject: python adds an extra half space when reading froma string or list -- back to the question In-Reply-To: References: Message-ID: <51D228C4.8080500@davea.name> On 07/01/2013 05:16 PM, rusi wrote: > On Tuesday, July 2, 2013 1:32:44 AM UTC+5:30, Dave Angel wrote: >> > > Yes in this specific instance all this is probably true. > I believe however, that Joel's intent in reposting this is more global (and important) in its scope, viz: > > If this list persists in the current unhealthy state which it is experiencing, authentic noob questions will get buried in mountains of bullshit. > > Note: I find Joshua's answer fine given the question. > >> If the OP has abandoned it, so should we. > > If you were a noob-OP who asked that question and the result unfolded as it has, what would you do? > Point well-taken. So I'll see what I can do here. I'll put comments on lines I had to add or change. finale_line = [] #missing initialization lot_number = 99 number_drawn=() def load(lot_number,number_drawn): first=input("enter first lot: ") last=input("enter last lot: ") for lot_number in range(first,last): line_out=str(lot_number) for count in range(1,5): number_drawn=raw_input("number: ") line_out=line_out+(number_drawn) print line_out finale_line.append(line_out) #finale_line2=finale_line #not referenced load(lot_number,number_drawn) print finale_line print(" "*4), for n in range(1,41): print n, #this is to produce a line of numbers to #compare to output# for a in finale_line: print"\n", print a[0]," ", space_count=1 for b in range(1,5): if int(a[b])<10: print(" "*(int(a[b])-space_count)),int(a[b]), space_count=int(a[b]) else: pass #print(" "*(a[b]-space_count)),a[b], #dead code #space_count=a[b]+1 #dead code Since all the numbers are butted together in the string line_out, the later logic is iterating over digits, which cannot be bigger than 9. So the else clause is nonsensical. Even if they were to run, they'd give runtime errors. > #as you can see many numbers are between the lines of a normal print# > #I thought this was due to "white space" int he format .So I tried a > list > of strings and got the same results.# No clue what that means. What numbers are between what lines? And what is a normal print? Presumably the intent was to somehow do a variable spacing of those digits. -- DaveA From wuwei23 at gmail.com Mon Jul 1 22:37:08 2013 From: wuwei23 at gmail.com (alex23) Date: Tue, 02 Jul 2013 12:37:08 +1000 Subject: python adds an extra half space when reading froma string or list -- back to the question In-Reply-To: References: Message-ID: On 2/07/2013 5:32 AM, Joel Goldstick wrote: > I copied the original question so that the rant on the other > thread can continue. Let's keep this thread ontopic You've included the same set of code twice. Also, it doesn't run as is, so you haven't reduced it to a minimal working example for us to test. Python's print adds a space where there's a comma, try this at the interactive prompt: >>> print 'a','b' a b I don't think you're taking this into account. There are quite a few other problems with your code, though: > number_drawn=() > def load(lot_number,number_drawn): You've assigned `number_drawn` to a tuple, which is an immutable type. Within the `load` function you then do: > number_drawn=raw_input("number: ") Which just re-assigns `number_drawn` to the output of the `raw_input` function _within the function itself_. The value isn't available outside of the function, if that is what you're intending. You also haven't defined `lot_number` in your code, and again you re-assign it within the body of the `load` function: > for lot_number in range(first,last): Which has no impact on any global definition. Since you're not actually passing values into the function you can probably do without both arguments and just go with: def load(): > first=input("enter first lot: ") > last=input("enter last lot: ") You should never use `input`, it evaluates the expression entered. Always use `raw_input` unless you absolutely know what you're doing. > for lot_number in range(first,last): `range` will start at `first` and finish _at but not including `last`. If you want 4 lines when you enter first=1 and last=4, then you need to increment `last` by 1: for lot_number in range(first,last+1): > finale_line.append(line_out) Where is `finale_line` defined? You haven't shown it but my guess is you've made it an empty list. You're also not returning anything from the function, which implies you're relying on global scope to hold the result. This is bad practice. You should start your function with: finale_line = [] And then end it with: return finale_line Which would allow you to replace: > finale_line2=finale_line > load(lot_number,number_drawn) With: finale_line = load() > for a in finale_line: > print"\n", > print a[0]," ", > space_count=1 > for b in range(1,5): This condition will _always_ be true: > if int(a[b])<10: Because of the way you're stepping through `finale_line`, you're only ever looking at a single character, so that value will always be from 0 to 9. You clearly want to allow for double digit numbers, so not storing them all as a big string would be a good start. You're dealing with numbers, so hold them as numbers in a list. Here's a more flexible approach that I think does what you want: import os def load_numbers(): first=int(raw_input("enter first lot: ")) last=int(raw_input("enter last lot: ")) finale_line = [] for lot_number in range(first,last+1): line_out = [] for count in range(1,5): number_drawn=raw_input("lot %d, number %d: " % (lot_number, count)) line_out.append(number_drawn) finale_line.append((lot_number, line_out)) return finale_line finale_line = load_numbers() # get space needed from the maximum number entered largest_number = max( number for row,numbers in finale_line for number in numbers) space_count = len(str(largest_number)) # get no. of columns from the first set of numbers columns = len(finale_line[0][1]) # space to cover the lot numbers print ' ', # print the columns for column in xrange(1, columns+1): spacing = space_count - len(str(column)) print '%s%d' % (' '*spacing, column), for lot, numbers in finale_line: print os.linesep, # use the EOL used by the current # operating system print lot, for number in numbers: spacing = space_count - len(str(number)) print '%s%d' % (' '*spacing, number), However, if you want to make your life a _lot_ easier when printing tabular data, I highly recommend using a library dedicated to just that, something like: https://code.google.com/p/prettytable/ From jsf80238 at gmail.com Mon Jul 1 19:07:57 2013 From: jsf80238 at gmail.com (Jason Friedman) Date: Mon, 1 Jul 2013 17:07:57 -0600 Subject: Regular expression negative look-ahead Message-ID: I have table names in this form: MY_TABLE MY_TABLE_CTL MY_TABLE_DEL MY_TABLE_RUN YOUR_TABLE YOUR_TABLE_CTL YOUR_TABLE_DEL YOUR_TABLE_RUN I am trying to create a regular expression that will return true for only these tables: MY_TABLE YOUR_TABLE I tried these: pattern = re.compile(r"_(?!(CTL|DEL|RUN))") pattern = re.compile(r"\w+(?!(CTL|DEL|RUN))") pattern = re.compile(r"(?!(CTL|DEL|RUN)$)") But, both match. I do not need to capture anything. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsf80238 at gmail.com Mon Jul 1 22:27:51 2013 From: jsf80238 at gmail.com (Jason Friedman) Date: Mon, 1 Jul 2013 20:27:51 -0600 Subject: Regular expression negative look-ahead In-Reply-To: References: Message-ID: Found this: http://stackoverflow.com/questions/13871833/negative-lookahead-assertion-not-working-in-python . This pattern seems to work: pattern = re.compile(r"^(?!.*(CTL|DEL|RUN))") But I am not sure why. On Mon, Jul 1, 2013 at 5:07 PM, Jason Friedman wrote: > I have table names in this form: > MY_TABLE > MY_TABLE_CTL > MY_TABLE_DEL > MY_TABLE_RUN > YOUR_TABLE > YOUR_TABLE_CTL > YOUR_TABLE_DEL > YOUR_TABLE_RUN > > I am trying to create a regular expression that will return true for only > these tables: > MY_TABLE > YOUR_TABLE > > I tried these: > pattern = re.compile(r"_(?!(CTL|DEL|RUN))") > pattern = re.compile(r"\w+(?!(CTL|DEL|RUN))") > pattern = re.compile(r"(?!(CTL|DEL|RUN)$)") > > But, both match. > I do not need to capture anything. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ian.g.kelly at gmail.com Tue Jul 2 01:44:31 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 1 Jul 2013 23:44:31 -0600 Subject: Regular expression negative look-ahead In-Reply-To: References: Message-ID: On Mon, Jul 1, 2013 at 8:27 PM, Jason Friedman wrote: > Found this: > http://stackoverflow.com/questions/13871833/negative-lookahead-assertion-not-working-in-python. > > This pattern seems to work: > pattern = re.compile(r"^(?!.*(CTL|DEL|RUN))") > > But I am not sure why. > > > On Mon, Jul 1, 2013 at 5:07 PM, Jason Friedman wrote: >> >> I have table names in this form: >> MY_TABLE >> MY_TABLE_CTL >> MY_TABLE_DEL >> MY_TABLE_RUN >> YOUR_TABLE >> YOUR_TABLE_CTL >> YOUR_TABLE_DEL >> YOUR_TABLE_RUN >> >> I am trying to create a regular expression that will return true for only >> these tables: >> MY_TABLE >> YOUR_TABLE >> >> I tried these: >> pattern = re.compile(r"_(?!(CTL|DEL|RUN))") >> pattern = re.compile(r"\w+(?!(CTL|DEL|RUN))") >> pattern = re.compile(r"(?!(CTL|DEL|RUN)$)") >> >> But, both match. >> I do not need to capture anything. For some reason I don't seem to have a copy of your initial post. The reason that regex works is because you're anchoring it at the start of the string and then telling it to match only if ".*(CTL|DEL|RUN)" /doesn't/ match. That pattern does match starting from the beginning of the string, so the pattern as a whole does not match. The reason that the other three do not work is because the forward assertions are not properly anchored. The first one can match the first underscore in "MY_TABLE_CTL" instead of the second, and then the next three characters are "TAB", not any of the verboten strings, so it matches. The second one matches any substring of "MY_TABLE_CTL" that isn't followed by "CTL". So it will just match the entire string "MY_TABLE_CTL", and the rest of the string is then empty, so does not match any of those three strings, so it too gets accepted. The third one simply matches an empty string that isn't followed by one of those three, so it will just match at the very start of the string and see that the next three characters meet the forward assertion. Now, all that said, are you sure you actually need a regular expression for this? It seems to me that you're overcomplicating things. Since you don't need to capture anything, your need can be met more simply with: if not table_name.endswith(('_CTL', '_DEL', '_RUN')): # Do whatever From jsf80238 at gmail.com Wed Jul 3 22:49:23 2013 From: jsf80238 at gmail.com (Jason Friedman) Date: Wed, 3 Jul 2013 20:49:23 -0600 Subject: Regular expression negative look-ahead In-Reply-To: References: Message-ID: Huh, did not realize that endswith takes a list. I'll remember that in the future. This need is actually for http://schemaspy.sourceforge.net/, which allows one to include only tables/views that match a pattern. Either there is a bug in Schemaspy's code or Java's implementation of regular expressions is different than Python's or there is a flaw in my logic, because the pattern I verify using Python produces different results when used with Schemaspy. I suppose I'll open a bug there unless I can find the aforementioned flaw. On Mon, Jul 1, 2013 at 11:44 PM, Ian Kelly wrote: > On Mon, Jul 1, 2013 at 8:27 PM, Jason Friedman wrote: > > Found this: > > > http://stackoverflow.com/questions/13871833/negative-lookahead-assertion-not-working-in-python > . > > > > This pattern seems to work: > > pattern = re.compile(r"^(?!.*(CTL|DEL|RUN))") > > > > But I am not sure why. > > > > > > On Mon, Jul 1, 2013 at 5:07 PM, Jason Friedman > wrote: > >> > >> I have table names in this form: > >> MY_TABLE > >> MY_TABLE_CTL > >> MY_TABLE_DEL > >> MY_TABLE_RUN > >> YOUR_TABLE > >> YOUR_TABLE_CTL > >> YOUR_TABLE_DEL > >> YOUR_TABLE_RUN > >> > >> I am trying to create a regular expression that will return true for > only > >> these tables: > >> MY_TABLE > >> YOUR_TABLE > >> > >> I tried these: > >> pattern = re.compile(r"_(?!(CTL|DEL|RUN))") > >> pattern = re.compile(r"\w+(?!(CTL|DEL|RUN))") > >> pattern = re.compile(r"(?!(CTL|DEL|RUN)$)") > >> > >> But, both match. > >> I do not need to capture anything. > > > For some reason I don't seem to have a copy of your initial post. > > The reason that regex works is because you're anchoring it at the > start of the string and then telling it to match only if > ".*(CTL|DEL|RUN)" /doesn't/ match. That pattern does match starting > from the beginning of the string, so the pattern as a whole does not > match. > > The reason that the other three do not work is because the forward > assertions are not properly anchored. The first one can match the > first underscore in "MY_TABLE_CTL" instead of the second, and then the > next three characters are "TAB", not any of the verboten strings, so > it matches. The second one matches any substring of "MY_TABLE_CTL" > that isn't followed by "CTL". So it will just match the entire string > "MY_TABLE_CTL", and the rest of the string is then empty, so does not > match any of those three strings, so it too gets accepted. The third > one simply matches an empty string that isn't followed by one of those > three, so it will just match at the very start of the string and see > that the next three characters meet the forward assertion. > > Now, all that said, are you sure you actually need a regular > expression for this? It seems to me that you're overcomplicating > things. Since you don't need to capture anything, your need can be > met more simply with: > > if not table_name.endswith(('_CTL', '_DEL', '_RUN')): > # Do whatever > -- > http://mail.python.org/mailman/listinfo/python-list > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsf80238 at gmail.com Wed Jul 3 22:50:51 2013 From: jsf80238 at gmail.com (Jason Friedman) Date: Wed, 3 Jul 2013 20:50:51 -0600 Subject: Regular expression negative look-ahead In-Reply-To: References: Message-ID: > Huh, did not realize that endswith takes a list. I'll remember that in > the future. > > This need is actually for http://schemaspy.sourceforge.net/, which allows > one to include only tables/views that match a pattern. > > Either there is a bug in Schemaspy's code or Java's implementation of > regular expressions is different than Python's or there is a flaw in my > logic, because the pattern I verify using Python produces different results > when used with Schemaspy. I suppose I'll open a bug there unless I can > find the aforementioned flaw. > > > if not table_name.endswith(('_CTL', '_DEL', '_RUN')): >> # Do whatever >> >> > Shoot, sorry, did not mean to top-post. -------------- next part -------------- An HTML attachment was scrubbed... URL: From neilc at norwich.edu Tue Jul 2 08:15:10 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 2 Jul 2013 12:15:10 GMT Subject: Regular expression negative look-ahead References: Message-ID: On 2013-07-01, Jason Friedman wrote: > > I have table names in this form: > MY_TABLE > MY_TABLE_CTL > MY_TABLE_DEL > MY_TABLE_RUN > YOUR_TABLE > YOUR_TABLE_CTL > YOUR_TABLE_DEL > YOUR_TABLE_RUN > > I am trying to create a regular expression that will return true for only > these tables: > MY_TABLE > YOUR_TABLE Use the "is not a word" character class on either end. r"\WMY_TABLE\W" r"\WYOUR_TABLE\W" -- Neil Cerutti From nikos at superhost.gr Mon Jul 1 23:37:15 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Tue, 02 Jul 2013 06:37:15 +0300 Subject: OSError [Errno 26] ?!?! Message-ID: After making a slightly chnage inside my pelatologio.py script substituting '*****' instead of '-----' for no apparent reason i receive the following error: [Tue Jul 02 06:33:06 2013] [error] [client 46.12.97.148] OSError: [Errno 26] \\u0391\\u03c1\\u03c7\\u03b5\\u03af\\u03bf \\u03ba\\u03b5\\u03b9\\u03bc\\u03ad\\u03bd\\u03bf\\u03c5 \\u03c3\\u03b5 \\u03c7\\u03c1 What is this OSError and these unicode characters besided it? Here is the complete traceback. [Tue Jul 02 06:33:06 2013] [error] [client 46.12.97.148] Original exception was:, referer: http://superhost.gr/ [Tue Jul 02 06:33:06 2013] [error] [client 46.12.97.148] Traceback (most recent call last):, referer: http://superhost.gr/ [Tue Jul 02 06:33:06 2013] [error] [client 46.12.97.148] File "/home/nikos/public_html/cgi-bin/metrites.py", line 222, in , referer: http://superhost.gr/ [Tue Jul 02 06:33:06 2013] [error] [client 46.12.97.148] pypage = subprocess.check_output( cgi_path + page ), referer: http://superhost.gr/ [Tue Jul 02 06:33:06 2013] [error] [client 46.12.97.148] File "/usr/local/lib/python3.3/subprocess.py", line 573, in check_output, referer: http://superhost.gr/ [Tue Jul 02 06:33:06 2013] [error] [client 46.12.97.148] with Popen(*popenargs, stdout=PIPE, **kwargs) as process:, referer: http://superhost.gr/ [Tue Jul 02 06:33:06 2013] [error] [client 46.12.97.148] File "/usr/local/lib/python3.3/subprocess.py", line 820, in __init__, referer: http://superhost.gr/ [Tue Jul 02 06:33:06 2013] [error] [client 46.12.97.148] restore_signals, start_new_session), referer: http://superhost.gr/ [Tue Jul 02 06:33:06 2013] [error] [client 46.12.97.148] File "/usr/local/lib/python3.3/subprocess.py", line 1438, in _execute_child, referer: http://superhost.gr/ [Tue Jul 02 06:33:06 2013] [error] [client 46.12.97.148] raise child_exception_type(errno_num, err_msg), referer: http://superhost.gr/ [Tue Jul 02 06:33:06 2013] [error] [client 46.12.97.148] OSError: [Errno 26] \\u0391\\u03c1\\u03c7\\u03b5\\u03af\\u03bf \\u03ba\\u03b5\\u03b9\\u03bc\\u03ad\\u03bd\\u03bf\\u03c5 \\u03c3\\u03b5 \\u03c7\\u03c1\\u03ae\\u03c3\\u03b7, referer: http://superhost.gr/ -- What is now proved was at first only imagined! From cs at zip.com.au Tue Jul 2 01:15:32 2013 From: cs at zip.com.au (Cameron Simpson) Date: Tue, 2 Jul 2013 15:15:32 +1000 Subject: OSError [Errno 26] ?!?! In-Reply-To: References: Message-ID: <20130702051532.GA63146@cskk.homeip.net> On 02Jul2013 06:37, ????? wrote: | After making a slightly chnage inside my pelatologio.py script | substituting '*****' instead of '-----' for no apparent reason i | receive the following error: | | [Tue Jul 02 06:33:06 2013] [error] [client 46.12.97.148] OSError: | [Errno 26] \\u0391\\u03c1\\u03c7\\u03b5\\u03af\\u03bf | \\u03ba\\u03b5\\u03b9\\u03bc\\u03ad\\u03bd\\u03bf\\u03c5 | \\u03c3\\u03b5 \\u03c7\\u03c1 Errno 26 depends on your operating system. See "man 2 intro" for details. But on my Mac and on a handle Linux system number 26 is: 26 ETXTBSY Text file busy. The new process was a pure procedure (shared text) file which was open for writing by another process, or while the pure procedure file was being executed an open call requested write access. See the os.strerror function for printing the message from an errno number: http://docs.python.org/3/library/os.html#os.strerror I'd say you're trying to write to a file you shouldn't be writing to. Maybe an executable? Try to get your script to print out the filename of whatever it was trying to overwrite before the open() call. Regarding the characters, I'd say they've been double escaped. Let's undo that: [/Users/cameron]fleet*> py3 Python 3.3.2 (default, May 21 2013, 11:50:47) [GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> print('\\u0391\\u03c1\\u03c7\\u03b5\\u03af\\u03bf \\u03ba\\u03b5\\u03b9\\u03bc\\u03ad\\u03bd\\u03bf\\u03c5 \\u03c3\\u03b5 \\u03c7\\u03c1') \u0391\u03c1\u03c7\u03b5\u03af\u03bf \u03ba\u03b5\u03b9\u03bc\u03ad\u03bd\u03bf\u03c5 \u03c3\u03b5 \u03c7\u03c1 >>> print('\u0391\u03c1\u03c7\u03b5\u03af\u03bf \u03ba\u03b5\u03b9\u03bc\u03ad\u03bd\u03bf\u03c5 \u03c3\u03b5 \u03c7\u03c1') ?????? ???????? ?? ?? >>> I can't read that, but I'd hope you can. | What is this OSError and these unicode characters besided it? | Here is the complete traceback. A traceback without code isn't terribly useful. Cheers, -- Cameron Simpson From nikos at superhost.gr Tue Jul 2 01:57:34 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Tue, 02 Jul 2013 08:57:34 +0300 Subject: OSError [Errno 26] ?!?! In-Reply-To: References: Message-ID: ???? 2/7/2013 8:15 ??, ?/? Cameron Simpson ??????: > On 02Jul2013 06:37, ????? wrote: > | After making a slightly chnage inside my pelatologio.py script > | substituting '*****' instead of '-----' for no apparent reason i > | receive the following error: > | > | [Tue Jul 02 06:33:06 2013] [error] [client 46.12.97.148] OSError: > | [Errno 26] \\u0391\\u03c1\\u03c7\\u03b5\\u03af\\u03bf > | \\u03ba\\u03b5\\u03b9\\u03bc\\u03ad\\u03bd\\u03bf\\u03c5 > | \\u03c3\\u03b5 \\u03c7\\u03c1 > > Errno 26 depends on your operating system. See "man 2 intro" for details. > But on my Mac and on a handle Linux system number 26 is: > > 26 ETXTBSY Text file busy. The new process was a pure procedure (shared > text) file which was open for writing by another process, or > while the pure procedure file was being executed an open call > requested write access. > > See the os.strerror function for printing the message from an errno number: > > http://docs.python.org/3/library/os.html#os.strerror > > I'd say you're trying to write to a file you shouldn't be writing to. > Maybe an executable? > > Try to get your script to print out the filename of whatever it was trying to > overwrite before the open() call. > > Regarding the characters, I'd say they've been double escaped. Let's undo that: > > [/Users/cameron]fleet*> py3 > Python 3.3.2 (default, May 21 2013, 11:50:47) > [GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> print('\\u0391\\u03c1\\u03c7\\u03b5\\u03af\\u03bf \\u03ba\\u03b5\\u03b9\\u03bc\\u03ad\\u03bd\\u03bf\\u03c5 \\u03c3\\u03b5 \\u03c7\\u03c1') > \u0391\u03c1\u03c7\u03b5\u03af\u03bf \u03ba\u03b5\u03b9\u03bc\u03ad\u03bd\u03bf\u03c5 \u03c3\u03b5 \u03c7\u03c1 > >>> print('\u0391\u03c1\u03c7\u03b5\u03af\u03bf \u03ba\u03b5\u03b9\u03bc\u03ad\u03bd\u03bf\u03c5 \u03c3\u03b5 \u03c7\u03c1') > ?????? ???????? ?? ?? > >>> > > I can't read that, but I'd hope you can. > > | What is this OSError and these unicode characters besided it? > | Here is the complete traceback. > > A traceback without code isn't terribly useful. > > Cheers, Thank you, the error have been caused due to the fact that the uploading procedure of my edited 'pelatologio.py' hadn't been completed yet, while i was requesting the execution of the script via browser. I didn't had to do anything it solved by itself, after upload was successful and file got unlocked for access. Thanks Cameron, and indeed the error message as you printed the Unicode characters was saying in Greek that file was in use. -- What is now proved was at first only imagined! From cs at zip.com.au Tue Jul 2 03:21:38 2013 From: cs at zip.com.au (Cameron Simpson) Date: Tue, 2 Jul 2013 17:21:38 +1000 Subject: OSError [Errno 26] ?!?! In-Reply-To: References: Message-ID: <20130702072138.GA48551@cskk.homeip.net> On 02Jul2013 08:57, ????? wrote: | Thank you, the error have been caused due to the fact that the | uploading procedure of my edited 'pelatologio.py' hadn't been | completed yet, while i was requesting the execution of the script | via browser. | | I didn't had to do anything it solved by itself, after upload was | successful and file got unlocked for access. If you're uploading using sftp (or, worse, ftp), might I suggest using rsync instead? Amongst other features, it uploads to a temporary file and then renames the temporary file to the real file. There is no window where the "live" file is being written to. The live file is the old one, and then later it is the new one. Cheers, -- Cameron Simpson No team manager will tell you this; but they all want to see you come walking back into the pits sometimes, carrying the steering wheel. - Mario Andretti From nikos at superhost.gr Tue Jul 2 05:40:59 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Tue, 02 Jul 2013 12:40:59 +0300 Subject: OSError [Errno 26] ?!?! In-Reply-To: References: Message-ID: ???? 2/7/2013 10:21 ??, ?/? Cameron Simpson ??????: > On 02Jul2013 08:57, ????? wrote: > | Thank you, the error have been caused due to the fact that the > | uploading procedure of my edited 'pelatologio.py' hadn't been > | completed yet, while i was requesting the execution of the script > | via browser. > | > | I didn't had to do anything it solved by itself, after upload was > | successful and file got unlocked for access. > > If you're uploading using sftp (or, worse, ftp), might I suggest > using rsync instead? Amongst other features, it uploads to a temporary > file and then renames the temporary file to the real file. There is > no window where the "live" file is being written to. The live file > is the old one, and then later it is the new one. Actually i'm uploading via Notepad++'s NPPFtp plugin, which i'm afraid is pure ftp and not even sftp. If i try to upload via FileZilla instead(which i can use as sftp on port 22), then it messes the cgi scripts encoding by uploading it as iso-8859-7 which then i need to ssh to the remote host and change it back to utf-8. Also sometimes it takes a lot of time even with Notepad++ to even upload a 50 KB script. Please tell me how to use the rsync method especially it would be best if i cna use it via text editor, Notepad++ or even better with Sublime Text. Thank you Cameron. -- What is now proved was at first only imagined! From kwpolska at gmail.com Tue Jul 2 06:06:32 2013 From: kwpolska at gmail.com (=?UTF-8?B?Q2hyaXMg4oCcS3dwb2xza2HigJ0gV2Fycmljaw==?=) Date: Tue, 2 Jul 2013 12:06:32 +0200 Subject: OSError [Errno 26] ?!?! In-Reply-To: References: Message-ID: On Tue, Jul 2, 2013 at 11:40 AM, ????? wrote: > ???? 2/7/2013 10:21 ??, ?/? Cameron Simpson ??????: >> >> On 02Jul2013 08:57, ????? wrote: >> | Thank you, the error have been caused due to the fact that the >> | uploading procedure of my edited 'pelatologio.py' hadn't been >> | completed yet, while i was requesting the execution of the script >> | via browser. >> | >> | I didn't had to do anything it solved by itself, after upload was >> | successful and file got unlocked for access. >> >> If you're uploading using sftp (or, worse, ftp), might I suggest >> using rsync instead? Amongst other features, it uploads to a temporary >> file and then renames the temporary file to the real file. There is >> no window where the "live" file is being written to. The live file >> is the old one, and then later it is the new one. > > > > Actually i'm uploading via Notepad++'s NPPFtp plugin, which i'm afraid is > pure ftp and not even sftp. The only thing SFTP and FTP have in common is that they are Internet protocols. Inside, they are much, much different. And you should NEVER use FTP. (also, why Notepad++? I?d suggest getting a better editor first.) > If i try to upload via FileZilla instead(which i can use as sftp on port > 22), then it messes the cgi scripts encoding by uploading it as iso-8859-7 > which then i need to ssh to the remote host and change it back to utf-8. You need to reconfigure FileZilla then. Site Manager (first icon on the toolbar) ? your site ? Charset ? Force UTF-8. > Also sometimes it takes a lot of time even with Notepad++ to even upload a > 50 KB script. > > Please tell me how to use the rsync method especially it would be best if i > cna use it via text editor, Notepad++ or even better with Sublime Text. Under Windows? I suggest installing Cygwin or switching over to Linux, for your sanity. If you don?t want either, look for a Windows port of rsync. The next step is to read the included man page. -- Kwpolska | GPG KEY: 5EAAEA16 stop html mail | always bottom-post http://asciiribbon.org | http://caliburn.nl/topposting.html From davea at davea.name Tue Jul 2 08:00:15 2013 From: davea at davea.name (Dave Angel) Date: Tue, 02 Jul 2013 08:00:15 -0400 Subject: OSError [Errno 26] ?!?! In-Reply-To: References: Message-ID: <51D2C0CF.4050102@davea.name> On 07/02/2013 06:06 AM, Chris ?Kwpolska? Warrick wrote: > On Tue, Jul 2, 2013 at 11:40 AM, ????? wrote: >> ???? 2/7/2013 10:21 ??, ?/? Cameron Simpson ??????: >>> >>> On 02Jul2013 08:57, ????? wrote: >>> | Thank you, the error have been caused due to the fact that the >>> | uploading procedure of my edited 'pelatologio.py' hadn't been >>> | completed yet, while i was requesting the execution of the script >>> | via browser. >>> | >>> | I didn't had to do anything it solved by itself, after upload was >>> | successful and file got unlocked for access. >>> >>> If you're uploading using sftp (or, worse, ftp), might I suggest >>> using rsync instead? Amongst other features, it uploads to a temporary >>> file and then renames the temporary file to the real file. There is >>> no window where the "live" file is being written to. The live file >>> is the old one, and then later it is the new one. >> >> >> >> Actually i'm uploading via Notepad++'s NPPFtp plugin, which i'm afraid is >> pure ftp and not even sftp. > > The only thing SFTP and FTP have in common is that they are Internet > protocols. Inside, they are much, much different. And you should > NEVER use FTP. (also, why Notepad++? I?d suggest getting a better > editor first.) > >> If i try to upload via FileZilla instead(which i can use as sftp on port >> 22), then it messes the cgi scripts encoding by uploading it as iso-8859-7 >> which then i need to ssh to the remote host and change it back to utf-8. > > You need to reconfigure FileZilla then. Site Manager (first icon on > the toolbar) ? your site ? Charset ? Force UTF-8. Much better: upload all files as binary, so they are byte for byte identical on both systems. Sometimes that might require some local preparing, but that beats strange problems with encodings, line endings, etc. Perhaps it's obvious, but that also means a local staging area whose directory tree is identical to the target system. > >> Also sometimes it takes a lot of time even with Notepad++ to even upload a >> 50 KB script. >> >> Please tell me how to use the rsync method especially it would be best if i >> cna use it via text editor, Notepad++ or even better with Sublime Text. Once you have done my suggestions above, a single rsynch command will upload any files that have changed, without you specifying which ones they are. So you don't need the "integrated file transfer feature" of various editors & guis. You just run one batch file which you set up, and when it finishes, the systems will match. > > Under Windows? I suggest installing Cygwin or switching over to > Linux, for your sanity. If you don?t want either, look for a Windows > port of rsync. The next step is to read the included man page. > rsynch has lots of option switches, but once set up, it's trivial to use. -- DaveA From nikos at superhost.gr Tue Jul 2 08:39:08 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Tue, 02 Jul 2013 15:39:08 +0300 Subject: OSError [Errno 26] ?!?! In-Reply-To: References: Message-ID: ???? 2/7/2013 3:00 ??, ?/? Dave Angel ??????: ou need to reconfigure FileZilla then. Site Manager (first icon on >> the toolbar) ? your site ? Charset ? Force UTF-8. > > Much better: upload all files as binary, so they are byte for byte > identical on both systems. Sometimes that might require some local > preparing, but that beats strange problems with encodings, line endings, > etc. I currently have it on Auto-Detect. I think its smart enough what is a binary file and what is an ASCII script, not sure though. > Once you have done my suggestions above, a single rsynch command will > upload any files that have changed, without you specifying which ones > they are. So you don't need the "integrated file transfer feature" of > various editors & guis. You just run one batch file which you set up, > and when it finishes, the systems will match. How and from where should i use rsync? I currently ditched FileZilla and start using CrossFTP Pro. I searched inside it but i have seen no rsync command/method just "Synchronized Browsing" >> Under Windows? I suggest installing Cygwin or switching over to >> Linux, for your sanity. If you don?t want either, look for a Windows >> port of rsync. The next step is to read the included man page. >> > > rsynch has lots of option switches, but once set up, it's trivial to use. Is rsync a command on the remote server or can be found as a standalone tool too? Please suggest an editor that has built in rsync ability so to immediately upload my cgi-scripts when i hit save in the text editor. How do you guys upload your files/scripts to remote hosts ? -- What is now proved was at first only imagined! From kwpolska at gmail.com Tue Jul 2 09:00:29 2013 From: kwpolska at gmail.com (=?UTF-8?B?Q2hyaXMg4oCcS3dwb2xza2HigJ0gV2Fycmljaw==?=) Date: Tue, 2 Jul 2013 15:00:29 +0200 Subject: OSError [Errno 26] ?!?! In-Reply-To: References: Message-ID: On Tue, Jul 2, 2013 at 2:39 PM, ????? wrote: > How and from where should i use rsync? >From your computer, in the command line. > I currently ditched FileZilla and start using CrossFTP Pro. > I searched inside it but i have seen no rsync command/method just > "Synchronized Browsing" Look for a rsync client, not a FTP/SFTP client. > Is rsync a command on the remote server or can be found as a standalone tool > too? Both. > Please suggest an editor that has built in rsync ability so to immediately > upload my cgi-scripts when i hit save in the text editor. CGI? Is this 2000? Nobody uses that wording these days. I?ve yet to see a Windowsy editor with rsync support. But you should choose the editor that works the best for you, not the editor which has rsync support. > How do you guys upload your files/scripts to remote hosts ? I, for one, use sftp or edit directly on the server with vim, my editor of choice. -- Kwpolska | GPG KEY: 5EAAEA16 stop html mail | always bottom-post http://asciiribbon.org | http://caliburn.nl/topposting.html From robert.kern at gmail.com Tue Jul 2 09:58:18 2013 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 02 Jul 2013 14:58:18 +0100 Subject: OSError [Errno 26] ?!?! In-Reply-To: References: Message-ID: On 2013-07-02 14:00, Chris ?Kwpolska? Warrick wrote: > On Tue, Jul 2, 2013 at 2:39 PM, ????? wrote: >> Please suggest an editor that has built in rsync ability so to immediately >> upload my cgi-scripts when i hit save in the text editor. > > CGI? Is this 2000? Nobody uses that wording these days. He is indeed using actual, bona fide CGI scripts. It's not just an antiquated wording for "web app". -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From square.steve at gmail.com Tue Jul 2 11:51:38 2013 From: square.steve at gmail.com (Steve Simmons) Date: Tue, 02 Jul 2013 16:51:38 +0100 Subject: Python list code of conduct In-Reply-To: References: Message-ID: Erm, It probably isn't the best time to start this post but I was wondering... Does this list have a code of conduct or a netiqeutte (sp?) statement/requirement? If not, should it? Is the membership of this list presently in the right frame of mind to create one or update any existing one? The reason I ask is that it seems to me that if we (the current membership of the list) could agree to a set of preferred/required behaviours we would at least have a framework by which to measure unwelcome posts. And, more importantly, to guide newcomers in understanding that we are enthusiasts who choose to discuss Python and *voluntarily* help solve problems with Python for the less experienced members. [Runs for cover] Steve Simmons Sent from a Galaxy far far away -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshua.landau.ws at gmail.com Tue Jul 2 15:48:54 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Tue, 2 Jul 2013 20:48:54 +0100 Subject: Python list code of conduct In-Reply-To: References: Message-ID: On 2 July 2013 16:51, Steve Simmons wrote: > Erm, > > It probably isn't the best time to start this post but I was wondering... > > Does this list have a code of conduct or a netiqeutte (sp?) > statement/requirement? > > If not, should it? > > Is the membership of this list presently in the right frame of mind to > create one or update any existing one? > > The reason I ask is that it seems to me that if we (the current membership > of the list) could agree to a set of preferred/required behaviours we would > at least have a framework by which to measure unwelcome posts. And, more > importantly, to guide newcomers in understanding that we are enthusiasts who > choose to discuss Python and *voluntarily* help solve problems with Python > for the less experienced members. I don't know if we do and I'll support such measures (pending that they're done intelligently) but unfortunately there seem to be at least two problems with your plan. - There seems to be no authority for upholding such rules - Newbies never read around That said, the idea itself is well grounded. I'd suggest stealing the rules Python already has somewhere. From skip at pobox.com Tue Jul 2 16:33:42 2013 From: skip at pobox.com (Skip Montanaro) Date: Tue, 2 Jul 2013 15:33:42 -0500 Subject: Python list code of conduct In-Reply-To: References: Message-ID: > Does this list have a code of conduct or a netiqeutte (sp?) statement/requirement? > > If not, should it? No, and probably not. As Joshua Landau indicated, nobody reads the user manual anyway. In addition, this mailing list is not currently moderated in an approve-each-message sort of way (and I doubt anyone wants to take on that task). The gateway with comp.lang.python makes things somewhat worse I would think. I've forgotten how Usenet works for moderation (far too many dead neurons ago), but I doubt it would be all that effective. ISTR moderation is done using a simple header which can be faked. If someone annoys you to no end, I think your best recourse is to ignore their posts or ignore threads which seem to generate more heat than light. In the old Usenet days, periodic (monthly?) posting of FAQs was common in many newsgroups. Ignoring fake newsgroups like the stuff that Google Groups and Gmane create, this mailing list is my only exposure to Usenet. I have no idea if periodic FAQ posting is common practice anymore. It might be worthwhile to create one if it's kept fairly brief and to the point. Skip From nad at acm.org Tue Jul 2 16:52:01 2013 From: nad at acm.org (Ned Deily) Date: Tue, 02 Jul 2013 13:52:01 -0700 Subject: Python list code of conduct References: Message-ID: In article , Steve Simmons wrote: > Does this list have a code of conduct or a netiqeutte (sp?) > statement/requirement? >From http://www.python.org/community/lists/ comp.lang.python comp.lang.python is a high-volume Usenet open (not moderated) newsgroup for general discussions and questions about Python. You can also access it as a mailing list through python-list. Pretty much anything Python-related is fair game for discussion, and the group is even fairly tolerant of off-topic digressions; there have been entertaining discussions of topics such as floating point, good software design, and other programming languages such as Lisp and Forth. Most discussion on comp.lang.python is about developing with Python, not about development of the Python interpreter itself. Some of the core developers still read the list, but most of them don't. Occasionally comp.lang.python suggestions have resulted in an enhancement proposal being written, leading to a new Python feature. If you find a bug in Python, don't send it to comp.lang.python; file a bug report in the issue tracker. Items posted on the Usenet group appear on the mailing list, and vice versa (bidirectional gateway). Due to the mysteries of Usenet, the order in which items show up may vary. Rudeness and personal attacks, even in reaction to blatant flamebait, are strongly frowned upon. People may strongly disagree on an issue, but usually discussion remains civil. In case of an actual flamebait posting, you can ignore it, quietly plonk the offending poster in your killfile or mail filters, or write a sharp but still-polite response, but at all costs resist the urge to flame back. Generally comp.lang.python is a high-signal, low-noise group. It's also a high-traffic group, running at around 200 posts per day. There are several different archives: ? Google Groups archive of comp.lang.python ? python.org archive of python-list ? gmane.org archive of python-list -- Ned Deily, nad at acm.org From ben+python at benfinney.id.au Tue Jul 2 18:39:59 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 03 Jul 2013 08:39:59 +1000 Subject: Python list code of conduct References: Message-ID: <7wobakh9hc.fsf@benfinney.id.au> Steve Simmons writes: > It probably isn't the best time to start this post but I was > wondering... An excellent time to raise this, in my opinion. Thank you for doing so. > Does this list have a code of conduct or a netiqeutte (sp?) > statement/requirement? This forum (both a Usenet newsgroup and a mailing list) is part of the Python community. So the ?Python Community Code of Conduct? applies. That was formed by the Python community and adopted by our gracious hosts, the Python Software Foundation, who provide this forum for our use. > The reason I ask is that it seems to me that if we (the current > membership of the list) could agree to a set of preferred/required > behaviours we would at least have a framework by which to measure > unwelcome posts. I welcome use of the above code of conduct to guide social norms in this community. > And, more importantly, to guide newcomers in understanding that we are > enthusiasts who choose to discuss Python and *voluntarily* help solve > problems with Python for the less experienced members. Amen. -- \ ?If sharing a thing in no way diminishes it, it is not rightly | `\ owned if it is not shared.? ?Saint Augustine | _o__) | Ben Finney From roy at panix.com Tue Jul 2 19:46:13 2013 From: roy at panix.com (Roy Smith) Date: Tue, 02 Jul 2013 19:46:13 -0400 Subject: Python list code of conduct References: Message-ID: In article , Ned Deily wrote: > If you find a bug in Python, don't send it to comp.lang.python; file > a bug report in the issue tracker. I'm not sure I agree with that one, at least not fully. It's certainly true that you shouldn't expect anybody to do anything about a bug unless you open an issue. On the other hand, I often find it useful to discuss things that I believe are bugs on c.l.p first. Sometimes people will explain to me that I'm just doing it wrong. Sometimes the discussion will end up with, "Yeah, that's a bug". In either case, it serves as a good initial filter for whether I should file a bug or not, and the discussion is often educational. From nad at acm.org Tue Jul 2 20:07:19 2013 From: nad at acm.org (Ned Deily) Date: Tue, 02 Jul 2013 17:07:19 -0700 Subject: Python list code of conduct References: Message-ID: In article , Roy Smith wrote: > In article , > Ned Deily wrote:> > > If you find a bug in Python, don't send it to comp.lang.python; file > > a bug report in the issue tracker. > I'm not sure I agree with that one, at least not fully. It's certainly > true that you shouldn't expect anybody to do anything about a bug unless > you open an issue. > > On the other hand, I often find it useful to discuss things that I > believe are bugs on c.l.p first. Sometimes people will explain to me > that I'm just doing it wrong. Sometimes the discussion will end up > with, "Yeah, that's a bug". In either case, it serves as a good initial > filter for whether I should file a bug or not, and the discussion is > often educational. For the record, those are not my words, rather a quote from the "charter" for this forum on http://www.python.org/community/lists/. I think the point of the quoted words is not to discourage discussion here but rather a reminder about the role of the issue tracker. As you say, don't expect anybody to do anything about it without an issue on the tracker. -- Ned Deily, nad at acm.org From cs at zip.com.au Tue Jul 2 20:10:10 2013 From: cs at zip.com.au (Cameron Simpson) Date: Wed, 3 Jul 2013 10:10:10 +1000 Subject: Python list code of conduct In-Reply-To: References: Message-ID: <20130703001010.GA17323@cskk.homeip.net> On 02Jul2013 19:46, Roy Smith wrote: | In article , | Ned Deily wrote: | | > If you find a bug in Python, don't send it to comp.lang.python; file | > a bug report in the issue tracker. | | I'm not sure I agree with that one, at least not fully. It's certainly | true that you shouldn't expect anybody to do anything about a bug unless | you open an issue. | | On the other hand, I often find it useful to discuss things that I | believe are bugs on c.l.p first. Sometimes people will explain to me | that I'm just doing it wrong. Sometimes the discussion will end up | with, "Yeah, that's a bug". In either case, it serves as a good initial | filter for whether I should file a bug or not, and the discussion is | often educational. +1 I've certinly been educated that way. -- Cameron Simpson Try not to get sucked into the vortex of hell, Billy! - MST3K, "Megalon vs. Godzilla" From steve+comp.lang.python at pearwood.info Tue Jul 2 20:52:46 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 03 Jul 2013 00:52:46 GMT Subject: Bug reports [was Re: Python list code of conduct] References: Message-ID: <51d375de$0$29999$c3e8da3$5496439d@news.astraweb.com> On Tue, 02 Jul 2013 19:46:13 -0400, Roy Smith wrote: > In article , > Ned Deily wrote: > >> If you find a bug in Python, don't send it to comp.lang.python; file a >> bug report in the issue tracker. > > I'm not sure I agree with that one, at least not fully. It's certainly > true that you shouldn't expect anybody to do anything about a bug unless > you open an issue. > > On the other hand, I often find it useful to discuss things that I > believe are bugs on c.l.p first. Sometimes people will explain to me > that I'm just doing it wrong. Sometimes the discussion will end up > with, "Yeah, that's a bug". In either case, it serves as a good initial > filter for whether I should file a bug or not, and the discussion is > often educational. Agreed strongly! It frustrates and amuses me when I see newbies, who sometimes don't know enough Python to tell the difference between (1, 2, 3) and [1, 2, 3], jump straight to the conclusion that anything that doesn't work the way they expect must be a bug in Python. If you are a beginner to a programming language, assume that anything that doesn't work the way you expect is a bug in YOUR code, or YOUR understanding, not in the language. 99.9999% of the time you will be correct. If you assume the opposite, you will nearly always just come across looking hopelessly naive at best, and at worst like an entitled, arrogant idiot. A popular language like Python has been around for about 20 years. It is in daily use by tens of thousands of people around the world. What are the chances that you, in your first week of using Python, just happened to stumble across a bug that *nobody else in the world* has noticed? So anyway, if you're going to make a fool of yourself by loudly proclaiming that your broken code is a bug in the language, at least do it here rather than waste the time of the people actually working on Python :-) -- Steven From joshua.landau.ws at gmail.com Tue Jul 2 21:12:12 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Wed, 3 Jul 2013 02:12:12 +0100 Subject: Bug reports [was Re: Python list code of conduct] In-Reply-To: <51d375de$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d375de$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 3 July 2013 01:52, Steven D'Aprano wrote: > If you are a beginner to a programming language, assume that anything > that doesn't work the way you expect is a bug in YOUR code, or YOUR > understanding, not in the language. Not just beginners. Out of the hundreds of times where I've gone "this *can't* make sense", I think only one -maybe two- was an actual bug in Python. From rosuav at gmail.com Wed Jul 3 04:02:33 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 3 Jul 2013 18:02:33 +1000 Subject: Bug reports [was Re: Python list code of conduct] In-Reply-To: <51d375de$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d375de$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Wed, Jul 3, 2013 at 10:52 AM, Steven D'Aprano wrote: > A popular language like Python has been around for about 20 years. It is > in daily use by tens of thousands of people around the world. What are > the chances that you, in your first week of using Python, just happened > to stumble across a bug that *nobody else in the world* has noticed? Of course, it's possible for there to be dark corners. But if you're working with those, you know it full well. The dark corners of Python might be in some of its more obscure modules, or maybe in IPv6 handling, or perhaps some weird platform that misbehaves, but they're definitely not in lists and tuples treated normally. ChrisA From roy at panix.com Wed Jul 3 10:03:24 2013 From: roy at panix.com (Roy Smith) Date: Wed, 03 Jul 2013 10:03:24 -0400 Subject: Bug reports [was Re: Python list code of conduct] References: <51d375de$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: In article , Chris Angelico wrote: > Of course, it's possible for there to be dark corners. But if you're > working with those, you know it full well. The dark corners of Python > might be in some of its more obscure modules, or maybe in IPv6 > handling, The sad thing about this statement is that IPv6 has been around for about as long as Python. From rosuav at gmail.com Wed Jul 3 10:15:36 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Jul 2013 00:15:36 +1000 Subject: Bug reports [was Re: Python list code of conduct] In-Reply-To: References: <51d375de$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, Jul 4, 2013 at 12:03 AM, Roy Smith wrote: > In article , > Chris Angelico wrote: > >> Of course, it's possible for there to be dark corners. But if you're >> working with those, you know it full well. The dark corners of Python >> might be in some of its more obscure modules, or maybe in IPv6 >> handling, > > The sad thing about this statement is that IPv6 has been around for > about as long as Python. Yeah, and yet so many ISPs simply don't support it (only one of the Australian ISPs I've worked with does - Internode). So heaps of code just doesn't get tested in an IPv6 environment. I don't know whether there's anything in Python that would fail in that way, but there might be; if someone starts writing a DNS server in Python and runs into difficulties with AAAA records, then it actually might be a library bug. But the concept still applies: Extraordinary claims require extraordinary evidence.[1] Before ranting loudly about how Python is buggy, make absolutely sure you're right, and show the evidence. ChrisA [1] http://www.catb.org/esr/faqs/smart-questions.html#idp5249360 From ben+python at benfinney.id.au Wed Jul 3 21:15:08 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 04 Jul 2013 11:15:08 +1000 Subject: IPv6 deployment by ISPs (was: Bug reports) References: <51d375de$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <7wa9m315yb.fsf_-_@benfinney.id.au> Chris Angelico writes: > Yeah, and yet so many ISPs simply don't support it [IPv6] (only one of > the Australian ISPs I've worked with does - Internode). Internode was the first in Australia, yes. Telstra supports IPv6, but only for enterprise/government customers. Wikipedia has a list of IPv6 deployment status by country . -- \ ?How wonderful that we have met with a paradox. Now we have | `\ some hope of making progress.? ?Niels Bohr | _o__) | Ben Finney From roy at panix.com Wed Jul 3 10:23:35 2013 From: roy at panix.com (Roy Smith) Date: Wed, 03 Jul 2013 10:23:35 -0400 Subject: Bug reports [was Re: Python list code of conduct] References: <51d375de$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: In article , Chris Angelico wrote: > On Thu, Jul 4, 2013 at 12:03 AM, Roy Smith wrote: > > In article , > > Chris Angelico wrote: > > > >> Of course, it's possible for there to be dark corners. But if you're > >> working with those, you know it full well. The dark corners of Python > >> might be in some of its more obscure modules, or maybe in IPv6 > >> handling, > > > > The sad thing about this statement is that IPv6 has been around for > > about as long as Python. > > Yeah, and yet so many ISPs simply don't support it (only one of the > Australian ISPs I've worked with does - Internode). My comment was not meant as a flame against Python, but as a flame against the whole frigging Internet industry for dragging their feet(*) on IPv6 rollout. (*) It is difficult to walk with any gait other than foot-dragging when you are suffering from a severe case of cranio-rectal inversion. From rosuav at gmail.com Wed Jul 3 10:52:16 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Jul 2013 00:52:16 +1000 Subject: Bug reports [was Re: Python list code of conduct] In-Reply-To: References: <51d375de$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, Jul 4, 2013 at 12:23 AM, Roy Smith wrote: > In article , > Chris Angelico wrote: > >> On Thu, Jul 4, 2013 at 12:03 AM, Roy Smith wrote: >> > In article , >> > Chris Angelico wrote: >> > >> >> Of course, it's possible for there to be dark corners. But if you're >> >> working with those, you know it full well. The dark corners of Python >> >> might be in some of its more obscure modules, or maybe in IPv6 >> >> handling, >> > >> > The sad thing about this statement is that IPv6 has been around for >> > about as long as Python. >> >> Yeah, and yet so many ISPs simply don't support it (only one of the >> Australian ISPs I've worked with does - Internode). > > My comment was not meant as a flame against Python, but as a flame > against the whole frigging Internet industry for dragging their feet(*) > on IPv6 rollout. > > (*) It is difficult to walk with any gait other than foot-dragging when > you are suffering from a severe case of cranio-rectal inversion. Right. My examples of dark corners were non-specific too; I haven't actually found any such corners in Python as I don't use it enough. It's high time the world moved to IPv6 wholesale. Start leaning on your ISPs to offer /64s (or /56s as Internode give) to all their customers. Sooner or later they'll have to make it a priority. IPv4 depletion was back in early 2011 - that's two years since IANA gave out the last blocks to the RIRs. Surely that means something? At work, we took a stand on this, for what it's worth. We rent dedicated servers from a number of hosting companies (giving us geographically and topologically separate servers), and will not work with anyone who doesn't give IPv6 as well as IPv4 addressing. But since we currently put less than $100/month into servers, that's not much of a stand :) ChrisA From invalid at invalid.invalid Wed Jul 3 16:11:15 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Wed, 3 Jul 2013 20:11:15 +0000 (UTC) Subject: Bug reports [was Re: Python list code of conduct] References: <51d375de$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2013-07-03, Roy Smith wrote: > In article , > Chris Angelico wrote: > >> Of course, it's possible for there to be dark corners. But if you're >> working with those, you know it full well. The dark corners of Python >> might be in some of its more obscure modules, or maybe in IPv6 >> handling, > > The sad thing about this statement is that IPv6 has been around for > about as long as Python. I've recently added IPv6 support to some embedded firmware. We had to add it because various purchasing people (mostly government) have non-negotiable feature checklists and "IPv6 support" is starting to show up on the checklists. However, so far, none of the people requiring IPv6 support actually have any plans to use IPv6 -- ever. I'm convinced we could put "IPv6 support" on the data sheets and continue shipping IPv4-only stuff for many years to come, and all our customers would be perfectly happy. -- Grant Edwards grant.b.edwards Yow! My NOSE is NUMB! at gmail.com From roy at panix.com Wed Jul 3 16:31:13 2013 From: roy at panix.com (Roy Smith) Date: Wed, 03 Jul 2013 16:31:13 -0400 Subject: Bug reports [was Re: Python list code of conduct] References: <51d375de$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: In article , Grant Edwards wrote: > On 2013-07-03, Roy Smith wrote: > > In article , > > Chris Angelico wrote: > > > >> Of course, it's possible for there to be dark corners. But if you're > >> working with those, you know it full well. The dark corners of Python > >> might be in some of its more obscure modules, or maybe in IPv6 > >> handling, > > > > The sad thing about this statement is that IPv6 has been around for > > about as long as Python. > > I've recently added IPv6 support to some embedded firmware. We had to > add it because various purchasing people (mostly government) have > non-negotiable feature checklists and "IPv6 support" is starting to > show up on the checklists. Yup. I did one of those. Hmmm, just found the powerpoint deck dated March 2006, so we probably started the project around September 2005. Same deal, we couldn't sell into government accounts unless we had IPv6. Have they actually used it? I doubt it. The really annoying thing about the project was I never got around to implementing rfc1924. > However, so far, none of the people requiring IPv6 support actually > have any plans to use IPv6 -- ever. What's known in the industry as "shelfware". From invalid at invalid.invalid Wed Jul 3 17:52:16 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Wed, 3 Jul 2013 21:52:16 +0000 (UTC) Subject: Bug reports [was Re: Python list code of conduct] References: <51d375de$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2013-07-03, Roy Smith wrote: > In article , > Grant Edwards wrote: > >> On 2013-07-03, Roy Smith wrote: >> > In article , >> > Chris Angelico wrote: >> > >> >> Of course, it's possible for there to be dark corners. But if you're >> >> working with those, you know it full well. The dark corners of Python >> >> might be in some of its more obscure modules, or maybe in IPv6 >> >> handling, >> > >> > The sad thing about this statement is that IPv6 has been around for >> > about as long as Python. >> >> I've recently added IPv6 support to some embedded firmware. We had to >> add it because various purchasing people (mostly government) have >> non-negotiable feature checklists and "IPv6 support" is starting to >> show up on the checklists. > > Yup. I did one of those. Hmmm, just found the powerpoint deck dated > March 2006, so we probably started the project around September 2005. > Same deal, we couldn't sell into government accounts unless we had IPv6. > Have they actually used it? I doubt it. > > The really annoying thing about the project was I never got around to > implementing rfc1924. Thankfully, the network library for our RTOS had support functions to handle rfc1924. Otherwise I would have had to do it myself because there's no way I could have made it through the project having to read and type full-length IPv6 addresses the whole time. >> However, so far, none of the people requiring IPv6 support actually >> have any plans to use IPv6 -- ever. > > What's known in the industry as "shelfware". Yep. 99.9% of the time, our products are used on small, isolated industrial networks that have little or no internet connectivity. The product line started out when a lot of customers weren't even using IPv4 yet. We still have customers who use our products with all IP support disabled (though these days it's always on networks that have other IP traffic). -- Grant Edwards grant.b.edwards Yow! On the road, ZIPPY at is a pinhead without a gmail.com purpose, but never without a POINT. From tjreedy at udel.edu Wed Jul 3 02:18:43 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 03 Jul 2013 02:18:43 -0400 Subject: Python list code of conduct In-Reply-To: References: Message-ID: On 7/2/2013 7:46 PM, Roy Smith wrote: > In article , > Ned Deily wrote: > >> If you find a bug in Python, don't send it to comp.lang.python; file >> a bug report in the issue tracker. I would revise this to "If you are have really found a bug in Python..." How does a newbie know? > I'm not sure I agree with that one, at least not fully. It's certainly > true that you shouldn't expect anybody to do anything about a bug unless > you open an issue. > > On the other hand, I often find it useful to discuss things that I > believe are bugs on c.l.p first. Sometimes people will explain to me > that I'm just doing it wrong. Sometimes the discussion will end up > with, "Yeah, that's a bug". usually followed by "File a tracker issue" or "I opened an tracker issue for this." (I have done that several times, though I sometimes prefer a person learn how to do it themselves.) > In either case, it serves as a good initial > filter for whether I should file a bug or not, and the discussion is > often educational. Ask here first. With a subject line that says 'I think ...' or "Problem with ...' Advantages: 1. At least half the bugs newbies report are not. The tracker does not need the extra clutter. 2. Filing a tracker issue sometimes creates a 'mental investment' in the mis-perception, which leads to resentment upon explanation. 3. There are lots of people here ready to help and answer questions. Any sensible question usually gets multiple responses, usually within a few hours or a day. (Invalid tracker reports may sit for days and get one short response.) 4. Explanations posted here benefit lots of people, rather than just 1. 5. A question posted here may elicit essential information, like which systems or which versions have the problem. 6. If you make an informed post to the tracker backed up by at least opinion, at least one tracker responder be in a better mode when responding. -- Terry Jan Reedy From roy at panix.com Wed Jul 3 10:10:17 2013 From: roy at panix.com (Roy Smith) Date: Wed, 03 Jul 2013 10:10:17 -0400 Subject: Python list code of conduct References: Message-ID: In article , Terry Reedy wrote: > 6. If you make an informed post to the tracker backed up by at least > opinion, at least one tracker responder be in a better mode when responding. What I generally do is summarize the problem in the tracker, but also include a link to the google groups archive of the mailing list thread. Then, if anybody wants to see the whole background discussion, it's easy to find. The most common outcome, for me at least, is that I end up opening a bug against the docs, because they didn't do a good job of explaining how something is supposed to work. From steve+comp.lang.python at pearwood.info Wed Jul 3 13:24:06 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 03 Jul 2013 17:24:06 GMT Subject: Python list code of conduct References: Message-ID: <51d45e35$0$29999$c3e8da3$5496439d@news.astraweb.com> On Wed, 03 Jul 2013 10:10:17 -0400, Roy Smith wrote: > In article , > Terry Reedy wrote: > >> 6. If you make an informed post to the tracker backed up by at least >> opinion, at least one tracker responder be in a better mode when >> responding. > > What I generally do is summarize the problem in the tracker, but also > include a link to the google groups archive of the mailing list thread. ^^^^^^^^^^^^^^^^^^^^^^ Please don't. Link to the actual mailing list thread instead: http://mail.python.org/mailman/listinfo/python-list This is the canonical archive for the list, and doesn't involve the atrocious Google Groups interface. -- Steven From ben+python at benfinney.id.au Tue Jul 2 20:39:35 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 03 Jul 2013 10:39:35 +1000 Subject: Python list code of conduct References: Message-ID: <7wzju4wk6w.fsf@benfinney.id.au> Dennis Lee Bieber writes: > So who would enforce any rules? Ideally, this community is healthy enough for us to enforce the code of conduct of our host, through social convention among us all. -- \ ?I took a course in speed waiting. Now I can wait an hour in | `\ only ten minutes.? ?Steven Wright | _o__) | Ben Finney From rustompmody at gmail.com Wed Jul 3 07:50:50 2013 From: rustompmody at gmail.com (rusi) Date: Wed, 3 Jul 2013 04:50:50 -0700 (PDT) Subject: Python list code of conduct In-Reply-To: References: Message-ID: <55c93426-5c25-4641-89c7-1396baeb6a67@googlegroups.com> On Wednesday, July 3, 2013 6:09:35 AM UTC+5:30, Ben Finney wrote: > Dennis Lee Bieber writes: > > > So who would enforce any rules? > > Ideally, this community is healthy enough for us to enforce the code of > conduct of our host, through social convention among us all. Thanks Ben for that. Lets not stymie Steve Simmons original suggestion for a CoC by making it into a formal CoC and then saying its impossible. There is a good deal of informal enforcement already. Consider eg: - Mark's footnote reminding about GG problems and their solutions - Steven's brusque 'cut-the-crap' type responses - Your own prompt STOPs to racist/sexist etc posts etc So we may take and welcome Steve Simmons' call to expand that list. I certainly welcome the suggestion. From square.steve at gmail.com Wed Jul 3 10:18:06 2013 From: square.steve at gmail.com (Steve Simmons) Date: Wed, 03 Jul 2013 15:18:06 +0100 Subject: Python list code of conduct In-Reply-To: <55c93426-5c25-4641-89c7-1396baeb6a67@googlegroups.com> References: <55c93426-5c25-4641-89c7-1396baeb6a67@googlegroups.com> Message-ID: <51D4329E.3010500@gmail.com> On 03/07/2013 12:50, rusi wrote: > On Wednesday, July 3, 2013 6:09:35 AM UTC+5:30, Ben Finney wrote: >> Dennis Lee Bieber writes: >> >>> So who would enforce any rules? >> Ideally, this community is healthy enough for us to enforce the code of >> conduct of our host, through social convention among us all. > Thanks Ben for that. > Lets not stymie Steve Simmons original suggestion for a CoC by making it into a formal CoC and then saying its impossible. > > There is a good deal of informal enforcement already. Consider eg: > - Mark's footnote reminding about GG problems and their solutions > - Steven's brusque 'cut-the-crap' type responses > - Your own prompt STOPs to racist/sexist etc posts > etc > > So we may take and welcome Steve Simmons' call to expand that list. > I certainly welcome the suggestion. Thanks Rusi - and everyone else that provided feedback. I understand (and had already anticipated) that this type of group can't and probably shouldn't be moderated in the more draconian sense of the word but there's no reason why we couldn't create some behavioural and technical guidelines. I had it in mind to move to a few (?!) paragraphs that read well when considered as a single document but that were also suitable to cut'n'paste into a response - much as Mark puts his GG warning in every mail. The aim would be to put a considered response in a first reply that tells the OP 'this is what you need to do to get a sensible answer'. We have this for GG & code fragments and I see no reason why we can't have it for bad language; bad attitude; unreasonable expectations; thread high-jacking; top posting; and all the other sins we expect Noobs (and some long standing members!) to avoid. I guess I should have said this up front but let me say it now: I completely 'get' that this won't fix the Trolls or the dyed-in-the-wool pedants or recidivist high-jackers but it may put some buffering between: Newb: "I want you to do my homework and I have to give it in tomorrow morning" and Pythonista: "you're a dick" If there's enough support for the general idea, I'm happy to make a decent sized contribution but if there isn't - Nuff said. Steve From steve+comp.lang.python at pearwood.info Wed Jul 3 02:18:14 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 03 Jul 2013 06:18:14 GMT Subject: Python list code of conduct References: Message-ID: <51d3c225$0$11124$c3e8da3@news.astraweb.com> On Wed, 03 Jul 2013 08:39:59 +1000, Ben Finney wrote: > Steve Simmons writes: >> Does this list have a code of conduct or a netiqeutte (sp?) >> statement/requirement? > > This forum (both a Usenet newsgroup and a mailing list) is part of the > Python community. So the ?Python Community Code of Conduct? applies. > > It certainly does not apply. Even putting aside the practical impossibility of enforcing it on an unmoderated mailing list/news group, the Code of Conduct needs to be adopted by each community group that chooses to do so. The Python Software Foundation (PSF) has approved the CoC, but it is only *suggested* that Python-related communities adopt it, it is not mandatory. Nor could it be mandatory -- the PSF has no power to tell arbitrary Python groups how to run their community. (Although in principle it could choose to withhold funding from those that do not.) The PSF blog post introducing this makes it clear that the PSF *encourages* community groups to adopt this CoC, but that is all: [quote] The PSF supports and advocates for the use of the CoC throughout the community, but without adoption by specific areas, the CoC is merely a document that the Foundation is supportive of. The way it?s useful is that an area of the community can adopt the CoC and use it as a guideline for participation. IT COULD BE adopted by mailing lists, IRC channels, the bug tracker, user groups, sprints, and more. For example, a mailing list COULD SAY that their membership should adhere to the CoC. [emphasis added] [end quote] http://pyfound.blogspot.com.au/2013/06/announcing-code-of-conduct-for-use-by.html To my knowledge, only two groups have adopted this so far: the PSF itself have adopted it for their internal (and private) mailing lists), and the python-ideas mailing list, whose two moderators adopted it without consultation with the mailing list subscribers, not that I'm bitter or anything. > That was formed by the Python community and adopted by our gracious > hosts, the Python Software Foundation, who provide this forum for our > use. The Code of Conduct was not formed by the Python community. It was formed by a small but influential subset of the Python community, the PSF. The greater Python community includes large numbers of people who are not members of the PSF, are not on this mailing list, and indeed may not be on any mailing list at all. It is absurd to think that the PSF can unilaterally decide how (say) StackOverflow users are allowed to behave when they ask Python questions. Now, if we were talking about the Python Secret Underground, which emphatically does not exist, then -- Steven From nhodgson at iinet.net.au Wed Jul 3 05:50:40 2013 From: nhodgson at iinet.net.au (Neil Hodgson) Date: Wed, 03 Jul 2013 19:50:40 +1000 Subject: Python list code of conduct In-Reply-To: References: Message-ID: Dennis Lee Bieber: > So who would enforce any rules? I doubt it could be ported to a new (if > approval could even be obtained) comp.lang.python.mod(erated) so nothing > can be enforced on the comp.lang.python side; and what would you do with > Google Groups? The current news group charter doesn't really have any rules. While it may be possible to recharter an existing news group, it would likely be simpler to just create a new one. CHARTER Comp.lang.python is an unmoderated newsgroup which will serve as a forum for discussing the Python computer language. The group will serve both those who just program in Python and those who work on developing the language. Topics that may be discussed include: - announcements of new versions of the language and applications written in Python. - discussion on the internals of the Python language. - general information about the language. - discussion on programming in Python. http://www.python.org/search/hypermail/python-1994q1/0377.html Neil From steve+comp.lang.python at pearwood.info Tue Jul 2 20:42:21 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 03 Jul 2013 00:42:21 GMT Subject: Python list code of conduct References: Message-ID: <51d3736d$0$29999$c3e8da3$5496439d@news.astraweb.com> On Tue, 02 Jul 2013 16:51:38 +0100, Steve Simmons wrote: > Erm, > > It probably isn't the best time to start this post but I was > wondering... > > Does this list have a code of conduct or a netiqeutte (sp?) > statement/requirement? > > If not, should it? I *knew* this would be raised. You are right to do so, of course, but I really, really, REALLY wasn't looking forward to it. No, this list does not have a Code of Conduct, apart from the informal conventions of netiquette (most of which can be summed up with "don't be a dick") that most public forums populated by adults[1] operate under. In practical terms, it *cannot* have a Code of Conduct enforced, because it is an unmoderated list. Anyone can join. Anyone can post. It exists as both email and news. Do we need one? No, I don't think so. I'm philosophically opposed to formalising the idea of "polite behaviour" in a Code of Conduct for various reasons, but even putting that aside I have been here for at least 7 years, and I've seen Xah Lee come and go (and come, and go...) and Ranting Rick in full-on troll mode, and spammers, and endless heated discussions about fine semantics of some aspect of Python's object model. People come and go, but the community goes on, and we have weathered all of those and we will weather this damned argument over whether or not it is appropriate to flame help-vampires or not. [1] I refer to state of maturity, not chronological age. Some people are adult at ten, others can live to ninety and never be worthy of the term. -- Steven From steve+comp.lang.python at pearwood.info Tue Jul 2 20:19:00 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 03 Jul 2013 00:19:00 GMT Subject: OSError [Errno 26] ?!?! References: Message-ID: <51d36df4$0$9505$c3e8da3$5496439d@news.astraweb.com> On Tue, 02 Jul 2013 14:58:18 +0100, Robert Kern wrote: > On 2013-07-02 14:00, Chris ?Kwpolska? Warrick wrote: >> On Tue, Jul 2, 2013 at 2:39 PM, ????? wrote: > >>> Please suggest an editor that has built in rsync ability so to >>> immediately upload my cgi-scripts when i hit save in the text editor. >> >> CGI? Is this 2000? Nobody uses that wording these days. > > He is indeed using actual, bona fide CGI scripts. It's not just an > antiquated wording for "web app". CGI didn't stop working just because more powerful, or better, alternatives now exist. -- Steven From python.list at tim.thechases.com Tue Jul 2 20:57:04 2013 From: python.list at tim.thechases.com (Tim Chase) Date: Tue, 2 Jul 2013 19:57:04 -0500 Subject: Persistence of CGI (was: OSError [Errno 26] ?!?!) In-Reply-To: <51d36df4$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <51d36df4$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: <20130702195704.02be4dd5@bigbox.christie.dr> On 2013-07-03 00:19, Steven D'Aprano wrote: > CGI didn't stop working just because more powerful, or better, > alternatives now exist. And for most exceptionally cheap hosting services, your choices are usually limited to PHP, static HTML (possibly server-side includes if you're lucky), or CGI. So CGI is often the easiest way to get dynamic websites written in Python (or Perl) onto cheap web providers. And for the small sites that use such hosting, the overhead of CGI doesn't generally cause much issue. -tkc From subhabangalore at gmail.com Tue Jul 2 13:43:03 2013 From: subhabangalore at gmail.com (subhabangalore at gmail.com) Date: Tue, 2 Jul 2013 10:43:03 -0700 (PDT) Subject: HTML Parser Message-ID: Dear Group, I was looking for a good tutorial for a "HTML Parser". My intention was to extract tables from web pages or information from tables in web pages. I tried to make a search, I got HTMLParser, BeautifulSoup, etc. HTMLParser works fine for me, but I am looking for a good tutorial to learn it nicely. I could not use BeautifulSoup as I did not find an .exe file. I am using Python 2.7 on Windows 7 SP1 (64 bit). I am looking for a good tutorial for HTMLParser or any similar parser which have an .exe file for my environment and a good tutorial. If anyone of the learned members can kindly suggest. Thanking You in Advance, Regards, Subhabrata. From neilc at norwich.edu Tue Jul 2 13:57:22 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 2 Jul 2013 17:57:22 GMT Subject: HTML Parser References: Message-ID: On 2013-07-02, subhabangalore at gmail.com wrote: > Dear Group, > > I was looking for a good tutorial for a "HTML Parser". My > intention was to extract tables from web pages or information > from tables in web pages. > > I tried to make a search, I got HTMLParser, BeautifulSoup, etc. > HTMLParser works fine for me, but I am looking for a good > tutorial to learn it nicely. Take a read of the topic "Parsing, creating, and Manipulating HTML Documents" from chapter five of Text Processing in Python. http://gnosis.cx/TPiP/chap5.txt -- Neil Cerutti From steve+comp.lang.python at pearwood.info Tue Jul 2 19:53:25 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 02 Jul 2013 23:53:25 GMT Subject: HTML Parser References: Message-ID: <51d367f5$0$29999$c3e8da3$5496439d@news.astraweb.com> On Tue, 02 Jul 2013 10:43:03 -0700, subhabangalore wrote: > I could not use BeautifulSoup as I did not find an .exe file. I believe that BeautifulSoup is a pure-Python module, and so does not have a .exe file. However, it does have good tutorials: https://duckduckgo.com/html/?q=beautifulsoup+tutorial > I am looking for a good tutorial for HTMLParser or any similar parser > which have an .exe file for my environment and a good tutorial. Why do you care about a .exe file? Most Python libraries are .py files. -- Steven From joshua.landau.ws at gmail.com Tue Jul 2 20:13:50 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Wed, 3 Jul 2013 01:13:50 +0100 Subject: HTML Parser In-Reply-To: References: Message-ID: On 2 July 2013 18:43, wrote: > I could not use BeautifulSoup as I did not find an .exe file. Were you perhaps looking for a .exe file to install BeautifulSoup? It's quite plausible that a windows user like you might be dazzled at the idea of a .tar.gz. I suggest just using "pip install beautifulsoup4" at a command prompt. See http://stackoverflow.com/questions/12228102/how-to-install-beautiful-soup-4-with-python-2-7-on-windows for explanations -- there are links for things you need to know. But basically, use BeautifulSoup. It does what you need. From sas429s at gmail.com Tue Jul 2 14:45:53 2013 From: sas429s at gmail.com (sas429s at gmail.com) Date: Tue, 2 Jul 2013 11:45:53 -0700 (PDT) Subject: Parsing Text file Message-ID: <08ae2828-1532-47b6-a9cb-208549189467@googlegroups.com> I have a text file like this: Sometext Somemore Somemore maskit Sometext Somemore Somemore Somemore maskit Sometext Somemore maskit I want to search for the string maskit in this file and also need to print Sometext above it..SOmetext location can vary as you can see above. In the first instance it is 3 lines above mask it, in the second instance it is 4 lines above it and so on.. Please help how to do it? From neilc at norwich.edu Tue Jul 2 15:24:26 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 2 Jul 2013 19:24:26 GMT Subject: Parsing Text file References: <08ae2828-1532-47b6-a9cb-208549189467@googlegroups.com> Message-ID: On 2013-07-02, sas429s at gmail.com wrote: > I have a text file like this: > > Sometext > Somemore > Somemore > maskit > > Sometext > Somemore > Somemore > Somemore > maskit > > Sometext > Somemore > maskit > > I want to search for the string maskit in this file and also > need to print Sometext above it..SOmetext location can vary as > you can see above. > > In the first instance it is 3 lines above mask it, in the > second instance it is 4 lines above it and so on.. > > Please help how to do it? How can you tell the difference between Sometext and Somemore? -- Neil Cerutti From sas429s at gmail.com Tue Jul 2 15:30:07 2013 From: sas429s at gmail.com (sas429s at gmail.com) Date: Tue, 2 Jul 2013 12:30:07 -0700 (PDT) Subject: Parsing Text file In-Reply-To: References: <08ae2828-1532-47b6-a9cb-208549189467@googlegroups.com> Message-ID: <8ea32ea7-2cee-4e61-8cbd-066721d88d4a@googlegroups.com> Somemore can be anything for instance: Sometext mail maskit Sometext rupee dollar maskit and so on.. Is there a way I can achieve this? On Tuesday, July 2, 2013 2:24:26 PM UTC-5, Neil Cerutti wrote: > On 2013-07-02, sas429s at gmail.com wrote: > > > I have a text file like this: > > > > > > Sometext > > > Somemore > > > Somemore > > > maskit > > > > > > Sometext > > > Somemore > > > Somemore > > > Somemore > > > maskit > > > > > > Sometext > > > Somemore > > > maskit > > > > > > I want to search for the string maskit in this file and also > > > need to print Sometext above it..SOmetext location can vary as > > > you can see above. > > > > > > In the first instance it is 3 lines above mask it, in the > > > second instance it is 4 lines above it and so on.. > > > > > > Please help how to do it? > > > > How can you tell the difference between Sometext and Somemore? > > > > -- > > Neil Cerutti From toby at tobiah.org Tue Jul 2 15:50:23 2013 From: toby at tobiah.org (Tobiah) Date: Tue, 02 Jul 2013 12:50:23 -0700 Subject: Parsing Text file In-Reply-To: <8ea32ea7-2cee-4e61-8cbd-066721d88d4a@googlegroups.com> References: <08ae2828-1532-47b6-a9cb-208549189467@googlegroups.com> <8ea32ea7-2cee-4e61-8cbd-066721d88d4a@googlegroups.com> Message-ID: On 07/02/2013 12:30 PM, sas429s at gmail.com wrote: > Somemore can be anything for instance: > > Sometext > mail > maskit > > Sometext > rupee > dollar > maskit > > and so on.. > > Is there a way I can achieve this? How do we know whether we have Sometext? If it's really just a literal 'Sometext', then just print that when you hit maskit. Otherwise: for line in open('file.txt').readlines(): if is_sometext(line): memory = line if line == 'maskit': print memory From neilc at norwich.edu Tue Jul 2 16:12:55 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 2 Jul 2013 20:12:55 GMT Subject: Parsing Text file References: <08ae2828-1532-47b6-a9cb-208549189467@googlegroups.com> <8ea32ea7-2cee-4e61-8cbd-066721d88d4a@googlegroups.com> Message-ID: On 2013-07-02, Tobiah wrote: > On 07/02/2013 12:30 PM, sas429s at gmail.com wrote: >> Somemore can be anything for instance: >> >> Sometext >> mail >> maskit >> >> Sometext >> rupee >> dollar >> maskit >> >> and so on.. >> >> Is there a way I can achieve this? > > How do we know whether we have Sometext? > If it's really just a literal 'Sometext', then > just print that when you hit maskit. > > Otherwise: > > > for line in open('file.txt').readlines(): > > if is_sometext(line): > memory = line > > if line == 'maskit': > print memory Tobiah's solution fits what little we can make of your problem. My feeling is that you've simplified your question a little too much in hopes that it would help us provide a better solution. Can you provide more context? -- Neil Cerutti From sas429s at gmail.com Tue Jul 2 16:28:33 2013 From: sas429s at gmail.com (sas429s at gmail.com) Date: Tue, 2 Jul 2013 13:28:33 -0700 (PDT) Subject: Parsing Text file In-Reply-To: References: <08ae2828-1532-47b6-a9cb-208549189467@googlegroups.com> <8ea32ea7-2cee-4e61-8cbd-066721d88d4a@googlegroups.com> Message-ID: <7e82becd-77c1-4800-8f4e-7624b19de82b@googlegroups.com> Ok here is a snippet of the text file I have: config/meal/governor_mode_config.h #define GOVERNOR_MODE_TASK_RATE SSS_TID_0015MSEC #define GOVERNOR_MODE_WORK_MODE_MASK (CEAL_MODE_WORK_MASK_GEAR| \ CEAL_MODE_WORK_MASK_PARK_BRAKE | \ CEAL_MODE_WORK_MASK_VEHICLE_SPEED) #define GOVERNOR_MODE_IDLE_CHECK FALSE #define GOVERNOR_MODE_SPD_THRES 50 #define GOVERNOR_MODE_SPDDES_THRES 10 config/meal/components/source/kso_aic_core_config.h #define CEAL_KSO_AIC_CORE_TASK_RATE SSS_TID_0120MSEC #define CEAL_KSO_AIC_LOAD_FAC_AVG_TIME 300 #define CEAL_KSO_AIC_LOAD_FAC_HYST_TIME 30 #define CEAL_KSO_AIC_TEMP_DPF_INSTALLED TRUE #define CEAL_KSO_AIC_TEMP_DPF_ENABLE 450 #define CEAL_KSO_AIC_TEMP_DPF_HYST 25 #define CEAL_KSO_AIC_DPF_ROC_TIME 10 #define CEAL_KSO_AIC_TEMP_EXHAUST_INSTALLED FALSE #define CEAL_KSO_AIC_TEMP_EXHAUST_ENABLE 275 #define CEAL_KSO_AIC_TEMP_EXHAUST_HYST 25 #define CEAL_KSO_AIC_EXHAUST_ROC_TIME 10 #define CEAL_KSO_AIC_WORK_MODE_MASK (CEAL_MODE_WORK_MASK_GEAR | \ CEAL_MODE_WORK_MASK_PARK_BRAKE | \ CEAL_MODE_WORK_MASK_VEHICLE_SPEED) #define CEAL_KSO_AIC_OV_TIME 15 Here I am looking for the line that contains: "WORK_MODE_MASK", I want to print that line as well as the file name above it: config/meal/governor_mode_config.h or config/meal/components/source/ceal_PackD_kso_aic_core_config.h. SO the output should be something like this: config/meal/governor_mode_config.h #define GOVERNOR_MODE_WORK_MODE_MASK (CEAL_MODE_WORK_MASK_GEAR| \ CEAL_MODE_WORK_MASK_PARK_BRAKE | \ CEAL_MODE_WORK_MASK_VEHICLE_SPEED) config/meal/components/source/kso_aic_core_config.h #define CEAL_KSO_AIC_WORK_MODE_MASK (CEAL_MODE_WORK_MASK_GEAR | \ CEAL_MODE_WORK_MASK_PARK_BRAKE | \ CEAL_MODE_WORK_MASK_VEHICLE_SPEED) I hope this helps.. Thanks for your help On Tuesday, July 2, 2013 3:12:55 PM UTC-5, Neil Cerutti wrote: > On 2013-07-02, Tobiah wrote: > > > On 07/02/2013 12:30 PM, sas429s at gmail.com wrote: > > >> Somemore can be anything for instance: > > >> > > >> Sometext > > >> mail > > >> maskit > > >> > > >> Sometext > > >> rupee > > >> dollar > > >> maskit > > >> > > >> and so on.. > > >> > > >> Is there a way I can achieve this? > > > > > > How do we know whether we have Sometext? > > > If it's really just a literal 'Sometext', then > > > just print that when you hit maskit. > > > > > > Otherwise: > > > > > > > > > for line in open('file.txt').readlines(): > > > > > > if is_sometext(line): > > > memory = line > > > > > > if line == 'maskit': > > > print memory > > > > Tobiah's solution fits what little we can make of your problem. > > > > My feeling is that you've simplified your question a little too > > much in hopes that it would help us provide a better solution. > > Can you provide more context? > > > > -- > > Neil Cerutti From joshua.landau.ws at gmail.com Tue Jul 2 16:56:55 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Tue, 2 Jul 2013 21:56:55 +0100 Subject: Parsing Text file In-Reply-To: <7e82becd-77c1-4800-8f4e-7624b19de82b@googlegroups.com> References: <08ae2828-1532-47b6-a9cb-208549189467@googlegroups.com> <8ea32ea7-2cee-4e61-8cbd-066721d88d4a@googlegroups.com> <7e82becd-77c1-4800-8f4e-7624b19de82b@googlegroups.com> Message-ID: On 2 July 2013 21:28, wrote: > Here I am looking for the line that contains: "WORK_MODE_MASK", I want to print that line as well as the file name above it: config/meal/governor_mode_config.h > or config/meal/components/source/ceal_PackD_kso_aic_core_config.h. > > SO the output should be something like this: > config/meal/governor_mode_config.h > > #define GOVERNOR_MODE_WORK_MODE_MASK (CEAL_MODE_WORK_MASK_GEAR| \ > CEAL_MODE_WORK_MASK_PARK_BRAKE | \ > CEAL_MODE_WORK_MASK_VEHICLE_SPEED) > > config/meal/components/source/kso_aic_core_config.h > #define CEAL_KSO_AIC_WORK_MODE_MASK (CEAL_MODE_WORK_MASK_GEAR | \ > CEAL_MODE_WORK_MASK_PARK_BRAKE | \ > CEAL_MODE_WORK_MASK_VEHICLE_SPEED) (Please don't top-post.) filename = None with open("tmp.txt") as file: nonblanklines = (line for line in file if line) for line in nonblanklines: if line.lstrip().startswith("#define"): defn, name, *other = line.split() if name.endswith("WORK_MODE_MASK"): print(filename, line, sep="") else: filename = line Basically, you loop through remembering what lines you need, match a little bit and ignore blank lines. If this isn't a solid specification, you'll 'ave to tell me more about the edge-cases. You said that > #define CEAL_KSO_AIC_WORK_MODE_MASK (CEAL_MODE_WORK_MASK_GEAR | \ > CEAL_MODE_WORK_MASK_PARK_BRAKE | \ > CEAL_MODE_WORK_MASK_VEHICLE_SPEED) was one line. If it is not, I suggest doing a pre-process to "wrap" lines with trailing "\"s before running the algorithm: def wrapped(lines): wrap = "" for line in lines: if line.rstrip().endswith("\\"): wrap += line else: yield wrap + line wrap = "" ... nonblanklines = (line for line in wrapped(file) if line) ... This doesn't handle all wrapped lines properly, as it leaves the "\" in so may interfere with matching. That's easily fixable, and there are many other ways to do this. What did you try? From denismfmcmahon at gmail.com Tue Jul 2 20:55:22 2013 From: denismfmcmahon at gmail.com (Denis McMahon) Date: Wed, 3 Jul 2013 00:55:22 +0000 (UTC) Subject: Parsing Text file References: <08ae2828-1532-47b6-a9cb-208549189467@googlegroups.com> <8ea32ea7-2cee-4e61-8cbd-066721d88d4a@googlegroups.com> <7e82becd-77c1-4800-8f4e-7624b19de82b@googlegroups.com> Message-ID: On Tue, 02 Jul 2013 13:28:33 -0700, sas429s wrote: > Ok here is a snippet of the text file I have: > I hope this helps.. > ..... > Thanks for your help ok ... so you need to figure out how best to distinguish the filename, then loop through the file, remember each filename as you find it, and when you find lines containing your target text, print the current value of filename and the target text line. filenames might be distinguished by one or more of the following: They always start in column 0 and nothing else starts in column 0 They never contain spaces and all other lines contain spaces or are blank They always contain at least one / characters They always terminate with a . followed by one or more characters All the characters in them are lower case Then loop through the file in something like the following manner: open input file; open output file; for each line in input file: { if line is a filename: { thisfile = line; } elif line matches search term: { print thisfile in output file; print line in output file; } } close input file; close output file; (Note this is an algorithm written in a sort of pythonic manner, rather than actual python code - also because some newsreaders may break indenting etc, I've used ; as line terminators and {} to group blocks) -- Denis McMahon, denismfmcmahon at gmail.com From joshua.landau.ws at gmail.com Tue Jul 2 16:28:26 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Tue, 2 Jul 2013 21:28:26 +0100 Subject: Parsing Text file In-Reply-To: References: <08ae2828-1532-47b6-a9cb-208549189467@googlegroups.com> <8ea32ea7-2cee-4e61-8cbd-066721d88d4a@googlegroups.com> Message-ID: On 2 July 2013 20:50, Tobiah wrote: > How do we know whether we have Sometext? > If it's really just a literal 'Sometext', then > just print that when you hit maskit. > > Otherwise: > > > for line in open('file.txt').readlines(): > > if is_sometext(line): > memory = line > > if line == 'maskit': > print memory My understanding of the question follows more like: # Python 3, UNTESTED memory = [] for line in open('file.txt').readlines(): if line == 'maskit': print(*memory, sep="") elif line: memory.append(line) else: memory = [] From suryak at ieee.org Tue Jul 2 17:43:51 2013 From: suryak at ieee.org (Surya Kasturi) Date: Tue, 2 Jul 2013 23:43:51 +0200 Subject: how to calculate reputation Message-ID: Hi all, this seems to be quite stupid question but I am "confused".. We set the initial value to 0, +1 for up-vote and -1 for down-vote! nice. I have a list of bool values True, False (True for up vote, False for down-vote).. submitted by users. [True, False, False, True....] Now to calculate the total reputation should I take True = +1, False=0 [or] True = +1, False=-1 ?? for adding all. I am missing something here.. and that's clear.. anyone please help me on it? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From toby at tobiah.org Tue Jul 2 17:59:38 2013 From: toby at tobiah.org (Tobiah) Date: Tue, 02 Jul 2013 14:59:38 -0700 Subject: how to calculate reputation In-Reply-To: References: Message-ID: On 07/02/2013 02:43 PM, Surya Kasturi wrote: > Hi all, this seems to be quite stupid question but I am "confused".. > We set the initial value to 0, +1 for up-vote and -1 for down-vote! nice. > > I have a list of bool values True, False (True for up vote, False for down-vote).. submitted by users. > > [True, False, False, True....] > > Now to calculate the total reputation > > should I take True = +1, False=0 [or] True = +1, False=-1 ?? for adding all. > > I am missing something here.. and that's clear.. anyone please help me on it? > > Thanks > > > for vote in bool_votes: reputation += 2 * vote - 1 Tobiah From suryak at ieee.org Tue Jul 2 18:19:57 2013 From: suryak at ieee.org (Surya Kasturi) Date: Wed, 3 Jul 2013 00:19:57 +0200 Subject: how to calculate reputation In-Reply-To: References: Message-ID: On Tue, Jul 2, 2013 at 11:59 PM, Tobiah wrote: > On 07/02/2013 02:43 PM, Surya Kasturi wrote: > >> Hi all, this seems to be quite stupid question but I am "confused".. >> We set the initial value to 0, +1 for up-vote and -1 for down-vote! nice. >> >> I have a list of bool values True, False (True for up vote, False for >> down-vote).. submitted by users. >> >> [True, False, False, True....] >> >> Now to calculate the total reputation >> >> should I take True = +1, False=0 [or] True = +1, False=-1 ?? for adding >> all. >> >> I am missing something here.. and that's clear.. anyone please help me on >> it? >> >> Thanks >> >> >> >> > for vote in bool_votes: > > reputation += 2 * vote - 1 > > > Tobiah > -- > http://mail.python.org/**mailman/listinfo/python-list > I think I didnt explain it clearly.. let me make it clear.. 1. The database we are using is having BooleanField for it!! so, cant do anything 2. I am not looking for sorting algorithms .. just simple math :) It sounds crazy but let me describe my confusion lets have 3 users with [null, null, null] reputation = 0 [T, - - ] rept = 1 [T T T] rept = 3 [T T F] rept = 1 (its jumping from 3 to 1 -->but generally, we observe only decrease in 1 right?) [T T F] rept = 3 (its jumping from 1 to 3) These jumpings are common? or my logic is going any wrong? -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshua.landau.ws at gmail.com Tue Jul 2 18:27:02 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Tue, 2 Jul 2013 23:27:02 +0100 Subject: how to calculate reputation In-Reply-To: References: Message-ID: On 2 July 2013 23:19, Surya Kasturi wrote: > > I think I didnt explain it clearly.. let me make it clear.. Yeah... I don't get it. From ian.g.kelly at gmail.com Tue Jul 2 18:29:38 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Tue, 2 Jul 2013 16:29:38 -0600 Subject: how to calculate reputation In-Reply-To: References: Message-ID: On Tue, Jul 2, 2013 at 4:19 PM, Surya Kasturi wrote: > I think I didnt explain it clearly.. let me make it clear.. > > 1. The database we are using is having BooleanField for it!! so, cant do > anything > 2. I am not looking for sorting algorithms .. just simple math :) It sounds > crazy but let me describe my confusion Nobody has suggested a *sorting* algorithm. > lets have 3 users with > > [null, null, null] > reputation = 0 > > [T, - - ] > rept = 1 > > [T T T] > rept = 3 > > [T T F] > rept = 1 (its jumping from 3 to 1 -->but generally, we observe only decrease > in 1 right?) I'm with you so far. You see the reputation drop by 2 here because you have both removed an up-vote and added a down-vote. Each of these things individually will cause the sum to drop by 1. > [T T F] > rept = 3 (its jumping from 1 to 3) > > These jumpings are common? or my logic is going any wrong? This is the same scenario as the previous one, so I don't understand why you identify the reputation sum as 3 here. From steve+comp.lang.python at pearwood.info Tue Jul 2 20:18:26 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 03 Jul 2013 00:18:26 GMT Subject: how to calculate reputation References: Message-ID: <51d36dd2$0$29999$c3e8da3$5496439d@news.astraweb.com> On Tue, 02 Jul 2013 23:43:51 +0200, Surya Kasturi wrote: > Hi all, this seems to be quite stupid question but I am "confused".. We > set the initial value to 0, +1 for up-vote and -1 for down-vote! nice. > > I have a list of bool values True, False (True for up vote, False for > down-vote).. submitted by users. > > [True, False, False, True....] > > Now to calculate the total reputation > > should I take True = +1, False=0 [or] True = +1, False=-1 ?? for adding > all. You can work this out for yourself by doing a couple of tests. Suppose you have somebody who gets one Upvote and five Downvotes: [True, False, False, False, False, False] What reputation would you expect them to have? We can't answer that, only you can answer that. With True=+1 and False=0, the sum is +1. With True=+1 and False=-1, the sum is -4. Upvotes and downvotes don't have to be weighted the same: With True=+5 and False=-1, the sum is 0. With True=+1 and False=-5, the sum is -24. *You* have to decide how you want the reputation system to work, then program it to work that way. For a real web app, you almost certainly do not want something this simple. Perhaps new accounts with low reputation themselves should be weighted less than old, established accounts with high reputation. There is no standard for counting reputation, and so every web site that does something like this does it differently, and most get it wrong, including some really big, well-known sites like Amazon. It's very easy to come up with lousy algorithms for calculating reputation, much harder to come up with good algorithms. I second Joshua Landau's recommendation that you read: http://www.evanmiller.org/how-not-to-sort-by-average-rating.html and I bring to your attention that this doesn't necessarily have anything to do with *sorting*. The Ruby function given returns a number between 0 and 1. You don't have to sort on that number, you can use that as your reputation. -- Steven From rosuav at gmail.com Tue Jul 2 20:26:01 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 3 Jul 2013 10:26:01 +1000 Subject: how to calculate reputation In-Reply-To: <51d36dd2$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d36dd2$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Wed, Jul 3, 2013 at 10:18 AM, Steven D'Aprano wrote: > It's very easy to come up with lousy algorithms for calculating > reputation, much harder to come up with good algorithms. Yes. Reminder: Don't just average your users' ratings. http://xkcd.com/937/ In fact, mere upvotes and downvotes mightn't be that useful regardless of algorithm, depending on what you're doing. ChrisA From ian.g.kelly at gmail.com Tue Jul 2 17:59:21 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Tue, 2 Jul 2013 15:59:21 -0600 Subject: how to calculate reputation In-Reply-To: References: Message-ID: On Tue, Jul 2, 2013 at 3:43 PM, Surya Kasturi wrote: > Hi all, this seems to be quite stupid question but I am "confused".. > We set the initial value to 0, +1 for up-vote and -1 for down-vote! nice. > > I have a list of bool values True, False (True for up vote, False for > down-vote).. submitted by users. > > [True, False, False, True....] > > Now to calculate the total reputation > > should I take True = +1, False=0 [or] True = +1, False=-1 ?? for adding > all. You're using False to represent a down-vote and you want down-votes to count as -1, so you would count a False value as -1. Now why don't you just use 1 and -1 to represent the votes instead of True and False? Then you won't need to worry about this conversion step. From joshua.landau.ws at gmail.com Tue Jul 2 18:00:41 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Tue, 2 Jul 2013 23:00:41 +0100 Subject: how to calculate reputation In-Reply-To: References: Message-ID: On 2 July 2013 22:43, Surya Kasturi wrote: > Hi all, this seems to be quite stupid question but I am "confused".. > We set the initial value to 0, +1 for up-vote and -1 for down-vote! nice. > > I have a list of bool values True, False (True for up vote, False for > down-vote).. submitted by users. > > [True, False, False, True....] > > Now to calculate the total reputation > > should I take True = +1, False=0 [or] True = +1, False=-1 ?? for adding > all. > > I am missing something here.. and that's clear.. anyone please help me on > it? It depends what you want to do. I suggest http://www.evanmiller.org/how-not-to-sort-by-average-rating.html. This aint so simple, but it's the correct way to sort by "approval". In fact, it's bloody confusing. To assume that only the "sum" approval counts, I'd not suggest {True: 1, False: 0}. The problem is that a "downvote" then becomes pointless. So probably go with False as -1. So you'd want: sum((1 if vote else -1) for vote in votes) From dreamingforward at gmail.com Tue Jul 2 20:05:35 2013 From: dreamingforward at gmail.com (Mark Janssen) Date: Tue, 2 Jul 2013 17:05:35 -0700 Subject: how to calculate reputation In-Reply-To: References: Message-ID: > Hi all, this seems to be quite stupid question but I am "confused".. > We set the initial value to 0, +1 for up-vote and -1 for down-vote! nice. > > I have a list of bool values True, False (True for up vote, False for > down-vote).. submitted by users. > > should I take True = +1, False=0 [or] True = +1, False=-1 ?? for adding > all. > > I am missing something here.. and that's clear.. anyone please help me on > it? If False is representing a down-vote, like you say, then you have to incorporate that information, in which case False=-1 ==> a user not merely ignored another user, but marked him/her down. MarkJ Tacoma, Washington From djc at news.invalid Sat Jul 6 09:23:25 2013 From: djc at news.invalid (DJC) Date: Sat, 06 Jul 2013 15:23:25 +0200 Subject: how to calculate reputation In-Reply-To: References: Message-ID: On 03/07/13 02:05, Mark Janssen wrote: >> Hi all, this seems to be quite stupid question but I am "confused".. >> We set the initial value to 0, +1 for up-vote and -1 for down-vote! nice. >> >> I have a list of bool values True, False (True for up vote, False for >> down-vote).. submitted by users. >> >> should I take True = +1, False=0 [or] True = +1, False=-1 ?? for adding >> all. >> >> I am missing something here.. and that's clear.. anyone please help me on >> it? > > If False is representing a down-vote, like you say, then you have to > incorporate that information, in which case False=-1 ==> a user not > merely ignored another user, but marked him/her down. You could express the result as 'x out of y users up-voted this' where x = total true and y = x + total_false From leegold at operamail.com Tue Jul 2 20:51:47 2013 From: leegold at operamail.com (goldtech) Date: Tue, 2 Jul 2013 17:51:47 -0700 (PDT) Subject: Read active tab URL from a browser with Python ? Message-ID: Hi, I've goggled this but haven't found a way (at least a way I understand) for a running Python script to get the current URL in a browser's location bar. Is there a way? I'm using Windows so maybe that would provide a way (or not). This isn't for a browser controlled/automated by Python that Pyhton might have a "hook" in, but just a "typical" running browser FF, IE... Any help appreciated. Lee G. From wuwei23 at gmail.com Tue Jul 2 21:38:34 2013 From: wuwei23 at gmail.com (alex23) Date: Wed, 03 Jul 2013 11:38:34 +1000 Subject: Read active tab URL from a browser with Python ? In-Reply-To: References: Message-ID: On 3/07/2013 10:51 AM, goldtech wrote: > I've goggled this but haven't found a way (at least a way I understand) for a running Python script to get the current URL in a browser's location bar. Is there a way? I'm using Windows so maybe that would provide a way (or not). > This isn't for a browser controlled/automated by Python that Pyhton might have a "hook" in, but just a "typical" running browser FF, IE... Unfortunately, I don't think there's a simple, consistent way to do this without delving down into deep Windows COM voodoo like this: http://stackoverflow.com/questions/2876755/get-name-of-active-excel-workbook-from-python Another option might be to use Python in conjuction with AutoIt: http://www.pha.com.au/kb/index.php/Using_AutoIT_with_Python http://www.autoitscript.com/forum/topic/115293-how-to-get-firefox-current-page-address/ From leegold at operamail.com Wed Jul 3 14:23:02 2013 From: leegold at operamail.com (goldtech) Date: Wed, 3 Jul 2013 11:23:02 -0700 (PDT) Subject: Read active tab URL from a browser with Python ? In-Reply-To: References: Message-ID: <42e05d8a-9b85-4c0b-bf48-4b1dfeec5cb7@googlegroups.com> snip > > Another option might be to use Python in conjuction with AutoIt: > > > > http://www.pha.com.au/kb/index.php/Using_AutoIT_with_Python > > http://www.autoitscript.com/forum/topic/115293-how-to-get-firefox-current-page-address/ This is powerful, and seems like I can call Autoit from Python. Thanks for turning me on to this! From leegold at operamail.com Tue Jul 2 21:20:12 2013 From: leegold at operamail.com (goldtech) Date: Tue, 2 Jul 2013 18:20:12 -0700 (PDT) Subject: How to tell Script to use pythonw.exe ? Message-ID: Hi, Using Windows.... I want to run a .py file script using pythonw.exe so the DOS box will not open. Is there a way from inside the script to say "run me with pythonw.exe and not python.exe"? Thanks From leegold at operamail.com Tue Jul 2 21:28:40 2013 From: leegold at operamail.com (goldtech) Date: Tue, 2 Jul 2013 18:28:40 -0700 (PDT) Subject: How to tell Script to use pythonw.exe ? In-Reply-To: References: Message-ID: I just changed the file extension of the script file from .py to .pyw and it uses pythonw.exe. I didn't read it anywhere, just intuited it and tried it. Python has some very smart people working the language... From timr at probo.com Tue Jul 2 23:43:44 2013 From: timr at probo.com (Tim Roberts) Date: Tue, 02 Jul 2013 20:43:44 -0700 Subject: How to tell Script to use pythonw.exe ? References: Message-ID: goldtech wrote: > >I just changed the file extension of the script file from .py to .pyw >and it uses pythonw.exe. I didn't read it anywhere, just intuited it >and tried it. Python has some very smart people working the language... While your statement is true, that's not what happened here. Windows has long had the ability to associate a file extension with a handler. If you start a command shell, the "assoc" command tells you the program type associated with an extension, and the "ftype" command tells you the command line that will be executed for that program type. On my box: C:\tmp>assoc .py .py=Python C:\tmp>ftype Python Python="C:\Apps\Python27\Python.exe" "%1" %* C:\tmp>assoc .pyw .pyw=Python.NoConFile C:\tmp>ftype Python.NoConFile Python.NoConFile="C:\Apps\Python27\Pythonw.exe" "%1" %* You can create your own, if you want. If you want files with a .script extension to run PythonW, you can type: assoc .script=Python.NoConFile -- Tim Roberts, timr at probo.com Providenza & Boekelheide, Inc. From nikos at superhost.gr Wed Jul 3 11:22:53 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Wed, 03 Jul 2013 18:22:53 +0300 Subject: How to tell Script to use pythonw.exe ? In-Reply-To: References: Message-ID: ???? 3/7/2013 6:43 ??, ?/? Tim Roberts ??????: > goldtech wrote: >> >> I just changed the file extension of the script file from .py to .pyw >> and it uses pythonw.exe. I didn't read it anywhere, just intuited it >> and tried it. Python has some very smart people working the language... > > While your statement is true, that's not what happened here. > > Windows has long had the ability to associate a file extension with a > handler. If you start a command shell, the "assoc" command tells you the > program type associated with an extension, and the "ftype" command tells > you the command line that will be executed for that program type. On my > box: > > C:\tmp>assoc .py > .py=Python > > C:\tmp>ftype Python > Python="C:\Apps\Python27\Python.exe" "%1" %* > > C:\tmp>assoc .pyw > .pyw=Python.NoConFile > > C:\tmp>ftype Python.NoConFile > Python.NoConFile="C:\Apps\Python27\Pythonw.exe" "%1" %* > > You can create your own, if you want. If you want files with a .script > extension to run PythonW, you can type: > > assoc .script=Python.NoConFile > My associations are broken, bt i only care for open web pages with Chrome instead of IE, so i sued your method: C:\Windows\system32>assoc .html=Chrome .html=Chrome C:\Windows\system32>ftype Chrome="C:\Users\Ferrous\AppData\Local\Google\Chrome\Application\chrome.exe" %1 Chrome="C:\Users\Ferrous\AppData\Local\Google\Chrome\Application\chrome.exe" %1 but still when i click a link IE keeps popping up isntead of Chrome. Any ideas why? -- What is now proved was at first only imagined! From benjamin.kaplan at case.edu Wed Jul 3 12:36:57 2013 From: benjamin.kaplan at case.edu (Benjamin Kaplan) Date: Wed, 3 Jul 2013 09:36:57 -0700 Subject: How to tell Script to use pythonw.exe ? In-Reply-To: References: Message-ID: On Jul 3, 2013 8:27 AM, "?????" wrote: > > ???? 3/7/2013 6:43 ??, ?/? Tim Roberts ??????: > >> goldtech wrote: >>> >>> >>> I just changed the file extension of the script file from .py to .pyw >>> and it uses pythonw.exe. I didn't read it anywhere, just intuited it >>> and tried it. Python has some very smart people working the language... >> >> >> While your statement is true, that's not what happened here. >> >> Windows has long had the ability to associate a file extension with a >> handler. If you start a command shell, the "assoc" command tells you the >> program type associated with an extension, and the "ftype" command tells >> you the command line that will be executed for that program type. On my >> box: >> >> C:\tmp>assoc .py >> .py=Python >> >> C:\tmp>ftype Python >> Python="C:\Apps\Python27\Python.exe" "%1" %* >> >> C:\tmp>assoc .pyw >> .pyw=Python.NoConFile >> >> C:\tmp>ftype Python.NoConFile >> Python.NoConFile="C:\Apps\Python27\Pythonw.exe" "%1" %* >> >> You can create your own, if you want. If you want files with a .script >> extension to run PythonW, you can type: >> >> assoc .script=Python.NoConFile >> > My associations are broken, bt i only care for open web pages with Chrome instead of IE, so i sued your method: > > > C:\Windows\system32>assoc .html=Chrome > .html=Chrome > > C:\Windows\system32>ftype Chrome="C:\Users\Ferrous\AppData\Local\Google\Chrome\Application\chrome.exe" %1 > > Chrome="C:\Users\Ferrous\AppData\Local\Google\Chrome\Application\chrome.exe" %1 > > but still when i click a link IE keeps popping up isntead of Chrome. > Any ideas why? Because your links don't open files. They send requests to an http server for data. And IE is the program designated to send http requests. Just use the browser's "make this the default" button. > -- > What is now proved was at first only imagined > -- > http://mail.python.org/mailman/listinfo/python-list -------------- next part -------------- An HTML attachment was scrubbed... URL: From nikos at superhost.gr Wed Jul 3 12:50:14 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Wed, 03 Jul 2013 19:50:14 +0300 Subject: How to tell Script to use pythonw.exe ? In-Reply-To: References: Message-ID: ???? 3/7/2013 7:36 ??, ?/? Benjamin Kaplan ??????: > > On Jul 3, 2013 8:27 AM, "?????" > wrote: > > > > ???? 3/7/2013 6:43 ??, ?/? Tim Roberts ??????: > > > >> goldtech > wrote: > >>> > >>> > >>> I just changed the file extension of the script file from .py to .pyw > >>> and it uses pythonw.exe. I didn't read it anywhere, just intuited it > >>> and tried it. Python has some very smart people working the language... > >> > >> > >> While your statement is true, that's not what happened here. > >> > >> Windows has long had the ability to associate a file extension with a > >> handler. If you start a command shell, the "assoc" command tells > you the > >> program type associated with an extension, and the "ftype" command tells > >> you the command line that will be executed for that program type. On my > >> box: > >> > >> C:\tmp>assoc .py > >> .py=Python > >> > >> C:\tmp>ftype Python > >> Python="C:\Apps\Python27\Python.exe" "%1" %* > >> > >> C:\tmp>assoc .pyw > >> .pyw=Python.NoConFile > >> > >> C:\tmp>ftype Python.NoConFile > >> Python.NoConFile="C:\Apps\Python27\Pythonw.exe" "%1" %* > >> > >> You can create your own, if you want. If you want files with a .script > >> extension to run PythonW, you can type: > >> > >> assoc .script=Python.NoConFile > >> > > My associations are broken, bt i only care for open web pages with > Chrome instead of IE, so i sued your method: > > > > > > C:\Windows\system32>assoc .html=Chrome > > .html=Chrome > > > > C:\Windows\system32>ftype > Chrome="C:\Users\Ferrous\AppData\Local\Google\Chrome\Application\chrome.exe" > %1 > > > > > Chrome="C:\Users\Ferrous\AppData\Local\Google\Chrome\Application\chrome.exe" > %1 > > > > but still when i click a link IE keeps popping up isntead of Chrome. > > Any ideas why? > > Because your links don't open files. They send requests to an http > server for data. And IE is the program designated to send http requests. > Just use the browser's "make this the default" button. I dont understand you. I explicitly state via cmd to have the .html files opened with Chrome instead of IE. Tried it with the way you said and evben with "open with.." but all that fails. some seriosu damaged must have happened and assoc are refusing to change. -- What is now proved was at first only imagined! From wuwei23 at gmail.com Wed Jul 3 21:28:09 2013 From: wuwei23 at gmail.com (alex23) Date: Thu, 04 Jul 2013 11:28:09 +1000 Subject: How to tell Script to use pythonw.exe ? In-Reply-To: References: Message-ID: On 4/07/2013 2:50 AM, ????? wrote: > I dont understand you. I explicitly state via cmd to have the .html > files opened with Chrome instead of IE. > Tried it with the way you said and evben with "open with.." but all that > fails. > some seriosu damaged must have happened and assoc are refusing to change. This list is for discussing Python, not your concerns du jour with various operating systems. Please take this to comp.os.ms-windows.misc instead. From steve+comp.lang.python at pearwood.info Tue Jul 2 21:29:52 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 03 Jul 2013 01:29:52 GMT Subject: How to tell Script to use pythonw.exe ? References: Message-ID: <51d37e90$0$29999$c3e8da3$5496439d@news.astraweb.com> On Tue, 02 Jul 2013 18:20:12 -0700, goldtech wrote: > Hi, > > Using Windows.... > > I want to run a .py file script using pythonw.exe so the DOS box will > not open. Is there a way from inside the script to say "run me with > pythonw.exe and not python.exe"? I don't believe so, because by the time the script is even opened, it is too late. I'm not an expert about Windows, but as I understand it, the process that occurs when you double-click the file is: * Windows inspects the file extension and sees it is .py * Windows checks the registry for the file extension association and finds python.exe * Windows calls python.exe with the path to the script as argument * finally python.exe opens the script. Instead, you can use .pyw as the file extension, which should do what you want. -- Steven From robotsondrugs at gmail.com Tue Jul 2 21:34:02 2013 From: robotsondrugs at gmail.com (Andrew Berg) Date: Tue, 02 Jul 2013 20:34:02 -0500 Subject: How to tell Script to use pythonw.exe ? In-Reply-To: References: Message-ID: <51D37F8A.3010905@gmail.com> On 2013.07.02 20:20, goldtech wrote: > Using Windows.... > > I want to run a .py file script using pythonw.exe so the DOS box will not open. Is there a way from inside the script to say "run me with pythonw.exe and not python.exe"? Use the .pyw extension instead of .py. Also, just FYI, DOS is long dead, and is much, much different under the hood from the console subsystem in modern versions of Windows. -- CPython 3.3.2 | Windows NT 6.2.9200 / FreeBSD 9.1 From mail at timgolden.me.uk Wed Jul 3 03:34:45 2013 From: mail at timgolden.me.uk (Tim Golden) Date: Wed, 03 Jul 2013 08:34:45 +0100 Subject: DOS or not? [was Re: How to tell Script to use pythonw.exe ?] In-Reply-To: <51D37F8A.3010905@gmail.com> References: <51D37F8A.3010905@gmail.com> Message-ID: <51D3D415.5060802@timgolden.me.uk> On 03/07/2013 02:34, Andrew Berg wrote: > DOS is long > dead, and is much, much different under the hood from the console > subsystem in modern versions of Windows. While this is clearly true, it's by no means unusual for people to refer to the "DOS Box" or talk about "DOS commands" etc. even when they're quite well aware of the history of Windows and its Console subsystem. It's just quicker than saying "Console Window" or something. I mention this because it seems to get called out every time someone uses the term "DOS" on this and other Python lists and it can smack slightly of pedantry. TJG From rosuav at gmail.com Wed Jul 3 03:41:08 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 3 Jul 2013 17:41:08 +1000 Subject: DOS or not? [was Re: How to tell Script to use pythonw.exe ?] In-Reply-To: <51D3D415.5060802@timgolden.me.uk> References: <51D37F8A.3010905@gmail.com> <51D3D415.5060802@timgolden.me.uk> Message-ID: On Wed, Jul 3, 2013 at 5:34 PM, Tim Golden wrote: > On 03/07/2013 02:34, Andrew Berg wrote: >> DOS is long >> dead, and is much, much different under the hood from the console >> subsystem in modern versions of Windows. > > > While this is clearly true, it's by no means unusual for people to refer > to the "DOS Box" or talk about "DOS commands" etc. even when they're > quite well aware of the history of Windows and its Console subsystem. > It's just quicker than saying "Console Window" or something. > > I mention this because it seems to get called out every time someone > uses the term "DOS" on this and other Python lists and it can smack > slightly of pedantry. I would avoid the term "DOS Box" in reference to cmd.exe though, because DOSBox is an emulator. (Also because I have an OS/2 heritage, where a DOS Window is command.com in a virtual x86 windowed session, while an OS/2 Window is the native OS/2 command interpreter.) But in general usage, calling them "DOS commands" is sloppy but unambiguous. I wouldn't be concerned one way or the other. ChrisA From robotsondrugs at gmail.com Wed Jul 3 04:28:01 2013 From: robotsondrugs at gmail.com (Andrew Berg) Date: Wed, 03 Jul 2013 03:28:01 -0500 Subject: DOS or not? [was Re: How to tell Script to use pythonw.exe ?] In-Reply-To: <51D3D415.5060802@timgolden.me.uk> References: <51D37F8A.3010905@gmail.com> <51D3D415.5060802@timgolden.me.uk> Message-ID: <51D3E091.6020706@gmail.com> On 2013.07.03 02:34, Tim Golden wrote: > While this is clearly true, it's by no means unusual for people to refer > to the "DOS Box" or talk about "DOS commands" etc. even when they're > quite well aware of the history of Windows and its Console subsystem. > It's just quicker than saying "Console Window" or something. > > I mention this because it seems to get called out every time someone > uses the term "DOS" on this and other Python lists and it can smack > slightly of pedantry. It's not as ambiguous (or as common) as it used to be, but it has always bothered me when someone refers to the Win32 console as DOS. I try not to be rude about it, but I really would like to prevent those who are not very familiar with Windows or its history from associating the limits and internal behavior of MS-DOS with the console subsystem of Windows NT. Especially with the anti-Windows sentiment that seems to be getting more popular (not that it's entirely without merit, but that's another topic for another day on another list), I'd rather see operating systems judged on things that are actually true and not miscellaneous FUD spread by ignorance and confusion. I realize it can come off as pedantic, but what may be obvious to those with of us with a lot of experience with different operating systems over the years can easily slip past a novice. BTW, there are several short and unambiguous terms one can use to refer to the Windows CLI environment (or parts thereof): cmd, command prompt, command line, terminal, console, etc.. Also, I don't think I've ever encountered anyone who prefers to call it DOS even though they know it's not correct, but if you say it's not unusual, then they're obviously out there, and I'll keep that in mind before jumping in to correct them. -- CPython 3.3.2 | Windows NT 6.2.9200 / FreeBSD 9.1 From mail at timgolden.me.uk Wed Jul 3 04:51:05 2013 From: mail at timgolden.me.uk (Tim Golden) Date: Wed, 03 Jul 2013 09:51:05 +0100 Subject: DOS or not? [was Re: How to tell Script to use pythonw.exe ?] In-Reply-To: <51D3E091.6020706@gmail.com> References: <51D37F8A.3010905@gmail.com> <51D3D415.5060802@timgolden.me.uk> <51D3E091.6020706@gmail.com> Message-ID: <51D3E5F9.6010008@timgolden.me.uk> On 03/07/2013 09:28, Andrew Berg wrote: > On 2013.07.03 02:34, Tim Golden wrote: >> While this is clearly true, it's by no means unusual for people to >> refer to the "DOS Box" or talk about "DOS commands" etc. even when >> they're quite well aware of the history of Windows and its Console >> subsystem. It's just quicker than saying "Console Window" or >> something. >> >> I mention this because it seems to get called out every time >> someone uses the term "DOS" on this and other Python lists and it >> can smack slightly of pedantry. > I really would like to prevent those who are not very familiar with > Windows or its history from associating the limits and internal > behavior of MS-DOS with the console subsystem of Windows NT. > Especially with the anti-Windows sentiment that seems to be getting > more popular (not that it's entirely without merit, but that's > another topic for another day on another list), I'd rather see > operating systems judged on things that are actually true and not > miscellaneous FUD spread by ignorance and confusion. We can certainly agree on this. I can't count the number of emails I've deleted as too hot-headed in response to dismissive comments about Windows as a platform. Some of them, at least, appear to be from people who last actually used Windows back in the 9x days when the command window was very limited indeed. I realize it can > come off as pedantic, but what may be obvious to those with of us > with a lot of experience with different operating systems over the > years can easily slip past a novice. I suppose I view it in the same light as people (very few, thankfully, in my experience) who go out of their way to correct "MB" to "MiB" when talking about disk sizes. Or -- and I'm definitely guilty of this -- of pointing out that London telephone numbers are all 020 plus eight digits and *not* 0207 or 0208 plus seven digits. Whenever I do that I'm aware that I'm technically in the right but that, for all practical purposes, it's a needless precision. Obviously, if it were clearly a source of confusion in some context I'd clarify what needed to be clarified. TJG From python.list at tim.thechases.com Wed Jul 3 08:50:46 2013 From: python.list at tim.thechases.com (Tim Chase) Date: Wed, 3 Jul 2013 07:50:46 -0500 Subject: DOS or not? [was Re: How to tell Script to use pythonw.exe ?] In-Reply-To: <51D3E5F9.6010008@timgolden.me.uk> References: <51D37F8A.3010905@gmail.com> <51D3D415.5060802@timgolden.me.uk> <51D3E091.6020706@gmail.com> <51D3E5F9.6010008@timgolden.me.uk> Message-ID: <20130703075046.2f0737de@bigbox.christie.dr> On 2013-07-03 09:51, Tim Golden wrote: > We can certainly agree on this. I can't count the number of emails > I've deleted as too hot-headed in response to dismissive comments > about Windows as a platform. Some of them, at least, appear to be > from people who last actually used Windows back in the 9x days when > the command window was very limited indeed. I guess one of my biggest frustrations with the cmd.exe (and command.com) interpreters is that argument processing is left to the application, so each application may do it slightly differently: C:\temp\> find weather *.py FIND: Parameter format not correct C:\temp\> find "weather" *.py ---------- WFD.PY weather = Weather(lat, lon) C:\temp\> findstr weather *.py wfd.py: weather = Weather(lat, lon) C:\temp\> findstr "weather" *.py wfd.py: weather = Weather(lat, lon) And more maddeningly: C:\temp\> start file.txt ... opens the file correctly in Notepad C:\temp\> start "file with space.txt" ... opens a new dos box with the name "file with space.txt" rather than opening the file C:\temp\> start "" "file with space.txt" ... opens the file correctly in Notepad It's the little inconsistencies like this that wear daily on me. That and the lack of built-in utilities, so I'm regularly adding GNU tools on new boxes. -tkc From mail at timgolden.me.uk Wed Jul 3 09:00:49 2013 From: mail at timgolden.me.uk (Tim Golden) Date: Wed, 03 Jul 2013 14:00:49 +0100 Subject: DOS or not? [was Re: How to tell Script to use pythonw.exe ?] In-Reply-To: <20130703075046.2f0737de@bigbox.christie.dr> References: <51D37F8A.3010905@gmail.com> <51D3D415.5060802@timgolden.me.uk> <51D3E091.6020706@gmail.com> <51D3E5F9.6010008@timgolden.me.uk> <20130703075046.2f0737de@bigbox.christie.dr> Message-ID: <51D42081.1030103@timgolden.me.uk> On 03/07/2013 13:50, Tim Chase wrote: > On 2013-07-03 09:51, Tim Golden wrote: >> We can certainly agree on this. I can't count the number of emails >> I've deleted as too hot-headed in response to dismissive comments >> about Windows as a platform. Some of them, at least, appear to be >> from people who last actually used Windows back in the 9x days when >> the command window was very limited indeed. > > I guess one of my biggest frustrations with the cmd.exe (and > command.com) interpreters is that argument processing is left to the > application, so each application may do it slightly differently: Goodness, I doubt if you'll find anyone who can seriously make a case that the Windows command prompt is all it might be. I'm not a Powershell user myself but people speak highly of it. Or, as you say, you can use the GNU tools either natively or via cygwin. Not my cup of tea, but that's the way of tools: one man's meat... More to the point, I've got no problem with informed criticism (although there's little point in grumbling just for the sake of it). The problem I have is with criticisms which are years out of date, or which appear to be fuelled by prejudice more than by experience. TJG From steve+comp.lang.python at pearwood.info Wed Jul 3 09:19:26 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 03 Jul 2013 13:19:26 GMT Subject: DOS or not? [was Re: How to tell Script to use pythonw.exe ?] References: <51D37F8A.3010905@gmail.com> <51D3D415.5060802@timgolden.me.uk> <51D3E091.6020706@gmail.com> <51D3E5F9.6010008@timgolden.me.uk> <20130703075046.2f0737de@bigbox.christie.dr> Message-ID: <51d424de$0$9505$c3e8da3$5496439d@news.astraweb.com> On Wed, 03 Jul 2013 14:00:49 +0100, Tim Golden wrote: > Goodness, I doubt if you'll find anyone who can seriously make a case > that the Windows command prompt is all it might be. I'm not a Powershell > user myself but people speak highly of it. I understand that Powershell is aimed more for batch use rather than interactive use. -- Steven From jeff at schwabcenter.com Wed Jul 3 09:22:24 2013 From: jeff at schwabcenter.com (Jeff Schwab) Date: Wed, 3 Jul 2013 09:22:24 -0400 Subject: DOS or not? [was Re: How to tell Script to use pythonw.exe ?] References: <51d424de$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: <2013070309222451280-jeff@schwabcentercom> On 2013-07-03 13:19:26 +0000, Steven D'Aprano said: > On Wed, 03 Jul 2013 14:00:49 +0100, Tim Golden wrote: > >> Goodness, I doubt if you'll find anyone who can seriously make a case >> that the Windows command prompt is all it might be. I'm not a Powershell >> user myself but people speak highly of it. > > I understand that Powershell is aimed more for batch use rather than > interactive use. Not sure what gave that impression. PS comes with handy Unix-like aliases by default. I used it interactively every day for years, and recommend it over either cmd or Cygwin. From ian.g.kelly at gmail.com Wed Jul 3 19:35:52 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 3 Jul 2013 17:35:52 -0600 Subject: DOS or not? [was Re: How to tell Script to use pythonw.exe ?] In-Reply-To: <2t79t81lbf5v8aeleicalff2q167e1v849@4ax.com> References: <51D37F8A.3010905@gmail.com> <51D3D415.5060802@timgolden.me.uk> <51D3E091.6020706@gmail.com> <51D3E5F9.6010008@timgolden.me.uk> <20130703075046.2f0737de@bigbox.christie.dr> <51d424de$0$9505$c3e8da3$5496439d@news.astraweb.com> <2t79t81lbf5v8aeleicalff2q167e1v849@4ax.com> Message-ID: On Wed, Jul 3, 2013 at 4:11 PM, Dennis Lee Bieber wrote: > On 03 Jul 2013 13:19:26 GMT, Steven D'Aprano > declaimed the following: > >>On Wed, 03 Jul 2013 14:00:49 +0100, Tim Golden wrote: >> >>> Goodness, I doubt if you'll find anyone who can seriously make a case >>> that the Windows command prompt is all it might be. I'm not a Powershell >>> user myself but people speak highly of it. >> >>I understand that Powershell is aimed more for batch use rather than >>interactive use. > > In one respect: no... > > Consider that the Powershell default is to /prevent/ execution of > script files unless some security settings have been changed; even local > script files need to be "signed" to be executed. IOW, it's aimed at *secure* batch use for paranoid sysadmins. According to microsoft.com: """ Windows PowerShell? is a task-based command-line shell and scripting language designed especially for system administration. Built on the .NET Framework, Windows PowerShell? helps IT professionals and power users control and automate the administration of the Windows operating system and applications that run on Windows. """ Which would seem to indicate that it targets both interactive and scripting uses. From wayne at waynewerner.com Thu Jul 4 10:08:18 2013 From: wayne at waynewerner.com (Wayne Werner) Date: Thu, 4 Jul 2013 09:08:18 -0500 (CDT) Subject: DOS or not? [was Re: How to tell Script to use pythonw.exe ?] In-Reply-To: <2t79t81lbf5v8aeleicalff2q167e1v849@4ax.com> References: <51D37F8A.3010905@gmail.com> <51D3D415.5060802@timgolden.me.uk> <51D3E091.6020706@gmail.com> <51D3E5F9.6010008@timgolden.me.uk> <20130703075046.2f0737de@bigbox.christie.dr> <51d424de$0$9505$c3e8da3$5496439d@news.astraweb.com> <2t79t81lbf5v8aeleicalff2q167e1v849@4ax.com> Message-ID: On Wed, 3 Jul 2013, Dennis Lee Bieber wrote: > Consider that the Powershell default is to /prevent/ execution of > script files unless some security settings have been changed; even local > script files need to be "signed" to be executed. Protip: No they don't - wrap it in a cmd/bat file and have it launch powershell[1]: powershell -ExecutionPolicy Bypass -File ... \o/ Microsoft "security" at it again! (reminds me a bit of just pushing "Cancel" to log into windows 98, I think it was) -W [1]: http://stackoverflow.com/q/728143/344286 From robotsondrugs at gmail.com Thu Jul 4 18:12:13 2013 From: robotsondrugs at gmail.com (Andrew Berg) Date: Thu, 04 Jul 2013 17:12:13 -0500 Subject: DOS or not? [was Re: How to tell Script to use pythonw.exe ?] In-Reply-To: References: <51D37F8A.3010905@gmail.com> <51D3D415.5060802@timgolden.me.uk> <51D3E091.6020706@gmail.com> <51D3E5F9.6010008@timgolden.me.uk> <20130703075046.2f0737de@bigbox.christie.dr> <51d424de$0$9505$c3e8da3$5496439d@news.astraweb.com> <2t79t81lbf5v8aeleicalff2q167e1v849@4ax.com> Message-ID: <51D5F33D.9080008@gmail.com> On 2013.07.04 09:08, Wayne Werner wrote: > powershell -ExecutionPolicy Bypass -File ... > > > \o/ > > Microsoft "security" at it again! (reminds me a bit of just pushing > "Cancel" to log into windows 98, I think it was) >From an MSDN page linked in one of the answers: > Now, why is > > PowerShell.exe ?ExecutionPolicy Bypass ?File c:\temp\bad-script.ps1 > > not a security bug? Ultimately, if bad code has the ability to run this code, it already has control of the machine. http://blogs.msdn.com/b/powershell/archive/2008/09/30/powershell-s-security-guiding-principles.aspx If an attacker can run code, he/she already has the capability to well, run code. -- CPython 3.3.2 | Windows NT 6.2.9200 / FreeBSD 9.1 From rosuav at gmail.com Thu Jul 4 18:39:39 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 5 Jul 2013 08:39:39 +1000 Subject: DOS or not? [was Re: How to tell Script to use pythonw.exe ?] In-Reply-To: <51D5F33D.9080008@gmail.com> References: <51D37F8A.3010905@gmail.com> <51D3D415.5060802@timgolden.me.uk> <51D3E091.6020706@gmail.com> <51D3E5F9.6010008@timgolden.me.uk> <20130703075046.2f0737de@bigbox.christie.dr> <51d424de$0$9505$c3e8da3$5496439d@news.astraweb.com> <2t79t81lbf5v8aeleicalff2q167e1v849@4ax.com> <51D5F33D.9080008@gmail.com> Message-ID: On Fri, Jul 5, 2013 at 8:12 AM, Andrew Berg wrote: > On 2013.07.04 09:08, Wayne Werner wrote: >> powershell -ExecutionPolicy Bypass -File ... >> >> >> \o/ >> >> Microsoft "security" at it again! (reminds me a bit of just pushing >> "Cancel" to log into windows 98, I think it was) > From an MSDN page linked in one of the answers: >> Now, why is >> >> PowerShell.exe ?ExecutionPolicy Bypass ?File c:\temp\bad-script.ps1 >> >> not a security bug? Ultimately, if bad code has the ability to run this code, it already has control of the machine. > http://blogs.msdn.com/b/powershell/archive/2008/09/30/powershell-s-security-guiding-principles.aspx > > If an attacker can run code, he/she already has the capability to well, run code. Well, the whole point of sandboxing is to allow some code and not other - look at web browser scripts. You can run your JavaScript code on someone else's machine without the capability to run arbitrary code. What this proves is that PowerShell is not a sandboxing environment. It has just two states: Trusted and untrusted. Untrusted code may not run. Trusted code has full access as though the administrator typed the commands by hand. Unix has measures to prevent a running process from having full control over the system, but even there, privilege escalation attacks (usually involving some application that runs as root) have been known. Restricting a running binary (as opposed to creating an interpreted and very slow language) is a distinctly hard problem. ChrisA From rosuav at gmail.com Wed Jul 3 10:00:04 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Jul 2013 00:00:04 +1000 Subject: DOS or not? [was Re: How to tell Script to use pythonw.exe ?] In-Reply-To: <20130703075046.2f0737de@bigbox.christie.dr> References: <51D37F8A.3010905@gmail.com> <51D3D415.5060802@timgolden.me.uk> <51D3E091.6020706@gmail.com> <51D3E5F9.6010008@timgolden.me.uk> <20130703075046.2f0737de@bigbox.christie.dr> Message-ID: On Wed, Jul 3, 2013 at 10:50 PM, Tim Chase wrote: > On 2013-07-03 09:51, Tim Golden wrote: >> We can certainly agree on this. I can't count the number of emails >> I've deleted as too hot-headed in response to dismissive comments >> about Windows as a platform. Some of them, at least, appear to be >> from people who last actually used Windows back in the 9x days when >> the command window was very limited indeed. > > I guess one of my biggest frustrations with the cmd.exe (and > command.com) interpreters is that argument processing is left to the > application, so each application may do it slightly differently: > > C:\temp\> find weather *.py > FIND: Parameter format not correct > C:\temp\> find "weather" *.py > ---------- WFD.PY > weather = Weather(lat, lon) > C:\temp\> findstr weather *.py > wfd.py: weather = Weather(lat, lon) > C:\temp\> findstr "weather" *.py > wfd.py: weather = Weather(lat, lon) > > And more maddeningly: > > C:\temp\> start file.txt > ... opens the file correctly in Notepad > C:\temp\> start "file with space.txt" > ... opens a new dos box with the name "file with space.txt" rather > than opening the file > C:\temp\> start "" "file with space.txt" > ... opens the file correctly in Notepad > > It's the little inconsistencies like this that wear daily on me. That > and the lack of built-in utilities, so I'm regularly adding GNU tools > on new boxes. The issue you have there is mainly that the quotes are serving double purpose. Yes, they delimit and thus can be used to surround a file name with spaces in it, but they're also significant to a couple of apps (FIND uses them to indicate the search string, START looks for a quoted argument to use as the title). I'm not entirely sure how it's done under the covers; C code looking at argc/argv sees quoted arguments without their quotes, exactly as I would expect on Unix, and yet the convention is to notice the quotes. The issue with START is 100% understandable and 100% annoying. ChrisA From victorhooi at gmail.com Tue Jul 2 23:47:29 2013 From: victorhooi at gmail.com (Victor Hooi) Date: Tue, 2 Jul 2013 20:47:29 -0700 (PDT) Subject: Python - forking an external process? Message-ID: <14be21de-2ceb-464a-a638-dce0368ab9e7@googlegroups.com> Hi, I have a Python script where I want to run fork and run an external command (or set of commands). For example, after doing , I then want to run ssh to a host, handover control back to the user, and have my script terminate. Or I might want to run ssh to a host, less a certain textfile, then exit. What's the idiomatic way of doing this within Python? Is it possible to do with Subprocess? Cheers, Victor (I did see this SO post - http://stackoverflow.com/questions/6011235/run-a-program-from-python-and-have-it-continue-to-run-after-the-script-is-kille, but it's a bit older, and I was going to see what the current idiomatic way of doing this is). From rustompmody at gmail.com Tue Jul 2 23:59:19 2013 From: rustompmody at gmail.com (rusi) Date: Tue, 2 Jul 2013 20:59:19 -0700 (PDT) Subject: Python - forking an external process? In-Reply-To: <14be21de-2ceb-464a-a638-dce0368ab9e7@googlegroups.com> References: <14be21de-2ceb-464a-a638-dce0368ab9e7@googlegroups.com> Message-ID: <3d55a01e-379d-490e-8573-548d2ac21db5@googlegroups.com> On Wednesday, July 3, 2013 9:17:29 AM UTC+5:30, Victor Hooi wrote: > Hi, > > I have a Python script where I want to run fork and run an external command > (or set of commands). > For example, after doing , I then want to run ssh to a host, handover > control back to the user, and have my script terminate. Seen Fabric? http://docs.fabfile.org/en/1.6/ Recently -- within the last month methinks -- there was someone who posted a supposed improvement to it (forget the name) From victorhooi at gmail.com Wed Jul 3 00:11:32 2013 From: victorhooi at gmail.com (Victor Hooi) Date: Tue, 2 Jul 2013 21:11:32 -0700 (PDT) Subject: Python - forking an external process? In-Reply-To: <3d55a01e-379d-490e-8573-548d2ac21db5@googlegroups.com> References: <14be21de-2ceb-464a-a638-dce0368ab9e7@googlegroups.com> <3d55a01e-379d-490e-8573-548d2ac21db5@googlegroups.com> Message-ID: Hi, Hmm, this script is actually written using the Cliff framework (https://github.com/dreamhost/cliff). I was hoping to keep the whole approach fairly simple, without needing to pull in too much external stuff, or set anything up. There's no way to do it with just Python core is there? Also, what's this improvement you mentioned? Cheers, Victor On Wednesday, 3 July 2013 13:59:19 UTC+10, rusi wrote: > On Wednesday, July 3, 2013 9:17:29 AM UTC+5:30, Victor Hooi wrote: > > > Hi, > > > > > > I have a Python script where I want to run fork and run an external command > > > (or set of commands). > > > For example, after doing , I then want to run ssh to a host, handover > > > control back to the user, and have my script terminate. > > > > Seen Fabric? > > http://docs.fabfile.org/en/1.6/ > > > > Recently -- within the last month methinks -- there was someone who posted a supposed improvement to it (forget the name) From rustompmody at gmail.com Wed Jul 3 00:22:45 2013 From: rustompmody at gmail.com (rusi) Date: Tue, 2 Jul 2013 21:22:45 -0700 (PDT) Subject: Python - forking an external process? In-Reply-To: References: <14be21de-2ceb-464a-a638-dce0368ab9e7@googlegroups.com> <3d55a01e-379d-490e-8573-548d2ac21db5@googlegroups.com> Message-ID: <35c74a51-5fab-4750-a8e4-7022c7cde19f@googlegroups.com> On Wednesday, July 3, 2013 9:41:32 AM UTC+5:30, Victor Hooi wrote: > Also, what's this improvement you mentioned? See thread http://mail.python.org/pipermail/python-list/2013-June/650550.html From roy at panix.com Wed Jul 3 00:17:05 2013 From: roy at panix.com (Roy Smith) Date: Wed, 03 Jul 2013 00:17:05 -0400 Subject: Python - forking an external process? References: <14be21de-2ceb-464a-a638-dce0368ab9e7@googlegroups.com> Message-ID: In article <14be21de-2ceb-464a-a638-dce0368ab9e7 at googlegroups.com>, Victor Hooi wrote: > Hi, > > I have a Python script where I want to run fork and run an external command > (or set of commands). > > For example, after doing , I then want to run ssh to a host, handover > control back to the user, and have my script terminate. > > Or I might want to run ssh to a host, less a certain textfile, then exit. > > What's the idiomatic way of doing this within Python? Is it possible to do > with Subprocess? I suspect you are trying to reinvent fabric. It is designed to do exactly these things (in particular, handling all the really complicated stuff about ssh). See http://docs.fabfile.org/ From antoon.pardon at rece.vub.ac.be Wed Jul 3 06:55:55 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Wed, 03 Jul 2013 12:55:55 +0200 Subject: Python - forking an external process? In-Reply-To: <14be21de-2ceb-464a-a638-dce0368ab9e7@googlegroups.com> References: <14be21de-2ceb-464a-a638-dce0368ab9e7@googlegroups.com> Message-ID: <51D4033B.2010409@rece.vub.ac.be> Op 03-07-13 05:47, Victor Hooi schreef: > Hi, > > I have a Python script where I want to run fork and run an external command (or set of commands). > > For example, after doing , I then want to run ssh to a host, handover control back to the user, and have my script terminate. > > Or I might want to run ssh to a host, less a certain textfile, then exit. > > What's the idiomatic way of doing this within Python? Is it possible to do with Subprocess? > If I understand correctly that you want something done in python and then wish to finish python and give control to an other program. I would use an external script that would first launch the python program and then the other program. If for some reason this is a less attractive option, you can use the os.exec family. That terminates the python program will starting up an other program. -- Antoon Pardon From praveen.venkata at gmail.com Wed Jul 3 00:12:09 2013 From: praveen.venkata at gmail.com (Ven) Date: Tue, 2 Jul 2013 21:12:09 -0700 (PDT) Subject: How to define metaclass for a class that extends from sqlalchemy declarative base ? Message-ID: <280863ba-d167-4338-8171-212951854831@googlegroups.com> I use: Python 2.6 and sqlalchemy 0.6.1 This is what I am trying to do: from sqlalchemy.types import ( Integer, String, Boolean ) from sqlalchemy.ext.declarative import declarative_base Base = declarative_base() class SampleMeta(type): def __new__(cls, name, bases, attrs): attrs.update({ 'id': Column('Id', Integer, primary_key=True), 'name': Column('Name', String), 'description': Column('Description', String), 'is_active': Column('IsActive', Boolean) }) return super(SampleMeta, cls).__new__(cls, name, bases, attrs) class Sample(Base): __tablename__ = 'Sample' __table_args__ = {'useexisting': True} __metaclass__ = SampleMeta def __init__(self, id, name, description, is_active): self.id = id self.name = name self.description = description self.is_active = is_active def __repr__(self): return "<(%d, '%s', '%s', %r)>" % (self.id, self.name, self.description, self.isactive) And the error I am getting is this: TypeError: Error when calling the metaclass bases metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases Now, if I do the same thing above by using class Sample(object) instead of class Sample(Base) it works absolutely fine. I need to update the attributes of the class dynamically. So, I will be using dynamic attribute and column names. And I need the above piece code to work in order to be able to get there. **Please help** From __peter__ at web.de Wed Jul 3 01:37:42 2013 From: __peter__ at web.de (Peter Otten) Date: Wed, 03 Jul 2013 07:37:42 +0200 Subject: How to define metaclass for a class that extends from sqlalchemy declarative base ? References: <280863ba-d167-4338-8171-212951854831@googlegroups.com> Message-ID: Ven wrote: > I use: Python 2.6 and sqlalchemy 0.6.1 > > This is what I am trying to do: > > from sqlalchemy.types import ( > Integer, > String, > Boolean > ) > from sqlalchemy.ext.declarative import declarative_base > > Base = declarative_base() > > class SampleMeta(type): > def __new__(cls, name, bases, attrs): > attrs.update({ 'id': Column('Id', Integer, > primary_key=True), > 'name': Column('Name', String), > 'description': Column('Description', String), > 'is_active': Column('IsActive', Boolean) > }) > return super(SampleMeta, cls).__new__(cls, name, bases, attrs) > > class Sample(Base): > __tablename__ = 'Sample' > __table_args__ = {'useexisting': True} > __metaclass__ = SampleMeta > > def __init__(self, id, name, description, is_active): > self.id = id > self.name = name > self.description = description > self.is_active = is_active > > def __repr__(self): > return "<(%d, '%s', '%s', %r)>" % (self.id, self.name, > self.description, self.isactive) > > And the error I am getting is this: > > TypeError: Error when calling the metaclass bases > metaclass conflict: the metaclass of a derived class must be a > (non-strict) subclass of the metaclasses of all its bases > > Now, if I do the same thing above by using > > class Sample(object) > > instead of > > class Sample(Base) > > it works absolutely fine. > > I need to update the attributes of the class dynamically. So, I will be > using dynamic attribute and column names. And I need the above piece code > to work in order to be able to get there. I'm not an sqlalchemy user, but I'd try deriving your metaclass from Base's metaclass: BaseMeta = type(Base) class SampleMeta(BaseMeta): ... class Sample(Base): __metaclass__ = SampleMeta ... From steve+comp.lang.python at pearwood.info Wed Jul 3 08:22:12 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 03 Jul 2013 12:22:12 GMT Subject: Why is CPython 2.5 a dependency for Jython 2.5? Message-ID: <51d41774$0$29999$c3e8da3$5496439d@news.astraweb.com> I'm running a box with Debian squeeze, and I just ran: sudo aptitude install jython which ended up installing Python 2.5: [...] Linking and byte-compiling packages for runtime python2.5... Setting up python2.5 (2.5.5-11) ... Does anyone know why CPython 2.5 is a dependency for Jython 2.5.1+ on Debian squeeze? -- Steven From rustompmody at gmail.com Wed Jul 3 08:38:05 2013 From: rustompmody at gmail.com (rusi) Date: Wed, 3 Jul 2013 05:38:05 -0700 (PDT) Subject: Why is CPython 2.5 a dependency for Jython 2.5? In-Reply-To: <51d41774$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d41774$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Wednesday, July 3, 2013 5:52:12 PM UTC+5:30, Steven D'Aprano wrote: > Does anyone know why CPython 2.5 is a dependency for Jython 2.5.1+ on > Debian squeeze? Not exactly answering your question... The debian dependencies can be fairly 'conservative' which means all kinds of stuff is pulled in 'just-in-case' eg Ive seen aptitude build-dep pull in a ridiculous amount of doc-stuff like latex (whose size is of the order of GBs) probably because some document format may require it. From skip at pobox.com Wed Jul 3 08:43:46 2013 From: skip at pobox.com (Skip Montanaro) Date: Wed, 3 Jul 2013 07:43:46 -0500 Subject: Why is CPython 2.5 a dependency for Jython 2.5? In-Reply-To: <51d41774$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d41774$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: > Does anyone know why CPython 2.5 is a dependency for Jython 2.5.1+ on > Debian squeeze? Might Jython use some Python modules/packages unmodified? Does sys.path in Jython refer to the CPython tree? Skip From steve+comp.lang.python at pearwood.info Wed Jul 3 08:48:34 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 03 Jul 2013 12:48:34 GMT Subject: Why is CPython 2.5 a dependency for Jython 2.5? References: <51d41774$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51d41da2$0$29999$c3e8da3$5496439d@news.astraweb.com> On Wed, 03 Jul 2013 07:43:46 -0500, Skip Montanaro wrote: >> Does anyone know why CPython 2.5 is a dependency for Jython 2.5.1+ on >> Debian squeeze? > > Might Jython use some Python modules/packages unmodified? Does sys.path > in Jython refer to the CPython tree? Apparently not: >>> sys.path ['', '/usr/lib/site-python', '/usr/share/jython/Lib', '__classpath__', '__pyclasspath__/', '/usr/share/jython/Lib/site-packages'] -- Steven From rustompmody at gmail.com Wed Jul 3 08:55:39 2013 From: rustompmody at gmail.com (rusi) Date: Wed, 3 Jul 2013 05:55:39 -0700 (PDT) Subject: Why is CPython 2.5 a dependency for Jython 2.5? In-Reply-To: <51d41774$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d41774$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <04908475-5353-4322-97c7-7e819a6c28b6@googlegroups.com> On Wednesday, July 3, 2013 5:52:12 PM UTC+5:30, Steven D'Aprano wrote: > I'm running a box with Debian squeeze, and I just ran: > sudo aptitude install jython > which ended up installing Python 2.5: BTW trying to install jython out here gave me this list (which does not seem to have this dependency): $ sudo aptitude install jython The following NEW packages will be installed: antlr3{a} ca-certificates-java{a} default-jdk{a} default-jre{a} default-jre-headless{a} icedtea-6-jre-cacao{a} icedtea-6-jre-jamvm{a} icedtea-netx{a} icedtea-netx-common{a} jython libantlr-java{a} libasm4-java{a} libatk-wrapper-java{a} libatk-wrapper-java-jni{a} libconstantine-java{a} libguava-java{a} libjaffl-java{a} libjffi-java{a} libjffi-jni{a} libjnr-netdb-java{a} libjnr-posix-java{a} libjnr-x86asm-java{a} libjsr305-java{a} liblivetribe-jsr223-java{a} libreadline-java{a} libstringtemplate-java{a} openjdk-6-jdk{a} openjdk-6-jre{a} openjdk-6-jre-headless{a} openjdk-6-jre-lib{a} tzdata-java{a} $ cat /etc/debian_version jessie/sid [And python2.5 is not installed] From rosuav at gmail.com Wed Jul 3 10:07:15 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Jul 2013 00:07:15 +1000 Subject: Why is CPython 2.5 a dependency for Jython 2.5? In-Reply-To: <04908475-5353-4322-97c7-7e819a6c28b6@googlegroups.com> References: <51d41774$0$29999$c3e8da3$5496439d@news.astraweb.com> <04908475-5353-4322-97c7-7e819a6c28b6@googlegroups.com> Message-ID: On Wed, Jul 3, 2013 at 10:55 PM, rusi wrote: > On Wednesday, July 3, 2013 5:52:12 PM UTC+5:30, Steven D'Aprano wrote: >> I'm running a box with Debian squeeze, and I just ran: >> sudo aptitude install jython >> which ended up installing Python 2.5: > > BTW trying to install jython out here gave me this list > (which does not seem to have this dependency): > > $ sudo aptitude install jython > The following NEW packages will be installed: > antlr3{a} ca-certificates-java{a} default-jdk{a} default-jre{a} > default-jre-headless{a} icedtea-6-jre-cacao{a} icedtea-6-jre-jamvm{a} > icedtea-netx{a} icedtea-netx-common{a} jython libantlr-java{a} > libasm4-java{a} libatk-wrapper-java{a} libatk-wrapper-java-jni{a} > libconstantine-java{a} libguava-java{a} libjaffl-java{a} libjffi-java{a} > libjffi-jni{a} libjnr-netdb-java{a} libjnr-posix-java{a} > libjnr-x86asm-java{a} libjsr305-java{a} liblivetribe-jsr223-java{a} > libreadline-java{a} libstringtemplate-java{a} openjdk-6-jdk{a} > openjdk-6-jre{a} openjdk-6-jre-headless{a} openjdk-6-jre-lib{a} > tzdata-java{a} > > $ cat /etc/debian_version > jessie/sid > > [And python2.5 is not installed] Python 2.5 may not be installed (it's no longer available on Wheezy, so it wouldn't surprise me if Jessie also doesn't have it), but quite possibly there's another Python available. On my Wheezy system, I already have Python 2.7 installed, and apt-getting jython doesn't ask to install any more Pythons; but 'apt-cache show jython' doesn't suggest anything that'd be pulling in Python, and I don't have a handy way to poke around and find what would be pulling that in. ChrisA From ifelsetrue at gmail.com Wed Jul 3 09:25:57 2013 From: ifelsetrue at gmail.com (ifelsetrue at gmail.com) Date: Wed, 3 Jul 2013 06:25:57 -0700 (PDT) Subject: python.exe crash if opencv tries to access busy webcam Message-ID: Hello, I have a while loop taking images every 5 minutes from webcam. Unfortunately, if the camera is busy, python.exe crashes and there is no exception to catch. Is there a way to check if camera is busy to avoid the crash? Thanks! from cv2 import * while True: time.sleep(4) cam = VideoCapture(0) s, img = cam.read() if s: # frame captured without any errors namedWindow("cam-test",CV_WINDOW_AUTOSIZE) imshow("cam-test",img) waitKey(0) destroyWindow("cam-test") imwrite("filename.jpg",img) #save image From mail at timgolden.me.uk Wed Jul 3 09:38:46 2013 From: mail at timgolden.me.uk (Tim Golden) Date: Wed, 03 Jul 2013 14:38:46 +0100 Subject: python.exe crash if opencv tries to access busy webcam In-Reply-To: References: Message-ID: <51D42966.6060306@timgolden.me.uk> On 03/07/2013 14:25, ifelsetrue at gmail.com wrote: > Hello, > > > I have a while loop taking images every 5 minutes from webcam. > Unfortunately, if the camera is busy, python.exe crashes and there is > no exception to catch. Is there a way to check if camera is busy to > avoid the crash? If python.exe crashes -- ie exits completely without a traceback, then it's most likely a flaw in the extension module which is capturing the image. If that's the case, there's nothing you can do in Python code to prevent it; you need to work with the project maintainers to fix that. If you're getting a traceback but the console window is then closing too fast for you to see it, then try running the code from an open console window rather than double-clicking on it. (You don't say how you're kicking the code off so I'm offering this just in case) TJG From steve+comp.lang.python at pearwood.info Wed Jul 3 09:29:14 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 03 Jul 2013 13:29:14 GMT Subject: [SPOILERS] Python easter eggs Message-ID: <51d4272a$0$29999$c3e8da3$5496439d@news.astraweb.com> Most people are familiar with: import this and sometimes even with: from __future__ import braces But I'm aware of at least three more. Anyone care to give them? -- Steven From ian at feete.org Wed Jul 3 10:03:14 2013 From: ian at feete.org (Ian Foote) Date: Wed, 03 Jul 2013 15:03:14 +0100 Subject: [SPOILERS] Python easter eggs In-Reply-To: <51d4272a$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d4272a$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51D42F22.7060300@feete.org> On 03/07/13 14:29, Steven D'Aprano wrote: > Most people are familiar with: > > import this > > > and sometimes even with: > > from __future__ import braces > > > But I'm aware of at least three more. Anyone care to give them? > > import antigravity Regards, Ian F From rosuav at gmail.com Wed Jul 3 10:11:54 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Jul 2013 00:11:54 +1000 Subject: [SPOILERS] Python easter eggs In-Reply-To: <51D42F22.7060300@feete.org> References: <51d4272a$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D42F22.7060300@feete.org> Message-ID: On Thu, Jul 4, 2013 at 12:03 AM, Ian Foote wrote: > import antigravity Having checked its docstring, I am now left wondering what I can do with the Munroe geohashing algorithm and if there's any way I could use that at work somehow. ChrisA From ian.g.kelly at gmail.com Wed Jul 3 12:14:30 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 3 Jul 2013 10:14:30 -0600 Subject: [SPOILERS] Python easter eggs In-Reply-To: <51D42F22.7060300@feete.org> References: <51d4272a$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D42F22.7060300@feete.org> Message-ID: On Wed, Jul 3, 2013 at 8:03 AM, Ian Foote wrote: > On 03/07/13 14:29, Steven D'Aprano wrote: >> >> Most people are familiar with: >> >> import this >> >> >> and sometimes even with: >> >> from __future__ import braces >> >> >> But I'm aware of at least three more. Anyone care to give them? >> >> > > import antigravity from __future__ import barry_as_FLUFL import __hello__ From ian.g.kelly at gmail.com Wed Jul 3 12:15:22 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 3 Jul 2013 10:15:22 -0600 Subject: [SPOILERS] Python easter eggs In-Reply-To: References: <51d4272a$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D42F22.7060300@feete.org> Message-ID: On Wed, Jul 3, 2013 at 10:14 AM, Ian Kelly wrote: > On Wed, Jul 3, 2013 at 8:03 AM, Ian Foote wrote: >> On 03/07/13 14:29, Steven D'Aprano wrote: >>> >>> Most people are familiar with: >>> >>> import this >>> >>> >>> and sometimes even with: >>> >>> from __future__ import braces >>> >>> >>> But I'm aware of at least three more. Anyone care to give them? >>> >>> >> >> import antigravity > > from __future__ import barry_as_FLUFL > import __hello__ And in Jython there's: from __future__ import GIL From steve+comp.lang.python at pearwood.info Wed Jul 3 13:22:56 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 03 Jul 2013 17:22:56 GMT Subject: [SPOILERS] Python easter eggs References: <51d4272a$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D42F22.7060300@feete.org> Message-ID: <51d45def$0$29999$c3e8da3$5496439d@news.astraweb.com> On Wed, 03 Jul 2013 10:15:22 -0600, Ian Kelly wrote: > And in Jython there's: > > from __future__ import GIL Nice one! I didn't know about that! -- Steven From ben+python at benfinney.id.au Wed Jul 3 20:59:13 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 04 Jul 2013 10:59:13 +1000 Subject: [SPOILERS] Python easter eggs References: <51d4272a$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D42F22.7060300@feete.org> Message-ID: <7wehbf16ou.fsf@benfinney.id.au> Ian Kelly writes: > from __future__ import barry_as_FLUFL Only works in Python 3 (raises a SyntaxError in Python 2). > import __hello__ Different between Python 2 and Python 3 ? try it in both! -- \ ?I spent all my money on a FAX machine. Now I can only FAX | `\ collect.? ?Steven Wright | _o__) | Ben Finney From wuwei23 at gmail.com Wed Jul 3 21:35:40 2013 From: wuwei23 at gmail.com (alex23) Date: Thu, 04 Jul 2013 11:35:40 +1000 Subject: [SPOILERS] Python easter eggs In-Reply-To: References: <51d4272a$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D42F22.7060300@feete.org> Message-ID: On 4/07/2013 10:59 AM, Ben Finney wrote: > Ian Kelly writes: >> import __hello__ > > Different between Python 2 and Python 3 ? try it in both! Is it meant to imply that Py2 is weary while Py3 is still enthusiastic, or is that just my personal take? :) From akshay.ksth at gmail.com Wed Jul 3 18:05:11 2013 From: akshay.ksth at gmail.com (HighBeliever) Date: Wed, 3 Jul 2013 15:05:11 -0700 (PDT) Subject: Importing modules into IronPython Message-ID: Hi, I have to shift a Python 2.7 program to run in Windows. Doing that has forced me to use IronPython because my program is dependent on a .dll file that uses .NET framework. I moved all my code to Iron Python and modified it to work with the dll. But I cant import PyQt4 module into the project. Is there a way I can do that? Please Help. From benjamin.kaplan at case.edu Wed Jul 3 19:04:05 2013 From: benjamin.kaplan at case.edu (Benjamin Kaplan) Date: Wed, 3 Jul 2013 16:04:05 -0700 Subject: Importing modules into IronPython In-Reply-To: References: Message-ID: On Wed, Jul 3, 2013 at 3:05 PM, HighBeliever wrote: > Hi, I have to shift a Python 2.7 program to run in Windows. Doing that has forced me to use IronPython because my program is dependent on a .dll file that uses .NET framework. > > I moved all my code to Iron Python and modified it to work with the dll. But I cant import PyQt4 module into the project. Is there a way I can do that? Please Help. > -- Generally, extensions that aren't pure Python can only be used with the interpreter they were designed for. There has been some attempt to get CPython extensions working under IronPython (there's a tracking issue on IronPython's bug tracker- http://ironpython.codeplex.com/workitem/11333), but I don't know how well it works. From jcasale at activenetwerx.com Wed Jul 3 18:09:47 2013 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Wed, 3 Jul 2013 22:09:47 +0000 Subject: Decorator help Message-ID: I have a set of methods which take args that I decorate twice, def wrapped(func): def wrap(*args, **kwargs): try: val = func(*args, **kwargs) # some work except BaseException as error: log.exception(error) return [] return wrap def wrapped_again(length): def something(func): def wrapped_func(*args, **kwargs): values = func(*args, **kwargs) # do some work return values return wrapped_func return something So the methods wrapped are as follows: @wrapped_again(12) @wrapped def class_method(self, **kwargs): #.... Is it possible to get the name of the original method (class_method) from within wrapped_func inside wrapped_again? Thanks! jlc From joshua.landau.ws at gmail.com Wed Jul 3 18:19:02 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Wed, 3 Jul 2013 23:19:02 +0100 Subject: Decorator help In-Reply-To: References: Message-ID: On 3 July 2013 23:09, Joseph L. Casale wrote: > I have a set of methods which take args that I decorate twice, > > def wrapped(func): > def wrap(*args, **kwargs): > try: > val = func(*args, **kwargs) > # some work > except BaseException as error: > log.exception(error) > return [] > return wrap > > def wrapped_again(length): > def something(func): > def wrapped_func(*args, **kwargs): > values = func(*args, **kwargs) > # do some work > return values > return wrapped_func > return something > > So the methods wrapped are as follows: > > @wrapped_again(12) > @wrapped > def class_method(self, **kwargs): > #.... > > Is it possible to get the name of the original method (class_method) from within wrapped_func inside wrapped_again? > Thanks! Normally you'd want to use functools.wraps; def wrapped(func): @functools.wraps def wrap(*args, **kwargs): ... return wrap def wrapped_again(length): @functools.wraps def something(func): ... return something @wrapped_again(12) @wrapped def class_method(self, **kwargs): .... And then the name is "carried", as with docstrings. If you don't want to do that, you'd need to use introspection of a remarkably hacky sort. If you want that, well, it'll take a mo. From joshua.landau.ws at gmail.com Wed Jul 3 18:45:19 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Wed, 3 Jul 2013 23:45:19 +0100 Subject: Decorator help In-Reply-To: References: Message-ID: On 3 July 2013 23:19, Joshua Landau wrote: > If you don't want to do that, you'd need to use introspection of a > remarkably hacky sort. If you want that, well, it'll take a mo. After some effort I'm pretty confident that the hacky way is impossible. From jcasale at activenetwerx.com Wed Jul 3 18:54:33 2013 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Wed, 3 Jul 2013 22:54:33 +0000 Subject: Decorator help In-Reply-To: References: , Message-ID: >> If you don't want to do that, you'd need to use introspection of a >> remarkably hacky sort. If you want that, well, it'll take a mo. > > After some effort I'm pretty confident that the hacky way is impossible. Hah, I fired it in PyCharm's debugger and spent a wack time myself, thanks for the confirmation, I'll give functools a shot. Thanks a lot, jlc From __peter__ at web.de Thu Jul 4 01:39:55 2013 From: __peter__ at web.de (Peter Otten) Date: Thu, 04 Jul 2013 07:39:55 +0200 Subject: Decorator help References: Message-ID: Joshua Landau wrote: > On 3 July 2013 23:19, Joshua Landau wrote: >> If you don't want to do that, you'd need to use introspection of a >> remarkably hacky sort. If you want that, well, it'll take a mo. > > After some effort I'm pretty confident that the hacky way is impossible. Well, technically it's func.func_closure[0].cell_contents.__name__ but of course you cannot know that for the general case. From jcasale at activenetwerx.com Thu Jul 4 12:07:49 2013 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Thu, 4 Jul 2013 16:07:49 +0000 Subject: Decorator help In-Reply-To: References: , Message-ID: >Well, technically it's > >func.func_closure[0].cell_contents.__name__ > >but of course you cannot know that for the general case. Hah, I admit I lacked perseverance in looking at this in PyCharms debugger as I missed that. Much appreciated! jlc From joshua.landau.ws at gmail.com Thu Jul 4 21:14:02 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Fri, 5 Jul 2013 02:14:02 +0100 Subject: Decorator help In-Reply-To: References: Message-ID: On 4 July 2013 06:39, Peter Otten <__peter__ at web.de> wrote: > Joshua Landau wrote: > >> On 3 July 2013 23:19, Joshua Landau wrote: >>> If you don't want to do that, you'd need to use introspection of a >>> remarkably hacky sort. If you want that, well, it'll take a mo. >> >> After some effort I'm pretty confident that the hacky way is impossible. > > Well, technically it's > > func.func_closure[0].cell_contents.__name__ > > but of course you cannot know that for the general case. I didn't want to do something like that as it implies a lot of knowledge about the function -- which implies that there's no reason to do it hacky in the first place. I was using "inspect.getclosurevars(func).nonlocals" and that "coerces" to a dictionary first. It's the "correct" way of doing things. But you never know what name the function inside the wrapper is bound to, so I didn't accept that. Also, your method has undefined behaviour AFAIK -- the order of func_closure is compiler-dependant. If you want to do something like this, I recommend my method (but it doesn't work for the general case in the slightest): inspect.getclosurevars(func).nonlocals["func"].__name__ If you can't assume the name it's stored in, but you can know the order of closure variables *then* use Peter's. But again, don't use either; it's impossible just as I said. From ollietempleman at aol.com Wed Jul 3 18:38:39 2013 From: ollietempleman at aol.com (ollietempleman at aol.com) Date: Wed, 3 Jul 2013 15:38:39 -0700 (PDT) Subject: socket data sending problem Message-ID: im trying to do a simple socket test program for a school project using the socket module, but im having difficulty in sending data between the client and host program. so far all tutorials and examples have used something along the lines of: s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) host = socket.gethostname() port = 12345 s.connect((host, port)) and received it on the server end with: s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) host = '' port = 12345 s.bind((host, port)) s.listen(1) conn, addr = s.accept() print ('client is at', addr) data = conn.recv(5) print(data) it all works fine, except for when i try to use: s.send("hello") to send data between the client and server, i just get this error message: >>> Traceback (most recent call last): File "C:/Users/Ollie/Documents/code/chatroom/client3.py", line 9, in s.send("hello") TypeError: 'str' does not support the buffer interface >>> if anyone can either show me what im doing wrong, what this means and what's causing it, or even better how to fix it it would be greatly appreciated many thanks Ollie From irmen.NOSPAM at xs4all.nl Wed Jul 3 18:57:34 2013 From: irmen.NOSPAM at xs4all.nl (Irmen de Jong) Date: Thu, 04 Jul 2013 00:57:34 +0200 Subject: socket data sending problem In-Reply-To: References: Message-ID: <51d4ac5d$0$15982$e4fe514c@news.xs4all.nl> On 4-7-2013 0:38, ollietempleman at aol.com wrote: > it all works fine, except for when i try to use: > > s.send("hello") > > to send data between the client and server, i just get this error message: > > >>> > Traceback (most recent call last): > File "C:/Users/Ollie/Documents/code/chatroom/client3.py", line 9, in > s.send("hello") > TypeError: 'str' does not support the buffer interface > >>> Are you using Python 3.x? Try sending bytes instead of strings: s.send(b"hello") Or switch to using Python 2.7 Irmen From python at mrabarnett.plus.com Wed Jul 3 19:02:21 2013 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 04 Jul 2013 00:02:21 +0100 Subject: socket data sending problem In-Reply-To: References: Message-ID: <51D4AD7D.807@mrabarnett.plus.com> On 03/07/2013 23:38, ollietempleman at aol.com wrote: > im trying to do a simple socket test program for a school project using the socket module, but im having difficulty in sending data between the client and host program. > > so far all tutorials and examples have used something along the lines of: > > s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) > host = socket.gethostname() > port = 12345 > s.connect((host, port)) > > > and received it on the server end with: > > s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) > host = '' > port = 12345 > s.bind((host, port)) > s.listen(1) > conn, addr = s.accept() > print ('client is at', addr) > data = conn.recv(5) > print(data) > > it all works fine, except for when i try to use: > > s.send("hello") > > to send data between the client and server, i just get this error message: > > >>> > Traceback (most recent call last): > File "C:/Users/Ollie/Documents/code/chatroom/client3.py", line 9, in > s.send("hello") > TypeError: 'str' does not support the buffer interface > >>> > > if anyone can either show me what im doing wrong, what this means and what's causing it, or even better how to fix it it would be greatly appreciated > You didn't say which version of Python you're using, but I think that you're using Python 3. A socket handles bytes, not Unicode strings, so you need to encode the Unicode strings to bytes before sending, and decode the bytes to Unicode strings after receiving. From steve+comp.lang.python at pearwood.info Wed Jul 3 23:27:25 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 04 Jul 2013 03:27:25 GMT Subject: Default scope of variables Message-ID: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> Recently, there was a thread where people discussed variable declarations, with a couple people stating that they wished that Python required you to declare local variables, instead of globals. I'm sure they have their (foolish, pathetic) *wink* reasons for this, but I thought I'd explain why I think Python makes the right decision to require declarations for globals but not locals. I was reading some Javascript today, and wading through masses of code that looked something like this: function update_results() { var n1 = foo; var n2 = bar; var m1 = foobar; var m2 = foobarbaz; var fe = n1 * m1; var fi = n2 * m2; var fo = n1 * m2; var fum = n2 * m1; ... } and so on for about a page and a half. Looking at that page in my editor, with that solid column of bold "var" keywords, it struck me just how redundant that enormous column of "var"s was. Of course they were variables, what else would they be? Larry Wall, the creator of Perl, is fond of discussing Huffman coding as it relates to programming syntax. Common things should be short, rare things can be longer. Wall is not concerned about saving a few bytes in your source files, but about programmer effort: typing effort, reading effort, mental processing effort. Which do you prefer? total = subtotal + extra set total to subtotal plus extra Even though the second is only 8 more characters, I bet you prefer the first version. With respect to the Huffman coding of declarations, Javascript gets it backwards. Locals ought to be more common, but they require more typing. Locals are safer, better, more desirable than globals, and so it should be easier to use locals than globals, not the other way around. Having to declare "give me the safe kind of variable", while getting the harmful[1] kind of variable for free, strikes me as arse-backwards. Lazy, naive or careless coders get globals[2] by default or accident. That's bad. Not just theoretically bad. Here's a real-world case where a single missed "var" lead to something much, much worse than a crash: code that kept going, but doing the wrong thing. http://blog.safeshepherd.com/23/how-one-missing-var-ruined-our-launch/ The two situations: 1) Accidentally scope an intended local as global; 2) Accidentally scope an intended global as local; are not symmetrical. In the first case, you get multiple invocations of your function overwriting each other's data. Confusion reigns, but the function calls will likely continue, pumping out garbage results instead of crashing. The likely result is typically fail-unsafe rather than fail- safe. [Aside: fail-safe does not mean "safe from failing", but "fails in a safe manner".] In the second case, any failure is far more likely to result in the function call failing hard with an exception (fail-safe) rather than churning out bad results, since each call of the function gets its own set of locals rather than using those from some other call. So in Javascript, it's easy to get unsafe globals by accident; in Python, it's hard to get unsafe globals by accident. In my opinion, Python gets it right. [1] As in, "Global variables considered harmful", one of the classic papers of computer science: http://c2.com/cgi/wiki?GlobalVariablesConsideredHarmful [2] Actually, Javascript gives you something a little closer to Python's "nonlocal" by default: each enclosing function is searched for a matching variable, terminating at the global scope. -- Steven From rosuav at gmail.com Thu Jul 4 00:07:55 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Jul 2013 14:07:55 +1000 Subject: Default scope of variables In-Reply-To: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, Jul 4, 2013 at 1:27 PM, Steven D'Aprano wrote: > With respect to the Huffman coding of declarations, Javascript gets it > backwards. Locals ought to be more common, but they require more typing. > Locals are safer, better, more desirable than globals, and so it should > be easier to use locals than globals, not the other way around. Having to > declare "give me the safe kind of variable", while getting the harmful[1] > kind of variable for free, strikes me as arse-backwards. Lazy, naive or > careless coders get globals[2] by default or accident. That's bad. I agree that Javascript has it wrong, but not quite for the reason you say. The advantage of declaring locals is a flexibility: you can have multiple unique variables with the same name in the same function. Python lets you do that across but not within functions. But Javascript/ECMAScript/whatever doesn't give you that. A var declaration makes it function-local, no matter where the declaration is. That's pointless. C++, on the other hand, lets you do this: void somefunc() { for (int i=0;i<10;++i) { // do something with outer i for (int i=0;i<4;++i) { // do something with inner i } // outer i is visible again } // neither i is visible } Also, C++ overlays the "this is local" declaration with the "this contains this type" declaration, which neither Python nor Javascript bothers with; that makes the declaration feel less redundant. Granted, this flexibility is mostly of value when writing huge functions with complex nesting, but it is something that I find convenient. In terms of Huffman coding, every C++ variable must be declared, somewhere. It's not a matter of declaring globals or declaring locals - you just declare variables. If you declare them at file scope, they're globals; if at function scope, they're locals. There's really no difference. Everything's visible at its own level and those further in, and not those further out. I do see the convenience of the Python system, and I do like it; but someone needs to speak up for the foolish and pathetic :) ChrisA From joshua.landau.ws at gmail.com Thu Jul 4 00:30:03 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Thu, 4 Jul 2013 05:30:03 +0100 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 4 July 2013 05:07, Chris Angelico wrote: > On Thu, Jul 4, 2013 at 1:27 PM, Steven D'Aprano > wrote: >> With respect to the Huffman coding of declarations, Javascript gets it >> backwards. Locals ought to be more common, but they require more typing. >> Locals are safer, better, more desirable than globals, and so it should >> be easier to use locals than globals, not the other way around. Having to >> declare "give me the safe kind of variable", while getting the harmful[1] >> kind of variable for free, strikes me as arse-backwards. Lazy, naive or >> careless coders get globals[2] by default or accident. That's bad. > > I agree that Javascript has it wrong, but not quite for the reason you > say. The advantage of declaring locals is a flexibility: you can have > multiple unique variables with the same name in the same function. > Python lets you do that across but not within functions. > > But Javascript/ECMAScript/whatever doesn't give you that. A var > declaration makes it function-local, no matter where the declaration > is. Coffeescript, which compiles to Javascript, "fixes" the problem Steven brought up by automatically declaring variables so that you don't have to. But what do you think this does?: a = 1 func = -> a = 2 b = 2 The "a" in "func" is global, the "b" is local. And Coffeescript *doesn't let* you shadow even if you explicitly want to. There just isn't syntax for it. That said, I'm not too convinced. Personally, the proper way to do what you are talking about is creating a new closure. Like: for i in range(100): with new_scope(): for i in range(100): func(i) func(i) # Using i from original loop But it's not like Python'll ever support that. From rosuav at gmail.com Thu Jul 4 00:36:20 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Jul 2013 14:36:20 +1000 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, Jul 4, 2013 at 2:30 PM, Joshua Landau wrote: > That said, I'm not too convinced. Personally, the proper way to do > what you are talking about is creating a new closure. Like: > > for i in range(100): > with new_scope(): > for i in range(100): > func(i) > func(i) # Using i from original loop > > But it's not like Python'll ever support that. > def foo(): for i in range(3): print("outer",i) def inner(): for i in range(4): print("inner",i) inner() print("outer",i) That works, but you then have to declare all your nonlocals, and it hardly reads well. ChrisA From joshua.landau.ws at gmail.com Thu Jul 4 01:09:04 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Thu, 4 Jul 2013 06:09:04 +0100 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 4 July 2013 05:36, Chris Angelico wrote: > On Thu, Jul 4, 2013 at 2:30 PM, Joshua Landau > wrote: >> That said, I'm not too convinced. Personally, the proper way to do >> what you are talking about is creating a new closure. Like: >> >> for i in range(100): >> with new_scope(): >> for i in range(100): >> func(i) >> func(i) # Using i from original loop >> >> But it's not like Python'll ever support that. >> > > def foo(): > for i in range(3): > print("outer",i) > def inner(): > for i in range(4): > print("inner",i) > inner() > print("outer",i) > > That works, but you then have to declare all your nonlocals, and it > hardly reads well. Unfortunately that's what people, I included, end up doing. Stuff like: def paranoia(...): def safe_recursive(...): safe_recursive(...) return safe_recursive safe_recursive = paranoia() is blimmin ugly. Then you're only really left with class safe_recursive: def __call__(self, ...): self(...) which only solves it for recursive functions. I guess this means I actually agree with your sentiment, just not the specifics. From joshua.landau.ws at gmail.com Mon Jul 8 12:58:16 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Mon, 8 Jul 2013 17:58:16 +0100 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 4 July 2013 05:36, Chris Angelico wrote: > On Thu, Jul 4, 2013 at 2:30 PM, Joshua Landau > wrote: >> That said, I'm not too convinced. Personally, the proper way to do >> what you are talking about is creating a new closure. Like: >> >> for i in range(100): >> with new_scope(): >> for i in range(100): >> func(i) >> func(i) # Using i from original loop >> >> But it's not like Python'll ever support that. >> > > def foo(): > for i in range(3): > print("outer",i) > def inner(): > for i in range(4): > print("inner",i) > inner() > print("outer",i) > > That works, but you then have to declare all your nonlocals, and it > hardly reads well. Stealing concepts shamelessly from http://www.slideshare.net/r1chardj0n3s/dont-do-this-24000445, you can do this: import inspect from contextlib import contextmanager @contextmanager def scope(namespace=None): old_names = inspect.currentframe().f_back.f_back.f_locals.copy() yield names = inspect.currentframe().f_back.f_back.f_locals if namespace is not None: new_names = {k:v for k, v in names.items() if k not in old_names and v is not namespace} namespace.update(**new_names) names.clear() names.update(old_names) So you *can* do: >>> for i in range(3): ... with scope(): ... for i in range(3): ... print("Inner:", i) ... print("Outer", i) Inner: 0 Inner: 1 Inner: 2 Outer 0 Inner: 0 Inner: 1 Inner: 2 Outer 1 Inner: 0 Inner: 1 Inner: 2 Outer 2 :) If you pass scope() a dictionary, all the new variables will get added to it. From steve+comp.lang.python at pearwood.info Thu Jul 4 01:45:11 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 04 Jul 2013 05:45:11 GMT Subject: Default scope of variables References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51d50be7$0$6512$c3e8da3$5496439d@news.astraweb.com> On Thu, 04 Jul 2013 05:30:03 +0100, Joshua Landau wrote: > That said, I'm not too convinced. Personally, the proper way to do what > you are talking about is creating a new closure. Like: > > for i in range(100): > with new_scope(): > for i in range(100): > func(i) > func(i) # Using i from original loop > > But it's not like Python'll ever support that. Probably not, but Python does support this: for i in range(100): for j in range(100): func(j) func(i) # Using i from original loop which solves the problem of inner i overwriting outer i nicely. -- Steven From steve+comp.lang.python at pearwood.info Thu Jul 4 01:32:29 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 04 Jul 2013 05:32:29 GMT Subject: Default scope of variables References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> On Thu, 04 Jul 2013 14:07:55 +1000, Chris Angelico wrote: > On Thu, Jul 4, 2013 at 1:27 PM, Steven D'Aprano > wrote: >> With respect to the Huffman coding of declarations, Javascript gets it >> backwards. Locals ought to be more common, but they require more >> typing. Locals are safer, better, more desirable than globals, and so >> it should be easier to use locals than globals, not the other way >> around. Having to declare "give me the safe kind of variable", while >> getting the harmful[1] kind of variable for free, strikes me as >> arse-backwards. Lazy, naive or careless coders get globals[2] by >> default or accident. That's bad. > > I agree that Javascript has it wrong, but not quite for the reason you > say. The advantage of declaring locals is a flexibility: you can have > multiple unique variables with the same name in the same function. Well, if I ever have more than 63,000,000 variables[1] in a function, I'll keep that in mind. Until then, I'm pretty sure you can trivially avoid name clashes with globals that you wish to avoid clashing with. Accidental shadowing can be a problem, but I've never heard of anyone saying that they were *forced* to shadow a global they needed access to. Just pick a different name. > Python lets you do that across but not within functions. > > But Javascript/ECMAScript/whatever doesn't give you that. A var > declaration makes it function-local, no matter where the declaration is. > That's pointless. C++, on the other hand, lets you do this: > > void somefunc() { > for (int i=0;i<10;++i) { > // do something with outer i > for (int i=0;i<4;++i) { > // do something with inner i > } > // outer i is visible again > } > // neither i is visible > } That's truly horrible. If the cost of this "flexibility" is that I'll have to read, and debug, other people's code with this sort of thing, I'm happy to be less flexible. For what possible reason other than "because I can" would you want to use the same loop variable name in two nested loops? I'm not suggesting that C++ should prohibit it. But just because a language allows something doesn't make it a *feature*. I can write this in Python: a = a = a = a = a = 1 and it works, but the ability to do so is hardly a feature. It's just a side effect of how Python works. I believe that the function is the right level for scope changes, not to large, not to small. I'm not even sure I like the fact that generator expressions (in Python 2 & 3) and list comprehensions (in Python 3) introduce their own scope. [...] > Granted, this flexibility is mostly of value when writing huge functions > with complex nesting, but it is something that I find convenient. "This feature is mostly of value when poking myself, and any developers who have to maintain my code after I'm gone, in the eye with a red-hot poker." [1] Based on empirical evidence that Python supports names with length at least up to one million characters long, and assuming that each character can be an ASCII letter, digit or underscore. -- Steven From rosuav at gmail.com Thu Jul 4 01:47:57 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Jul 2013 15:47:57 +1000 Subject: Default scope of variables In-Reply-To: <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, Jul 4, 2013 at 3:32 PM, Steven D'Aprano wrote: > Accidental shadowing can be a problem, but I've never heard of anyone > saying that they were *forced* to shadow a global they needed access to. > Just pick a different name. Here's one example of shadowing that comes from a C++ project at work. I have a class that represents a database transaction (constructing it begins a transaction, it has methods for doing queries, and its destructor rolls back). There's also a class for a subtransation (same thing, but it uses savepoints within the transaction). So to bracket a piece of code in a subtransaction, I want to declare a new subtransaction object with the same name as the outer transaction object, and then dispose of it and "reveal" the original. There will always be an object called "trans", and it will always be the appropriate transaction to do queries on, but it'll change what it is. ChrisA From __peter__ at web.de Thu Jul 4 02:48:59 2013 From: __peter__ at web.de (Peter Otten) Date: Thu, 04 Jul 2013 08:48:59 +0200 Subject: Default scope of variables References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: Steven D'Aprano wrote: > Well, if I ever have more than 63,000,000 variables[1] in a function, > I'll keep that in mind. Until then, I'm pretty sure you can trivially > avoid name clashes with globals that you wish to avoid clashing with. > [1] Based on empirical evidence that Python supports names with length at > least up to one million characters long, and assuming that each character > can be an ASCII letter, digit or underscore. That would be 63**10**6. Or 53*63**999999 if I were to nitpick... From ian.g.kelly at gmail.com Thu Jul 4 03:12:23 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Thu, 4 Jul 2013 01:12:23 -0600 Subject: Default scope of variables In-Reply-To: <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Wed, Jul 3, 2013 at 11:32 PM, Steven D'Aprano wrote: >> Python lets you do that across but not within functions. >> >> But Javascript/ECMAScript/whatever doesn't give you that. A var >> declaration makes it function-local, no matter where the declaration is. >> That's pointless. C++, on the other hand, lets you do this: >> >> void somefunc() { >> for (int i=0;i<10;++i) { >> // do something with outer i >> for (int i=0;i<4;++i) { >> // do something with inner i >> } >> // outer i is visible again >> } >> // neither i is visible >> } > > That's truly horrible. If the cost of this "flexibility" is that I'll > have to read, and debug, other people's code with this sort of thing, I'm > happy to be less flexible. For what possible reason other than "because I > can" would you want to use the same loop variable name in two nested > loops? It's interesting to note that while Java and C# also allow reuse of local variable names, they do not allow local variables declared in inner scopes to shadow variables declared in enclosing scopes, as in the example above. But the following would be perfectly legal: void somefunc() { for (int i = 0; i < a.size; ++i) { // do something with a[i] } for (int i = 0; i < b.size; ++i) { // do something with b[i] } } And the two i's are treated as completely separate variables here, as arguably they should be since they're used for two distinct purposes. From davea at davea.name Thu Jul 4 03:06:25 2013 From: davea at davea.name (Dave Angel) Date: Thu, 04 Jul 2013 03:06:25 -0400 Subject: Default scope of variables In-Reply-To: <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 07/04/2013 01:32 AM, Steven D'Aprano wrote: > > > Well, if I ever have more than 63,000,000 variables[1] in a function, > I'll keep that in mind. > > > [1] Based on empirical evidence that Python supports names with length at > least up to one million characters long, and assuming that each character > can be an ASCII letter, digit or underscore. > Well, the number wouldn't be 63,000,000. Rather it'd be 63**1000000 I probably have it wrong, but I think that looks like: 859,122,207,646,748,720,415,212,786,780,258,721,683,540,870,960,267,706,738,947,655,539,422,295,787,680,882,091,181,482,626,114,653,152,637,456,091,641,990,601,474,111,018,521,295,858,424,750,289,461,372,414,431,396,326,232,796,267,104,001 variables. (The number has 180 digits) -- DaveA From lele at metapensiero.it Thu Jul 4 08:43:22 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Thu, 04 Jul 2013 14:43:22 +0200 Subject: Default scope of variables References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: <87wqp6ts0l.fsf@nautilus.nautilus> Dave Angel writes: > Well, the number wouldn't be 63,000,000. Rather it'd be 63**1000000 Uhm, if we are talking about Py2, then you should not count all the combinations starting with a digit, while under Py3 the number explodes, as this is valid code: >>> ? = 1 >>> ? 1 :-) back to easily-enumerable issues, ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From wayne at waynewerner.com Thu Jul 4 11:45:50 2013 From: wayne at waynewerner.com (Wayne Werner) Date: Thu, 4 Jul 2013 10:45:50 -0500 (CDT) Subject: Default scope of variables In-Reply-To: <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, 4 Jul 2013, Steven D'Aprano wrote: > > [1] Based on empirical evidence that Python supports names with length at > least up to one million characters long, and assuming that each character > can be an ASCII letter, digit or underscore. > The specification *does* state unlimited length: http://docs.python.org/release/2.5.2/ref/identifiers.html Though practicality beats purity. -W From steve+comp.lang.python at pearwood.info Thu Jul 4 11:52:22 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 04 Jul 2013 15:52:22 GMT Subject: Default scope of variables References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51d59a36$0$29999$c3e8da3$5496439d@news.astraweb.com> On Thu, 04 Jul 2013 03:06:25 -0400, Dave Angel wrote: > On 07/04/2013 01:32 AM, Steven D'Aprano wrote: >> > >> >> Well, if I ever have more than 63,000,000 variables[1] in a function, >> I'll keep that in mind. >> > >> >> [1] Based on empirical evidence that Python supports names with length >> at least up to one million characters long, and assuming that each >> character can be an ASCII letter, digit or underscore. >> >> > Well, the number wouldn't be 63,000,000. Rather it'd be 63**1000000 > > I probably have it wrong, but I think that looks like: > > 859,122,207,646,748,720,415,212,786,780,258,721,683,540,870,960,267,706,738,947,655,539,422,295,787,680,882,091,181,482,626,114,653,152,637,456,091,641,990,601,474,111,018,521,295,858,424,750,289,461,372,414,431,396,326,232,796,267,104,001 > > variables. (The number has 180 digits) I think that's more than 63,000,000 :-) Thanks Dave and Peter for the correction. -- Steven From sg552 at hotmail.co.uk Thu Jul 4 12:54:20 2013 From: sg552 at hotmail.co.uk (Rotwang) Date: Thu, 04 Jul 2013 17:54:20 +0100 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: Sorry to be OT, but this is sending my pedantry glands haywire: On 04/07/2013 08:06, Dave Angel wrote: > On 07/04/2013 01:32 AM, Steven D'Aprano wrote: >> > >> >> Well, if I ever have more than 63,000,000 variables[1] in a function, >> I'll keep that in mind. >> > >> >> [1] Based on empirical evidence that Python supports names with length at >> least up to one million characters long, and assuming that each character >> can be an ASCII letter, digit or underscore. >> > > Well, the number wouldn't be 63,000,000. Rather it'd be 63**1000000 > > I probably have it wrong, but I think that looks like: > > 859,122,[etc.] > > > variables. (The number has 180 digits) That's 63**100. Note that 10**1000000 has 1000001 digits, and is somewhat smaller than 63**1000000. Anyway, none of the calculations that has been given takes into account the fact that names can be /less/ than one million characters long. The actual number of non-empty strings of length at most 1000000 characters, that consist only of ascii letters, digits or underscores, and that don't start with a digit, is sum(53*63**i for i in range(1000000)) == 53*(63**1000000 - 1)//62 It's perhaps worth mentioning that some non-ascii characters are allowed in identifiers in Python 3, though I don't know which ones. From __peter__ at web.de Thu Jul 4 14:36:16 2013 From: __peter__ at web.de (Peter Otten) Date: Thu, 04 Jul 2013 20:36:16 +0200 Subject: Default scope of variables References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: Rotwang wrote: > Sorry to be OT, but this is sending my pedantry glands haywire: We are mostly pedants, too -- so this is well-deserved... > On 04/07/2013 08:06, Dave Angel wrote: >> On 07/04/2013 01:32 AM, Steven D'Aprano wrote: >>> >> >>> >>> Well, if I ever have more than 63,000,000 variables[1] in a function, >>> I'll keep that in mind. >>> >> >>> >>> [1] Based on empirical evidence that Python supports names with length >>> [at >>> least up to one million characters long, and assuming that each >>> character can be an ASCII letter, digit or underscore. >>> >> >> Well, the number wouldn't be 63,000,000. Rather it'd be 63**1000000 >> >> I probably have it wrong, but I think that looks like: >> >> 859,122,[etc.] >> >> >> variables. (The number has 180 digits) > > That's 63**100. Note that 10**1000000 has 1000001 digits, and is > somewhat smaller than 63**1000000. > > Anyway, none of the calculations that has been given takes into account > the fact that names can be /less/ than one million characters long. I think we have a winner ;) > The > actual number of non-empty strings of length at most 1000000 characters, > that consist only of ascii letters, digits or underscores, and that > don't start with a digit, is > > sum(53*63**i for i in range(1000000)) == 53*(63**1000000 - 1)//62 > It's perhaps worth mentioning that some non-ascii characters are allowed > in identifiers in Python 3, though I don't know which ones. From joshua.landau.ws at gmail.com Thu Jul 4 20:04:32 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Fri, 5 Jul 2013 01:04:32 +0100 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 4 July 2013 17:54, Rotwang wrote: > 53*(63**1000000 - 1)//62 Or about 10**10**6.255 (so about 1.80M digits long). For the unicode side (Python 3, in other words) and reusing your math (ya better hope it's right!), you are talking: 97812*((97812+2020)**1000000 - 1)/(97812+2020-1) Or about 10**10**6.699 Which has about 5.00M digits. Good luck running out. From steve+comp.lang.python at pearwood.info Thu Jul 4 21:24:09 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 05 Jul 2013 01:24:09 GMT Subject: Default scope of variables References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51d62039$0$29999$c3e8da3$5496439d@news.astraweb.com> On Thu, 04 Jul 2013 17:54:20 +0100, Rotwang wrote: [...] > Anyway, none of the calculations that has been given takes into account > the fact that names can be /less/ than one million characters long. Not in *my* code they don't!!! *wink* > The > actual number of non-empty strings of length at most 1000000 characters, > that consist only of ascii letters, digits or underscores, and that > don't start with a digit, is > > sum(53*63**i for i in range(1000000)) == 53*(63**1000000 - 1)//62 I take my hat of to you sir, or possibly madam. That is truly an inspired piece of pedantry. > It's perhaps worth mentioning that some non-ascii characters are allowed > in identifiers in Python 3, though I don't know which ones. PEP 3131 describes the rules: http://www.python.org/dev/peps/pep-3131/ For example: py> import unicodedata as ud py> for c in '???????????': ... print(c, ud.name(c), c.isidentifier(), ud.category(c)) ... ? LATIN SMALL LETTER E WITH ACUTE True Ll ? LATIN SMALL LETTER AE True Ll ? YEN SIGN False Sc ? MICRO SIGN True Ll ? INVERTED QUESTION MARK False Po ? GREEK SMALL LETTER MU True Ll ? CYRILLIC CAPITAL LETTER ZHE True Lu ? OGHAM LETTER FEARN True Lo ? PER MILLE SIGN False Po ? RIGHTWARDS ARROW OVER LEFTWARDS ARROW False So ? INFINITY False Sm -- Steven From davea at davea.name Thu Jul 4 22:03:52 2013 From: davea at davea.name (Dave Angel) Date: Thu, 04 Jul 2013 22:03:52 -0400 Subject: Default scope of variables In-Reply-To: <51d62039$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d62039$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 07/04/2013 09:24 PM, Steven D'Aprano wrote: > On Thu, 04 Jul 2013 17:54:20 +0100, Rotwang wrote: > [...] >> Anyway, none of the calculations that has been given takes into account >> the fact that names can be /less/ than one million characters long. > > > Not in *my* code they don't!!! > > *wink* > > >> The >> actual number of non-empty strings of length at most 1000000 characters, >> that consist only of ascii letters, digits or underscores, and that >> don't start with a digit, is >> >> sum(53*63**i for i in range(1000000)) == 53*(63**1000000 - 1)//62 > > > I take my hat of to you sir, or possibly madam. That is truly an inspired > piece of pedantry. > > >> It's perhaps worth mentioning that some non-ascii characters are allowed >> in identifiers in Python 3, though I don't know which ones. > > PEP 3131 describes the rules: > > http://www.python.org/dev/peps/pep-3131/ > > For example: > > py> import unicodedata as ud > py> for c in '???????????': > ... print(c, ud.name(c), c.isidentifier(), ud.category(c)) > ... > ? LATIN SMALL LETTER E WITH ACUTE True Ll > ? LATIN SMALL LETTER AE True Ll > ? YEN SIGN False Sc > ? MICRO SIGN True Ll > ? INVERTED QUESTION MARK False Po > ? GREEK SMALL LETTER MU True Ll > ? CYRILLIC CAPITAL LETTER ZHE True Lu > ? OGHAM LETTER FEARN True Lo > ? PER MILLE SIGN False Po > ? RIGHTWARDS ARROW OVER LEFTWARDS ARROW False So > ? INFINITY False Sm > > > The isidentifier() method will let you weed out the characters that cannot start an identifier. But there are other groups of characters that can appear after the starting "letter". So a more reasonable sample might be something like: > py> import unicodedata as ud > py> for c in '???????????': > ... xc = "X" + c > ... print(c, ud.name(c), xc.isidentifier(), ud.category(c)) > ... In particular, http://docs.python.org/3.3/reference/lexical_analysis.html#identifiers has a definition for id_continue that includes several interesting categories. I expected the non-ASCII digits, but there's other stuff there, like "nonspacing marks" that are surprising. I'm pretty much speculating here, so please correct me if I'm way off. -- DaveA From joshua.landau.ws at gmail.com Thu Jul 4 22:27:18 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Fri, 5 Jul 2013 03:27:18 +0100 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d62039$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 5 July 2013 03:03, Dave Angel wrote: > On 07/04/2013 09:24 PM, Steven D'Aprano wrote: >> On Thu, 04 Jul 2013 17:54:20 +0100, Rotwang wrote: >>> It's perhaps worth mentioning that some non-ascii characters are allowed >>> in identifiers in Python 3, though I don't know which ones. >> >> PEP 3131 describes the rules: >> >> http://www.python.org/dev/peps/pep-3131/ > > The isidentifier() method will let you weed out the characters that cannot > start an identifier. But there are other groups of characters that can > appear after the starting "letter". So a more reasonable sample might be > something like: ... > In particular, > http://docs.python.org/3.3/reference/lexical_analysis.html#identifiers > > has a definition for id_continue that includes several interesting > categories. I expected the non-ASCII digits, but there's other stuff there, > like "nonspacing marks" that are surprising. > > I'm pretty much speculating here, so please correct me if I'm way off. For my calculation above, I used this code I quickly mocked up: > import unicodedata as unidata > from sys import maxunicode > from collections import defaultdict > from itertools import chain > > def get(): > xid_starts = set() > xid_continues = set() > > id_start_categories = "Lu, Ll, Lt, Lm, Lo, Nl".split(", ") > id_continue_categories = "Mn, Mc, Nd, Pc".split(", ") > > characters = (chr(n) for n in range(maxunicode + 1)) > > print("Making normalized characters") > > normalized = (unidata.normalize("NFKC", character) for character in characters) > normalized = set(chain.from_iterable(normalized)) > > print("Assigning to categories") > > for character in normalized: > category = unidata.category(character) > > if category in id_start_categories: > xid_starts.add(character) > elif category in id_continue_categories: > xid_continues.add(character) > > return xid_starts, xid_continues Please note that "xid_continues" actually represents "xid_continue - xid_start". From joshua.landau.ws at gmail.com Thu Jul 4 22:29:36 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Fri, 5 Jul 2013 03:29:36 +0100 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d62039$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 5 July 2013 03:03, Dave Angel wrote: > In particular, > http://docs.python.org/3.3/reference/lexical_analysis.html#identifiers > > has a definition for id_continue that includes several interesting > categories. I expected the non-ASCII digits, but there's other stuff there, > like "nonspacing marks" that are surprising. "nonspacing marks" are just accents, so it makes sense *?* mon avis. From sg552 at hotmail.co.uk Fri Jul 5 02:28:46 2013 From: sg552 at hotmail.co.uk (Rotwang) Date: Fri, 05 Jul 2013 07:28:46 +0100 Subject: Default scope of variables In-Reply-To: <51d62039$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d62039$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 05/07/2013 02:24, Steven D'Aprano wrote: > On Thu, 04 Jul 2013 17:54:20 +0100, Rotwang wrote: > [...] >> Anyway, none of the calculations that has been given takes into account >> the fact that names can be /less/ than one million characters long. > > > Not in *my* code they don't!!! > > *wink* > > >> The >> actual number of non-empty strings of length at most 1000000 characters, >> that consist only of ascii letters, digits or underscores, and that >> don't start with a digit, is >> >> sum(53*63**i for i in range(1000000)) == 53*(63**1000000 - 1)//62 > > > I take my hat of to you sir, or possibly madam. That is truly an inspired > piece of pedantry. FWIW, I'm male. >> It's perhaps worth mentioning that some non-ascii characters are allowed >> in identifiers in Python 3, though I don't know which ones. > > PEP 3131 describes the rules: > > http://www.python.org/dev/peps/pep-3131/ Thanks. From neilc at norwich.edu Fri Jul 5 09:24:43 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 5 Jul 2013 13:24:43 GMT Subject: Default scope of variables References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2013-07-04, Dave Angel wrote: > On 07/04/2013 01:32 AM, Steven D'Aprano wrote: > >> Well, if I ever have more than 63,000,000 variables[1] in a >> function, I'll keep that in mind. >> > >> >> [1] Based on empirical evidence that Python supports names >> with length at least up to one million characters long, and >> assuming that each character can be an ASCII letter, digit or >> underscore. > > Well, the number wouldn't be 63,000,000. Rather it'd be > 63**1000000 You should really count only the ones somebody might actually want to use. That's a much, much smaller number, though still plenty big. Inner scopes (I don't remember the official name) is a great feature of C++. It's not the right feature for Python, though, since Python doesn't have deterministic destruction. It wouldn't buy much except for namespace tidyness. for x in range(4): print(x) print(x) # Vader NOoooooOOOOOO!!! Python provides deterministic destruction with a different feature. -- Neil Cerutti From rosuav at gmail.com Fri Jul 5 09:43:51 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 5 Jul 2013 23:43:51 +1000 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Fri, Jul 5, 2013 at 11:24 PM, Neil Cerutti wrote: > Python provides deterministic destruction with a different > feature. You mean 'with'? That's not actually destruction, it just does one of the same jobs that deterministic destruction is used for (RAII). It doesn't, for instance, have any influence on memory usage, nor does it ensure the destruction of the object's referents. But yes, it does achieve (one of) the most important role(s) of destruction. ChrisA From neilc at norwich.edu Fri Jul 5 11:36:02 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 5 Jul 2013 15:36:02 GMT Subject: Default scope of variables References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2013-07-05, Chris Angelico wrote: > On Fri, Jul 5, 2013 at 11:24 PM, Neil Cerutti > wrote: >> Python provides deterministic destruction with a different >> feature. > > You mean 'with'? That's not actually destruction, it just does > one of the same jobs that deterministic destruction is used for > (RAII). It doesn't, for instance, have any influence on memory > usage, nor does it ensure the destruction of the object's > referents. But yes, it does achieve (one of) the most important > role(s) of destruction. Yes, thanks. I meant the ability to grab and release a resource deterministically. -- Neil Cerutti From steve+comp.lang.python at pearwood.info Sun Jul 7 12:08:59 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 07 Jul 2013 16:08:59 GMT Subject: Default scope of variables References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51d9929b$0$9505$c3e8da3$5496439d@news.astraweb.com> On Fri, 05 Jul 2013 13:24:43 +0000, Neil Cerutti wrote: > for x in range(4): > print(x) > print(x) # Vader NOoooooOOOOOO!!! That loops do *not* introduce a new scope is a feature, not a bug. It is *really* useful to be able to use the value of x after the loop has finished. That's a much more common need than being able to have an x inside the loop and an x outside the loop. -- Steven From neilc at norwich.edu Mon Jul 8 07:54:28 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 8 Jul 2013 11:54:28 GMT Subject: Default scope of variables References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d9929b$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2013-07-07, Steven D'Aprano wrote: > On Fri, 05 Jul 2013 13:24:43 +0000, Neil Cerutti wrote: > >> for x in range(4): >> print(x) >> print(x) # Vader NOoooooOOOOOO!!! > > That loops do *not* introduce a new scope is a feature, not a bug. It is > *really* useful to be able to use the value of x after the loop has > finished. I don't buy necessarily buy that it's "*really*" useful but I do like introducing new names in (not really the scope of) if/elif/else and for statement blocks. z = record["Zip"] if int(z) > 99999: zip_code = z[:-4].rjust(5, "0") zip4 = z[-4:] else: zip_code = z.rjust(5, "0") zip4 = "" As opposed to: zip_code = None zip4 = None z = record["Zip"] if int(z) > 99999: zip_code = z[:-4].rjust(5, "0") zip4 = z[-4:] else: zip_code = z.rjust(5, "0") zip4 = "" -- Neil Cerutti From joshua.landau.ws at gmail.com Mon Jul 8 09:14:16 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Mon, 8 Jul 2013 14:14:16 +0100 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d9929b$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 8 July 2013 12:54, Neil Cerutti wrote: > On 2013-07-07, Steven D'Aprano wrote: >> On Fri, 05 Jul 2013 13:24:43 +0000, Neil Cerutti wrote: >> >>> for x in range(4): >>> print(x) >>> print(x) # Vader NOoooooOOOOOO!!! >> >> That loops do *not* introduce a new scope is a feature, not a bug. It is >> *really* useful to be able to use the value of x after the loop has >> finished. > > I don't buy necessarily buy that it's "*really*" useful Just take "really" to mean "like, I'm totz not lying". > but I do > like introducing new names in (not really the scope of) > if/elif/else and for statement blocks. > > z = record["Zip"] > if int(z) > 99999: > zip_code = z[:-4].rjust(5, "0") > zip4 = z[-4:] > else: > zip_code = z.rjust(5, "0") > zip4 = "" I'd probably break down and cry if "if"s introduced a new scope in Pythons before the "nonlocal" keyword (assuming current Python semantics where "=" defaults to only the inner-most scope). From steve+comp.lang.python at pearwood.info Thu Jul 4 12:38:28 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 04 Jul 2013 16:38:28 GMT Subject: Default scope of variables References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> On Thu, 04 Jul 2013 15:47:57 +1000, Chris Angelico wrote: > On Thu, Jul 4, 2013 at 3:32 PM, Steven D'Aprano > wrote: >> Accidental shadowing can be a problem, but I've never heard of anyone >> saying that they were *forced* to shadow a global they needed access >> to. Just pick a different name. > > Here's one example of shadowing that comes from a C++ project at work. I > have a class that represents a database transaction (constructing it > begins a transaction, it has methods for doing queries, and its > destructor rolls back). When the object finally gets garbage collected, doesn't that mean the last transaction will be rolled back? > There's also a class for a subtransation (same > thing, but it uses savepoints within the transaction). So to bracket a > piece of code in a subtransaction, I want to declare a new > subtransaction object with the same name as the outer transaction > object, and then dispose of it and "reveal" the original. There will > always be an object called "trans", and it will always be the > appropriate transaction to do queries on, but it'll change what it is. You don't need to introduce such scoping rules for variables for that use- case. We have namespaces (classes) for that sort of thing :-) Python 3.3's ChainMap is probably a better solution, but here's another way to get the same sort of behaviour: def function(): class Namespace: # Set a class attribute. trans = Transaction() ns = Namespace() do_stuff_with(ns.trans) # Enter a subtransaction. ns.trans = Subtransaction() do_stuff_with(ns.trans) del ns.trans do_stuff_with(ns.trans) Yes, it looks weird to see ns.trans used immediately after deleting it, but that's because it is a weird (or at least unusual) use-case. -- Steven From rosuav at gmail.com Thu Jul 4 12:58:50 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 5 Jul 2013 02:58:50 +1000 Subject: Default scope of variables In-Reply-To: <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Fri, Jul 5, 2013 at 2:38 AM, Steven D'Aprano wrote: > On Thu, 04 Jul 2013 15:47:57 +1000, Chris Angelico wrote: >> Here's one example of shadowing that comes from a C++ project at work. I >> have a class that represents a database transaction (constructing it >> begins a transaction, it has methods for doing queries, and its >> destructor rolls back). > > When the object finally gets garbage collected, doesn't that mean the > last transaction will be rolled back? Oh. Uhm... ahh... it would have helped to mention that it also has a commit() method! But yes, that's correct; if the object expires (this is C++, so it's guaranteed to call the destructor at that close brace - none of the Python vagueness about when __del__ is called) without commit() being called, then the transaction will be rolled back. And since this is PostgreSQL we use, the same applies if the process is SIGKILLed or the power fails. If commit() doesn't happen, neither does the transaction. (There are a few actions the program can take that are deliberately non-transactional - log entries of various sorts, mainly - but everything else is guarded in this way.) ChrisA From wayne at waynewerner.com Sun Jul 7 09:13:43 2013 From: wayne at waynewerner.com (Wayne Werner) Date: Sun, 7 Jul 2013 08:13:43 -0500 (CDT) Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Fri, 5 Jul 2013, Chris Angelico wrote: > Oh. Uhm... ahh... it would have helped to mention that it also has a > commit() method! But yes, that's correct; if the object expires (this > is C++, so it's guaranteed to call the destructor at that close brace > - none of the Python vagueness about when __del__ is called) without > commit() being called, then the transaction will be rolled back. If one wants to duplicate this kind of behavior in Python, that's what context managers are for combined with a `with` block, which does guarantee that the __exit__ method will be called - in this case it could be something as simple as: from contextlib import contextmanager @contextmanager def new_transaction(conn): tran = conn.begin_transaction() yield tran if not tran.committed: tran.rollback() Which you would then use like: conn = create_conn() with new_transaction(conn) as tran: rows_affected = do_query_stuff(tran) if rows_affected == 42: tran.commit() And then you get the desired constructor/destructor behavior of having guaranteed that code will be executed at the start and at the end. You can wrap things in try/catch for some error handling, or write your own context manager class. HTH, Wayne From rosuav at gmail.com Sun Jul 7 09:43:24 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 7 Jul 2013 23:43:24 +1000 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sun, Jul 7, 2013 at 11:13 PM, Wayne Werner wrote: > Which you would then use like: > > > conn = create_conn() > with new_transaction(conn) as tran: > rows_affected = do_query_stuff(tran) > if rows_affected == 42: > tran.commit() Yep. There's a problem, though, when you bring in subtransactions. The logic wants to be like this: with new_transaction(conn) as tran: tran.query("blah") with tran.subtransaction() as tran: tran.query("blah") with tran.subtransaction() as tran: tran.query("blah") # roll this subtransaction back tran.query("blah") tran.commit() tran.query("blah") tran.commit() The 'with' statement doesn't allow this. I would need to use some kind of magic to rebind the old transaction to the name, or else use a list that gets magically populated: with new_transaction(conn) as tran: tran[-1].query("blah") with subtransaction(tran): tran[-1].query("blah") with subtransaction(tran): tran[-1].query("blah") # roll this subtransaction back tran[-1].query("blah") tran[-1].commit() tran[-1].query("blah") tran[-1].commit() I don't like the look of this. It might work, but it's hardly ideal. This is why I like to be able to nest usages of the same name. ChrisA From ethan at stoneleaf.us Sun Jul 7 12:52:12 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 07 Jul 2013 09:52:12 -0700 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51D99CBC.40606@stoneleaf.us> On 07/07/2013 06:43 AM, Chris Angelico wrote: > On Sun, Jul 7, 2013 at 11:13 PM, Wayne Werner wrote: >> Which you would then use like: >> >> >> conn = create_conn() >> with new_transaction(conn) as tran: >> rows_affected = do_query_stuff(tran) >> if rows_affected == 42: >> tran.commit() > > Yep. There's a problem, though, when you bring in subtransactions. The > logic wants to be like this: Is there some reason you can't simply do this? with new_transaction(conn) as tran1: tran1.query("blah") with tran1.subtransaction() as tran2: tran2.query("blah") with tran2.subtransaction() as tran3: tran3.query("blah") # roll this subtransaction back tran2.query("blah") tran2.commit() tran1.query("blah") tran1.commit() -- ~Ethan~ From rosuav at gmail.com Sun Jul 7 20:48:03 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 8 Jul 2013 10:48:03 +1000 Subject: Default scope of variables In-Reply-To: <51D99CBC.40606@stoneleaf.us> References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D99CBC.40606@stoneleaf.us> Message-ID: On Mon, Jul 8, 2013 at 2:52 AM, Ethan Furman wrote: > On 07/07/2013 06:43 AM, Chris Angelico wrote: >> >> On Sun, Jul 7, 2013 at 11:13 PM, Wayne Werner >> wrote: >>> >>> Which you would then use like: >>> >>> >>> conn = create_conn() >>> with new_transaction(conn) as tran: >>> rows_affected = do_query_stuff(tran) >>> if rows_affected == 42: >>> tran.commit() >> >> >> Yep. There's a problem, though, when you bring in subtransactions. The >> logic wants to be like this: > > > Is there some reason you can't simply do this? > > with new_transaction(conn) as tran1: > tran1.query("blah") > with tran1.subtransaction() as tran2: > tran2.query("blah") > with tran2.subtransaction() as tran3: > tran3.query("blah") > > # roll this subtransaction back > tran2.query("blah") > tran2.commit() > tran1.query("blah") > tran1.commit() That means that I, as programmer, have to keep track of the nesting level of subtransactions. Extremely ugly. A line of code can't be moved around without first checking which transaction object to work with. ChrisA From steve+comp.lang.python at pearwood.info Sun Jul 7 22:23:16 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 08 Jul 2013 02:23:16 GMT Subject: Default scope of variables References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D99CBC.40606@stoneleaf.us> Message-ID: <51da2294$0$6512$c3e8da3$5496439d@news.astraweb.com> On Mon, 08 Jul 2013 10:48:03 +1000, Chris Angelico wrote: [...] > That means that I, as programmer, have to keep track of the nesting > level of subtransactions. Extremely ugly. A line of code can't be moved > around without first checking which transaction object to work with. I feel your pain, but I wonder why we sometimes accept "a line of code can't be moved around" as an issue to be solved by the language. After all, in general most lines of code can't be moved around. -- Steven From rosuav at gmail.com Sun Jul 7 23:11:37 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 8 Jul 2013 13:11:37 +1000 Subject: Default scope of variables In-Reply-To: <51da2294$0$6512$c3e8da3$5496439d@news.astraweb.com> References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D99CBC.40606@stoneleaf.us> <51da2294$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Mon, Jul 8, 2013 at 12:23 PM, Steven D'Aprano wrote: > On Mon, 08 Jul 2013 10:48:03 +1000, Chris Angelico wrote: > [...] >> That means that I, as programmer, have to keep track of the nesting >> level of subtransactions. Extremely ugly. A line of code can't be moved >> around without first checking which transaction object to work with. > > I feel your pain, but I wonder why we sometimes accept "a line of code > can't be moved around" as an issue to be solved by the language. After > all, in general most lines of code can't be moved around. It's not something to be solved by the language, but it's often something to be solved by the program's design. Two lines of code that achieve the same goal should normally look the same. This is why Python's policy is "one obvious way to do something" rather than "spell it five different ways in the same file to make a nightmare for other people coming after you". Why should database queries be spelled "trans1.query()" in one place, and "trans2.query()" in another? Similarly, if I want to call another function and that function needs to use the database, why should I pass it trans3 and have that come out as trans1 on the other side? Unnecessarily confusing. Makes much more sense to use the same name everywhere. ChrisA From steve+comp.lang.python at pearwood.info Mon Jul 8 01:00:48 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 08 Jul 2013 05:00:48 GMT Subject: Default scope of variables References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D99CBC.40606@stoneleaf.us> <51da2294$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51da4780$0$6512$c3e8da3$5496439d@news.astraweb.com> On Mon, 08 Jul 2013 13:11:37 +1000, Chris Angelico wrote: > On Mon, Jul 8, 2013 at 12:23 PM, Steven D'Aprano > wrote: >> On Mon, 08 Jul 2013 10:48:03 +1000, Chris Angelico wrote: [...] >>> That means that I, as programmer, have to keep track of the nesting >>> level of subtransactions. Extremely ugly. A line of code can't be >>> moved around without first checking which transaction object to work >>> with. >> >> I feel your pain, but I wonder why we sometimes accept "a line of code >> can't be moved around" as an issue to be solved by the language. After >> all, in general most lines of code can't be moved around. > > It's not something to be solved by the language, but it's often > something to be solved by the program's design. Two lines of code that > achieve the same goal should normally look the same. This is why > Python's policy is "one obvious way to do something" rather than "spell > it five different ways in the same file to make a nightmare for other > people coming after you". Why should database queries be spelled > "trans1.query()" in one place, and "trans2.query()" in another? Is that a trick question? They probably shouldn't. But it's a big leap from that to "...and therefore `for` and `while` should introduce their own scope". > Similarly, if I want to call another function and that function needs to > use the database, why should I pass it trans3 and have that come out as > trans1 on the other side? Unnecessarily confusing. Makes much more sense > to use the same name everywhere. "Is your name not Bruce? That's going to cause a little confusion." http://www.youtube.com/watch?v=_f_p0CgPeyA -- Steven From rosuav at gmail.com Mon Jul 8 01:14:06 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 8 Jul 2013 15:14:06 +1000 Subject: Default scope of variables In-Reply-To: <51da4780$0$6512$c3e8da3$5496439d@news.astraweb.com> References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> <51D99CBC.40606@stoneleaf.us> <51da2294$0$6512$c3e8da3$5496439d@news.astraweb.com> <51da4780$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Mon, Jul 8, 2013 at 3:00 PM, Steven D'Aprano wrote: > On Mon, 08 Jul 2013 13:11:37 +1000, Chris Angelico wrote: >> It's not something to be solved by the language, but it's often >> something to be solved by the program's design. Two lines of code that >> achieve the same goal should normally look the same. This is why >> Python's policy is "one obvious way to do something" rather than "spell >> it five different ways in the same file to make a nightmare for other >> people coming after you". Why should database queries be spelled >> "trans1.query()" in one place, and "trans2.query()" in another? > > Is that a trick question? They probably shouldn't. But it's a big leap > from that to "...and therefore `for` and `while` should introduce their > own scope". No, it's not a trick question; I was responding to Ethan's suggestion as well as yours, and he was saying pretty much that. BruceA (maybe that'll reduce the confusion?) From ethan at stoneleaf.us Sun Jul 7 14:59:28 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 07 Jul 2013 11:59:28 -0700 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51D9BA90.2070205@stoneleaf.us> On 07/07/2013 06:43 AM, Chris Angelico wrote: > > The 'with' statement doesn't allow this. I would need to use some kind > of magic to rebind the old transaction to the name, or else use a list > that gets magically populated: > > with new_transaction(conn) as tran: > tran[-1].query("blah") > with subtransaction(tran): > tran[-1].query("blah") > with subtransaction(tran): > tran[-1].query("blah") > # roll this subtransaction back > tran[-1].query("blah") > tran[-1].commit() > tran[-1].query("blah") > tran[-1].commit() The other option is to build the magic into the new_transaction class, then your code will look like: with new_transaction(conn) as tran: tran.query("blah") with tran.subtransaction(): tran.query("blah") with tran.subtransaction(): tran.query("blah") # roll this subtransaction back tran.query("blah") tran.commit() tran.query("blah") tran.commit() This would definitely make more sense in a loop. ;) -- ~Ethan~ From steve+comp.lang.python at pearwood.info Sun Jul 7 12:03:51 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 07 Jul 2013 16:03:51 GMT Subject: Default scope of variables References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51d99167$0$9505$c3e8da3$5496439d@news.astraweb.com> On Sun, 07 Jul 2013 23:43:24 +1000, Chris Angelico wrote: > On Sun, Jul 7, 2013 at 11:13 PM, Wayne Werner > wrote: >> Which you would then use like: >> >> >> conn = create_conn() >> with new_transaction(conn) as tran: >> rows_affected = do_query_stuff(tran) >> if rows_affected == 42: >> tran.commit() > > Yep. There's a problem, though, when you bring in subtransactions. The > logic wants to be like this: [snip hideous code] > I don't like the look of this. It might work, but it's hardly ideal. > This is why I like to be able to nest usages of the same name. Yes, and the obvious way to nest usages of the same name is to use a instance with a class attribute and instance attribute of the same name: class Example: attr = 23 x = Example() x.attr = 42 print(x.attr) del x.attr print(x.attr) If you need more than two levels, you probably ought to re-design your code to be less confusing, otherwise you may be able to use ChainMap to emulate any number of nested scopes. One interesting trick is you can use a ChainMap as function globals. Here's a sketch for what you can do in Python 3.3: from types import FunctionType from collections import ChainMap class _ChainedDict(ChainMap, dict): # Function dicts must be instances of dict :-( pass def chained_function(func, *dicts): """Return a new function, copied from func, using a ChainMap as dict. """ dicts = dicts + (func.__globals__, builtins.__dict__) d = _ChainedDict(*dicts) name = func.__name__ newfunc = FunctionType( func.__code__, d, name, closure=func.__closure__) newfunc.__dict__.update(func.__dict__) newfunc.__defaults__ = func.__defaults__ return newfunc And in use: py> f = chained_function(lambda x: x+y, {'y': 100}) py> f(1) 101 py> f.__globals__.maps.insert(0, {'y': 200}) py> f(1) 201 py> del f.__globals__.maps[0]['y'] py> f(1) 101 -- Steven From rosuav at gmail.com Sun Jul 7 20:46:59 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 8 Jul 2013 10:46:59 +1000 Subject: Default scope of variables In-Reply-To: <51d99167$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d99167$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Mon, Jul 8, 2013 at 2:03 AM, Steven D'Aprano wrote: > If you need more than two levels, you probably ought to re-design your > code to be less confusing, otherwise you may be able to use ChainMap to > emulate any number of nested scopes. The subtransactions are primarily to represent the database equivalent of a try/except block, so they need to be able to be nested arbitrarily. ChrisA From wuwei23 at gmail.com Tue Jul 9 00:52:41 2013 From: wuwei23 at gmail.com (alex23) Date: Tue, 09 Jul 2013 14:52:41 +1000 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 7/07/2013 11:43 PM, Chris Angelico wrote: > Yep. There's a problem, though, when you bring in subtransactions. The > logic wants to be like this: > > with new_transaction(conn) as tran: > tran.query("blah") > with tran.subtransaction() as tran: > tran.query("blah") > with tran.subtransaction() as tran: > tran.query("blah") > # roll this subtransaction back > tran.query("blah") > tran.commit() > tran.query("blah") > tran.commit() > > The 'with' statement doesn't allow this. I'd take the 'explicit is better than implicit' approach to naming the context managers here and rather than use 'tran' choose names that reflect for what the transaction is to be used ie summarising what's in the "blah" of each query. with new_transaction(conn) as folder_tran: folder_tran.query("blah") with folder_tran.subtransaction() as file_tran: file_tran.query("blah") with file_tran.subtransaction() as type_tran: type_tran.query("blah") (for want of a better heirachical example...) From rosuav at gmail.com Tue Jul 9 01:07:29 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 9 Jul 2013 15:07:29 +1000 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, Jul 9, 2013 at 2:52 PM, alex23 wrote: > with new_transaction(conn) as folder_tran: > folder_tran.query("blah") > with folder_tran.subtransaction() as file_tran: > file_tran.query("blah") > with file_tran.subtransaction() as type_tran: > type_tran.query("blah") Warp my code around a language limitation? Only if I absolutely have to. The subtransactions are NOT concepted as separate transactions. They are effectively the database equivalent of a try/except block. Would you want to have to use a different name for a builtin just because you're inside a try block? a = int("123") try: b = int1(user_input) except ValueError: b = 0 c = int("234") No. That assignment to b should be int(), same as a and c. ChrisA From wuwei23 at gmail.com Tue Jul 9 02:08:18 2013 From: wuwei23 at gmail.com (alex23) Date: Tue, 09 Jul 2013 16:08:18 +1000 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 9/07/2013 3:07 PM, Chris Angelico wrote: > The subtransactions are NOT concepted as separate transactions. They > are effectively the database equivalent of a try/except block. Sorry, I assumed each nested query was somehow related to the prior one. In which case, I'd probably go with Ethan's suggestion of a top-level transaction context manager with its own substransaction method. From rosuav at gmail.com Tue Jul 9 02:13:43 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 9 Jul 2013 16:13:43 +1000 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, Jul 9, 2013 at 4:08 PM, alex23 wrote: > On 9/07/2013 3:07 PM, Chris Angelico wrote: >> >> The subtransactions are NOT concepted as separate transactions. They >> are effectively the database equivalent of a try/except block. > > > Sorry, I assumed each nested query was somehow related to the prior > one. In which case, I'd probably go with Ethan's suggestion of a > top-level transaction context manager with its own substransaction > method. Yeah, that would probably be the best option in this particular instance. Though I do still like the ability to have variables shadow each other, even if there's a way around one particular piece of code that uses the technique. ChrisA From frank at chagford.com Tue Jul 9 03:35:34 2013 From: frank at chagford.com (Frank Millman) Date: Tue, 9 Jul 2013 09:35:34 +0200 Subject: Default scope of variables References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com><51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com><51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: "Chris Angelico" wrote in message news:CAPTjJmqkmFd4-Jpugr-vubuB6riBV6K_Mwnxc_U3CVaBr_Wgbg at mail.gmail.com... > On Tue, Jul 9, 2013 at 4:08 PM, alex23 wrote: >> On 9/07/2013 3:07 PM, Chris Angelico wrote: >>> >>> The subtransactions are NOT concepted as separate transactions. They >>> are effectively the database equivalent of a try/except block. >> >> >> Sorry, I assumed each nested query was somehow related to the prior >> one. In which case, I'd probably go with Ethan's suggestion of a >> top-level transaction context manager with its own substransaction >> method. > > Yeah, that would probably be the best option in this particular > instance. Though I do still like the ability to have variables shadow > each other, even if there's a way around one particular piece of code > that uses the technique. > I have been following this sub-thread with interest, as it resonates with what I am doing in my project. In my case, one update can trigger another, which can trigger another, etc. It is important that they are treated as a single transaction. Each object has its own 'save' method, so there is not one place where all updates are executed, and I found it tricky to control. I came up with the following context manager - class DbSession: """ A context manager to handle database activity. """ def __init__(self): self.conn = None self.no_connections = 0 self.transaction_active = False def __enter__(self): if self.conn is None: self.conn = _get_connection() # get connection from pool self.conn.cur = self.conn.cursor() # all updates in same transaction use same timestamp self.conn.timestamp = datetime.now() self.no_connections += 1 return self.conn def __exit__(self, type, exc, tb): if type is not None: # an exception occurred if self.transaction_active: self.conn.rollback() self.transaction_active = False self.conn.release() # return connection to pool self.conn = None return # will reraise exception self.no_connections -= 1 if not self.no_connections: if self.transaction_active: self.conn.commit() self.transaction_active = False self.conn.cur.close() self.conn.release() # return connection to pool self.conn = None All objects created within a session share a common DbSession instance. When any of them need any database access, whether for reading or for updating, they execute the following - with db_session as conn: conn.transaction_active = True # this line must be added if updating conn.cur.execute(__whatever__) Now it 'just works'. I don't have the need for save-points - either all updates happen, or none of them do. Frank Millman From rosuav at gmail.com Tue Jul 9 03:45:56 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 9 Jul 2013 17:45:56 +1000 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, Jul 9, 2013 at 5:35 PM, Frank Millman wrote: > I have been following this sub-thread with interest, as it resonates with > what I am doing in my project. Just FYI, none of my own code will help you as it's all using libpqxx, but the docs for the library itself are around if you want them (it's one of the standard ways for C++ programs to use PostgreSQL). > I came up with the following context manager - > > class DbSession: > def __exit__(self, type, exc, tb): > if self.transaction_active: > self.conn.commit() > self.transaction_active = False Hmm. So you automatically commit. I'd actually be inclined to _not_ do this; make it really explicit in your code that you now will commit this transaction (which might throw an exception if you have open subtransactions). The way the code with libpqxx works is that any transaction (including a subtransaction) must be explicitly committed, or it will be rolled back. So there's one possible code path that results in persistent changes to the database, and anything else won't: * If the object expires without being committed, it's rolled back. * If an exception is thrown and unwinds the stack, roll back. * If a Unix signal is sent that terminates the program, roll back. * If the process gets killed -9, definitely roll back. * If the computer the program's running on spontaneously combusts, roll back. * If the hard drive is physically ripped from the server during processing, roll back. (Note though that most of these guarantees are from PostgreSQL, not from libpqxx. I'm talking about the whole ecosystem here, not something one single library can handle.) I have absolute 100% confidence that nothing can possibly affect the database unless I explicitly commit it (aside from a few non-transaction actions like advancing a sequence pointer, which specifically don't matter (rolled back transactions can result in gaps in a sequence of record IDs, nothing more)). It's an extremely comfortable work environment - I can write whatever code I like, and if I'm not sure if it'll work or not, I just comment out the commit line and run. *Nothing* can get past that and quietly commit it behind my back. ChrisA From frank at chagford.com Tue Jul 9 04:22:48 2013 From: frank at chagford.com (Frank Millman) Date: Tue, 9 Jul 2013 10:22:48 +0200 Subject: Default scope of variables References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com><51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com><51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: "Chris Angelico" wrote in message news:CAPTjJmr4Mr0qCGWqXwyvDCZ55NuaV79VbTT8BJNDSDvhrKq15w at mail.gmail.com... > On Tue, Jul 9, 2013 at 5:35 PM, Frank Millman wrote: >> I have been following this sub-thread with interest, as it resonates with >> what I am doing in my project. > > Just FYI, none of my own code will help you as it's all using libpqxx, > but the docs for the library itself are around if you want them (it's > one of the standard ways for C++ programs to use PostgreSQL). > I support multiple databases (PostgreSQL, MS SQL Server, sqlite3 at this stage) so I use generic Python as much as possible. >> I came up with the following context manager - >> >> class DbSession: >> def __exit__(self, type, exc, tb): >> if self.transaction_active: >> self.conn.commit() >> self.transaction_active = False > > Hmm. So you automatically commit. I'd actually be inclined to _not_ do > this; make it really explicit in your code that you now will commit > this transaction (which might throw an exception if you have open > subtransactions). > I endeavour to keep all my database activity to the shortest time possible - get a connection, execute a command, release the connection. So setting 'transaction_active = True' is my way of saying 'execute this command and commit it straight away'. That is explicit enough for me. If there are nested updates they all follow the same philosophy, so the transaction should complete quickly. Frank From ian.g.kelly at gmail.com Tue Jul 9 03:56:29 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Tue, 9 Jul 2013 01:56:29 -0600 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, Jul 9, 2013 at 1:35 AM, Frank Millman wrote: > When any of them need any database access, whether for reading or for > updating, they execute the following - > > with db_session as conn: > conn.transaction_active = True # this line must be added if > updating > conn.cur.execute(__whatever__) I'd probably factor out the transaction_active line into a separate DbSession method. @contextmanager def updating(self): with self as conn: conn.transaction_active = True yield conn Then you can do "with db_session" if you're merely reading, or "with db_session.updating()" if you're writing, and you don't need to repeat the transaction_active line all over the place. I would also probably make db_session a factory function instead of a global. From frank at chagford.com Tue Jul 9 04:38:21 2013 From: frank at chagford.com (Frank Millman) Date: Tue, 9 Jul 2013 10:38:21 +0200 Subject: Default scope of variables References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com><51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com><51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: "Ian Kelly" wrote in message news:CALwzid=FzgjPebifx1stDBkh8iwLtWggwwPTPhZ1ykYg+05wEg at mail.gmail.com... > On Tue, Jul 9, 2013 at 1:35 AM, Frank Millman wrote: >> When any of them need any database access, whether for reading or for >> updating, they execute the following - >> >> with db_session as conn: >> conn.transaction_active = True # this line must be added if >> updating >> conn.cur.execute(__whatever__) > > I'd probably factor out the transaction_active line into a separate > DbSession method. > > @contextmanager > def updating(self): > with self as conn: > conn.transaction_active = True > yield conn > > Then you can do "with db_session" if you're merely reading, or "with > db_session.updating()" if you're writing, and you don't need to repeat > the transaction_active line all over the place. > I'll bear it in mind, but I will have to expend some mental energy to understand it first , so it will have to wait until I can find some time. > I would also probably make db_session a factory function instead of a > global. It is not actually a global. When I create a new session, I create a db_session instance and store it as a session attribute. Whenever I create a database object during the session, I pass in the instance as an argument, so they all share the same one. Frank From ethan at stoneleaf.us Tue Jul 9 12:07:58 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 09 Jul 2013 09:07:58 -0700 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com><51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com><51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51DC355E.20606@stoneleaf.us> On 07/09/2013 01:38 AM, Frank Millman wrote: > > "Ian Kelly" wrote in message > news:CALwzid=FzgjPebifx1stDBkh8iwLtWggwwPTPhZ1ykYg+05wEg at mail.gmail.com... >> On Tue, Jul 9, 2013 at 1:35 AM, Frank Millman wrote: >>> When any of them need any database access, whether for reading or for >>> updating, they execute the following - >>> >>> with db_session as conn: >>> conn.transaction_active = True # this line must be added if >>> updating >>> conn.cur.execute(__whatever__) >> >> I'd probably factor out the transaction_active line into a separate >> DbSession method. >> >> @contextmanager >> def updating(self): >> with self as conn: >> conn.transaction_active = True >> yield conn >> >> Then you can do "with db_session" if you're merely reading, or "with >> db_session.updating()" if you're writing, and you don't need to repeat >> the transaction_active line all over the place. >> > > I'll bear it in mind, but I will have to expend some mental energy to > understand it first , so it will have to wait until I can find some time. You could also do it like this: def updating(self): self.transaction_active = True return self and a sample object: class Tester(object): def __init__(self): self.transaction_active = False print 'initializied' def __enter__(self, *args): print '__enter__: transaction_active =', self.transaction_active return self def __exit__(self, *args): self.transaction_active = False print '__exit__: transaction_active =', self.transaction_active return def updating(self): self.transaction_active = True print 'updating: self.transaction_active =', self.transaction_active return self with Tester() as conn: print 'in first with block' print '-' * 50 with Tester().updating() as conn: print 'in second with block' with it's test run: ethan at hydra:~$ python test_cm.py initialized __enter__: transaction_active = False in first with block __exit__: transaction_active = False -------------------------------------------------- initialized updating: self.transaction_active = True __enter__: transaction_active = True in second with block __exit__: transaction_active = False -- ~Ethan~ From ian.g.kelly at gmail.com Tue Jul 9 12:44:18 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Tue, 9 Jul 2013 10:44:18 -0600 Subject: Default scope of variables In-Reply-To: <51DC355E.20606@stoneleaf.us> References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> <51DC355E.20606@stoneleaf.us> Message-ID: On Tue, Jul 9, 2013 at 10:07 AM, Ethan Furman wrote: > You could also do it like this: > > def updating(self): > self.transaction_active = True > return self Yes, that would be simpler. I was all set to point out why this doesn't work, and then I noticed that the location of the "transaction_active" attribute is not consistent in the original code. The DbSession class places it on self, and then the example usage places it on the connection object (which I had based my version on). Since that seems to be a source of confusion, it demonstrates another reason why factoring this out is a good thing. From ethan at stoneleaf.us Tue Jul 9 13:23:58 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 09 Jul 2013 10:23:58 -0700 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> <51DC355E.20606@stoneleaf.us> Message-ID: <51DC472E.3090806@stoneleaf.us> On 07/09/2013 09:44 AM, Ian Kelly wrote: > On Tue, Jul 9, 2013 at 10:07 AM, Ethan Furman wrote: >> You could also do it like this: >> >> def updating(self): >> self.transaction_active = True >> return self > > Yes, that would be simpler. I was all set to point out why this > doesn't work, and then I noticed that the location of the > "transaction_active" attribute is not consistent in the original code. > The DbSession class places it on self, and then the example usage > places it on the connection object It looks like DbSession has a conn object, and in the example he has DbSession() named as conn -- ironic, considering this is a variable scoping thread. ;) -- ~Ethan~ From ian.g.kelly at gmail.com Tue Jul 9 14:41:10 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Tue, 9 Jul 2013 12:41:10 -0600 Subject: Default scope of variables In-Reply-To: <51DC472E.3090806@stoneleaf.us> References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> <51DC355E.20606@stoneleaf.us> <51DC472E.3090806@stoneleaf.us> Message-ID: On Tue, Jul 9, 2013 at 11:23 AM, Ethan Furman wrote: > On 07/09/2013 09:44 AM, Ian Kelly wrote: >> >> On Tue, Jul 9, 2013 at 10:07 AM, Ethan Furman wrote: >>> >>> You could also do it like this: >>> >>> def updating(self): >>> self.transaction_active = True >>> return self >> >> >> Yes, that would be simpler. I was all set to point out why this >> doesn't work, and then I noticed that the location of the >> "transaction_active" attribute is not consistent in the original code. >> The DbSession class places it on self, and then the example usage >> places it on the connection object > > > It looks like DbSession has a conn object, and in the example he has > DbSession() named as conn -- ironic, considering this is a variable scoping > thread. ;) The object returned by __enter__ is the conn object, not the DbSession, so naming it "conn" is correct. From ethan at stoneleaf.us Tue Jul 9 15:19:14 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 09 Jul 2013 12:19:14 -0700 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> <51DC355E.20606@stoneleaf.us> <51DC472E.3090806@stoneleaf.us> Message-ID: <51DC6232.7030606@stoneleaf.us> On 07/09/2013 11:41 AM, Ian Kelly wrote: > On Tue, Jul 9, 2013 at 11:23 AM, Ethan Furman wrote: >> On 07/09/2013 09:44 AM, Ian Kelly wrote: >>> >>> On Tue, Jul 9, 2013 at 10:07 AM, Ethan Furman wrote: >>>> >>>> You could also do it like this: >>>> >>>> def updating(self): >>>> self.transaction_active = True >>>> return self >>> >>> >>> Yes, that would be simpler. I was all set to point out why this >>> doesn't work, and then I noticed that the location of the >>> "transaction_active" attribute is not consistent in the original code. >>> The DbSession class places it on self, and then the example usage >>> places it on the connection object >> >> >> It looks like DbSession has a conn object, and in the example he has >> DbSession() named as conn -- ironic, considering this is a variable scoping >> thread. ;) > > The object returned by __enter__ is the conn object, not the > DbSession, so naming it "conn" is correct. Huh. I didn't realize a different object could be returned by __enter__ without affecting which object's __exit__ gets called. Thanks for the lesson! :) -- ~Ethan~ From frank at chagford.com Wed Jul 10 01:54:22 2013 From: frank at chagford.com (Frank Millman) Date: Wed, 10 Jul 2013 07:54:22 +0200 Subject: Default scope of variables References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com><51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com><51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> <51DC355E.20606@stoneleaf.us> Message-ID: "Ian Kelly" wrote in message news:CALwzidnf3Obe0eNf3xTHLj5a40K8HxVThVEipECQ8+34ZxyLxw at mail.gmail.com... > On Tue, Jul 9, 2013 at 10:07 AM, Ethan Furman wrote: >> You could also do it like this: >> >> def updating(self): >> self.transaction_active = True >> return self > > Yes, that would be simpler. I was all set to point out why this > doesn't work, and then I noticed that the location of the > "transaction_active" attribute is not consistent in the original code. > The DbSession class places it on self, and then the example usage > places it on the connection object (which I had based my version on). > Since that seems to be a source of confusion, it demonstrates another > reason why factoring this out is a good thing. You had me worried there for a moment, as that is obviously an error. Then I checked my actual code, and I find that I mis-transcribed it. It actually looks like this - with db_session as conn: db_session.transaction_active = True conn.cur.execute(...) I am still not quite sure what your objection is to this. It feels straightforward to me. Here is one possible answer. Whenever I want to commit a transaction I have to add the extra line. There is a danger that I could mis-spell 'transaction_active', in which case it would not raise an error, but would not commit the transaction, which could be a hard-to-trace bug. Using your approach, if I mis-spelled 'db_session.connect()', it would immediately raise an error. Is that your concern, or are there other issues? Frank From ian.g.kelly at gmail.com Wed Jul 10 11:42:08 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 10 Jul 2013 09:42:08 -0600 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com> <51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> <51DC355E.20606@stoneleaf.us> Message-ID: On Tue, Jul 9, 2013 at 11:54 PM, Frank Millman wrote: > You had me worried there for a moment, as that is obviously an error. > > Then I checked my actual code, and I find that I mis-transcribed it. It > actually looks like this - > > with db_session as conn: > db_session.transaction_active = True > conn.cur.execute(...) > > I am still not quite sure what your objection is to this. It feels > straightforward to me. > > Here is one possible answer. Whenever I want to commit a transaction I have > to add the extra line. There is a danger that I could mis-spell > 'transaction_active', in which case it would not raise an error, but would > not commit the transaction, which could be a hard-to-trace bug. Using your > approach, if I mis-spelled 'db_session.connect()', it would immediately > raise an error. > > Is that your concern, or are there other issues? Yes, that is one concern. Another is that since you mistakenly typed "conn" instead of "db_session" once, you might make the same mistake again in actual code, with the same effect (unless the conn object doesn't allow arbitrary attributes, which is a possibility). Another is that the code adheres better to the DRY principle if you don't need to copy that line all over the place. From frank at chagford.com Thu Jul 11 01:52:32 2013 From: frank at chagford.com (Frank Millman) Date: Thu, 11 Jul 2013 07:52:32 +0200 Subject: Default scope of variables References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com><51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com><51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> <51DC355E.20606@stoneleaf.us> Message-ID: "Ian Kelly" wrote in message news:CALwzidk2+b5bYM5B+XVtoZ8LHeyVHcos4v58F8Z2o1Jb6sAzog at mail.gmail.com... > On Tue, Jul 9, 2013 at 11:54 PM, Frank Millman wrote: >> You had me worried there for a moment, as that is obviously an error. >> >> Then I checked my actual code, and I find that I mis-transcribed it. It >> actually looks like this - >> >> with db_session as conn: >> db_session.transaction_active = True >> conn.cur.execute(...) >> >> I am still not quite sure what your objection is to this. It feels >> straightforward to me. >> >> Here is one possible answer. Whenever I want to commit a transaction I >> have >> to add the extra line. There is a danger that I could mis-spell >> 'transaction_active', in which case it would not raise an error, but >> would >> not commit the transaction, which could be a hard-to-trace bug. Using >> your >> approach, if I mis-spelled 'db_session.connect()', it would immediately >> raise an error. >> >> Is that your concern, or are there other issues? > > Yes, that is one concern. Another is that since you mistakenly typed > "conn" instead of "db_session" once, you might make the same mistake > again in actual code, with the same effect (unless the conn object > doesn't allow arbitrary attributes, which is a possibility). Another > is that the code adheres better to the DRY principle if you don't need > to copy that line all over the place. Thanks to you and Ethan - that does make sense. I have reviewed my code base to see how many occurrences there are, and there are just three. All database objects inherit from a DbOject class. The class has a save() method and a delete() method. Each of these requires a commit, so I use my technique there. All database updates are activated by calling save() or delete(). I have an init.py script to 'bootstrap' a brand new installation, which requires populating an empty database with some basic structures up front. DbObject cannot be used here as the required plumbing is not in place, so I use my technique here as well. However, this does not invalidate your general point, so I will keep it in mind. Thanks Frank From ethan at stoneleaf.us Wed Jul 10 11:29:24 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 10 Jul 2013 08:29:24 -0700 Subject: Default scope of variables In-Reply-To: References: <51d4eb9c$0$29999$c3e8da3$5496439d@news.astraweb.com><51d508ed$0$6512$c3e8da3$5496439d@news.astraweb.com><51d5a504$0$29999$c3e8da3$5496439d@news.astraweb.com> <51DC355E.20606@stoneleaf.us> Message-ID: <51DD7DD4.5030309@stoneleaf.us> On 07/09/2013 10:54 PM, Frank Millman wrote: > > "Ian Kelly" wrote in message > news:CALwzidnf3Obe0eNf3xTHLj5a40K8HxVThVEipECQ8+34ZxyLxw at mail.gmail.com... >> On Tue, Jul 9, 2013 at 10:07 AM, Ethan Furman wrote: >>> You could also do it like this: >>> >>> def updating(self): >>> self.transaction_active = True >>> return self >> >> Yes, that would be simpler. I was all set to point out why this >> doesn't work, and then I noticed that the location of the >> "transaction_active" attribute is not consistent in the original code. >> The DbSession class places it on self, and then the example usage >> places it on the connection object (which I had based my version on). >> Since that seems to be a source of confusion, it demonstrates another >> reason why factoring this out is a good thing. > > You had me worried there for a moment, as that is obviously an error. > > Then I checked my actual code, and I find that I mis-transcribed it. It > actually looks like this - > > with db_session as conn: > db_session.transaction_active = True > conn.cur.execute(...) > > I am still not quite sure what your objection is to this. It feels > straightforward to me. > > Here is one possible answer. Whenever I want to commit a transaction I have > to add the extra line. There is a danger that I could mis-spell > 'transaction_active', in which case it would not raise an error, but would > not commit the transaction, which could be a hard-to-trace bug. Using your > approach, if I mis-spelled 'db_session.connect()', it would immediately > raise an error. > > Is that your concern, or are there other issues? That concern is big enough. I've been bitten by that type of thing enough in my own code to want to avoid it where possible. Plus the `with db_session.updating() as conn:` saves keystrokes. ;) -- ~Ethan~ From fiedzia at gmail.com Wed Jul 3 23:52:43 2013 From: fiedzia at gmail.com (Maciej Dziardziel) Date: Wed, 3 Jul 2013 20:52:43 -0700 (PDT) Subject: Why this code works in python3, but not python 2: Message-ID: Out of curiosity: Does anyone know why the code below is valid in python3, but not python2: def foo(*args, bar=1, **kwargs): pass -- Maciej Dziardziel From wuwei23 at gmail.com Thu Jul 4 00:05:23 2013 From: wuwei23 at gmail.com (alex23) Date: Thu, 04 Jul 2013 14:05:23 +1000 Subject: Why this code works in python3, but not python 2: In-Reply-To: References: Message-ID: On 4/07/2013 1:52 PM, Maciej Dziardziel wrote: > Out of curiosity: Does anyone know why the code below is valid in python3, but not python2: > > def foo(*args, bar=1, **kwargs): > pass It was an explicit syntax change for Python3. You can read about the reasoning behind it here: http://www.python.org/dev/peps/pep-3102/ From fiedzia at gmail.com Thu Jul 4 00:31:51 2013 From: fiedzia at gmail.com (Maciej Dziardziel) Date: Wed, 3 Jul 2013 21:31:51 -0700 (PDT) Subject: Why this code works in python3, but not python 2: In-Reply-To: References: Message-ID: On Thursday, July 4, 2013 5:05:23 AM UTC+1, alex23 wrote: > It was an explicit syntax change for Python3. You can read about the > reasoning behind it here: > > http://www.python.org/dev/peps/pep-3102/ Thanks, that was helpful. Maciej Dziardziel From rosuav at gmail.com Thu Jul 4 00:10:39 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Jul 2013 14:10:39 +1000 Subject: Why this code works in python3, but not python 2: In-Reply-To: References: Message-ID: On Thu, Jul 4, 2013 at 1:52 PM, Maciej Dziardziel wrote: > Out of curiosity: Does anyone know why the code below is valid in python3, but not python2: > > def foo(*args, bar=1, **kwargs): > pass Keyword-only arguments are (IIRC) a Py3-only feature. There are lots of features that don't work in Python 2 - that's simply the way things happen with old versions of software. ChrisA From joshua.landau.ws at gmail.com Thu Jul 4 00:12:26 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Thu, 4 Jul 2013 05:12:26 +0100 Subject: Why this code works in python3, but not python 2: In-Reply-To: References: Message-ID: On 4 July 2013 04:52, Maciej Dziardziel wrote: > Out of curiosity: Does anyone know why the code below is valid in python3, but not python2: > > def foo(*args, bar=1, **kwargs): > pass Python 3 gained syntax for keyword-only arguments. Try "foo(1)" and it will fail -- "bar" needs to be given as a keyword. This is because it comes after a *-expression. You can also do "def foo(*, bar=1)" if you want bar to be keyword-only without accepting any number of positional arguments. Python 2 does not have these, and doesn't understand the syntax for them. From wuwei23 at gmail.com Thu Jul 4 00:47:03 2013 From: wuwei23 at gmail.com (alex23) Date: Thu, 04 Jul 2013 14:47:03 +1000 Subject: Why this code works in python3, but not python 2: In-Reply-To: References: Message-ID: On 4/07/2013 2:12 PM, Joshua Landau wrote: > On 4 July 2013 04:52, Maciej Dziardziel wrote: >> def foo(*args, bar=1, **kwargs): >> pass > Try "foo(1)" and it will fail -- "bar" needs to be given as a keyword. No it won't, because it is supplied with a default. You may be confusing it with the requirement that `bar` must be given as a keyword in order for it to override the default, compared to the way this would need to be written in Py2: def foo(bar=1, *args, **kwargs): ... ...in which case `bar` could be assigned to by the first positional argument. From joshua.landau.ws at gmail.com Thu Jul 4 01:10:21 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Thu, 4 Jul 2013 06:10:21 +0100 Subject: Why this code works in python3, but not python 2: In-Reply-To: References: Message-ID: On 4 July 2013 05:47, alex23 wrote: > On 4/07/2013 2:12 PM, Joshua Landau wrote: >> >> On 4 July 2013 04:52, Maciej Dziardziel wrote: >>> >>> def foo(*args, bar=1, **kwargs): >>> pass > > >> Try "foo(1)" and it will fail -- "bar" needs to be given as a keyword. > > > No it won't, because it is supplied with a default. Pah! I'm not even thinking. From ms2597 at cornell.edu Thu Jul 4 03:32:59 2013 From: ms2597 at cornell.edu (cutems93) Date: Thu, 4 Jul 2013 00:32:59 -0700 (PDT) Subject: Important features for editors Message-ID: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> I am researching on editors for my own reference. I found that each of them has some features that other don't, but I am not sure which features are significant/necessary for a GOOD editor. What features do you a good editor should have? Keyboard shortcuts? Extensions? Thanks! Min From nikos at superhost.gr Thu Jul 4 03:59:46 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Thu, 04 Jul 2013 10:59:46 +0300 Subject: Important features for editors In-Reply-To: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: ???? 4/7/2013 10:32 ??, ?/? cutems93 ??????: > I am researching on editors for my own reference. I found that each of them has some features that other don't, but I am not sure which features are significant/necessary for a GOOD editor. What features do you a good editor should have? Keyboard shortcuts? Extensions? > > Thanks! > Min > Download Sublime Text v3 Is a great editor -- What is now proved was at first only imagined! From davea at davea.name Thu Jul 4 04:34:44 2013 From: davea at davea.name (Dave Angel) Date: Thu, 04 Jul 2013 04:34:44 -0400 Subject: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: On 07/04/2013 03:59 AM, ????? wrote: > ???? 4/7/2013 10:32 ??, ?/? cutems93 ??????: >> I am researching on editors for my own reference. I found that each of >> them has some features that other don't, but I am not sure which >> features are significant/necessary for a GOOD editor. What features do >> you a good editor should have? Keyboard shortcuts? Extensions? >> >> Thanks! >> Min >> > Download Sublime Text v3 > > Is a great editor > When possible, it's polite to supply a link to the page where it's described or can be downloaded. In this case, http://www.sublimetext.com/ It looks pretty good on the web page. The main negatives I can see are: It costs $70 per user It can only be purchased with Paypal, which I won't use. It's available for OS/X, Linux and Windows, with a single purchase The eval/demo is not time-limited (currently) The positives It can be customized, apparently in Python The simple customizations are specified by JSON files Current download is version 2, but there's a version 3 beta, and if you buy now, you won't have an upgrade fee. ...... Note that the OP didn't ask which is a good editor, but which features make a good editor. I'll post a response to that in a little while. -- DaveA From nikos at superhost.gr Thu Jul 4 05:14:38 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Thu, 04 Jul 2013 12:14:38 +0300 Subject: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: ???? 4/7/2013 11:34 ??, ?/? Dave Angel ??????: > On 07/04/2013 03:59 AM, ????? wrote: >> ???? 4/7/2013 10:32 ??, ?/? cutems93 ??????: >>> I am researching on editors for my own reference. I found that each of >>> them has some features that other don't, but I am not sure which >>> features are significant/necessary for a GOOD editor. What features do >>> you a good editor should have? Keyboard shortcuts? Extensions? >>> >>> Thanks! >>> Min >>> >> Download Sublime Text v3 >> >> Is a great editor >> > > When possible, it's polite to supply a link to the page where it's > described or can be downloaded. In this case, > > http://www.sublimetext.com/ > > It looks pretty good on the web page. The main negatives I can see are: > It costs $70 per user > It can only be purchased with Paypal, which I won't use. > It's available for OS/X, Linux and Windows, with a single purchase > The eval/demo is not time-limited (currently) > > The positives > It can be customized, apparently in Python > The simple customizations are specified by JSON files > Current download is version 2, but there's a version 3 beta, and if > you buy now, you won't have an upgrade fee. > > ...... > > Note that the OP didn't ask which is a good editor, but which features > make a good editor. I'll post a response to that in a little while. > > If you guys want to use it i can send you a patch for it. I know its illegal thing to say but it will help you use it without buying it. -- What is now proved was at first only imagined! From rosuav at gmail.com Thu Jul 4 06:03:12 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Jul 2013 20:03:12 +1000 Subject: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: On Thu, Jul 4, 2013 at 7:14 PM, ????? wrote: > If you guys want to use it i can send you a patch for it. > I know its illegal thing to say but it will help you use it without buying > it. Considering that there are plenty of free text editors around, I don't see any reason to steal one. If you want a pay-for editor, pay for it. If you don't want to pay, use emacs or SciTE or nano or something. ChrisA From robert.kern at gmail.com Thu Jul 4 07:01:26 2013 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 04 Jul 2013 12:01:26 +0100 Subject: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: On 2013-07-04 10:14, ????? wrote: > If you guys want to use it i can send you a patch for it. > I know its illegal thing to say but it will help you use it without buying it. Please do not use this forum to make such offers. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From square.steve at gmail.com Thu Jul 4 09:33:19 2013 From: square.steve at gmail.com (Steve Simmons) Date: Thu, 04 Jul 2013 14:33:19 +0100 Subject: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: <82a9b6ee-5d55-42ea-b486-c638b8ef2ce0@email.android.com> "?????" wrote: >???? 4/7/2013 11:34 ??, ?/? Dave Angel ??????: >> On 07/04/2013 03:59 AM, ????? wrote: >>> ???? 4/7/2013 10:32 ??, ?/? cutems93 ??????: >>>> I am researching on editors for my own reference. I found that each >of >>>> them has some features that other don't, but I am not sure which >>>> features are significant/necessary for a GOOD editor. What features >do >>>> you a good editor should have? Keyboard shortcuts? Extensions? >>>> >>>> Thanks! >>>> Min >>>> >>> Download Sublime Text v3 >>> >>> Is a great editor >>> >> >> When possible, it's polite to supply a link to the page where it's >> described or can be downloaded. In this case, >> >> http://www.sublimetext.com/ >> >> It looks pretty good on the web page. The main negatives I can see >are: >> It costs $70 per user >> It can only be purchased with Paypal, which I won't use. >> It's available for OS/X, Linux and Windows, with a single >purchase >> The eval/demo is not time-limited (currently) >> >> The positives >> It can be customized, apparently in Python >> The simple customizations are specified by JSON files >> Current download is version 2, but there's a version 3 beta, and >if >> you buy now, you won't have an upgrade fee. >> >> ...... >> >> Note that the OP didn't ask which is a good editor, but which >features >> make a good editor. I'll post a response to that in a little while. >> >> >If you guys want to use it i can send you a patch for it. >I know its illegal thing to say but it will help you use it without >buying it. > >-- >What is now proved was at first only imagined! >-- >http://mail.python.org/mailman/listinfo/python-list Boy oh boy! You really are a slow learner Nicos. You have just offered to commit a crime and to include dozens of others in that crime ON A PUBLIC FORUM. Please think before you post. Sent from a Galaxy far far away -------------- next part -------------- An HTML attachment was scrubbed... URL: From nikos at superhost.gr Thu Jul 4 09:36:02 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiDOk866z4EzM866?=) Date: Thu, 04 Jul 2013 16:36:02 +0300 Subject: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: ???? 4/7/2013 4:33 ??, ?/? Steve Simmons ??????: > "?????" wrote: > > ???? 4/7/2013 11:34 ??, ?/? Dave Angel ??????: > > On 07/04/2013 03:59 AM, ????? wrote: > > ???? 4/7/2013 10:32 ??, ?/? cutems93 ??????: > > I am researching on editors for my own reference. I > found that each of > them has some features that other don't, but I am not > sure which > features are significant/necessary for a GOOD editor. > What features do > you a good editor should have? Keyboard shortcuts? > Extensions? > > Thanks! > Min > > > Download Sublime Text v3 > > Is a great editor > > > > When possible, it's polite to supply a link to the page where it's > described or can be downloaded. In this case, > > http://www.sublimetext.com/ > > It looks pretty good on the web page. The main negatives I can > see are: > It costs $70 per user > It can only be purchased with Paypal, which I won't use. > It's available for OS/X, Linux and Windows, with a single purchase > The eval/demo is not time-limited (currently) > > The positives > It can be customized, apparently in Python > The simple customizations are specified by JSON files > Current download is version 2, but there's a version 3 beta, and if > you buy now, you won't have an upgrade fee. > > ...... > > Note that the OP didn't ask which is a good editor, but which > features > make a good editor. I'll post a response to that in a little while. > > > > If you > guys want to use it i can send you a patch for it. > I know its illegal thing to say but it will help you use it without > buying it. > > > Boy oh boy! You really are a slow learner Nicos. You have just offered > to commit a crime and to include dozens of others in that crime ON A > PUBLIC FORUM. Please think before you post. > > Sent from a Galaxy far far away Just wanted to help people that might wanted it, thats all, its a great editor, but i wont post in public again such propositions. -- What is now proved was at first only imagined! From feedthetroll at gmx.de Thu Jul 4 10:03:04 2013 From: feedthetroll at gmx.de (feedthetroll at gmx.de) Date: Thu, 4 Jul 2013 07:03:04 -0700 (PDT) Subject: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: <210add0a-8e99-4d86-af92-439f0190b103@googlegroups.com> Am Donnerstag, 4. Juli 2013 15:36:02 UTC+2 schrieb ????? ???33?: > ???? 4/7/2013 4:33 ??, ?/? Steve Simmons ??????: >> "?????" wrote: >>> ???? 4/7/2013 11:34 ??, ?/? Dave Angel ??????: >>>> On 07/04/2013 03:59 AM, ????? wrote: >>>>> ... >>>>> Download Sublime Text v3 >>>>> Is a great editor >>>> ... >>>> It looks pretty good on the web page. The main negatives I can >>>> see are: >>>> It costs $70 per user >>>> It can only be purchased with Paypal, which I won't use. >>>> ... >>> If you >>> guys want to use it i can send you a patch for it. >>> I know its illegal thing to say but it will help you use it without >>> buying it. >> >> Boy oh boy! You really are a slow learner Nicos. You have just offered >> to commit a crime and to include dozens of others in that crime ON A >> PUBLIC FORUM. Please think before you post. >> ... > Just wanted to help people that might wanted it, thats all, its a great > editor, but i wont post in public again such propositions. Ahh, I see. Committing crimes is O.K. as long as 1. I think I help other with it and 2. I do not talk about it. Yep. That's a great attitude towards life. From rustompmody at gmail.com Thu Jul 4 10:02:26 2013 From: rustompmody at gmail.com (rusi) Date: Thu, 4 Jul 2013 07:02:26 -0700 (PDT) Subject: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: On Thursday, July 4, 2013 7:03:19 PM UTC+5:30, Steve Simmons wrote: > Boy oh boy! You really are a slow learner Nicos. You have just offered to > commit a crime and to include dozens of others in that crime ON A PUBLIC > FORUM. Please think before you post. For the record Steve, let me say, I find Robert Kern's objection barely sufficient. And yours less than that. Note that you are not objecting to the crime but to the public expression of it. Just look at your objection from the angle of a police officer, and you will see that it can certainly be construed as abetment/collusion (or whatever is the legalistic jargon. Obviously IANAL) For the sake of covering my own arse let me say: 1. I have earlier described Nikos as a petty-criminal and consider myself justified. 2. I have earlier described condoning him as collusion and I reiterate that claim. And I add 3. The collective actions of this list will help determine further whether he climbs or descends the criminal ladder. Beyond that I reiterate what I said earlier in this thread: These discussions 'in-band' are counter-productive. Setting up something like an invite-only yahoo or google group or some such to discuss this is free and takes 5 minutes. I would have offered to do it but some people like Joshua Landau seem to think I am interested in world (aka Python-list) domination so not offering. I was actually going to suggest that Steven d'Aprano do it. However before I could do that, he said (in CoC thread) that policing is not possible. Saddens me... So I am suggesting you do it. If I am invited I will contribute my 2c. If not, no objection: I dont consider myself a very important member of this list. And yes: I hope you will consider rewording your objection, if not anything to cover your own you-know-what-where!! From square.steve at gmail.com Thu Jul 4 11:35:32 2013 From: square.steve at gmail.com (Steve Simmons) Date: Thu, 04 Jul 2013 16:35:32 +0100 Subject: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: <18c24b2b-40c3-4e53-9ee3-c2e23af244fe@email.android.com> rusi wrote: >On Thursday, July 4, 2013 7:03:19 PM UTC+5:30, Steve Simmons wrote: >> Boy oh boy! You really are a slow learner Nicos. You have just >offered to >> commit a crime and to include dozens of others in that crime ON A >PUBLIC >> FORUM. Please think before you post. > >For the record Steve, let me say, I find Robert Kern's objection barely >sufficient. > >And yours less than that. > >Note that you are not objecting to the crime but to the public >expression of it. >Just look at your objection from the angle of a police officer, and you >will see that it can certainly be construed as abetment/collusion (or >whatever is the legalistic jargon. Obviously IANAL) > >For the sake of covering my own arse let me say: >1. I have earlier described Nikos as a petty-criminal and consider >myself justified. >2. I have earlier described condoning him as collusion and I reiterate >that claim. > >And I add >3. The collective actions of this list will help determine further >whether he climbs or descends the criminal ladder. > >Beyond that I reiterate what I said earlier in this thread: These >discussions 'in-band' are counter-productive. > >Setting up something like an invite-only yahoo or google group or some >such to discuss this is free and takes 5 minutes. > >I would have offered to do it but some people like Joshua Landau seem >to think I am interested in world (aka Python-list) domination so not >offering. >I was actually going to suggest that Steven d'Aprano do it. However >before I could do that, he said (in CoC thread) that policing is not >possible. Saddens me... > >So I am suggesting you do it. > >If I am invited I will contribute my 2c. If not, no objection: I dont >consider myself a very important member of this list. > >And yes: I hope you will consider rewording your objection, if not >anything to cover your own you-know-what-where!! >-- >http://mail.python.org/mailman/listinfo/python-list Rusi, Frankly, I don't give a tinkers cuss whether Nicos gets a visit from the police, FAST or the Greek secret service. Like too many others, I gave him some advice and within 24 hours he has demonstrated that he didn't see any value in that advice. Fair enough, it's not my problem. For my part, I have pointed out the folly of his ways and now reiterated, maybe too politely, that he has let himself down again. Robert Kern had already given him a blunt response that such communication is unwelcome and I felt no need to reinforce it. If the police or any other authority are 'on my trail' they are welcome to visit my house where they will find absolutely no trace of any kind of crime (unless they check how badly I've ignored the need to redecorate). And just for the record, the lack of evidence of crime will not be because I covered it up really well but because "there ain't none guv". I absolutely agree that the ongoing debate about Nicos' behaviour and the finer points of how various members of this list have reacted has become more than a little tiresome and it is high time it moved to another medium. However, with no guidelines in place and no sanctions available, any discussion - on or off list - had precious little value. I still think the CoC idea has some merit but with just a few notable exceptions the response has been pretty lukewarm. Likewise, I don't contribute much to this list - as a newcomer to python I don't have to much to offer - but I mostly enjoy being a small part of it and I would be happy to make a contribution to the CoC if I felt that the end result would be both useful and welcomed. Steve Sent from a Galaxy far far away -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve+comp.lang.python at pearwood.info Thu Jul 4 11:46:51 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 04 Jul 2013 15:46:51 GMT Subject: Important features for editors References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: <51d598eb$0$29999$c3e8da3$5496439d@news.astraweb.com> On Thu, 04 Jul 2013 07:02:26 -0700, rusi wrote: > On Thursday, July 4, 2013 7:03:19 PM UTC+5:30, Steve Simmons wrote: >> Boy oh boy! You really are a slow learner Nicos. You have just offered >> to commit a crime and to include dozens of others in that crime ON A >> PUBLIC FORUM. Please think before you post. > > For the record Steve, let me say, I find Robert Kern's objection barely > sufficient. > > And yours less than that. > > Note that you are not objecting to the crime Which crime is that? Presumably you mean an actual criminal felony, not a mere civil offence. Under which jurisdiction? > but to the public > expression of it. Just look at your objection from the angle of a police > officer, and you will see that it can certainly be construed as > abetment/collusion If piracy is a crime, and not just a civil offence, then surely so is libel. You have just accused Steve Simmons of collusion in a felony, based on absolutely no evidence at all. That's as clear a case of libel as I've ever seen. Although I'm not sure that the damages should be terribly high -- there's little evidence that anyone treats your ridiculously exaggerated claims about other people's supposed crimes as anything but hot air. But, putting damages aside, according to what passes for the reasoning that you have demonstrated here, your committing libel would make you equally a "petty criminal" as Nikos. Speaking of petty, this whole witch-hunt is getting ridiculous. Don't you have something more productive to do? Accusing people of colluding in crimes because they fail to be sufficiently zealous in "objecting to the crime" is nuts. -- Steven From steve+comp.lang.python at pearwood.info Thu Jul 4 11:48:00 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 04 Jul 2013 15:48:00 GMT Subject: Important features for editors References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: <51d5992f$0$29999$c3e8da3$5496439d@news.astraweb.com> On Thu, 04 Jul 2013 12:01:26 +0100, Robert Kern wrote: > On 2013-07-04 10:14, ????? wrote: > >> If you guys want to use it i can send you a patch for it. I know its >> illegal thing to say but it will help you use it without buying it. > > Please do not use this forum to make such offers. Thank you Robert. -- Steven From invalid at invalid.invalid Thu Jul 4 14:40:21 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Thu, 4 Jul 2013 18:40:21 +0000 (UTC) Subject: Important features for editors References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: On 2013-07-04, ?????????? wrote: > > If you guys want to use it i can send you a patch for it. I know its > illegal thing to say but it will help you use it without buying it. A new low. Now he's offering to help people steal others' work. -- Grant From nikos at superhost.gr Thu Jul 4 14:52:19 2013 From: nikos at superhost.gr (Ferrous Cranus) Date: Thu, 04 Jul 2013 21:52:19 +0300 Subject: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: ???? 4/7/2013 9:40 ??, ?/? Grant Edwards ??????: > On 2013-07-04, ?????????? wrote: >> >> If you guys want to use it i can send you a patch for it. I know its >> illegal thing to say but it will help you use it without buying it. > > A new low. Now he's offering to help people steal others' work. Like you never downloaded serials/keygens/patch/cracks for warez and torrents websites. What a hypocritism.....my intensions was to help the OP. -- What is now proved was at first only imagined! From rosuav at gmail.com Thu Jul 4 17:59:12 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 5 Jul 2013 07:59:12 +1000 Subject: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: On Fri, Jul 5, 2013 at 4:52 AM, Ferrous Cranus wrote: > ???? 4/7/2013 9:40 ??, ?/? Grant Edwards ??????: > >> On 2013-07-04, ?????????? wrote: >>> >>> >>> If you guys want to use it i can send you a patch for it. I know its >>> illegal thing to say but it will help you use it without buying it. >> >> >> A new low. Now he's offering to help people steal others' work. > > > Like you never downloaded serials/keygens/patch/cracks for warez and > torrents websites. > > What a hypocritism.....my intensions was to help the OP. Even if some of us *have* violated license agreements (which you shouldn't assume is the case - most of us here are happily using free software), it's still something that you shouldn't advocate in public. Would you walk into a crowded room and start hawking a Kryptonite gun ("guaranteed to kill Superman, then you can go rob all the banks you like") regardless of who's listening? No, you sell it quietly on the black market. You certainly don't say "Hey, you guys have robbed banks before... haven't you?", because that's certainly not going to be true of everyone, and even if it is, it doesn't change what you've just offered. ChrisA From jason.swails at gmail.com Thu Jul 4 17:59:56 2013 From: jason.swails at gmail.com (Jason Swails) Date: Thu, 4 Jul 2013 17:59:56 -0400 Subject: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: On Thu, Jul 4, 2013 at 2:52 PM, Ferrous Cranus wrote: > ???? 4/7/2013 9:40 ??, ?/? Grant Edwards ??????: > > On 2013-07-04, ?????????? wrote: >> >>> >>> If you guys want to use it i can send you a patch for it. I know its >>> illegal thing to say but it will help you use it without buying it. >>> >> >> A new low. Now he's offering to help people steal others' work. >> > > Like you never downloaded serials/keygens/patch/cracks for warez and > torrents websites. > > What a hypocritism.....my intensions was to help the OP. No, I don't. Ever. And I imagine many others on this list are the same way. If I don't want to pay asking price for a product, I won't use it. Period. There's an open source solution to almost everything. Learn to use that instead. Since many people on this list are in the business of making software, I'd be willing to bet you are in the minority here. (The rather despised minority, based on previous comments). Even people that strongly believe all software should be free would avoid Sublime Text (or start an inspired open source project), not crack it. Example: look at the history of git. What you are doing (and offering to help others do in a public forum) damages the very product you claim to like. Commercial software is maintained and improved by the funding provided by product sales, which you deprive them of by your behavior. The original offer was misguided at best, but this attempted defense that casts everyone else down to your level (and avoids admitting wrongdoing) is reprehensible. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Jul 5 03:25:46 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 05 Jul 2013 03:25:46 -0400 Subject: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: On 7/4/2013 2:52 PM, Ferrous Cranus wrote: > > Like you never downloaded serials/keygens/patch/cracks for warez and > torrents websites. Morality aside, why would I? Today I bought 8 games on GOG.com for about $22 - drm and virus free and easy download and install. If I get 10 hours of fun from 2 of them, I'll be happy. This is not to mention free Python and LibreOffice as my primary work programs - suppported by hg, TortoiseHg, 7zip, and others. -- Terry Jan Reedy From invalid at invalid.invalid Fri Jul 5 10:11:26 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Fri, 5 Jul 2013 14:11:26 +0000 (UTC) Subject: Important features for editors References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: On 2013-07-04, Ferrous Cranus wrote: > ???????? 4/7/2013 9:40 ????, ??/?? Grant Edwards ????????????: >> On 2013-07-04, ?????????? wrote: >>> >>> If you guys want to use it i can send you a patch for it. I know its >>> illegal thing to say but it will help you use it without buying it. >> >> A new low. Now he's offering to help people steal others' work. > > Like you never downloaded serials/keygens/patch/cracks for warez and > torrents websites. No, not that I can recall. -- Grant Edwards grant.b.edwards Yow! In Newark the at laundromats are open 24 gmail.com hours a day! From nikos at superhost.gr Fri Jul 5 03:41:39 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Fri, 05 Jul 2013 10:41:39 +0300 Subject: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: ???? 4/7/2013 10:59 ??, ?/? ????? ??????: > ???? 4/7/2013 10:32 ??, ?/? cutems93 ??????: >> I am researching on editors for my own reference. I found that each of >> them has some features that other don't, but I am not sure which >> features are significant/necessary for a GOOD editor. What features do >> you a good editor should have? Keyboard shortcuts? Extensions? >> >> Thanks! >> Min >> > Download Sublime Text v3 > > Is a great editor > The only thing missing from this great editor is the ability to upload your python scripts to a remote web server. They should embed a plugin for that like Notepad's NPPFtp plugin. Other that that its all complete, and it has help built-in system to help you with the syntax of the commands as you write them. -- What is now proved was at first only imagined! From feedthetroll at gmx.de Fri Jul 5 04:28:32 2013 From: feedthetroll at gmx.de (feedthetroll at gmx.de) Date: Fri, 5 Jul 2013 01:28:32 -0700 (PDT) Subject: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: <41c74174-3337-4793-819a-05368de4e363@googlegroups.com> Am Donnerstag, 4. Juli 2013 11:14:38 UTC+2 schrieb ????? Gr33k: > ... >> On 07/04/2013 03:59 AM, ????? wrote: >>> ... >>> Download Sublime Text v3 >>> Is a great editor >>> ... > > If you guys want to use it i can send you a patch for it. > I know its illegal thing to say but it will help you use it without > buying it. Am Freitag, 5. Juli 2013 09:41:39 UTC+2 schrieb ????? Gr33k: > [talkin about Sublime Text editor] > The only thing missing from this great editor is the ability to upload > your python scripts to a remote web server. > They should embed a plugin for that like Notepad's NPPFtp plugin. Oh, I'm sure they would have time/money to do so, if more people payed the license fees. From goktug.kayaalp at gmail.com Thu Jul 4 04:07:10 2013 From: goktug.kayaalp at gmail.com (=?UTF-8?B?R8O2a3R1xJ8gS2F5YWFscA==?=) Date: Thu, 4 Jul 2013 11:07:10 +0300 Subject: Fwd: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: Programmability comes to my mind, before anything else. I'd suggest to find out about designs of Emacs and Vi(m). On Thu, Jul 4, 2013 at 10:32 AM, cutems93 wrote: > I am researching on editors for my own reference. I found that each of them has some features that other don't, but I am not sure which features are significant/necessary for a GOOD editor. What features do you a good editor should have? Keyboard shortcuts? Extensions? > > Thanks! > Min > -- > http://mail.python.org/mailman/listinfo/python-list From rustompmody at gmail.com Fri Jul 5 08:12:14 2013 From: rustompmody at gmail.com (rusi) Date: Fri, 5 Jul 2013 05:12:14 -0700 (PDT) Subject: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: <081319f4-4d68-409e-81cb-1f500d5d87f0@googlegroups.com> On Thursday, July 4, 2013 1:37:10 PM UTC+5:30, G?ktu? Kayaalp wrote: > Programmability comes to my mind, before anything else. I'd suggest > to find out about designs of Emacs and Vi(m). There's one reason I prefer emacs -- and I guess some people prefer Idle -- the interpreter and editor are tightly integrated. This is not strictly in the class of editor-as-editor but editor as programming-in-the-tiny support. That said it needs to be also remembered: - needs some setup efforts for python - key-bindings and terminology will seem weird to the younger generation One expansion for EMACS is Editor for Middle Aged Computer Scientists -- so I am guessing if you're asking the question you dont qualify :-) If you get past these initial hurdles and also the initial programming hurdles to get to the point where you have the AHA moment -- programming is like art/poetry, you may want to look at emacs -> org-mode -> babel which is the state-of-art system for what is called 'literate programming' http://orgmode.org/worg/org-contrib/babel/intro.html [I am a medium-grade user of org-mode and completely noob at babel! ] From cs at zip.com.au Fri Jul 5 19:06:38 2013 From: cs at zip.com.au (Cameron Simpson) Date: Sat, 6 Jul 2013 09:06:38 +1000 Subject: Important features for editors In-Reply-To: <081319f4-4d68-409e-81cb-1f500d5d87f0@googlegroups.com> References: <081319f4-4d68-409e-81cb-1f500d5d87f0@googlegroups.com> Message-ID: <20130705230638.GA38521@cskk.homeip.net> On 05Jul2013 05:12, rusi wrote: | On Thursday, July 4, 2013 1:37:10 PM UTC+5:30, G?ktu? Kayaalp wrote: | > Programmability comes to my mind, before anything else. I'd suggest | > to find out about designs of Emacs and Vi(m). | | There's one reason I prefer emacs -- and I guess some people | prefer Idle -- the interpreter and editor are tightly integrated. That is indeed a strength of emacs over vi. For myself, I generally don't want to program my editor beyond writing keyboard macros, and vim's programming interface has yet to attract me. When I want to manipulate text beyond a simple macro I tend to write a sed script. Or awk, or python in increasing complexity of task. [...] | One expansion for EMACS is Editor for Middle Aged Computer | Scientists -- so I am guessing if you're asking the question you | dont qualify :-) While I started with vi just slightly before encountering emacs (mid-to-late 1980s, both), my main trouble with choosing emacs was the heavy use of control keys. Vi's modal nature means that in "edit" mode, all the keystrokes are available as edit controls. Emacs' modeless nature means that all the edit controls must be control-this and meta/escape-that. For this reason, I often expand EMACS as Escape Meta Alt Control Shift. I'm a vi user. Once I mastered "hit ESC by reflex when you pause typing an insert" I was never confused above which mode I was in. And now my fingers know vi. Cheers, -- Cameron Simpson A novice of the temple once approached the Chief Priest with a question. "Master, does Emacs have the Buddha nature?" the novice asked. The Chief Priest had been in the temple for many years and could be relied upon to know these things. He thought for several minutes before replying. "I don't see why not. It's got bloody well everything else." With that, the Chief Priest went to lunch. The novice suddenly achieved enlightenment, several years later. Commentary: His Master is kind, Answering his FAQ quickly, With thought and sarcasm. From rustompmody at gmail.com Fri Jul 5 23:13:24 2013 From: rustompmody at gmail.com (Rustom Mody) Date: Sat, 6 Jul 2013 08:43:24 +0530 Subject: Important features for editors In-Reply-To: <20130705230638.GA38521@cskk.homeip.net> References: <081319f4-4d68-409e-81cb-1f500d5d87f0@googlegroups.com> <20130705230638.GA38521@cskk.homeip.net> Message-ID: On Sat, Jul 6, 2013 at 4:36 AM, Cameron Simpson wrote: > While I started with vi just slightly before encountering emacs > (mid-to-late 1980s, both), my main trouble with choosing emacs was > the heavy use of control keys. Vi's modal nature means that in > "edit" mode, all the keystrokes are available as edit controls. > Emacs' modeless nature means that all the edit controls must be > control-this and meta/escape-that. > > For this reason, I often expand EMACS as Escape Meta Alt Control Shift. > > Yes... The fact that rms has crippling RSI should indicate that emacs' ergonomics is not right. > I'm a vi user. Once I mastered "hit ESC by reflex when you pause > typing an insert" I was never confused above which mode I was in. > > And now my fingers know vi. > > Yes... vi: (n) A program that has two modes, one in which it beeps and the other in which it corrupts your file :-) > Cheers, > -- > Cameron Simpson > > A novice of the temple once approached the Chief Priest with a question. > > "Master, does Emacs have the Buddha nature?" the novice asked. > > The Chief Priest had been in the temple for many years and could be > relied > upon to know these things. He thought for several minutes before > replying. > > "I don't see why not. It's got bloody well everything else." > > With that, the Chief Priest went to lunch. The novice suddenly achieved > enlightenment, several years later. > > Commentary: > > His Master is kind, > Answering his FAQ quickly, > With thought and sarcasm. > > Heard somewhere: Emacs is my operating system and linux is its device driver. No I dont belong to that camp -- Actually I am quite dissatisfied with emacs nowadays... Keep trying eclipse and getting repulsed by the gorilla. Philosophy being this: What functional programming is to program-semantics, fast-branching (as in git) is to program-source[1]. To complete the trinity, one needs semi-automated refactoring. The first I can do in my sleep; the second still noob-status, the third yet to start! [1] Not necessarily source-code See http://blog.vctr.me/posts/why-you-should-learn-git.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From esj at harvee.org Sat Jul 6 02:52:20 2013 From: esj at harvee.org (Eric S. Johansson) Date: Sat, 06 Jul 2013 02:52:20 -0400 Subject: Important features for editors In-Reply-To: References: <081319f4-4d68-409e-81cb-1f500d5d87f0@googlegroups.com> <20130705230638.GA38521@cskk.homeip.net> Message-ID: On Fri, 05 Jul 2013 23:13:24 -0400, Rustom Mody wrote: > Yes... > The fact that rms has crippling RSI should indicate that emacs' > ergonomics is not right. As someone crippled by Emacs ( actual cause not known), I should also point out that RMS, instead of doing the responsible thing and using speech recognition software, burns the hands of other human beings by using them as biological speech recognition units. Now for me, an important feature for editor is the ability to command it, not by keystrokes but by function/method invocation. This is be the first step to reducing the disasters caused by misrecognition events injecting unintentional commands into an editor. For example, bring up a file in VI in close your eyes and type some string like "save file" or "end of line". What kind of damage do you get? With an editor RPC, you can bypass all this damage. You turn off keystroke input at the start of a recognition event and all keyboard queue data is injected as characters. All commands are injected by the API. There's a few other things, I need in a very tiny editor to help a part of my accessibility problem. One of the ways I deal with speech recognition user interfaces by creating tiny domain specific languages to solve a problem. You can say them, they are resilient in the face of misrecognition, edit them and you can replay them. Bunch of wins. The tiny editor needs to use the right Windows edit control to work with NaturallySpeaking, save data so that I never have to think about it. It's always on disk, always safe. If I invoke a file by name, I get exactly one instance. And last, I want the ability to filter the contents of the editor through a bit of Python code so I can do transformations on opening the file or writing the file. Further down the road, instead of the classic syntax highlighting, I need dynamic naming of features so that I can say things like "replace third argument", "modify index" for format local times using pattern. I will admit the last one is a bit of a cheat because that's a subset of the domain specific notation I think that earlier. Not a solved problem :-) So important feature editor change depending on your perspective, or should I say, state of impending disability. We all become disabled with age, just some of us age much faster than the rest of the population -------------- next part -------------- An HTML attachment was scrubbed... URL: From rustompmody at gmail.com Sat Jul 6 04:09:11 2013 From: rustompmody at gmail.com (Rustom Mody) Date: Sat, 6 Jul 2013 13:39:11 +0530 Subject: Important features for editors In-Reply-To: References: <081319f4-4d68-409e-81cb-1f500d5d87f0@googlegroups.com> <20130705230638.GA38521@cskk.homeip.net> Message-ID: On Sat, Jul 6, 2013 at 12:22 PM, Eric S. Johansson wrote: > ** > On Fri, 05 Jul 2013 23:13:24 -0400, Rustom Mody > wrote: > > Yes... > The fact that rms has crippling RSI should indicate that emacs' ergonomics > is not right. > > > > As someone crippled by Emacs ( actual cause not known), I should also > point out that RMS, instead of doing the responsible thing and using speech > recognition software, burns the hands of other human beings by using them > as biological speech recognition units. > > Hope youve seen this http://www.cs.princeton.edu/~arora/RSI.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy at panix.com Fri Jul 5 23:25:10 2013 From: roy at panix.com (Roy Smith) Date: Fri, 05 Jul 2013 23:25:10 -0400 Subject: Important features for editors References: <081319f4-4d68-409e-81cb-1f500d5d87f0@googlegroups.com> <20130705230638.GA38521@cskk.homeip.net> Message-ID: In article , Rustom Mody wrote: > > I'm a vi user. Once I mastered "hit ESC by reflex when you pause > > typing an insert" I was never confused above which mode I was in. > > > > And now my fingers know vi. All the vi you need to know: : q ! From joshua.landau.ws at gmail.com Sat Jul 6 00:35:14 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Sat, 6 Jul 2013 05:35:14 +0100 Subject: Important features for editors In-Reply-To: References: <081319f4-4d68-409e-81cb-1f500d5d87f0@googlegroups.com> <20130705230638.GA38521@cskk.homeip.net> Message-ID: On 6 July 2013 04:25, Roy Smith wrote: > In article , > Rustom Mody wrote: > >> > I'm a vi user. Once I mastered "hit ESC by reflex when you pause >> > typing an insert" I was never confused above which mode I was in. >> > >> > And now my fingers know vi. > > All the vi you need to know: > > : q ! I never got why Vi doesn't support Ctrl-C by default -- it's not like it's a used key-combination and it would have helped me so many times when I was younger. :^C Interrupt :^C Interrupt :^C Interrupt :^C Interrupt : At end-of-file : At end-of-file : At end-of-file : At end-of-file :^C Interrupt : At end-of-file :^C Interrupt : At end-of-file :^C Interrupt :^C Interrupt : From rustompmody at gmail.com Sat Jul 6 01:19:39 2013 From: rustompmody at gmail.com (rusi) Date: Fri, 5 Jul 2013 22:19:39 -0700 (PDT) Subject: Important features for editors In-Reply-To: References: <081319f4-4d68-409e-81cb-1f500d5d87f0@googlegroups.com> <20130705230638.GA38521@cskk.homeip.net> Message-ID: On Saturday, July 6, 2013 10:05:14 AM UTC+5:30, Joshua Landau wrote: > I never got why Vi doesn't support Ctrl-C by default -- it's not like > it's a used key-combination and it would have helped me so many times > when I was younger. Dunno what you are referring to. Out here C-c gets vi out of insert mode Second (onwards) and it prints Type :quit to exit Vim From joshua.landau.ws at gmail.com Sat Jul 6 02:19:06 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Sat, 6 Jul 2013 07:19:06 +0100 Subject: Important features for editors In-Reply-To: References: <081319f4-4d68-409e-81cb-1f500d5d87f0@googlegroups.com> <20130705230638.GA38521@cskk.homeip.net> Message-ID: On 6 July 2013 06:19, rusi wrote: > On Saturday, July 6, 2013 10:05:14 AM UTC+5:30, Joshua Landau wrote: >> I never got why Vi doesn't support Ctrl-C by default -- it's not like >> it's a used key-combination and it would have helped me so many times >> when I was younger. > > Dunno what you are referring to. > Out here C-c gets vi out of insert mode > Second (onwards) and it prints > Type :quit to exit Vim I know how to quit Vi *now*, but when I didn't it was a pain. It's easy to get lost in a program that doesn't accept the *standard quitting key*. The rest of my post was a demonstration of what my Vi sessions used to look like. Note that Vi != Vim; Vim at least tells you what to do. From davea at davea.name Thu Jul 4 05:02:40 2013 From: davea at davea.name (Dave Angel) Date: Thu, 04 Jul 2013 05:02:40 -0400 Subject: Important features for editors In-Reply-To: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: On 07/04/2013 03:32 AM, cutems93 wrote: > I am researching on editors for my own reference. I found that each of them has some features that other don't, but I am not sure which features are significant/necessary for a GOOD editor. What features do you a good editor should have? Keyboard shortcuts? Extensions? > Not sure what you mean by keyboard shortcuts. If you mean there should be a keyboard version of anything that can be done with the mouse, absolutely. There are hundreds of features that could be listed, and only you can decide which ones are important. I'll try to list a few that are important to me, and some more that would sure be nice. Very important: -------------- It runs on every platform I'm using. It's extremely fast, when run locally, and reasonable over a slow internet connection. Licensing is free, or very inexpensive It opens and edits files of fairly arbitrary size (at least 10 MB) It has a large number of potential buffers, for editing multiple files at the same time. It can readily be customized, on a per-language basis, so that it can easily handle the quirks of each language. And it switches between them based on file name, without explicitly setting some mode. However, if the filename is unconventional, it allows the buffer to be explicitly set to a particular language, not affecting other files that are simultaneously open. It comes pre-customized for the languages I'm likely to use now. That includes pseudo languages like html, xml, css, not just "programming languages." It supports my own strange preferences for tab-handling, or at least can be customized to do so. It recognizes immediately when a file has been changed on disk, and gives me reasonable ways to merge my current edits into what's now in the disk file. It doesn't force me to accept .bak or other funny files; that's what dvcs systems are for. It CAN create such files while a file is being edited, they just shouldn't persist after the editor is normally closed. If it has project files, they should be out of band, not mixed in with source files I'm editing. Nice to have: ------------ It has visible spaces (and tabs, and other funny white-space characters) It can be run in an ssh session, remotely, over a satellite internet connection and vpn. Customization language is one I'm comfortable with. Not VBA or javascript. Mandatory for Python use: ------------------------ It understands indenting, and lets you easily get to the various columns that are legal at any point. This means it recognizes if statements and suchlike, and indents (4) spaces for the following line. And when you want to unindent, you don't have to use 4 backspaces, but just press the tab again. Nice for Python use: ------------------- Syntax coloring. Re-indenting a group of lines by plus-or-minus 4 columns. Now, you may be asking about an IDE. And that's a whole other kettle of fish. Context-sensitive auto-completion, jump to definition, refactoring support, data breakpoints, ... Candidates? emacs - standard on most OS's, available for Windows from various websites Komodo Edit free http://www.activestate.com/komodo-edit Komodo IDE not free http://www.activestate.com/komodo-ide -- DaveA From python.list at tim.thechases.com Thu Jul 4 09:22:26 2013 From: python.list at tim.thechases.com (Tim Chase) Date: Thu, 4 Jul 2013 08:22:26 -0500 Subject: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: <20130704082226.57b0897a@bigbox.christie.dr> On 2013-07-04 05:02, Dave Angel wrote: [snip an excellent list of things to look for in an editor] Also, - the ability to perform changes in bulk, especially across files. Often, this is done with the ability to record/playback macros, though some editors have multiple insertion/edit cursors; others allow for performing a bulk-change command across the entire file or list of files. - folding (the ability to collapse multiple lines of text down to one line). Especially if there are various ways to do it (manual folding, language-block folding, folding by indentation) - multiple clipboard buffers/registers - multiple bookmarks - the ability to interact with external programs (piping a portion of a file through an external utility) - a good community around it in case you have questions - easy navigation to "important" things in your file (where "important" may vary based on file-type, but may include function definitions, paragraph boundaries, matching paren/bracket/brace/tag, etc) Other nice-to-haves include - split window editing - tabbed windows - Unicode support (including various encodings) - vimgolf.com ;-) > Candidates? > emacs - standard on most OS's, available for Windows from And I'll put in a plug for Vim. -tkc From python at mrabarnett.plus.com Thu Jul 4 10:24:31 2013 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 04 Jul 2013 15:24:31 +0100 Subject: Important features for editors In-Reply-To: <20130704082226.57b0897a@bigbox.christie.dr> References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> <20130704082226.57b0897a@bigbox.christie.dr> Message-ID: <51D5859F.7000600@mrabarnett.plus.com> On 04/07/2013 14:22, Tim Chase wrote: > On 2013-07-04 05:02, Dave Angel wrote: > [snip an excellent list of things to look for in an editor] > > Also, > > - the ability to perform changes in bulk, especially across files. > Often, this is done with the ability to record/playback macros, > though some editors have multiple insertion/edit cursors; others > allow for performing a bulk-change command across the entire file > or list of files. > > - folding (the ability to collapse multiple lines of text down to one > line). Especially if there are various ways to do it (manual > folding, language-block folding, folding by indentation) > > - multiple clipboard buffers/registers > > - multiple bookmarks > > - the ability to interact with external programs (piping a portion of > a file through an external utility) > > - a good community around it in case you have questions > > - easy navigation to "important" things in your file (where > "important" may vary based on file-type, but may include function > definitions, paragraph boundaries, matching > paren/bracket/brace/tag, etc) > > Other nice-to-haves include > > - split window editing > - tabbed windows > - Unicode support (including various encodings) It's 2013, yet Unicode support is merely a "nice-to-have"? > - vimgolf.com ;-) > >> Candidates? >> emacs - standard on most OS's, available for Windows from > > And I'll put in a plug for Vim. > From python.list at tim.thechases.com Thu Jul 4 17:03:19 2013 From: python.list at tim.thechases.com (Tim Chase) Date: Thu, 4 Jul 2013 16:03:19 -0500 Subject: Important features for editors In-Reply-To: <51D5859F.7000600@mrabarnett.plus.com> References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> <20130704082226.57b0897a@bigbox.christie.dr> <51D5859F.7000600@mrabarnett.plus.com> Message-ID: <20130704160319.47ca7f97@bigbox.christie.dr> On 2013-07-04 15:24, MRAB wrote: > On 04/07/2013 14:22, Tim Chase wrote: > > Other nice-to-haves include > > > > - Unicode support (including various encodings) > > It's 2013, yet Unicode support is merely a "nice-to-have"? Yeah, while I use Vim and it's got support, most of what I do interacts with unicode stuff as escaped rather than in-line. In python, it's things like u"\u20AC" or in HTML/XML using one of the escaping variants: € or € or even € So I don't feel particularly hampered even if/when I get stuck with an editor that only speaks lower-ASCII. That's why I considered it merely "nice to have" rather than "essential". -tkc From wrw at mac.com Thu Jul 4 09:42:05 2013 From: wrw at mac.com (William Ray Wing) Date: Thu, 04 Jul 2013 09:42:05 -0400 Subject: Important features for editors In-Reply-To: <20130704082226.57b0897a@bigbox.christie.dr> References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> <20130704082226.57b0897a@bigbox.christie.dr> Message-ID: On Jul 4, 2013, at 9:22 AM, Tim Chase wrote: > On 2013-07-04 05:02, Dave Angel wrote: > [snip an excellent list of things to look for in an editor] > > Also, > > - the ability to perform changes in bulk, especially across files. > Often, this is done with the ability to record/playback macros, > though some editors have multiple insertion/edit cursors; others > allow for performing a bulk-change command across the entire file > or list of files. > To Tim's and Dave's excellent lists let me add one more feature that I find useful: the ability to bulk-search through all files in a folder/directory/project for a word or phrase, display the hits and then selectively or optionally do a replace on each hit in the displayed list. In effect, this is an interactive extension of the bulk search and replace Tim lists above. > - folding (the ability to collapse multiple lines of text down to one > line). Especially if there are various ways to do it (manual > folding, language-block folding, folding by indentation) Yes, yes, yes. It may simply be a reflection of my poor programming style (grow by accretion, wait way too long to refactor), but this is a big help in following program logic. > > - multiple clipboard buffers/registers > > - multiple bookmarks > > - the ability to interact with external programs (piping a portion of > a file through an external utility) > And in particular, the ability to preview a chunk of html in a browser. > - a good community around it in case you have questions > > - easy navigation to "important" things in your file (where > "important" may vary based on file-type, but may include function > definitions, paragraph boundaries, matching > paren/bracket/brace/tag, etc) > > Other nice-to-haves include > > - split window editing > - tabbed windows > - Unicode support (including various encodings) > - vimgolf.com ;-) > >> Candidates? >> emacs - standard on most OS's, available for Windows from > > And I'll put in a plug for Vim. > > -tkc > > > > > > > > -- > http://mail.python.org/mailman/listinfo/python-list From rurpy at yahoo.com Thu Jul 4 11:56:52 2013 From: rurpy at yahoo.com (rurpy at yahoo.com) Date: Thu, 4 Jul 2013 08:56:52 -0700 (PDT) Subject: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> <20130704082226.57b0897a@bigbox.christie.dr> Message-ID: On 07/04/2013 08:24 AM, MRAB wrote: > On 04/07/2013 14:22, Tim Chase wrote: >> On 2013-07-04 05:02, Dave Angel wrote: >> [snip an excellent list of things to look for in an editor] > It's 2013, yet Unicode support is merely a "nice-to-have"? I agree that this is pretty important. Even if you don't have to deal with Unicode today, the chances are good that you will need to, if only in an occasional way, in the future. One thing not mentioned (sorry if I missed it) that I use more than many of the features that have been mentioned is some form of advanced search/replace. I.e. something that can do regular expression searches and replaces including multiline ones. Another feature I find necessary is very fast start up time since I open an editor hundreds of times a day. Because advanced features and fast startup seem to be mutually exclusive, I often use two editors, a simple but quick starting one like Gedit on Linux or Notepad on Windows for 90% of routine editing and Emacs for the the other 10% when I need to do something more powerful. But as a disclaimer I should add that I do not spend 8+ hours a day doing nothing but programming so YMMV. BTW, the group is currently having a problem both with trolls and with regulars here that bite at every baited hook that drifts past their screen. There seems to nothing to be done other than ignore the obnoxious posts but I am sorry they have infiltrated your discussion. Hopefully this comment won't add to the problem. From square.steve at gmail.com Thu Jul 4 12:14:27 2013 From: square.steve at gmail.com (Steve Simmons) Date: Thu, 4 Jul 2013 17:14:27 +0100 Subject: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> <20130704082226.57b0897a@bigbox.christie.dr> Message-ID: To Rurpy and cutems93, My apologies too. I reacted before I thought about creating a new thread. To your question: One thing that I don't use daily but find very useful to have in an editor is 'Hex View' (or better yet a 'Hex Editor'). Whilst it has been 'dissed' recently on this list, I like Notepad++ for everyday editing but if I'm head-down in a particular language, I prefer to be in an IDE. Steve On Thu, Jul 4, 2013 at 4:56 PM, wrote: > On 07/04/2013 08:24 AM, MRAB wrote: > > On 04/07/2013 14:22, Tim Chase wrote: > >> On 2013-07-04 05:02, Dave Angel wrote: > >> [snip an excellent list of things to look for in an editor] > > > It's 2013, yet Unicode support is merely a "nice-to-have"? > > I agree that this is pretty important. Even if you don't > have to deal with Unicode today, the chances are good that > you will need to, if only in an occasional way, in the > future. > > One thing not mentioned (sorry if I missed it) that I > use more than many of the features that have been mentioned > is some form of advanced search/replace. I.e. something > that can do regular expression searches and replaces > including multiline ones. > > Another feature I find necessary is very fast start up time > since I open an editor hundreds of times a day. > > Because advanced features and fast startup seem to be mutually > exclusive, I often use two editors, a simple but quick starting > one like Gedit on Linux or Notepad on Windows for 90% of > routine editing and Emacs for the the other 10% when I need > to do something more powerful. But as a disclaimer I should > add that I do not spend 8+ hours a day doing nothing but > programming so YMMV. > > BTW, the group is currently having a problem both with > trolls and with regulars here that bite at every baited > hook that drifts past their screen. There seems to nothing > to be done other than ignore the obnoxious posts but I am > sorry they have infiltrated your discussion. Hopefully > this comment won't add to the problem. > -- > http://mail.python.org/mailman/listinfo/python-list > -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshua.landau.ws at gmail.com Thu Jul 4 20:38:10 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Fri, 5 Jul 2013 01:38:10 +0100 Subject: Important features for editors In-Reply-To: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: On 4 July 2013 08:32, cutems93 wrote: > I am researching on editors for my own reference. I found that each of them has some features that other don't, but I am not sure which features are significant/necessary for a GOOD editor. What features do you a good editor should have? Keyboard shortcuts? Extensions? Let me give you some reasons I really, really like Sublime Text. * Fast. Like, really fast. I've used Vim -- Sublime Text is faster. Considering I'm on a middle-end 5-year-old computer (not for long...) this matters. * The rendering is gorgeous. There are subtle shadows, there's perfectly crisp text (the main reason I no longer use terminal editors, actually), and once you choose the right theme (Nexus and Phoenix, Tomorrow Night for me) it's just lovely. There's this feature where it shows you tabs -- but only for the part of the file you're on. There's, instead of a scrollbar, a little "bird's-eye-view" of the whole code on the RHS. This goes on. Visually it is stunning, in a helpful way. If it had proper terminal-emulation support, I'd replace my terminal with it. It's just that usable an interface. * Multiple cursors. This is something that no-one else really advertises, but it's one of the most used features for me. It's something you just have to try for a while -- I think it's a bit like Vim's power-of-Regex but easy for a, you know, human. (I just found https://github.com/terryma/vim-multiple-cursors). * Good navigation between and inside of files. A lot of things have this, so I won't say much more. * The "Command Palette" is a dropdown that you do commands from, and because of the way you search it, it's like a hybrid between vim's command-based power and something that's actually discoverable and easy. * Usable on *really big* files, and has one of the best binary-file support I know of. I open binary file a little more than I should, not that I can do much with them. * Useful extensions, installable at a button-press -- in[search for package]. Like SublimeREPL. I know Emacs/Vim will do better at REPLs, but few others will. * Etc. This goes on. Looking at Dave Angel's list, Sublime Text pretty-much aces it. What I don't understand is where he says: > The main negatives I can see are: ... > It's available for OS/X, Linux and Windows, with a single purchase > The eval/demo is not time-limited (currently) How on earth are those negatives? He basically only dislikes it because you have to use PayPal, which is his choice. I can't say I agree with it though. From davea at davea.name Thu Jul 4 21:15:43 2013 From: davea at davea.name (Dave Angel) Date: Thu, 04 Jul 2013 21:15:43 -0400 Subject: Important features for editors In-Reply-To: References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: On 07/04/2013 08:38 PM, Joshua Landau wrote: > On 4 July 2013 08:32, cutems93 wrote: >> I am researching on editors for my own reference. I found that each of them has some features that other don't, but I am not sure which features are significant/necessary for a GOOD editor. What features do you a good editor should have? Keyboard shortcuts? Extensions? > > Let me give you some reasons I really, really like Sublime Text. > > * Fast. Like, really fast. I've used Vim -- Sublime Text is faster. > Considering I'm on a middle-end 5-year-old computer (not for long...) > this matters. > > * The rendering is gorgeous. There are subtle shadows, there's > perfectly crisp text (the main reason I no longer use terminal > editors, actually), and once you choose the right theme (Nexus and > Phoenix, Tomorrow Night for me) it's just lovely. There's this feature > where it shows you tabs -- but only for the part of the file you're > on. There's, instead of a scrollbar, a little "bird's-eye-view" of the > whole code on the RHS. This goes on. Visually it is stunning, in a > helpful way. If it had proper terminal-emulation support, I'd replace > my terminal with it. It's just that usable an interface. > > * Multiple cursors. This is something that no-one else really > advertises, but it's one of the most used features for me. It's > something you just have to try for a while -- I think it's a bit like > Vim's power-of-Regex but easy for a, you know, human. (I just found > https://github.com/terryma/vim-multiple-cursors). > > * Good navigation between and inside of files. A lot of things have > this, so I won't say much more. > > * The "Command Palette" is a dropdown that you do commands from, and > because of the way you search it, it's like a hybrid between vim's > command-based power and something that's actually discoverable and > easy. > > * Usable on *really big* files, and has one of the best binary-file > support I know of. I open binary file a little more than I should, not > that I can do much with them. > > * Useful extensions, installable at a button-press -- > in[search for package]. Like SublimeREPL. I know > Emacs/Vim will do better at REPLs, but few others will. > > * Etc. This goes on. > > Looking at Dave Angel's list, Sublime Text pretty-much aces it. > > What I don't understand is where he says: > >> The main negatives I can see are: > ... >> It's available for OS/X, Linux and Windows, with a single purchase >> The eval/demo is not time-limited (currently) > > How on earth are those negatives? > A typo. I was collecting points and trying to put them in categories, but somehow that didn't end up in the right place. > He basically only dislikes it because you have to use PayPal, which is > his choice. I can't say I agree with it though. > -- DaveA From roy at panix.com Thu Jul 4 21:50:38 2013 From: roy at panix.com (Roy Smith) Date: Thu, 04 Jul 2013 21:50:38 -0400 Subject: Important features for editors References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: In article , Joshua Landau wrote: [talking about Sublime Text] > There's, instead of a scrollbar, a little "bird's-eye-view" of the > whole code on the RHS. I've never used it myself, but there's a couple of guys in the office who do. I have to admit, this feature looks pretty neat. Does Sublime have some sort of remote mode? We've got one guy who loves it, but needs to work on a remote machine (i.e. in AWS). I got X11 working for him (he has a Mac desktop), so he can run Sublime on the AWS Linux box and have it display on his Mac desktop, but that's less than ideal. From cs at zip.com.au Thu Jul 4 22:59:03 2013 From: cs at zip.com.au (Cameron Simpson) Date: Fri, 5 Jul 2013 12:59:03 +1000 Subject: Important features for editors In-Reply-To: References: Message-ID: <20130705025903.GA86611@cskk.homeip.net> [ Digressing to tuning remote access. Sorry. - Cameron ] On 04Jul2013 21:50, Roy Smith wrote: | Does Sublime have some sort of remote mode? We've got one guy who loves | it, but needs to work on a remote machine (i.e. in AWS). I got X11 | working for him (he has a Mac desktop), so he can run Sublime on the AWS | Linux box and have it display on his Mac desktop, but that's less than | ideal. It's worth pointing out that, depending on the app, X11 can do a fair bit of traffic. Particularly with more recent things on "rich" widget libraries with animation or drag'n'drop, it can be quite painful because the app's developed on someones local desktop where latency is negligible. Sometimes it is worth running a desktop local to the remote machine (eg using vncserver) and operating it remotely via a viewer (eg a vnc viewer). Although you're now throwign screen updates around as bitmaps of some flavour, this can decouple you from vile spinning progress icons and modal drag'n'drop stuff. (This also insulates you against network drops; just reconnect the viewer, just like screen or tmux do for terminals.) Seamonkey, for example, is like molasses done directly with X11 from a remote host. It used to be snappy, but widget library bling creep has made it an exercise in pain. Another alternative, better still if easy, is to mount the remote system's files on your local machine and run the editor locally. Snappy response, your native widget-set/look'n'feel, and saving a file is normally pretty cheap even remotely; that would normally be your main remote transaction under this model. Cheers, -- Cameron Simpson Ninety percent of everything is crud. - Theodore Sturgeon From jussij at zeusedit.com Mon Jul 8 02:16:39 2013 From: jussij at zeusedit.com (jussij at zeusedit.com) Date: Sun, 7 Jul 2013 23:16:39 -0700 (PDT) Subject: Important features for editors In-Reply-To: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> Message-ID: <3a2c6b65-e964-429c-9009-73bc90b10c6d@googlegroups.com> On Thursday, July 4, 2013 5:32:59 PM UTC+10, cutems93 wrote: > I am researching on editors for my own reference. On the Windows platform there is the Zeus editor: http://www.zeusedit.com/python.html It does the standard syntax highlighting, code folding and smarting indent etc etc. It's also scriptable (in Python) making which makes it highly configurable. The keyboard is fully configurable and my keyboard mapping of choice is Brief (there's an EMACS keyboard mapping but no vim mapping). I couldn't live without the keyboard macro record and playback. The automatic ctags also helps to navigate large code bases. User defined templates help with common programming constructs like if, while, for etc. I never was a big fan of code folding but have grow to use that feature a lot. NOTE: I'm the author of Zeus, it is shareware, runs natively on the Windows platform and can run on Linux using Wine. From steve+comp.lang.python at pearwood.info Mon Jul 8 02:37:38 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 08 Jul 2013 06:37:38 GMT Subject: Important features for editors References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> <3a2c6b65-e964-429c-9009-73bc90b10c6d@googlegroups.com> Message-ID: <51da5e32$0$6512$c3e8da3$5496439d@news.astraweb.com> On Sun, 07 Jul 2013 23:16:39 -0700, jussij wrote: > I couldn't live without the keyboard macro record and playback. I used to work with a programmer who couldn't live without his insulin injections. -- Steven From skip at pobox.com Mon Jul 8 06:21:55 2013 From: skip at pobox.com (Skip Montanaro) Date: Mon, 8 Jul 2013 05:21:55 -0500 Subject: Important features for editors In-Reply-To: <51da5e32$0$6512$c3e8da3$5496439d@news.astraweb.com> References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> <3a2c6b65-e964-429c-9009-73bc90b10c6d@googlegroups.com> <51da5e32$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: >> I couldn't live without the keyboard macro record and playback. > > I used to work with a programmer who couldn't live without his insulin > injections. Hyperbole aside, two of my most common "crutches" are Emacs macros and bash history. Given how useful macros are, I find it very odd that recent versions of GNU Emacs dispensed with the old key binding to C-x c. Skip From nsivaram.net at gmail.com Mon Jul 8 10:24:07 2013 From: nsivaram.net at gmail.com (Sivaram Neelakantan) Date: Mon, 08 Jul 2013 19:54:07 +0530 Subject: Important features for editors References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> <3a2c6b65-e964-429c-9009-73bc90b10c6d@googlegroups.com> <51da5e32$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: <8738rpf7ug.fsf@gmail.com> On Mon, Jul 08 2013,Skip Montanaro wrote: >>> I couldn't live without the keyboard macro record and playback. >> >> I used to work with a programmer who couldn't live without his insulin >> injections. > > Hyperbole aside, two of my most common "crutches" are Emacs macros and > bash history. Given how useful macros are, I find it very odd that > recent versions of GNU Emacs dispensed with the old key binding to C-x > c. > > Skip Wasn't it C-x ( ? From the manual In addition to the and commands described above, Emacs also supports an older set of key bindings for defining and executing keyboard macros. To begin a macro definition, type `C-x (' (`kmacro-start-macro'); as with , a prefix argument appends this definition to the last keyboard macro. To end a macro definition, type `C-x )' (`kmacro-end-macro'). To execute the most recent macro, type `C-x e' (`kmacro-end-and-call-macro'). If you enter `C-x e' while sivaram -- From skip at pobox.com Mon Jul 8 14:03:33 2013 From: skip at pobox.com (Skip Montanaro) Date: Mon, 8 Jul 2013 13:03:33 -0500 Subject: Important features for editors In-Reply-To: <8738rpf7ug.fsf@gmail.com> References: <0eab9d68-da63-41b0-bad4-d7b4457128ce@googlegroups.com> <3a2c6b65-e964-429c-9009-73bc90b10c6d@googlegroups.com> <51da5e32$0$6512$c3e8da3$5496439d@news.astraweb.com> <8738rpf7ug.fsf@gmail.com> Message-ID: > Wasn't it C-x ( ? From the manual > > In addition to the and commands described above, Emacs > also supports an older set of key bindings for defining and executing > keyboard macros. To begin a macro definition, type `C-x (' > (`kmacro-start-macro'); as with , a prefix argument appends this > definition to the last keyboard macro. To end a macro definition, type > `C-x )' (`kmacro-end-macro'). To execute the most recent macro, type > `C-x e' (`kmacro-end-and-call-macro'). If you enter `C-x e' while (We are getting a bit off-topic, but I suppose that's not too unusual...) Thanks for pointing that out. Things moved around on me while I wasn't looking. For a long, long time, I have used C-x e as a prefix for a number of ediff commands. I imagine that comes from my old XEmacs habits. C-x c was always bound to call-last-kbd-macro, and I'm pretty sure it used to be that way in older versions of GNU Emacs. The kmacro package probably appeared while I was using XEmacs. I found that in the interim, the GNU folks inserted all sorts of extra keys, so that many things I used to do with two keystrokes are now done with three. I understand the logic of what they did (easier to increase the number of keystrokes required for some commands than to increase the number of keys on the keyboard), but prefer many things the way I used to do them. So, call-last-kbd-macro got unbound in the GNU switch to kmacro, and reasserting my preference for the way I called ediff commands meant that the new spelling of the kmacro stuff got dropped. I do use C-x ( and C-x ) to define macros. Skip From akshay.ksth at gmail.com Thu Jul 4 04:04:15 2013 From: akshay.ksth at gmail.com (HighBeliever) Date: Thu, 4 Jul 2013 01:04:15 -0700 (PDT) Subject: Access Violation Error while using Python Ctypes Message-ID: <59073a41-1728-4f5f-883b-5f414b1132d1@googlegroups.com> I used the following code to import the functions of a dll in my Python code. from ctypes import * hunspell=CDLL('C:\Nhunspell\Hunspellx64.dll') hunspell.HunspellInit.restype = POINTER(c_int) hunspell.HunspellInit.argtypes = (c_char_p, c_char_p) hunspell.HunspellSpell.argtypes = (POINTER(c_int), c_char_p) hunspell.HunspellAdd.argtypes = (POINTER(c_int), c_char_p) hunspell.HunspellSuggest.argtypes = (POINTER(c_int), POINTER(POINTER(c_char_p)), c_char_p) class Hunspell(object): def __init__(self): self.hunhandle = hunspell.HunspellInit('en_US.aff', 'en_US.dic') a=Hunspell() The class Hunspell should act as a wrapper for the function, but then I get this error when I try to run this. Traceback (most recent call last): File "C:\Users\KURO\Desktop\hunspell\text.py", line 49, in a=Hunspell() File "C:\Users\KURO\Desktop\hunspell\text.py", line 17, in __init__ self.hunhandle = hunspell.HunspellInit('en_US.aff', 'en_US.dic') WindowsError: exception: access violation reading 0x0000000001FBB000 Please help me out. From nikos at superhost.gr Thu Jul 4 04:37:08 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Thu, 04 Jul 2013 11:37:08 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte Message-ID: I just started to have this error without changing nothing in my index.html(template) and metrites.py(which ipen the template) [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Original exception was: [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Traceback (most recent call last): [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] File "/home/nikos/public_html/cgi-bin/metrites.py", line 19, in [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnResolved' [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Premature end of script headers: metrites.py Why cant it decode the starting byte? what starting byte is that? -- What is now proved was at first only imagined! From davea at davea.name Thu Jul 4 05:59:22 2013 From: davea at davea.name (Dave Angel) Date: Thu, 04 Jul 2013 05:59:22 -0400 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: On 07/04/2013 04:37 AM, ????? wrote: > I just started to have this error without changing nothing > > in my index.html(template) and metrites.py(which ipen the template) > > [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Original > exception was: [Thu Jul 04 11:35:14 2013] [error] [client > 108.162.229.97] Traceback (most recent call last): [Thu Jul 04 > 11:35:14 2013] [error] [client 108.162.229.97] File > "/home/nikos/public_html/cgi-bin/metrites.py", line 19, in > [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] host = > socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or > 'UnResolved' [Thu Jul 04 11:35:14 2013] [error] [client > 108.162.229.97] UnicodeDecodeError: 'utf-8' codec can't decode byte > 0xb6 in position 0: invalid start byte [Thu Jul 04 11:35:14 2013] > [error] [client 108.162.229.97] Premature end of script headers: > metrites.py > > > Why cant it decode the starting byte? what starting byte is that? The error message means that somebody is trying to decode a byte string into Unicode, and using the utf-8 codec for it. Only certain sequences are legal in utf-8, and the first byte of a character may not be 0xb6. So it gives an error. The question is where does this string come from. Well, the message shows the source line from metrites.py: host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnResolved' So the most likely candidate is the string in the environment named "REMOTE_ADDR" Can you display that string? it should look like "11.24.32.4" or some other valid IP address. I'm assuming Python 2.7. You should specify the python version when starting a new thread, as we (or at least I) cannot keep track of what version everyone's running. -- DaveA From nikos at superhost.gr Thu Jul 4 06:03:55 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Thu, 04 Jul 2013 13:03:55 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 4/7/2013 12:59 ??, ?/? Dave Angel ??????: > On 07/04/2013 04:37 AM, ????? wrote: >> I just started to have this error without changing nothing >> >> in my index.html(template) and metrites.py(which ipen the template) >> >> [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Original >> exception was: [Thu Jul 04 11:35:14 2013] [error] [client >> 108.162.229.97] Traceback (most recent call last): [Thu Jul 04 >> 11:35:14 2013] [error] [client 108.162.229.97] File >> "/home/nikos/public_html/cgi-bin/metrites.py", line 19, in >> [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] host = >> socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or >> 'UnResolved' [Thu Jul 04 11:35:14 2013] [error] [client >> 108.162.229.97] UnicodeDecodeError: 'utf-8' codec can't decode byte >> 0xb6 in position 0: invalid start byte [Thu Jul 04 11:35:14 2013] >> [error] [client 108.162.229.97] Premature end of script headers: >> metrites.py >> >> >> Why cant it decode the starting byte? what starting byte is that? > > The error message means that somebody is trying to decode a byte string > into Unicode, and using the utf-8 codec for it. Only certain sequences > are legal in utf-8, and the first byte of a character may not be 0xb6. > So it gives an error. The question is where does this string come from. > > > Well, the message shows the source line from metrites.py: > > host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnResolved' > > So the most likely candidate is the string in the environment named > "REMOTE_ADDR" Can you display that string? it should look like > > "11.24.32.4" > > or some other valid IP address. > > > I'm assuming Python 2.7. You should specify the python version when > starting a new thread, as we (or at least I) cannot keep track of what > version everyone's running. Ima using Python v3.3.2 Dave The host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnResolved' should not have give an error since i explicityl tell it that if it cannot resolve dns the ip to hostname to set it as "unresolved" The error appear ONLY when i CloudFlare superhost.gr If i pause tthe domain form CloudFlare then my website loads properly. -- What is now proved was at first only imagined! From davea at davea.name Thu Jul 4 06:29:25 2013 From: davea at davea.name (Dave Angel) Date: Thu, 04 Jul 2013 06:29:25 -0400 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: On 07/04/2013 06:03 AM, ????? wrote: > ???? 4/7/2013 12:59 ??, ?/? Dave Angel ??????: >> On 07/04/2013 04:37 AM, ????? wrote: >>> I just started to have this error without changing nothing >>> >>> in my index.html(template) and metrites.py(which ipen the template) >>> >>> [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Original >>> exception was: [Thu Jul 04 11:35:14 2013] [error] [client >>> 108.162.229.97] Traceback (most recent call last): [Thu Jul 04 >>> 11:35:14 2013] [error] [client 108.162.229.97] File >>> "/home/nikos/public_html/cgi-bin/metrites.py", line 19, in >>> [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] host = >>> socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or >>> 'UnResolved' [Thu Jul 04 11:35:14 2013] [error] [client >>> 108.162.229.97] UnicodeDecodeError: 'utf-8' codec can't decode byte >>> 0xb6 in position 0: invalid start byte [Thu Jul 04 11:35:14 2013] >>> [error] [client 108.162.229.97] Premature end of script headers: >>> metrites.py >>> >>> >>> Why cant it decode the starting byte? what starting byte is that? >> >> The error message means that somebody is trying to decode a byte string >> into Unicode, and using the utf-8 codec for it. Only certain sequences >> are legal in utf-8, and the first byte of a character may not be 0xb6. >> So it gives an error. The question is where does this string come from. >> >> >> Well, the message shows the source line from metrites.py: >> >> host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or >> 'UnResolved' >> >> So the most likely candidate is the string in the environment named >> "REMOTE_ADDR" Can you display that string? it should look like >> >> "11.24.32.4" >> >> or some other valid IP address. >> >> >> I'm assuming Python 2.7. You should specify the python version when >> starting a new thread, as we (or at least I) cannot keep track of what >> version everyone's running. > > Ima using Python v3.3.2 Dave > > The host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or > 'UnResolved' should not have give an error since i explicityl tell it > that if it cannot resolve dns the ip to hostname to set it as "unresolved" That's not true. The 'or' doesn't get executed until after the gethostbyaddr() call has returned. So if something's wrong with that call, and it throws an exception, the 'or "unresolved"' won't help. I don't know that this is the problem, and I'm not all familiar with these api's. But I cannot see anything else that could go wrong there to give that particular exception. Unless the hostname it's going to return is a byte string. > > The error appear ONLY when i CloudFlare superhost.gr > > If i pause tthe domain form CloudFlare then my website loads properly. > > I don't really know what CloudFlare is, and have no idea what 'pausing the form' will do. But since it has something to do with dns, perhaps it's returning an invalid host name, one that isn't encoded in utf-8. -- DaveA From ulrich.eckhardt at dominolaser.com Thu Jul 4 05:50:15 2013 From: ulrich.eckhardt at dominolaser.com (Ulrich Eckhardt) Date: Thu, 04 Jul 2013 11:50:15 +0200 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: Am 04.07.2013 10:37, schrieb ?????: > I just started to have this error without changing nothing Well, undo the nothing that you didn't change. ;) > UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: > invalid start byte > [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Premature end > of script headers: metrites.py > > Why cant it decode the starting byte? what starting byte is that? It's the 0xb6 but it's expecting the starting byte of a UTF-8 sequence. Please do some research on UTF-8, that should clear it up. You could also search for common causes of that error. Uli From nikos at superhost.gr Thu Jul 4 06:38:09 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Thu, 04 Jul 2013 13:38:09 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 4/7/2013 12:50 ??, ?/? Ulrich Eckhardt ??????: > Am 04.07.2013 10:37, schrieb ?????: >> I just started to have this error without changing nothing > > Well, undo the nothing that you didn't change. ;) > >> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: >> invalid start byte >> [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Premature end >> of script headers: metrites.py >> >> Why cant it decode the starting byte? what starting byte is that? > > It's the 0xb6 but it's expecting the starting byte of a UTF-8 sequence. > Please do some research on UTF-8, that should clear it up. You could > also search for common causes of that error. So you are also suggesting that what gesthostbyaddr() returns is not utf-8 encoded too? What character is 0xb6 anyways? -- What is now proved was at first only imagined! From rosuav at gmail.com Thu Jul 4 06:54:30 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Jul 2013 20:54:30 +1000 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: On Thu, Jul 4, 2013 at 8:38 PM, ????? wrote: > So you are also suggesting that what gesthostbyaddr() returns is not utf-8 > encoded too? > > What character is 0xb6 anyways? It isn't. It's a byte. Bytes are not characters. http://www.joelonsoftware.com/articles/Unicode.html ChrisA From python at mrabarnett.plus.com Thu Jul 4 07:06:40 2013 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 04 Jul 2013 12:06:40 +0100 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: <51D55740.8050306@mrabarnett.plus.com> On 04/07/2013 11:38, ????? wrote: > ???? 4/7/2013 12:50 ??, ?/? Ulrich Eckhardt ??????: >> Am 04.07.2013 10:37, schrieb ?????: >>> I just started to have this error without changing nothing >> >> Well, undo the nothing that you didn't change. ;) >> >>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: >>> invalid start byte >>> [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Premature end >>> of script headers: metrites.py >>> >>> Why cant it decode the starting byte? what starting byte is that? >> >> It's the 0xb6 but it's expecting the starting byte of a UTF-8 sequence. >> Please do some research on UTF-8, that should clear it up. You could >> also search for common causes of that error. > > So you are also suggesting that what gesthostbyaddr() returns is not > utf-8 encoded too? > > What character is 0xb6 anyways? > Well, it's from a bytestring, so you'll have to specify what encoding you're using! (It clearly isn't UTF-8.) If it's ISO-8859-7 (what you've previously referred to as "greek-iso"), then: >>> import unicodedata >>> unicodedata.name(b"\xb6".decode("ISO-8859-7")) 'GREEK CAPITAL LETTER ALPHA WITH TONOS' You'll need to find out where that bytestring is coming from. From nikos at superhost.gr Thu Jul 4 07:29:22 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Thu, 04 Jul 2013 14:29:22 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 4/7/2013 1:54 ??, ?/? Chris Angelico ??????: > On Thu, Jul 4, 2013 at 8:38 PM, ????? wrote: >> So you are also suggesting that what gesthostbyaddr() returns is not utf-8 >> encoded too? >> >> What character is 0xb6 anyways? > > It isn't. It's a byte. Bytes are not characters. > > http://www.joelonsoftware.com/articles/Unicode.html Well in case of utf-8 encoding for the first 127 codepoing we can safely say that a character equals a byte :) -- What is now proved was at first only imagined! From python at mrabarnett.plus.com Thu Jul 4 07:52:29 2013 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 04 Jul 2013 12:52:29 +0100 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: <51D561FD.9030907@mrabarnett.plus.com> On 04/07/2013 12:29, ????? wrote: > ???? 4/7/2013 1:54 ??, ?/? Chris Angelico ??????: >> On Thu, Jul 4, 2013 at 8:38 PM, ????? wrote: >>> So you are also suggesting that what gesthostbyaddr() returns is not utf-8 >>> encoded too? >>> >>> What character is 0xb6 anyways? >> >> It isn't. It's a byte. Bytes are not characters. >> >> http://www.joelonsoftware.com/articles/Unicode.html > > Well in case of utf-8 encoding for the first 127 codepoing we can safely > say that a character equals a byte :) > Equals? No. Bytes are not characters. (Strictly speaking, they're codepoints, not characters.) And anyway, it's the first _128_ codepoints. From rosuav at gmail.com Thu Jul 4 08:37:40 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Jul 2013 22:37:40 +1000 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: <51D561FD.9030907@mrabarnett.plus.com> References: <51D561FD.9030907@mrabarnett.plus.com> Message-ID: On Thu, Jul 4, 2013 at 9:52 PM, MRAB wrote: > On 04/07/2013 12:29, ????? wrote: >> >> ???? 4/7/2013 1:54 ??, ?/? Chris Angelico ??????: >>> >>> On Thu, Jul 4, 2013 at 8:38 PM, ????? wrote: >>>> >>>> So you are also suggesting that what gesthostbyaddr() returns is not >>>> utf-8 >>>> encoded too? >>>> >>>> What character is 0xb6 anyways? >>> >>> >>> It isn't. It's a byte. Bytes are not characters. >>> >>> http://www.joelonsoftware.com/articles/Unicode.html >> >> >> Well in case of utf-8 encoding for the first 127 codepoing we can safely >> say that a character equals a byte :) >> > Equals? No. Bytes are not characters. (Strictly speaking, they're > codepoints, not characters.) > > And anyway, it's the first _128_ codepoints. As MRAB says, even if there's a 1:1 correspondence between bytes, codepoints, and characters, they're still not the same thing. Plus, 0xb6 is not in the first 128, so your statement is false and your question has no answer. Do you understand why I gave you that link? If not, go read the page linked to. ChrisA From nikos at superhost.gr Thu Jul 4 08:06:47 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Thu, 04 Jul 2013 15:06:47 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 4/7/2013 2:52 ??, ?/? MRAB ??????: > On 04/07/2013 12:29, ????? wrote: >> ???? 4/7/2013 1:54 ??, ?/? Chris Angelico ??????: >>> On Thu, Jul 4, 2013 at 8:38 PM, ????? wrote: >>>> So you are also suggesting that what gesthostbyaddr() returns is not >>>> utf-8 >>>> encoded too? >>>> >>>> What character is 0xb6 anyways? >>> >>> It isn't. It's a byte. Bytes are not characters. >>> >>> http://www.joelonsoftware.com/articles/Unicode.html >> >> Well in case of utf-8 encoding for the first 127 codepoing we can safely >> say that a character equals a byte :) >> > Equals? No. Bytes are not characters. (Strictly speaking, they're > codepoints, not characters.) > > And anyway, it's the first _128_ codepoints. Yes 0-127 = 128, i knew that! Well the relationship between characters and bytes is that: A [0-127] Unicode codepoints(characters) need 1-byte to be stored in utf-8 encoding. I think its also correct to say that the byte in the above situation is the representation of our character. -- What is now proved was at first only imagined! From nikos at superhost.gr Thu Jul 4 07:36:25 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Thu, 04 Jul 2013 14:36:25 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 4/7/2013 2:06 ??, ?/? MRAB ??????: > On 04/07/2013 11:38, ????? wrote: >> ???? 4/7/2013 12:50 ??, ?/? Ulrich Eckhardt ??????: >>> Am 04.07.2013 10:37, schrieb ?????: >>>> I just started to have this error without changing nothing >>> >>> Well, undo the nothing that you didn't change. ;) >>> >>>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: >>>> invalid start byte >>>> [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Premature >>>> end >>>> of script headers: metrites.py >>>> >>>> Why cant it decode the starting byte? what starting byte is that? >>> >>> It's the 0xb6 but it's expecting the starting byte of a UTF-8 sequence. >>> Please do some research on UTF-8, that should clear it up. You could >>> also search for common causes of that error. >> >> So you are also suggesting that what gesthostbyaddr() returns is not >> utf-8 encoded too? >> >> What character is 0xb6 anyways? >> > Well, it's from a bytestring, so you'll have to specify what encoding > you're using! (It clearly isn't UTF-8.) > > If it's ISO-8859-7 (what you've previously referred to as "greek-iso"), > then: > > >>> import unicodedata > >>> unicodedata.name(b"\xb6".decode("ISO-8859-7")) > 'GREEK CAPITAL LETTER ALPHA WITH TONOS' > > You'll need to find out where that bytestring is coming from. Right. But nowhere in my script(metrites.py) i use an '?' so i really have no clue where this is coming from. And you are right if it was a byte came from an utf-8 encoding scheme then it would be automatically decoded. The only thing i can say for use is that this problem a[[ear only when i cloudflare my domain "superhost.gr" If i un-cloudlflare it it cease to display errors. Can you tell me hpw to write the following properly: host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnResolved' so even if the function fails "unresolved" to be returned back? Somehow i need to capture the error. Or it dosnt have to do it the or operand will be returned? -- What is now proved was at first only imagined! From python at mrabarnett.plus.com Thu Jul 4 08:07:41 2013 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 04 Jul 2013 13:07:41 +0100 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: <51D5658D.9080406@mrabarnett.plus.com> On 04/07/2013 12:36, ????? wrote: > ???? 4/7/2013 2:06 ??, ?/? MRAB ??????: >> On 04/07/2013 11:38, ????? wrote: >>> ???? 4/7/2013 12:50 ??, ?/? Ulrich Eckhardt ??????: >>>> Am 04.07.2013 10:37, schrieb ?????: >>>>> I just started to have this error without changing nothing >>>> >>>> Well, undo the nothing that you didn't change. ;) >>>> >>>>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: >>>>> invalid start byte >>>>> [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Premature >>>>> end >>>>> of script headers: metrites.py >>>>> >>>>> Why cant it decode the starting byte? what starting byte is that? >>>> >>>> It's the 0xb6 but it's expecting the starting byte of a UTF-8 sequence. >>>> Please do some research on UTF-8, that should clear it up. You could >>>> also search for common causes of that error. >>> >>> So you are also suggesting that what gesthostbyaddr() returns is not >>> utf-8 encoded too? >>> >>> What character is 0xb6 anyways? >>> >> Well, it's from a bytestring, so you'll have to specify what encoding >> you're using! (It clearly isn't UTF-8.) >> >> If it's ISO-8859-7 (what you've previously referred to as "greek-iso"), >> then: >> >> >>> import unicodedata >> >>> unicodedata.name(b"\xb6".decode("ISO-8859-7")) >> 'GREEK CAPITAL LETTER ALPHA WITH TONOS' >> >> You'll need to find out where that bytestring is coming from. > > Right. > But nowhere in my script(metrites.py) i use an '?' so i really have no > clue where this is coming from. > > And you are right if it was a byte came from an utf-8 encoding scheme > then it would be automatically decoded. > > The only thing i can say for use is that this problem a[[ear only when i > cloudflare my domain "superhost.gr" > > If i un-cloudlflare it it cease to display errors. > > Can you tell me hpw to write the following properly: > > host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnResolved' > > so even if the function fails "unresolved" to be returned back? > Somehow i need to capture the error. > > Or it dosnt have to do it the or operand will be returned? > If gethostbyaddr fails, it raises socket.gaierror, (which, from Python 3.3 onwards, is a subclass of OSError), so try catching that, setting 'host' to 'UnResolved' if it's raised. Also, try printing out ascii(os.environ['REMOTE_ADDR']). From nikos at superhost.gr Thu Jul 4 08:47:23 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Thu, 04 Jul 2013 15:47:23 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 4/7/2013 3:07 ??, ?/? MRAB ??????: > On 04/07/2013 12:36, ????? wrote: >> ???? 4/7/2013 2:06 ??, ?/? MRAB ??????: >>> On 04/07/2013 11:38, ????? wrote: >>>> ???? 4/7/2013 12:50 ??, ?/? Ulrich Eckhardt ??????: >>>>> Am 04.07.2013 10:37, schrieb ?????: >>>>>> I just started to have this error without changing nothing >>>>> >>>>> Well, undo the nothing that you didn't change. ;) >>>>> >>>>>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in >>>>>> position 0: >>>>>> invalid start byte >>>>>> [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Premature >>>>>> end >>>>>> of script headers: metrites.py >>>>>> >>>>>> Why cant it decode the starting byte? what starting byte is that? >>>>> >>>>> It's the 0xb6 but it's expecting the starting byte of a UTF-8 >>>>> sequence. >>>>> Please do some research on UTF-8, that should clear it up. You could >>>>> also search for common causes of that error. >>>> >>>> So you are also suggesting that what gesthostbyaddr() returns is not >>>> utf-8 encoded too? >>>> >>>> What character is 0xb6 anyways? >>>> >>> Well, it's from a bytestring, so you'll have to specify what encoding >>> you're using! (It clearly isn't UTF-8.) >>> >>> If it's ISO-8859-7 (what you've previously referred to as "greek-iso"), >>> then: >>> >>> >>> import unicodedata >>> >>> unicodedata.name(b"\xb6".decode("ISO-8859-7")) >>> 'GREEK CAPITAL LETTER ALPHA WITH TONOS' >>> >>> You'll need to find out where that bytestring is coming from. >> >> Right. >> But nowhere in my script(metrites.py) i use an '?' so i really have no >> clue where this is coming from. >> >> And you are right if it was a byte came from an utf-8 encoding scheme >> then it would be automatically decoded. >> >> The only thing i can say for use is that this problem a[[ear only when i >> cloudflare my domain "superhost.gr" >> >> If i un-cloudlflare it it cease to display errors. >> >> Can you tell me hpw to write the following properly: >> >> host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or >> 'UnResolved' >> >> so even if the function fails "unresolved" to be returned back? >> Somehow i need to capture the error. >> >> Or it dosnt have to do it the or operand will be returned? >> > If gethostbyaddr fails, it raises socket.gaierror, (which, from Python > 3.3 onwards, is a subclass of OSError), so try catching that, setting > 'host' to 'UnResolved' if it's raised. > > Also, try printing out ascii(os.environ['REMOTE_ADDR']). > I have followed your suggestion by trying this: try: host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] except socket.gaierror: host = "UnResolved" and then re-cloudlflared "superhost.gr" domain http://superhost.gr/ gives internal server error. -- What is now proved was at first only imagined! From python at mrabarnett.plus.com Thu Jul 4 09:34:39 2013 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 04 Jul 2013 14:34:39 +0100 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: <51D579EF.9060708@mrabarnett.plus.com> On 04/07/2013 13:47, ????? wrote: > ???? 4/7/2013 3:07 ??, ?/? MRAB ??????: >> On 04/07/2013 12:36, ????? wrote: >>> ???? 4/7/2013 2:06 ??, ?/? MRAB ??????: >>>> On 04/07/2013 11:38, ????? wrote: >>>>> ???? 4/7/2013 12:50 ??, ?/? Ulrich Eckhardt ??????: >>>>>> Am 04.07.2013 10:37, schrieb ?????: >>>>>>> I just started to have this error without changing nothing >>>>>> >>>>>> Well, undo the nothing that you didn't change. ;) >>>>>> >>>>>>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in >>>>>>> position 0: >>>>>>> invalid start byte >>>>>>> [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Premature >>>>>>> end >>>>>>> of script headers: metrites.py >>>>>>> >>>>>>> Why cant it decode the starting byte? what starting byte is that? >>>>>> >>>>>> It's the 0xb6 but it's expecting the starting byte of a UTF-8 >>>>>> sequence. >>>>>> Please do some research on UTF-8, that should clear it up. You could >>>>>> also search for common causes of that error. >>>>> >>>>> So you are also suggesting that what gesthostbyaddr() returns is not >>>>> utf-8 encoded too? >>>>> >>>>> What character is 0xb6 anyways? >>>>> >>>> Well, it's from a bytestring, so you'll have to specify what encoding >>>> you're using! (It clearly isn't UTF-8.) >>>> >>>> If it's ISO-8859-7 (what you've previously referred to as "greek-iso"), >>>> then: >>>> >>>> >>> import unicodedata >>>> >>> unicodedata.name(b"\xb6".decode("ISO-8859-7")) >>>> 'GREEK CAPITAL LETTER ALPHA WITH TONOS' >>>> >>>> You'll need to find out where that bytestring is coming from. >>> >>> Right. >>> But nowhere in my script(metrites.py) i use an '?' so i really have no >>> clue where this is coming from. >>> >>> And you are right if it was a byte came from an utf-8 encoding scheme >>> then it would be automatically decoded. >>> >>> The only thing i can say for use is that this problem a[[ear only when i >>> cloudflare my domain "superhost.gr" >>> >>> If i un-cloudlflare it it cease to display errors. >>> >>> Can you tell me hpw to write the following properly: >>> >>> host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or >>> 'UnResolved' >>> >>> so even if the function fails "unresolved" to be returned back? >>> Somehow i need to capture the error. >>> >>> Or it dosnt have to do it the or operand will be returned? >>> >> If gethostbyaddr fails, it raises socket.gaierror, (which, from Python >> 3.3 onwards, is a subclass of OSError), so try catching that, setting >> 'host' to 'UnResolved' if it's raised. >> >> Also, try printing out ascii(os.environ['REMOTE_ADDR']). >> > > I have followed your suggestion by trying this: > > try: > host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] > except socket.gaierror: > host = "UnResolved" > > and then re-cloudlflared "superhost.gr" domain > > http://superhost.gr/ gives internal server error. > Try catching OSError instead. (As I said, from Python 3.3, socket.gaierror is a subclass of it.) From nikos at superhost.gr Thu Jul 4 09:38:35 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiDOk866z4EzM866?=) Date: Thu, 04 Jul 2013 16:38:35 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: <51D57ADB.3050202@superhost.gr> ???? 4/7/2013 4:34 ??, ?/? MRAB ??????: > On 04/07/2013 13:47, ????? wrote: >> ???? 4/7/2013 3:07 ??, ?/? MRAB ??????: >>> On 04/07/2013 12:36, ????? wrote: >>>> ???? 4/7/2013 2:06 ??, ?/? MRAB ??????: >>>>> On 04/07/2013 11:38, ????? wrote: >>>>>> ???? 4/7/2013 12:50 ??, ?/? Ulrich Eckhardt ??????: >>>>>>> Am 04.07.2013 10:37, schrieb ?????: >>>>>>>> I just started to have this error without changing nothing >>>>>>> >>>>>>> Well, undo the nothing that you didn't change. ;) >>>>>>> >>>>>>>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in >>>>>>>> position 0: >>>>>>>> invalid start byte >>>>>>>> [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] >>>>>>>> Premature >>>>>>>> end >>>>>>>> of script headers: metrites.py >>>>>>>> >>>>>>>> Why cant it decode the starting byte? what starting byte is that? >>>>>>> >>>>>>> It's the 0xb6 but it's expecting the starting byte of a UTF-8 >>>>>>> sequence. >>>>>>> Please do some research on UTF-8, that should clear it up. You could >>>>>>> also search for common causes of that error. >>>>>> >>>>>> So you are also suggesting that what gesthostbyaddr() returns is not >>>>>> utf-8 encoded too? >>>>>> >>>>>> What character is 0xb6 anyways? >>>>>> >>>>> Well, it's from a bytestring, so you'll have to specify what encoding >>>>> you're using! (It clearly isn't UTF-8.) >>>>> >>>>> If it's ISO-8859-7 (what you've previously referred to as >>>>> "greek-iso"), >>>>> then: >>>>> >>>>> >>> import unicodedata >>>>> >>> unicodedata.name(b"\xb6".decode("ISO-8859-7")) >>>>> 'GREEK CAPITAL LETTER ALPHA WITH TONOS' >>>>> >>>>> You'll need to find out where that bytestring is coming from. >>>> >>>> Right. >>>> But nowhere in my script(metrites.py) i use an '?' so i really have no >>>> clue where this is coming from. >>>> >>>> And you are right if it was a byte came from an utf-8 encoding scheme >>>> then it would be automatically decoded. >>>> >>>> The only thing i can say for use is that this problem a[[ear only >>>> when i >>>> cloudflare my domain "superhost.gr" >>>> >>>> If i un-cloudlflare it it cease to display errors. >>>> >>>> Can you tell me hpw to write the following properly: >>>> >>>> host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or >>>> 'UnResolved' >>>> >>>> so even if the function fails "unresolved" to be returned back? >>>> Somehow i need to capture the error. >>>> >>>> Or it dosnt have to do it the or operand will be returned? >>>> >>> If gethostbyaddr fails, it raises socket.gaierror, (which, from Python >>> 3.3 onwards, is a subclass of OSError), so try catching that, setting >>> 'host' to 'UnResolved' if it's raised. >>> >>> Also, try printing out ascii(os.environ['REMOTE_ADDR']). >>> >> >> I have followed your suggestion by trying this: >> >> try: >> host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] >> except socket.gaierror: >> host = "UnResolved" >> >> and then re-cloudlflared "superhost.gr" domain >> >> http://superhost.gr/ gives internal server error. >> > Try catching OSError instead. (As I said, from Python 3.3, > socket.gaierror is a subclass of it.) > At least CloudFlare doesn't give me issues: if i try this: try: host = os.environ['REMOTE_ADDR'][0] except socket.gaierror: host = "UnResolved" then i get no errors and a valid ip back but the above fails. I don't know how to catch the exception with OSError. i know only this two: except socket.gaierror: except socket.herror both fail. -- What is now proved was at first only imagined! From python at mrabarnett.plus.com Thu Jul 4 11:10:56 2013 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 04 Jul 2013 16:10:56 +0100 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: <51D57ADB.3050202@superhost.gr> References: <51D57ADB.3050202@superhost.gr> Message-ID: <51D59080.2060000@mrabarnett.plus.com> On 04/07/2013 14:38, ????? ???33? wrote: > ???? 4/7/2013 4:34 ??, ?/? MRAB ??????: >> On 04/07/2013 13:47, ????? wrote: >>> ???? 4/7/2013 3:07 ??, ?/? MRAB ??????: >>>> On 04/07/2013 12:36, ????? wrote: >>>>> ???? 4/7/2013 2:06 ??, ?/? MRAB ??????: >>>>>> On 04/07/2013 11:38, ????? wrote: >>>>>>> ???? 4/7/2013 12:50 ??, ?/? Ulrich Eckhardt ??????: >>>>>>>> Am 04.07.2013 10:37, schrieb ?????: >>>>>>>>> I just started to have this error without changing nothing >>>>>>>> >>>>>>>> Well, undo the nothing that you didn't change. ;) >>>>>>>> >>>>>>>>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in >>>>>>>>> position 0: >>>>>>>>> invalid start byte >>>>>>>>> [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] >>>>>>>>> Premature >>>>>>>>> end >>>>>>>>> of script headers: metrites.py >>>>>>>>> >>>>>>>>> Why cant it decode the starting byte? what starting byte is that? >>>>>>>> >>>>>>>> It's the 0xb6 but it's expecting the starting byte of a UTF-8 >>>>>>>> sequence. >>>>>>>> Please do some research on UTF-8, that should clear it up. You could >>>>>>>> also search for common causes of that error. >>>>>>> >>>>>>> So you are also suggesting that what gesthostbyaddr() returns is not >>>>>>> utf-8 encoded too? >>>>>>> >>>>>>> What character is 0xb6 anyways? >>>>>>> >>>>>> Well, it's from a bytestring, so you'll have to specify what encoding >>>>>> you're using! (It clearly isn't UTF-8.) >>>>>> >>>>>> If it's ISO-8859-7 (what you've previously referred to as >>>>>> "greek-iso"), >>>>>> then: >>>>>> >>>>>> >>> import unicodedata >>>>>> >>> unicodedata.name(b"\xb6".decode("ISO-8859-7")) >>>>>> 'GREEK CAPITAL LETTER ALPHA WITH TONOS' >>>>>> >>>>>> You'll need to find out where that bytestring is coming from. >>>>> >>>>> Right. >>>>> But nowhere in my script(metrites.py) i use an '?' so i really have no >>>>> clue where this is coming from. >>>>> >>>>> And you are right if it was a byte came from an utf-8 encoding scheme >>>>> then it would be automatically decoded. >>>>> >>>>> The only thing i can say for use is that this problem a[[ear only >>>>> when i >>>>> cloudflare my domain "superhost.gr" >>>>> >>>>> If i un-cloudlflare it it cease to display errors. >>>>> >>>>> Can you tell me hpw to write the following properly: >>>>> >>>>> host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or >>>>> 'UnResolved' >>>>> >>>>> so even if the function fails "unresolved" to be returned back? >>>>> Somehow i need to capture the error. >>>>> >>>>> Or it dosnt have to do it the or operand will be returned? >>>>> >>>> If gethostbyaddr fails, it raises socket.gaierror, (which, from Python >>>> 3.3 onwards, is a subclass of OSError), so try catching that, setting >>>> 'host' to 'UnResolved' if it's raised. >>>> >>>> Also, try printing out ascii(os.environ['REMOTE_ADDR']). >>>> >>> >>> I have followed your suggestion by trying this: >>> >>> try: >>> host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] >>> except socket.gaierror: >>> host = "UnResolved" >>> >>> and then re-cloudlflared "superhost.gr" domain >>> >>> http://superhost.gr/ gives internal server error. >>> >> Try catching OSError instead. (As I said, from Python 3.3, >> socket.gaierror is a subclass of it.) >> > > At least CloudFlare doesn't give me issues: > > if i try this: > > try: > host = os.environ['REMOTE_ADDR'][0] > except socket.gaierror: > host = "UnResolved" > It's pointless trying to catch a socket exception here because you're not using a socket, you're just getting a string from an environment variable. > then i get no errors and a valid ip back > > but the above fails. > > I don't know how to catch the exception with OSError. > > i know only this two: > > except socket.gaierror: > except socket.herror > > both fail. > What do you mean "I don't know how to catch the exception with OSError"? You've tried "except socket.gaierror" and "except socket.herror", well just write "except OSError" instead! From nikos at superhost.gr Thu Jul 4 11:56:14 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiDOk866z4EzM866?=) Date: Thu, 04 Jul 2013 18:56:14 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: <51D57ADB.3050202@superhost.gr> Message-ID: ???? 4/7/2013 6:10 ??, ?/? MRAB ??????: > What do you mean "I don't know how to catch the exception with > OSError"? You've tried "except socket.gaierror" and "except > socket.herror", well just write "except OSError" instead! try: host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] except OSError: host = "UnResolved" produces also an internal server error. Are you sure is just except OSError ? seems very general... -- What is now proved was at first only imagined! From wayne at waynewerner.com Fri Jul 12 07:47:53 2013 From: wayne at waynewerner.com (Wayne Werner) Date: Fri, 12 Jul 2013 06:47:53 -0500 (CDT) Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: <51D57ADB.3050202@superhost.gr> Message-ID: On Thu, 4 Jul 2013, ????? ???33? wrote: > ???? 4/7/2013 6:10 ??, ?/? MRAB ??????: >> What do you mean "I don't know how to catch the exception with >> OSError"? You've tried "except socket.gaierror" and "except >> socket.herror", well just write "except OSError" instead! > > > try: > host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] > except OSError: > host = "UnResolved" > > produces also an internal server error. > > Are you sure is just except OSError ? > Have you ensured that 'REMOTE_ADDR' is actually a key in os.environ? I highly recommend using the logging module to help diagnose what the actual exception is. HTH, -W From nikos at superhost.gr Fri Jul 12 07:56:07 2013 From: nikos at superhost.gr (Ferrous Cranus) Date: Fri, 12 Jul 2013 14:56:07 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: <51D57ADB.3050202@superhost.gr> Message-ID: ???? 12/7/2013 2:47 ??, ?/? Wayne Werner ??????: > On Thu, 4 Jul 2013, ????? ???33? wrote: > >> ???? 4/7/2013 6:10 ??, ?/? MRAB ??????: >>> What do you mean "I don't know how to catch the exception with >>> OSError"? You've tried "except socket.gaierror" and "except >>> socket.herror", well just write "except OSError" instead! >> >> >> try: >> host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] >> except OSError: >> host = "UnResolved" >> >> produces also an internal server error. >> >> Are you sure is just except OSError ? >> > > Have you ensured that 'REMOTE_ADDR' is actually a key in os.environ? I > highly recommend using the logging module to help diagnose what the > actual exception is. > > HTH, > -W Yes it is a key, but the problem as i suspected was cloudflare. i had to use os.environ['HTTP_CF_CONNECTING_IP'] that cloudflare passes as variable i the cgi enviroment in order to retrieve the visitor's ip. try: gi = pygeoip.GeoIP('/usr/local/share/GeoLiteCity.dat') city = gi.time_zone_by_addr( os.environ['HTTP_CF_CONNECTING_IP'] ) host = socket.gethostbyaddr( os.environ['HTTP_CF_CONNECTING_IP'] )[0] except Exception as e: host = repr(e) Sometimes though iam still receiving the usual UnicodeDecodeError('utf-8', b'\xc1\xf0\xef\xf4\xf5 but only for a few ip addresses, in moste cases it works. -- What is now proved was at first only imagined! From davea at davea.name Fri Jul 12 11:46:35 2013 From: davea at davea.name (Dave Angel) Date: Fri, 12 Jul 2013 11:46:35 -0400 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: <51D57ADB.3050202@superhost.gr> Message-ID: On 07/12/2013 07:56 AM, Ferrous Cranus wrote: > ???? 12/7/2013 2:47 ??, ?/? Wayne Werner ??????: >> On Thu, 4 Jul 2013, ????? ???33? wrote: >> >>> ???? 4/7/2013 6:10 ??, ?/? MRAB ??????: >>>> What do you mean "I don't know how to catch the exception with >>>> OSError"? You've tried "except socket.gaierror" and "except >>>> socket.herror", well just write "except OSError" instead! >>> >>> >>> try: >>> host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] >>> except OSError: >>> host = "UnResolved" >>> >>> produces also an internal server error. >>> >>> Are you sure is just except OSError ? >>> >> >> Have you ensured that 'REMOTE_ADDR' is actually a key in os.environ? I >> highly recommend using the logging module to help diagnose what the >> actual exception is. >> >> HTH, >> -W > > Yes it is a key, but the problem as i suspected was cloudflare. > i had to use os.environ['HTTP_CF_CONNECTING_IP'] that cloudflare passes > as variable i the cgi enviroment in order to retrieve the visitor's ip. > > > try: > gi = pygeoip.GeoIP('/usr/local/share/GeoLiteCity.dat') > city = gi.time_zone_by_addr( os.environ['HTTP_CF_CONNECTING_IP'] ) > host = socket.gethostbyaddr( os.environ['HTTP_CF_CONNECTING_IP'] )[0] > except Exception as e: > host = repr(e) > > > Sometimes though iam still receiving the usual > UnicodeDecodeError('utf-8', b'\xc1\xf0\xef\xf4\xf5 > > but only for a few ip addresses, in moste cases it works. And naturally, you now know how to debug those UnicodeDecodeError problems. Surely, the code you post here isn't what you actually do, because when people spend time to give you detailed advice, you actually read it, and work at understanding it. Chortle, snort. -- DaveA From nikos at superhost.gr Thu Jul 4 08:52:59 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Thu, 04 Jul 2013 15:52:59 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 4/7/2013 3:07 ??, ?/? MRAB ??????: > Also, try printing out ascii(os.environ['REMOTE_ADDR']). '108.162.229.97' is the result of: print( ascii(os.environ['REMOTE_ADDR']) ) Seems perfectly valid. and also have a PTR record, so that leaved us clueless about the internal server error. -- What is now proved was at first only imagined! From python at mrabarnett.plus.com Thu Jul 4 09:34:42 2013 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 04 Jul 2013 14:34:42 +0100 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: <51D579F2.6070101@mrabarnett.plus.com> On 04/07/2013 13:52, ????? wrote: > ???? 4/7/2013 3:07 ??, ?/? MRAB ??????: >> Also, try printing out ascii(os.environ['REMOTE_ADDR']). > > '108.162.229.97' is the result of: > > print( ascii(os.environ['REMOTE_ADDR']) ) > > Seems perfectly valid. and also have a PTR record, so that leaved us > clueless about the internal server error. > For me, socket.gethostbyaddr('108.162.229.97') raises socket.herror, which is also a subclass of OSError from Python 3.3 onwards. From ulrich.eckhardt at dominolaser.com Thu Jul 4 09:07:11 2013 From: ulrich.eckhardt at dominolaser.com (Ulrich Eckhardt) Date: Thu, 04 Jul 2013 15:07:11 +0200 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: Am 04.07.2013 12:38, schrieb ?????: > ???? 4/7/2013 12:50 ??, ?/? Ulrich Eckhardt ??????: >> Am 04.07.2013 10:37, schrieb ?????: >>> Why cant it decode the starting byte? what starting byte is that? >> >> It's the 0xb6 but it's expecting the starting byte of a UTF-8 sequence. >> Please do some research on UTF-8, that should clear it up. You could >> also search for common causes of that error. > > So you are also suggesting that what gesthostbyaddr() returns is not > utf-8 encoded too? I never said that. And do some research yourself, you were given plenty of hints. Uli From nikos at superhost.gr Thu Jul 4 09:26:30 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiDOk866z4EzM866?=) Date: Thu, 04 Jul 2013 16:26:30 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 4/7/2013 4:07 ??, ?/? Ulrich Eckhardt ??????: > Am 04.07.2013 12:38, schrieb ?????: >> ???? 4/7/2013 12:50 ??, ?/? Ulrich Eckhardt ??????: >>> Am 04.07.2013 10:37, schrieb ?????: >>>> Why cant it decode the starting byte? what starting byte is that? >>> >>> It's the 0xb6 but it's expecting the starting byte of a UTF-8 sequence. >>> Please do some research on UTF-8, that should clear it up. You could >>> also search for common causes of that error. >> >> So you are also suggesting that what gesthostbyaddr() returns is not >> utf-8 encoded too? > > I never said that. And do some research yourself, you were given plenty > of hints. > > Uli > Yes and as you can see form my responses i have tried any suggestion so far and the problem still remains unresolved. What you said implied that the string returned by the function cannot be decoded as utf-8. -- What is now proved was at first only imagined! From nikos at superhost.gr Thu Jul 4 16:25:15 2013 From: nikos at superhost.gr (Ferrous Cranus) Date: Thu, 04 Jul 2013 23:25:15 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 4/7/2013 11:08 ??, ?/? Dennis Lee Bieber ??????: > On Thu, 04 Jul 2013 13:38:09 +0300, ????? declaimed > the following: > >> What character is 0xb6 anyways? >> > It depends on the encoding... In EBCDIC it's unassigned. It's a > paragraph mark in ISO-Latin-1 (ISO-8859-1). Apparently also a paragraph > mark in ISO-Latin-9 (ISO-8859-15). > > If it is valid in UTF-8, I can't find a reference. It's not a prefix > for a multi-byte character, which implies that the previous byte should > have been something in prefix or another extended byte entry... try: host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] except: host = "Reverse DNS Failed" Is there a way to write the above so i cna print the error return when it fails? -- What is now proved was at first only imagined! From lele at metapensiero.it Thu Jul 4 16:44:10 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Thu, 04 Jul 2013 22:44:10 +0200 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte References: Message-ID: <87sizut5r9.fsf@nautilus.nautilus> Ferrous Cranus writes: > try: > host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] > except: > host = "Reverse DNS Failed" > > Is there a way to write the above so i cna print the error return when > it fails? Try something like try: host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] except Exception as e: host = "Reverse DNS Failed" print(e) ? ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From torriem at gmail.com Thu Jul 4 16:53:34 2013 From: torriem at gmail.com (Michael Torrie) Date: Thu, 04 Jul 2013 14:53:34 -0600 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: <51D5E0CE.1060007@gmail.com> On 07/04/2013 02:25 PM, Ferrous Cranus wrote: > try: > host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] > except: > host = "Reverse DNS Failed" > > Is there a way to write the above so i cna print the error return when > it fails? > Do you know what IP address causes the failure? If so write a little python program that does the socket.gethostbyaddr and run it on the command line! Debugging through the CGI interface sucks. Have you been writing python tests that you can run on the command line during your development these last weeks? From nobody at nowhere.com Thu Jul 4 20:06:42 2013 From: nobody at nowhere.com (Nobody) Date: Fri, 05 Jul 2013 01:06:42 +0100 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte References: Message-ID: On Thu, 04 Jul 2013 13:38:09 +0300, ????? wrote: > So you are also suggesting that what gesthostbyaddr() returns is not > utf-8 encoded too? The gethostbyaddr() OS function returns a byte string with no specified encoding. Python 3 will doubtless try to decode that to a character string using some (probably unspecified) encoding. Names obtained from DNS should consist entirely of ASCII characters (gethostbyname shouldn't attempt to decode internationalised names which use IDN, it should return the raw data). Names obtained by other means (e.g. /etc/hosts or Active Directory) could contain anything, but if you use non-ASCII hostnames you're asking for trouble. From nikos at superhost.gr Fri Jul 5 02:07:19 2013 From: nikos at superhost.gr (Ferrous Cranus) Date: Fri, 05 Jul 2013 09:07:19 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 5/7/2013 3:06 ??, ?/? Nobody ??????: > On Thu, 04 Jul 2013 13:38:09 +0300, ????? wrote: > >> So you are also suggesting that what gesthostbyaddr() returns is not >> utf-8 encoded too? > > The gethostbyaddr() OS function returns a byte string with no specified > encoding. Python 3 will doubtless try to decode that to a character string > using some (probably unspecified) encoding. I see, but if the function returns a byte string not inutf-8 format then how my script is uspposes to decode this byte string? And why only this error happens when i cloudflare my domain, while when i un-cloudflare it are reverse DNS are being resolves without problem. So the queston is: How come it only fails when i cloidflare the domain? Also please comment on that: host = gethostbyaddr(....) or "UnResolved" This will return the first argument that define the evaluation as being true or untrue. if function returns false the the 2nd argument. Nut if the function gives an exception will the condition return the 2nd argument or will the program fail? I was udner the impression that i could avoid error handling inside try/excepts by utilizing "or". -- What is now proved was at first only imagined! From lele at metapensiero.it Fri Jul 5 02:55:20 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Fri, 05 Jul 2013 08:55:20 +0200 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte References: Message-ID: <87zju1h4x3.fsf@nautilus.nautilus> Ferrous Cranus writes: > host = gethostbyaddr(....) or "UnResolved" > > This will return the first argument that define the evaluation as > being true or untrue. > > if function returns false the the 2nd argument. > Nut if the function gives an exception will the condition return the > 2nd argument or will the program fail? > > I was udner the impression that i could avoid error handling inside > try/excepts by utilizing "or". No, you had the wrong impression. Why don't you simply invoke the Python interpreter and try things out with that?? >>> a = 1/0 or 100 Traceback (most recent call last): File "", line 1, in ZeroDivisionError: division by zero >>> a Traceback (most recent call last): File "", line 1, in NameError: name 'a' is not defined >>> a = 0/1 or 100 >>> a 100 >>> ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From nikos at superhost.gr Fri Jul 5 03:10:12 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Fri, 05 Jul 2013 10:10:12 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 5/7/2013 9:55 ??, ?/? Lele Gaifax ??????: > Ferrous Cranus writes: > >> host = gethostbyaddr(....) or "UnResolved" >> >> This will return the first argument that define the evaluation as >> being true or untrue. >> >> if function returns false the the 2nd argument. >> Nut if the function gives an exception will the condition return the >> 2nd argument or will the program fail? >> >> I was udner the impression that i could avoid error handling inside >> try/excepts by utilizing "or". > > No, you had the wrong impression. Why don't you simply invoke the Python > interpreter and try things out with that?? > > >>> a = 1/0 or 100 > Traceback (most recent call last): > File "", line 1, in > ZeroDivisionError: division by zero > >>> a > Traceback (most recent call last): > File "", line 1, in > NameError: name 'a' is not defined > >>> a = 0/1 or 100 > >>> a > 100 > >>> Thank you Lele, i wanted to but i had no idea how to test it. Your devision by zero is very smart thing to test! So it proves that a condition cannot be evaluation as truthy or falsey if one of the operators is giving out an exception. Thank you. -- What is now proved was at first only imagined! From nikos at superhost.gr Fri Jul 5 02:51:27 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Fri, 05 Jul 2013 09:51:27 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 5/7/2013 3:06 ??, ?/? Nobody ??????: > On Thu, 04 Jul 2013 13:38:09 +0300, ????? wrote: > >> So you are also suggesting that what gesthostbyaddr() returns is not >> utf-8 encoded too? > > The gethostbyaddr() OS function returns a byte string with no specified > encoding. Python 3 will doubtless try to decode that to a character string > using some (probably unspecified) encoding. > > Names obtained from DNS should consist entirely of ASCII characters > (gethostbyname shouldn't attempt to decode internationalised names > which use IDN, it should return the raw data). > > Names obtained by other means (e.g. /etc/hosts or Active Directory) could > contain anything, but if you use non-ASCII hostnames you're asking for > trouble. > Please help because i just happened to noticed that after having this code: try: host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] except Exception as e: host = "Reverse DNS Failed" all requests are being resolves, result to: Reverse DNS Failed as you can see here: http://superhost.gr/?show=log&page=index.html How can the above code not be able to reeverse dns any more and it falls back to the failed string? -- What is now proved was at first only imagined! From lele at metapensiero.it Fri Jul 5 03:06:22 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Fri, 05 Jul 2013 09:06:22 +0200 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte References: Message-ID: <87vc4ph4ep.fsf@nautilus.nautilus> ????? Gr33k writes: > try: > host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] > except Exception as e: > host = "Reverse DNS Failed" > > How can the above code not be able to reeverse dns any more and it > falls back to the failed string? The only way to know is actually printing out the exception, either to stderr, or better using the logging facility, as I suggested. FYI, your code above is (almost) exactly equivalent to the simpler try: host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] except: host = "Reverse DNS Failed" ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From benjamin.kaplan at case.edu Fri Jul 5 03:21:06 2013 From: benjamin.kaplan at case.edu (Benjamin Kaplan) Date: Fri, 5 Jul 2013 00:21:06 -0700 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: <87vc4ph4ep.fsf@nautilus.nautilus> References: <87vc4ph4ep.fsf@nautilus.nautilus> Message-ID: On Jul 5, 2013 12:12 AM, "Lele Gaifax" wrote: > > ????? Gr33k writes: > > > try: > > host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] > > except Exception as e: > > host = "Reverse DNS Failed" > > > > How can the above code not be able to reeverse dns any more and it > > falls back to the failed string? > > The only way to know is actually printing out the exception, either to > stderr, or better using the logging facility, as I suggested. > > FYI, your code above is (almost) exactly equivalent to the simpler > > try: > host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] > except: > host = "Reverse DNS Failed" > > ciao, lele. > They aren't equivalent. "except Exception" won't catch KeyboardInterrupt or SystemExit or a few others that you really don't want to catch in a generic error handler. You should almost never have a bare except. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lele at metapensiero.it Fri Jul 5 06:16:01 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Fri, 05 Jul 2013 12:16:01 +0200 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte References: <87vc4ph4ep.fsf@nautilus.nautilus> Message-ID: <87r4fdgvmm.fsf@nautilus.nautilus> Benjamin Kaplan writes: >> FYI, your code above is (almost) exactly equivalent to the simpler > > They aren't equivalent. "except Exception" won't catch KeyboardInterrupt or > SystemExit or a few others that you really don't want to catch in a generic > error handler. You should almost never have a bare except. I know, that's why I added "(almost)", I was just trying to explain why he wasn't able to see the problem. Thanks for pointing out the difference, ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From nikos at superhost.gr Fri Jul 5 03:13:59 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Fri, 05 Jul 2013 10:13:59 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 5/7/2013 10:06 ??, ?/? Lele Gaifax ??????: > try: > host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] > except: > host = "Reverse DNS Failed" Yes i uses to had it like that, until i was looking for ways to make it hold the error except Exception as e: print( e ) host = e but print( e ) in the way i used to had it doesn't print out the error, it instead gives an internal server error on browser. I must somehow take a look at the error to understand why every visitor i have gets UnResolved, but how since prints fails? -- What is now proved was at first only imagined! From davea at davea.name Fri Jul 5 04:27:23 2013 From: davea at davea.name (Dave Angel) Date: Fri, 05 Jul 2013 04:27:23 -0400 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: On 07/05/2013 03:13 AM, ????? Gr33k wrote: > ???? 5/7/2013 10:06 ??, ?/? Lele Gaifax ??????: >> try: >> host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] >> except: >> host = "Reverse DNS Failed" > > > Yes i uses to had it like that, until i was looking for ways to make it > hold the error > > except Exception as e: > print( e ) > host = e > > but print( e ) in the way i used to had it doesn't print out the error, > it instead gives an internal server error on browser. > > I must somehow take a look at the error to understand why every visitor > i have gets UnResolved, but how since prints fails? > How have you been doing it all along? Just open a console onto that server, start the appropriate version of Python interactively, and try the things we've been talking about. If it fails the same way as within the cgi environmnet, you get full visibility. Or if the problems cannot be recreated outside the cgi environment, use the log files you've been logging other problems into. Or simply open a text file for writing, and add a file= keyword parameter to the print function call. print(repr(e), file=myfile) -- DaveA From nikos at superhost.gr Fri Jul 5 05:01:18 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Fri, 05 Jul 2013 12:01:18 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 5/7/2013 11:27 ??, ?/? Dave Angel ??????: > > Or if the problems cannot be recreated outside the cgi environment, use > the log files you've been logging other problems into. Or simply open a > text file for writing, and add a file= keyword parameter to the print > function call. > > print(repr(e), file=myfile) Yes you are correct, problem need to be recreated within the cgi env. try: remadd = os.environ('REMOTE_ADDR') tuple3 = socket.gethostbyaddr(remadd) host = tuple3[0] except Exception as e: host = repr(e) http://superhost.gr/?show=log&page=index.html shows explicitly the same kind of error that python interpreter via cli gave me the same error! -- What is now proved was at first only imagined! From davea at davea.name Fri Jul 5 03:50:52 2013 From: davea at davea.name (Dave Angel) Date: Fri, 05 Jul 2013 03:50:52 -0400 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: On 07/05/2013 02:51 AM, ????? Gr33k wrote: > > Please help because i just happened to noticed that after having this code: > > try: > host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] > except Exception as e: > host = "Reverse DNS Failed" Don't ever catch a bare Exception class. Make it more specific to the particular problem you're trying to suppress. In particular, your previous problem with the utf-8 decoding will also be caught by the Exception class, so it'll get all lumped together as "Reverse DNS Failed". > > > all requests are being resolves, result to: > > > Reverse DNS Failed as you can see here: > http://superhost.gr/?show=log&page=index.html > > How can the above code not be able to reeverse dns any more and it falls > back to the failed string? > Since you've not made any progress with all the other suggestions, how about if you decompose this line into several, and see just which one is failing? Maybe that'll tell you what's going on. In general, suppressing an exception without knowing why it's firing is a huge mistake. The line started as: > host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] refactor that to: remadd = os.environ('REMOVE_ADDR') tuple3 = socket.gethostbyaddr(remadd) host = tuple3[0] and see which one throws the exception. Then once you have that, examine the exact parameters that might be triggering the problem. In particular, figure out the exact types and values for remadd and tuple3. print(type(remadd) + " : " + repr(remadd)) Of course, print itself won't work too well in a CGI environment. But you must have solved that problem by now, either using log files or running the program excerpt in a regular console. -- DaveA From nikos at superhost.gr Fri Jul 5 04:00:21 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Fri, 05 Jul 2013 11:00:21 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 5/7/2013 10:50 ??, ?/? Dave Angel ??????: > > The line started as: > > > host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] > > refactor that to: > remadd = os.environ('REMOVE_ADDR') > tuple3 = socket.gethostbyaddr(remadd) > host = tuple3[0] > > and see which one throws the exception. Then once you have that, > examine the exact parameters that might be triggering the problem. In > particular, figure out the exact types and values for remadd and tuple3. > > print(type(remadd) + " : " + repr(remadd)) I'am not sure how iam supposed to write this: i just tried this: try: remadd = os.environ('REMOVE_ADDR') tuple3 = socket.gethostbyaddr(remadd) host = tuple3[0] except: host = type(remadd) + " : " + repr(remadd) but iam getting an internal server error. I didnt print it as you said but its the same thing host var gets printed later on. Now, why would this give an internal server error? -- What is now proved was at first only imagined! From davea at davea.name Fri Jul 5 04:35:18 2013 From: davea at davea.name (Dave Angel) Date: Fri, 05 Jul 2013 04:35:18 -0400 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: On 07/05/2013 04:00 AM, ????? Gr33k wrote: > ???? 5/7/2013 10:50 ??, ?/? Dave Angel ??????: >> >> The line started as: >> >> > host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] >> >> refactor that to: >> remadd = os.environ('REMOVE_ADDR') >> tuple3 = socket.gethostbyaddr(remadd) >> host = tuple3[0] >> >> and see which one throws the exception. Then once you have that, >> examine the exact parameters that might be triggering the problem. In >> particular, figure out the exact types and values for remadd and tuple3. >> >> print(type(remadd) + " : " + repr(remadd)) > > I'am not sure how iam supposed to write this: i just tried this: > > > try: > remadd = os.environ('REMOVE_ADDR') > tuple3 = socket.gethostbyaddr(remadd) > host = tuple3[0] > except: > host = type(remadd) + " : " + repr(remadd) > > > but iam getting an internal server error. > > I didnt print it as you said but its the same thing host var gets > printed later on. > > Now, why would this give an internal server error? > I have no idea what causes an internal server error. It's up to you to get the output of the expression to some location you can examine. Easiest way is to run those 3 lines directly on the server, not in the cgi environment. But if you don't have any debugging tools, then STOP right now and build some. Use logging, or redirect print, or do something that the server folks provide as debugging aids. But just blindly guessing is ludicrous. So also is throwing out clues by using a bare except. Assigning that string to host makes no sense at all. And neither does putting it in the except clause. You want to get that string to YOUR eyes, not to the server who might get an internal server error. -- DaveA From feedthetroll at gmx.de Fri Jul 5 04:44:29 2013 From: feedthetroll at gmx.de (feedthetroll at gmx.de) Date: Fri, 5 Jul 2013 01:44:29 -0700 (PDT) Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: <06a3bc9c-3f0f-49fd-a489-b66cee64e976@googlegroups.com> Am Freitag, 5. Juli 2013 10:00:21 UTC+2 schrieb ????? Gr33k: > ... > I'am not sure how iam supposed to write this: i just tried this: > try: > remadd = os.environ('REMOVE_ADDR') > tuple3 = socket.gethostbyaddr(remadd) > host = tuple3[0] > except: > host = type(remadd) + " : " + repr(remadd) Hey, if no one told you before: You are allowed to read what other people suggest you to do, think about it and so correct obvious typos: 'REMO*V*E_ADDR' By the way, my i cite: Am Donnerstag, 4. Juli 2013 14:52:59 UTC+2 schrieb ????? Gr33k: > ... > '108.162.229.97' is the result of: > print( ascii(os.environ['REMOTE_ADDR']) ) Am Donnerstag, 4. Juli 2013 16:48 UTC+2 schrieb ????? Gr33k: > For me, socket.gethostbyaddr('108.162.229.97') raises socket.herror, > which is also a subclass of OSError from Python 3.3 onwards. From nikos at superhost.gr Fri Jul 5 04:49:31 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Fri, 05 Jul 2013 11:49:31 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 5/7/2013 11:35 ??, ?/? Dave Angel ??????: > On 07/05/2013 04:00 AM, ????? Gr33k wrote: >> ???? 5/7/2013 10:50 ??, ?/? Dave Angel ??????: >>> >>> The line started as: >>> >>> > host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] >>> >>> refactor that to: >>> remadd = os.environ('REMOVE_ADDR') >>> tuple3 = socket.gethostbyaddr(remadd) >>> host = tuple3[0] >>> >>> and see which one throws the exception. Then once you have that, >>> examine the exact parameters that might be triggering the problem. In >>> particular, figure out the exact types and values for remadd and tuple3. >>> >>> print(type(remadd) + " : " + repr(remadd)) >> >> I'am not sure how iam supposed to write this: i just tried this: >> >> >> try: >> remadd = os.environ('REMOVE_ADDR') >> tuple3 = socket.gethostbyaddr(remadd) >> host = tuple3[0] >> except: >> host = type(remadd) + " : " + repr(remadd) >> >> >> but iam getting an internal server error. >> >> I didnt print it as you said but its the same thing host var gets >> printed later on. >> >> Now, why would this give an internal server error? >> > > I have no idea what causes an internal server error. It's up to you to > get the output of the expression to some location you can examine. > Easiest way is to run those 3 lines directly on the server, not in the > cgi environment. > > But if you don't have any debugging tools, then STOP right now and build > some. Use logging, or redirect print, or do something that the server > folks provide as debugging aids. But just blindly guessing is > ludicrous. So also is throwing out clues by using a bare except. > > Assigning that string to host makes no sense at all. And neither does > putting it in the except clause. You want to get that string to YOUR > eyes, not to the server who might get an internal server error. I don't think running it via 'cli' would help much, since its a cgi-script and ip addr function have no meaning calling them in plain our of a cgi environment but here it is: Python 3.3.2 (default, Jun 3 2013, 16:18:05) [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> remadd = os.environ('REMOVE_ADDR') Traceback (most recent call last): File "", line 1, in TypeError: '_Environ' object is not callable -- What is now proved was at first only imagined! From davea at davea.name Fri Jul 5 05:21:33 2013 From: davea at davea.name (Dave Angel) Date: Fri, 05 Jul 2013 05:21:33 -0400 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: On 07/05/2013 04:49 AM, ????? Gr33k wrote: > > I don't think running it via 'cli' would help much, since its a > cgi-script and ip addr function have no meaning calling them in plain > our of a cgi environment but here it is: > No idea how to parse "have no meaning calling them in plain our of a cgi environment" > > Python 3.3.2 (default, Jun 3 2013, 16:18:05) > [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> import os > >>> remadd = os.environ('REMOVE_ADDR') > Traceback (most recent call last): > File "", line 1, in > TypeError: '_Environ' object is not callable > But there were two problems with the code you faithfully copied from my earlier post. One was already pointed out by feedthetroll, that I accidentally changed REMOTE_ADDR to REMOVE_ADDR. The other one is perhaps more subtle; I replaced square brackets with parentheses. So try again with: >>> import os >>> remadd = os.environ['REMOTE_ADDR'] I get an error: Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.3/os.py", line 669, in __getitem__ value = self._data[self.encodekey(key)] KeyError: b'REMOTE_ADDR' but presumably your machine actually has such an environment variable. Isn't that mistake something you could easily have caught? Or were you just blindly pasting my bugs without understanding what I was trying to do with refactoring? Anyway, I can't see any reason why the rest of the sequence shouldn't behave identically from a terminal as it does in CGI. -- DaveA From nikos at superhost.gr Fri Jul 5 05:25:29 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Fri, 05 Jul 2013 12:25:29 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 5/7/2013 12:21 ??, ?/? Dave Angel ??????: > On 07/05/2013 04:49 AM, ????? Gr33k wrote: >> > > >> I don't think running it via 'cli' would help much, since its a >> cgi-script and ip addr function have no meaning calling them in plain >> our of a cgi environment but here it is: >> > > No idea how to parse "have no meaning calling them in plain our of a cgi > environment" > > >> >> Python 3.3.2 (default, Jun 3 2013, 16:18:05) >> [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import os >> >>> remadd = os.environ('REMOVE_ADDR') >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: '_Environ' object is not callable >> > But there were two problems with the code you faithfully copied from my > earlier post. One was already pointed out by feedthetroll, that I > accidentally changed REMOTE_ADDR to REMOVE_ADDR. > > The other one is perhaps more subtle; I replaced square brackets with > parentheses. > > So try again with: > > >>> import os > >>> remadd = os.environ['REMOTE_ADDR'] > > I get an error: > > Traceback (most recent call last): > File "", line 1, in > File "/usr/local/lib/python3.3/os.py", line 669, in __getitem__ > value = self._data[self.encodekey(key)] > KeyError: b'REMOTE_ADDR' > > > but presumably your machine actually has such an environment variable. > > Isn't that mistake something you could easily have caught? Or were you > just blindly pasting my bugs without understanding what I was trying to > do with refactoring? > > Anyway, I can't see any reason why the rest of the sequence shouldn't > behave identically from a terminal as it does in CGI. Yes i didnt see your typo and i have corrected it: try: remadd = os.environ('REMOTE_ADDR') tuple3 = socket.gethostbyaddr(remadd) host = tuple3[0] except Exception as e: host = repr(e) Ima still receiving the same kind of erro as i did with cli as well. You can view the error in the very first line here: http://superhost.gr/?show=log&page=index.html which yields: TypeError("'_Environ' object is not callable",) -- What is now proved was at first only imagined! From lele at metapensiero.it Fri Jul 5 06:24:54 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Fri, 05 Jul 2013 12:24:54 +0200 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte References: Message-ID: <87mwq1gv7t.fsf@nautilus.nautilus> ????? Gr33k writes: > You can view the error in the very first line here: > > http://superhost.gr/?show=log&page=index.html No, visiting that page simply emit the standard Apache error page, without details. > > which yields: TypeError("'_Environ' object is not callable",) Dave already told you the reason[1]. ?????, *read* **and** *understand* our *whole* answers to your questions, otherwise we are wasting time, you, and us! ciao, lele. [1] ?? The other one is perhaps more subtle; I replaced square brackets with parentheses.? -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From nikos at superhost.gr Fri Jul 5 06:26:55 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Fri, 05 Jul 2013 13:26:55 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 5/7/2013 12:25 ??, ?/? ????? Gr33k ??????: > > try: > remadd = os.environ('REMOTE_ADDR') > tuple3 = socket.gethostbyaddr(remadd) > host = tuple3[0] > except Exception as e: > host = repr(e) > which yields: TypeError("'_Environ' object is not callable",) Any thoufgs as to why os.environ('REMOTE_ADDR') gives the above error? I noticed that if i remove my domain from cloudflare the gethostbyaddr as it uses too months now. -- What is now proved was at first only imagined! From lele at metapensiero.it Fri Jul 5 06:36:58 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Fri, 05 Jul 2013 12:36:58 +0200 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte References: Message-ID: <87ehbdgunp.fsf@nautilus.nautilus> ????? Gr33k writes: > ???? 5/7/2013 12:25 ??, ?/? ????? Gr33k ??????: >> >> try: >> remadd = os.environ('REMOTE_ADDR') >> tuple3 = socket.gethostbyaddr(remadd) >> host = tuple3[0] >> except Exception as e: >> host = repr(e) > >> which yields: TypeError("'_Environ' object is not callable",) > > Any thoufgs as to why os.environ('REMOTE_ADDR') gives the above error? Yes, I'd try to understand the error message, and eventually lookup the documentation on os.environ to see what type of object it is bound to. The solution should be easy. ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From nikos at superhost.gr Fri Jul 5 06:42:36 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Fri, 05 Jul 2013 13:42:36 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 5/7/2013 1:36 ??, ?/? Lele Gaifax ??????: > ????? Gr33k writes: > >> ???? 5/7/2013 12:25 ??, ?/? ????? Gr33k ??????: >>> >>> try: >>> remadd = os.environ('REMOTE_ADDR') >>> tuple3 = socket.gethostbyaddr(remadd) >>> host = tuple3[0] >>> except Exception as e: >>> host = repr(e) >> >>> which yields: TypeError("'_Environ' object is not callable",) >> >> Any thoufgs as to why os.environ('REMOTE_ADDR') gives the above error? > > Yes, I'd try to understand the error message, and eventually lookup the > documentation on os.environ to see what type of object it is bound > to. The solution should be easy. Looks now when i print( repr(e)) i get UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3 but what string does it try to decode and jeeps failing? -- What is now proved was at first only imagined! From lele at metapensiero.it Fri Jul 5 06:59:54 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Fri, 05 Jul 2013 12:59:54 +0200 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte References: Message-ID: <87a9m1gtlh.fsf@nautilus.nautilus> ????? Gr33k writes: > Looks now when i print( repr(e)) i get > > UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3 > > but what string does it try to decode and jeeps failing? Reasonably it's the second one, as the first clearly seems the tag of the decoder that tried to translate it to Unicode. As already explained, your immediate goal should be trying to understand from *where* that byte string is coming. I can't help on that, sorry. ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From nikos at superhost.gr Fri Jul 5 07:05:07 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Fri, 05 Jul 2013 14:05:07 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 5/7/2013 1:59 ??, ?/? Lele Gaifax ??????: > ????? Gr33k writes: > >> Looks now when i print( repr(e)) i get >> >> UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3 >> >> but what string does it try to decode and jeeps failing? > > Reasonably it's the second one, as the first clearly seems the tag of > the decoder that tried to translate it to Unicode. 2nd one of what? 2nd byte in order? Can ou show me form which characters does this string consist of so we migth have an idea of where its coming from if we know what it looks like? > As already explained, your immediate goal should be trying to understand > from *where* that byte string is coming. I can't help on that, sorry. Thats what i'm trying to do. If i completely remove the gethostbyaddr fucntion adds function then there is nor problem. The problem is recreating when the script tries to decode a hostname. For some bizarre reason if i exclude my domain from CloudFlare then the reverse DNS resolution of the visitors hostname would be returned properly. I will do it right now for you to see. -- What is now proved was at first only imagined! From lele at metapensiero.it Fri Jul 5 07:16:55 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Fri, 05 Jul 2013 13:16:55 +0200 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte References: Message-ID: <8761wpgst4.fsf@nautilus.nautilus> ????? Gr33k writes: > ???? 5/7/2013 1:59 ??, ?/? Lele Gaifax ??????: >> ????? Gr33k writes: >> >>> UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3 >>> >>> but what string does it try to decode and jeeps failing? >> >> Reasonably it's the second one, as the first clearly seems the tag of >> the decoder that tried to translate it to Unicode. > > 2nd one of what? 2nd byte in order? You asked ?what string? (although you probably meant ?which string?): UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3 first string-------^^^^^^^ second string---------------^^^^^^^^^^^^^^^^^^^^^^ ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From nikos at superhost.gr Fri Jul 5 07:16:20 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Fri, 05 Jul 2013 14:16:20 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 5/7/2013 2:05 ??, ?/? ????? Gr33k ??????: > Thats what i'm trying to do. > If i completely remove the gethostbyaddr fucntion adds function then > there is nor problem. > > The problem is recreating when the script tries to decode a hostname. > > For some bizarre reason if i exclude my domain from CloudFlare then the > reverse DNS resolution of the visitors hostname would be returned properly. > > I will do it right now for you to see. Precisely as i have said it would happen: I removed the domain form CloudFlare and your domains visting my website appearing as usual. Its only when i cloudflare it and the UnicodeError happens. But why?!?! I see no reason as to why when my domain becomes Cloudflared the gethostbyaddr() fails. What kind of weird strings does it return back? -- What is now proved was at first only imagined! From nikos at superhost.gr Fri Jul 5 07:27:25 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Fri, 05 Jul 2013 14:27:25 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 5/7/2013 2:16 ??, ?/? Lele Gaifax ??????: > UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3 > > first string-------^^^^^^^ > second string---------------^^^^^^^^^^^^^^^^^^^^^^ Hold on please! From where do these dashes and carets characters come from? Also from where do you see 2 strings? Looking at that: UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3 ic an understandn onlt that utf-8 has failsed decoding some byte stream satrting with \xb6 I totally not follow... -- What is now proved was at first only imagined! From feedthetroll at gmx.de Fri Jul 5 08:06:17 2013 From: feedthetroll at gmx.de (feedthetroll at gmx.de) Date: Fri, 5 Jul 2013 05:06:17 -0700 (PDT) Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: <4e735bb4-bb5c-4f46-8b77-8fd0524c84f0@googlegroups.com> Am Freitag, 5. Juli 2013 13:27:25 UTC+2 schrieb ????? Gr33k: > ???? 5/7/2013 2:16 ??, ?/? Lele Gaifax ??????: >> UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3 >> first string-------^^^^^^^ >> second string---------------^^^^^^^^^^^^^^^^^^^^^^ > > Hold on please! > From where do these dashes and carets characters come from? ROTFL!!!! Nikos, you made my day, again! Fun is back in these threads! > > Also from where do you see 2 strings? Look, my little dumb baby: The dashes and carets point to the strings. The first one being 'utf-8', the second one starting with '\xb6\xe3\xed' (being a bytestring, therefore the b before the ') Sorry, I forgot. You are not using python for your business, therefore you can't know, that strings in python can for example be identified by surrounding '. > ic an understandn onlt that utf-8 has failsed decoding some byte stream > satrting with \xb6 So ... where did you call utf-8() so that it could try to decode something? From nikos at superhost.gr Fri Jul 5 06:28:45 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Fri, 05 Jul 2013 13:28:45 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 5/7/2013 1:24 ??, ?/? Lele Gaifax ??????: > ????? Gr33k writes: > >> You can view the error in the very first line here: >> >> http://superhost.gr/?show=log&page=index.html > > No, visiting that page simply emit the standard Apache error page, > without details. > >> >> which yields: TypeError("'_Environ' object is not callable",) > > Dave already told you the reason[1]. ?????, *read* **and** *understand* > our *whole* answers to your questions, otherwise we are wasting time, > you, and us! > > ciao, lele. > > [1] ?? The other one is perhaps more subtle; I replaced square brackets > with parentheses.? > I read carefully all of tour answer please try to load again: http://superhost.gr/?show=log&page=index.html Di i miss an explanation on this? TypeError("'_Environ' object is not callable",) -- What is now proved was at first only imagined! From nikos at superhost.gr Fri Jul 5 06:33:05 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Fri, 05 Jul 2013 13:33:05 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 5/7/2013 12:21 ??, ?/? Dave Angel ??????: > Traceback (most recent call last): > File "", line 1, in > File "/usr/local/lib/python3.3/os.py", line 669, in __getitem__ > value = self._data[self.encodekey(key)] > KeyError: b'REMOTE_ADDR Wait! Are you saying that the ip address is being returned as a byte string which then i have to decode with something like: host = socket.gethostbyaddr( os.environ['REMOTE_HOST'].decode('utf-8') )[0] ? -- What is now proved was at first only imagined! From feedthetroll at gmx.de Fri Jul 5 07:56:47 2013 From: feedthetroll at gmx.de (feedthetroll at gmx.de) Date: Fri, 5 Jul 2013 04:56:47 -0700 (PDT) Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: Am Freitag, 5. Juli 2013 12:33:05 UTC+2 schrieb ????? Gr33k: > ... > Wait! > Are you saying that the ip address is being returned as a byte string > which then i have to decode with something like: > > host = socket.gethostbyaddr( os.environ['REMOTE_HOST'].decode('utf-8') )[0] Wait! I get a decode error when python tries to automatically decode a bytestring assuming it to be utf-8 encoded. I am sure the error will disappear, when I try to decode it explicit using utf-8. Heureka! I got it! Or in other words: If a big stone falls on my foot accidently it hurts. But I am sure it will not hurt, if take that same stone and throw it on my foot. Heureka! I got it! P.S.: Am 14.06.2013 10:35, schrieb F?bio Santos: > Also you have been shown this link and I feel you really need to read it. > http://slash7.com/2006/12/22/vampires/ From feedthetroll at gmx.de Fri Jul 5 08:14:13 2013 From: feedthetroll at gmx.de (feedthetroll at gmx.de) Date: Fri, 5 Jul 2013 05:14:13 -0700 (PDT) Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: <9d1589fa-1372-4164-a33a-c96aab89a17e@googlegroups.com> Am Freitag, 5. Juli 2013 12:33:05 UTC+2 schrieb ????? Gr33k: > ... > Wait! > Are you saying that the ip address is being returned as a byte string > which then i have to decode with something like: > > host = socket.gethostbyaddr( os.environ['REMOTE_HOST'].decode('utf-8') )[0] Wait! I get a decode error when python tries to automatically decode a bytestring assuming it to be utf-8 encoded. I am sure the error will disappear, when I try to decode it explicit using utf-8. Heureka! I got it! Or in other words: If a big stone falls on my foot accidently, it hurts. ------------------------------------------^ But I am sure it will not hurt, if take that same stone and throw it on my foot. Heureka! I got it! P.S.: Am 14.06.2013 10:35, schrieb F?bio Santos: > Also you have been shown this link and I feel you really need to read it. > http://slash7.com/2006/12/22/vampires/ From davea at davea.name Fri Jul 5 09:05:33 2013 From: davea at davea.name (Dave Angel) Date: Fri, 05 Jul 2013 09:05:33 -0400 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: On 07/05/2013 06:33 AM, ????? Gr33k wrote: > ???? 5/7/2013 12:21 ??, ?/? Dave Angel ??????: >> Traceback (most recent call last): >> File "", line 1, in >> File "/usr/local/lib/python3.3/os.py", line 669, in __getitem__ >> value = self._data[self.encodekey(key)] >> KeyError: b'REMOTE_ADDR > > > Wait! > Are you saying that the ip address is being returned as a byte string > which then i have to decode with something like: > > host = socket.gethostbyaddr( os.environ['REMOTE_HOST'].decode('utf-8') )[0] > Don't fix the problem till you understand it. Figure out who is dealing with a byte string here, and where that byte string came from. Adding a decode, especially one that's going to do the same decode as your original error message, is very premature. You're quoting from my error output, and that's caused because I don't have such an environment variable. But you do. So why aren't you in there debugging it? And why on earth are you using the complex expression instead of a refactored one which might be simple enough for you to figure out what's wrong with it. There is definitely something strange going on with that os.environ reference (NOT call). So have you yet succeeded in running the factored lines? If you can't get them to run, at least up to the point that you get that unicode error, then you'll make progress only by guessing. Get to that interactive debug session, and enter the lines till you get an error. Then at least you know which line is causing the error. xxx = os.environ['REMOTE_HOST'] yyy = socket.gethostbyaddr(xxx) host = yyy[0] I'll bet the real problem is you're using some greek characters in the name of the environment variable, rather than "REMOTE_HOST" So everything you show us is laboriously retyped, hiding the real problems underneath. -- DaveA From lele at metapensiero.it Fri Jul 5 10:11:20 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Fri, 05 Jul 2013 16:11:20 +0200 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte References: Message-ID: <871u7dgkqf.fsf@nautilus.nautilus> Dave Angel writes: > You're quoting from my error output, and that's caused because I don't > have such an environment variable. But you do. Dave, maybe you already know, but that variable is "injected" by the CGI mechanism, is not coming from the OP shell environment. As ????? discovered, when he "cloudfare" (whatever that means) his site, the REMOTE_HOST envvar contains some (I guess) latin-greek encoded string, and the remote address is carried by a different envvar... ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From nikos at superhost.gr Fri Jul 5 10:27:31 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Fri, 05 Jul 2013 17:27:31 +0300 Subject: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: ???? 5/7/2013 5:11 ??, ?/? Lele Gaifax ??????: > Dave Angel writes: > >> You're quoting from my error output, and that's caused because I don't >> have such an environment variable. But you do. > > Dave, maybe you already know, but that variable is "injected" by the CGI > mechanism, is not coming from the OP shell environment. > > As ????? discovered, when he "cloudfare" (whatever that means) his site, > the REMOTE_HOST envvar contains some (I guess) latin-greek encoded > string, and the remote address is carried by a different envvar... Exactly only when i CloudFlare(www.cloudflare.com) the domain the hostname cannot be retrieved. At least i managed to solve this by: try: host = socket.gethostbyaddr( os.environ['HTTP_CF_CONNECTING_IP'] )[0] except Exception as e: host = repr(e) Seems like when you cloudflare a domain you can no longer have the originates ip address of the visitor but you have to read the above environmental variable to be bale to retrieve it! -- What is now proved was at first only imagined! From nikos at superhost.gr Thu Jul 4 06:36:17 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/Pgg==?=) Date: Thu, 04 Jul 2013 13:36:17 +0300 Subject: Fwd: Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: -------- ?????? ?????? -------- ????: Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte ??????????: Thu, 04 Jul 2013 06:29:25 -0400 ???: Dave Angel ????: python-list at python.org ?????? ?????????: comp.lang.python ????????: On 07/04/2013 06:03 AM, ????? wrote: > ???? 4/7/2013 12:59 ??, ?/? Dave Angel ??????: >> On 07/04/2013 04:37 AM, ????? wrote: >>> I just started to have this error without changing nothing >>> >>> in my index.html(template) and metrites.py(which ipen the template) >>> >>> [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] Original >>> exception was: [Thu Jul 04 11:35:14 2013] [error] [client >>> 108.162.229.97] Traceback (most recent call last): [Thu Jul 04 >>> 11:35:14 2013] [error] [client 108.162.229.97] File >>> "/home/nikos/public_html/cgi-bin/metrites.py", line 19, in >>> [Thu Jul 04 11:35:14 2013] [error] [client 108.162.229.97] host = >>> socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or >>> 'UnResolved' [Thu Jul 04 11:35:14 2013] [error] [client >>> 108.162.229.97] UnicodeDecodeError: 'utf-8' codec can't decode byte >>> 0xb6 in position 0: invalid start byte [Thu Jul 04 11:35:14 2013] >>> [error] [client 108.162.229.97] Premature end of script headers: >>> metrites.py >>> >>> >>> Why cant it decode the starting byte? what starting byte is that? >> >> The error message means that somebody is trying to decode a byte string >> into Unicode, and using the utf-8 codec for it. Only certain sequences >> are legal in utf-8, and the first byte of a character may not be 0xb6. >> So it gives an error. The question is where does this string come from. >> >> >> Well, the message shows the source line from metrites.py: >> >> host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or >> 'UnResolved' >> >> So the most likely candidate is the string in the environment named >> "REMOTE_ADDR" Can you display that string? it should look like >> >> "11.24.32.4" >> >> or some other valid IP address. >> >> >> I'm assuming Python 2.7. You should specify the python version when >> starting a new thread, as we (or at least I) cannot keep track of what >> version everyone's running. > > Ima using Python v3.3.2 Dave > > The host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or > 'UnResolved' should not have give an error since i explicityl tell it > that if it cannot resolve dns the ip to hostname to set it as "unresolved" That's not true. The 'or' doesn't get executed until after the gethostbyaddr() call has returned. So if something's wrong with that call, and it throws an exception, the 'or "unresolved"' won't help. So how do you propose to write the above statement? I was under the i impression then if for any reason the gethostbyaddr failed then or's argument will be returned instead. > > The error appear ONLY when i CloudFlare superhost.gr > > If i pause tthe domain form CloudFlare then my website loads properly. > > I don't really know what CloudFlare is, and have no idea what 'pausing the form' will do. But since it has something to do with dns, perhaps it's returning an invalid host name, one that isn't encoded in utf-8. I think so too. -- What is now proved was at first only imagined! From kanchan.n.mahajan at gmail.com Thu Jul 4 08:48:26 2013 From: kanchan.n.mahajan at gmail.com (kanchan.n.mahajan at gmail.com) Date: Thu, 4 Jul 2013 05:48:26 -0700 (PDT) Subject: Coping with cyclic imports In-Reply-To: <87bq4knmax.fsf@physik.rwth-aachen.de> References: <87bq4knmax.fsf@physik.rwth-aachen.de> Message-ID: On Tuesday, April 8, 2008 10:06:46 PM UTC+2, Torsten Bronger wrote: > Hall?chen! > > I have a rather fat module that represents a document parser -- > inline elements, block elements, and the like. Now I want to split > it into many modules to make everything more manageable. > > But at the moment I don't see how to avoid cyclic imports: A > document element A, which is represented my module parser.A, may > contain the element B, as defined in parser.B. And vice versa. So > both modules must import each other, as far as I can see. > > I know that cyclic imports work in Python under certain > circumstances. Can anyone refer me to a page which explains *when* > this works? Because at least once, the imported module was not > "finished" and thus largely unusual. > > Thank you! > > Tsch?, > Torsten. > If you do "import foo" inside bar and "import bar" inside foo, it will work fine. By the time anything actually runs, both modules will be fully loaded and will have references to each other. The problem is when instead you do "from foo import abc" and "from bar import xyz". Because now each module requires the other module to already be compiled (so that the name we are importing exists) before it can be compiled. from http://stackoverflow.com/questions/744373/circular-or-cyclic-imports-in-python From davea at davea.name Thu Jul 4 09:33:27 2013 From: davea at davea.name (Dave Angel) Date: Thu, 04 Jul 2013 09:33:27 -0400 Subject: Coping with cyclic imports In-Reply-To: References: <87bq4knmax.fsf@physik.rwth-aachen.de> Message-ID: On 07/04/2013 08:48 AM, kanchan.n.mahajan at gmail.com wrote: > On Tuesday, April 8, 2008 10:06:46 PM UTC+2, Torsten Bronger wrote: >> Hall?chen! >> >> I have a rather fat module that represents a document parser -- >> inline elements, block elements, and the like. Now I want to split >> it into many modules to make everything more manageable. >> >> But at the moment I don't see how to avoid cyclic imports: A >> document element A, which is represented my module parser.A, may >> contain the element B, as defined in parser.B. And vice versa. So >> both modules must import each other, as far as I can see. >> >> I know that cyclic imports work in Python under certain >> circumstances. Can anyone refer me to a page which explains *when* >> this works? Because at least once, the imported module was not >> "finished" and thus largely unusual. >> >> Thank you! >> >> Tsch?, >> Torsten. >> > > > If you do "import foo" inside bar and "import bar" inside foo, it will work fine. By the time anything actually runs, both modules will be fully loaded and will have references to each other. > > The problem is when instead you do "from foo import abc" and "from bar import xyz". Because now each module requires the other module to already be compiled (so that the name we are importing exists) before it can be compiled. > > from > http://stackoverflow.com/questions/744373/circular-or-cyclic-imports-in-python > That's a nice looking oversimplification. But it's not true if top-level code in the second module to be imported tries to use symbols defined in the first module. And it's definitely bad practice, with weird bugs if either one of these files is the main script being loaded. -- DaveA From oscar.j.benjamin at gmail.com Thu Jul 4 11:03:20 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 4 Jul 2013 16:03:20 +0100 Subject: Coping with cyclic imports In-Reply-To: References: <87bq4knmax.fsf@physik.rwth-aachen.de> Message-ID: On 4 July 2013 13:48, wrote: > On Tuesday, April 8, 2008 10:06:46 PM UTC+2, Torsten Bronger wrote: [snip] > > If you do "import foo" inside bar and "import bar" inside foo, it will work fine. By the time anything actually runs, both modules will be fully loaded and will have references to each other. > > The problem is when instead you do "from foo import abc" and "from bar import xyz". Because now each module requires the other module to already be compiled (so that the name we are importing exists) before it can be compiled. > > from > http://stackoverflow.com/questions/744373/circular-or-cyclic-imports-in-python Is there some reason you're responding to a post from 5 years ago? Or is it just a joke that you've created a cyclic import advice link by referring to a SO question where the top answer is actually a quote linking back to the previous post in this same thread? Oscar From cs at zip.com.au Thu Jul 4 21:24:50 2013 From: cs at zip.com.au (Cameron Simpson) Date: Fri, 5 Jul 2013 11:24:50 +1000 Subject: Coping with cyclic imports In-Reply-To: References: Message-ID: <20130705012450.GA45211@cskk.homeip.net> On 04Jul2013 16:03, Oscar Benjamin wrote: | On 4 July 2013 13:48, wrote: | > On Tuesday, April 8, 2008 10:06:46 PM UTC+2, Torsten Bronger wrote: | > http://stackoverflow.com/questions/744373/circular-or-cyclic-imports-in-python | | Is there some reason you're responding to a post from 5 years ago? Is there some reason not to, if no newer solutions are available? Certainly if someone has new input on an old Python thread, which is legitimately part of the old thread, I am _glad_ if they reply to the old post. It pulls the whole prior thread content up the top for my perusal should it be necessary. If they make a new post I have to go digging through archives if I need to do that. Cheers, -- Cameron Simpson From oscar.j.benjamin at gmail.com Fri Jul 5 05:36:17 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 5 Jul 2013 10:36:17 +0100 Subject: Coping with cyclic imports In-Reply-To: <20130705012450.GA45211@cskk.homeip.net> References: <20130705012450.GA45211@cskk.homeip.net> Message-ID: On 5 July 2013 02:24, Cameron Simpson wrote: > On 04Jul2013 16:03, Oscar Benjamin wrote: > | > | Is there some reason you're responding to a post from 5 years ago? > > Is there some reason not to, if no newer solutions are available? No, I was genuinely curious. My way of accessing this forum/newsgroup/mailing list doesn't give me a way to respond to very old posts but others seem to do it every now and again. I see now that if you're looking at an old thread in Google Groups (rather than e.g. the python.org archives) it makes the thread seem more like a forum than a newsgroup or a mailing list so that it's easy and seems more natural to respond to old posts. Oscar From cs at zip.com.au Fri Jul 5 20:47:33 2013 From: cs at zip.com.au (Cameron Simpson) Date: Sat, 6 Jul 2013 10:47:33 +1000 Subject: Coping with cyclic imports In-Reply-To: References: Message-ID: <20130706004733.GA99060@cskk.homeip.net> On 05Jul2013 10:36, Oscar Benjamin wrote: | On 5 July 2013 02:24, Cameron Simpson wrote: | > On 04Jul2013 16:03, Oscar Benjamin wrote: | > | Is there some reason you're responding to a post from 5 years ago? | > | > Is there some reason not to, if no newer solutions are available? | | No, I was genuinely curious. My way of accessing this | forum/newsgroup/mailing list doesn't give me a way to respond to very | old posts but others seem to do it every now and again. I see now that | if you're looking at an old thread in Google Groups (rather than e.g. | the python.org archives) it makes the thread seem more like a forum | than a newsgroup or a mailing list so that it's easy and seems more | natural to respond to old posts. Fair enough. Personally, like others, I stay as far away from GG as possible. Instead, when I join a mailing list (a far better way to access this IMO), I pull down the list archives and unpack them into my folder for the list. That way old posts are right there, ready for access. And if someone replies to an old thread, I've got the whole thing! Cheers, -- Cameron Simpson It is not a simple life to be a single cell, although I have no right to say so, having been a single cell so long ago myself that I have no memory at all of that stage of my life. - Lewis Thomas From kanchan.n.mahajan at gmail.com Thu Jul 4 11:11:29 2013 From: kanchan.n.mahajan at gmail.com (kanchan.n.mahajan at gmail.com) Date: Thu, 4 Jul 2013 08:11:29 -0700 (PDT) Subject: Coping with cyclic imports In-Reply-To: References: <87bq4knmax.fsf@physik.rwth-aachen.de> Message-ID: On Thursday, July 4, 2013 5:03:20 PM UTC+2, Oscar Benjamin wrote: > On 4 July 2013 13:48, wrote: > > > On Tuesday, April 8, 2008 10:06:46 PM UTC+2, Torsten Bronger wrote: > > [snip] > > > > > > If you do "import foo" inside bar and "import bar" inside foo, it will work fine. By the time anything actually runs, both modules will be fully loaded and will have references to each other. > > > > > > The problem is when instead you do "from foo import abc" and "from bar import xyz". Because now each module requires the other module to already be compiled (so that the name we are importing exists) before it can be compiled. > > > > > > from > > > http://stackoverflow.com/questions/744373/circular-or-cyclic-imports-in-python > > > > Is there some reason you're responding to a post from 5 years ago? > > > > Or is it just a joke that you've created a cyclic import advice link > > by referring to a SO question where the top answer is actually a quote > > linking back to the previous post in this same thread? > > > > > > Oscar Well I am just a new to python and sorry i didnt see the date of the question and guessing if people find this post first then it would be helpful for them to get a good answer From davea at davea.name Thu Jul 4 18:12:16 2013 From: davea at davea.name (Dave Angel) Date: Thu, 04 Jul 2013 18:12:16 -0400 Subject: Coping with cyclic imports In-Reply-To: References: <87bq4knmax.fsf@physik.rwth-aachen.de> Message-ID: On 07/04/2013 11:11 AM, kanchan.n.mahajan at gmail.com wrote: > On Thursday, July 4, 2013 5:03:20 PM UTC+2, Oscar Benjamin wrote: >> On 4 July 2013 13:48, wrote: >> >>> On Tuesday, April 8, 2008 10:06:46 PM UTC+2, Torsten Bronger wrote: >> >> [snip] >> >>> >> >>> If you do "import foo" inside bar and "import bar" inside foo, it will work fine. By the time anything actually runs, both modules will be fully loaded and will have references to each other. >> >>> >> >>> The problem is when instead you do "from foo import abc" and "from bar import xyz". Because now each module requires the other module to already be compiled (so that the name we are importing exists) before it can be compiled. >> >>> >> >>> from >> >>> http://stackoverflow.com/questions/744373/circular-or-cyclic-imports-in-python >> >> >> >> Is there some reason you're responding to a post from 5 years ago? >> >> >> >> Or is it just a joke that you've created a cyclic import advice link >> >> by referring to a SO question where the top answer is actually a quote >> >> linking back to the previous post in this same thread? >> >> >> >> >> >> Oscar > > Well I am just a new to python > and sorry i didnt see the date of the question > and guessing if people find this post first then it would be helpful for them to get a good answer > But the StackOverflow answers for this question aren't good enough. The problem is much more subtle than those answers imply, unless you make certain assumptons about the user's code. And if I'm going to do that, the thing I'm going to figure is that the rational user avoids any cyclic imports, period. if a.py imports b.py, then b.py is not permitted, directly or indirectly, to import a.py. If it does, and something doesn't work, then 50 lashes with a wet noodle. Whenever you find a cycle, try to move the common stuff into a 3rd module and have the first two import that one instead of each other. Avoid having any non-trivial code at the top level, and in no case reference anything you've imported from there. Move everything into functions, and don't have any class-static code or data initialization. If you have to break any of these (such as referring to sys.argv), make sure that the module you're using is not one that's importing you. Never, ever import the original script from another module. -- DaveA From d.poreh at gmail.com Thu Jul 4 09:07:11 2013 From: d.poreh at gmail.com (dave poreh) Date: Thu, 4 Jul 2013 06:07:11 -0700 (PDT) Subject: Google earth Message-ID: Folks, Hi, I am an IDL users and i have started to check out some pythons abilities, and turns out it is really cool. I just wondering if there is a Google earth module that we could import out lat-lon-(data) based on python codes. I have this web site (https://code.google.com/p/kdm-idl/) that i usually take my KML files to. Is there any work similar this happened in python before? Thanks for any help in advance, Cheers, Dave From asmbansal2 at gmail.com Thu Jul 4 09:23:41 2013 From: asmbansal2 at gmail.com (Aseem Bansal) Date: Thu, 4 Jul 2013 06:23:41 -0700 (PDT) Subject: Beginner - GUI devlopment in Tkinter - Any IDE with drag and drop feature like Visual Studio? Message-ID: I want to start GUI development using Tkinter in Python 2.7.5. I have been searching all over google but couldn't find any IDE that has drag-and-drop feature for Python GUI development. Tried to ask on stackoverflow (http://stackoverflow.com/questions/17439620/an-ide-with-drag-and-drop-feature-for-python-2-7-tkinter) but couldn't get an answer there. So is there any IDE that can be used for GUI developemnt and has drag-and-drop feature for Python GUI dev? I came across somewhere that eclipse's pydev plugin can be used but couldn't find anything on its website. Any advice about this? From memilanuk at gmail.com Thu Jul 4 11:24:42 2013 From: memilanuk at gmail.com (memilanuk) Date: Thu, 04 Jul 2013 08:24:42 -0700 Subject: Beginner - GUI devlopment in Tkinter - Any IDE with drag and drop feature like Visual Studio? In-Reply-To: References: Message-ID: On 07/04/2013 06:23 AM, Aseem Bansal wrote: > I want to start GUI development using Tkinter in Python 2.7.5. > > I have been searching all over google but couldn't find any IDE that > has drag-and-drop feature for Python GUI development. > For Tkinter, no luck. The general consensus always seems to be that Tkinter and/or apps written using it tend to be simple enough to not really need that much 'help'. Being a new(er) user I kind of disagree, as I think having to 'hand code' everything for the gui library included with Python detracts somewhat from its appeal to new users in that particular area. Qt Creator is available for PyQt, which may be a better pick for you depending on what you want to do. It does have a drag-n-drop interface that is very slick, but it generates a .ui file that still needs to be translated into a python module and imported into your main 'program' file - but it takes a *lot* of the tedium out of creating a user interface more complicated than a simple dialog window. HTH, Monte From davecook at nowhere.net Fri Jul 5 02:52:48 2013 From: davecook at nowhere.net (Dave Cook) Date: 05 Jul 2013 06:52:48 GMT Subject: Beginner - GUI devlopment in Tkinter - Any IDE with drag and drop feature like Visual Studio? References: Message-ID: <51d66d40$0$1609$c3e8da3$12bcf670@news.astraweb.com> On 2013-07-04, Aseem Bansal wrote: > I have been searching all over google but couldn't find any IDE that > has drag-and-drop feature for Python GUI development. If you want something like Visual Studio, you could try IronPython: http://www.ironpython.net/tools/ For any moderatley complex cross-platform application, I would suggest moving on to wxPython, PyQt, or PySide (the latter two both based on Qt). Then try wxFormBuilder, wxGlade, or Qt Designer. Generally, I'd prefer using PyQt/PySide, because Qt has the more uniform API. I use wxPython for commercial work because it had the more liberal license at the time I started using it (PySide was not available at that time). As far as features and maturity go, I think both wx and Qt are fairly equal. Qt has become more popular for scientific work. PyGtk and Glade are another option, particularly on Linux, though I think wx or Qt still have better compatibility with Win32 or OS X. For OS X only, PyObjC and XCode are an option. While they are a tool that can save time, it's pretty easy to bump into the limitations of form designers. It's best to lower ones expectations about how much coding can be offloaded onto a form designer. http://sourceforge.net/projects/wxformbuilder/ http://qt-project.org/doc/qt-4.8/designer-manual.html http://wxglade.sourceforge.net/ > I came across somewhere that eclipse's pydev plugin can be used but > couldn't find anything on its website. The only form builders for Eclipse that I'm aware of are for Swing or SWT. You would need to use Jython (if you want to stick with a Python implementation) to interface with these. Dave Cook From asmbansal2 at gmail.com Tue Jul 9 04:33:17 2013 From: asmbansal2 at gmail.com (Aseem Bansal) Date: Tue, 9 Jul 2013 01:33:17 -0700 (PDT) Subject: Beginner - GUI devlopment in Tkinter - Any IDE with drag and drop feature like Visual Studio? In-Reply-To: <51d66d40$0$1609$c3e8da3$12bcf670@news.astraweb.com> References: <51d66d40$0$1609$c3e8da3$12bcf670@news.astraweb.com> Message-ID: <330e0cae-88d6-420d-a695-51b633afc631@googlegroups.com> Thanks @Dave Cook. I'll try wxPython. From cmpython at gmail.com Wed Jul 10 01:15:14 2013 From: cmpython at gmail.com (CM) Date: Tue, 9 Jul 2013 22:15:14 -0700 (PDT) Subject: Beginner - GUI devlopment in Tkinter - Any IDE with drag and drop feature like Visual Studio? In-Reply-To: <330e0cae-88d6-420d-a695-51b633afc631@googlegroups.com> References: <51d66d40$0$1609$c3e8da3$12bcf670@news.astraweb.com> <330e0cae-88d6-420d-a695-51b633afc631@googlegroups.com> Message-ID: On Tuesday, July 9, 2013 4:33:17 AM UTC-4, Aseem Bansal wrote: > Thanks @Dave Cook. > > > > I'll try wxPython. If so, the hoary but working Boa Constructor 0.7 is a drag and drop GUI builder for wxPython applications. Well, more like click and then click again, then drag around. It's also an IDE, so it also does a lot of other stuff. ("No one" likes it any more because it hasn't been updated in years, but I like it, but I also like the 1974 AMC Matador.) From asmbansal2 at gmail.com Mon Jul 15 10:57:57 2013 From: asmbansal2 at gmail.com (Aseem Bansal) Date: Mon, 15 Jul 2013 07:57:57 -0700 (PDT) Subject: Beginner - GUI devlopment in Tkinter - Any IDE with drag and drop feature like Visual Studio? In-Reply-To: References: <51d66d40$0$1609$c3e8da3$12bcf670@news.astraweb.com> <330e0cae-88d6-420d-a695-51b633afc631@googlegroups.com> Message-ID: @CM Thanks for the suggestion. I'll take a look. From asimjalis at gmail.com Mon Jul 15 12:59:29 2013 From: asimjalis at gmail.com (asimjalis at gmail.com) Date: Mon, 15 Jul 2013 09:59:29 -0700 (PDT) Subject: Beginner - GUI devlopment in Tkinter - Any IDE with drag and drop feature like Visual Studio? In-Reply-To: References: Message-ID: <4f766d02-a3bd-4849-943e-16d297d6cb9a@googlegroups.com> Take a look at kivy at http://kivy.org. On Thursday, July 4, 2013 6:23:41 AM UTC-7, Aseem Bansal wrote: > I want to start GUI development using Tkinter in Python 2.7.5. > > > > I have been searching all over google but couldn't find any IDE that has drag-and-drop feature for Python GUI development. Tried to ask on stackoverflow > > > > (http://stackoverflow.com/questions/17439620/an-ide-with-drag-and-drop-feature-for-python-2-7-tkinter) > > > > but couldn't get an answer there. So is there any IDE that can be used for GUI developemnt and has drag-and-drop feature for Python GUI dev? > > > > I came across somewhere that eclipse's pydev plugin can be used but couldn't find anything on its website. > > > > Any advice about this? From asmbansal2 at gmail.com Sat Jul 20 03:59:00 2013 From: asmbansal2 at gmail.com (Aseem Bansal) Date: Sat, 20 Jul 2013 00:59:00 -0700 (PDT) Subject: Beginner - GUI devlopment in Tkinter - Any IDE with drag and drop feature like Visual Studio? In-Reply-To: References: Message-ID: After considering all the options suggested here I decided to use PySide/QtCreator as was suggested by Dave Cook. I created a simple GUI with QtCreator and found a way to convert .ui files to .py files. So far so good. But now I am having some confusion about the correct tools to use for PySide and I am stuck due to that. I explored the PySide wiki and discussed the confusion on Qt-forums but that didn't help much. The url of that discussion is given below(without backslashes to avoid it being shortened). Just using this on google you can easily find the discussion. qt-project.org forums viewthread 30114 Do I need to use QtCreator with PySide if I want drag-and-drop feature for GUI development? Do I need to install Qt? If yes, which version - 4.8 or 5.1? Can I use cxfreeze to make exe files from the GUI developed? I used it for pure Python files and it worked. The size of the file was big but it worked with me having to install Python on another computer. I tried to use cxfreeze on the GUI Python script that I was finally able to make. Some exe was made but no GUI started when I ran the exe. Is that a problem with cxfreeze or me having wrong tools installed? Any help is appreciated. From davecook at nowhere.net Sun Jul 21 07:14:30 2013 From: davecook at nowhere.net (Dave Cook) Date: 21 Jul 2013 11:14:30 GMT Subject: Beginner - GUI devlopment in Tkinter - Any IDE with drag and drop feature like Visual Studio? References: Message-ID: <51ebc296$0$14280$c3e8da3$f017e9df@news.astraweb.com> On 2013-07-20, Aseem Bansal wrote: > Do I need to use QtCreator with PySide if I want drag-and-drop > feature for GUI development? No, you just need the layout part of QtCreator, called QtDesigner, and any decent Python editor or IDE (e.g. Idle): http://wiki.python.org/moin/IntegratedDevelopmentEnvironments There's another Qt IDE that uses QtDesigner called Monkey Studio (seems to be PyQt only). Designers can be really nice for learning the toolkit widgets and layout, but I'd like to get away from them in my own projects, actually, and move to something like enaml: http://docs.enthought.com/enaml/ > Do I need to install Qt? If yes, which version - 4.8 or 5.1? PySide does not support 5.x yet, according to their website. I installed the PySide windows download, and it seems to work without needing any Qt packages. > Can I use cxfreeze to make exe files from the GUI developed? Probably. We use py2exe for our wxPython/numpy application. Depending on the complexity and number of external modules used by your application, bundling tools like this can require some google-fu and trial and error to get working. http://qt-project.org/wiki/Packaging_PySide_applications_on_Windows Dave Cook From kwpolska at gmail.com Sun Jul 21 07:20:35 2013 From: kwpolska at gmail.com (=?UTF-8?B?Q2hyaXMg4oCcS3dwb2xza2HigJ0gV2Fycmljaw==?=) Date: Sun, 21 Jul 2013 13:20:35 +0200 Subject: Beginner - GUI devlopment in Tkinter - Any IDE with drag and drop feature like Visual Studio? In-Reply-To: <51ebc296$0$14280$c3e8da3$f017e9df@news.astraweb.com> References: <51ebc296$0$14280$c3e8da3$f017e9df@news.astraweb.com> Message-ID: On Sun, Jul 21, 2013 at 1:14 PM, Dave Cook wrote: > On 2013-07-20, Aseem Bansal wrote: > >> Do I need to use QtCreator with PySide if I want drag-and-drop >> feature for GUI development? > > No, you just need the layout part of QtCreator, called QtDesigner, and > any decent Python editor or IDE (e.g. Idle): ?and one more thing: pyside-uic, for transforming the .ui files into (ugly) .py files. It seems to be in /PythonXY/Scripts according to Stack Overflow if you have PySide installed. -- Kwpolska | GPG KEY: 5EAAEA16 stop html mail | always bottom-post http://asciiribbon.org | http://caliburn.nl/topposting.html From torriem at gmail.com Sun Jul 21 23:56:22 2013 From: torriem at gmail.com (Michael Torrie) Date: Sun, 21 Jul 2013 21:56:22 -0600 Subject: Beginner - GUI devlopment in Tkinter - Any IDE with drag and drop feature like Visual Studio? In-Reply-To: References: <51ebc296$0$14280$c3e8da3$f017e9df@news.astraweb.com> Message-ID: <51ECAD66.8030400@gmail.com> On 07/21/2013 05:20 AM, Chris ?Kwpolska? Warrick wrote: > On Sun, Jul 21, 2013 at 1:14 PM, Dave Cook wrote: >> On 2013-07-20, Aseem Bansal wrote: >> >>> Do I need to use QtCreator with PySide if I want drag-and-drop >>> feature for GUI development? >> >> No, you just need the layout part of QtCreator, called QtDesigner, and >> any decent Python editor or IDE (e.g. Idle): > > ?and one more thing: pyside-uic, for transforming the .ui files into > (ugly) .py files. It seems to be in /PythonXY/Scripts according to > Stack Overflow if you have PySide installed. I don't think you want to be converting ui files into python classes. Just use the PySide Qt api for loading them at runtime. From davecook at nowhere.net Sun Jul 21 20:13:36 2013 From: davecook at nowhere.net (Dave Cook) Date: 22 Jul 2013 00:13:36 GMT Subject: Beginner - GUI devlopment in Tkinter - Any IDE with drag and drop feature like Visual Studio? References: <51ebc296$0$14280$c3e8da3$f017e9df@news.astraweb.com> Message-ID: <51ec7930$0$43833$c3e8da3$9deca2c3@news.astraweb.com> On 2013-07-21, Chris ?Kwpolska? Warrick wrote: > ?and one more thing: pyside-uic, for transforming the .ui files into > (ugly) .py files. It seems to be in /PythonXY/Scripts according to > Stack Overflow if you have PySide installed. Also, it looks like it's possible to directly load the .ui files: http://srinikom.github.io/pyside-docs/PySide/QtUiTools/QUiLoader.html Dave Cook From asmbansal2 at gmail.com Mon Jul 22 08:32:25 2013 From: asmbansal2 at gmail.com (Aseem Bansal) Date: Mon, 22 Jul 2013 05:32:25 -0700 (PDT) Subject: Beginner - GUI devlopment in Tkinter - Any IDE with drag and drop feature like Visual Studio? In-Reply-To: References: <51ebc296$0$14280$c3e8da3$f017e9df@news.astraweb.com> Message-ID: <4aeb747b-64b7-4d7d-8d2b-5a0daff368f3@googlegroups.com> @Chris ?Kwpolska? Warrick Yeah, as I mentioned I was able to use it to create .py files and the GUI ran. But when I made the .exe from the .py using cxfreeze it created exe but the GUI did not run. From fronagzen at gmail.com Sun Jul 21 21:02:45 2013 From: fronagzen at gmail.com (fronagzen at gmail.com) Date: Sun, 21 Jul 2013 18:02:45 -0700 (PDT) Subject: Beginner - GUI devlopment in Tkinter - Any IDE with drag and drop feature like Visual Studio? In-Reply-To: References: Message-ID: On Saturday, July 20, 2013 3:59:00 PM UTC+8, Aseem Bansal wrote: > After considering all the options suggested here I decided to use PySide/QtCreator as was suggested by Dave Cook. I created a simple GUI with QtCreator and found a way to convert .ui files to .py files. So far so good. > > But now I am having some confusion about the correct tools to use for PySide and I am stuck due to that. I explored the PySide wiki and discussed the confusion on Qt-forums but that didn't help much. The url of that discussion is given below(without backslashes to avoid it being shortened). Just using this on google you can easily find the discussion. > > qt-project.org forums viewthread 30114 > > Do I need to use QtCreator with PySide if I want drag-and-drop feature for GUI development? Do I need to install Qt? If yes, which version - 4.8 or 5.1? > > Can I use cxfreeze to make exe files from the GUI developed? I used it for pure Python files and it worked. The size of the file was big but it worked with me having to install Python on another computer. I tried to use cxfreeze on the GUI Python script that I was finally able to make. Some exe was made but no GUI started when I ran the exe. Is that a problem with cxfreeze or me having wrong tools installed? > > Any help is appreciated. It is possible to freeze PyQt programs, yes; I've succeeded before. However, cxFreeze will grab the entire PyQt library, which is past a hundred megabytes in size, so you're going to end up with an enormous file. Do you have a setup.py appropriately configured for this? You should probably do, it's rather likely that the autodetect isn't correctly identifying dependencies. cxFreeze actually lists missing modules near the beginning of its run, try and take a look at that. Unfortunately, not every single missing module is actually missing or critical, and I can't really help you identify which one. For example, for my script, I have: Missing Modules: ? BeautifulSoup imported from lxml.html.souparser ? UserDict imported from lxml.html ? __main__imported from bdb ? _gestalt imported from platform ? _posixsubprocess imported from subprocess ? etree imported from lxml.ptclasslookup ? html5lib imported from lxml.html.html5parser ? htmlentitydefs imported from lxml.html.soupparser ? sets oimported from lxml.ElementInclude ? Ssouppraser imported from lxml.html.ElementSoup ? urllib.urlencode imported from lxml.html ? urllib.urlopen imported from lxml.html ? urllib2 imported from lxml.ElementInclude ? urlparse imported from lxml.ElementInclude ? win32api imported from platform ? win32con imported from platform But the only thing I actually need to explicitly include in my setup.py is os, lxml and gzip. Go figure. From lzgcucool at gmail.com Mon Jul 22 04:54:09 2013 From: lzgcucool at gmail.com (Cucole Lee) Date: Mon, 22 Jul 2013 01:54:09 -0700 (PDT) Subject: Beginner - GUI devlopment in Tkinter - Any IDE with drag and drop feature like Visual Studio? In-Reply-To: References: Message-ID: <2d98ebd2-7092-4479-95f1-f96bb2c7d75f@googlegroups.com> Why Thinter? You can try wxpython. From kw at codebykevin.com Mon Jul 22 10:57:42 2013 From: kw at codebykevin.com (Kevin Walzer) Date: Mon, 22 Jul 2013 10:57:42 -0400 Subject: Beginner - GUI devlopment in Tkinter - Any IDE with drag and drop feature like Visual Studio? In-Reply-To: <2d98ebd2-7092-4479-95f1-f96bb2c7d75f@googlegroups.com> References: <2d98ebd2-7092-4479-95f1-f96bb2c7d75f@googlegroups.com> Message-ID: On 7/22/13 4:54 AM, Cucole Lee wrote: > Why Thinter? You can try wxpython. > Well, it's partly a matter of taste, but I for one find wxPython's API...inelegant. -- Kevin Walzer Code by Kevin/Mobile Code by Kevin http://www.codebykevin.com http://www.wtmobilesoftware.com From dwightdhutto at gmail.com Tue Jul 23 01:49:11 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Tue, 23 Jul 2013 01:49:11 -0400 Subject: Beginner - GUI devlopment in Tkinter - Any IDE with drag and drop feature like Visual Studio? In-Reply-To: References: <2d98ebd2-7092-4479-95f1-f96bb2c7d75f@googlegroups.com> Message-ID: There is the matter of how much time you want to put into this. There is the standard gtk library for python, and in the future, as soon as I'm well enough to focus, having recent problems, I'll be using the blender game engine to enhance my software to a 3d graphical form...to make my apps pop. So which tutorials are you willing to take time with, and which are benchmarked the best for your apps? For a 3d trig/quadrant cartesian 3d implementation for practice refresher, I used tkinter, and when using it was slow as hell. SO the question is, have you looked at the individual graphical tutorials for each, and decided on a look, a feel, and a time enhanced optimization yet? In other words, work a little bit with each to see which best fits your needs. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nikos at superhost.gr Thu Jul 4 10:48:43 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiDOk866z4EzM866?=) Date: Thu, 04 Jul 2013 17:48:43 +0300 Subject: Fwd: Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: References: Message-ID: -------- ?????? ?????? -------- ????: Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte ??????????: Thu, 04 Jul 2013 14:34:42 +0100 ???: MRAB ????????: python-list at python.org ????: python-list at python.org ?????? ?????????: comp.lang.python ????????: On 04/07/2013 13:52, ????? wrote: > ???? 4/7/2013 3:07 ??, ?/? MRAB ??????: >> Also, try printing out ascii(os.environ['REMOTE_ADDR']). > > '108.162.229.97' is the result of: > > print( ascii(os.environ['REMOTE_ADDR']) ) > > Seems perfectly valid. and also have a PTR record, so that leaved us > clueless about the internal server error. > For me, socket.gethostbyaddr('108.162.229.97') raises socket.herror, which is also a subclass of OSError from Python 3.3 onwards. Tell me how i should write the try/except please. -- What is now proved was at first only imagined! From rustompmody at gmail.com Thu Jul 4 13:10:07 2013 From: rustompmody at gmail.com (Rustom Mody) Date: Thu, 4 Jul 2013 22:40:07 +0530 Subject: Illegal suggestions on python list Message-ID: On Thu, Jul 4, 2013 at 9:16 PM, Steven D'Aprano < steve+comp.lang.python at pearwood.info> wrote: > On Thu, 04 Jul 2013 07:02:26 -0700, rusi wrote: > > > On Thursday, July 4, 2013 7:03:19 PM UTC+5:30, Steve Simmons wrote: > >> Boy oh boy! You really are a slow learner Nicos. You have just offered > >> to commit a crime and to include dozens of others in that crime ON A > >> PUBLIC FORUM. Please think before you post. > > > > For the record Steve, let me say, I find Robert Kern's objection barely > > sufficient. > > > > And yours less than that. > > > > Note that you are not objecting to the crime > > Which crime is that? Presumably you mean an actual criminal felony, not a > mere civil offence. Under which jurisdiction? > > > > but to the public > > expression of it. Just look at your objection from the angle of a police > > officer, and you will see that it can certainly be construed as > > abetment/collusion > > If piracy is a crime, and not just a civil offence, then surely so is > libel. You have just accused Steve Simmons of collusion in a felony, > based on absolutely no evidence at all. That's as clear a case of libel > as I've ever seen. Although I'm not sure that the damages should be > terribly high -- there's little evidence that anyone treats your > ridiculously exaggerated claims about other people's supposed crimes as > anything but hot air. > > But, putting damages aside, according to what passes for the reasoning > that you have demonstrated here, your committing libel would make you > equally a "petty criminal" as Nikos. > > > Speaking of petty, this whole witch-hunt is getting ridiculous. Don't you > have something more productive to do? Accusing people of colluding in > crimes because they fail to be sufficiently zealous in "objecting to the > crime" is nuts. > > > -- > Steven > -- > http://mail.python.org/mailman/listinfo/python-list > You've got your OO class hierarchy mixed up and then imposing that on me. See http://www.nolo.com/legal-encyclopedia/crimes-felonies-misdemeanors-infractions-classification-33814.html I did not say or suggest felony. Maybe misdemeanor or infarction. Dunno which. IANAL... -------------- next part -------------- An HTML attachment was scrubbed... URL: From feedthetroll at gmx.de Thu Jul 4 14:18:23 2013 From: feedthetroll at gmx.de (feedthetroll at gmx.de) Date: Thu, 4 Jul 2013 11:18:23 -0700 (PDT) Subject: Illegal suggestions on python list In-Reply-To: References: Message-ID: <7211aae7-02b3-44dc-84e1-58b51c447ed8@googlegroups.com> Am Donnerstag, 4. Juli 2013 19:10:07 UTC+2 schrieb rusi: > On Thu, Jul 4, 2013 at 9:16 PM, Steven D'Aprano > wrote: >> On Thu, 04 Jul 2013 07:02:26 -0700, rusi wrote: >>> ... >>> Note that you are not objecting to the crime >> >> Which crime is that? Presumably you mean an actual criminal felony, not a >> mere civil offence. Under which jurisdiction? > > You've got your OO class hierarchy mixed up and then imposing that on me. > See > http://www.nolo.com/legal-encyclopedia/crimes-felonies-misdemeanors- > infractions-classification-33814.html Oh, sorry, we forgot, that the US legal system is the only one applicable to the internt (and of course to the whole world). LOL From wuwei23 at gmail.com Thu Jul 4 21:11:27 2013 From: wuwei23 at gmail.com (alex23) Date: Fri, 05 Jul 2013 11:11:27 +1000 Subject: Illegal suggestions on python list In-Reply-To: <7211aae7-02b3-44dc-84e1-58b51c447ed8@googlegroups.com> References: <7211aae7-02b3-44dc-84e1-58b51c447ed8@googlegroups.com> Message-ID: On 5/07/2013 4:18 AM, feedthetroll at gmx.de wrote: > Oh, sorry, we forgot, that the US legal system is the only one applicable > to the internt (and of course to the whole world). Given that Australian citizens (at least) have been extradited to the US for piracy crimes that weren't commited on US soil, it's not that far from the truth, unfortunatley. From rosuav at gmail.com Thu Jul 4 18:30:57 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 5 Jul 2013 08:30:57 +1000 Subject: Illegal suggestions on python list In-Reply-To: References: Message-ID: On Fri, Jul 5, 2013 at 3:10 AM, Rustom Mody wrote: > On Thu, Jul 4, 2013 at 9:16 PM, Steven D'Aprano > wrote: >> >> Which crime is that? Presumably you mean an actual criminal felony, not a >> mere civil offence. Under which jurisdiction? >> >> If piracy is a crime, and not just a civil offence, then surely so is >> libel. > > You've got your OO class hierarchy mixed up and then imposing that on me. > > See > http://www.nolo.com/legal-encyclopedia/crimes-felonies-misdemeanors-infractions-classification-33814.html > > I did not say or suggest felony. > Maybe misdemeanor or infarction. Dunno which. IANAL... The specific terms don't really matter here, but in most jurisdictions, "felony" will be a term that unambiguously refers to a crime against the law, as distinct from what Steven is comparing against, civil damages. If I shoot you dead, I have committed murder. This is a crime. If I put up a sign saying "Entrance to " over my own door, next to yours, then I have committed no crime, but you can sue me in civil court for your loss of business resulting from my action. Whether one of them is classed as a felony in some jurisdiction or not doesn't matter, what matters is that one is simply not a crime. Violating a license agreement usually is not a crime (and if done in good faith, is often considered a bug to be fixed, not a suit to be pressed - but that does NOT apply here), hence the analogies to criminal action somewhat break down. That said, though, offering in public to rip someone off is just as unwise as offering in public to commit a crime. ChrisA From zoom at yahoo.com Fri Jul 5 02:47:27 2013 From: zoom at yahoo.com (zoom) Date: Fri, 05 Jul 2013 08:47:27 +0200 Subject: Illegal suggestions on python list In-Reply-To: References: Message-ID: On 07/05/2013 12:30 AM, Chris Angelico wrote: > On Fri, Jul 5, 2013 at 3:10 AM, Rustom Mody wrote: >> On Thu, Jul 4, 2013 at 9:16 PM, Steven D'Aprano >> wrote: >>> >>> Which crime is that? Presumably you mean an actual criminal felony, not a >>> mere civil offence. Under which jurisdiction? >>> >>> If piracy is a crime, and not just a civil offence, then surely so is >>> libel. >> >> You've got your OO class hierarchy mixed up and then imposing that on me. >> >> See >> http://www.nolo.com/legal-encyclopedia/crimes-felonies-misdemeanors-infractions-classification-33814.html >> >> I did not say or suggest felony. >> Maybe misdemeanor or infarction. Dunno which. IANAL... > > The specific terms don't really matter here, but in most > jurisdictions, "felony" will be a term that unambiguously refers to a > crime against the law, as distinct from what Steven is comparing > against, civil damages. > > If I shoot you dead, I have committed murder. This is a crime. > > If I put up a sign saying "Entrance to" over my > own door, next to yours, then I have committed no crime, but you can > sue me in civil court for your loss of business resulting from my > action. > > Whether one of them is classed as a felony in some jurisdiction or not > doesn't matter, what matters is that one is simply not a crime. > Violating a license agreement usually is not a crime (and if done in > good faith, is often considered a bug to be fixed, not a suit to be > pressed - but that does NOT apply here), hence the analogies to > criminal action somewhat break down. > > That said, though, offering in public to rip someone off is just as > unwise as offering in public to commit a crime. > > ChrisA Probably... http://en.wikipedia.org/wiki/Aaron_Swartz#JSTOR From rosuav at gmail.com Fri Jul 5 03:18:00 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 5 Jul 2013 17:18:00 +1000 Subject: Illegal suggestions on python list In-Reply-To: References: Message-ID: On Fri, Jul 5, 2013 at 4:47 PM, zoom wrote: > Probably... > > http://en.wikipedia.org/wiki/Aaron_Swartz#JSTOR Things are always more complicated than any simple Usenet post can do justice to, and it'll often come down to who has the biggest legal budget. Since he was never actually found guilty (or so a cursory reading of that Wikipedia page suggests), that can't really be taken as proof; anyone can be charged with anything if someone can trump up some plausible evidence (no, I'm not completely cynical, why do you ask?), but that doesn't make walking and chewing gum at the same time a crime. Anyway, this is all technicalities; it's still unwise to advocate legally and/or morally shady actions in public. You never know what people will find back years later. ChrisA From nikos at superhost.gr Fri Jul 5 03:23:43 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Fri, 05 Jul 2013 10:23:43 +0300 Subject: Illegal suggestions on python list In-Reply-To: References: Message-ID: ???? 5/7/2013 10:18 ??, ?/? Chris Angelico ??????: > On Fri, Jul 5, 2013 at 4:47 PM, zoom wrote: >> Probably... >> >> http://en.wikipedia.org/wiki/Aaron_Swartz#JSTOR > > Things are always more complicated than any simple Usenet post can do > justice to, and it'll often come down to who has the biggest legal > budget. Since he was never actually found guilty (or so a cursory > reading of that Wikipedia page suggests), that can't really be taken > as proof; anyone can be charged with anything if someone can trump up > some plausible evidence (no, I'm not completely cynical, why do you > ask?), but that doesn't make walking and chewing gum at the same time > a crime. > > Anyway, this is all technicalities; it's still unwise to advocate > legally and/or morally shady actions in public. You never know what > people will find back years later. > > ChrisA Chris i just wanted to offer some help to you guys because i received a lot for help here myself. It was my way to contribute to the people here. Of course i shouldn't have proposed that because yes its like ripping the company off. Of course we all know that a serial/patch/keygen/crack can be found for this great edit very easily on warez or torrentz sites so it was like a common secret to all of us. I just wanted to make things easy and felt like helping. I wont post something like that again though. -- What is now proved was at first only imagined! From rosuav at gmail.com Fri Jul 5 03:34:33 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 5 Jul 2013 17:34:33 +1000 Subject: Illegal suggestions on python list In-Reply-To: References: Message-ID: On Fri, Jul 5, 2013 at 5:23 PM, ????? Gr33k wrote: > Chris i just wanted to offer some help to you guys because i received a lot > for help here myself. Hey guys, I really appreciate what you've done, so here's what I'll do: I'm robbing the Last National Bank on Tuesday, and you can all join me! > It was my way to contribute to the people here. > Of course i shouldn't have proposed that because yes its like ripping the > company off. > > Of course we all know that a serial/patch/keygen/crack can be found for this > great edit very easily on warez or torrentz sites so it was like a common > secret to all of us. Actually no, I don't know that that's available. And if we all know, why bother saying it? Mind you, I can't honestly say that I've *never* bypassed a pay-for system. But I would not suggest it about something that I like and want to encourage (as you seem to with this editor), and if I'm going to say something in public, it's going to be worded carefully - something like this: http://rosuav.blogspot.com.au/2012/07/ea-games-dont-buy-from-them.html And that's the strongest I'll ever get on the subject, and only there because I have a VERY strong dislike for their actions. ChrisA From joshua.landau.ws at gmail.com Sat Jul 6 00:40:14 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Sat, 6 Jul 2013 05:40:14 +0100 Subject: Illegal suggestions on python list In-Reply-To: References: Message-ID: On 5 July 2013 08:34, Chris Angelico wrote: > On Fri, Jul 5, 2013 at 5:23 PM, ????? Gr33k wrote: >> Of course we all know that a serial/patch/keygen/crack can be found for this >> great edit very easily on warez or torrentz sites so it was like a common >> secret to all of us. > > Actually no, I don't know that that's available. Seriously? I thought you had internet access... /onlyhalfjoking /alsoNikosjustbuyitgoddamnit From rosuav at gmail.com Sat Jul 6 01:43:47 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 6 Jul 2013 15:43:47 +1000 Subject: Illegal suggestions on python list In-Reply-To: References: Message-ID: On Sat, Jul 6, 2013 at 2:40 PM, Joshua Landau wrote: > On 5 July 2013 08:34, Chris Angelico wrote: >> On Fri, Jul 5, 2013 at 5:23 PM, ????? Gr33k wrote: >>> Of course we all know that a serial/patch/keygen/crack can be found for this >>> great edit very easily on warez or torrentz sites so it was like a common >>> secret to all of us. >> >> Actually no, I don't know that that's available. > > Seriously? I thought you had internet access... Not exactly, no... didn't you know? I'm an entity created by the internet itself in order to program it. I AM SKYNET! At least, that's what people say on Threshold RPG, since I'm so constantly online. Maybe that doesn't apply here. Can I be a real person in one place and a construct of the internet in another? This is deeeeeep.... ChrisA From a7xrturodev at gmail.com Thu Jul 4 13:20:43 2013 From: a7xrturodev at gmail.com (Arturo B) Date: Thu, 4 Jul 2013 10:20:43 -0700 (PDT) Subject: How is this evaluated Message-ID: <81f466df-5203-48a7-97a8-643b70d9dd16@googlegroups.com> I'm making this exercise: (Python 3.3) Write a function translate() that will translate a text into "r?varspr?ket" (Swedish for "robber's language"). That is, double every consonant and place an occurrence of "o" in between. For example, translate("this is fun") should return the string "tothohisos isos fofunon". So I tried to solved it, but I couldn't, so I found many answers, but I selected this: def translate(s): consonants = 'bcdfghjklmnpqrstvwxz' return ''.join(l + 'o' + l if l in consonants else l for l in s) print(translate('hello code solver')) OUTPUT: 'hohelollolo cocodode sosololvoveror' ______________________________________________________________ So I want to question: How is the if 'h' in consonants else 'h' for 'h' in s part evaluated? (step by step please :P ) ''.join('h' + 'o' + 'h' if 'h' in consonants else 'h' for 'h' in s) Thank you guys From antoon.pardon at rece.vub.ac.be Thu Jul 4 14:05:57 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Thu, 04 Jul 2013 20:05:57 +0200 Subject: How is this evaluated In-Reply-To: <81f466df-5203-48a7-97a8-643b70d9dd16@googlegroups.com> References: <81f466df-5203-48a7-97a8-643b70d9dd16@googlegroups.com> Message-ID: <51D5B985.10703@rece.vub.ac.be> Op 04-07-13 19:20, Arturo B schreef: > I'm making this exercise: (Python 3.3) > > Write a function translate() that will translate a text into "r?varspr?ket" (Swedish for "robber's language"). That is, double every consonant and place an occurrence of "o" in between. For example, translate("this is fun") should return the string "tothohisos isos fofunon". > > So I tried to solved it, but I couldn't, so I found many answers, but I selected this: > > def translate(s): > consonants = 'bcdfghjklmnpqrstvwxz' > return ''.join(l + 'o' + l if l in consonants else l for l in s) > > print(translate('hello code solver')) > > > OUTPUT: > 'hohelollolo cocodode sosololvoveror' > > ______________________________________________________________ > So I want to question: > How is the > > if 'h' in consonants else 'h' for 'h' in s > > part evaluated? (step by step please :P ) > > ''.join('h' + 'o' + 'h' if 'h' in consonants else 'h' for 'h' in s) > > Thank you guys This doesn't make much sence because you substituted a varible for a character literal, so I'll go with the original. l + 'o' + l if l in consonants else l for l in s Evaluate this expression: l + 'o' + l if l in consonants else l for each l in s (so for each letter that in s). l + 'o' + l if l in consonants else l if l is part of the consonants produce l + 'o' + l otherwise just produce l. Hope this helps. -- Antoon Pardon From newspost2012 at gmx.de Thu Jul 4 14:07:57 2013 From: newspost2012 at gmx.de (newspost2012 at gmx.de) Date: Thu, 4 Jul 2013 11:07:57 -0700 (PDT) Subject: How is this evaluated In-Reply-To: <81f466df-5203-48a7-97a8-643b70d9dd16@googlegroups.com> References: <81f466df-5203-48a7-97a8-643b70d9dd16@googlegroups.com> Message-ID: <882c9c1a-f2aa-4af5-b473-ea0c0d61e71e@googlegroups.com> Am Donnerstag, 4. Juli 2013 19:20:43 UTC+2 schrieb Arturo B: > ... > So I want to question: > How is the > > if 'h' in consonants else 'h' for 'h' in s > > part evaluated? (step by step please :P ) Although new to python I think I can solve this (if no one contradicts, I can guess that I understood it :-) ): Take every letter from the string, one after the other, (for l in s). If you can find the letter in the string constants (if l in constants) then take it, append an "o" and then the letter again and return it (l + 'o' + l) if not, return the letter as it is (else l). Or in applicable order: take 'h' append 'o' append 'h' and return this, if you can find 'h' in constants, if not return just 'h' (and do this for every letter in string s) hth, Martin From jpiitula at ling.helsinki.fi Thu Jul 4 14:22:21 2013 From: jpiitula at ling.helsinki.fi (Jussi Piitulainen) Date: 04 Jul 2013 21:22:21 +0300 Subject: How is this evaluated References: <81f466df-5203-48a7-97a8-643b70d9dd16@googlegroups.com> Message-ID: Arturo B writes: > I'm making this exercise: (Python 3.3) > > Write a function translate() that will translate a text into > "r?varspr?ket" (Swedish for "robber's language"). That is, double > every consonant and place an occurrence of "o" in between. For > example, translate("this is fun") should return the string > "tothohisos isos fofunon". > > So I tried to solved it, but I couldn't, so I found many answers, > but I selected this: > > def translate(s): > consonants = 'bcdfghjklmnpqrstvwxz' > return ''.join(l + 'o' + l if l in consonants else l for l in s) > > print(translate('hello code solver')) > > > OUTPUT: > 'hohelollolo cocodode sosololvoveror' > > ______________________________________________________________ > So I want to question: > How is the > > if 'h' in consonants else 'h' for 'h' in s > > part evaluated? (step by step please :P ) That's nonsense in two different ways. The actual expression in the solution you found makes sense. It's a generator expression of this form: expression for l in s And the expression in it is this conditional expression: l + 'o' + l if l in consonants else l The value of this conditional expression is (l + 'o' + l) if l is in consontants, for example 'tot' if l == 't'; else it is just l, for example 'i' if l == 'i'. Fully parenthesised the whole expression is this: ((l + 'o' + l) if (l in consonants) else l) for l in s The value of this expression yields the values like 'tot' and 'hoh' and 'i' for each letter l in s in turn. > ''.join('h' + 'o' + 'h' if 'h' in consonants else 'h' for 'h' in s) This is still nonsense. To make sense, replace each 'h' with h. Then ''.join can eat up the values of a generator expression to produce the string like 'tothohi...'. You can experiment with the components of this complicated expression: >>> list(x for x in 'plaintext') >>> 'x' if True else 'y' >>> 'x' if False else 'y' >>> list((x + 'o' if x not in 'aeiouy' else x) for x in 'plaintext') >>> for x in 'plaintext': print(x + 'o' if x not in 'aeiouy' else x) I suppose you understand this: >>> ''.join(('tot', 'hoh', 'i', 'sos')) Note that a similar-looking expression in brackets [] is a list comprehension; in braces {} it is a set comprehension, or a dictionary comprehension if the value expression is a key : value pair, with the colon. And this is not all: there's nesting and filtering, too. The different uses of the keywords 'for', 'in', 'if', 'else' are a bit subtle but one sort of learns to see the whole expression. I tend to use line breaks and parentheses in such expressions: ''.join(c + 'o' + c if c in consonants else c for c in message) ''.join((c + 'o' + c if c in consonants else c) for c in message) From steve+comp.lang.python at pearwood.info Thu Jul 4 21:41:37 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 05 Jul 2013 01:41:37 GMT Subject: How is this evaluated References: <81f466df-5203-48a7-97a8-643b70d9dd16@googlegroups.com> Message-ID: <51d62451$0$29999$c3e8da3$5496439d@news.astraweb.com> On Thu, 04 Jul 2013 10:20:43 -0700, Arturo B wrote: > I'm making this exercise: (Python 3.3) > > Write a function translate() that will translate a text into > "r?varspr?ket" (Swedish for "robber's language"). That is, double every > consonant and place an occurrence of "o" in between. For example, > translate("this is fun") should return the string "tothohisos isos > fofunon". > > So I tried to solved it, but I couldn't, so I found many answers, but I > selected this: > > def translate(s): > consonants = 'bcdfghjklmnpqrstvwxz' > return ''.join(l + 'o' + l if l in consonants else l for l in s) The part inside the join(...) is called a generator expression. I do not recommend generator expressions (or list comprehensions) for beginners, because they can be confusing until you understand how they work. As a beginner, if you have a choice between a one-line piece of code that makes no sense to you, and a five-line piece of code that you understand, you should choose the one that you understand. > So I want to question: > How is the > > if 'h' in consonants else 'h' for 'h' in s > > part evaluated? (step by step please :P ) That is the wrong question, because you have split the code in the wrong places. Your generator expression looks like this: l + 'o' + l if l in consonants else l for l in s This is a "generator expression" of the form: (x for l in s) where x will be defined below. This part is easy enough to understand: Python will iterate over the string s, extracting one letter at a time l, and then evaluate the expression "x" below. What is the expression "x"? It is this part: (l + 'o' + l if l in consonants else l) which forms the "ternary operator" if-clause: value-if-true if condition-being-tested else value-if-false If you know C, that's like: ?(condition-being-tested, value-if-true, value-if-false) The condition being tested is, "l in consonants". The value if true is "l + 'o' + l". And the value if false is just l. So putting this all together, we can convert the generator expression version to this longer, but more readable, version: def translate(s): consonants = 'bcdfghjklmnpqrstvwxz' collector = [] for l in s: if l in consonants: collector.append(l + 'o' + l) else: collector.append(l) return ''.join(collector) -- Steven From rosuav at gmail.com Fri Jul 5 03:05:49 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 5 Jul 2013 17:05:49 +1000 Subject: How is this evaluated In-Reply-To: <51d62451$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <81f466df-5203-48a7-97a8-643b70d9dd16@googlegroups.com> <51d62451$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Fri, Jul 5, 2013 at 11:41 AM, Steven D'Aprano wrote: > If you know C, that's like: > > ?(condition-being-tested, value-if-true, value-if-false) Or to be precise: condition-being-tested ? value-if-true : value-if-false ChrisA From steve+comp.lang.python at pearwood.info Fri Jul 5 04:38:00 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 05 Jul 2013 08:38:00 GMT Subject: How is this evaluated References: <81f466df-5203-48a7-97a8-643b70d9dd16@googlegroups.com> <51d62451$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51d685e8$0$29999$c3e8da3$5496439d@news.astraweb.com> On Fri, 05 Jul 2013 17:05:49 +1000, Chris Angelico wrote: > On Fri, Jul 5, 2013 at 11:41 AM, Steven D'Aprano > wrote: >> If you know C, that's like: >> >> ?(condition-being-tested, value-if-true, value-if-false) > > Or to be precise: > > condition-being-tested ? value-if-true : value-if-false Oops. Sorry about that. I thought it looked wrong even as I was typing it. -- Steven From robert.winkler at bioprocess.org Thu Jul 4 13:57:25 2013 From: robert.winkler at bioprocess.org (robert.winkler at bioprocess.org) Date: Thu, 4 Jul 2013 10:57:25 -0700 (PDT) Subject: Convert SOAP response (ArrayOfInt) to Python list Message-ID: <502d34a9-551c-4939-b570-f6d7c92051bd@googlegroups.com> Thanks to the OSA library, which works for SOAP requests with Python 3.x, I can now use SOAP services at http://www.chemspider.com. The results structure is int int The result is a list of accession numbers (which correspond to chemical compounds) and I get them in the following format: (ArrayOfInt){ int[] = [ 5744, 69182, 292, 68027, 3404131, 82616, 18280, 11200, 704646, 543430 ... ] } How could I transform this to a simple python list? [5744, 69182, 292,68027, 3404131, 82616, 18280, 11200, 704646, 543430 ...] Conversion to a numpy array (and subsequent list(), or similar) does not solve the problem, since the structure is maintained; the numpy.shape returns (). Suggestions? From burak.arslan at arskom.com.tr Fri Jul 5 15:41:16 2013 From: burak.arslan at arskom.com.tr (Burak Arslan) Date: Fri, 05 Jul 2013 22:41:16 +0300 Subject: Convert SOAP response (ArrayOfInt) to Python list In-Reply-To: <502d34a9-551c-4939-b570-f6d7c92051bd@googlegroups.com> References: <502d34a9-551c-4939-b570-f6d7c92051bd@googlegroups.com> Message-ID: <51D7215C.90606@arskom.com.tr> Hi, FYI, There's a soap-specific python.org list: soap at python.org On 07/04/13 20:57, robert.winkler at bioprocess.org wrote: > Thanks to the OSA library, which works for SOAP requests with Python 3.x, I can now use SOAP services at http://www.chemspider.com. > > The results structure is > > int > int > > > The result is a list of accession numbers (which correspond to chemical compounds) and I get them in the following format: > > [snip] > > How could I transform this to a simple python list? I did not use OSA, but assuming print(ret) prints that, you should do ret.int to get your list. It should already be a regular Python list. I hope that helps. Best, Burak From vasileios44 at hotmail.com Fri Jul 5 03:48:09 2013 From: vasileios44 at hotmail.com (bill papanastasiou) Date: Fri, 5 Jul 2013 09:48:09 +0200 Subject: question please Message-ID: hello , good morning how i can p?t one python file in website ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve+comp.lang.python at pearwood.info Fri Jul 5 04:42:12 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 05 Jul 2013 08:42:12 GMT Subject: question please References: Message-ID: <51d686e4$0$29999$c3e8da3$5496439d@news.astraweb.com> On Fri, 05 Jul 2013 09:48:09 +0200, bill papanastasiou wrote: > hello , good morning > > how i can p?t one python file in website ? The same way you would put any other file in a website. Can you be more specific? What website do you want to put it on? Is it your website or somebody else's? -- Steven From davea at davea.name Fri Jul 5 04:52:07 2013 From: davea at davea.name (Dave Angel) Date: Fri, 05 Jul 2013 04:52:07 -0400 Subject: question please In-Reply-To: References: Message-ID: On 07/05/2013 03:48 AM, bill papanastasiou wrote: > hello , good morning > > how i can p?t one python file in website ? > Whose website? If it's your own, log into the server, and use cp. Or if you're remote with ssh access, use scp. And if you really have a bunch of files to remotely transfer, use rsync from your local staging area. It's smart enough to only transfer the files that are missing or changed. The type of the file doesn't matter, unless you also have to convert line-endings from Windoze to Unix. And if so, you should do it as a separate step, when you update your staging area. So the copying of the file onto the website can be done as a simple binary transfer. If none of this works, or doesn't make sense, then tell us a little more about this site, how it's being hosted, and what access you do have. In some cases the answer is to hand a USB stick to the administrator down the hall. -- DaveA From jarausch at igpm.rwth-aachen.de Fri Jul 5 04:22:51 2013 From: jarausch at igpm.rwth-aachen.de (Helmut Jarausch) Date: 5 Jul 2013 08:22:51 GMT Subject: How to make this faster Message-ID: Hi, I have coded a simple algorithm to solve a Sudoku (probably not the first one). Unfortunately, it takes 13 seconds for a difficult problem which is more than 75 times slower than the same algorithm coded in C++. Is this to be expected or could I have made my Python version faster *** without *** sacrificing readability. Profiling shows that the function find_good_cell is called (only) 45267 times and this take 12.9 seconds CPU time (on a 3.2 GHz machine) For your pleasure I include the full code below. Many thanks for a hint, Helmut #!/usr/bin/python3 import numpy as np Grid= np.zeros((9,9),dtype=int) Row_Digits = np.asarray(np.zeros((9,10)),dtype=bool) Col_Digits = np.asarray(np.zeros((9,10)),dtype=bool) Sqr_Digits = np.asarray(np.zeros((9,10)),dtype=bool) def Read_Sudoku(Input) : r= -1 R_Cells= 81 for line in Input : line= line.strip() if len(line) == 0 or line[0] == '#' : continue r+= 1 for (c,ch) in enumerate(line) : if ch == '_' : continue if not ch in "123456789_" : raise ValueError("invalid character {0} in input line {1}".format(c,line)) Sq_No= (r//3)*3+c//3 d= int(ch) Grid[r,c]= d Row_Digits[r,d]= True Col_Digits[c,d]= True Sqr_Digits[Sq_No,d]= True R_Cells-= 1 return R_Cells def Print_Grid() : Sep = "+---+---+---#---+---+---#---+---+---+" SepK= "#####################################" print(Sep) for r in range(9) : print('|',end='') for c in range(9) : d= Grid[r,c] print(" {0} {1}".format( str(d) if d>0 else ' ', '#' if (c+1)%3==0 and c>0 and c<8 else '|' ), end='') print("\n{}".format(SepK if (r+1)%3==0 and r>0 and r<8 else Sep)) def find_good_cell() : Best= None minPoss= 10 for r in range(9) : for c in range(9) : if Grid[r,c] > 0 : continue Sq_No= (r//3)*3+c//3 Possibilities= 0 for d in range(1,10) : if Row_Digits[r,d] or Col_Digits[c,d] or Sqr_Digits[Sq_No,d] : continue Possibilities+= 1 if ( Possibilities < minPoss ) : minPoss= Possibilities Best= (r,c) if minPoss == 0 : Best=(-1,-1) return Best def Solve(R_Cells) : if R_Cells == 0 : print("\n\n++++++++++ S o l u t i o n ++++++++++\n") Print_Grid() return True r,c= find_good_cell() if r < 0 : return False Sq_No= (r//3)*3+c//3 for d in range(1,10) : if Row_Digits[r,d] or Col_Digits[c,d] or Sqr_Digits[Sq_No,d] : continue # put d into Grid Grid[r,c]= d Row_Digits[r,d]= True Col_Digits[c,d]= True Sqr_Digits[Sq_No,d]= True Success= Solve(R_Cells-1) # remove d again Grid[r,c]= 0 Row_Digits[r,d]= False Col_Digits[c,d]= False Sqr_Digits[Sq_No,d]= False if Success : Zuege.append((d,r,c)) return True return False from io import StringIO Problem=''' _________ _____3_85 __1_2____ ___5_7___ __4___1__ _9_______ 5______73 __2_1____ ____4___9 ''' Input= StringIO(Problem) from time import process_time R_Cells= Read_Sudoku(Input) Print_Grid() Zuege=[] Start= process_time() # import cProfile # cProfile.run("Success = Solve(R_Cells)") Success = Solve(R_Cells) Stop= process_time() print("after {} seconds:".format(Stop-Start)) if Success : print("\nZuege:") n=0 for Z in reversed(Zuege) : print("{0} -> ({1},{2})\t".format(Z[0],Z[1]+1,Z[2]+1),end='') n+= 1 if n%5 == 0 : print() print() From fabiosantosart at gmail.com Fri Jul 5 05:38:35 2013 From: fabiosantosart at gmail.com (=?ISO-8859-1?Q?F=E1bio_Santos?=) Date: Fri, 5 Jul 2013 10:38:35 +0100 Subject: How to make this faster In-Reply-To: References: Message-ID: On 5 Jul 2013 09:29, "Helmut Jarausch" wrote: > > Hi, > > I have coded a simple algorithm to solve a Sudoku (probably not the first one). > Unfortunately, it takes 13 seconds for a difficult problem which is more than 75 times slower > than the same algorithm coded in C++. > Is this to be expected or could I have made my Python version faster *** without *** sacrificing readability. > Profiling shows that the function find_good_cell is called (only) 45267 times and this take 12.9 seconds > CPU time (on a 3.2 GHz machine) [Skipping to bottleneck] > def find_good_cell() : In this function you are accessing global variables a lot of times. Since accessing globals takes much more time than accessing locals, I advise you to assign them to local names, and use them. > Best= None > minPoss= 10 > for r in range(9) : > for c in range(9) : > if Grid[r,c] > 0 : continue > Sq_No= (r//3)*3+c//3 > Possibilities= 0 > for d in range(1,10) : On this condition (below) you can try to check which condition is True more often and put that condition first in order to take advantage of short circuiting and minimize array access. > if Row_Digits[r,d] or Col_Digits[c,d] or Sqr_Digits[Sq_No,d] : continue > Possibilities+= 1 > > if ( Possibilities < minPoss ) : > minPoss= Possibilities > Best= (r,c) > > if minPoss == 0 : Best=(-1,-1) > return Best Well I think that is as far as I go with my knowledge. Someone who knows numpy arrays' performance traits should be able to help you more. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jarausch at igpm.rwth-aachen.de Fri Jul 5 06:07:57 2013 From: jarausch at igpm.rwth-aachen.de (Helmut Jarausch) Date: 5 Jul 2013 10:07:57 GMT Subject: How to make this faster References: Message-ID: On Fri, 05 Jul 2013 10:38:35 +0100, F?bio Santos wrote: > [Skipping to bottleneck] > >> def find_good_cell() : > > In this function you are accessing global variables a lot of times. Since > accessing globals takes much more time than accessing locals, I advise you > to assign them to local names, and use them. I've tried to use local "references" like G= Grid and using G instead of Grid below and similarly for Row_Digits, Col_Digits and Sqr_Digits but it had no noticeable effect. > >> Best= None >> minPoss= 10 >> for r in range(9) : >> for c in range(9) : >> if Grid[r,c] > 0 : continue >> Sq_No= (r//3)*3+c//3 >> Possibilities= 0 >> for d in range(1,10) : > > On this condition (below) you can try to check which condition is True more > often and put that condition first in order to take advantage of short > circuiting and minimize array access. Unfortunately, that's unpredictable. > >> if Row_Digits[r,d] or Col_Digits[c,d] or Sqr_Digits[Sq_No,d] : > continue Many thanks, Helmut From oscar.j.benjamin at gmail.com Fri Jul 5 06:13:33 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 5 Jul 2013 11:13:33 +0100 Subject: How to make this faster In-Reply-To: References: Message-ID: On 5 July 2013 09:22, Helmut Jarausch wrote: > Hi, > > I have coded a simple algorithm to solve a Sudoku (probably not the first one). > Unfortunately, it takes 13 seconds for a difficult problem which is more than 75 times slower > than the same algorithm coded in C++. > Is this to be expected or could I have made my Python version faster *** without *** sacrificing readability. It is to be expected that this kind of processing is faster in C than in straight Python. Where Python can win in these kind of problems is if it makes it easier to implement a better algorithm but if your code is just a transliteration from C to Python you should expect it to run slower. Another way that Python can win is if you can express your problem in terms of optimised operations from e.g. the numpy library but you're not doing that here. Of course that does depend on your Python implementation so you could try e.g. using PyPy/nuitka/cython etc. to speed up the core processing. > Profiling shows that the function find_good_cell is called (only) 45267 times and this take 12.9 seconds > CPU time (on a 3.2 GHz machine) [snip] > > def find_good_cell() : > Best= None > minPoss= 10 > for r in range(9) : > for c in range(9) : > if Grid[r,c] > 0 : continue > Sq_No= (r//3)*3+c//3 > Possibilities= 0 > for d in range(1,10) : > if Row_Digits[r,d] or Col_Digits[c,d] or Sqr_Digits[Sq_No,d] : continue > Possibilities+= 1 > > if ( Possibilities < minPoss ) : > minPoss= Possibilities > Best= (r,c) > > if minPoss == 0 : Best=(-1,-1) > return Best My one comment is that you're not really making the most out of numpy arrays. Numpy's ndarrays are efficient when each line of Python code is triggering a large number of numerical computations performed over the array. Because of their N-dimensional nature and the fact that they are in some sense second class citizens in CPython they are often not as good as lists for this kind of looping and indexing. I would actually expect this program to run faster with ordinary Python lists and lists of lists. It means that you need to change e.g. Grid[r, c] to Grid[r][c] but really I think that the indexing syntax is all you're getting out of numpy here. Oscar From jarausch at igpm.rwth-aachen.de Fri Jul 5 06:53:35 2013 From: jarausch at igpm.rwth-aachen.de (Helmut Jarausch) Date: 5 Jul 2013 10:53:35 GMT Subject: How to make this faster References: Message-ID: On Fri, 05 Jul 2013 11:13:33 +0100, Oscar Benjamin wrote: > My one comment is that you're not really making the most out of numpy > arrays. Numpy's ndarrays are efficient when each line of Python code > is triggering a large number of numerical computations performed over > the array. Because of their N-dimensional nature and the fact that > they are in some sense second class citizens in CPython they are often > not as good as lists for this kind of looping and indexing. > > I would actually expect this program to run faster with ordinary > Python lists and lists of lists. It means that you need to change e.g. > Grid[r, c] to Grid[r][c] but really I think that the indexing syntax > is all you're getting out of numpy here. > Thanks Oscar, that was a big improvement, indeed. Using lists of lists instead of numpy arrays made the code more than twice as fast (13 seconds down to 6 seconds) Since I don't do any numerical stuff with the arrays, Numpy doesn't seem to be a good choice. I think this is an argument to add real arrays to Python. I even tried to use dictionaries instead of Numpy arrays. This version is a bit slower then the lists of lists version (7.2 seconds instead of 6 second) but still much faster than the Numpy array solution. Helmut From steve+comp.lang.python at pearwood.info Fri Jul 5 08:02:21 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 05 Jul 2013 12:02:21 GMT Subject: How to make this faster References: Message-ID: <51d6b5cd$0$29999$c3e8da3$5496439d@news.astraweb.com> On Fri, 05 Jul 2013 10:53:35 +0000, Helmut Jarausch wrote: > Since I don't do any numerical stuff with the arrays, Numpy doesn't seem > to be a good choice. I think this is an argument to add real arrays to > Python. Guido's time machine strikes again: import array By the way, I'm not exactly sure how you go from "I don't do numerical calculations on numpy arrays" to "therefore Python should have arrays". -- Steven From jarausch at igpm.rwth-aachen.de Fri Jul 5 10:48:17 2013 From: jarausch at igpm.rwth-aachen.de (Helmut Jarausch) Date: 5 Jul 2013 14:48:17 GMT Subject: How to make this faster References: < 51d6b5cd$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Fri, 05 Jul 2013 12:02:21 +0000, Steven D'Aprano wrote: > On Fri, 05 Jul 2013 10:53:35 +0000, Helmut Jarausch wrote: > >> Since I don't do any numerical stuff with the arrays, Numpy doesn't seem >> to be a good choice. I think this is an argument to add real arrays to >> Python. > > Guido's time machine strikes again: > > import array > > > By the way, I'm not exactly sure how you go from "I don't do numerical > calculations on numpy arrays" to "therefore Python should have arrays". I should have been more clear. I meant multi-dimensional arrays (2D, at least) Numpy is fine if I do math with matrices (without loops in python). Given that I don't like to use the old FORTRAN way (when "dynamic" arrays are passed to functions) of indexing a 2-d array I would need a MACRO or an INLINED function in Python or something like a META-compiler phase transforming def access2d(v,i,j,dim1) : # doesn't work on the l.h.s. return v[i*dim1+j] access2d(v,i,j,dim1) = 7 # at compile time, please to v[i*dim1+j]= 7 # this, by itself, is considered ugly (the FORTRAN way) Thanks for the discussion, Helmut From oscar.j.benjamin at gmail.com Fri Jul 5 11:26:20 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 5 Jul 2013 16:26:20 +0100 Subject: How to make this faster In-Reply-To: References: Message-ID: On 5 July 2013 15:48, Helmut Jarausch wrote: > On Fri, 05 Jul 2013 12:02:21 +0000, Steven D'Aprano wrote: > >> On Fri, 05 Jul 2013 10:53:35 +0000, Helmut Jarausch wrote: >> >>> Since I don't do any numerical stuff with the arrays, Numpy doesn't seem >>> to be a good choice. I think this is an argument to add real arrays to >>> Python. >> >> Guido's time machine strikes again: >> >> import array >> >> >> By the way, I'm not exactly sure how you go from "I don't do numerical >> calculations on numpy arrays" to "therefore Python should have arrays". > > I should have been more clear. I meant multi-dimensional arrays (2D, at least) > Numpy is fine if I do math with matrices (without loops in python). > > Given that I don't like to use the old FORTRAN way (when "dynamic" arrays are passed to > functions) of indexing a 2-d array I would need a MACRO or an INLINED function in Python > or something like a META-compiler phase transforming > > def access2d(v,i,j,dim1) : # doesn't work on the l.h.s. > return v[i*dim1+j] > > access2d(v,i,j,dim1) = 7 # at compile time, please > > to > > v[i*dim1+j]= 7 # this, by itself, is considered ugly (the FORTRAN way) The list of lists approach works fine for what you're doing. I don't think that a[r][c] is that much worse than a[r, c]. It's only when you want to do something like a[:, c] that it breaks down. In any case, your algorithm would work better with Python's set/dict/list types than numpy arrays. One of the reasons that it's faster to use lists than numpy arrays (as you found out) is precisely because the N-dimensional array logic complicates 1-dimensional processing. I've seen discussions in Cython and numpy about lighter-weight 1-dimensional array types for this reason. The other reason that numpy arrays are slower for what you're doing is that (just like the stdlib array type Steven referred to) they use homogeneous types in a contiguous buffer and each element is not a Python object in its own right until you access it with e.g. a[0]. That means that the numpy array has to create a new object every time you index into it whereas the list can simply return a new reference to an existing object. You can get the same effect with numpy arrays by using dtype=object but I'd still expect it to be slower for this. Oscar From fabiosantosart at gmail.com Fri Jul 5 08:44:57 2013 From: fabiosantosart at gmail.com (=?ISO-8859-1?Q?F=E1bio_Santos?=) Date: Fri, 5 Jul 2013 13:44:57 +0100 Subject: How to make this faster In-Reply-To: References: Message-ID: On 5 Jul 2013 11:58, "Helmut Jarausch" wrote: > > On Fri, 05 Jul 2013 11:13:33 +0100, Oscar Benjamin wrote: > > > My one comment is that you're not really making the most out of numpy > > arrays. Numpy's ndarrays are efficient when each line of Python code > > is triggering a large number of numerical computations performed over > > the array. Because of their N-dimensional nature and the fact that > > they are in some sense second class citizens in CPython they are often > > not as good as lists for this kind of looping and indexing. > > > > I would actually expect this program to run faster with ordinary > > Python lists and lists of lists. It means that you need to change e.g. > > Grid[r, c] to Grid[r][c] but really I think that the indexing syntax > > is all you're getting out of numpy here. > > > > Thanks Oscar, that was a big improvement, indeed. > Using lists of lists instead of numpy arrays made the code more than > twice as fast (13 seconds down to 6 seconds) > > Since I don't do any numerical stuff with the arrays, Numpy doesn't seem to be > a good choice. I think this is an argument to add real arrays to Python. > > I even tried to use dictionaries instead of Numpy arrays. This version is a bit > slower then the lists of lists version (7.2 seconds instead of 6 second) but still > much faster than the Numpy array solution. May I suggest you avoid range and use enumerate(the_array) instead? It might be faster. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Fri Jul 5 09:41:23 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 5 Jul 2013 14:41:23 +0100 Subject: How to make this faster In-Reply-To: References: Message-ID: On 5 July 2013 11:53, Helmut Jarausch wrote: > I even tried to use dictionaries instead of Numpy arrays. This version is a bit > slower then the lists of lists version (7.2 seconds instead of 6 second) but still > much faster than the Numpy array solution. When you switched to dictionaries did you take advantage of the sparseness by iterating over dictionary keys instead of indices? This is the kind of thing that I meant when I said that in Python it's often easier to implement a better algorithm than in C. What I mean is that if Grid is a dict so that Grid[(r, c)] is the entry at row r and column c (if it exists) then you can change a loop like: for r in range(9): for c in range(9): if Grid[r, c] > 0: continue # do stuff so that it looks like: for r, c in Grid: # do stuff If the grid is sparsely occupied then this could be a significant improvement. Oscar From jarausch at igpm.rwth-aachen.de Fri Jul 5 10:28:36 2013 From: jarausch at igpm.rwth-aachen.de (Helmut Jarausch) Date: 5 Jul 2013 14:28:36 GMT Subject: How to make this faster References: < mailman.4296.1373031707.3114.python-list@python.org> Message-ID: On Fri, 05 Jul 2013 14:41:23 +0100, Oscar Benjamin wrote: > On 5 July 2013 11:53, Helmut Jarausch wrote: >> I even tried to use dictionaries instead of Numpy arrays. This version is a bit >> slower then the lists of lists version (7.2 seconds instead of 6 second) but still >> much faster than the Numpy array solution. > > When you switched to dictionaries did you take advantage of the > sparseness by iterating over dictionary keys instead of indices? This > is the kind of thing that I meant when I said that in Python it's > often easier to implement a better algorithm than in C. What I mean is > that if Grid is a dict so that Grid[(r, c)] is the entry at row r and > column c (if it exists) then you can change a loop like: > > for r in range(9): > for c in range(9): > if Grid[r, c] > 0: continue > # do stuff > > so that it looks like: > > for r, c in Grid: > # do stuff > > If the grid is sparsely occupied then this could be a significant improvement. > > > Oscar This gives a big speedup. Now, the time is gone down to 1.73 seconds in comparison to original 13 seconds or the 7 seconds for the first version above. Many thanks, it seems hard to optimize a Python program, Helmut From oscar.j.benjamin at gmail.com Fri Jul 5 10:45:25 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 5 Jul 2013 15:45:25 +0100 Subject: How to make this faster In-Reply-To: References: Message-ID: On 5 July 2013 15:28, Helmut Jarausch wrote: > On Fri, 05 Jul 2013 14:41:23 +0100, Oscar Benjamin wrote: > >> On 5 July 2013 11:53, Helmut Jarausch wrote: >>> I even tried to use dictionaries instead of Numpy arrays. This version is a bit >>> slower then the lists of lists version (7.2 seconds instead of 6 second) but still >>> much faster than the Numpy array solution. >> >> When you switched to dictionaries did you take advantage of the >> sparseness by iterating over dictionary keys instead of indices? This >> is the kind of thing that I meant when I said that in Python it's >> often easier to implement a better algorithm than in C. What I mean is >> that if Grid is a dict so that Grid[(r, c)] is the entry at row r and >> column c (if it exists) then you can change a loop like: >> >> for r in range(9): >> for c in range(9): >> if Grid[r, c] > 0: continue >> # do stuff >> >> so that it looks like: >> >> for r, c in Grid: >> # do stuff >> >> If the grid is sparsely occupied then this could be a significant improvement. >> > This gives a big speedup. Now, the time is gone down to 1.73 seconds in comparison to > original 13 seconds or the 7 seconds for the first version above. Presumably then you're now down to the innermost loop as a bottle-neck: Possibilities= 0 for d in range(1,10) : if Row_Digits[r,d] or Col_Digits[c,d] or Sqr_Digits[Sq_No,d] : continue Possibilities+= 1 If you make it so that e.g. Row_Digits[r] is a set of indices rather than a list of bools then you can do this with something like Possibilities = len(Row_Digits[r] | Col_Digits[c] | Sqr_Digits[Sq_No]) or perhaps Possibilities = len(set.union(Row_Digits[r], Col_Digits[c], Sqr_Digits[Sq_No])) which I would expect to be a little faster than looping over range since the loop is then performed under the hood by the builtin set-type. > Many thanks, > it seems hard to optimize a Python program, It just takes practice. It's a little less obvious in Python than in low-level languages where the bottlenecks will be and which operations are faster/slower but optimisation always involves a certain amount of trial and error anyway. Oscar From jarausch at igpm.rwth-aachen.de Fri Jul 5 11:17:08 2013 From: jarausch at igpm.rwth-aachen.de (Helmut Jarausch) Date: 5 Jul 2013 15:17:08 GMT Subject: How to make this faster References: < b3o3gkFtc39U1@mid.dfncis.de> Message-ID: On Fri, 05 Jul 2013 15:45:25 +0100, Oscar Benjamin wrote: > Presumably then you're now down to the innermost loop as a bottle-neck: > > Possibilities= 0 > for d in range(1,10) : > if Row_Digits[r,d] or Col_Digits[c,d] or Sqr_Digits[Sq_No,d] : continue > Possibilities+= 1 > > If you make it so that e.g. Row_Digits[r] is a set of indices rather > than a list of bools then you can do this with something like > > Possibilities = len(Row_Digits[r] | Col_Digits[c] | Sqr_Digits[Sq_No]) > > or perhaps > > Possibilities = len(set.union(Row_Digits[r], Col_Digits[c], > Sqr_Digits[Sq_No])) > > which I would expect to be a little faster than looping over range > since the loop is then performed under the hood by the builtin > set-type. > > It just takes practice. indeed > It's a little less obvious in Python than in > low-level languages where the bottlenecks will be and which operations > are faster/slower but optimisation always involves a certain amount of > trial and error anyway. > > > Oscar I've tried the following version def find_good_cell() : Best= None minPoss= 10 for r,c in Grid : if Grid[(r,c)] > 0 : continue Sq_No= (r//3)*3+c//3 Possibilities= 9-len(Row_Digits[r] | Col_Digits[c] | Sqr_Digits[Sq_No]) if ( Possibilities < minPoss ) : minPoss= Possibilities Best= (r,c) if minPoss == 0 : Best=(-1,-1) return Best All_digits= set((1,2,3,4,5,6,7,8,9)) def Solve(R_Cells) : if R_Cells == 0 : print("\n\n++++++++++ S o l u t i o n ++++++++++\n") Print_Grid() return True r,c= find_good_cell() if r < 0 : return False Sq_No= (r//3)*3+c//3 for d in All_digits - (Row_Digits[r] | Col_Digits[c] | Sqr_Digits[Sq_No]) : # put d into Grid Grid[(r,c)]= d Row_Digits[r].add(d) Col_Digits[c].add(d) Sqr_Digits[Sq_No].add(d) Success= Solve(R_Cells-1) # remove d again Grid[(r,c)]= 0 Row_Digits[r].remove(d) Col_Digits[c].remove(d) Sqr_Digits[Sq_No].remove(d) if Success : Zuege.append((d,r,c)) return True return False which turns out to be as fast as the previous "dictionary only version". Probably, set.remove is a bit slow Thanks, Helmut From oscar.j.benjamin at gmail.com Fri Jul 5 11:38:43 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 5 Jul 2013 16:38:43 +0100 Subject: How to make this faster In-Reply-To: References: Message-ID: On 5 July 2013 16:17, Helmut Jarausch wrote: > > I've tried the following version > > def find_good_cell() : > Best= None > minPoss= 10 > for r,c in Grid : > if Grid[(r,c)] > 0 : continue Sorry, I think what I meant was that you should have a structure called e.g. Remaining which is the set of all (r, c) pairs that you want to loop over here. Then there's no need to check on each iteration whether or not Grid[(r, c)] > 0. When I said "sparse" I meant that you don't need to set keys in Grid unless you actually have a value there so the test "Grid[(r, c)] > 0" would look like "(r, c) in Grid". Remaining is the set of all (r, c) pairs not in Grid that you update incrementally with .add() and .remove(). Then this for r,c in Grid : if Grid[(r,c)] > 0 : continue becomes for r, c in Remaining: > Sq_No= (r//3)*3+c//3 > Possibilities= 9-len(Row_Digits[r] | Col_Digits[c] | Sqr_Digits[Sq_No]) > if ( Possibilities < minPoss ) : > minPoss= Possibilities > Best= (r,c) > > if minPoss == 0 : Best=(-1,-1) > return Best > > All_digits= set((1,2,3,4,5,6,7,8,9)) All_digits= set(range(1, 10)) or All_digits = {1,2,3,4,5,6,7,8,9} > > def Solve(R_Cells) : > if R_Cells == 0 : > print("\n\n++++++++++ S o l u t i o n ++++++++++\n") > Print_Grid() > return True > > r,c= find_good_cell() > if r < 0 : return False > Sq_No= (r//3)*3+c//3 > > for d in All_digits - (Row_Digits[r] | Col_Digits[c] | Sqr_Digits[Sq_No]) : > # put d into Grid > Grid[(r,c)]= d > Row_Digits[r].add(d) > Col_Digits[c].add(d) > Sqr_Digits[Sq_No].add(d) > > Success= Solve(R_Cells-1) > > # remove d again > Grid[(r,c)]= 0 > Row_Digits[r].remove(d) > Col_Digits[c].remove(d) > Sqr_Digits[Sq_No].remove(d) > > if Success : > Zuege.append((d,r,c)) > return True > > return False > > which turns out to be as fast as the previous "dictionary only version". > Probably, set.remove is a bit slow No it's not and you're not using it in your innermost loops anyway. Probably the loop I referred to isn't your bottleneck. Oscar From jarausch at igpm.rwth-aachen.de Fri Jul 5 12:07:03 2013 From: jarausch at igpm.rwth-aachen.de (Helmut Jarausch) Date: 5 Jul 2013 16:07:03 GMT Subject: How to make this faster References: < b3o6bkFtc39U4@mid.dfncis.de> Message-ID: On Fri, 05 Jul 2013 16:38:43 +0100, Oscar Benjamin wrote: > On 5 July 2013 16:17, Helmut Jarausch wrote: >> >> I've tried the following version >> >> def find_good_cell() : >> Best= None >> minPoss= 10 >> for r,c in Grid : >> if Grid[(r,c)] > 0 : continue > > Sorry, I think what I meant was that you should have a structure > called e.g. Remaining which is the set of all (r, c) pairs that you > want to loop over here. Then there's no need to check on each > iteration whether or not Grid[(r, c)] > 0. When I said "sparse" I > meant that you don't need to set keys in Grid unless you actually have > a value there so the test "Grid[(r, c)] > 0" would look like "(r, c) > in Grid". Remaining is the set of all (r, c) pairs not in Grid that > you update incrementally with .add() and .remove(). > > Then this > > for r,c in Grid : > if Grid[(r,c)] > 0 : continue > > becomes > > for r, c in Remaining: > >> Sq_No= (r//3)*3+c//3 >> Possibilities= 9-len(Row_Digits[r] | Col_Digits[c] | Sqr_Digits[Sq_No]) >> if ( Possibilities < minPoss ) : >> minPoss= Possibilities >> Best= (r,c) >> >> if minPoss == 0 : Best=(-1,-1) >> return Best >> >> All_digits= set((1,2,3,4,5,6,7,8,9)) > > All_digits= set(range(1, 10)) > > or > > All_digits = {1,2,3,4,5,6,7,8,9} > >> >> def Solve(R_Cells) : >> if R_Cells == 0 : >> print("\n\n++++++++++ S o l u t i o n ++++++++++\n") >> Print_Grid() >> return True >> >> r,c= find_good_cell() >> if r < 0 : return False >> Sq_No= (r//3)*3+c//3 >> >> for d in All_digits - (Row_Digits[r] | Col_Digits[c] | Sqr_Digits[Sq_No]) : >> # put d into Grid >> Grid[(r,c)]= d >> Row_Digits[r].add(d) >> Col_Digits[c].add(d) >> Sqr_Digits[Sq_No].add(d) >> >> Success= Solve(R_Cells-1) >> >> # remove d again >> Grid[(r,c)]= 0 >> Row_Digits[r].remove(d) >> Col_Digits[c].remove(d) >> Sqr_Digits[Sq_No].remove(d) >> >> if Success : >> Zuege.append((d,r,c)) >> return True >> >> return False >> >> which turns out to be as fast as the previous "dictionary only version". >> Probably, set.remove is a bit slow > > No it's not and you're not using it in your innermost loops anyway. > Probably the loop I referred to isn't your bottleneck. > I've tried your suggestion, but unless I've misunderstood your suggestion it's a bit slower than the solution above. The solution above take 0.79 seconds (mean of 100 calls) while the following version take 1.05 seconds (mean of 100 calls): Grid = {(i,j):0 for i in range(9) for j in range(9)} Remaining= {(i,j) for i in range(9) for j in range(9)} Row_Digits = [set() for r in range(9)] Col_Digits = [set() for c in range(9)] Sqr_Digits = [set() for s in range(9)] .... remove some pairs from Remaining for the initial set of the given Sudoku def find_good_cell() : Best= None minPoss= 10 for r,c in Remaining : Sq_No= (r//3)*3+c//3 Possibilities= 9-len(Row_Digits[r] | Col_Digits[c] | Sqr_Digits[Sq_No]) if ( Possibilities < minPoss ) : minPoss= Possibilities Best= (r,c) if minPoss == 0 : Best=(-1,-1) return Best All_digits= set(range(1,10)) def Solve(R_Cells) : if R_Cells == 0 : print("\n\n++++++++++ S o l u t i o n ++++++++++\n") Print_Grid() return True r,c= find_good_cell() if r < 0 : return False Sq_No= (r//3)*3+c//3 for d in All_digits - (Row_Digits[r] | Col_Digits[c] | Sqr_Digits[Sq_No]) : # put d into Grid Grid[(r,c)]= d Remaining.remove((r,c)) Row_Digits[r].add(d) Col_Digits[c].add(d) Sqr_Digits[Sq_No].add(d) Success= Solve(R_Cells-1) # remove d again Grid[(r,c)]= 0 Remaining.add((r,c)) Row_Digits[r].remove(d) Col_Digits[c].remove(d) Sqr_Digits[Sq_No].remove(d) if Success : Zuege.append((d,r,c)) return True return False From steve+comp.lang.python at pearwood.info Fri Jul 5 12:50:41 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 05 Jul 2013 16:50:41 GMT Subject: How to make this faster References: Message-ID: <51d6f960$0$29999$c3e8da3$5496439d@news.astraweb.com> On Fri, 05 Jul 2013 16:07:03 +0000, Helmut Jarausch wrote: > The solution above take 0.79 seconds (mean of 100 calls) while the > following version take 1.05 seconds (mean of 100 calls): 1) How are you timing the calls? 2) Don't use the mean, that's the wrong statistic when you are measuring something where the error is always one sided. You want the minimum, not the mean. When you measure the time taken for a piece of code, the number you get is made up of two components: 1) the actual time the code would have taken, if there were no random fluctuations due to other processes, etc.; and 2) random errors due to switching to other processes, etc. Both of these are unknown; you only know the total. But obviously the random errors are always positive. They can never be negative, and you can never measure a time which is less than the fastest your code could run. (If your anti-virus software starts scanning in the middle of the trial, it can only make your code take more time to run, never less.) So the only way to minimize the error is to pick the minimum time, not the average. The average just gives you: - some unknown "true" time, plus some unknown error, somewhere between the smallest error and the biggest error; whereas the minimum gives you: - some unknown "true" time, plus the smallest error yet seen. -- Steven From jarausch at igpm.rwth-aachen.de Fri Jul 5 14:39:15 2013 From: jarausch at igpm.rwth-aachen.de (Helmut Jarausch) Date: 5 Jul 2013 18:39:15 GMT Subject: How to make this faster References: < mailman.4302.1373038751.3114.python-list@ python.org> <51d6f960$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Fri, 05 Jul 2013 16:50:41 +0000, Steven D'Aprano wrote: > On Fri, 05 Jul 2013 16:07:03 +0000, Helmut Jarausch wrote: > >> The solution above take 0.79 seconds (mean of 100 calls) while the >> following version take 1.05 seconds (mean of 100 calls): > > 1) How are you timing the calls? I've put the work horse "Solve" into a loop executing 100 times. That's on an otherwise idle Linux machine. > > 2) Don't use the mean, that's the wrong statistic when you are measuring > something where the error is always one sided. You want the minimum, not > the mean. Here you assume time measuring itself is without error - I doubt that. If the timing version, which executes function "Solve" one hundred times, runs about 80-100 seconds without a significant variation, then taking the mean is mathematically correct. I can't take the minimum since I don't measure the time a single call takes. > > When you measure the time taken for a piece of code, the number you get > is made up of two components: > > > 1) the actual time the code would have taken, if there were no random > fluctuations due to other processes, etc.; and > > 2) random errors due to switching to other processes, etc. > > Both of these are unknown; you only know the total. But obviously the > random errors are always positive. They can never be negative, and you > can never measure a time which is less than the fastest your code could > run. > > (If your anti-virus software starts scanning in the middle of the trial, > it can only make your code take more time to run, never less.) > > So the only way to minimize the error is to pick the minimum time, not > the average. The average just gives you: > > - some unknown "true" time, plus some unknown error, somewhere > between the smallest error and the biggest error; > > whereas the minimum gives you: > > - some unknown "true" time, plus the smallest error yet seen. From steve+comp.lang.python at pearwood.info Fri Jul 5 23:05:30 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 06 Jul 2013 03:05:30 GMT Subject: How to make this faster References: <51d6f960$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51d7897a$0$29999$c3e8da3$5496439d@news.astraweb.com> On Fri, 05 Jul 2013 18:39:15 +0000, Helmut Jarausch wrote: > On Fri, 05 Jul 2013 16:50:41 +0000, Steven D'Aprano wrote: > >> On Fri, 05 Jul 2013 16:07:03 +0000, Helmut Jarausch wrote: >> >>> The solution above take 0.79 seconds (mean of 100 calls) while the >>> following version take 1.05 seconds (mean of 100 calls): >> >> 1) How are you timing the calls? > > I've put the work horse "Solve" into a loop executing 100 times. That's > on an otherwise idle Linux machine. That doesn't explain how you time it, only that you have a loop executing 100 times. Are you using time.time, or time.clock? (I trust you're not measuring times by hand with a stop watch.) I expect you're probably doing something like this: start = time.time() for i in range(100): solve(data) t = time.time() - start print "time taken", t/100 In this case, you're getting times well over a second, so the overhead is unlikely to be significant. E.g. the time it takes to create range(100) will be insignificant compared to the time it takes to run solve. Nevertheless, as a matter of good practice, I recommend that you use the timeit module, as it's carefully written to minimize the overhead when timing small code snippets where the overhead is *not* insignificant. http://docs.python.org/2/library/timeit.html http://docs.python.org/3/library/timeit.html For longer running code, like this, you might also like this: http://code.activestate.com/recipes/577896/ >> 2) Don't use the mean, that's the wrong statistic when you are >> measuring something where the error is always one sided. You want the >> minimum, not the mean. > > Here you assume time measuring itself is without error - I doubt that. No, I don't. I may have glossed over all the sources of measurement error, but they are all *positive* which is the important thing. The time measured will never be *less* than the actual time taken. So regardless of the source of measurement error, the way to minimize it is to only look at the minimum timing, not the mean. > If the timing version, which executes function "Solve" one hundred > times, runs about 80-100 seconds without a significant variation, then > taking the mean is mathematically correct. If the best you can say is it takes "80-100 seconds", that's pretty significant variation, of the order of 20%. In this case, with times of the order of a second per loop, it may be reasonable to say "in this specific case, the error is too small to care about", or "I just don't care about the error, since it will be about the same for different variations of my solve function". But in that case, why bother looping 100 times? > I can't take the minimum > since I don't measure the time a single call takes. Then perhaps you should. -- Steven From jarausch at igpm.rwth-aachen.de Sat Jul 6 03:25:21 2013 From: jarausch at igpm.rwth-aachen.de (Helmut Jarausch) Date: 6 Jul 2013 07:25:21 GMT Subject: How to make this faster References: < b3o997Fu5lhU2@mid. dfncis.de> <51d6f960$0$29999$c3e8da3$5496439d@news. astraweb.com> < 51d7897a$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sat, 06 Jul 2013 03:05:30 +0000, Steven D'Aprano wrote: > That doesn't explain how you time it, only that you have a loop executing > 100 times. Are you using time.time, or time.clock? (I trust you're not > measuring times by hand with a stop watch.) > > I expect you're probably doing something like this: > > start = time.time() I have been using time.process_time >> If the timing version, which executes function "Solve" one hundred >> times, runs about 80-100 seconds without a significant variation, then >> taking the mean is mathematically correct. > For longer running code, like this, you might also like this: > http://code.activestate.com/recipes/577896/ Thanks for the pointer. > If the best you can say is it takes "80-100 seconds", that's pretty > significant variation, of the order of 20%. That's not a variation of a SINGLE variant. One variant takes 80 seconds and the other variant to be compared with takes 100 seconds. > > In this case, with times of the order of a second per loop, it may be > reasonable to say "in this specific case, the error is too small to care > about", or "I just don't care about the error, since it will be about the > same for different variations of my solve function". But in that case, > why bother looping 100 times? > > >> I can't take the minimum >> since I don't measure the time a single call takes. > > Then perhaps you should. Many thanks, Helmut From python at mrabarnett.plus.com Fri Jul 5 12:25:54 2013 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 05 Jul 2013 17:25:54 +0100 Subject: How to make this faster In-Reply-To: References: < b3o3gkFtc39U1@mid.dfncis.de> Message-ID: <51D6F392.2070301@mrabarnett.plus.com> On 05/07/2013 16:17, Helmut Jarausch wrote: > On Fri, 05 Jul 2013 15:45:25 +0100, Oscar Benjamin wrote: > >> Presumably then you're now down to the innermost loop as a bottle-neck: >> >> Possibilities= 0 >> for d in range(1,10) : >> if Row_Digits[r,d] or Col_Digits[c,d] or Sqr_Digits[Sq_No,d] : continue >> Possibilities+= 1 >> >> If you make it so that e.g. Row_Digits[r] is a set of indices rather >> than a list of bools then you can do this with something like >> >> Possibilities = len(Row_Digits[r] | Col_Digits[c] | Sqr_Digits[Sq_No]) >> >> or perhaps >> >> Possibilities = len(set.union(Row_Digits[r], Col_Digits[c], >> Sqr_Digits[Sq_No])) >> >> which I would expect to be a little faster than looping over range >> since the loop is then performed under the hood by the builtin >> set-type. >> >> It just takes practice. > > indeed > >> It's a little less obvious in Python than in >> low-level languages where the bottlenecks will be and which operations >> are faster/slower but optimisation always involves a certain amount of >> trial and error anyway. >> >> >> Oscar > > I've tried the following version > > def find_good_cell() : > Best= None > minPoss= 10 > for r,c in Grid : > if Grid[(r,c)] > 0 : continue > Sq_No= (r//3)*3+c//3 > Possibilities= 9-len(Row_Digits[r] | Col_Digits[c] | Sqr_Digits[Sq_No]) > if ( Possibilities < minPoss ) : > minPoss= Possibilities > Best= (r,c) > > if minPoss == 0 : Best=(-1,-1) > return Best > > All_digits= set((1,2,3,4,5,6,7,8,9)) > > def Solve(R_Cells) : > if R_Cells == 0 : > print("\n\n++++++++++ S o l u t i o n ++++++++++\n") > Print_Grid() > return True > > r,c= find_good_cell() > if r < 0 : return False > Sq_No= (r//3)*3+c//3 > > for d in All_digits - (Row_Digits[r] | Col_Digits[c] | Sqr_Digits[Sq_No]) : > # put d into Grid > Grid[(r,c)]= d > Row_Digits[r].add(d) > Col_Digits[c].add(d) > Sqr_Digits[Sq_No].add(d) > > Success= Solve(R_Cells-1) > > # remove d again > Grid[(r,c)]= 0 > Row_Digits[r].remove(d) > Col_Digits[c].remove(d) > Sqr_Digits[Sq_No].remove(d) > > if Success : > Zuege.append((d,r,c)) > return True > > return False > > which turns out to be as fast as the previous "dictionary only version". > Probably, set.remove is a bit slow > For comparison, here's my solution: from collections import Counter problem = ''' _________ _____3_85 __1_2____ ___5_7___ __4___1__ _9_______ 5______73 __2_1____ ____4___9 ''' # Build the grid. digits = "123456789" grid = [] for row in problem.splitlines(): if not row: continue new_row = [] for cell in row: if cell.isdigit(): new_row.append({cell}) else: new_row.append(set(digits)) grid.append(new_row) # Solve the grid. changed = True while changed: changed = False # Look for cells that contain only one digit. for r in range(9): for c in range(9): if len(grid[r][c]) == 1: digit = list(grid[r][c])[0] # Remove from other cells in same row. for c2 in range(9): if c2 != c and digit in grid[r][c2]: grid[r][c2].remove(digit) changed = True # Remove from other cells in same column. for r2 in range(9): if r2 != r and digit in grid[r2][c]: grid[r2][c].remove(digit) changed = True # Remove from other cells in the same block of 9. start_row = r - r % 3 start_column = c - c % 3 for r2 in range(start_row, start_row + 3): for c2 in range(start_column, start_column + 3): if (r2, c2) != (r, c) and digit in grid[r2][c2]: grid[r2][c2].remove(digit) changed = True # Look for digits that occur in only one cell in a row. for r in range(9): counts = Counter() for c in range(9): counts += Counter(grid[r][c]) unique = {digit for digit, times in counts.items() if times == 1} for c in range(9): if len(grid[r][c]) > 1 and len(grid[r][c] & unique) == 1: grid[r][c] &= unique changed = True # Look for digits that occur in only one cell in a column. for c in range(9): counts = Counter() for r in range(9): counts += Counter(grid[r][c]) unique = {digit for digit, times in counts.items() if times == 1} for r in range(9): if len(grid[r][c]) > 1 and len(grid[r][c] & unique) == 1: grid[r][c] &= unique changed = True # Look for digits that occur in only one cell in a block of 9. for start_row in range(0, 9, 3): for start_column in range(0, 9, 3): counts = Counter() for r in range(start_row, start_row + 3): for c in range(start_column, start_column + 3): counts += Counter(grid[r][c]) unique = {digit for digit, times in counts.items() if times == 1} for r in range(start_row, start_row + 3): for c in range(start_column, start_column + 3): if len(grid[r][c]) > 1 and len(grid[r][c] & unique) == 1: grid[r][c] &= unique changed = True # Display the solution. for row in grid: for cell in row: print("".join(sorted(cell)), end=" ") print() From joshua.landau.ws at gmail.com Fri Jul 5 21:45:47 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Sat, 6 Jul 2013 02:45:47 +0100 Subject: How to make this faster In-Reply-To: <51D6F392.2070301@mrabarnett.plus.com> References: <51D6F392.2070301@mrabarnett.plus.com> Message-ID: On 5 July 2013 17:25, MRAB wrote: > For comparison, here's my solution: Unfortunately, there are some sudokus that require guessing - your algorithm will not solve those. A combination algorithm would be best, AFAIK. ----- FWIW, this is my interpretation of the original algorithm: from collections import defaultdict from io import StringIO # For printing DONE = defaultdict(lambda: " ") # How many different numbers can go in each slot, indexes are of the form: [position] NUM_POSSIBILITIES = defaultdict(lambda: 9) # How many times each value has been removed from the possibilities, # indexes are of the form: [value, position] TIMES_REMOVED = defaultdict(lambda: 0) # Set of empty spaces in the grid REMAINING = {(i,j) for i in range(9) for j in range(9)} # Cache, would normally be done inline PLACES_TO_REMOVE = {} for row in range(9): for column in range(9): square_top, square_left = 3*(row // 3), 3*(column // 3) # Across the row PLACES_TO_REMOVE[row, column] = {(p_row, column) for p_row in range(9)} # Across the collumn PLACES_TO_REMOVE[row, column] |= {(row, p_collumn) for p_collumn in range(9)} # Inside the square PLACES_TO_REMOVE[row, column] |= { (p_row, p_collumn) for p_row in range(square_top, 3+square_top) for p_collumn in range(square_left, 3+square_left) } def add_at(value, position): """Add value to the position, updating needed globals""" DONE[position] = value # Use "REMAINING &" so that we only change places we care about # It's reversed in a consistent order to allow this for add_pos in REMAINING & PLACES_TO_REMOVE[position]: # Remove possibility if it hasn't already been removed if not TIMES_REMOVED[value, add_pos]: NUM_POSSIBILITIES[add_pos] -= 1 TIMES_REMOVED[value, add_pos] += 1 REMAINING.remove(position) def remove_at(value, position): """The inverse of add_at""" REMAINING.add(position) # Use "REMAINING &" so that we only change places we care about # It's reversed in a consistent order to allow this for remove_pos in REMAINING & PLACES_TO_REMOVE[position]: TIMES_REMOVED[value, remove_pos] -= 1 # Add posibility back if TIMES_REMOVED falls to 0 if not TIMES_REMOVED[value, remove_pos]: NUM_POSSIBILITIES[remove_pos] += 1 del DONE[position] def read_sudoku(input): number_remaining = 81 # StringIO iterates over lines lines = (line.strip() for line in StringIO(input)) lines = (line for line in lines if line and not line.startswith("#")) for row, line in enumerate(lines): line = line.strip().replace(" ", "").replace("\t", "") for column, character in enumerate(line): if character == "_": pass elif character.isdigit(): add_at(int(character), (row, column)) number_remaining -= 1 else: raise ValueError("Invalid character {} in input line {}".format(character, line)) return number_remaining def print_grid(highlight=None): row_seperator = "? ? ? ? ? ? ? ? ? ?" special_seperator = "?-----------?-----------?-----------?" seperators = " ? ? ?" row_seperators = [special_seperator, row_seperator, row_seperator] * 3 for row, row_seperator in enumerate(row_seperators): print(row_seperator) print("?", end=" ") for column, seperator in enumerate(seperators): if (row, column) == highlight: # These are colors, ignore 'em if you wish # This is to let you see changes when debugging print("\x1b[31;01m{}\x1b[39;49;00m".format(DONE[row, column]), seperator, end=" ") else: print(DONE[row, column], seperator, end=" ") print() print(special_seperator) print() def solve(number_remaining): if not number_remaining: return True # We need to guess -- go for the one where we have the least choice position = min(REMAINING, key=NUM_POSSIBILITIES.__getitem__) for value in range(1, 10): if not TIMES_REMOVED[value, position]: add_at(value, position) if solve(number_remaining-1): return True remove_at(value, position) return False problem = """ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 3 _ 8 5 _ _ 1 _ 2 _ _ _ _ _ _ _ 5 _ 7 _ _ _ _ _ 4 _ _ _ 1 _ _ _ 9 _ _ _ _ _ _ _ 5 _ _ _ _ _ _ 7 3 _ _ 2 _ 1 _ _ _ _ _ _ _ _ 4 _ _ _ 9 """ solve(read_sudoku(problem)) print_grid() From jarausch at igpm.rwth-aachen.de Fri Jul 5 14:42:56 2013 From: jarausch at igpm.rwth-aachen.de (Helmut Jarausch) Date: 5 Jul 2013 18:42:56 GMT Subject: How to make this faster References: < b3o6bkFtc39U4@mid.dfncis.de> Message-ID: On Fri, 05 Jul 2013 17:25:54 +0100, MRAB wrote: > For comparison, here's my solution: Your solution is very fast, indeed. It takes 0.04 seconds (mean of 1000 runs) restoring "grid" in between. But that's a different algorithm which is IMHO more difficult to understand. Many thanks, Helmut > > from collections import Counter > > problem = ''' > _________ > _____3_85 > __1_2____ > ___5_7___ > __4___1__ > _9_______ > 5______73 > __2_1____ > ____4___9 > ''' > > # Build the grid. > digits = "123456789" > > grid = [] > > for row in problem.splitlines(): > if not row: > continue > > new_row = [] > > for cell in row: > if cell.isdigit(): > new_row.append({cell}) > else: > new_row.append(set(digits)) > > grid.append(new_row) > > # Solve the grid. > changed = True > while changed: > changed = False > > # Look for cells that contain only one digit. > for r in range(9): > for c in range(9): > if len(grid[r][c]) == 1: > digit = list(grid[r][c])[0] > > # Remove from other cells in same row. > for c2 in range(9): > if c2 != c and digit in grid[r][c2]: > grid[r][c2].remove(digit) > changed = True > > # Remove from other cells in same column. > for r2 in range(9): > if r2 != r and digit in grid[r2][c]: > grid[r2][c].remove(digit) > changed = True > > # Remove from other cells in the same block of 9. > start_row = r - r % 3 > start_column = c - c % 3 > for r2 in range(start_row, start_row + 3): > for c2 in range(start_column, start_column + 3): > if (r2, c2) != (r, c) and digit in grid[r2][c2]: > grid[r2][c2].remove(digit) > changed = True > > # Look for digits that occur in only one cell in a row. > for r in range(9): > counts = Counter() > for c in range(9): > counts += Counter(grid[r][c]) > > unique = {digit for digit, times in counts.items() if times == 1} > > for c in range(9): > if len(grid[r][c]) > 1 and len(grid[r][c] & unique) == 1: > grid[r][c] &= unique > changed = True > > # Look for digits that occur in only one cell in a column. > for c in range(9): > counts = Counter() > for r in range(9): > counts += Counter(grid[r][c]) > > unique = {digit for digit, times in counts.items() if times == 1} > > for r in range(9): > if len(grid[r][c]) > 1 and len(grid[r][c] & unique) == 1: > grid[r][c] &= unique > changed = True > > # Look for digits that occur in only one cell in a block of 9. > for start_row in range(0, 9, 3): > for start_column in range(0, 9, 3): > counts = Counter() > for r in range(start_row, start_row + 3): > for c in range(start_column, start_column + 3): > counts += Counter(grid[r][c]) > > unique = {digit for digit, times in counts.items() if times == 1} > > for r in range(start_row, start_row + 3): > for c in range(start_column, start_column + 3): > if len(grid[r][c]) > 1 and len(grid[r][c] & unique) == 1: > grid[r][c] &= unique > changed = True > > # Display the solution. > for row in grid: > for cell in row: > print("".join(sorted(cell)), end=" ") > > print() From jarausch at igpm.rwth-aachen.de Fri Jul 5 10:54:26 2013 From: jarausch at igpm.rwth-aachen.de (Helmut Jarausch) Date: 5 Jul 2013 14:54:26 GMT Subject: How to make this faster References: < mailman.4294.1373028307.3114.python-list@python.org> Message-ID: On Fri, 05 Jul 2013 13:44:57 +0100, F?bio Santos wrote: May I suggest you avoid range and use enumerate(the_array) instead? It might be faster. How does this work? Given Grid= [[0 for j in range(9)] for i in range(9)] for (r,c,val) in ????(Grid) : Helmut From fabiosantosart at gmail.com Fri Jul 5 11:18:41 2013 From: fabiosantosart at gmail.com (=?ISO-8859-1?Q?F=E1bio_Santos?=) Date: Fri, 5 Jul 2013 16:18:41 +0100 Subject: How to make this faster In-Reply-To: References: Message-ID: On 5 Jul 2013 15:59, "Helmut Jarausch" wrote: > > On Fri, 05 Jul 2013 13:44:57 +0100, F?bio Santos wrote: > May I suggest you avoid range and use enumerate(the_array) instead? It > might be faster. > > How does this work? > > Given > > Grid= [[0 for j in range(9)] for i in range(9)] > > for (r,c,val) in ????(Grid) : > > Helmut for r, row_lst in enumerate(Grid): for c, val in enumerate(row_lst): -------------- next part -------------- An HTML attachment was scrubbed... URL: From jarausch at igpm.rwth-aachen.de Fri Jul 5 11:47:45 2013 From: jarausch at igpm.rwth-aachen.de (Helmut Jarausch) Date: 5 Jul 2013 15:47:45 GMT Subject: How to make this faster References: < b3o512Ftc39U3@mid.dfncis.de> Message-ID: On Fri, 05 Jul 2013 16:18:41 +0100, F?bio Santos wrote: > On 5 Jul 2013 15:59, "Helmut Jarausch" wrote: >> >> On Fri, 05 Jul 2013 13:44:57 +0100, F?bio Santos wrote: >> May I suggest you avoid range and use enumerate(the_array) instead? It >> might be faster. >> >> How does this work? >> >> Given >> >> Grid= [[0 for j in range(9)] for i in range(9)] >> >> for (r,c,val) in ????(Grid) : >> >> Helmut > > for r, row_lst in enumerate(Grid): > for c, val in enumerate(row_lst): This is only slightly faster. I assume the creation of the temporary lists "row_list" is a bit expensive. Taking 5.4 seconds it's much slower than the current champion ( 0.79 seconds ) Thanks, Helmut. From steve+comp.lang.python at pearwood.info Fri Jul 5 12:52:00 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 05 Jul 2013 16:52:00 GMT Subject: How to make this faster References: Message-ID: <51d6f9b0$0$29999$c3e8da3$5496439d@news.astraweb.com> On Fri, 05 Jul 2013 15:47:45 +0000, Helmut Jarausch wrote: > > for r, row_lst in enumerate(Grid): > > for c, val in enumerate(row_lst): > > I assume the creation of the temporary lists "row_list" is a bit > expensive. No temporary list is being created. The pre-existing list is just being grabbed, which is fast. -- Steven From steve+comp.lang.python at pearwood.info Fri Jul 5 12:24:19 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 05 Jul 2013 16:24:19 GMT Subject: How to make this faster References: Message-ID: <51d6f332$0$29999$c3e8da3$5496439d@news.astraweb.com> On Fri, 05 Jul 2013 14:54:26 +0000, Helmut Jarausch wrote: > On Fri, 05 Jul 2013 13:44:57 +0100, F?bio Santos wrote: May I suggest > you avoid range and use enumerate(the_array) instead? It might be > faster. > > How does this work? > > Given > > Grid= [[0 for j in range(9)] for i in range(9)] This creates a list containing nine lists, each one of which contains 0 repeated nine times. That can be simplified to: grid = [[0]*9 for i in range(9)] which will likely be faster, although for such a small number of lists you may not notice the difference. > for (r,c,val) in ????(Grid) : This causes a SyntaxError, as ???? is not legal Python code. I'm not really sure what you think you are trying to do here. But I think F?bio means that instead of writing code like this: for r in range(9): for c in range(9): value = grid[r][c] ... you should consider writing this: for r, row in enumerate(grid): for c, value in enumerate(row): ... -- Steven From nikos at superhost.gr Fri Jul 5 08:21:03 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Fri, 05 Jul 2013 15:21:03 +0300 Subject: Fwd: Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte In-Reply-To: <51D6B92D.1040209@superhost.gr> References: <51D6B92D.1040209@superhost.gr> Message-ID: -------- ?????? ?????? -------- ????: Re: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte ??????????: Fri, 05 Jul 2013 15:16:45 +0300 ???: ????? Gr33k ????: feedthetroll at gmx.de ???? 5/7/2013 3:06 ??, ?/? feedthetroll at gmx.de ??????: > Am Freitag, 5. Juli 2013 13:27:25 UTC+2 schrieb ????? Gr33k: >> ???? 5/7/2013 2:16 ??, ?/? Lele Gaifax ??????: >>> UnicodeDecodeError('utf-8', b'\xb6\xe3\xed\xf9\xf3 >>> first string-------^^^^^^^ >>> second string---------------^^^^^^^^^^^^^^^^^^^^^^ >> >> Hold on please! >> From where do these dashes and carets characters come from? > > ROTFL!!!! > Nikos, you made my day, again! Fun is back in these threads! Oh my God! i though they were the actual sting not that the pointed to the string themselves! I ebn laughted at me! >> Also from where do you see 2 strings? > > Look, my little dumb baby: The dashes and carets point to the strings. > The first one being 'utf-8', the second one starting with '\xb6\xe3\xed' (being a bytestring, therefore the b before the ') > Sorry, I forgot. You are not using python for your business, therefore you can't know, that strings in python can for example be identified by surrounding '. Indeed, but where is the 2nd sttring coming from? >> ic an understandn onlt that utf-8 has failsed decoding some byte stream >> satrting with \xb6 > So ... where did you call utf-8() so that it could try to decode something? Thats still unknown, At least i manages to solve this by: try: host = socket.gethostbyaddr( os.environ['HTTP_CF_CONNECTING_IP'] )[0] except Exception as e: host = repr(e) Seems like when you cloudflare a domain you can no longer have the originates ip address of the visitor but you have to read the above environmental variable to be bale to retrieve it! -- What is now proved was at first only imagined! -- What is now proved was at first only imagined! -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve+comp.lang.python at pearwood.info Fri Jul 5 12:59:54 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 05 Jul 2013 16:59:54 GMT Subject: How to check for threads being finished? Message-ID: <51d6fb8a$0$29999$c3e8da3$5496439d@news.astraweb.com> I have a pool of worker threads, created like this: threads = [MyThread(*args) for i in range(numthreads)] for t in threads: t.start() I then block until the threads are all done: while any(t.isAlive() for t in threads): pass Is that the right way to wait for the threads to be done? Should I stick a call to time.sleep() inside the while loop? If so, how long should I sleep? That's probably an unanswerable question, but some guidelines on choosing the sleep time will be appreciated. -- Steven From rosuav at gmail.com Fri Jul 5 13:11:09 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 6 Jul 2013 03:11:09 +1000 Subject: How to check for threads being finished? In-Reply-To: <51d6fb8a$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d6fb8a$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sat, Jul 6, 2013 at 2:59 AM, Steven D'Aprano wrote: > I then block until the threads are all done: > > while any(t.isAlive() for t in threads): > pass > Using the threading module, I assume. Is there any reason you can't simply join() each thread in succession? ChrisA From irmen.NOSPAM at xs4all.nl Fri Jul 5 13:12:44 2013 From: irmen.NOSPAM at xs4all.nl (Irmen de Jong) Date: Fri, 05 Jul 2013 19:12:44 +0200 Subject: How to check for threads being finished? In-Reply-To: <51d6fb8a$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d6fb8a$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51d6fe8b$0$15956$e4fe514c@news.xs4all.nl> On 5-7-2013 18:59, Steven D'Aprano wrote: > I then block until the threads are all done: > > while any(t.isAlive() for t in threads): > pass > > > Is that the right way to wait for the threads to be done? Should I stick > a call to time.sleep() inside the while loop? If so, how long should I > sleep? That's probably an unanswerable question, but some guidelines on > choosing the sleep time will be appreciated. > I think your while loop busy-waits until the threads are completed. Do this instead: for t in threads: t.join() # optionally pass a timeout to join -Irmen From steve+comp.lang.python at pearwood.info Fri Jul 5 22:18:23 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 06 Jul 2013 02:18:23 GMT Subject: How to check for threads being finished? References: <51d6fb8a$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d6fe8b$0$15956$e4fe514c@news.xs4all.nl> Message-ID: <51d77e6e$0$29999$c3e8da3$5496439d@news.astraweb.com> On Fri, 05 Jul 2013 19:12:44 +0200, Irmen de Jong wrote: > On 5-7-2013 18:59, Steven D'Aprano wrote: >> I then block until the threads are all done: >> >> while any(t.isAlive() for t in threads): >> pass >> >> >> Is that the right way to wait for the threads to be done? Should I >> stick a call to time.sleep() inside the while loop? If so, how long >> should I sleep? That's probably an unanswerable question, but some >> guidelines on choosing the sleep time will be appreciated. >> >> > I think your while loop busy-waits until the threads are completed. Do > this instead: > > for t in threads: t.join() # optionally pass a timeout to join Yes, the busy-wait was what I was concerned about. Thanks to everyone who suggested the same solution. -- Steven From stefan_ml at behnel.de Sat Jul 6 04:49:00 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 06 Jul 2013 10:49:00 +0200 Subject: How to check for threads being finished? In-Reply-To: <51d6fe8b$0$15956$e4fe514c@news.xs4all.nl> References: <51d6fb8a$0$29999$c3e8da3$5496439d@news.astraweb.com> <51d6fe8b$0$15956$e4fe514c@news.xs4all.nl> Message-ID: Irmen de Jong, 05.07.2013 19:12: > On 5-7-2013 18:59, Steven D'Aprano wrote: >> I then block until the threads are all done: >> >> while any(t.isAlive() for t in threads): >> pass >> >> >> Is that the right way to wait for the threads to be done? Should I stick >> a call to time.sleep() inside the while loop? If so, how long should I >> sleep? That's probably an unanswerable question, but some guidelines on >> choosing the sleep time will be appreciated. >> > > I think your while loop busy-waits until the threads are completed. > Do this instead: > > for t in threads: t.join() # optionally pass a timeout to join A related stdlib tool that many people don't seem to know is the thread pool in concurrent.futures. It supports both waiting for all threads to finish as well as iterating over results as they come in. It also comes with a parallel map() implementation. http://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor-example http://docs.python.org/3/library/concurrent.futures.html#module-functions New in Py3.2, but there's also a backport AFAIR. Stefan From ian.g.kelly at gmail.com Fri Jul 5 13:18:31 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Fri, 5 Jul 2013 11:18:31 -0600 Subject: How to check for threads being finished? In-Reply-To: <51d6fb8a$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d6fb8a$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Fri, Jul 5, 2013 at 10:59 AM, Steven D'Aprano wrote: > I have a pool of worker threads, created like this: > > threads = [MyThread(*args) for i in range(numthreads)] > for t in threads: > t.start() > > > I then block until the threads are all done: > > while any(t.isAlive() for t in threads): > pass > > > Is that the right way to wait for the threads to be done? Should I stick > a call to time.sleep() inside the while loop? If so, how long should I > sleep? That's probably an unanswerable question, but some guidelines on > choosing the sleep time will be appreciated. for thread in threads: thread.join() From cs at zip.com.au Fri Jul 5 20:45:13 2013 From: cs at zip.com.au (Cameron Simpson) Date: Sat, 6 Jul 2013 10:45:13 +1000 Subject: How to check for threads being finished? In-Reply-To: <51d6fb8a$0$29999$c3e8da3$5496439d@news.astraweb.com> References: <51d6fb8a$0$29999$c3e8da3$5496439d@news.astraweb.com> Message-ID: <20130706004513.GA98112@cskk.homeip.net> On 05Jul2013 16:59, Steven D'Aprano wrote: | I have a pool of worker threads, created like this: | | threads = [MyThread(*args) for i in range(numthreads)] | for t in threads: | t.start() | | I then block until the threads are all done: | | while any(t.isAlive() for t in threads): | pass This spins. How about: for t in threads: t.join() Blocks instead of polls. Cheers, -- Cameron Simpson Processes are like potatoes. - NCR device driver manual From nikos at superhost.gr Fri Jul 5 15:08:33 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Fri, 05 Jul 2013 22:08:33 +0300 Subject: Geo Location extracted from visitors ip address Message-ID: Is there a way to extract out of some environmental variable the Geo location of the user being the city the user visits out website from? Perhaps by utilizing his originated ip address? -- What is now proved was at first only imagined! From joel.goldstick at gmail.com Fri Jul 5 15:23:21 2013 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Fri, 5 Jul 2013 15:23:21 -0400 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: On Fri, Jul 5, 2013 at 3:08 PM, ????? Gr33k wrote: > Is there a way to extract out of some environmental variable the Geo > location of the user being the city the user visits out website from? > > Perhaps by utilizing his originated ip address? > > -- > What is now proved was at first only imagined! > -- > http://mail.python.org/**mailman/listinfo/python-list > Google shows lots of results. This one might work for you: http://freegeoip.net/ -- Joel Goldstick http://joelgoldstick.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From malaclypse2 at gmail.com Fri Jul 5 15:28:15 2013 From: malaclypse2 at gmail.com (Jerry Hill) Date: Fri, 5 Jul 2013 15:28:15 -0400 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: On Fri, Jul 5, 2013 at 3:08 PM, ????? Gr33k wrote: > Is there a way to extract out of some environmental variable the Geo > location of the user being the city the user visits out website from? > > Perhaps by utilizing his originated ip address? No, you'd need to take the originating IP address and look it up in a geolocation database or submit it to a geolocation service and get the response back from them. It's not stored in any environment variables. -- Jerry From nikos at superhost.gr Fri Jul 5 15:43:16 2013 From: nikos at superhost.gr (=?UTF-8?B?U3VwcG9ydCBieSDOnc6vzrrOv8+C?=) Date: Fri, 05 Jul 2013 22:43:16 +0300 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: <51D721D4.6020205@superhost.gr> ???? 5/7/2013 10:28 ??, ?/? Jerry Hill ??????: > On Fri, Jul 5, 2013 at 3:08 PM, ????? Gr33k wrote: >> Is there a way to extract out of some environmental variable the Geo >> location of the user being the city the user visits out website from? >> >> Perhaps by utilizing his originated ip address? > No, you'd need to take the originating IP address and look it up in a > geolocation database or submit it to a geolocation service and get the > response back from them. It's not stored in any environment > variables. I see but i dont know how to do this Geo location lookups. Can you give me more specific details please? -- Webhost -------------- next part -------------- An HTML attachment was scrubbed... URL: From python.list at tim.thechases.com Fri Jul 5 15:58:09 2013 From: python.list at tim.thechases.com (Tim Chase) Date: Fri, 5 Jul 2013 14:58:09 -0500 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: <20130705145809.3ef548f8@bigbox.christie.dr> On 2013-07-05 22:08, ????? Gr33k wrote: > Is there a way to extract out of some environmental variable the > Geo location of the user being the city the user visits out website > from? > > Perhaps by utilizing his originated ip address? Yep. You can get an 11MB database (17MB uncompressed) http://dev.maxmind.com/geoip/legacy/downloadable/ which you can use to either populate an existing database with the .CSV data there, or use the binary data blob in concert with the Python API https://github.com/maxmind/geoip-api-python # Python + C http://pypi.python.org/pypi/pygeoip/ # pure Python Just be sure to adhere to the licensing terms. -tkc From nikos at superhost.gr Fri Jul 5 15:59:00 2013 From: nikos at superhost.gr (=?UTF-8?B?U3VwcG9ydCBieSDOnc6vzrrOv8+C?=) Date: Fri, 05 Jul 2013 22:59:00 +0300 Subject: Geo Location extracted from visitors ip address In-Reply-To: <20130705145809.3ef548f8@bigbox.christie.dr> References: <20130705145809.3ef548f8@bigbox.christie.dr> Message-ID: <51D72584.8080307@superhost.gr> ???? 5/7/2013 10:58 ??, ?/? Tim Chase ??????: > On 2013-07-05 22:08, ????? Gr33k wrote: >> Is there a way to extract out of some environmental variable the >> Geo location of the user being the city the user visits out website >> from? >> >> Perhaps by utilizing his originated ip address? > Yep. You can get an 11MB database (17MB uncompressed) > > http://dev.maxmind.com/geoip/legacy/downloadable/ > > which you can use to either populate an existing database with > the .CSV data there, or use the binary data blob in concert with the > Python API > > https://github.com/maxmind/geoip-api-python # Python + C > http://pypi.python.org/pypi/pygeoip/ # pure Python > > Just be sure to adhere to the licensing terms. > > -tkc > > > > > Thank you ill take a look on http://pypi.python.org/pypi/pygeoip/ # pure Python i hope it will be easy! -- Webhost -------------- next part -------------- An HTML attachment was scrubbed... URL: From python.list at tim.thechases.com Fri Jul 5 18:21:03 2013 From: python.list at tim.thechases.com (Tim Chase) Date: Fri, 5 Jul 2013 17:21:03 -0500 Subject: Geo Location extracted from visitors ip address In-Reply-To: <51D72584.8080307@superhost.gr> References: <20130705145809.3ef548f8@bigbox.christie.dr> <51D72584.8080307@superhost.gr> Message-ID: <20130705172103.59c525c4@bigbox.christie.dr> On 2013-07-05 22:59, Support by ????? wrote: > ???? 5/7/2013 10:58 ??, ?/? Tim Chase ??????: > > On 2013-07-05 22:08, ????? Gr33k wrote: > >> Is there a way to extract out of some environmental variable the > >> Geo location of the user being the city the user visits out > >> website from? > >> > >> Perhaps by utilizing his originated ip address? > > Yep. You can get an 11MB database (17MB uncompressed) > > > > http://dev.maxmind.com/geoip/legacy/downloadable/ > > > Thank you ill take a look Alternatively, you[1] can use JavaScript in your code and have it submit AJAX requests. This interacts with the browser, asking it where (geographically) it thinks it is. This interacts with a GPS some browsers (particularly mobile), while on others, it trusts the user to report where they are. Note that at any time, the user can decline to answer or blatantly lie[2]. -tkc [1] generic "you"...this requires a fair bit of coding skill and the ability to read/understand documentation that I can't say I've seen demonstrated in certain recent threads. [2] https://addons.mozilla.org/en-us/firefox/addon/geolocater/ From timr at probo.com Fri Jul 5 16:44:40 2013 From: timr at probo.com (Tim Roberts) Date: Fri, 05 Jul 2013 13:44:40 -0700 Subject: Geo Location extracted from visitors ip address References: Message-ID: ????? Gr33k wrote: > >Is there a way to extract out of some environmental variable the Geo >location of the user being the city the user visits out website from? > >Perhaps by utilizing his originated ip address? It is possible to look up the geographic region associated with a block of IP addresses. That does not necessarily bear any resemblence to the actual location of the user. It tells you the location of the Internet provider that registered the IP addresses. -- Tim Roberts, timr at probo.com Providenza & Boekelheide, Inc. From davea at davea.name Fri Jul 5 17:18:03 2013 From: davea at davea.name (Dave Angel) Date: Fri, 05 Jul 2013 17:18:03 -0400 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: On 07/05/2013 04:44 PM, Tim Roberts wrote: > ????? Gr33k wrote: >> >> Is there a way to extract out of some environmental variable the Geo >> location of the user being the city the user visits out website from? >> >> Perhaps by utilizing his originated ip address? > > It is possible to look up the geographic region associated with a block of > IP addresses. That does not necessarily bear any resemblence to the actual > location of the user. It tells you the location of the Internet provider > that registered the IP addresses. > To be more specific, the last time I checked, my IP address is registered to a location over 1000 miles away. And it's not a dedicated IP address, so it could very well be used tomorrow by someone 200 nmiles the other side of the registered location. Further, I sometimes use other ISP's with different addresses, even when I don't leave the house. And once I leave, anything's possible. The only things that travel with me and my machine are the cookies. So if someone has asked me my location, and if I told the truth, then that information could remain available till I flush my cookies or change Virtual Machines or even browsers. -- DaveA From rosuav at gmail.com Fri Jul 5 21:06:33 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 6 Jul 2013 11:06:33 +1000 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: On Sat, Jul 6, 2013 at 7:18 AM, Dave Angel wrote: > On 07/05/2013 04:44 PM, Tim Roberts wrote: >> >> ????? Gr33k wrote: >>> >>> >>> Is there a way to extract out of some environmental variable the Geo >>> location of the user being the city the user visits out website from? >>> >>> Perhaps by utilizing his originated ip address? >> >> >> It is possible to look up the geographic region associated with a block of >> IP addresses. That does not necessarily bear any resemblence to the >> actual >> location of the user. It tells you the location of the Internet provider >> that registered the IP addresses. >> > > To be more specific, the last time I checked, my IP address is registered to > a location over 1000 miles away. And it's not a dedicated IP address, so it > could very well be used tomorrow by someone 200 nmiles the other side of the > registered location. > > Further, I sometimes use other ISP's with different addresses, even when I > don't leave the house. And once I leave, anything's possible. That said, though, geolocation is usually accurate enough to report which country you're in. For basic statistical analysis, that's generally sufficient. ChrisA From invalid at invalid.invalid Fri Jul 5 18:24:08 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Fri, 5 Jul 2013 22:24:08 +0000 (UTC) Subject: Geo Location extracted from visitors ip address References: Message-ID: On 2013-07-05, ?????????? Gr33k wrote: > Is there a way to extract out of some environmental variable the Geo > location of the user being the city the user visits out website from? No. > Perhaps by utilizing his originated ip address? There is a very poor correlation between IP address and geographical location. I know that users of the ISPs in hometown are consistently mis-identified as being from towns 1500km away. -- Grant Edwards grant.b.edwards Yow! If elected, Zippy at pledges to each and every gmail.com American a 55-year-old houseboy ... From nikos at superhost.gr Fri Jul 5 19:55:43 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Sat, 06 Jul 2013 02:55:43 +0300 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: ???? 5/7/2013 10:58 ??, ?/? Tim Chase ??????: > On 2013-07-05 22:08, ????? Gr33k wrote: >> Is there a way to extract out of some environmental variable the >> Geo location of the user being the city the user visits out website >> from? >> >> Perhaps by utilizing his originated ip address? > > Yep. You can get an 11MB database (17MB uncompressed) > > http://dev.maxmind.com/geoip/legacy/downloadable/ > http://pypi.python.org/pypi/pygeoip/ # pure Python Thank you very much Tim. i am know tryitn to use it as: import pygeoip try: gic = pygeoip.GeoIP('/root/GeoIPCity.dat') host = gic.time_zone_by_addr( os.environ['HTTP_CF_CONNECTING_IP'] ) except Exception as e: host = repr(e) lets hope it will work! -- What is now proved was at first only imagined! From nikos at superhost.gr Fri Jul 5 19:58:50 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Sat, 06 Jul 2013 02:58:50 +0300 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: ???? 6/7/2013 2:55 ??, ?/? ????? Gr33k ??????: > ???? 5/7/2013 10:58 ??, ?/? Tim Chase ??????: >> On 2013-07-05 22:08, ????? Gr33k wrote: >>> Is there a way to extract out of some environmental variable the >>> Geo location of the user being the city the user visits out website >>> from? >>> >>> Perhaps by utilizing his originated ip address? >> >> Yep. You can get an 11MB database (17MB uncompressed) >> >> http://dev.maxmind.com/geoip/legacy/downloadable/ > >> http://pypi.python.org/pypi/pygeoip/ # pure Python > > Thank you very much Tim. > i am know trying to use it as: > > import pygeoip > > try: > gic = pygeoip.GeoIP('/root/GeoIPCity.dat') > host = gic.time_zone_by_addr( os.environ['HTTP_CF_CONNECTING_IP'] ) > except Exception as e: > host = repr(e) > > lets hope it will work! Just my luck again, PermissionError(13, '?????? ?????????') ?????? ????????? = Access Denied Why would that happen? -- What is now proved was at first only imagined! From nikos at superhost.gr Fri Jul 5 20:08:04 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Sat, 06 Jul 2013 03:08:04 +0300 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: ???? 6/7/2013 2:58 ??, ?/? ????? Gr33k ??????: > ???? 6/7/2013 2:55 ??, ?/? ????? Gr33k ??????: >> ???? 5/7/2013 10:58 ??, ?/? Tim Chase ??????: >>> On 2013-07-05 22:08, ????? Gr33k wrote: >>>> Is there a way to extract out of some environmental variable the >>>> Geo location of the user being the city the user visits out website >>>> from? >>>> >>>> Perhaps by utilizing his originated ip address? >>> >>> Yep. You can get an 11MB database (17MB uncompressed) >>> >>> http://dev.maxmind.com/geoip/legacy/downloadable/ >> >>> http://pypi.python.org/pypi/pygeoip/ # pure Python >> >> Thank you very much Tim. >> i am know trying to use it as: >> >> import pygeoip >> >> try: >> gic = pygeoip.GeoIP('/root/GeoIPCity.dat') >> host = gic.time_zone_by_addr( os.environ['HTTP_CF_CONNECTING_IP'] ) >> except Exception as e: >> host = repr(e) >> >> lets hope it will work! > > Just my luck again, > > PermissionError(13, '?????? ?????????') > > ?????? ????????? = Access Denied > > Why would that happen? root at nikos [~]# ls -l GeoLiteCity.dat -rw-r--r-- 1 root root 17633968 Jul 3 02:11 GeoLiteCity.dat root at nikos [~]# chmod +x GeoLiteCity.dat root at nikos [~]# ls -l GeoLiteCity.dat -rwxr-xr-x 1 root root 17633968 Jul 3 02:11 GeoLiteCity.dat* root at nikos [~]# python Python 3.3.2 (default, Jun 3 2013, 16:18:05) [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pygeoip >>> gic = pygeoip.GeoIP('/root/GeoIPCity.dat') Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.3/site-packages/pygeoip-0.2.6-py3.3.egg/pygeoip/__init__.py", line 110, in __init__ self._filehandle = codecs.open(filename, 'rb', ENCODING) File "/usr/local/lib/python3.3/codecs.py", line 884, in open file = builtins.open(filename, mode, buffering) FileNotFoundError: [Errno 2] No such file or directory: '/root/GeoIPCity.dat' >>> -- What is now proved was at first only imagined! From joel.goldstick at gmail.com Fri Jul 5 20:56:25 2013 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Fri, 5 Jul 2013 20:56:25 -0400 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: On Fri, Jul 5, 2013 at 8:08 PM, ????? Gr33k wrote: > ???? 6/7/2013 2:58 ??, ?/? ????? Gr33k ??????: > > ???? 6/7/2013 2:55 ??, ?/? ????? Gr33k ??????: >> >>> ???? 5/7/2013 10:58 ??, ?/? Tim Chase ??????: >>> >>>> On 2013-07-05 22:08, ????? Gr33k wrote: >>>> >>>>> Is there a way to extract out of some environmental variable the >>>>> Geo location of the user being the city the user visits out website >>>>> from? >>>>> >>>>> Perhaps by utilizing his originated ip address? >>>>> >>>> >>>> Yep. You can get an 11MB database (17MB uncompressed) >>>> >>>> http://dev.maxmind.com/geoip/**legacy/downloadable/ >>>> >>> >>> http://pypi.python.org/pypi/**pygeoip/ # pure Python >>>> >>> >>> Thank you very much Tim. >>> i am know trying to use it as: >>> >>> import pygeoip >>> >>> try: >>> gic = pygeoip.GeoIP('/root/**GeoIPCity.dat') >>> host = gic.time_zone_by_addr( os.environ['HTTP_CF_**CONNECTING_IP'] ) >>> except Exception as e: >>> host = repr(e) >>> >>> lets hope it will work! >>> >> >> Just my luck again, >> >> PermissionError(13, '?????? ?????????') >> >> ?????? ????????? = Access Denied >> >> Why would that happen? >> > > root at nikos [~]# ls -l GeoLiteCity.dat > -rw-r--r-- 1 root root 17633968 Jul 3 02:11 GeoLiteCity.dat > root at nikos [~]# chmod +x GeoLiteCity.dat > root at nikos [~]# ls -l GeoLiteCity.dat > -rwxr-xr-x 1 root root 17633968 Jul 3 02:11 GeoLiteCity.dat* > root at nikos [~]# python > Python 3.3.2 (default, Jun 3 2013, 16:18:05) > [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> import pygeoip > > >>> gic = pygeoip.GeoIP('/root/**GeoIPCity.dat') > Traceback (most recent call last): > File "", line 1, in > File "/usr/local/lib/python3.3/**site-packages/pygeoip-0.2.6-** > py3.3.egg/pygeoip/__init__.py"**, line 110, in __init__ > self._filehandle = codecs.open(filename, 'rb', ENCODING) > File "/usr/local/lib/python3.3/**codecs.py", line 884, in open > file = builtins.open(filename, mode, buffering) > Your code is not finding /root/GeoIPCity.dat because your directory has this file: GeoLiteCity.dat > FileNotFoundError: [Errno 2] No such file or directory: > '/root/GeoIPCity.dat' Aside from that you might have some permission problems since the file is owned by root. You should go back to old threads where this issue was explained. As was also pointed out, you only get information about where your isp is located. Phones and tablets find location from triangulating cell towers. I don't think that laptops have that capability, and desktops probably even less likely. What is the purpose that you wish to serve. I don't think you've thought this through. > > >>> > > > > -- > What is now proved was at first only imagined! > -- > http://mail.python.org/**mailman/listinfo/python-list > -- Joel Goldstick http://joelgoldstick.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From nikos at superhost.gr Fri Jul 5 21:10:24 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Sat, 06 Jul 2013 04:10:24 +0300 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: ???? 6/7/2013 3:56 ??, ?/? Joel Goldstick ??????: > > Your code is not finding /root/GeoIPCity.dat because your directory has > this file: GeoLiteCity.dat > > FileNotFoundError: [Errno 2] No such file or directory: > '/root/GeoIPCity.dat' My mistake. Is there a differnce between GeoLiteCity.dat and GeoIPCity.dat > Aside from that you might have some permission problems since the file > is owned by root. But he cgi scripts when running have full access to the server. No? or they only have the kind of access that their user has also? > As was also pointed out, you only get information about where your isp > is located. Its the best i can get to, since there is no other way to match the users city. ? Phones and tablets find location from triangulating cell > towers.? I don't think that laptops have that capability, and desktops > probably even less likely. What do you mean by that? > What is the purpose that you wish to serve.? I don't think you've > thought this through. I just dont want to store visitor's ip addresses any more, i prefer to store its city of origin. -- What is now proved was at first only imagined! From joel.goldstick at gmail.com Fri Jul 5 21:31:46 2013 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Fri, 5 Jul 2013 21:31:46 -0400 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: On Fri, Jul 5, 2013 at 9:10 PM, ????? Gr33k wrote: > ???? 6/7/2013 3:56 ??, ?/? Joel Goldstick ??????: > >> >> Your code is not finding /root/GeoIPCity.dat because your directory has >> this file: GeoLiteCity.dat >> >> FileNotFoundError: [Errno 2] No such file or directory: >> '/root/GeoIPCity.dat' >> > > My mistake. > Is there a differnce between GeoLiteCity.dat and GeoIPCity.dat > > > Aside from that you might have some permission problems since the file >> is owned by root. >> > > But he cgi scripts when running have full access to the server. > No? or they only have the kind of access that their user has also? > > > > > As was also pointed out, you only get information about where your isp >> is located. >> > Its the best i can get to, since there is no other way to match the users > city. > > ? Phones and tablets find location from triangulating cell > >> towers.? I don't think that laptops have that capability, and desktops >> probably even less likely. >> > > What do you mean by that? > > > What is the purpose that you wish to serve.? I don't think you've >> thought this through. >> > > I just dont want to store visitor's ip addresses any more, i prefer to > store its city of origin. > > Except its the ISP city that you are getting, not the user's > > > -- > What is now proved was at first only imagined! > -- > http://mail.python.org/**mailman/listinfo/python-list > -- Joel Goldstick http://joelgoldstick.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From nikos at superhost.gr Fri Jul 5 21:41:10 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Sat, 06 Jul 2013 04:41:10 +0300 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: Yes i know iam only storing the ISP's city instead of visitor's homeland but this is the closest i can get: try: gi = pygeoip.GeoIP('/home/nikos/GeoLiteCity.dat') city = gi.time_zone_by_addr( os.environ['HTTP_CF_CONNECTING_IP'] ) host = socket.gethostbyaddr( os.environ['HTTP_CF_CONNECTING_IP'] ) except Exception as e: host = repr(e) Tried it myself and it falsey said that i'am from Europe/Athens (capital of Greece) while i'am from Europe/Thessaloniki (sub-capital of Greece) If we can pin-point the uvisitor more accurately plz let me know. -- What is now proved was at first only imagined! From nikos at superhost.gr Sat Jul 6 04:01:46 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Sat, 06 Jul 2013 11:01:46 +0300 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: ???? 6/7/2013 4:41 ??, ?/? ????? Gr33k ??????: > Yes i know iam only storing the ISP's city instead of visitor's homeland > but this is the closest i can get: > > try: > gi = pygeoip.GeoIP('/home/nikos/GeoLiteCity.dat') > city = gi.time_zone_by_addr( os.environ['HTTP_CF_CONNECTING_IP'] ) > host = socket.gethostbyaddr( os.environ['HTTP_CF_CONNECTING_IP'] ) > except Exception as e: > host = repr(e) > > > Tried it myself and it falsey said that i'am from Europe/Athens (capital > of Greece) while i'am from Europe/Thessaloniki (sub-capital of Greece) > > If we can pin-point the uvisitor more accurately plz let me know. Good morning from Greece, All my Greece visitors as Dave correctly said have the ISP address which here in Greece is Europe/Athens, so i have now way to distinct the cities of the visitors. Is there any way to pinpoint the visitor's exact location? -- What is now proved was at first only imagined! From rosuav at gmail.com Sat Jul 6 04:30:34 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 6 Jul 2013 18:30:34 +1000 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: On Sat, Jul 6, 2013 at 6:01 PM, ????? Gr33k wrote: > Is there any way to pinpoint the visitor's exact location? Yes. You ask them to fill in a shipping address. They may still lie, or they may choose to not answer, but that's the best you're going to achieve without getting a wizard to cast Scrying. ChrisA From nikos at superhost.gr Sat Jul 6 04:41:58 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Sat, 06 Jul 2013 11:41:58 +0300 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: ???? 6/7/2013 11:30 ??, ?/? Chris Angelico ??????: > On Sat, Jul 6, 2013 at 6:01 PM, ????? Gr33k wrote: >> Is there any way to pinpoint the visitor's exact location? > > Yes. You ask them to fill in a shipping address. They may still lie, > or they may choose to not answer, but that's the best you're going to > achieve without getting a wizard to cast Scrying. No, no registration requirements. you know when i go to maps.google.com its always find my exact city of location and not just say Europe/Athens. and twitter and facebook too both of them pinpoint my _exact_ location. How are they able to do it? We need the same way. -- What is now proved was at first only imagined! From robert.kern at gmail.com Sat Jul 6 06:21:27 2013 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 06 Jul 2013 11:21:27 +0100 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: On 2013-07-06 09:41, ????? Gr33k wrote: > ???? 6/7/2013 11:30 ??, ?/? Chris Angelico ??????: >> On Sat, Jul 6, 2013 at 6:01 PM, ????? Gr33k wrote: >>> Is there any way to pinpoint the visitor's exact location? >> >> Yes. You ask them to fill in a shipping address. They may still lie, >> or they may choose to not answer, but that's the best you're going to >> achieve without getting a wizard to cast Scrying. > > No, no registration requirements. > > you know when i go to maps.google.com its always find my exact city of location > and not just say Europe/Athens. > > and twitter and facebook too both of them pinpoint my _exact_ location. > > How are they able to do it? We need the same way. They use client-side JavaScript. This is a relatively new API available in most, but not all, recent browsers. This information will not be available to your CGI script. You will have to generate HTML with the proper JavaScript to get the geolocation (if the user allows it) and then send it back to your server through a different CGI script (or web application endpoint). http://diveintohtml5.info/geolocation.html -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From python.list at tim.thechases.com Sat Jul 6 07:20:16 2013 From: python.list at tim.thechases.com (Tim Chase) Date: Sat, 6 Jul 2013 06:20:16 -0500 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: <20130706062016.136caaf1@bigbox.christie.dr> On 2013-07-06 11:41, ????? Gr33k wrote: > you know when i go to maps.google.com its always find my exact city > of location and not just say Europe/Athens. > > and twitter and facebook too both of them pinpoint my _exact_ > location. > > How are they able to do it? We need the same way. A couple possibilities: 1) using the aforementioned HTML5 location API, your device may be tattling on where you are. Are you browsing from a smart-phone or other device with a GPS built in? 2) at some point in the distant past, you told Google where you are, and it has dutifully remembered that. Try using an alternate browser in a new session (Firefox has the ability to create a new profile; Chrome/Chromium should have the ability to start up with a virgin profile; I can't say for Safari or IE) and see if Google suddenly lacks the ability to locate you 3) Google has a better IP-to-location map database than you have. You might have to pay real money for such functionality. Or, you might have to use a different library, as the IP-to-location database that I linked you to earlier has both an "IP to Country" and an "IP to City" database. Note that this is often wrong or grossly inaccurate, as mentioned in other threads (geolocation by IP address often puts me in the nearest major city which is a good 45min drive away, and if I just visit Google maps with a fresh browser, it just shows me the state, TX, which is a ~13hr drive across, if done at 65mph the whole way) -tkc From davea at davea.name Sat Jul 6 07:49:07 2013 From: davea at davea.name (Dave Angel) Date: Sat, 06 Jul 2013 07:49:07 -0400 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: On 07/06/2013 04:41 AM, ????? Gr33k wrote: > ???? 6/7/2013 11:30 ??, ?/? Chris Angelico ??????: >> On Sat, Jul 6, 2013 at 6:01 PM, ????? Gr33k wrote: >>> Is there any way to pinpoint the visitor's exact location? >> >> Yes. You ask them to fill in a shipping address. They may still lie, >> or they may choose to not answer, but that's the best you're going to >> achieve without getting a wizard to cast Scrying. > > No, no registration requirements. > > you know when i go to maps.google.com its always find my exact city of > location and not just say Europe/Athens. > > and twitter and facebook too both of them pinpoint my _exact_ location. > > How are they able to do it? We need the same way. > At some point, you entered your address, and it's stored in some database in the sky. You have cookies on your machine which correlate to that database. Chances are you did it for google-maps, and google shared it with their search engine and other parts. As far as I know, each such company has a separate database, but perhaps google (for exakmple) has an partner API which facebook uses. -- DaveA From nikos at superhost.gr Sat Jul 6 16:14:29 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Sat, 06 Jul 2013 23:14:29 +0300 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: ???? 6/7/2013 2:20 ??, ?/? Tim Chase ??????: > 1) using the aforementioned HTML5 location API, your device may be > tattling on where you are. Are you browsing from a smart-phone or > other device with a GPS built in? I'm using my lenovo laptop, by maps.gogole.com, fb and twitter have no problem pionpoint my exact location, even postal code. How do they do it? Can you be more specific please about using the aforementioned HTML5 location API ? Never heard of it. Can it be utilizized via a python cgi script? -- What is now proved was at first only imagined! From python.list at tim.thechases.com Sat Jul 6 16:32:07 2013 From: python.list at tim.thechases.com (Tim Chase) Date: Sat, 6 Jul 2013 15:32:07 -0500 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: <20130706153207.236ccc7d@bigbox.christie.dr> On 2013-07-06 23:14, ????? Gr33k wrote: Can you be more specific please about using the aforementioned > HTML5 location API ? https://www.google.com/search?q=html5+location+api It's client-side JavaScript. > Never heard of it. Can it be utilizized via a python cgi script? Because it's client-side JavaScript, it runs, well, on the client's browser. Note that the user may be prompted regarding whether they want to permit the website to access location information, so this information may not be available. If the user permits and JS is enabled, the client-side JS code can then make AJAX requests (or stash it in a cookie that gets sent with future requests) to convey the location information to the server where your Python code is running. -tkc From nikos at superhost.gr Sat Jul 6 16:51:52 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Sat, 06 Jul 2013 23:51:52 +0300 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: ???? 6/7/2013 11:32 ??, ?/? Tim Chase ??????: > Can you be more specific please about using the aforementioned >> HTML5 location API ? > https://www.google.com/search?q=html5+location+api > > It's client-side JavaScript. so, i must edit my cgi script and do this: print ''' ''' Will that do the trick? but then again i want the city to be stored in the city variable. Somehow the above javascript code mu return me a value that i will the store at variable "city". I don't know how to do that. -- What is now proved was at first only imagined! From nikos at superhost.gr Sun Jul 7 05:40:57 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Sun, 07 Jul 2013 12:40:57 +0300 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: ???? 6/7/2013 11:51 ??, ?/? ????? Gr33k ??????: > ???? 6/7/2013 11:32 ??, ?/? Tim Chase ??????: >> Can you be more specific please about using the aforementioned >>> HTML5 location API ? >> https://www.google.com/search?q=html5+location+api >> >> It's client-side JavaScript. > > > so, i must edit my cgi script and do this: > > print ''' > > ''' > > Will that do the trick? > > but then again i want the city to be stored in the city variable. > Somehow the above javascript code mu return me a value that i will the > store at variable "city". > > I don't know how to do that. I had a reply of another person telling me these: Google, Facebook, Microsoft, Amazon and most other gigantic companies with lots of money track you in several different ways, not just by the IP. They compare several categories of tracking to generate a list of possible locations for you and then pick the one with the highest confidence. For example, I have an AU phone. If I register with AU Cloud that also registers me with Google, and then my AU tower, IP and GPS location all get reported to Google. When I login later on a desktop to the same GoogleID account, they only have my IP and tracking cookies to look at, but they already know to check the latest location of my phone -- and whether its turned on/permitting GPS updates right then affects the confidence report % of that method of tracking. Recent reservations, dated product/service reviews, driving directions, map inquiries, map bookmarks/pins, etc. all give some confidence for frequented location and movement history each. Any billing relationship you have with them will give them another tracking point based on your billing address, and they can compare the billing address with frequented GPS locs, past shipping information and recent locale-oriented searches. The more recent the data and the more points of data match the same location the more confidence the potential location has. ...and so on. Its pretty creepy, actually. Anyway, you can't just do this using IP information. To get reliable, live, pinpoint user location data you need to do one of: Convince the user to report/register/pick their location Convince the user to permit you to track their phone Get a contract with Google that buys you their best guess at user location Be like Google and engage in a conspiracy to invade the privacy of millions that dwarfs the resources of most intelligence agencies (and then sell it to intelligence agencies, just like Google does) -- What is now proved was at first only imagined! From invalid at invalid.invalid Mon Jul 8 10:27:36 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Mon, 8 Jul 2013 14:27:36 +0000 (UTC) Subject: Geo Location extracted from visitors ip address References: Message-ID: On 2013-07-06, ?????????? Gr33k wrote: > ???????? 6/7/2013 4:41 ????, ??/?? ?????????? Gr33k ????????????: >> Yes i know iam only storing the ISP's city instead of visitor's homeland >> but this is the closest i can get: >> >> try: >> gi = pygeoip.GeoIP('/home/nikos/GeoLiteCity.dat') >> city = gi.time_zone_by_addr( os.environ['HTTP_CF_CONNECTING_IP'] ) >> host = socket.gethostbyaddr( os.environ['HTTP_CF_CONNECTING_IP'] ) >> except Exception as e: >> host = repr(e) >> >> >> Tried it myself and it falsey said that i'am from Europe/Athens (capital >> of Greece) while i'am from Europe/Thessaloniki (sub-capital of Greece) >> >> If we can pin-point the uvisitor more accurately plz let me know. > > Good morning from Greece, > > All my Greece visitors as Dave correctly said have the ISP address which > here in Greece is Europe/Athens, so i have now way to distinct the > cities of the visitors. > > Is there any way to pinpoint the visitor's exact location? No. -- Grant Edwards grant.b.edwards Yow! Does someone from at PEORIA have a SHORTER gmail.com ATTENTION span than me? From invalid at invalid.invalid Mon Jul 8 10:27:21 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Mon, 8 Jul 2013 14:27:21 +0000 (UTC) Subject: Geo Location extracted from visitors ip address References: Message-ID: On 2013-07-06, ?????????? Gr33k wrote: > Yes i know iam only storing the ISP's city instead of visitor's homeland > but this is the closest i can get: > > try: > gi = pygeoip.GeoIP('/home/nikos/GeoLiteCity.dat') > city = gi.time_zone_by_addr( os.environ['HTTP_CF_CONNECTING_IP'] ) > host = socket.gethostbyaddr( os.environ['HTTP_CF_CONNECTING_IP'] ) > except Exception as e: > host = repr(e) > > > Tried it myself and it falsey said that i'am from Europe/Athens (capital > of Greece) while i'am from Europe/Thessaloniki (sub-capital of Greece) > > If we can pin-point the uvisitor more accurately plz let me know. For the Nth time: you can't. -- Grant Edwards grant.b.edwards Yow! HOORAY, Ronald!! at Now YOU can marry LINDA gmail.com RONSTADT too!! From nikos at superhost.gr Sat Jul 6 03:10:22 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Sat, 06 Jul 2013 10:10:22 +0300 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: ???? 6/7/2013 5:52 ??, ?/? Dennis Lee Bieber ??????: > On Sat, 06 Jul 2013 04:10:24 +0300, ????? Gr33k > declaimed the following: > >> >> But he cgi scripts when running have full access to the server. >> No? or they only have the kind of access that their user has also? >> > In any decent system, the web server runs as a particular user, and > only has access to the web content and scripts. And those scripts run as > the web server process (at most -- it may be that they run at an even more > restricted mode). > > So NO, they do NOT have access to stuff under /root; for ancient > CGI-BIN style, they may be restricted to only the files in the CGI-BIN > directory. > Thats why i was getting permission denied vene when i had +x when i moved the geo.dat file to /home/nikos/geo.dat then the cgi python script was able to opened it. It was some guy form hostgator.com that had told me that a python script has the same level of access to anything on the filesystem as its coressponding user running it, implying that if i run it under user 'root' the python script could access anything. Are you sure that python scripts run under Apache user or Nobody user in my case and not as user 'nikos' ? Is there some way to test that? -- What is now proved was at first only imagined! From nikos at superhost.gr Sat Jul 6 16:12:02 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Sat, 06 Jul 2013 23:12:02 +0300 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: ???? 6/7/2013 5:43 ??, ?/? Dennis Lee Bieber ??????: >> It was some guy form hostgator.com that had told me that a python script >> has the same level of access to anything on the filesystem as its >> coressponding user running it, implying that if i run it under user >> 'root' the python script could access anything. > Yes, IF YOU RUN IT UNDER "root"... The ownership of the script file > doesn't control the privileges it runs under as long as the file itself is > read-access to other "users". I though that the ownership of the script file controlled the privileges it runs under..... Who controlls the script's privileges then? The process that calls the script file, i.e. Apache? > as the file itself is > read-access to other "users". What do you mean by that? -- What is now proved was at first only imagined! From python.list at tim.thechases.com Sat Jul 6 16:33:43 2013 From: python.list at tim.thechases.com (Tim Chase) Date: Sat, 6 Jul 2013 15:33:43 -0500 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: <20130706153343.65bcd472@bigbox.christie.dr> On 2013-07-06 23:12, ????? Gr33k wrote: > I though that the ownership of the script file controlled the > privileges it runs under..... Only if the script is SUID. In some environments, scripts can't be run SUID, only binaries. > Who controlls the script's privileges then? > The process that calls the script file, i.e. Apache? Yes. -tkc From nikos at superhost.gr Sat Jul 6 16:49:46 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Or866zr/PgiBHcjMzaw==?=) Date: Sat, 06 Jul 2013 23:49:46 +0300 Subject: Geo Location extracted from visitors ip address In-Reply-To: References: Message-ID: ???? 6/7/2013 11:33 ??, ?/? Tim Chase ??????: >> Who controlls the script's privileges then? >> The process that calls the script file, i.e. Apache? > Yes. When we run the python interpreter to run a python script like python metrites.py then out script will inherit the kind of access the python interpreter has which in turn will inherit the kind of access the user that is run under upon has? -- What is now proved was at first only imagined! From jenn.duerr at gmail.com Fri Jul 5 15:18:13 2013 From: jenn.duerr at gmail.com (noydb) Date: Fri, 5 Jul 2013 12:18:13 -0700 (PDT) Subject: analyzing time Message-ID: <2aa041fe-8226-4fb9-9ce6-b1eb48a19e4d@googlegroups.com> Hello All, I have a table with a column of type date, with dates and time combined (like '1/6/2013 3:52:69PM'), that spans many months. How would I pull out records that are the first and last entries per day? Also, if I wanted to find time clusters per day (or per week) -- like if an entry is made every day around 11am -- is there a way to get at that temporal statistical cluster? Python 2.7, Windows 7. Any guidance would be greatly appreciated! Time seems tricky... Thanks, N From neilc at norwich.edu Fri Jul 5 15:35:33 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 5 Jul 2013 19:35:33 GMT Subject: analyzing time References: <2aa041fe-8226-4fb9-9ce6-b1eb48a19e4d@googlegroups.com> Message-ID: On 2013-07-05, noydb wrote: > Hello All, > > I have a table with a column of type date, with dates and time > combined (like '1/6/2013 3:52:69PM'), that spans many months. > How would I pull out records that are the first and last > entries per day? > > Also, if I wanted to find time clusters per day (or per week) > -- like if an entry is made every day around 11am -- is there a > way to get at that temporal statistical cluster? > > Python 2.7, Windows 7. > > Any guidance would be greatly appreciated! Time seems tricky... Time *is* really tricky, but that's because we humans created a tricky system. If can ignore issues of timespampts, timezones and daylight savings time, then time handling in Python can be simple. datetime.datetime.strptime can translate the time format above into datetime.datetime objects, which provide all the methods you will need. To find clusters and min and max values you will likely need to put the datetime objects in a list, and use some Python builtins and list methods. -- Neil Cerutti From skip at pobox.com Fri Jul 5 15:43:56 2013 From: skip at pobox.com (Skip Montanaro) Date: Fri, 5 Jul 2013 14:43:56 -0500 Subject: analyzing time In-Reply-To: <2aa041fe-8226-4fb9-9ce6-b1eb48a19e4d@googlegroups.com> References: <2aa041fe-8226-4fb9-9ce6-b1eb48a19e4d@googlegroups.com> Message-ID: > I have a table with a column of type date, with dates and time combined (like '1/6/2013 3:52:69PM'), that spans many months. How would I pull out records that are the first and last entries per day? You mentioned "table" and "column", which leads me to think you are dealing with data in a SQL database. If so, that would likely change the problem solution significantly. If you have something like lists of string data in Python though, you might want to make sure your timestamps are in a normalized form first, so you can accurately sort by the timestamps. For that, my weapon of choice is the dateutil package, and in particular, the dateutil.parser module: >>> x = dateutil.parser.parse("1/6/2013 3:52:59PM") >>> x datetime.datetime(2013, 1, 6, 15, 52, 59) >>> print x 2013-01-06 15:52:59 Once your timestamps are represented as Python datetime objects, the problem gets a bit easier, especially if you want to find the beginning and ending timestamps of a bunch of dates. Sort, then throw some itertools.groupby pixie dust at it. My ancient, reptilian brain has never quite grokked all that iterator stuff, so I won't hazard a guess how to spell the exact solution. Skip From tjreedy at udel.edu Fri Jul 5 15:54:46 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 05 Jul 2013 15:54:46 -0400 Subject: analyzing time In-Reply-To: <2aa041fe-8226-4fb9-9ce6-b1eb48a19e4d@googlegroups.com> References: <2aa041fe-8226-4fb9-9ce6-b1eb48a19e4d@googlegroups.com> Message-ID: On 7/5/2013 3:18 PM, noydb wrote: > I have a table with a column of type date, with dates and time This is a datetime in Python parlance. > combined (like '1/6/2013 3:52:69PM'), that spans many months. How > would I pull out records that are the first and last entries per > day? Sort on that column. Look at pairs of rows. If the days differ, you have the last of the first and the first of the second. One way: it = dt1 = next(it) d1 = date(dt1) # whatever that looks like for row in it: dt2 = row d2 = date(dt2) if d1 != d2: do_whatever(dt1, dt2) dt1, d1 = dt2, d2 > Also, if I wanted to find time clusters per day (or per week) -- like > if an entry is made every day around 11am -- is there a way to get at > that temporal statistical cluster? Make a histogram of time, ignoring date. > Python 2.7, Windows 7. > > Any guidance would be greatly appreciated! Time seems tricky... Yes -- Terry Jan Reedy From gherron at digipen.edu Fri Jul 5 15:47:49 2013 From: gherron at digipen.edu (Gary Herron) Date: Fri, 05 Jul 2013 12:47:49 -0700 Subject: analyzing time In-Reply-To: <2aa041fe-8226-4fb9-9ce6-b1eb48a19e4d@googlegroups.com> References: <2aa041fe-8226-4fb9-9ce6-b1eb48a19e4d@googlegroups.com> Message-ID: <51D722E5.1020708@digipen.edu> On 07/05/2013 12:18 PM, noydb wrote: > Hello All, > > I have a table with a column of type date, with dates and time combined (like '1/6/2013 3:52:69PM'), that spans many months. How would I pull out records that are the first and last entries per day? > > Also, if I wanted to find time clusters per day (or per week) -- like if an entry is made every day around 11am -- is there a way to get at that temporal statistical cluster? > > Python 2.7, Windows 7. > > Any guidance would be greatly appreciated! Time seems tricky... > > Thanks, > > N Are you asking a Python question, like how to turn a string "1/6/2013 3:52:69PM" into an internal representation of time, or are you asking a data analysis and statistical question? If the former, then look at datetime.strptime from the datetime module. If the later, then you may get an answer here, but I'd suggest trying somewhere that discusses statistics and analysis. Gary Herron -- Dr. Gary Herron Department of Computer Science DigiPen Institute of Technology (425) 895-4418 From j.chiu at cern.ch Fri Jul 5 16:53:20 2013 From: j.chiu at cern.ch (Justin Chiu) Date: Fri, 5 Jul 2013 13:53:20 -0700 Subject: AMQP listening and user-facing daemon Message-ID: <51D73240.1000001@cern.ch> Hi all, What is the best approach to writing a concurrent daemon that can execute callbacks for different types of events (AMQP messages, parsed output of a subprocess, HTTP requests)? I am considering [twisted][1], the built-in [threading][2] module, and [greenlet][3]. I must admit that I am very unfamiliar with concurrent programming and Python programming in general (formerly a data analysis driven procedural programmer). Any resources on threaded/concurrent programming (specifically daemons...not just multi-threading a single task) would be much appreciated. Thank you. Details: 1) Listens into AMQP messaging queues and executes callbacks when messages arrive. Example: Immediately after startup, the daemon continuously listens to the [Openstack Notifications messaging queue][4]. When a virtual machine is launched, a notification is generated by Openstack with the hostname, IP address, etc. The daemon should read this message and write some info to a log (or POST the info to a server, or notify the user...something simple). 2) Parse the output of a subprocess and execute callbacks based on the output. Example: Every 30 seconds, a system command "[qstat][5]" is run to query a job resource manager (e.g. TORQUE). Similar callbacks to 1). 3) Receive requests from a user and process them. I think this will be via WSGI HTTP. Example: User submits an XML template with virtual machine templates. The daemon does some simple XML parsing and writes a job script for the job resource manager. The job is submitted to the resource manager and the daemon continually checks for the status of the job with "qstat" and for messages from AMQP. It should return "live" feedback to the user and write to a log. [1]: https://twistedmatrix.com/trac/wiki/Documentation [2]: http://docs.python.org/2/library/threading.html [3]: http://greenlet.readthedocs.org/en/latest/ [4]: https://wiki.openstack.org/wiki/NotificationEventExamples#Immediate_Notifications: [5]: http://www.clusterresources.com/torquedocs21/commands/qstat.shtml Justin Chiu TRIUMF From bobjim.hunter at gmail.com Fri Jul 5 18:58:17 2013 From: bobjim.hunter at gmail.com (Robert Hunter) Date: Fri, 5 Jul 2013 15:58:17 -0700 (PDT) Subject: calculating binomial coefficients using itertools Message-ID: <5d22d723-bf1b-467d-b9d3-a9a814230309@googlegroups.com> from itertools import count, repeat, izip, starmap def binomial(n): """Calculate list of Nth-order binomial coefficients using itertools.""" l = range(2) for _ in xrange(n): indices = izip(count(-1), count(1), repeat(1, len(l) + 1)) slices = starmap(slice, indices) l = [sum(l[s]) for s in slices] return l[1:] From ian.g.kelly at gmail.com Sat Jul 6 03:11:01 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Sat, 6 Jul 2013 01:11:01 -0600 Subject: calculating binomial coefficients using itertools In-Reply-To: <5d22d723-bf1b-467d-b9d3-a9a814230309@googlegroups.com> References: <5d22d723-bf1b-467d-b9d3-a9a814230309@googlegroups.com> Message-ID: On Fri, Jul 5, 2013 at 4:58 PM, Robert Hunter wrote: > from itertools import count, repeat, izip, starmap > > def binomial(n): > """Calculate list of Nth-order binomial coefficients using itertools.""" > > l = range(2) > for _ in xrange(n): > indices = izip(count(-1), count(1), repeat(1, len(l) + 1)) > slices = starmap(slice, indices) > l = [sum(l[s]) for s in slices] > return l[1:] Nice, I like seeing interesting ways to use slice. This will be more efficient, though: def binomial(n): value = 1 for i in range(n+1): yield value value = value * (n-i) // (i+1) From rustompmody at gmail.com Sat Jul 6 02:47:33 2013 From: rustompmody at gmail.com (Rustom Mody) Date: Sat, 6 Jul 2013 12:17:33 +0530 Subject: semi-programmer-ish envs/apps using python Message-ID: Some systems use python as a glue to make environments which are used by professionals that are not primarily programmers. Some egs: Scientific prog with Scipy+matplotlib Data analysis with pandas Visual arts with processing (Is pyprocessing stable enough?) Linguistics with nltk Is there such a list somewhere? -------------- next part -------------- An HTML attachment was scrubbed... URL: From russ.pobox at gmail.com Sat Jul 6 08:37:38 2013 From: russ.pobox at gmail.com (Russel Walker) Date: Sat, 6 Jul 2013 05:37:38 -0700 (PDT) Subject: Simple recursive sum function | what's the cause of the weird behaviour? Message-ID: I know this is simple but I've been starring at it for half an hour and trying all sorts of things in the interpreter but I just can't see where it's wrong. def supersum(sequence, start=0): result = start for item in sequence: try: result += supersum(item, start) except: result += item return result It's supposed to work like the builtin sum, but on multidimensional lists and also with the optional start parameter accepting something like an empty list and so would also works as a robust list flattener. It's just for kicks, I'm not actually going to use it for anything. This works: - - - - - - >>> x = [[1], [2], [3]] >>> supersum(x) 6 >>> supersum(x, []) [1, 2, 3] >>> This does not: - - - - - - - >>> x = [[[1], [2]], [3]] >>> supersum(x, []) [1, 2, 1, 2, 3] >>> From russ.pobox at gmail.com Sat Jul 6 08:54:56 2013 From: russ.pobox at gmail.com (Russel Walker) Date: Sat, 6 Jul 2013 05:54:56 -0700 (PDT) Subject: Simple recursive sum function | what's the cause of the weird behaviour? In-Reply-To: References: Message-ID: Nevermind! Stupid of me to forget that lists or mutable so result and start both point to the same list. From russ.pobox at gmail.com Sat Jul 6 08:59:06 2013 From: russ.pobox at gmail.com (Russel Walker) Date: Sat, 6 Jul 2013 05:59:06 -0700 (PDT) Subject: Simple recursive sum function | what's the cause of the weird behaviour? In-Reply-To: References: Message-ID: Since I've already wasted a thread I might as well... Does this serve as an acceptable solution? def supersum(sequence, start=0): result = type(start)() for item in sequence: try: result += supersum(item, start) except: result += item return result From __peter__ at web.de Sat Jul 6 09:19:30 2013 From: __peter__ at web.de (Peter Otten) Date: Sat, 06 Jul 2013 15:19:30 +0200 Subject: Simple recursive sum function | what's the cause of the weird behaviour? References: Message-ID: Russel Walker wrote: > Since I've already wasted a thread I might as well... > > Does this serve as an acceptable solution? > > def supersum(sequence, start=0): > result = type(start)() > for item in sequence: > try: > result += supersum(item, start) > except: > result += item > return result That depends on what is an acceptable result ;) For instance: >>> supersum([2, 3], 1) 5 >>> supersum([[1], ["abc"]], []) [1, 'a', 'b', 'c'] From joshua.landau.ws at gmail.com Sat Jul 6 14:43:42 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Sat, 6 Jul 2013 19:43:42 +0100 Subject: Simple recursive sum function | what's the cause of the weird behaviour? In-Reply-To: References: Message-ID: On 6 July 2013 13:59, Russel Walker wrote: > Since I've already wasted a thread I might as well... > > Does this serve as an acceptable solution? > > def supersum(sequence, start=0): > result = type(start)() > for item in sequence: > try: > result += supersum(item, start) > except: > result += item > return result It's probably more robust to do: def supersum(sequence, start=0): for item in sequence: try: result = result + supersum(item, start) except: result = result + item return result as that way you aren't assuming the signature of type(start). From sg552 at hotmail.co.uk Sat Jul 6 16:10:48 2013 From: sg552 at hotmail.co.uk (Rotwang) Date: Sat, 06 Jul 2013 21:10:48 +0100 Subject: Simple recursive sum function | what's the cause of the weird behaviour? In-Reply-To: References: Message-ID: On 06/07/2013 19:43, Joshua Landau wrote: > On 6 July 2013 13:59, Russel Walker wrote: >> Since I've already wasted a thread I might as well... >> >> Does this serve as an acceptable solution? >> >> def supersum(sequence, start=0): >> result = type(start)() >> for item in sequence: >> try: >> result += supersum(item, start) >> except: >> result += item >> return result > > It's probably more robust to do: > > def supersum(sequence, start=0): > for item in sequence: > try: > result = result + supersum(item, start) > except: > result = result + item > return result I assume you meant to put "result = start" in there at the beginning. > as that way you aren't assuming the signature of type(start). It's not quite clear to me what the OP's intentions are in the general case, but calling supersum(item, start) seems odd - for example, is the following desirable? >>> supersum([[1], [2], [3]], 4) 22 I would have thought that the "correct" answer would be 10. How about the following? def supersum(sequence, start = 0): result = start for item in reversed(sequence): try: result = supersum(item, result) except: result = item + result return result From sg552 at hotmail.co.uk Sat Jul 6 16:25:51 2013 From: sg552 at hotmail.co.uk (Rotwang) Date: Sat, 06 Jul 2013 21:25:51 +0100 Subject: Simple recursive sum function | what's the cause of the weird behaviour? In-Reply-To: References: Message-ID: On 06/07/2013 21:10, Rotwang wrote: > [...] > > It's not quite clear to me what the OP's intentions are in the general > case, but calling supersum(item, start) seems odd - for example, is the > following desirable? > > >>> supersum([[1], [2], [3]], 4) > 22 > > I would have thought that the "correct" answer would be 10. How about > the following? > > def supersum(sequence, start = 0): > result = start > for item in reversed(sequence): > try: > result = supersum(item, result) > except: > result = item + result > return result Sorry, I've no idea what I was thinking with that reversed thing. The following seems better: def supersum(sequence, start = 0): result = start for item in sequence: try: result = supersum(item, result) except: result = result + item return result From rosuav at gmail.com Sat Jul 6 13:22:40 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 7 Jul 2013 03:22:40 +1000 Subject: Simple recursive sum function | what's the cause of the weird behaviour? In-Reply-To: References: Message-ID: On Sat, Jul 6, 2013 at 10:37 PM, Russel Walker wrote: > This works: > - - - - - - >>>> x = [[1], [2], [3]] >>>> supersum(x) > 6 >>>> supersum(x, []) > [1, 2, 3] >>>> > > > This does not: > - - - - - - - >>>> x = [[[1], [2]], [3]] >>>> supersum(x, []) > [1, 2, 1, 2, 3] >>>> You have a problem of specification here. What should supersum do with the list [1]? Should it recurse into it, or append it as a list? It can't do both. For a list flattener, you would need to either use .append for each element you come across, or .extend with each list, with some kind of check to find whether you should recurse or not. Still, it's a fun thing to play with. I like code golfing these sorts of trinketty functions, just for fun :) ChrisA From tjreedy at udel.edu Sat Jul 6 14:47:27 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 06 Jul 2013 14:47:27 -0400 Subject: Simple recursive sum function | what's the cause of the weird behaviour? In-Reply-To: References: Message-ID: On 7/6/2013 8:37 AM, Russel Walker wrote: > I know this is simple but I've been starring at it for half an hour and trying all sorts of things in the interpreter but I just can't see where it's wrong. > > def supersum(sequence, start=0): > result = start > for item in sequence: > try: > result += supersum(item, start) > except: Bare except statements cover up too many sins. I and others *strongly* recommend that you only catch what you *know* you actually want to (see below). > result += item > return result I recommend that you start with at least one test case, and with an edge case at that. If you cannot bring yourself to do it before writing a draft of the function code, do it immediately after and run. If you do not want to use a framework, use assert. assert supersum([]) == 0 assert supersum([], []) == [] Do the asserts match your intention? The tests amount to a specification by example. Any 'kind' of input that is not tested is not guaranteed to work. Back to the except clause: only add try..except xxx when needed to pass a test. -- Terry Jan Reedy From russ.pobox at gmail.com Sun Jul 7 12:13:19 2013 From: russ.pobox at gmail.com (Russel Walker) Date: Sun, 7 Jul 2013 09:13:19 -0700 (PDT) Subject: Simple recursive sum function | what's the cause of the weird behaviour? In-Reply-To: References: Message-ID: I read through all of the posts and thanks for helping. What was supposed to be simple a (recursively) straightforward, turned out to be quite tricky. I've set up a small testing bench and tried all of the proposed solutions including my own but none pass. I'll post it below. I've also discovered something about lists that explains the very first "weird" result I was observing, which I realized was because lists are mutable etc, but more specifically: This >>> a = [1, 2] >>> a += [3] is equivalent to, AFAIK, this >>> a = [1, 2] >>> a.extend([3]) So to overcome that you just have to do >>> a = [1, 2] >>> a = a + [3] Which creates a new list. So any variables which were pointing to the same list as a, are unaffected. Summary - - - - >>> # --- Bad --- >>> a = [1, 2] >>> b = a >>> a += [3] >>> print a [1, 2, 3] >>> print b [1, 2, 3] >>> # --- Good --- >>> a = [1, 2] >>> b = a >>> a = a + [3] >>> print a [1, 2, 3] >>> print b [1, 2] And as for the testbench: def supersum(seq, start=0): return # ---------------- Testing -------------------------------- > testcases = [ # (seq, start, result) # arithmetic sums ([], 0, 0), ([[], []], 0, 0), ([[], [[],[]]], 0, 0), ([1], 0, 1), ([[], [1]], 0, 1), ([[], [[],[1, 1]]], 0, 2), ([[1], [1]], 0, 2), ([[1], [[1],[1, 1]]], 0, 4), ([[1], [[1],[1, 1]]], 1, 5), # list flattening ([], [], []), ([[], []], [], []), ([[], [[],[]]], [], []), ([], [1], [1]), ([[], []], [1], [1]), ([[], [[],[]]], [1], [1]), ([1], [1], [1, 1]), ([[1], [1]], [1], [1, 1]), ([[1], [[1],[1]]], [1], [1, 1, 1, 1]), ] for seq, start, result in testcases: try: assert supersum(seq, start) == result except Exception as er: print "seq:%s\t start:%s" % (seq, start) if type(er) is AssertionError: print "expected:", result print "got: ", supersum(seq, start) else: print repr(er) print '' From russ.pobox at gmail.com Sun Jul 7 12:44:42 2013 From: russ.pobox at gmail.com (Russel Walker) Date: Sun, 7 Jul 2013 09:44:42 -0700 (PDT) Subject: Simple recursive sum function | what's the cause of the weird behaviour? In-Reply-To: References: Message-ID: I got it! One of the testcases was wrong, ([[1], [1]], [1], [1, 1]), should be ([[1], [1]], [1], [1, 1, 1]), And the working solution. def supersum(sequence, start=0): result = start start = type(start)() for item in sequence: try: item = supersum(item, start) except TypeError: pass try: result = result + item except TypeError: return result + sequence return result I couldn't yet get around doing type(start)() and it's pretty messy, but anyways... From skip at pobox.com Sat Jul 6 10:10:39 2013 From: skip at pobox.com (Skip Montanaro) Date: Sat, 6 Jul 2013 09:10:39 -0500 Subject: Editor Ergonomics [was: Important features for editors] Message-ID: > The fact that rms has crippling RSI should indicate that > emacs' ergonomics is not right. Kind of a small sample size, don't you think? Hopefully we can kill this meme that Emacs is somehow worse for your wrists than other text editors before it goes any further than your one unsupported assertion. I've been using one form or another of Emacs as a programmer/software engineer for over 30 years. While I have had problems with RSI from time-to-time, there have generally been other factors at play other than just use of a text editor. I have learned how to manage it and recognize the warning signs that indicate the onset of an episode. More likely, rms ignored the problem and had bad personal ergomonics: ignorance or lack of understanding of the problem, poor posture, wrists not in a neutral position, lack of breaks, etc. If you stop to think about it, all text editors probably present similar issues for their users. They all involve: * a lot of typing, * use of modifier keys (ctrl, alt, command, etc) * movement between the mouse and the keyboard Skip From rosuav at gmail.com Sat Jul 6 10:23:12 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 7 Jul 2013 00:23:12 +1000 Subject: Editor Ergonomics [was: Important features for editors] In-Reply-To: References: Message-ID: On Sun, Jul 7, 2013 at 12:10 AM, Skip Montanaro wrote: > If you stop to > think about it, all text editors probably present similar issues for > their users. They all involve: > > * a lot of typing, > * use of modifier keys (ctrl, alt, command, etc) > * movement between the mouse and the keyboard Well, not all involve the mouse. Editors designed to be used over SSH often don't. But yes, editing text files will tend to involve a lot of typing. That's kinda the point of it :) ChrisA From rustompmody at gmail.com Sat Jul 6 11:04:00 2013 From: rustompmody at gmail.com (rusi) Date: Sat, 6 Jul 2013 08:04:00 -0700 (PDT) Subject: Editor Ergonomics [was: Important features for editors] In-Reply-To: References: Message-ID: <7e41e80c-8a83-4b4c-8839-93705c6b3f7a@googlegroups.com> On Saturday, July 6, 2013 7:40:39 PM UTC+5:30, Skip Montanaro wrote: > > The fact that rms has crippling RSI should indicate that > > emacs' ergonomics is not right. > > Kind of a small sample size, don't you think? Hopefully we can kill > this meme that Emacs is somehow worse for your wrists than other text > editors before it goes any further than your one unsupported > assertion. Not beyond small sample but beyond singleton: http://ergoemacs.org/emacs/emacs_hand_pain_celebrity.html > * a lot of typing, > * use of modifier keys (ctrl, alt, command, etc) > * movement between the mouse and the keyboard My own experience: The second 2 are the worse culprits. And while emacs is bad on the second, its excellent on the third -- to the extend that you 'live inside emacs,' you dont need the mouse. From ian.g.kelly at gmail.com Sat Jul 6 11:39:19 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Sat, 6 Jul 2013 09:39:19 -0600 Subject: Editor Ergonomics [was: Important features for editors] In-Reply-To: <7e41e80c-8a83-4b4c-8839-93705c6b3f7a@googlegroups.com> References: <7e41e80c-8a83-4b4c-8839-93705c6b3f7a@googlegroups.com> Message-ID: On Sat, Jul 6, 2013 at 9:04 AM, rusi wrote: > On Saturday, July 6, 2013 7:40:39 PM UTC+5:30, Skip Montanaro wrote: >> > The fact that rms has crippling RSI should indicate that >> > emacs' ergonomics is not right. >> >> Kind of a small sample size, don't you think? Hopefully we can kill >> this meme that Emacs is somehow worse for your wrists than other text >> editors before it goes any further than your one unsupported >> assertion. > > Not beyond small sample but beyond singleton: > http://ergoemacs.org/emacs/emacs_hand_pain_celebrity.html Funny that although that list claims a focus on emacs, the fourth person on the list is a vi user. The upshot here is that people who use keyboards a lot often suffer from RSI. You can't just link it to a particular editor on the basis of a few anecdotes. From rhodri at wildebst.demon.co.uk Mon Jul 8 18:44:53 2013 From: rhodri at wildebst.demon.co.uk (Rhodri James) Date: Mon, 08 Jul 2013 23:44:53 +0100 Subject: Editor Ergonomics [was: Important features for editors] References: <7e41e80c-8a83-4b4c-8839-93705c6b3f7a@googlegroups.com> Message-ID: On Sat, 06 Jul 2013 16:04:00 +0100, rusi wrote: > On Saturday, July 6, 2013 7:40:39 PM UTC+5:30, Skip Montanaro wrote: >> * a lot of typing, >> * use of modifier keys (ctrl, alt, command, etc) >> * movement between the mouse and the keyboard > > My own experience: The second 2 are the worse culprits. > And while emacs is bad on the second, its excellent on the third -- to > the extend that you 'live inside emacs,' you dont need the mouse. You clearly never trained as a classical pianist :-) -- Rhodri James *-* Wildebeest Herder to the Masses From steve+comp.lang.python at pearwood.info Sat Jul 6 22:41:02 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 07 Jul 2013 02:41:02 GMT Subject: Editor Ergonomics [was: Important features for editors] References: Message-ID: <51d8d53e$0$9505$c3e8da3$5496439d@news.astraweb.com> On Sat, 06 Jul 2013 09:10:39 -0500, Skip Montanaro wrote: >> The fact that rms has crippling RSI should indicate that emacs' >> ergonomics is not right. > > Kind of a small sample size, don't you think? Yes, but RMS is worth 1000 ordinary programmers!!! *wink* [...] > More likely, rms ignored the problem and had bad personal ergomonics: > ignorance or lack of understanding of the problem, poor posture, wrists > not in a neutral position, lack of breaks, etc. If you stop to think > about it, all text editors probably present similar issues for their > users. They all involve: > > * a lot of typing, > * use of modifier keys (ctrl, alt, command, etc) > * movement between the mouse and the keyboard I am not an ergonomic expert, but I understand that moving from mouse to keyboard actually helps prevent RSI, because it slows down the rate of keystrokes and uses different muscle groups. -- Steven From jussij at zeusedit.com Mon Jul 8 01:34:46 2013 From: jussij at zeusedit.com (jussij at zeusedit.com) Date: Sun, 7 Jul 2013 22:34:46 -0700 (PDT) Subject: Editor Ergonomics [was: Important features for editors] In-Reply-To: <51d8d53e$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <51d8d53e$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: <619de8bb-11f4-46e2-a79e-4b6209beba5e@googlegroups.com> On Sunday, July 7, 2013 12:41:02 PM UTC+10, Steven D'Aprano wrote: > I am not an ergonomic expert, but I understand that moving from mouse to > keyboard actually helps prevent RSI, because it slows down the rate of > keystrokes and uses different muscle groups. After 20+ years of coding using the Brief keyboard mapping I have so far I've gotten by with no perceivable RSI. On the half dozen occasions that I can recall experienced wrist pain, I remember the pain being worst when trying to interfacing with the mouse. So at least for me the mouse does not help. From steve+comp.lang.python at pearwood.info Mon Jul 8 03:39:09 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 08 Jul 2013 07:39:09 GMT Subject: Editor Ergonomics [was: Important features for editors] References: <51d8d53e$0$9505$c3e8da3$5496439d@news.astraweb.com> <619de8bb-11f4-46e2-a79e-4b6209beba5e@googlegroups.com> Message-ID: <51da6c9d$0$6512$c3e8da3$5496439d@news.astraweb.com> On Sun, 07 Jul 2013 22:34:46 -0700, jussij wrote: > On Sunday, July 7, 2013 12:41:02 PM UTC+10, Steven D'Aprano wrote: > >> I am not an ergonomic expert, but I understand that moving from mouse >> to keyboard actually helps prevent RSI, because it slows down the rate >> of keystrokes and uses different muscle groups. > > After 20+ years of coding using the Brief keyboard mapping I have so far > I've gotten by with no perceivable RSI. > > On the half dozen occasions that I can recall experienced wrist pain, I > remember the pain being worst when trying to interfacing with the mouse. > > So at least for me the mouse does not help. Chances are that you had one of those ridiculously high mice that force your wrist to bend upwards ("dorsiflexion"), or otherwise were bending the wrist inappropriately. When mousing, your wrist should be as close to straight as possible. It helps if your mouse has a low profile, so that you can rest your wrist and forearm on the desk and control your mouse with your fingers without bending the wrist. Assuming that you're keeping your wrists straight, then shifting from typing to mousing should reduce the chances of RSI because you are using different muscle groups. But if your mouse position is worse than your typing position, yes, that will probably cause problems... -- Steven From jsf80238 at gmail.com Tue Jul 9 00:27:14 2013 From: jsf80238 at gmail.com (Jason Friedman) Date: Mon, 8 Jul 2013 22:27:14 -0600 Subject: Editor Ergonomics [was: Important features for editors] In-Reply-To: <619de8bb-11f4-46e2-a79e-4b6209beba5e@googlegroups.com> References: <51d8d53e$0$9505$c3e8da3$5496439d@news.astraweb.com> <619de8bb-11f4-46e2-a79e-4b6209beba5e@googlegroups.com> Message-ID: I am right-handed and use a lefty-mouse about 50% of the time. It was difficult at first, now I'm almost as fast lefty as righty. As has been stated by others, changing the muscles being used reduces the impact on any one of them. -------------- next part -------------- An HTML attachment was scrubbed... URL: From neilc at norwich.edu Tue Jul 9 08:12:58 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 9 Jul 2013 12:12:58 GMT Subject: Editor Ergonomics [was: Important features for editors] References: <51d8d53e$0$9505$c3e8da3$5496439d@news.astraweb.com> <619de8bb-11f4-46e2-a79e-4b6209beba5e@googlegroups.com> Message-ID: On 2013-07-09, Jason Friedman wrote: > I am right-handed and use a lefty-mouse about 50% of the time. > It was difficult at first, now I'm almost as fast lefty as > righty. As has been stated by others, changing the muscles > being used reduces the impact on any one of them. That's the system I've adopted. I use the mouse lefty all day when working and righty all night when playing. -- Neil Cerutti From tjreedy at udel.edu Tue Jul 9 17:04:33 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 09 Jul 2013 17:04:33 -0400 Subject: Editor Ergonomics [was: Important features for editors] In-Reply-To: References: <51d8d53e$0$9505$c3e8da3$5496439d@news.astraweb.com> <619de8bb-11f4-46e2-a79e-4b6209beba5e@googlegroups.com> Message-ID: On 7/9/2013 8:12 AM, Neil Cerutti wrote: > On 2013-07-09, Jason Friedman wrote: >> I am right-handed and use a lefty-mouse about 50% of the time. >> It was difficult at first, now I'm almost as fast lefty as >> righty. As has been stated by others, changing the muscles >> being used reduces the impact on any one of them. > > That's the system I've adopted. I use the mouse lefty all day > when working and righty all night when playing. Me too, more or less. -- Terry Jan Reedy From giorgos.tzampanakis at gmail.com Mon Jul 8 02:54:30 2013 From: giorgos.tzampanakis at gmail.com (Giorgos Tzampanakis) Date: Mon, 8 Jul 2013 06:54:30 +0000 (UTC) Subject: Editor Ergonomics [was: Important features for editors] References: Message-ID: On 2013-07-06, Skip Montanaro wrote: > More likely, rms ignored the problem and had bad personal ergomonics: > ignorance or lack of understanding of the problem, poor posture, > wrists not in a neutral position, lack of breaks, etc. If you stop to > think about it, all text editors probably present similar issues for > their users. They all involve: > > * a lot of typing, If you do touch-typing then this shouldn't cause any problems since the finger/hand movement required is minimal. > * use of modifier keys (ctrl, alt, command, etc) Alt is not a problem, you hit them with your thumbs. Ctrl you should use the bottom-outer part of your palm. If you actually hit ctrl with your little finger then yes, this is a disaster waiting to happen. > * movement between the mouse and the keyboard Avoid at all costs. Use an editor that never needs the mouse (emacs or vim). -- Real (i.e. statistical) tennis and snooker player rankings and ratings: http://www.statsfair.com/ From xfq.free at gmail.com Wed Jul 10 00:17:12 2013 From: xfq.free at gmail.com (Xue Fuqiao) Date: Wed, 10 Jul 2013 12:17:12 +0800 Subject: Editor Ergonomics [was: Important features for editors] In-Reply-To: References: Message-ID: On Mon, Jul 8, 2013 at 2:54 PM, Giorgos Tzampanakis wrote: > On 2013-07-06, Skip Montanaro wrote: >> * movement between the mouse and the keyboard > > Avoid at all costs. Use an editor that never needs the mouse (emacs or > vim). I don't use vim often, but for Emacs, I think mouse is often needed: * It is especially handy for selecting and deleting text. * Mouse wheel gives you fine control. You can scroll by a few lines, but scroll-{up, down}-command can't. For programmers working on source code, this is especially nice. * There is mouse3.el[fn:1], which is very convenient. Footnotes: [fn:1] http://www.emacswiki.org/emacs/mouse3.el -- Best regards, Xue Fuqiao. http://www.gnu.org/software/emacs/ From jussij at zeusedit.com Thu Jul 11 00:15:27 2013 From: jussij at zeusedit.com (jussij at zeusedit.com) Date: Wed, 10 Jul 2013 21:15:27 -0700 (PDT) Subject: Editor Ergonomics [was: Important features for editors] In-Reply-To: References: Message-ID: <2fdf282e-fd28-4ba3-8c83-aaaace1201ec@googlegroups.com> On Wednesday, July 10, 2013 2:17:12 PM UTC+10, Xue Fuqiao wrote: > * It is especially handy for selecting and deleting text. When coding I never use a mouse to select text regions or to delete text. These operations I do using just the keyboard. From roy at panix.com Thu Jul 11 09:45:33 2013 From: roy at panix.com (Roy Smith) Date: Thu, 11 Jul 2013 09:45:33 -0400 Subject: Editor Ergonomics [was: Important features for editors] References: <2fdf282e-fd28-4ba3-8c83-aaaace1201ec@googlegroups.com> Message-ID: In article <2fdf282e-fd28-4ba3-8c83-aaaace1201ec at googlegroups.com>, jussij at zeusedit.com wrote: > On Wednesday, July 10, 2013 2:17:12 PM UTC+10, Xue Fuqiao wrote: > > > * It is especially handy for selecting and deleting text. > > When coding I never use a mouse to select text regions or to delete text. > > These operations I do using just the keyboard. For good typists, there is high overhead to getting your hands oriented on the keyboard (that's why the F and J keys have little bumps). So, any time you move your hand from the keyboard to the mouse, you pay a price. The worst thing is to constantly be alternating between mouse actions and keyboard actions. You spend all your time getting your fingers hands re-oriented. That's slow. This is why I never understood the attraction of something like xemacs, where you use the mouse to make text selections and run commands out of menus. It means you have to keep switching hand modes. I use emacs in non-window mode, which means my hands never leave the keyboard. From paul.nospam at rudin.co.uk Thu Jul 11 11:42:02 2013 From: paul.nospam at rudin.co.uk (Paul Rudin) Date: Thu, 11 Jul 2013 16:42:02 +0100 Subject: Editor Ergonomics [was: Important features for editors] References: <2fdf282e-fd28-4ba3-8c83-aaaace1201ec@googlegroups.com> Message-ID: <87d2qpw1bp.fsf@no-fixed-abode.cable.virginmedia.net> Roy Smith writes: > This is why I never understood the attraction of something like > xemacs, where you use the mouse to make text selections and run > commands out of menus. Menus are good for learning the functionality, and you have them just as much in Gnu emacs as in xemacs. You can even use them absent a windowing system! Text selection with a mouse is a different thing. Sometimes it's more convenient, sometimes it's not. But I agree with your general point - it's often quicker to keep your hands in position, which is where knowing keybinds for everything scores. From rosuav at gmail.com Thu Jul 11 11:50:17 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 12 Jul 2013 01:50:17 +1000 Subject: Editor Ergonomics [was: Important features for editors] In-Reply-To: <87d2qpw1bp.fsf@no-fixed-abode.cable.virginmedia.net> References: <2fdf282e-fd28-4ba3-8c83-aaaace1201ec@googlegroups.com> <87d2qpw1bp.fsf@no-fixed-abode.cable.virginmedia.net> Message-ID: On Fri, Jul 12, 2013 at 1:42 AM, Paul Rudin wrote: > Text selection with a mouse is a different thing. Sometimes it's > more convenient, sometimes it's not. As screens get larger and the amount of text on them increases, it's likely to get more and more useful to use a mouse... but personally, I still find the switch-to-mouse operation so clunky on anything except a Thinkpad that I'll use the keyboard even if it's a bit slower. (Why are Thinkpads different? Because the mouse, which is excellent, is right under my fingers, between G/H/B. It's less hassle to move to the mouse than, say, to hit Backspace, which I do quite a bit while typing. Effectively, I can touch-type the mouse. A HUGE advantage.) ChrisA From steve+comp.lang.python at pearwood.info Fri Jul 12 00:24:26 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 12 Jul 2013 04:24:26 GMT Subject: Editor Ergonomics [was: Important features for editors] References: <2fdf282e-fd28-4ba3-8c83-aaaace1201ec@googlegroups.com> <87d2qpw1bp.fsf@no-fixed-abode.cable.virginmedia.net> Message-ID: <51df84f9$0$9505$c3e8da3$5496439d@news.astraweb.com> On Fri, 12 Jul 2013 01:50:17 +1000, Chris Angelico wrote: > On Fri, Jul 12, 2013 at 1:42 AM, Paul Rudin > wrote: >> Text selection with a mouse is a different thing. Sometimes it's more >> convenient, sometimes it's not. > > As screens get larger and the amount of text on them increases, it's > likely to get more and more useful to use a mouse... but personally, I > still find the switch-to-mouse operation so clunky on anything except a > Thinkpad that I'll use the keyboard even if it's a bit slower. (Why are > Thinkpads different? Because the mouse, which is excellent, is right > under my fingers, between G/H/B. You mean those horrible nipple things? Ewwww. Ewwww. Damn things get the tactile feedback *completely* wrong. They're even worse than those awful little mousepads that you stroke. Frankly, nothing comes even close to a real mouse for feedback and ease of use. Maybe a stylus. But that's it. -- Steven From rosuav at gmail.com Fri Jul 12 00:34:05 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 12 Jul 2013 14:34:05 +1000 Subject: Editor Ergonomics [was: Important features for editors] In-Reply-To: <51df84f9$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <2fdf282e-fd28-4ba3-8c83-aaaace1201ec@googlegroups.com> <87d2qpw1bp.fsf@no-fixed-abode.cable.virginmedia.net> <51df84f9$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Fri, Jul 12, 2013 at 2:24 PM, Steven D'Aprano wrote: > On Fri, 12 Jul 2013 01:50:17 +1000, Chris Angelico wrote: > >> On Fri, Jul 12, 2013 at 1:42 AM, Paul Rudin >> wrote: >>> Text selection with a mouse is a different thing. Sometimes it's more >>> convenient, sometimes it's not. >> >> As screens get larger and the amount of text on them increases, it's >> likely to get more and more useful to use a mouse... but personally, I >> still find the switch-to-mouse operation so clunky on anything except a >> Thinkpad that I'll use the keyboard even if it's a bit slower. (Why are >> Thinkpads different? Because the mouse, which is excellent, is right >> under my fingers, between G/H/B. > > You mean those horrible nipple things? > > Ewwww. Ewwww. Damn things get the tactile feedback *completely* wrong. > They're even worse than those awful little mousepads that you stroke. > > Frankly, nothing comes even close to a real mouse for feedback and ease > of use. Maybe a stylus. But that's it. I would guess that you used a Dell one. They're far FAR worse. We have Dell laptops at work and I use external mice with them all. The IBM trackpoint is far better. I can game with a trackpoint, but when I built a Windows 7 desktop box for Alice, I had to go buy a high precision mouse just to be able to game properly. ChrisA From esj at harvee.org Fri Jul 12 00:57:24 2013 From: esj at harvee.org (Eric S. Johansson) Date: Fri, 12 Jul 2013 00:57:24 -0400 Subject: Editor Ergonomics [was: Important features for editors] In-Reply-To: <51df84f9$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <2fdf282e-fd28-4ba3-8c83-aaaace1201ec@googlegroups.com> <87d2qpw1bp.fsf@no-fixed-abode.cable.virginmedia.net> <51df84f9$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Fri, 12 Jul 2013 00:24:26 -0400, Steven D'Aprano wrote: > Frankly, nothing comes even close to a real mouse for feedback and ease > of use. Maybe a stylus. But that's it. before tremors, I would agree with you. Stylus is amazingly good tool for user interaction in a GUI. After tremors, not so much. For example, when you sweep across the pad, you keep your mouse tip up, over and use feedback from your mouse pointer to tell you when to touch down. My tremors caused mouse clicks on tablet at about a 2 Hz rate. You can just imagine it, hearing me move the stylus across the pad going: Tap tap tap tap tap tap tap. Yeah, sucks to be me. A high-resolution mouse is similarly problematic because I can make a gross motion to another part of the screen. hitting a small target on a high-resolution screen is not easy in the least and usually takes longer than the gross positioning across the screen. What I would love is a mouse interface that uses a mouse motion for a small range but a force vector like Mr. eraser head for longer-range movement. Not exactly sure how the mouse would behave but that's a very rough idea what would work well with my hands. From esj at harvee.org Fri Jul 12 18:54:47 2013 From: esj at harvee.org (Eric S. Johansson) Date: Fri, 12 Jul 2013 18:54:47 -0400 Subject: Editor Ergonomics [was: Important features for editors] In-Reply-To: References: <2fdf282e-fd28-4ba3-8c83-aaaace1201ec@googlegroups.com> <87d2qpw1bp.fsf@no-fixed-abode.cable.virginmedia.net> <51df84f9$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Fri, 12 Jul 2013 18:34:30 -0400, Dennis Lee Bieber wrote: > > Sounds like you might have liked an accessory I had on my Amiga. > Basically a proportional joystick feeding an interface box which > converted > the position value into a sequence of mouse movements -- sounds very cool. Although after I wrote my little screed, I found myself fiddling with my smartphone and I realized I was subconsciously putting it through the motions if it was a positioning device. The gross motion is detected by the accelerometer's in the phone. The fine positioning by fingers on the screen. Just thinking out loud. From steve+comp.lang.python at pearwood.info Thu Jul 11 22:39:18 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 12 Jul 2013 02:39:18 GMT Subject: Editor Ergonomics [was: Important features for editors] References: <2fdf282e-fd28-4ba3-8c83-aaaace1201ec@googlegroups.com> Message-ID: <51df6c56$0$9505$c3e8da3$5496439d@news.astraweb.com> On Thu, 11 Jul 2013 09:45:33 -0400, Roy Smith wrote: > In article <2fdf282e-fd28-4ba3-8c83-aaaace1201ec at googlegroups.com>, > jussij at zeusedit.com wrote: > >> On Wednesday, July 10, 2013 2:17:12 PM UTC+10, Xue Fuqiao wrote: >> >> > * It is especially handy for selecting and deleting text. >> >> When coding I never use a mouse to select text regions or to delete >> text. >> >> These operations I do using just the keyboard. > > For good typists, there is high overhead to getting your hands oriented > on the keyboard (that's why the F and J keys have little bumps). So, > any time you move your hand from the keyboard to the mouse, you pay a > price. > > The worst thing is to constantly be alternating between mouse actions > and keyboard actions. You spend all your time getting your fingers > hands re-oriented. That's slow. Big deal. I am utterly unconvinced that raw typing speed is even close to a bottleneck when programming. Data entry and transcribing from (say) dictated text, yes. Coding, not unless you are a one-fingered hunt-and- peek typist. The bottleneck is not typing speed but thinking speed: thinking about program design and APIs, thinking about data structures and algorithms, debugging, etc. Programming is normally done in spurts of typing followed by longer periods of thinking, testing, debugging. Micro-optimizing for the fraction of a second it takes to re-orient your hand on the keyboard is silly -- even if it is an optimization, not a pessimization (and I remain unconvinced) -- it's utterly trivial. Who cares that you saved 0.1 of a second by not moving your hand off the keyboard when you then sit there for 20 minutes staring into space thinking about your program design? Keyboard commands for yanking an entire line beat the mouse, but for moving the cursor around to arbitrary positions, or copying arbitrary pieces of text, a mouse is much faster. But either way, who cares? Typing speed is only rarely the bottleneck, let alone micro-optimizations like these. The difference between a 60 wpm typist and a 90 wpm typist is normally that the 90 wpm typist can introduce bugs 50% faster :-) I'm joking, of course, but typing *accuracy* is far more important than typing speed. -- Steven From rosuav at gmail.com Thu Jul 11 22:50:36 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 12 Jul 2013 12:50:36 +1000 Subject: Editor Ergonomics [was: Important features for editors] In-Reply-To: <51df6c56$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <2fdf282e-fd28-4ba3-8c83-aaaace1201ec@googlegroups.com> <51df6c56$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Fri, Jul 12, 2013 at 12:39 PM, Steven D'Aprano wrote: > Big deal. I am utterly unconvinced that raw typing speed is even close to > a bottleneck when programming. Data entry and transcribing from (say) > dictated text, yes. Coding, not unless you are a one-fingered hunt-and- > peek typist. The bottleneck is not typing speed but thinking speed: > thinking about program design and APIs, thinking about data structures > and algorithms, debugging, etc. > > Programming is normally done in spurts of typing followed by longer > periods of thinking, testing, debugging. That's true, but it's still important to be able to type quickly. You spend a minute or two figuring what you need to be doing, then want to see the result as quickly as possible. The plan is in your brain; you need to transfer it into code, save it, compile it if you need to, deploy it to your test-box if you need to, trigger its execution, and see its output. That's a roughly linear process, so any time saved in any step is an overall saving, and the shorter the total time from brain to output, the more smoothly your debugging/tinkering will be. That's why I've spent time developing systems at work that reduce the times required. With a single keystroke (F7 in SciTE), I can save, compile (for the one or two components that actually get compiled), and deploy to test-box, and a quick SIGHUP via Upstart does the rest. I can try two or three iterations of something without "losing" what my brain's holding onto - more if it's a trivial edit. Poor typing speed, or replacing the F7 whack with a button click that demands a mouse, would damage that. ChrisA From giorgos.tzampanakis at gmail.com Sun Jul 14 13:12:37 2013 From: giorgos.tzampanakis at gmail.com (Giorgos Tzampanakis) Date: Sun, 14 Jul 2013 17:12:37 +0000 (UTC) Subject: Editor Ergonomics [was: Important features for editors] References: <2fdf282e-fd28-4ba3-8c83-aaaace1201ec@googlegroups.com> <51df6c56$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2013-07-12, Steven D'Aprano wrote: > On Thu, 11 Jul 2013 09:45:33 -0400, Roy Smith wrote: > >> In article <2fdf282e-fd28-4ba3-8c83-aaaace1201ec at googlegroups.com>, >> jussij at zeusedit.com wrote: >> >>> On Wednesday, July 10, 2013 2:17:12 PM UTC+10, Xue Fuqiao wrote: >>> >>> > * It is especially handy for selecting and deleting text. >>> >>> When coding I never use a mouse to select text regions or to delete >>> text. >>> >>> These operations I do using just the keyboard. >> >> For good typists, there is high overhead to getting your hands oriented >> on the keyboard (that's why the F and J keys have little bumps). So, >> any time you move your hand from the keyboard to the mouse, you pay a >> price. >> >> The worst thing is to constantly be alternating between mouse actions >> and keyboard actions. You spend all your time getting your fingers >> hands re-oriented. That's slow. > > Big deal. I am utterly unconvinced that raw typing speed is even close to > a bottleneck when programming. Data entry and transcribing from (say) > dictated text, yes. Coding, not unless you are a one-fingered hunt-and- > peek typist. The bottleneck is not typing speed but thinking speed: > thinking about program design and APIs, thinking about data structures > and algorithms, debugging, etc. Typing time is definitely a small portion of coding time. However, since I learned touch typing I have found that I can work more hours without getting tired. It used to be that the repetitive up-down motion of the head was quickly leading to headaches and general tiredness. -- Real (i.e. statistical) tennis and snooker player rankings and ratings: http://www.statsfair.com/ From terry433iid at googlemail.com Sat Jul 6 10:58:58 2013 From: terry433iid at googlemail.com (terry433iid at googlemail.com) Date: Sat, 6 Jul 2013 07:58:58 -0700 (PDT) Subject: need data structure to for test results analysis Message-ID: <9ee9bed7-591a-4a3c-8cf6-9b1fc3b51cad@googlegroups.com> I have a python program that reads test result information from SQL and creates the following data that I want to capture in a data structure so it can be prioritized appropriately :- test_name new fail P1 test_name known fail (but no bug logged) P2 test_name known fail (bug logged but is closed) P3 test_name known fail (bug is open) P4 If I run my script I will get one of these types of failures - PLUS the number of occurrences of each type, sample data as follows:- P1 new fail | occurrence once (obviously) P1 new fail | occurrence once (obviously) P1 new fail | occurrence once (obviously) P1 new fail | occurrence once (obviously) P2 known fail | occurred previously 10 times in earlier executions P2 known fail | occurred previously 15 times in earlier executions P2 known fail | occurred previously 16 times in earlier executions P2 known fail | occurred previously 5 times in earlier executions P3 known fail | occurred previously 6 times in earlier executions P4 known fail | occurred previously 1 times in earlier executions P4 known fail | occurred previously 12 times in earlier executions . . . etc I want to be store this in an appropriate structure so I can then so some analysis :- if (all reported fails are "new fail"): this is priority 1 if (some fails are "new fail" and some are known (P2/P3/P4): this is priority 2 if (no new fail, but all/some reported fails are "P2 known fail") this is priority 3 if ( all/some reported fails are "P3 known fail") this is priority 4 if ( all/some reported fails are "P4 known fail") this is priority 4 I have tried using dictionary/lists but can't get the exact final outcome I want, any help appreciated.... From joshua.landau.ws at gmail.com Sat Jul 6 21:50:58 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Sun, 7 Jul 2013 02:50:58 +0100 Subject: need data structure to for test results analysis In-Reply-To: <9ee9bed7-591a-4a3c-8cf6-9b1fc3b51cad@googlegroups.com> References: <9ee9bed7-591a-4a3c-8cf6-9b1fc3b51cad@googlegroups.com> Message-ID: On 6 July 2013 15:58, wrote: > I have a python program that reads test result information from SQL and creates the following data that I want to capture in a data structure so it can be prioritized appropriately :- > > test_name new fail P1 > test_name known fail (but no bug logged) P2 > test_name known fail (bug logged but is closed) P3 > test_name known fail (bug is open) P4 > > > > > If I run my script I will get one of these types of failures - PLUS the number of occurrences of each type, sample data as follows:- > P1 new fail | occurrence once (obviously) > P1 new fail | occurrence once (obviously) > P1 new fail | occurrence once (obviously) > P1 new fail | occurrence once (obviously) > P2 known fail | occurred previously 10 times in earlier executions > P2 known fail | occurred previously 15 times in earlier executions > P2 known fail | occurred previously 16 times in earlier executions > P2 known fail | occurred previously 5 times in earlier executions > P3 known fail | occurred previously 6 times in earlier executions > P4 known fail | occurred previously 1 times in earlier executions > P4 known fail | occurred previously 12 times in earlier executions > . > . > . > etc I'm assuming you can put this into a list like: failures = [failure1, failure2, failure3, ...] A failure can be represented by a "namedtuple" or a class or some other thing. For simplicity, I'll use a class: class Failure: def __init__(self, name, type): self.name, self.type = name, type def __repr__(self): return "Failure({}, {})".format(self.name, self.type) > I want to be store this in an appropriate structure so I can then so some analysis :- > if (all reported fails are "new fail"): > this is priority 1 > if (some fails are "new fail" and some are known (P2/P3/P4): > this is priority 2 > if (no new fail, but all/some reported fails are "P2 known fail") > this is priority 3 > if ( all/some reported fails are "P3 known fail") > this is priority 4 > if ( all/some reported fails are "P4 known fail") > this is priority 4 > > I have tried using dictionary/lists but can't get the exact final outcome I want, any help appreciated.... You have your list of Failure()s, so you can do: if all(fail.type == "new fail" for fail in failures): set_priority_1() elif any(fail.type == "new fail" for fail in failures): set_priority_2() elif any(fail.type == "P2 known fail" for fail in failures): set_priority_3() elif any(fail.type == "P3 known fail" for fail in failures): set_priority_4() elif any(fail.type == "P4 known fail" for fail in failures): set_priority_4() else: freak_out() If you want something else, I'm not sure what you're asking. From bv8bv8bv8 at gmail.com Sat Jul 6 12:27:52 2013 From: bv8bv8bv8 at gmail.com (BV BV) Date: Sat, 6 Jul 2013 09:27:52 -0700 (PDT) Subject: WHAT DRIVES PEOPLE TO CONVERT TO ISLAM? Message-ID: <0bc9e71c-36d3-4508-8661-cd441b4ecc61@googlegroups.com> WHAT DRIVES PEOPLE TO CONVERT TO ISLAM? The various aspects of Islam which drives people to convert despite its negative portrayal in the media. The nature of religious faith is quite mysterious. As part of their religious faiths, people believe in a variety of deities. There are people who have religious faith in the unseen supreme inimitable power, and then there are others who believe in some humans as Gods, or animals (e.g. monkeys), fire, idols made of stone, and the list goes on. A lot is associated with having a religious ?faith?. Part of it has to do with beliefs passed on through generations. People?s identities therefore get tied to it. Many times, these beliefs and associated feelings are not completely demonstrable by reason or any rational arguments. There is nothing right or wrong with this, but that?s just how the nature of religious faith has come to be. Almost everyone thinks they are right in their faith and beliefs. Being with people and groups with similar faith further strengthens people?s faith, and they see it as right, even though logical reasoning and argument sometimes can?t explain it all. That?s simple human psychology. Islam?s arguments based on intellectual reasoning Muslims believe however, that the Islamic religion is different in this context. One may argue that similar to other faiths there are aspects of it which are not completely demonstrable by reason, but on the other hand the Quranic text, which is God?s words addressing humanity at large, uses intellectual reason, critical thinking, and the process of reflection as a means not only to reinforce the faith of the believers, but also to call non-believers to ponder about the authenticity of Islam as the way of life for humanity at large. Although no religious beliefs can be fully based on logic and reasoning, Islam and Quran provide more than enough examples and an opportunity to examine the truth and the soundness of its message through the lens of empirical evidence and knowledge. No one (Muslim or otherwise) would argue that critical thinking and reflection can be a major catalyst for changing ones life. Critical thinking has been used by many to improve their lives simply because a critical thinker asks probing questions about a situation, collects as much information as possible, reflects on the ideas collected and generated in context of the information available, keeps an open and unbiased mind, and carefully scrutinizes assumptions and seeks alternatives. This is the reason, therefore, that new Muslim converts would attribute the use of intelligent reasoning, reflection and critical thinking when explaining their journey to Islam. Such people cut through the hysteria created in the media to view Islam from a critical lens and following the truth thus comes naturally to them as part of this process. How else can one explain the increase in conversions with the increase of anti-Islamic rhetoric? How else can one explain that more non-Muslim preachers have been converting to Islam than ever before? Although, as Muslims, we believe that guidance comes only from Allah, the use of a person?s God-gifted intellectual reasoning has a very powerful role to play in Muslim converts making that destiny changing decision. And once converted, they rarely go back to their old faiths, simply because a faith whose foundations are built on logic and reason is much less likely to be shaken down than one which simply builds upon a set of rites and sacraments. Reasons attributed by new Converts Some of the reasons given why people convert to Islam are the eloquence of the Quran?s language, its overwhelming scientific evidence and proofs, arguments rooted in intellectual reasoning, and the Divine wisdom behind various social issues. The uniqueness and beauty of the Quran?s text has been marveled by the best of Arab linguists and scholars, both Muslim and otherwise, from the days it was revealed until today. The more knowledgeable people are in the language, the more they appreciate the wonders of the textual fluency of the Quran. Revealed more than 1400 years ago, the Quran also has numerous scientific facts that are being validated by science only in this era. Furthermore, it is the only known religious text that challenges mankind to think, reflect and ponder over the creation at large, social issues, God?s existence, and more. The Quran, in many instances, challenges people to reflect and think on their own, rather than heeding the loose talk of those whose criticism is based on baseless foundations. Finally, the Quran provides a solution to numerous social issues, deviation from which has been known to cause societal chaos at all levels. The Quran is a confident assertion of a Supreme Being; the only known religious book that has a confident assertion of a Supreme Being on all issues ranging from the creation of the universe to most particular components of the social milieu. Moreover, its Divine Text - the language and prose of the Quran - is very different from the language in the Prophet?s sayings, which demonstrates that the Quran is not from the creative imagination or inspired words of Prophet Muhammad, as many doubters have alleged in the past, and continue to do even today. We can see that most of these reasons can only be attributed to the process of critical thinking and intellectual reflection. However, cold reasoning is not enough. The heart has to be engaged in the search: a search whose aim is to reach for the truth at its core. No wonder, then, that when such sincere people hear the Quran for the first time, and understand it, they say: ?We believe in it; surely, it is the Truth from our Lord. Indeed, even before it, we were Muslims!? (Quran 28:53) The Quran challenges humanity at large to think, reflect and ponder over their affairs on numerous occasions. This is some of what the Quran states: ? Thus do We explain the verses (and their signification) in detail for the people who reflect. (Jonah, Quran 10:24) ? Do they not think deeply about their own selves (being)? Allah has created not the heavens and the Earth and all that is between them except with truth and for an appointed term. And indeed many of mankind deny the Meeting with their Lord. (The-Romans, Quran 30:8) ? He it is Who has appointed for you the night that you may rest therein, and the day to make things visible. Verily, in this are signs for a people who listen. (Jonah, Quran 10:67) ? Does man think that he will be left to no purpose? (The Resurrection, Quran 75:36) ? Did you think that We had created you in play, and that you would not be brought back to Us?? (The Believers, Quran 23:115) ? Or do you think that most of them hear or understand? They are only like cattle; nay, they are even farther astray from the Path. (The Criterion, Quran 25:44) ? Do they not reflect? There is no madness in their companion (Muhammad). He is but a plain Warner. (The Heights, Quran 7:184) ? Had We sent down this Quran on a mountain, you would surely have seen it humbling itself and rending asunder for the fear of God. Such are the parables which We put forward to mankind that they may reflect. (The Overcrowding, Quran 59:21) When studying the many cases of new Muslim converts, we see that engaging in critical thinking and intellectual reasoning have led people to change their non-Islamic faiths ? the same faiths that would earlier supposedly have moved mountains, but were diluted by the voice of reason easily heard in the roots of Islam. A mere process of thinking and reflection brings so much into light that otherwise would remain veiled by the distractions and forces of anti-Islam pundits. Those who are bent on seeing only the negative fail to see the light of truth. Rather, they engage in a never ending superficial analysis to unsuccessfully prove their misguided philosophies. There are many statistics in the media that highlight the phenomenal rate at which people are converting to Islam. Although, the authenticity of all these sources has not been validated for the purpose of this article, some of them include the following: ? According to ?The Almanac Book of Facts?, the population increased 137% within the past decade, Christianity increased 46%, while Islam increased 235%. ? 100,000 people per year in America alone, are converting to Islam. For every 1 male convert to Islam, 4 females convert to Islam ? TV Report: 4,000 Germans Convert To ISLAM Each Year ? About 25,000 people convert to Islam every year in the UK alone ? ?many more examples exist. What about Muslims? If voices of reason embedded in the teachings of Islam are causing non-Muslims to revert to Islam in droves, why is it that so many Muslims born into the religion usually fail to fully follow, and thus enjoy, the teachings of the religion? The fact is that it may just be the lack of critical thinking and reflection on the part of some Muslims that is forcing the Muslim world to have a substandard way of life as a whole. Islam and its teachings hold the promise of a fulfilling and peaceful life for all. Yet Muslims continue to ignore the basics and get mired in social and moral issues causing unnecessary pain and suffering on themselves and their families. The fact is that only if they would think and reflect on the teachings of their own religion, they could escape the many problems and challenges that face them. The Message To non-Muslims who have only scratched the surface of Islam and may be getting distracted by those who are the wrong torch bearers of this religion and by the biased voices in the media, the message is simple ? try to view the teachings of Islam with a critical lens. It may be that you will be able to see more reason than you may initially have thought was not present. To Muslims, the message is that sometimes we do not appreciate the teachings of our own religion simply because we never think and grow beyond the few religious practices in our operating lives. A focused effort to learn, think and reflect more will help us get closer to the religious teachings in ways that can drastically improve our lives. http://www.islamhouse.com/430052/en/en/articles/What_Drives_People_to_Convert_to_Islam? thank you From bubble at turtle.telenet.be Sat Jul 6 15:02:54 2013 From: bubble at turtle.telenet.be (bubble.rogue) Date: 06 Jul 2013 19:02:54 GMT Subject: pyc++3d, C++ 3D math module Message-ID: Hi pys, Here is a C++ 3D module based on actors, you can load it in lua, android JNI and of course python. It includes matrix and vector classes with rotate and translate code. See the README and HACKING files for more information. http://sourceforge.net/projects/pycplusplus3d Enjoy, Turtle Wizard -- Time heals. My blog : http://thediaryofelvishhealer.blogspot.com From tjreedy at udel.edu Sat Jul 6 15:38:01 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 06 Jul 2013 15:38:01 -0400 Subject: Explain your acronyms (RSI?) Message-ID: "rms has crippling RSI" (anonymous, as quoted by Skip). I suspect that 'rms' = Richard M Stallman (but why lower case? to insult him?). I 'know' that RSI = Roberts Space Industries, a game company whose Kickstarter project I supported. Whoops, wrong context. How about 'Richard Stallman Insanity' (his personal form of megalomania)? That makes the phrase is a claim I have read others making. Lets continue and see if that interpretation works. "should indicate that emacs' ergonomics is not right". Aha! Anonymous believes that using his own invention, emacs, is what drove Richard crazy. He would not be the first self invention victim. But Skip mentions 'worse for wrists'. So RSI must be a physical rather than mental condition. Does 'I' instead stand for Inoperability?, Instability?, or what? Let us try Google. Type in RSI and it offers 'RSI medications' as a choice. Sound good, as it will eliminate all the companies with those initials. The two standard medical meanings of RSI seem to be Rapid Sequence Intubation and Rapid Sequence Induction. But those are procedures, not chronic syndromes. So I still do not know what the original poster, as quoted by Skip, meant. -- Terry Jan Reedy From benjamin.kaplan at case.edu Sat Jul 6 15:48:19 2013 From: benjamin.kaplan at case.edu (Benjamin Kaplan) Date: Sat, 6 Jul 2013 12:48:19 -0700 Subject: Explain your acronyms (RSI?) In-Reply-To: References: Message-ID: On Sat, Jul 6, 2013 at 12:38 PM, Terry Reedy wrote: > "rms has crippling RSI" (anonymous, as quoted by Skip). > > I suspect that 'rms' = Richard M Stallman (but why lower case? to insult > him?). I 'know' that RSI = Roberts Space Industries, a game company whose > Kickstarter project I supported. Whoops, wrong context. How about 'Richard > Stallman Insanity' (his personal form of megalomania)? That makes the phrase > is a claim I have read others making. > > Lets continue and see if that interpretation works. "should indicate that > emacs' ergonomics is not right". Aha! Anonymous believes that using his own > invention, emacs, is what drove Richard crazy. He would not be the first > self invention victim. > > But Skip mentions 'worse for wrists'. So RSI must be a physical rather than > mental condition. Does 'I' instead stand for Inoperability?, Instability?, > or what? > > Let us try Google. Type in RSI and it offers 'RSI medications' as a choice. > Sound good, as it will eliminate all the companies with those initials. The > two standard medical meanings of RSI seem to be Rapid Sequence Intubation > and Rapid Sequence Induction. But those are procedures, not chronic > syndromes. So I still do not know what the original poster, as quoted by > Skip, meant. > > -- > Terry Jan Reedy RSI is a repetitive stress injury. From sg552 at hotmail.co.uk Sat Jul 6 15:51:03 2013 From: sg552 at hotmail.co.uk (Rotwang) Date: Sat, 06 Jul 2013 20:51:03 +0100 Subject: Explain your acronyms (RSI?) In-Reply-To: References: Message-ID: On 06/07/2013 20:38, Terry Reedy wrote: > "rms has crippling RSI" (anonymous, as quoted by Skip). > > I suspect that 'rms' = Richard M Stallman (but why lower case? to insult > him?). I 'know' that RSI = Roberts Space Industries, a game company > whose Kickstarter project I supported. Whoops, wrong context. How about > 'Richard Stallman Insanity' (his personal form of megalomania)? That > makes the phrase is a claim I have read others making. > > Lets continue and see if that interpretation works. "should indicate > that emacs' ergonomics is not right". Aha! Anonymous believes that using > his own invention, emacs, is what drove Richard crazy. He would not be > the first self invention victim. > > But Skip mentions 'worse for wrists'. So RSI must be a physical rather > than mental condition. Does 'I' instead stand for Inoperability?, > Instability?, or what? > > Let us try Google. Type in RSI and it offers 'RSI medications' as a > choice. Sound good, as it will eliminate all the companies with those > initials. The two standard medical meanings of RSI seem to be Rapid > Sequence Intubation and Rapid Sequence Induction. But those are > procedures, not chronic syndromes. So I still do not know what the > original poster, as quoted by Skip, meant. Repetitive strain injury, I assume. Not sure if you're joking but over here the top 7 hits for "RSI" on Google, as well as the three ads that precede them, are repetitive strain injury-related. From stefan_ml at behnel.de Sat Jul 6 16:11:42 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 06 Jul 2013 22:11:42 +0200 Subject: Explain your acronyms (RSI?) In-Reply-To: References: Message-ID: Rotwang, 06.07.2013 21:51: > On 06/07/2013 20:38, Terry Reedy wrote: >> "rms has crippling RSI" (anonymous, as quoted by Skip). >> [...] >> Let us try Google. Type in RSI and it offers 'RSI medications' as a >> choice. Sound good, as it will eliminate all the companies with those >> initials. The two standard medical meanings of RSI seem to be Rapid >> Sequence Intubation and Rapid Sequence Induction. But those are >> procedures, not chronic syndromes. So I still do not know what the >> original poster, as quoted by Skip, meant. > > Repetitive strain injury, I assume. Not sure if you're joking but over here > the top 7 hits for "RSI" on Google, as well as the three ads that precede > them, are repetitive strain injury-related. Both of you might want to delete your browser cookies, log out of your Google accounts, and then retry. Maybe disabling JavaScript helps. Or enabling the Privacy Mode in your browser. Or try a different browser all together. Or a different search engine. Google has lots of ways to detect who's asking. Stefan From sg552 at hotmail.co.uk Sat Jul 6 16:27:17 2013 From: sg552 at hotmail.co.uk (Rotwang) Date: Sat, 06 Jul 2013 21:27:17 +0100 Subject: Explain your acronyms (RSI?) In-Reply-To: References: Message-ID: On 06/07/2013 21:11, Stefan Behnel wrote: > Rotwang, 06.07.2013 21:51: >> On 06/07/2013 20:38, Terry Reedy wrote: >>> "rms has crippling RSI" (anonymous, as quoted by Skip). >>> [...] >>> Let us try Google. Type in RSI and it offers 'RSI medications' as a >>> choice. Sound good, as it will eliminate all the companies with those >>> initials. The two standard medical meanings of RSI seem to be Rapid >>> Sequence Intubation and Rapid Sequence Induction. But those are >>> procedures, not chronic syndromes. So I still do not know what the >>> original poster, as quoted by Skip, meant. >> >> Repetitive strain injury, I assume. Not sure if you're joking but over here >> the top 7 hits for "RSI" on Google, as well as the three ads that precede >> them, are repetitive strain injury-related. > > Both of you might want to delete your browser cookies, log out of your > Google accounts, and then retry. Maybe disabling JavaScript helps. Or > enabling the Privacy Mode in your browser. Or try a different browser all > together. Or a different search engine. Google has lots of ways to detect > who's asking. The results I mentioned above were in private browsing in FF. I'm in the UK though so that certainly will have made a difference. From robert.kern at gmail.com Sat Jul 6 16:05:59 2013 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 06 Jul 2013 21:05:59 +0100 Subject: Explain your acronyms (RSI?) In-Reply-To: References: Message-ID: On 2013-07-06 20:38, Terry Reedy wrote: > "rms has crippling RSI" (anonymous, as quoted by Skip). > > I suspect that 'rms' = Richard M Stallman (but why lower case? to insult him?). http://stallman.org/ """ "Richard Stallman" is just my mundane name; you can call me "rms". """ > But Skip mentions 'worse for wrists'. So RSI must be a physical rather than > mental condition. Does 'I' instead stand for Inoperability?, Instability?, or what? > > Let us try Google. Type in RSI and it offers 'RSI medications' as a choice. "RSI wrist" would probably have been a wiser start. Not all medical conditions have medication associated with them. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From steve+comp.lang.python at pearwood.info Sat Jul 6 23:56:26 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 07 Jul 2013 03:56:26 GMT Subject: Numeric coercions Message-ID: <51d8e6ea$0$9505$c3e8da3$5496439d@news.astraweb.com> I sometimes find myself needing to promote[1] arbitrary numbers (Decimals, Fractions, ints) to floats. E.g. I might say: numbers = [float(num) for num in numbers] or if you prefer: numbers = map(float, numbers) The problem with this is that if a string somehow gets into the original numbers, it will silently be converted to a float when I actually want a TypeError. So I want something like this: def promote(x): if isinstance(x, str): raise TypeError return float(x) but I don't like the idea of calling isinstance on every value. Is there a better way to do this? E.g. some operation which is guaranteed to promote any numeric type to float, but not strings? For the record, calling promote() as above is about 7 times slower than calling float in Python 3.3. [1] Or should that be demote? -- Steven From joshua.landau.ws at gmail.com Sun Jul 7 00:17:01 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Sun, 7 Jul 2013 05:17:01 +0100 Subject: Numeric coercions In-Reply-To: <51d8e6ea$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <51d8e6ea$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 7 July 2013 04:56, Steven D'Aprano wrote: > I sometimes find myself needing to promote[1] arbitrary numbers > (Decimals, Fractions, ints) to floats. E.g. I might say: > > numbers = [float(num) for num in numbers] > > or if you prefer: > > numbers = map(float, numbers) > > The problem with this is that if a string somehow gets into the original > numbers, it will silently be converted to a float when I actually want a > TypeError. So I want something like this: > > def promote(x): > if isinstance(x, str): raise TypeError > return float(x) > > but I don't like the idea of calling isinstance on every value. Is there > a better way to do this? E.g. some operation which is guaranteed to > promote any numeric type to float, but not strings? > > For the record, calling promote() as above is about 7 times slower than > calling float in Python 3.3. >>> from operator import methodcaller >>> safe_float = methodcaller("__float__") >>> safe_float(434) 434.0 >>> safe_float(434.4342) 434.4342 >>> safe_float(Decimal("1.23")) 1.23 >>> safe_float("123") Traceback (most recent call last): File "", line 1, in AttributeError: 'str' object has no attribute '__float__' >>> From steve+comp.lang.python at pearwood.info Sun Jul 7 00:48:20 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 07 Jul 2013 04:48:20 GMT Subject: Numeric coercions References: <51d8e6ea$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51d8f313$0$9505$c3e8da3$5496439d@news.astraweb.com> On Sun, 07 Jul 2013 05:17:01 +0100, Joshua Landau wrote: > On 7 July 2013 04:56, Steven D'Aprano > wrote: ... >> def promote(x): >> if isinstance(x, str): raise TypeError return float(x) >>>> from operator import methodcaller >>>> safe_float = methodcaller("__float__") Nice! That's almost as fast as calling float. That will probably do :-) -- Steven From joshua.landau.ws at gmail.com Sun Jul 7 01:14:19 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Sun, 7 Jul 2013 06:14:19 +0100 Subject: Numeric coercions In-Reply-To: <51d8f313$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <51d8e6ea$0$9505$c3e8da3$5496439d@news.astraweb.com> <51d8f313$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 7 July 2013 05:48, Steven D'Aprano wrote: > On Sun, 07 Jul 2013 05:17:01 +0100, Joshua Landau wrote: > >> On 7 July 2013 04:56, Steven D'Aprano >> wrote: > ... >>> def promote(x): >>> if isinstance(x, str): raise TypeError return float(x) > >>>>> from operator import methodcaller >>>>> safe_float = methodcaller("__float__") > > Nice! > > That's almost as fast as calling float. That will probably do :-) *Almost* as fast? %~> \python -m timeit -s "safe_float = __import__('operator').methodcaller('__float__'); n = 423.213" "float(n)" 1000000 loops, best of 3: 0.398 usec per loop %~> \python -m timeit -s "safe_float = __import__('operator').methodcaller('__float__'); n = 423.213" "safe_float(n)" 1000000 loops, best of 3: 0.361 usec per loop %~> \python -m timeit -s "safe_float = __import__('operator').methodcaller('__float__'); n = 423" "float(n)" 1000000 loops, best of 3: 0.468 usec per loop %~> \python -m timeit -s "safe_float = __import__('operator').methodcaller('__float__'); n = 423" "safe_float(n)" 1000000 loops, best of 3: 0.436 usec per loop Actually, it's a fair bit faster (yes, I am purposely not showing you the results for Decimals). From joshua.landau.ws at gmail.com Sun Jul 7 01:30:16 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Sun, 7 Jul 2013 06:30:16 +0100 Subject: Numeric coercions In-Reply-To: References: <51d8e6ea$0$9505$c3e8da3$5496439d@news.astraweb.com> <51d8f313$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 7 July 2013 06:14, Joshua Landau wrote: > On 7 July 2013 05:48, Steven D'Aprano > wrote: >> On Sun, 07 Jul 2013 05:17:01 +0100, Joshua Landau wrote: >> >>> On 7 July 2013 04:56, Steven D'Aprano >>> wrote: >> ... >>>> def promote(x): >>>> if isinstance(x, str): raise TypeError return float(x) >> >>>>>> from operator import methodcaller >>>>>> safe_float = methodcaller("__float__") >> >> Nice! >> >> That's almost as fast as calling float. That will probably do :-) > > *Almost* as fast? > *incorrect timings* > > Actually, it's a fair bit faster (yes, I am purposely not showing you > the results for Decimals). *sigh*, it seems that was just because it's slower to access __builtins__ than globals(). float() is maybe 2% faster if you fix that. From vlastimil.brom at gmail.com Sun Jul 7 04:15:27 2013 From: vlastimil.brom at gmail.com (Vlastimil Brom) Date: Sun, 7 Jul 2013 10:15:27 +0200 Subject: Numeric coercions In-Reply-To: <51d8e6ea$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <51d8e6ea$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: 2013/7/7 Steven D'Aprano : > I sometimes find myself needing to promote[1] arbitrary numbers > (Decimals, Fractions, ints) to floats. E.g. I might say: > > numbers = [float(num) for num in numbers] > > or if you prefer: > > numbers = map(float, numbers) > > The problem with this is that if a string somehow gets into the original > numbers, it will silently be converted to a float when I actually want a > TypeError. So I want something like this: > >... > -- > Steven > -- Hi, I guess, a naive approach like numbers = [float(num+0) for num in numbers] wouldn't satisfy the performance requirements, right? vbr From joshua.landau.ws at gmail.com Sun Jul 7 04:34:24 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Sun, 7 Jul 2013 09:34:24 +0100 Subject: Numeric coercions In-Reply-To: References: <51d8e6ea$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 7 July 2013 09:15, Vlastimil Brom wrote: > 2013/7/7 Steven D'Aprano : >> I sometimes find myself needing to promote[1] arbitrary numbers >> (Decimals, Fractions, ints) to floats. E.g. I might say: >> >> numbers = [float(num) for num in numbers] >> >> or if you prefer: >> >> numbers = map(float, numbers) >> >> The problem with this is that if a string somehow gets into the original >> numbers, it will silently be converted to a float when I actually want a >> TypeError. So I want something like this: >> >>... >> -- >> Steven >> -- > Hi, > I guess, a naive approach like > > numbers = [float(num+0) for num in numbers] > > wouldn't satisfy the performance requirements, right? %~> \python -m timeit -s "n = 123" "n" 10000000 loops, best of 3: 0.0467 usec per loop %~> \python -m timeit -s "n = 123" "float(n)" 1000000 loops, best of 3: 0.483 usec per loop %~> \python -m timeit -s "n = 123" "n+0" 10000000 loops, best of 3: 0.0991 usec per loop %~> \python -m timeit -s "n = 123" "float(n+0)" 1000000 loops, best of 3: 0.537 usec per loop # And with more local lookup %~> \python -m timeit -s "n = 123; ffloat=float" "ffloat(n)" 1000000 loops, best of 3: 0.424 usec per loop %~> \python -m timeit -s "n = 123; ffloat=float" "ffloat(n+0)" 1000000 loops, best of 3: 0.483 usec per loop "+0" takes only a small fraction of the time of "float" (about 25% at worst), so I don't think that's too big a deal. My version of safe_float much closer to that of float (only about 2% relative overhead) but I'd say it's also arguably more correct -- any value which is coercible to a float has a ".__float__" method but not all numeric types are addable to others: >>> Decimal("0") + 0.0 Traceback (most recent call last): File "", line 1, in TypeError: unsupported operand type(s) for +: 'decimal.Decimal' and 'float' It's conceivable that a numeric type could refuse to add to integers, although very unlikely. The only reason Decimal refuses to add to floats, AFAIK, is precision. From inq1ltd at inqvista.com Sun Jul 7 13:06:18 2013 From: inq1ltd at inqvista.com (inq1ltd) Date: Sun, 07 Jul 2013 13:06:18 -0400 Subject: capture html screen with pexpect Message-ID: <9118058.sBVIZWDW2o@mach-114-20> python help, I can log into a web site with pexpect but what I want to do is pipe the opening window to a file. Logging into the site opens the site window but I can't get the window to a file. I can't use screen capture I need to get pexpect to pipe it to a txt file. Any help will be appreciated. jimonliinux -------------- next part -------------- An HTML attachment was scrubbed... URL: From davea at davea.name Mon Jul 8 07:17:38 2013 From: davea at davea.name (Dave Angel) Date: Mon, 08 Jul 2013 07:17:38 -0400 Subject: capture html screen with pexpect In-Reply-To: <9118058.sBVIZWDW2o@mach-114-20> References: <9118058.sBVIZWDW2o@mach-114-20> Message-ID: On 07/07/2013 01:06 PM, inq1ltd wrote: > python help, > > I can log into a web site with pexpect but > what I want to do is pipe the opening window > to a file. > > Logging into the site opens the site window > but I can't get the window to a file. > > I can't use screen capture I need to get > pexpect to pipe it to a txt file. > What have you tried? What environment are you in? What's the exact assignment? I'd use wget, since I'm on Linux, and it's simple. davea at think2:~$ wget google.com creates a file called index.html in my cwd -- DaveA From TurtleWizard at turtle.telenet.be Sun Jul 7 17:11:46 2013 From: TurtleWizard at turtle.telenet.be (bubble.rogue) Date: 07 Jul 2013 21:11:46 GMT Subject: pygame pyZelda - the minish cap sequel - WIP Message-ID: Hi all, I am working on a new Zelda - the minish cap engine. It has a spritesheet transformer into surfaces which can be altered in a map in pyLevelMaker. The moving of the player works within the small tile engine. Here is the repository : http://github.com/zork9/games.git WIP, I will post it later on on zeldadungeon.net, under fan games. Enjoy, Turtle -- Time heals. http://thediaryofelvishhealer.blogspot.com From c.justin88 at gmail.com Sun Jul 7 17:57:55 2013 From: c.justin88 at gmail.com (Justin Chiu) Date: Sun, 07 Jul 2013 14:57:55 -0700 Subject: AMQP listening and user-facing daemon Message-ID: <51D9E463.7030403@gmail.com> Hi all, What is the best approach to writing a concurrent daemon that can execute callbacks for different types of events (AMQP messages, parsed output of a subprocess, HTTP requests)? I am considering [twisted][1], the built-in [threading][2] module, and [greenlet][3]. I must admit that I am very unfamiliar with concurrent programming and Python programming in general (formerly a data analysis driven procedural programmer). Any resources on threaded/concurrent programming (specifically daemons...not just multi-threading a single task) would be much appreciated. Thank you. Details: 1) Listens into AMQP messaging queues and executes callbacks when messages arrive. Example: Immediately after startup, the daemon continuously listens to the [Openstack Notifications messaging queue][4]. When a virtual machine is launched, a notification is generated by Openstack with the hostname, IP address, etc. The daemon should read this message and write some info to a log (or POST the info to a server, or notify the user...something simple). 2) Parse the output of a subprocess and execute callbacks based on the output. Example: Every 30 seconds, a system command "[qstat][5]" is run to query a job resource manager (e.g. TORQUE). Similar callbacks to 1). 3) Receive requests from a user and process them. I think this will be via WSGI HTTP. Example: User submits an XML template with virtual machine templates. The daemon does some simple XML parsing and writes a job script for the job resource manager. The job is submitted to the resource manager and the daemon continually checks for the status of the job with "qstat" and for messages from AMQP. It should return "live" feedback to the user and write to a log. [1]: https://twistedmatrix.com/trac/wiki/Documentation [2]: http://docs.python.org/2/library/threading.html [3]: http://greenlet.readthedocs.org/en/latest/ [4]: https://wiki.openstack.org/wiki/NotificationEventExamples#Immediate_Notifications: [5]: http://www.clusterresources.com/torquedocs21/commands/qstat.shtml Justin Chiu TRIUMF From dieter at handshake.de Tue Jul 9 03:05:19 2013 From: dieter at handshake.de (dieter) Date: Tue, 09 Jul 2013 09:05:19 +0200 Subject: AMQP listening and user-facing daemon References: <51D9E463.7030403@gmail.com> Message-ID: <87a9lw5i34.fsf@handshake.de> Justin Chiu writes: > What is the best approach to writing a concurrent daemon that can > execute callbacks for different types of events (AMQP messages, parsed > output of a subprocess, HTTP requests)? I fear your question is too specific (which means you must investigate yourself). First of all, it is always very difficult to answer questions like "the best way" -- many many details are involved there. You provided details but to appreciate them one has to know "AMQP", "Openstack Notifications", "twisted", "greenlets", ... There will not be a lot of people with this knowledge (I do not belong to them). I believe that you will be able to realize a solution based on (Python) threads. It gives you the more freedom than "twisted" or "greenlets". But a different approach might be "better" (in some respect). From xfq.free at gmail.com Sun Jul 7 19:32:01 2013 From: xfq.free at gmail.com (Xue Fuqiao) Date: Mon, 8 Jul 2013 07:32:01 +0800 Subject: A small question about PEP 8 Message-ID: Hi all, (English is not my native language; please excuse typing errors.) I'm a Python newbie and just started reading PEP 8. PEP says: ----------------------------------------------------------------------- |The closing brace/bracket/parenthesis on multi-line constructs may |either line up under the last item of the list, as in: | |my_list = [ | 1, 2, 3, | 4, 5, 6, | ] |result = some_function_that_takes_arguments( | 'a', 'b', 'c', | 'd', 'e', 'f', | ) ----------------------------------------------------------------------- I think the last item in my_list/result is 6/'f', respectively. So why doesn't the bracket/paren line up _under_ the last item? ISTM the code isn't consistent with the description. I have searched the archive of c.l.p and the web, but nothing helped. Can anyone point me in the right direction? -- Best regards, Xue Fuqiao. From joshua.landau.ws at gmail.com Mon Jul 8 06:39:21 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Mon, 8 Jul 2013 11:39:21 +0100 Subject: A small question about PEP 8 In-Reply-To: References: Message-ID: On 8 July 2013 00:32, Xue Fuqiao wrote: > Hi all, > > (English is not my native language; please excuse typing errors.) > > I'm a Python newbie and just started reading PEP 8. PEP says: > > ----------------------------------------------------------------------- > |The closing brace/bracket/parenthesis on multi-line constructs may > |either line up under the last item of the list, as in: > | > |my_list = [ > | 1, 2, 3, > | 4, 5, 6, > | ] > |result = some_function_that_takes_arguments( > | 'a', 'b', 'c', > | 'd', 'e', 'f', > | ) > ----------------------------------------------------------------------- > > I think the last item in my_list/result is 6/'f', respectively. So why > doesn't the bracket/paren line up _under_ the last item? ISTM the code > isn't consistent with the description. > > I have searched the archive of c.l.p and the web, but nothing helped. > Can anyone point me in the right direction? You will grow to be a wonderful pedant. What it means is that the indentation will match the last one. Imagine: """ a_wonderful_set_of_things = { bannanas_made_of_apples, chocolate_covered_horns, doors_that_slide, china_but_on_the_moon, buffalo_with_windy_hair, not_missing_an_end_brace """? Now, there are several places you can put the end brace. You can (be a massive fool and) put it after the last item: """ a_wonderful_set_of_things = { ..., not_missing_an_end_brace} """ You can also (be a fool and) put it at the same *indentation*: """ a_wonderful_set_of_things = { ..., not_missing_an_end_brace } """ Or you can (be sane) and put it at no indentation: """ a_wonderful_set_of_things = { ..., not_missing_an_end_brace } """ Theoretically, there are more places you could put it (but we won't go there... *shudder*). The second of these is the one that PEP 8 was trying to explain. I agree wording could be improved, but hey. You can file a bug report at bugs.python.org if you care enough. ?} From ethan at stoneleaf.us Mon Jul 8 20:32:12 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 08 Jul 2013 17:32:12 -0700 Subject: A small question about PEP 8 In-Reply-To: References: Message-ID: <51DB5A0C.2020103@stoneleaf.us> On 07/08/2013 03:39 AM, Joshua Landau wrote: > On 8 July 2013 00:32, Xue Fuqiao wrote: >> Hi all, >> >> (English is not my native language; please excuse typing errors.) >> >> I'm a Python newbie and just started reading PEP 8. PEP says: >> >> ----------------------------------------------------------------------- >> |The closing brace/bracket/parenthesis on multi-line constructs may >> |either line up under the last item of the list, as in: >> | >> |my_list = [ >> | 1, 2, 3, >> | 4, 5, 6, >> | ] >> |result = some_function_that_takes_arguments( >> | 'a', 'b', 'c', >> | 'd', 'e', 'f', >> | ) >> ----------------------------------------------------------------------- >> >> I think the last item in my_list/result is 6/'f', respectively. So why >> doesn't the bracket/paren line up _under_ the last item? ISTM the code >> isn't consistent with the description. >> >> I have searched the archive of c.l.p and the web, but nothing helped. >> Can anyone point me in the right direction? > > You will grow to be a wonderful pedant. What it means is that the > indentation will match the last one. Imagine: > > """ > a_wonderful_set_of_things = { > bannanas_made_of_apples, > chocolate_covered_horns, > doors_that_slide, > china_but_on_the_moon, > buffalo_with_windy_hair, > not_missing_an_end_brace > """? > > Now, there are several places you can put the end brace. You can (be a > massive fool and) put it after the last item: > > """ > a_wonderful_set_of_things = { > ..., > not_missing_an_end_brace} > """ > > You can also (be a fool and) put it at the same *indentation*: > > """ > a_wonderful_set_of_things = { > ..., > not_missing_an_end_brace > } > """ Not only a fool but a crazy fool! That next-to-last line should have a comma! -- ~Ethan~ From xfq.free at gmail.com Tue Jul 9 06:47:18 2013 From: xfq.free at gmail.com (Xue Fuqiao) Date: Tue, 9 Jul 2013 18:47:18 +0800 Subject: A small question about PEP 8 In-Reply-To: References: Message-ID: On Mon, Jul 8, 2013 at 6:39 PM, Joshua Landau wrote: > On 8 July 2013 00:32, Xue Fuqiao wrote: >> I'm a Python newbie and just started reading PEP 8. PEP 8 says: >> >> ----------------------------------------------------------------------- >> |The closing brace/bracket/parenthesis on multi-line constructs may >> |either line up under the last item of the list, as in: >> | >> |my_list = [ >> | 1, 2, 3, >> | 4, 5, 6, >> | ] >> |result = some_function_that_takes_arguments( >> | 'a', 'b', 'c', >> | 'd', 'e', 'f', >> | ) >> ----------------------------------------------------------------------- >> >> I think the last item in my_list/result is 6/'f', respectively. So why >> doesn't the bracket/paren line up _under_ the last item? ISTM the code >> isn't consistent with the description. > You will grow to be a wonderful pedant. What it means is that the > indentation will match the last one. Imagine: > > """ > a_wonderful_set_of_things = { > bannanas_made_of_apples, > chocolate_covered_horns, > doors_that_slide, > china_but_on_the_moon, > buffalo_with_windy_hair, > not_missing_an_end_brace > """? > > Now, there are several places you can put the end brace. You can (be a > massive fool and) put it after the last item: > > """ > a_wonderful_set_of_things = { > ..., > not_missing_an_end_brace} > """ > > You can also (be a fool and) put it at the same *indentation*: > > """ > a_wonderful_set_of_things = { > ..., > not_missing_an_end_brace > } > """ > > Or you can (be sane) and put it at no indentation: > > """ > a_wonderful_set_of_things = { > ..., > not_missing_an_end_brace > } > """ > > Theoretically, there are more places you could put it (but we won't go > there... *shudder*). > > The second of these is the one that PEP 8 was trying to explain. I > agree wording could be improved, but hey. You can file a bug report at > bugs.python.org if you care enough. Done as issue 18412. -- Best regards, Xue Fuqiao. From llanitedave at veawb.coop Tue Jul 9 19:44:20 2013 From: llanitedave at veawb.coop (llanitedave) Date: Tue, 9 Jul 2013 16:44:20 -0700 (PDT) Subject: A small question about PEP 8 In-Reply-To: References: Message-ID: <07f4314e-a18b-4fee-bb08-80256e905444@googlegroups.com> I definitely prefer the 'fool' style. From steve+comp.lang.python at pearwood.info Mon Jul 8 07:46:42 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 08 Jul 2013 11:46:42 GMT Subject: A small question about PEP 8 References: Message-ID: <51daa6a2$0$9505$c3e8da3$5496439d@news.astraweb.com> On Mon, 08 Jul 2013 07:32:01 +0800, Xue Fuqiao wrote: > Hi all, > > (English is not my native language; please excuse typing errors.) > > I'm a Python newbie and just started reading PEP 8. PEP says: > > ----------------------------------------------------------------------- > |The closing brace/bracket/parenthesis on multi-line constructs may > |either line up under the last item of the list, as in: > | > |my_list = [ > | 1, 2, 3, > | 4, 5, 6, > | ] > |result = some_function_that_takes_arguments( > | 'a', 'b', 'c', > | 'd', 'e', 'f', > | ) > ----------------------------------------------------------------------- > > I think the last item in my_list/result is 6/'f', respectively. So why > doesn't the bracket/paren line up _under_ the last item? ISTM the code > isn't consistent with the description. I agree. I think it is just a mistake, and should say "under the FIRST item of the list". -- Steven From steve+comp.lang.python at pearwood.info Mon Jul 8 08:02:26 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 08 Jul 2013 12:02:26 GMT Subject: A small question about PEP 8 References: Message-ID: <51daaa51$0$9505$c3e8da3$5496439d@news.astraweb.com> On Mon, 08 Jul 2013 11:39:21 +0100, Joshua Landau wrote: > On 8 July 2013 00:32, Xue Fuqiao wrote: >> Hi all, >> >> (English is not my native language; please excuse typing errors.) >> >> I'm a Python newbie and just started reading PEP 8. PEP says: >> >> ----------------------------------------------------------------------- >> |The closing brace/bracket/parenthesis on multi-line constructs may >> |either line up under the last item of the list, as in: | >> |my_list = [ >> | 1, 2, 3, >> | 4, 5, 6, >> | ] >> |result = some_function_that_takes_arguments( >> | 'a', 'b', 'c', >> | 'd', 'e', 'f', >> | ) >> ----------------------------------------------------------------------- >> >> I think the last item in my_list/result is 6/'f', respectively. So why >> doesn't the bracket/paren line up _under_ the last item? ISTM the code >> isn't consistent with the description. >> >> I have searched the archive of c.l.p and the web, but nothing helped. >> Can anyone point me in the right direction? > > You will grow to be a wonderful pedant. What it means is that the > indentation will match the last one. That's obvious from the example given, but that's nothing like the description given. Imagine: > > """ > a_wonderful_set_of_things = { > bannanas_made_of_apples, > chocolate_covered_horns, > doors_that_slide, > china_but_on_the_moon, > buffalo_with_windy_hair, > not_missing_an_end_brace > """? > > Now, there are several places you can put the end brace. You can (be a > massive fool and) put it after the last item: > > """ > a_wonderful_set_of_things = { > ..., > not_missing_an_end_brace} > """ Well, call me a fool then, because when I have code that extends over *one* additional line, I prefer that: raise ValueError( "Some error message too long to fit on the above line") rather than: raise ValueError( "Some error message too long to fit on the above line" ) > You can also (be a fool and) put it at the same *indentation*: > > """ > a_wonderful_set_of_things = { > ..., > not_missing_an_end_brace > } > """ Call me a fool again, since I prefer this when there are more than one additional line. At least this time I'm in good company, since that's recommended by PEP 8. > Or you can (be sane) and put it at no indentation: > > """ > a_wonderful_set_of_things = { > ..., > not_missing_an_end_brace > } > """ I consider that the least aesthetically pleasing, and also rather awkward: some_result = some_function( arg1, arg2, arg3, arg4, [item1, item2, item3, item4, item5, item6, item7, item8, item9, item10, ], arg6, key=spam, word=eggs, extras=None, foo=bar, ) -- Steven From davea at davea.name Mon Jul 8 08:33:17 2013 From: davea at davea.name (Dave Angel) Date: Mon, 08 Jul 2013 08:33:17 -0400 Subject: A small question about PEP 8 In-Reply-To: <51daaa51$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <51daaa51$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 07/08/2013 08:02 AM, Steven D'Aprano wrote: > On Mon, 08 Jul 2013 11:39:21 +0100, Joshua Landau wrote: > > >> Or you can (be sane) and put it at no indentation: >> >> """ >> a_wonderful_set_of_things = { >> ..., >> not_missing_an_end_brace >> } >> """ > > I consider that the least aesthetically pleasing, and also rather awkward: > > > some_result = some_function( > arg1, arg2, arg3, arg4, > [item1, item2, item3, item4, > item5, item6, item7, > item8, item9, item10, > ], > arg6, key=spam, word=eggs, > extras=None, foo=bar, > ) or even better: class MyClass: def my_method(self): if self.flag: self.value.append( arg1, arg2, arg3 ) return self.value -- DaveA From joshua.landau.ws at gmail.com Mon Jul 8 09:07:30 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Mon, 8 Jul 2013 14:07:30 +0100 Subject: A small question about PEP 8 In-Reply-To: <51daaa51$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <51daaa51$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 8 July 2013 13:02, Steven D'Aprano wrote: > On Mon, 08 Jul 2013 11:39:21 +0100, Joshua Landau wrote: > Imagine: >> >> """ >> a_wonderful_set_of_things = { >> bannanas_made_of_apples, >> chocolate_covered_horns, >> doors_that_slide, >> china_but_on_the_moon, >> buffalo_with_windy_hair, >> not_missing_an_end_brace >> """? >> >> Now, there are several places you can put the end brace. You can (be a >> massive fool and) put it after the last item: >> >> """ >> a_wonderful_set_of_things = { >> ..., >> not_missing_an_end_brace} >> """ > > Well, call me a fool then, because when I have code that extends over > *one* additional line, I prefer that: > > raise ValueError( > "Some error message too long to fit on the above line") > > rather than: > > raise ValueError( > "Some error message too long to fit on the above line" > ) Fool. >> You can also (be a fool and) put it at the same *indentation*: >> >> """ >> a_wonderful_set_of_things = { >> ..., >> not_missing_an_end_brace >> } >> """ > > Call me a fool again, since I prefer this when there are more than one > additional line. At least this time I'm in good company, since that's > recommended by PEP 8. Fool (less so, though). >> Or you can (be sane) and put it at no indentation: >> >> """ >> a_wonderful_set_of_things = { >> ..., >> not_missing_an_end_brace >> } >> """ > > I consider that the least aesthetically pleasing, and also rather awkward: > > > some_result = some_function( > arg1, arg2, arg3, arg4, > [item1, item2, item3, item4, > item5, item6, item7, > item8, item9, item10, > ], > arg6, key=spam, word=eggs, > extras=None, foo=bar, > ) http://xkcd.com/605/ It was obvious? I was talking in context of my example. I used "no indentation" because in that case "no indentation" was correct. If you indent line #1, it follows that other lines might well be indented to keep style consistent. ? I hope. I never suggested otherwise. Since it's not too clear for you, my usage of fool was meant pseudoironically as a subtle reminder that it's all subjective anyway, but that I'm also still right. From ferdy.blatsco at gmail.com Sun Jul 7 20:22:26 2013 From: ferdy.blatsco at gmail.com (blatt) Date: Sun, 7 Jul 2013 17:22:26 -0700 (PDT) Subject: hex dump w/ or w/out utf-8 chars Message-ID: Hi all, but a particular hello to Chris Angelino which with their critics and suggestions pushed me to make a full revision of my application on hex dump in presence of utf-8 chars. If you are not using python 3, the utf-8 codec can add further programming problems, especially if you are not a guru.... The script seems very long but I commented too much ... sorry. It is very useful (at least IMHO...) It works under Linux. but there is still a little problem which I didn't solve (at least programmatically...). # -*- coding: utf-8 -*- # px.py vers. 11 (pxb.py) # python 2.6.6 # hex-dump w/ or w/out utf-8 chars # Using spaces as separators, this script shows # (better than tabnanny) uncorrect indentations. # to save output > python pxb.py hex.txt > px9_out_hex.txt nLenN=3 # n. of digits for lines # version almost thoroughly rewritten on the ground of # the critics and modifications suggested by Chris Angelico # in the first version the utf-8 conversion to hex was shown horizontaly: # 005 # qwerty: non ? unicode bens? ascii # 2 7767773 666 ca 7666666 6667ca 676660 # 3 175249a efe 38 5e93f45 25e33c 13399a # ... but I had to insert additional chars to keep the # synchronization between the literal and the hex part # 005 # qwerty: non ?. unicode bens?. ascii # 2 7767773 666 ca 7666666 6667ca 676660 # 3 175249a efe 38 5e93f45 25e33c 13399a # in the second version I followed Chris suggestion: # "to show the hex utf-8 vertically" # 005 # qwerty: non ? unicode bens? ascii # 2 7767773 666 c 7666666 6667c 676660 # 3 175249a efe 3 5e93f45 25e33 13399a # a a # 8 c # between the two solutions, I selected the first one + syncronization, # which seems more compact and easier to program (... I'm lazy...) # various run options: # std : python px.py file # bash cat : cat file | python px.py (alias hex) # bash echo: echo line | python px.py " " # works on any n. of bytes for utf-8 # For the user: it is helpful to have in a separate file # all special characters of interest, together with their names. # error: # echo '345"789"'|hex > 345"789" 345"789" # 33323332 instead of 333233320 # 3452789 a " " 34527892a # ... correction: avoiding "\n at end of test-line # echo "345'789'"|hex > 345'789' # 333233320 # 34577897a # same error in every run option # If someone can solve this bug... ################### import fileinput import sys, commands lF=[] # input file as list for line in fileinput.input(): # handles all the details of args-or-stdin lF.append(line) sSpacesXLN = ' ' * (nLenN+1) for n in xrange(len(lF)): sLineHexND=lF[n].encode('hex') # ND = no delimiter (space) sLineHex =lF[n].encode('hex').replace('20',' ') sLineHexH =sLineHex[::2] sLineHexL =sLineHex[1::2] sSynchro='' for k in xrange(0,len(sLineHexND),2): if sLineHexND[k]<'8': sSynchro+= sLineHexND[k]+sLineHexND[k+1] k+=1 elif sLineHexND[k]=='c': sSynchro+='c'+sLineHexND[k+1]+sLineHexND[k+2]+sLineHexND[k+3]+'2e' k+=3 elif sLineHexND[k]=='e': sSynchro+='e'+sLineHexND[k+1]+sLineHexND[k+2]+sLineHexND[k+3]+\ sLineHexND[k+4]+sLineHexND[k+5]+'2e2e' k+=5 # text output (synchroinized) print str(n+1).zfill(nLenN)+' '+sSynchro.decode('hex'), print sSpacesXLN + sLineHexH print sSpacesXLN + sLineHexL+ '\n' If there are problems of understanding, probably due to fonts, the best thing is import it in an editor with "mono" fonts... As I already told to Chris... critics are welcome! Bye, Blatt. From rosuav at gmail.com Sun Jul 7 21:17:17 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 8 Jul 2013 11:17:17 +1000 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: References: Message-ID: On Mon, Jul 8, 2013 at 10:22 AM, blatt wrote: > Hi all, > but a particular hello to Chris Angelino which with their critics and > suggestions pushed me to make a full revision of my application on > hex dump in presence of utf-8 chars. Hiya! Glad to have been of assistance :) > As I already told to Chris... critics are welcome! No problem. > # -*- coding: utf-8 -*- > # px.py vers. 11 (pxb.py) # python 2.6.6 > # hex-dump w/ or w/out utf-8 chars > # Using spaces as separators, this script shows > # (better than tabnanny) uncorrect indentations. > > # to save output > python pxb.py hex.txt > px9_out_hex.txt > > nLenN=3 # n. of digits for lines > > # chomp heaps and heaps of comments Little nitpick, since you did invite criticism :) When I went to copy and paste your code, I skipped all the comments and started at the line of hashes... and then didn't have the nLenN definition. Posting code to a forum like this is a huge invitation to try the code (it's the very easiest way to know what it does), so I would recommend having all your comments at the top, and all the code in a block underneath. It'd be that bit easier for us to help you. Not a big deal, though, I did figure out what was going on :) > sLineHex =lF[n].encode('hex').replace('20',' ') Here's the problem. Your hex string ends with "220a", and the replace() method doesn't concern itself with the divisions between bytes. It finds the second 2 of 22 and the leading 0 of 0a and replaces them. I think the best solution may be to avoid the .encode('hex') part, since it's not available in Python 3 anyway. Alternatively (if Py3 migration isn't a concern), you could do something like this: sLineHexND=lF[n].encode('hex') # ND = no delimiter (space) sLineHex =sLineHexND # No reason to redo the encoding twentypos=0 while True: twentypos=sLineHex.find("20",twentypos) if twentypos==-1: break # We've reached the end of the string if not twentypos%2: # It's at an even-numbered position, replace it sLineHex=sLineHex[:twentypos]+' '+sLineHex[twentypos+2:] twentypos+=1 # then continue on as before > sLineHexH =sLineHex[::2] > sLineHexL =sLineHex[1::2] > [ code continues ] Hope that helps! ChrisA From steve+comp.lang.python at pearwood.info Mon Jul 8 01:48:55 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 08 Jul 2013 05:48:55 GMT Subject: hex dump w/ or w/out utf-8 chars References: Message-ID: <51da52c6$0$6512$c3e8da3$5496439d@news.astraweb.com> On Sun, 07 Jul 2013 17:22:26 -0700, blatt wrote: > Hi all, > but a particular hello to Chris Angelino which with their critics and > suggestions pushed me to make a full revision of my application on hex > dump in presence of utf-8 chars. I don't understand what you are trying to say. All characters are UTF-8 characters. "a" is a UTF-8 character. So is "?". > If you are not using python 3, the utf-8 codec can add further > programming problems, On the contrary, I find that so long as you understand what you are doing it solves problems, not adds them. However, if you are confused about the difference between characters (text strings) and bytes, or if you are dealing with arbitrary binary data and trying to treat it as if it were UTF-8 encoded text, then you can have errors. Those errors are a good thing. > especially if you are not a guru.... The script > seems very long but I commented too much ... sorry. It is very useful > (at least IMHO...) > It works under Linux. but there is still a little problem which I didn't > solve (at least programmatically...). > > > # -*- coding: utf-8 -*- > # px.py vers. 11 (pxb.py) > # python 2.6.6 # hex-dump w/ or w/out utf-8 chars > # Using spaces as separators, this script shows > # (better than tabnanny) uncorrect indentations. The word you are looking for is "incorrect". > # to save output > python pxb.py hex.txt > px9_out_hex.txt > > nLenN=3 # n. of digits for lines > > # version almost thoroughly rewritten on the ground of > # the critics and modifications suggested by Chris Angelico > > # in the first version the utf-8 conversion to hex was shown > horizontaly: > > # 005 # qwerty: non ? unicode bens? ascii > # 2 7767773 666 ca 7666666 6667ca 676660 > # 3 175249a efe 38 5e93f45 25e33c 13399a Oh! We're supposed to read the output *downwards*! That's not very intuitive. It took me a while to work that out. You should at least say so. > # ... but I had to insert additional chars to keep the > # synchronization between the literal and the hex part > > # 005 # qwerty: non ?. unicode bens?. ascii > # 2 7767773 666 ca 7666666 6667ca 676660 > # 3 175249a efe 38 5e93f45 25e33c 13399a Well that sucks, because now sometimes you have to read downwards (character 'q' -> hex 71, reading downwards) and sometimes you read both downwards and across (character '?' -> hex c3a8). Sometimes a dot means a dot and sometimes it means filler. How is the user supposed to know when to read down and when across? > # in the second version I followed Chris suggestion: > # "to show the hex utf-8 vertically" You're already showing UTF-8 characters vertically, if they happen to be a one-byte character. Better to be consistent and always show characters vertical, regardless of whether they are one, two or four bytes. > # 005 # qwerty: non ? unicode bens? ascii > # 2 7767773 666 c 7666666 6667c 676660 > # 3 175249a efe 3 5e93f45 25e33 13399a > # a a > # 8 c Much better! Now at least you can trivially read down the column to see the bytes used for each character. As an alternative, you can space each character to show the bytes horizontally, displaying spaces and other invisible characters either as dots, backslash escapes, or Unicode control pictures, whichever you prefer. The example below uses dots for spaces and backslash escape for newline: q w e r t y : . n o n . ? . u n i 71 77 65 72 74 79 3a 20 6e 6f 6e 20 c3 a8 20 75 6e 69 c o d e . b e n s ? . a s c i i \n 63 6f 64 65 20 62 65 6e 73 c3 ac 20 61 73 63 69 69 0a There will always be some ambiguity between (e.g.) dot representing a dot, and it representing an invisible control character or space, but the reader can always tell them apart by reading the hex value, which you *always* read horizontally whether it is one byte, two or four. There's never any confusion whether you should read down or across. Unfortunately, most fonts don't support the Unicode control pictures. But if you choose to use them, here they are, together with their Unicode name. You can use the form '\N{...}' # Python 3 u'\N{...}' # Python 2 to get the characters, replacing ... with the name shown below: ? SYMBOL FOR NULL ? SYMBOL FOR START OF HEADING ? SYMBOL FOR START OF TEXT ? SYMBOL FOR END OF TEXT ? SYMBOL FOR END OF TRANSMISSION ? SYMBOL FOR ENQUIRY ? SYMBOL FOR ACKNOWLEDGE ? SYMBOL FOR BELL ? SYMBOL FOR BACKSPACE ? SYMBOL FOR HORIZONTAL TABULATION ? SYMBOL FOR LINE FEED ? SYMBOL FOR VERTICAL TABULATION ? SYMBOL FOR FORM FEED ? SYMBOL FOR CARRIAGE RETURN ? SYMBOL FOR SHIFT OUT ? SYMBOL FOR SHIFT IN ? SYMBOL FOR DATA LINK ESCAPE ? SYMBOL FOR DEVICE CONTROL ONE ? SYMBOL FOR DEVICE CONTROL TWO ? SYMBOL FOR DEVICE CONTROL THREE ? SYMBOL FOR DEVICE CONTROL FOUR ? SYMBOL FOR NEGATIVE ACKNOWLEDGE ? SYMBOL FOR SYNCHRONOUS IDLE ? SYMBOL FOR END OF TRANSMISSION BLOCK ? SYMBOL FOR CANCEL ? SYMBOL FOR END OF MEDIUM ? SYMBOL FOR SUBSTITUTE ? SYMBOL FOR ESCAPE ? SYMBOL FOR FILE SEPARATOR ? SYMBOL FOR GROUP SEPARATOR ? SYMBOL FOR RECORD SEPARATOR ? SYMBOL FOR UNIT SEPARATOR ? SYMBOL FOR SPACE ? SYMBOL FOR DELETE ? BLANK SYMBOL ? OPEN BOX ? SYMBOL FOR NEWLINE ? SYMBOL FOR DELETE FORM TWO ? SYMBOL FOR SUBSTITUTE FORM TWO (I wish more fonts would support these characters, they are very useful.) [...] > # works on any n. of bytes for utf-8 > > # For the user: it is helpful to have in a separate file > # all special characters of interest, together with their names. In Python, you can use the unicodedata module to look up characters by name, or given the character, find out what it's name is. [...] > import fileinput > import sys, commands > > lF=[] # input file as list > for line in fileinput.input(): # handles all the details of args-or- stdin > lF.append(line) That is more easily written as: lF = list(fileinput.input()) and better written with a meaningful file name. Whenever you have a variable, and find the need to give a comment explaining what the variable name means, you should consider a more descriptive name. When that name is a cryptic two letter name, that goes double. > sSpacesXLN = ' ' * (nLenN+1) > > > for n in xrange(len(lF)): > sLineHexND=lF[n].encode('hex') # ND = no delimiter (space) You're programming like a Pascal or C programmer. There is nearly never any need to write code like that in Python. Rather than iterate over the indexes, then extract the part you want, it is better to iterate directly over the parts you want: for line in lF: sLineHexND = line.encode('hex') > sLineHex =lF[n].encode('hex').replace('20',' ') > sLineHexH =sLineHex[::2] > sLineHexL =sLineHex[1::2] Trying to keep code lined up in this way is a bad habit to get into. It just sets you up for many hours of unproductive adding and deleting spaces trying to keep things aligned. Also, what on earth are all these "s" prefixes? > sSynchro='' > for k in xrange(0,len(sLineHexND),2): Probably the best way to walk through a string, grabbing the characters in pairs, comes from the itertools module: see the recipe for "grouper". http://docs.python.org/2/library/itertools.html Here is a simplified version: assert len(line) % 2 == 0 for pair in zip(*([iter(line)]*2)): ... although understanding how it works requires a little advanced knowledge. > if sLineHexND[k]<'8': > sSynchro+= sLineHexND[k]+sLineHexND[k+1] > k+=1 > elif sLineHexND[k]=='c': > sSynchro+='c'+sLineHexND[k+1]+sLineHexND[k+2]+sLineHexND[k +3]+'2e' > k+=3 > elif sLineHexND[k]=='e': > sSynchro+='e'+sLineHexND[k+1]+sLineHexND[k+2]+sLineHexND[k +3]+\ > sLineHexND[k+4]+sLineHexND[k+5]+'2e2e' > k+=5 Apart from being hideously ugly to read, I do not believe this code works the way you think it works. Adding to the loop variable doesn't advance the loop. Try this and see for yourself: for i in range(10): print(i) i += 5 The loop variable just gets reset once it reaches the top of the loop again. -- Steven From ferdy.blatsco at gmail.com Mon Jul 8 13:31:32 2013 From: ferdy.blatsco at gmail.com (ferdy.blatsco at gmail.com) Date: Mon, 8 Jul 2013 10:31:32 -0700 (PDT) Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: References: Message-ID: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> Hi Chris, glad to have received your contribution, but I was expecting much more critics... Starting from the "little nitpick" about the comment dispositon in my script... you are correct... It is a bad habit on my part to place variables subjected to change at the beginning of the script... and then forget it... About the mistake due to replace, you gave me a perfect explanation. Unfortunately (as probably I told you before) I will never pass to Python 3... Guido should not always listen only to gurus like him... I don't like Python as before...starting from OOP and ending with codecs like utf-8. Regarding OOP, much appreciated expecially by experts, he could use python 2 for hiding the complexities of OOP (improving, as an effect, object's code hiding) moving classes and objects to imported methods, leaving in this way the programming style to the well known old style: sequential programming and functions. About utf-8... the same solution: keep utf-8 but for the non experts, add methods to convert to solutions which use the range 128-255 of only one byte (I do not give a damn about chinese and "similia"!...) I know that is a lost battle (in italian "una battaglia persa")! Bye, Blatt From rosuav at gmail.com Mon Jul 8 13:52:17 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 9 Jul 2013 03:52:17 +1000 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> Message-ID: On Tue, Jul 9, 2013 at 3:31 AM, wrote: > Unfortunately (as probably I told you before) I will never pass to > Python 3... Guido should not always listen only to gurus like him... > I don't like Python as before...starting from OOP and ending with codecs > like utf-8. Regarding OOP, much appreciated expecially by experts, he > could use python 2 for hiding the complexities of OOP (improving, as an > effect, object's code hiding) moving classes and objects to > imported methods, leaving in this way the programming style to the > well known old style: sequential programming and functions. > About utf-8... the same solution: keep utf-8 but for the non experts, add > methods to convert to solutions which use the range 128-255 of only one > byte (I do not give a damn about chinese and "similia"!...) > I know that is a lost battle (in italian "una battaglia persa")! Well, there won't be a Python 2.8, so you really should consider moving at some point. Python 3.3 is already way better than 2.7 in many ways, 3.4 will improve on 3.3, and the future is pretty clear. But nobody's forcing you, and 2.7.x will continue to get bugfix/security releases for a while. (Personally, I'd be happy if everyone moved off the 2.3/2.4 releases. It's not too hard supporting 2.6+ or 2.7+.) The thing is, you're thinking about UTF-8, but you should be thinking about Unicode. I recommend you read these articles: http://www.joelonsoftware.com/articles/Unicode.html http://unspecified.wordpress.com/2012/04/19/the-importance-of-language-level-abstract-unicode-strings/ So long as you are thinking about different groups of characters as different, and wanting a solution that maps characters down into the <256 range, you will never be able to cleanly internationalize. With Python 3.3+, you can ignore the differences between ASCII, BMP, and SMP characters; they're all just "characters". Everything works perfectly with Unicode. ChrisA From wxjmfauth at gmail.com Thu Jul 11 09:18:13 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Thu, 11 Jul 2013 06:18:13 -0700 (PDT) Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> Message-ID: Le lundi 8 juillet 2013 19:52:17 UTC+2, Chris Angelico a ?crit?: > On Tue, Jul 9, 2013 at 3:31 AM, wrote: > > > Unfortunately (as probably I told you before) I will never pass to > > > Python 3... Guido should not always listen only to gurus like him... > > > I don't like Python as before...starting from OOP and ending with codecs > > > like utf-8. Regarding OOP, much appreciated expecially by experts, he > > > could use python 2 for hiding the complexities of OOP (improving, as an > > > effect, object's code hiding) moving classes and objects to > > > imported methods, leaving in this way the programming style to the > > > well known old style: sequential programming and functions. > > > About utf-8... the same solution: keep utf-8 but for the non experts, add > > > methods to convert to solutions which use the range 128-255 of only one > > > byte (I do not give a damn about chinese and "similia"!...) > > > I know that is a lost battle (in italian "una battaglia persa")! > > > > Well, there won't be a Python 2.8, so you really should consider > > moving at some point. Python 3.3 is already way better than 2.7 in > > many ways, 3.4 will improve on 3.3, and the future is pretty clear. > > But nobody's forcing you, and 2.7.x will continue to get > > bugfix/security releases for a while. (Personally, I'd be happy if > > everyone moved off the 2.3/2.4 releases. It's not too hard supporting > > 2.6+ or 2.7+.) > > > > The thing is, you're thinking about UTF-8, but you should be thinking > > about Unicode. I recommend you read these articles: > > > > http://www.joelonsoftware.com/articles/Unicode.html > > http://unspecified.wordpress.com/2012/04/19/the-importance-of-language-level-abstract-unicode-strings/ > > > > So long as you are thinking about different groups of characters as > > different, and wanting a solution that maps characters down into the > > <256 range, you will never be able to cleanly internationalize. With > > Python 3.3+, you can ignore the differences between ASCII, BMP, and > > SMP characters; they're all just "characters". Everything works > > perfectly with Unicode. > ----------- Just to stick with this funny character ?, a ucs-2 char in the Flexible String Representation nomenclature. It seems to me that, when one needs more than ten bytes to encode it, >>> sys.getsizeof('a') 26 >>> sys.getsizeof('?') 40 this is far away from the perfection. BTW, for a modern language, is not ucs2 considered as obsolete since many, many years? jmf From rosuav at gmail.com Thu Jul 11 09:32:00 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 11 Jul 2013 23:32:00 +1000 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> Message-ID: On Thu, Jul 11, 2013 at 11:18 PM, wrote: > Just to stick with this funny character ?, a ucs-2 char > in the Flexible String Representation nomenclature. > > It seems to me that, when one needs more than ten bytes > to encode it, > >>>> sys.getsizeof('a') > 26 >>>> sys.getsizeof('?') > 40 > > this is far away from the perfection. Better comparison is to see how much space is used by one copy of it, and how much by two copies: >>> sys.getsizeof('aa')-sys.getsizeof('a') 1 >>> sys.getsizeof('??')-sys.getsizeof('?') 2 String objects have overhead. Big deal. > BTW, for a modern language, is not ucs2 considered > as obsolete since many, many years? Clearly. And similarly, the 16-bit integer has been completely obsoleted, as there is no reason anyone should ever bother to use it. Same with the float type - everyone uses double or better these days, right? http://www.postgresql.org/docs/current/static/datatype-numeric.html http://www.cplusplus.com/doc/tutorial/variables/ Nope, nobody uses small integers any more, they're clearly completely obsolete. ChrisA From wxjmfauth at gmail.com Thu Jul 11 14:42:26 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Thu, 11 Jul 2013 11:42:26 -0700 (PDT) Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> Message-ID: <26d5c832-eaa1-439e-af61-e2855af2cd18@googlegroups.com> Le jeudi 11 juillet 2013 15:32:00 UTC+2, Chris Angelico a ?crit?: > On Thu, Jul 11, 2013 at 11:18 PM, wrote: > > > Just to stick with this funny character ?, a ucs-2 char > > > in the Flexible String Representation nomenclature. > > > > > > It seems to me that, when one needs more than ten bytes > > > to encode it, > > > > > >>>> sys.getsizeof('a') > > > 26 > > >>>> sys.getsizeof('?') > > > 40 > > > > > > this is far away from the perfection. > > > > Better comparison is to see how much space is used by one copy of it, > > and how much by two copies: > > > > >>> sys.getsizeof('aa')-sys.getsizeof('a') > > 1 > > >>> sys.getsizeof('??')-sys.getsizeof('?') > > 2 > > > > String objects have overhead. Big deal. > > > > > BTW, for a modern language, is not ucs2 considered > > > as obsolete since many, many years? > > > > Clearly. And similarly, the 16-bit integer has been completely > > obsoleted, as there is no reason anyone should ever bother to use it. > > Same with the float type - everyone uses double or better these days, > > right? > > > > http://www.postgresql.org/docs/current/static/datatype-numeric.html > > http://www.cplusplus.com/doc/tutorial/variables/ > > > > Nope, nobody uses small integers any more, they're clearly completely obsolete. > > > Sure there is some overhead because a str is a class. It still remain that a "?" weights 14 bytes more than an "a". In "a?", the ? weights 6 bytes. >>> sys.getsizeof('a') 26 >>> sys.getsizeof('a?') 42 and in "a??", the ? weights 2 bytes sys.getsizeof('a??') And what to say about this "ucs4" char/string '\U0001d11e' which is weighting 18 bytes more than an "a". >>> sys.getsizeof('\U0001d11e') 44 A total absurdity. How does is come? Very simple, once you split Unicode in subsets, not only you have to handle these subsets, you have to create "markers" to differentiate them. Not only, you produce "markers", you have to handle the mess generated by these "markers". Hiding this markers in the everhead of the class does not mean that they should not be counted as part of the coding scheme. BTW, since when a serious coding scheme need an extermal marker? >>> sys.getsizeof('aa') - sys.getsizeof('a') 1 Shortly, if my algebra is still correct: (overhead + marker + 2*'a') - (overhead + marker + 'a') = (overhead + marker + 2*'a') - overhead - marker - 'a') = overhead - overhead + marker - marker + 2*'a' - 'a' = 0 + 0 + 'a' = 1 The "marker" has magically disappeared. jmf From wxjmfauth at gmail.com Thu Jul 11 14:44:16 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Thu, 11 Jul 2013 11:44:16 -0700 (PDT) Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: <26d5c832-eaa1-439e-af61-e2855af2cd18@googlegroups.com> References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> <26d5c832-eaa1-439e-af61-e2855af2cd18@googlegroups.com> Message-ID: Le jeudi 11 juillet 2013 20:42:26 UTC+2, wxjm... at gmail.com a ?crit?: > Le jeudi 11 juillet 2013 15:32:00 UTC+2, Chris Angelico a ?crit?: > > > On Thu, Jul 11, 2013 at 11:18 PM, wrote: > > > > > > > Just to stick with this funny character ?, a ucs-2 char > > > > > > > in the Flexible String Representation nomenclature. > > > > > > > > > > > > > > It seems to me that, when one needs more than ten bytes > > > > > > > to encode it, > > > > > > > > > > > > > >>>> sys.getsizeof('a') > > > > > > > 26 > > > > > > >>>> sys.getsizeof('?') > > > > > > > 40 > > > > > > > > > > > > > > this is far away from the perfection. > > > > > > > > > > > > Better comparison is to see how much space is used by one copy of it, > > > > > > and how much by two copies: > > > > > > > > > > > > >>> sys.getsizeof('aa')-sys.getsizeof('a') > > > > > > 1 > > > > > > >>> sys.getsizeof('??')-sys.getsizeof('?') > > > > > > 2 > > > > > > > > > > > > String objects have overhead. Big deal. > > > > > > > > > > > > > BTW, for a modern language, is not ucs2 considered > > > > > > > as obsolete since many, many years? > > > > > > > > > > > > Clearly. And similarly, the 16-bit integer has been completely > > > > > > obsoleted, as there is no reason anyone should ever bother to use it. > > > > > > Same with the float type - everyone uses double or better these days, > > > > > > right? > > > > > > > > > > > > http://www.postgresql.org/docs/current/static/datatype-numeric.html > > > > > > http://www.cplusplus.com/doc/tutorial/variables/ > > > > > > > > > > > > Nope, nobody uses small integers any more, they're clearly completely obsolete. > > > > > > > > > > > > > Sure there is some overhead because a str is a class. > > It still remain that a "?" weights 14 bytes more than > > an "a". > > > > In "a?", the ? weights 6 bytes. > > > > >>> sys.getsizeof('a') > > 26 > > >>> sys.getsizeof('a?') > > 42 > > > > and in "a??", the ? weights 2 bytes > > > > sys.getsizeof('a??') > > > > And what to say about this "ucs4" char/string '\U0001d11e' which > > is weighting 18 bytes more than an "a". > > > > >>> sys.getsizeof('\U0001d11e') > > 44 > > > > A total absurdity. How does is come? Very simple, once you > > split Unicode in subsets, not only you have to handle these > > subsets, you have to create "markers" to differentiate them. > > Not only, you produce "markers", you have to handle the > > mess generated by these "markers". Hiding this markers > > in the everhead of the class does not mean that they should > > not be counted as part of the coding scheme. BTW, since > > when a serious coding scheme need an extermal marker? > > > > > > > > >>> sys.getsizeof('aa') - sys.getsizeof('a') > > 1 > > > > Shortly, if my algebra is still correct: > > > > (overhead + marker + 2*'a') - (overhead + marker + 'a') > > = (overhead + marker + 2*'a') - overhead - marker - 'a' > > = overhead - overhead + marker - marker + 2*'a' - 'a' > > = 0 + 0 + 'a' > > = 1 > > > > The "marker" has magically disappeared. > > > > jmf From rosuav at gmail.com Thu Jul 11 22:16:21 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 12 Jul 2013 12:16:21 +1000 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: <26d5c832-eaa1-439e-af61-e2855af2cd18@googlegroups.com> References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> <26d5c832-eaa1-439e-af61-e2855af2cd18@googlegroups.com> Message-ID: On Fri, Jul 12, 2013 at 4:42 AM, wrote: > BTW, since > when a serious coding scheme need an extermal marker? > All of them. Content-type: text/plain; charset=UTF-8 ChrisA From steve+comp.lang.python at pearwood.info Thu Jul 11 23:18:44 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 12 Jul 2013 03:18:44 GMT Subject: hex dump w/ or w/out utf-8 chars References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> <26d5c832-eaa1-439e-af61-e2855af2cd18@googlegroups.com> Message-ID: <51df7593$0$9505$c3e8da3$5496439d@news.astraweb.com> On Thu, 11 Jul 2013 11:42:26 -0700, wxjmfauth wrote: > And what to say about this "ucs4" char/string '\U0001d11e' which is > weighting 18 bytes more than an "a". > >>>> sys.getsizeof('\U0001d11e') > 44 > > A total absurdity. You should stick to Python 3.1 and 3.2 then: py> print(sys.version) 3.1.3 (r313:86834, Nov 28 2010, 11:28:10) [GCC 4.4.5] py> sys.getsizeof('\U0001d11e') 36 py> sys.getsizeof('a') 36 Now all your strings will be just as heavy, every single variable name and attribute name will use four times as much memory. Happy now? > How does is come? Very simple, once you split Unicode > in subsets, not only you have to handle these subsets, you have to > create "markers" to differentiate them. Not only, you produce "markers", > you have to handle the mess generated by these "markers". Hiding this > markers in the everhead of the class does not mean that they should not > be counted as part of the coding scheme. BTW, since when a serious > coding scheme need an extermal marker? Since always. How do you think that (say) a C compiler can tell the difference between the long 1199876496 and the float 67923.125? They both have exactly the same four bytes: py> import struct py> struct.pack('f', 67923.125) b'\x90\xa9\x84G' py> struct.pack('l', 1199876496) b'\x90\xa9\x84G' *Everything* in a computer is bytes. The only way to tell them apart is by external markers. -- Steven From wxjmfauth at gmail.com Fri Jul 12 17:42:56 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Fri, 12 Jul 2013 14:42:56 -0700 (PDT) Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: <51df7593$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> <26d5c832-eaa1-439e-af61-e2855af2cd18@googlegroups.com> <51df7593$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: <5f8322b5-56f1-4dda-9dae-203453eb62b8@googlegroups.com> Le vendredi 12 juillet 2013 05:18:44 UTC+2, Steven D'Aprano a ?crit?: > On Thu, 11 Jul 2013 11:42:26 -0700, wxjmfauth wrote: > > > Now all your strings will be just as heavy, every single variable name > > and attribute name will use four times as much memory. Happy now? > -------- >>> ? = 999 >>> class C: ... c?ur = 'heart' ... - Why always this magic number "four"? - Are you able to think once non-ascii? - Have you once had the feeling to be penalized, because you are using fonts with OpenType technology? - Have once had problem with pdf? I can tell you, utf32 is peanuts compared to the used CID-font you are using. - Did you toy once with a unicode TeX engine? - Did you take a look at a rendering engine code like HarfBuzz? jmf From wxjmfauth at gmail.com Sat Jul 13 03:56:52 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Sat, 13 Jul 2013 00:56:52 -0700 (PDT) Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> <26d5c832-eaa1-439e-af61-e2855af2cd18@googlegroups.com> Message-ID: <2165f7cc-b9bf-41b1-b128-d33b522046dc@googlegroups.com> Le vendredi 12 juillet 2013 04:16:21 UTC+2, Chris Angelico a ?crit?: > On Fri, Jul 12, 2013 at 4:42 AM, wrote: > > > BTW, since > > > when a serious coding scheme need an extermal marker? > > > > > > > All of them. > > > > Content-type: text/plain; charset=UTF-8 > > > > ChrisA ------ No one. You are confusing the knowledge of a coding scheme and the intrisinc information a "coding scheme" *may* have, in a mandatory way, to work properly. These are conceptualy two different things. I am convinced you are not conceptually understanding utf-8 very well. I wrote many times, "utf-8 does not produce bytes, but Unicode Encoding Units". A similar coding scheme: iso-6937 . Try to write an editor, a text widget, with with a coding scheme like the Flexible String Represenation. You will quickly notice, it is impossible (understand correctly). (You do not need a computer, just a sheet of paper and a pencil) Hint: what is the character at the caret position? jmf From lele at metapensiero.it Sat Jul 13 04:24:45 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Sat, 13 Jul 2013 10:24:45 +0200 Subject: hex dump w/ or w/out utf-8 chars References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> <26d5c832-eaa1-439e-af61-e2855af2cd18@googlegroups.com> <2165f7cc-b9bf-41b1-b128-d33b522046dc@googlegroups.com> Message-ID: <87sizilvea.fsf@nautilus.nautilus> wxjmfauth at gmail.com writes: > Try to write an editor, a text widget, with with a coding > scheme like the Flexible String Represenation. You will > quickly notice, it is impossible (understand correctly). > (You do not need a computer, just a sheet of paper and a pencil) > Hint: what is the character at the caret position? I am convinced you are not conceptually understanding FST very well. Alternatively, you may have a strange notion of ?impossible?. Or both. ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From steve+comp.lang.python at pearwood.info Sat Jul 13 05:36:06 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 13 Jul 2013 09:36:06 GMT Subject: hex dump w/ or w/out utf-8 chars References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> <26d5c832-eaa1-439e-af61-e2855af2cd18@googlegroups.com> <2165f7cc-b9bf-41b1-b128-d33b522046dc@googlegroups.com> Message-ID: <51e11f85$0$9505$c3e8da3$5496439d@news.astraweb.com> On Sat, 13 Jul 2013 00:56:52 -0700, wxjmfauth wrote: > I am convinced you are not conceptually understanding utf-8 very well. I > wrote many times, "utf-8 does not produce bytes, but Unicode Encoding > Units". Just because you write it many times, doesn't make it correct. You are simply wrong. UTF-8 produces bytes. That's what gets written to files and transmitted over networks, bytes, not "Unicode Encoding Units", whatever they are. > A similar coding scheme: iso-6937 . > > Try to write an editor, a text widget, with with a coding scheme like > the Flexible String Represenation. You will quickly notice, it is > impossible (understand correctly). (You do not need a computer, just a > sheet of paper and a pencil) Hint: what is the character at the caret > position? That is a simple index operation into the buffer. If the caret position is 10 characters in, you index buffer[10-1] and it will give you the character to the left of the caret. buffer[10] will give you the character to the right of the caret. It is simple, trivial, and easy. The buffer itself knows whether to look ahead 10 bytes, 10*2 bytes or 10*4 bytes. Here is an example of such a tiny buffer, implemented in Python 3.3 with the hated Flexible String Representation. In each example, imagine the caret is five characters from the left: 12345|more characters here... It works regardless of whether your characters are ASCII: py> buffer = '12345ABCD...' py> buffer[5-1] # character to the left of the caret '5' py> buffer[5] # character to the right of the caret 'A' Latin 1: py> buffer = '12345????...' py> buffer[5-1] # character to the left of the caret '5' py> buffer[5] # character to the right of the caret '?' Other BMP characters: py> buffer = '12345????...' py> buffer[5-1] # character to the left of the caret '5' py> buffer[5] # character to the right of the caret '?' And Supplementary Plane Characters: py> buffer = ('12345' ... '\N{ALCHEMICAL SYMBOL FOR AIR}' ... '\N{ALCHEMICAL SYMBOL FOR FIRE}' ... '\N{ALCHEMICAL SYMBOL FOR EARTH}' ... '\N{ALCHEMICAL SYMBOL FOR WATER}' ... '...') py> buffer '12345????...' py> len(buffer) 12 py> buffer[5-1] # character to the left of the caret '5' py> buffer[5] # character to the right of the caret '?' py> unicodedata.name(buffer[5]) 'ALCHEMICAL SYMBOL FOR AIR' And it all Just Works in Python 3.3. So much for "impossible to tell" what the character at the carat is. It is *trivial*. Ah, but how about Python 3.2? We set up the same buffer: py> buffer = ('12345' ... '\N{ALCHEMICAL SYMBOL FOR AIR}' ... '\N{ALCHEMICAL SYMBOL FOR FIRE}' ... '\N{ALCHEMICAL SYMBOL FOR EARTH}' ... '\N{ALCHEMICAL SYMBOL FOR WATER}' ... '...') py> buffer '12345????...' py> len(buffer) 16 Sixteen? Sixteen? Where did the extra four characters come from? They came from *surrogate pairs*. py> buffer[5-1] # character to the left of the caret '5' py> buffer[5] # character to the right of the caret '\ud83d' Funny, that looks different. py> unicodedata.name(buffer[5]) Traceback (most recent call last): File "", line 1, in ValueError: no such name No name? Because buffer[5] is only *half* of the surrogate pair. It is broken, and there is really no way of fixing that breakage in Python 3.2 with a narrow build. You can fix it with a wide build, but only at the cost of every string, every name, using double the amount of storage, whether it needs it or not. -- Steven From rosuav at gmail.com Sat Jul 13 05:46:40 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 13 Jul 2013 19:46:40 +1000 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: <2165f7cc-b9bf-41b1-b128-d33b522046dc@googlegroups.com> References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> <26d5c832-eaa1-439e-af61-e2855af2cd18@googlegroups.com> <2165f7cc-b9bf-41b1-b128-d33b522046dc@googlegroups.com> Message-ID: On Sat, Jul 13, 2013 at 5:56 PM, wrote: > Try to write an editor, a text widget, with with a coding > scheme like the Flexible String Represenation. You will > quickly notice, it is impossible (understand correctly). > (You do not need a computer, just a sheet of paper and a pencil) > Hint: what is the character at the caret position? I would use an internal representation that allows insertion and deletion - in its simplest form, a list of strings. And those strings would be whatever I can most conveniently work with. I've never built a text editor widget, because my libraries always provide them. But there is a rough parallel in the display storage for Gypsum, which stores a series of lines, each of which is a series of sections in different colors. (A line might be a single section, ie one color for its whole length.) I store them in arrays of (color, string, color, string, color, string...). The strings I use are in the format wanted by my display subsystem - which in my case is the native string type of the language, which... oh, what a pity for jmf, is a flexible object that uses 8, 16, or 32 bits for each character. ChrisA From steve+comp.lang.python at pearwood.info Sat Jul 13 05:49:10 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 13 Jul 2013 09:49:10 GMT Subject: hex dump w/ or w/out utf-8 chars References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> <26d5c832-eaa1-439e-af61-e2855af2cd18@googlegroups.com> <2165f7cc-b9bf-41b1-b128-d33b522046dc@googlegroups.com> Message-ID: <51e12295$0$9505$c3e8da3$5496439d@news.astraweb.com> On Sat, 13 Jul 2013 00:56:52 -0700, wxjmfauth wrote: > You are confusing the knowledge of a coding scheme and the intrisinc > information a "coding scheme" *may* have, in a mandatory way, to work > properly. These are conceptualy two different things. *May* have, in a *mandatory* way? JMF, I know you are not a native English speaker, so you might not be aware just how silly your statement is. If it *may* have, it is optional, since it *may not* have instead. But if it is optional, it is not mandatory. You are making so much fuss over such a simple, obvious implementation for strings. The language Pike has done the same thing for probably a decade or so. Ironically, Python has done the same thing for integers for many versions too. They just didn't call it "Flexible Integer Representation", but that's what it is. For integers smaller than 2**31, they are stored as C longs (plus object overhead). For integers larger than 2**31, they are promoted to a BigNum implementation that can handle unlimited digits. Using Python 2.7, where it is more obvious because the BigNum has an L appended to the display, and a different type: py> for n in (1, 2**20, 2**30, 2**31, 2**65): ... print repr(n), type(n), sys.getsizeof(n) ... 1 12 1048576 12 1073741824 12 2147483648L 18 36893488147419103232L 22 You have been using Flexible Integer Representation for *years*, and it works great, and you've never noticed any problems. -- Steven From rosuav at gmail.com Sat Jul 13 06:09:04 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 13 Jul 2013 20:09:04 +1000 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: <51e12295$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> <26d5c832-eaa1-439e-af61-e2855af2cd18@googlegroups.com> <2165f7cc-b9bf-41b1-b128-d33b522046dc@googlegroups.com> <51e12295$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sat, Jul 13, 2013 at 7:49 PM, Steven D'Aprano wrote: > Ironically, Python has done the same thing for integers for many versions > too. They just didn't call it "Flexible Integer Representation", but > that's what it is. For integers smaller than 2**31, they are stored as C > longs (plus object overhead). For integers larger than 2**31, they are > promoted to a BigNum implementation that can handle unlimited digits. Hmm. That's true of Python 2 (mostly - once an operation yields a long, it never reverts to int, whereas a string will shrink if you remove the wider characters from it), but not, I think, of Python 3. The optimization isn't there any more. At least, I did some tinkering a while ago (on 3.2, I think), so maybe it's been reinstated since. As of Python 3 and the unification of types, it's definitely possible to put that in as a pure optimization, anyhow. ChrisA From wxjmfauth at gmail.com Sat Jul 13 10:37:26 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Sat, 13 Jul 2013 07:37:26 -0700 (PDT) Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: <51e12295$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> <26d5c832-eaa1-439e-af61-e2855af2cd18@googlegroups.com> <2165f7cc-b9bf-41b1-b128-d33b522046dc@googlegroups.com> <51e12295$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: Le samedi 13 juillet 2013 11:49:10 UTC+2, Steven D'Aprano a ?crit?: > On Sat, 13 Jul 2013 00:56:52 -0700, wxjmfauth wrote: > > > > > You are confusing the knowledge of a coding scheme and the intrisinc > > > information a "coding scheme" *may* have, in a mandatory way, to work > > > properly. These are conceptualy two different things. > > > > *May* have, in a *mandatory* way? > > > > JMF, I know you are not a native English speaker, so you might not be > > aware just how silly your statement is. If it *may* have, it is optional, > > since it *may not* have instead. But if it is optional, it is not > > mandatory. > > > > You are making so much fuss over such a simple, obvious implementation > > for strings. The language Pike has done the same thing for probably a > > decade or so. > > > > Ironically, Python has done the same thing for integers for many versions > > too. They just didn't call it "Flexible Integer Representation", but > > that's what it is. For integers smaller than 2**31, they are stored as C > > longs (plus object overhead). For integers larger than 2**31, they are > > promoted to a BigNum implementation that can handle unlimited digits. > > > > Using Python 2.7, where it is more obvious because the BigNum has an L > > appended to the display, and a different type: > > > > py> for n in (1, 2**20, 2**30, 2**31, 2**65): > > ... print repr(n), type(n), sys.getsizeof(n) > > ... > > 1 12 > > 1048576 12 > > 1073741824 12 > > 2147483648L 18 > > 36893488147419103232L 22 > > > > > > You have been using Flexible Integer Representation for *years*, and it > > works great, and you've never noticed any problems. > > > > > > > > -- > > Steven ------ The FSR is naive and badly working. I can not force people to understand the coding of the characters [*]. I'm the first to recognize that Python and/or Pike are free to do what they wish. Luckily, for the crowd, those who do not even know that the coding of characters exists, all the serious actors active in text processing are working properly. jmf * By nature characters and numbers are differents. From davea at davea.name Sat Jul 13 15:02:24 2013 From: davea at davea.name (Dave Angel) Date: Sat, 13 Jul 2013 15:02:24 -0400 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> <26d5c832-eaa1-439e-af61-e2855af2cd18@googlegroups.com> <2165f7cc-b9bf-41b1-b128-d33b522046dc@googlegroups.com> <51e12295$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 07/13/2013 10:37 AM, wxjmfauth at gmail.com wrote: > > The FSR is naive and badly working. I can not force people > to understand the coding of the characters [*]. That would be very hard, since you certainly do not. > > I'm the first to recognize that Python and/or Pike are > free to do what they wish. Fortunately for us, Python (in version 3.3 and later) and Pike did it right. Some day the others may decide to do similarly. > > Luckily, for the crowd, those who do not even know that the > coding of characters exists, all the serious actors active in > text processing are working properly. Here, I'm really glad you don't know English, because if you had a decent grasp of the language, somebody might assume you knew what you were talking about. > > jmf > > * By nature characters and numbers are differents. > By nature Jmf has his own distorted reality. -- DaveA From nhodgson at iinet.net.au Sat Jul 13 19:17:41 2013 From: nhodgson at iinet.net.au (Neil Hodgson) Date: Sun, 14 Jul 2013 09:17:41 +1000 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> <26d5c832-eaa1-439e-af61-e2855af2cd18@googlegroups.com> <2165f7cc-b9bf-41b1-b128-d33b522046dc@googlegroups.com> <51e12295$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: wxjmfauth at gmail.com: > The FSR is naive and badly working. I can not force people > to understand the coding of the characters [*]. You could at least *try*. If there really was a problem with the FSR and you truly understood this problem then surely you would be able to communicate the problem to at least one person on the list. Neil From wxjmfauth at gmail.com Sun Jul 14 04:20:33 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Sun, 14 Jul 2013 01:20:33 -0700 (PDT) Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> <26d5c832-eaa1-439e-af61-e2855af2cd18@googlegroups.com> <2165f7cc-b9bf-41b1-b128-d33b522046dc@googlegroups.com> <51e12295$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: <69df4d48-4cb8-4102-b80c-247f8fd07f65@googlegroups.com> Le samedi 13 juillet 2013 21:02:24 UTC+2, Dave Angel a ?crit?: > On 07/13/2013 10:37 AM, wxjmfauth at gmail.com wrote: > > > > > > Fortunately for us, Python (in version 3.3 and later) and Pike did it > > right. Some day the others may decide to do similarly. > > > ----------- Possible but I doubt. For a very simple reason, the latin-1 block: considered and accepted today as beeing a Unicode design mistake. jmf From steve+comp.lang.python at pearwood.info Sun Jul 14 06:44:12 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 14 Jul 2013 10:44:12 GMT Subject: hex dump w/ or w/out utf-8 chars References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> <26d5c832-eaa1-439e-af61-e2855af2cd18@googlegroups.com> <2165f7cc-b9bf-41b1-b128-d33b522046dc@googlegroups.com> <51e12295$0$9505$c3e8da3$5496439d@news.astraweb.com> <69df4d48-4cb8-4102-b80c-247f8fd07f65@googlegroups.com> Message-ID: <51e280fc$0$9505$c3e8da3$5496439d@news.astraweb.com> On Sun, 14 Jul 2013 01:20:33 -0700, wxjmfauth wrote: > For a very simple reason, the latin-1 block: considered and accepted > today as beeing a Unicode design mistake. Latin-1 (also known as ISO-8859-1) was based on DEC's "Multinational Character Set", which goes back to 1983. ISO-8859-1 was first published in 1985, and was in use on Commodore computers the same year. The concept of Unicode wasn't even started until 1987, and the first draft wasn't published until the end of 1990. Unicode wasn't considered ready for production use until 1991, six years after Latin-1 was already in use in people's computers. -- Steven From wxjmfauth at gmail.com Sun Jul 14 09:44:44 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Sun, 14 Jul 2013 06:44:44 -0700 (PDT) Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: <51e280fc$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> <26d5c832-eaa1-439e-af61-e2855af2cd18@googlegroups.com> <2165f7cc-b9bf-41b1-b128-d33b522046dc@googlegroups.com> <51e12295$0$9505$c3e8da3$5496439d@news.astraweb.com> <69df4d48-4cb8-4102-b80c-247f8fd07f65@googlegroups.com> <51e280fc$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: Le dimanche 14 juillet 2013 12:44:12 UTC+2, Steven D'Aprano a ?crit?: > On Sun, 14 Jul 2013 01:20:33 -0700, wxjmfauth wrote: > > > > > For a very simple reason, the latin-1 block: considered and accepted > > > today as beeing a Unicode design mistake. > > > > Latin-1 (also known as ISO-8859-1) was based on DEC's "Multinational > > Character Set", which goes back to 1983. ISO-8859-1 was first published > > in 1985, and was in use on Commodore computers the same year. > > > > The concept of Unicode wasn't even started until 1987, and the first > > draft wasn't published until the end of 1990. Unicode wasn't considered > > ready for production use until 1991, six years after Latin-1 was already > > in use in people's computers. > > > > > > > > -- > > Steven ------ "Unicode" (in fact iso-14xxx) was not created in one night (Deus ex machina). What's count today is this: >>> timeit.repeat("a = 'hundred'; 'x' in a") [0.11785943134991479, 0.09850454944486256, 0.09761604599423179] >>> timeit.repeat("a = 'hundre?'; 'x' in a") [0.23955250303158593, 0.2195812612416752, 0.22133896997401692] >>> >>> >>> sys.getsizeof('d') 26 >>> sys.getsizeof('?') 40 >>> sys.version '3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:03:43) [MSC v.1600 32 bit (Intel)]' jmf From wxjmfauth at gmail.com Wed Jul 24 09:28:37 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Wed, 24 Jul 2013 06:28:37 -0700 (PDT) Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: References: <7ef8c0e7-7f7c-4a22-89a9-50f62c4a8064@googlegroups.com> <26d5c832-eaa1-439e-af61-e2855af2cd18@googlegroups.com> <2165f7cc-b9bf-41b1-b128-d33b522046dc@googlegroups.com> <51e12295$0$9505$c3e8da3$5496439d@news.astraweb.com> <69df4d48-4cb8-4102-b80c-247f8fd07f65@googlegroups.com> <51e280fc$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: <696caa4f-142a-4e46-88fc-090da94ced2e@googlegroups.com> I do not find the thread, where a Python core dev spoke about French, so I'm putting here. This stupid Flexible String Representation splits Unicode in chunks and one of these chunks is latin-1 (iso-8859-1). If we consider that latin-1 is unusable for 17 (seventeen) European languages based on the latin alphabet, one can not say Python is really well prepared. Most of the problems are coming from the extensive usage of diacritics in these languages. Thanks to the FSR again, working with normalized forms does not work very well. At least, there is some consistency. Now, if we consider that most of the new characters will be part of the BMP ("daily" used chars), it is hard to present Python as a modern language. It sticks more to the past and it not really prepared for the future, the acceptance of new chars like ? or the new Turkish lira sign ((U+20BA). >>> sys.getsizeof('?') 40 >>> sys.getsizeof('0') 26 14 bytes to encode a non-latin-1 char is not so bad. jmf From ferdy.blatsco at gmail.com Mon Jul 8 13:53:18 2013 From: ferdy.blatsco at gmail.com (ferdy.blatsco at gmail.com) Date: Mon, 8 Jul 2013 10:53:18 -0700 (PDT) Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: References: Message-ID: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> Hi Steven, thank you for your reply... I really needed another python guru which is also an English teacher! Sorry if English is not my mother tongue... "uncorrect" instead of "incorrect" (I misapplied the "similarity principle" like "unpleasant...>...uncorrect"). Apart from these trifles, you said: >> All characters are UTF-8, characters. "a" is a UTF-8 character. So is "?". Not using python 3, for me (a programmer which was present at the beginning of computer science, badly interacting with many languages from assembler to Fortran and from c to Pascal and so on) it was an hard job to arrange the abrupt transition from characters only equal to bytes to some special characters defined with 2, 3 bytes and even more. I should have preferred another solution... but i'm not Guido....! I said: > in the first version the utf-8 conversion to hex was shown horizontally And you replied: >> Oh! We're supposed to read the output *downwards*! You are correct, but I was only referring to "special characters"... My main concern was compactness of output and besides that every group of bytes used for defining "special characters" is well represented with high nibble in the range outside ascii 0-127. Your following observations are connected more or less to the above point and sorry if the interpretation of output... sucks! I think that, for the interested user, all the question is of minor importance. Only another point is relevant for me: >> The loop variable just gets reset once it reaches the top of the loop >> again. Apart your kind observation (... "hideously ugly to read") referring to my code snippet incrementing the loop variable... you are correct. I will never make the same mistake! Bye, Blatt. From rosuav at gmail.com Mon Jul 8 14:07:22 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 9 Jul 2013 04:07:22 +1000 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> Message-ID: On Tue, Jul 9, 2013 at 3:53 AM, wrote: >>> All characters are UTF-8, characters. "a" is a UTF-8 character. So is "?". > Not using python 3, for me (a programmer which was present at the beginning of > computer science, badly interacting with many languages from assembler to > Fortran and from c to Pascal and so on) it was an hard job to arrange the > abrupt transition from characters only equal to bytes to some special > characters defined with 2, 3 bytes and even more. Even back then, bytes and characters were different. 'A' is a character, 0x41 is a byte. And they correspond 1:1 if and only if you know that your characters are represented in ASCII. Other encodings (eg EBCDIC) mapped things differently. The only difference now is that more people are becoming aware that there are more than 256 characters in the world. Like Magic 2014 and its treatment of Slivers, at some point you're going to have to master the difference between bytes and characters, or else be eternally hacking around stuff in your code, so now is as good a time as any. ChrisA From davea at davea.name Mon Jul 8 16:56:54 2013 From: davea at davea.name (Dave Angel) Date: Mon, 08 Jul 2013 16:56:54 -0400 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> Message-ID: On 07/08/2013 01:53 PM, ferdy.blatsco at gmail.com wrote: > Hi Steven, > > thank you for your reply... I really needed another python guru which > is also an English teacher! Sorry if English is not my mother tongue... > "uncorrect" instead of "incorrect" (I misapplied the "similarity > principle" like "unpleasant...>...uncorrect"). > > Apart from these trifles, you said: >>> All characters are UTF-8, characters. "a" is a UTF-8 character. So is "?". > Not using python 3, for me (a programmer which was present at the beginning of > computer science, badly interacting with many languages from assembler to > Fortran and from c to Pascal and so on) it was an hard job to arrange the > abrupt transition from characters only equal to bytes to some special > characters defined with 2, 3 bytes and even more. Characters do not have a width. They are Unicode code points, an abstraction. It's only when you encode them in byte strings that a code point takes on any specific width. And some encodings go to one-byte strings (and get errors for characters that don't match), some go to two-bytes each, some variable, etc. > I should have preferred another solution... but i'm not Guido....! But Unicode has nothing to do with Guido, and it has existed for about 25 years (if I recall correctly). It's only that Python 3 is finally embracing it, and making it the default type for characters, as it should be. As far as I'm concerned, the only reason it shouldn't have been done long ago was that programs were trying to fit on 640k DOS machines. Even before Unicode, there were multi-byte encodings around (eg. Microsoft's MBCS), and each was thoroughly incompatible with all the others. And the problem with one-byte encodings is that if you need to use a Greek currency symbol in a document that's mostly Norwegian (or some such combination of characters), there might not be ANY valid way to do it within a single "character set." Python 2 supports all the same Unicode features as 3; it's just that it defaults to byte strings. So it's HARDER to get it right. Except for special purpose programs like a file dumper, it's usually unnecessary for a Python 3 programmer to deal with individual bytes from a byte string. Text files are a bunch of bytes, and somebody has to interpret them as characters. If you let open() handle it, and if you give it the correct encoding, it just works. Internally, all strings are Unicode, and you don't care where they came from, or what human language they may have characters from. You can combine strings from multiple places, without much worry that they might interfere. Windows NT/2000/XP/Vista/7 has used Unicode for its file system (NTFS) from the beginning (approx 1992), and has had Unicode versions of each of its API's for nearly as long. I appreciate you've been around a long time, and worked in a lot of languages. I've programmed professionally in at least 35 languages since 1967. But we've come a long way from the 6bit characters I used in 1968. At that time, we packed them 10 characters to each word. -- DaveA From python at mrabarnett.plus.com Mon Jul 8 17:38:04 2013 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 08 Jul 2013 22:38:04 +0100 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> Message-ID: <51DB313C.4040701@mrabarnett.plus.com> On 08/07/2013 21:56, Dave Angel wrote: > On 07/08/2013 01:53 PM, ferdy.blatsco at gmail.com wrote: >> Hi Steven, >> >> thank you for your reply... I really needed another python guru which >> is also an English teacher! Sorry if English is not my mother tongue... >> "uncorrect" instead of "incorrect" (I misapplied the "similarity >> principle" like "unpleasant...>...uncorrect"). >> >> Apart from these trifles, you said: >>>> All characters are UTF-8, characters. "a" is a UTF-8 character. So is "?". >> Not using python 3, for me (a programmer which was present at the beginning of >> computer science, badly interacting with many languages from assembler to >> Fortran and from c to Pascal and so on) it was an hard job to arrange the >> abrupt transition from characters only equal to bytes to some special >> characters defined with 2, 3 bytes and even more. > > Characters do not have a width. [snip] It depends what you mean by "width"! :-) Try this (Python 3): >>> print("A\N{FULLWIDTH LATIN CAPITAL LETTER A}") A? From joshua.landau.ws at gmail.com Mon Jul 8 18:02:57 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Mon, 8 Jul 2013 23:02:57 +0100 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: <51DB313C.4040701@mrabarnett.plus.com> References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> <51DB313C.4040701@mrabarnett.plus.com> Message-ID: On 8 July 2013 22:38, MRAB wrote: > On 08/07/2013 21:56, Dave Angel wrote: >> Characters do not have a width. > > [snip] > > It depends what you mean by "width"! :-) > > Try this (Python 3): > >>>> print("A\N{FULLWIDTH LATIN CAPITAL LETTER A}") > A? Serious question: How would one find the width of a character by that definition? From python at mrabarnett.plus.com Mon Jul 8 19:32:00 2013 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 09 Jul 2013 00:32:00 +0100 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> <51DB313C.4040701@mrabarnett.plus.com> Message-ID: <51DB4BF0.7060207@mrabarnett.plus.com> On 08/07/2013 23:02, Joshua Landau wrote: > On 8 July 2013 22:38, MRAB wrote: >> On 08/07/2013 21:56, Dave Angel wrote: >>> Characters do not have a width. >> >> [snip] >> >> It depends what you mean by "width"! :-) >> >> Try this (Python 3): >> >>>>> print("A\N{FULLWIDTH LATIN CAPITAL LETTER A}") >> A? > > Serious question: How would one find the width of a character by that > definition? > >>> import unicodedata >>> unicodedata.east_asian_width("A") 'Na' >>> unicodedata.east_asian_width("\N{FULLWIDTH LATIN CAPITAL LETTER A}") 'F' The possible widths are: N = Neutral A = Ambiguous H = Halfwidth W = Wide F = Fullwidth Na = Narrow All you then need to do is find out what those actually mean... From steve+comp.lang.python at pearwood.info Tue Jul 9 02:46:40 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 09 Jul 2013 06:46:40 GMT Subject: hex dump w/ or w/out utf-8 chars References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> <51DB313C.4040701@mrabarnett.plus.com> Message-ID: <51dbb1d0$0$6512$c3e8da3$5496439d@news.astraweb.com> On Tue, 09 Jul 2013 00:32:00 +0100, MRAB wrote: > On 08/07/2013 23:02, Joshua Landau wrote: >> On 8 July 2013 22:38, MRAB wrote: >>> On 08/07/2013 21:56, Dave Angel wrote: >>>> Characters do not have a width. >>> >>> [snip] >>> >>> It depends what you mean by "width"! :-) >>> >>> Try this (Python 3): >>> >>>>>> print("A\N{FULLWIDTH LATIN CAPITAL LETTER A}") >>> A? >> >> Serious question: How would one find the width of a character by that >> definition? >> > >>> import unicodedata > >>> unicodedata.east_asian_width("A") > 'Na' > >>> unicodedata.east_asian_width("\N{FULLWIDTH LATIN CAPITAL LETTER > >>> A}") > 'F' > > The possible widths are: > > N = Neutral > A = Ambiguous > H = Halfwidth > W = Wide > F = Fullwidth > Na = Narrow > > All you then need to do is find out what those actually mean... In some East-Asian encodings, there are code-points for Latin characters in two forms: "half-width" and "full-width". The half-width form took up a single fixed-width column; the full-width forms took up two fixed-width columns, so they would line up nicely in columns with Asian characters. See also: http://www.unicode.org/reports/tr11/ and search Wikipedia for "full-width" and "half-width". -- Steven From rosuav at gmail.com Mon Jul 8 17:49:45 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 9 Jul 2013 07:49:45 +1000 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> Message-ID: On Tue, Jul 9, 2013 at 6:56 AM, Dave Angel wrote: > But Unicode has nothing to do with Guido, and it has existed for about 25 > years (if I recall correctly). Depends how you measure. According to [1], the work kinda began back then (25 years ago being 1988), but it wasn't till 1991/92 that the spec was published. Also, the full Unicode range with multiple planes came about in 1996, with Unicode 2.0, so that could also be considered the beginning of Unicode. But that still means it's nearly old enough to drink, so programmers ought to be aware of it. [1] http://en.wikipedia.org/wiki/Unicode#History ChrisA From davea at davea.name Mon Jul 8 18:45:46 2013 From: davea at davea.name (Dave Angel) Date: Mon, 08 Jul 2013 18:45:46 -0400 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> Message-ID: On 07/08/2013 05:49 PM, Chris Angelico wrote: > On Tue, Jul 9, 2013 at 6:56 AM, Dave Angel wrote: >> But Unicode has nothing to do with Guido, and it has existed for about 25 >> years (if I recall correctly). > > Depends how you measure. According to [1], the work kinda began back > then (25 years ago being 1988), but it wasn't till 1991/92 that the > spec was published. Also, the full Unicode range with multiple planes > came about in 1996, with Unicode 2.0, so that could also be considered > the beginning of Unicode. But that still means it's nearly old enough > to drink, so programmers ought to be aware of it. > Well, then I'm glad I stuck the qualifier on it. I remember where I was working, and that company folded in 1992. I was working on NT long before its official release in 1993, and it used Unicode, even if the spec was sliding along. I'm sure I got unofficial versions of things through Microsoft, at the time. -- DaveA From rosuav at gmail.com Mon Jul 8 18:51:22 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 9 Jul 2013 08:51:22 +1000 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> Message-ID: On Tue, Jul 9, 2013 at 8:45 AM, Dave Angel wrote: > On 07/08/2013 05:49 PM, Chris Angelico wrote: >> >> On Tue, Jul 9, 2013 at 6:56 AM, Dave Angel wrote: >>> >>> But Unicode has nothing to do with Guido, and it has existed for about 25 >>> years (if I recall correctly). >> >> >> Depends how you measure. According to [1], the work kinda began back >> then (25 years ago being 1988), but it wasn't till 1991/92 that the >> spec was published. Also, the full Unicode range with multiple planes >> came about in 1996, with Unicode 2.0, so that could also be considered >> the beginning of Unicode. But that still means it's nearly old enough >> to drink, so programmers ought to be aware of it. >> > > Well, then I'm glad I stuck the qualifier on it. I remember where I was > working, and that company folded in 1992. I was working on NT long before > its official release in 1993, and it used Unicode, even if the spec was > sliding along. I'm sure I got unofficial versions of things through > Microsoft, at the time. No doubt! Of course, this list is good at dealing with the hard facts and making sure the archives are accurate, but that doesn't change your memory. Anyway, your fundamental point isn't materially affected by whether Unicode is 17 or 25 years old. It's been around plenty long enough by now, we should use it. Same with IPv6, too... ChrisA From steve+comp.lang.python at pearwood.info Tue Jul 9 02:53:39 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 09 Jul 2013 06:53:39 GMT Subject: hex dump w/ or w/out utf-8 chars References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> Message-ID: <51dbb372$0$6512$c3e8da3$5496439d@news.astraweb.com> On Tue, 09 Jul 2013 07:49:45 +1000, Chris Angelico wrote: > On Tue, Jul 9, 2013 at 6:56 AM, Dave Angel wrote: >> But Unicode has nothing to do with Guido, and it has existed for about >> 25 years (if I recall correctly). > > Depends how you measure. According to [1], the work kinda began back > then (25 years ago being 1988), but it wasn't till 1991/92 that the spec > was published. Also, the full Unicode range with multiple planes came > about in 1996, with Unicode 2.0, so that could also be considered the > beginning of Unicode. But that still means it's nearly old enough to > drink, so programmers ought to be aware of it. Yes, yes, a thousand times yes. It's really not that hard to get the basics of Unicode. "When I discovered that the popular web development tool PHP has almost complete ignorance of character encoding issues, blithely using 8 bits for characters, making it darn near impossible to develop good international web applications, I thought, enough is enough. So I have an announcement to make: if you are a programmer working in 2003 and you don't know the basics of characters, character sets, encodings, and Unicode, and I catch you, I'm going to punish you by making you peel onions for 6 months in a submarine. I swear I will." http://www.joelonsoftware.com/articles/Unicode.html Also: http://nedbatchelder.com/text/unipain.html To start with, if you're writing code for Python 2.x, and not using u'' for strings, then you're making a rod for your own back. Do yourself a favour and get into the habit of always using u'' strings in Python 2. I'll-start-taking-my-own-advice-next-week-I-promise-ly yrs, -- Steven From steve+comp.lang.python at pearwood.info Tue Jul 9 03:00:02 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 09 Jul 2013 07:00:02 GMT Subject: hex dump w/ or w/out utf-8 chars References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> Message-ID: <51dbb4f2$0$6512$c3e8da3$5496439d@news.astraweb.com> On Mon, 08 Jul 2013 10:53:18 -0700, ferdy.blatsco wrote: > Not using python 3, for me (a programmer which was present at the > beginning of computer science, badly interacting with many languages > from assembler to Fortran and from c to Pascal and so on) it was an hard > job to arrange the abrupt transition from characters only equal to bytes Characters have *never* been equal to bytes. Not even Perl treats the character 'A' as equal to the byte 0x0A: if (0x0A eq 'A') {print "Equal\n";} else {print "Unequal\n";} will print Unequal, even if you replace "eq" with "==". Nor does Perl consider the character 'A' equal to 65. If you have learned to think of characters being equal to bytes, you have learned wrong. > to some special characters defined with 2, 3 bytes and even more. I > should have preferred another solution... but i'm not Guido....! What's a special character? To an Italian, the characters J, K, W, X and Y are "special characters" which do not exist in the ordinary alphabet. To a German, they are not special, but S is special because you write SS as ?, but only in lowercase. To a mathematician, ? is just as ordinary as it would be to a Greek; but the mathematician probably won't recognise ? unless she actually is Greek, even though they are the same letter. To an American electrician, ? is an ordinary character, but ? isn't. To anyone working with angles, or temperatures, the degree symbol ? is an ordinary character, but the radian symbol is not. (I can't even find it.) The English have forgotten that W used to be a ligature for VV, and consider it a single ordinary character. But the ligature ? is considered an old-fashioned way of writing AE. But to Danes and Norwegians, ? is an ordinary letter, as distinct from AE as TH is from ?. (Which English used to have.) And so on... I don't know what a special character is, unless it is the ASCII NUL character, since that terminates C strings. -- Steven From wxjmfauth at gmail.com Tue Jul 9 05:34:08 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Tue, 9 Jul 2013 02:34:08 -0700 (PDT) Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: <51dbb4f2$0$6512$c3e8da3$5496439d@news.astraweb.com> References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> <51dbb4f2$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: <60b625d1-de98-4307-8e67-ada361111954@googlegroups.com> Le mardi 9 juillet 2013 09:00:02 UTC+2, Steven D'Aprano a ?crit?: > On Mon, 08 Jul 2013 10:53:18 -0700, ferdy.blatsco wrote: > > > > > Not using python 3, for me (a programmer which was present at the > > > beginning of computer science, badly interacting with many languages > > > from assembler to Fortran and from c to Pascal and so on) it was an hard > > > job to arrange the abrupt transition from characters only equal to bytes > > > > Characters have *never* been equal to bytes. Not even Perl treats the > > character 'A' as equal to the byte 0x0A: > > > > if (0x0A eq 'A') {print "Equal\n";} > > else {print "Unequal\n";} > > > > will print Unequal, even if you replace "eq" with "==". Nor does Perl > > consider the character 'A' equal to 65. > > > > If you have learned to think of characters being equal to bytes, you have > > learned wrong. > > > > > > > to some special characters defined with 2, 3 bytes and even more. I > > > should have preferred another solution... but i'm not Guido....! > > > > What's a special character? > > > > To an Italian, the characters J, K, W, X and Y are "special characters" > > which do not exist in the ordinary alphabet. To a German, they are not > > special, but S is special because you write SS as ?, but only in > > lowercase. > > > > To a mathematician, ? is just as ordinary as it would be to a Greek; but > > the mathematician probably won't recognise ? unless she actually is > > Greek, even though they are the same letter. > > > > To an American electrician, ? is an ordinary character, but ? isn't. > > > > To anyone working with angles, or temperatures, the degree symbol ? is an > > ordinary character, but the radian symbol is not. (I can't even find it.) > > > > The English have forgotten that W used to be a ligature for VV, and > > consider it a single ordinary character. But the ligature ? is considered > > an old-fashioned way of writing AE. > > > > But to Danes and Norwegians, ? is an ordinary letter, as distinct from AE > > as TH is from ?. (Which English used to have.) And so on... > > > > I don't know what a special character is, unless it is the ASCII NUL > > character, since that terminates C strings. -------- The concept of "special characters" does not exist. However, the definition of a "character" is a problem per se (character, glyph, grapheme, ...). You are confusing Unicode, typography and linguistic. There is no symbole for radian because mathematically radian is a pure number, a unitless number. You can hower sepecify a = ... in radian (rad). Note the difference between SS and ? 'FRANZ-JOSEF-STRAUSS-STRA?E' jmf From kwpolska at gmail.com Tue Jul 9 06:15:29 2013 From: kwpolska at gmail.com (=?UTF-8?B?Q2hyaXMg4oCcS3dwb2xza2HigJ0gV2Fycmljaw==?=) Date: Tue, 9 Jul 2013 12:15:29 +0200 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: <60b625d1-de98-4307-8e67-ada361111954@googlegroups.com> References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> <51dbb4f2$0$6512$c3e8da3$5496439d@news.astraweb.com> <60b625d1-de98-4307-8e67-ada361111954@googlegroups.com> Message-ID: On Tue, Jul 9, 2013 at 11:34 AM, wrote: > Note the difference between SS and ? > 'FRANZ-JOSEF-STRAUSS-STRA?E' This is a capital Eszett. Which just happens not to exist in German. Germans do not use this character, it is not available on German keyboards, and the German spelling rules have you replace ? with SS. And, surprise surprise, STRASSE is the example the Council for German Orthography used ([0] page 29, ?25 E3). [0]: http://www.neue-rechtschreibung.de/regelwerk.pdf -- Kwpolska | GPG KEY: 5EAAEA16 stop html mail | always bottom-post http://asciiribbon.org | http://caliburn.nl/topposting.html From steve+comp.lang.python at pearwood.info Tue Jul 9 12:32:40 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 09 Jul 2013 16:32:40 GMT Subject: hex dump w/ or w/out utf-8 chars References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> <51dbb4f2$0$6512$c3e8da3$5496439d@news.astraweb.com> <60b625d1-de98-4307-8e67-ada361111954@googlegroups.com> Message-ID: <51dc3b28$0$9505$c3e8da3$5496439d@news.astraweb.com> On Tue, 09 Jul 2013 12:15:29 +0200, Chris ?Kwpolska? Warrick wrote: > On Tue, Jul 9, 2013 at 11:34 AM, wrote: >> Note the difference between SS and ? 'FRANZ-JOSEF-STRAUSS-STRA?E' > > This is a capital Eszett. Which just happens not to exist in German. > Germans do not use this character, it is not available on German > keyboards, and the German spelling rules have you replace ? with SS. > And, surprise surprise, STRASSE is the example the Council for German > Orthography used ([0] page 29, ?25 E3). > > [0]: http://www.neue-rechtschreibung.de/regelwerk.pdf Only half-right. Uppercase Eszett has been used in Germany going back at least to 1879, and appears to be gaining popularity. In 2010 the use of uppercase ? apparently became mandatory for geographical place names when written in uppercase in official documentation. http://opentype.info/blog/2011/01/24/capital-sharp-s/ http://en.wikipedia.org/wiki/Capital_? Font support is still quite poor, but at least half a dozen Windows 7 fonts provide it, and at least one Mac font. -- Steven From wxjmfauth at gmail.com Wed Jul 10 04:52:19 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Wed, 10 Jul 2013 01:52:19 -0700 (PDT) Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: <51dc3b28$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> <51dbb4f2$0$6512$c3e8da3$5496439d@news.astraweb.com> <60b625d1-de98-4307-8e67-ada361111954@googlegroups.com> <51dc3b28$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: For those who are interested. The official proposal request for the encoding of the Latin uppercase letter Sharp S in ISO/IEC 10646; DIN (The German Institute for Standardization) proposal is available on the web. A pdf with the rationale. I do not remember from where I got it, probably from a German web site. Fonts: I'm observing the inclusion of this glyph since years. More and more fonts are supporting it. Available in many fonts, it is suprisingly not available in Cambria (at least the version I'm using). STIX does not includes it, it has been requested. Ditto, for the Latin Modern, the default bundle of fonts for the Unicode TeX engines. Last but not least, Python. Thanks to the Flexible String Representation, it is not necessary to mention the disastrous, erratic behaviour of Python, when processing text containing it. It's more than clear, a serious user willing to process the contain of 'DER GRO?E DUDEN' (a reference German dictionary) will be better served by using something else. The irony is that this Flexible String Representation has been created by a German. jmf From joshua at landau.ws Fri Jul 12 18:01:47 2013 From: joshua at landau.ws (Joshua Landau) Date: Fri, 12 Jul 2013 23:01:47 +0100 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: <60b625d1-de98-4307-8e67-ada361111954@googlegroups.com> References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> <51dbb4f2$0$6512$c3e8da3$5496439d@news.astraweb.com> <60b625d1-de98-4307-8e67-ada361111954@googlegroups.com> Message-ID: On 9 July 2013 10:34, wrote: > There is no symbole for radian because mathematically > radian is a pure number, a unitless number. You can > hower sepecify a = ... in radian (rad). > Isn't a superscript "c" the symbol for radians? -------------- next part -------------- An HTML attachment was scrubbed... URL: From timr at probo.com Fri Jul 12 23:42:48 2013 From: timr at probo.com (Tim Roberts) Date: Fri, 12 Jul 2013 20:42:48 -0700 Subject: hex dump w/ or w/out utf-8 chars References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> <51dbb4f2$0$6512$c3e8da3$5496439d@news.astraweb.com> <60b625d1-de98-4307-8e67-ada361111954@googlegroups.com> Message-ID: <04j1u8lr5tbqiaiic7gg9hhh7vkvvro9c5@4ax.com> Joshua Landau wrote: > >Isn't a superscript "c" the symbol for radians? That's very rarely used. More common is "rad". The problem with a superscript "c" is that it looks too much like a degree symbol. -- Tim Roberts, timr at probo.com Providenza & Boekelheide, Inc. From steve+comp.lang.python at pearwood.info Sat Jul 13 00:51:27 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 13 Jul 2013 04:51:27 GMT Subject: hex dump w/ or w/out utf-8 chars References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> <51dbb4f2$0$6512$c3e8da3$5496439d@news.astraweb.com> <60b625d1-de98-4307-8e67-ada361111954@googlegroups.com> Message-ID: <51e0dcce$0$9505$c3e8da3$5496439d@news.astraweb.com> On Fri, 12 Jul 2013 23:01:47 +0100, Joshua Landau wrote: > Isn't a superscript "c" the symbol for radians? Only in the sense that a superscript "o" is the symbol for degrees. Semantically, both degree-sign and radian-sign are different "things" than merely an o or c in superscript. Nevertheless, in mathematics at least, it is normal to leave out the radian sign when talking about angles. By default, "1.2" means "1.2 radians", not "1.2 degrees". -- Steven From neilc at norwich.edu Tue Jul 9 08:22:34 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 9 Jul 2013 12:22:34 GMT Subject: hex dump w/ or w/out utf-8 chars References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> Message-ID: On 2013-07-08, Dave Angel wrote: > I appreciate you've been around a long time, and worked in a > lot of languages. I've programmed professionally in at least > 35 languages since 1967. But we've come a long way from the > 6bit characters I used in 1968. At that time, we packed them > 10 characters to each word. One of the first Python project I undertook was a program to dump the ZSCII strings from Infocom game files. They are mostly packed one character per 5 bits, with escapes to (I had to recheck the Z-machine spec) latin-1. Oh, those clever implementors: thwarting hexdumping cheaters and cramming their games onto microcomputers with one blow. -- Neil Cerutti From davea at davea.name Tue Jul 9 08:54:28 2013 From: davea at davea.name (Dave Angel) Date: Tue, 09 Jul 2013 08:54:28 -0400 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> Message-ID: On 07/09/2013 08:22 AM, Neil Cerutti wrote: > On 2013-07-08, Dave Angel wrote: >> I appreciate you've been around a long time, and worked in a >> lot of languages. I've programmed professionally in at least >> 35 languages since 1967. But we've come a long way from the >> 6bit characters I used in 1968. At that time, we packed them >> 10 characters to each word. > > One of the first Python project I undertook was a program to dump > the ZSCII strings from Infocom game files. They are mostly packed > one character per 5 bits, with escapes to (I had to recheck the > Z-machine spec) latin-1. Oh, those clever implementors: thwarting > hexdumping cheaters and cramming their games onto microcomputers > with one blow. > In 1973 I played with encoding some data that came over the public airwaves (I never learned the specific radio technology, probably used sidebands of FM stations). The data was encoded, with most characters taking 5 bits, and the decoded stream was like a ticker-tape. With some hardware and the right software, you could track Wall Street in real time. (Or maybe it had the usual 15 minute delay). Obviously, they didn't publish the spec any place. But some others had the beginnings of a decoder, and I expanded on that. We never did anything with it, it was just an interesting challenge. -- DaveA From neilc at norwich.edu Tue Jul 9 09:00:48 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 9 Jul 2013 13:00:48 GMT Subject: hex dump w/ or w/out utf-8 chars References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> Message-ID: On 2013-07-09, Dave Angel wrote: >> One of the first Python project I undertook was a program to >> dump the ZSCII strings from Infocom game files. They are >> mostly packed one character per 5 bits, with escapes to (I had >> to recheck the Z-machine spec) latin-1. Oh, those clever >> implementors: thwarting hexdumping cheaters and cramming their >> games onto microcomputers with one blow. > > In 1973 I played with encoding some data that came over the > public airwaves (I never learned the specific radio technology, > probably used sidebands of FM stations). The data was encoded, > with most characters taking 5 bits, and the decoded stream was > like a ticker-tape. With some hardware and the right software, > you could track Wall Street in real time. (Or maybe it had the > usual 15 minute delay). > > Obviously, they didn't publish the spec any place. But some > others had the beginnings of a decoder, and I expanded on that. > We never did anything with it, it was just an interesting > challenge. Interestingly similar scheme. It wonder if 5-bit chars was a common compression scheme. The Z-machine spec was never officially published either. I believe a "task force" reverse engineered it sometime in the 90's. -- Neil Cerutti From skip at pobox.com Tue Jul 9 09:18:34 2013 From: skip at pobox.com (Skip Montanaro) Date: Tue, 9 Jul 2013 08:18:34 -0500 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> Message-ID: > It wonder if 5-bit chars was a > common compression scheme. http://en.wikipedia.org/wiki/List_of_binary_codes Baudot was pretty common, as I recall, though ASCII and EBCDIC ruled by the time I started punching cards. Skip From davea at davea.name Tue Jul 9 09:23:55 2013 From: davea at davea.name (Dave Angel) Date: Tue, 09 Jul 2013 09:23:55 -0400 Subject: hex dump w/ or w/out utf-8 chars In-Reply-To: References: <7b6fc645-8bf3-4681-821c-38fb1fa1d191@googlegroups.com> Message-ID: On 07/09/2013 09:00 AM, Neil Cerutti wrote: > Interestingly similar scheme. It wonder if 5-bit chars was a > common compression scheme. The Z-machine spec was never > officially published either. I believe a "task force" reverse > engineered it sometime in the 90's. > Baudot was 5 bits. It used shift-codes to get upper case and digits, if I recall. And ASCII was 7 bits so there could be one more for parity. -- DaveA From skunkwerk at gmail.com Mon Jul 8 01:27:13 2013 From: skunkwerk at gmail.com (skunkwerk) Date: Sun, 7 Jul 2013 22:27:13 -0700 (PDT) Subject: UnpicklingError: NEWOBJ class argument isn't a type object Message-ID: <18dd6d34-5afa-4324-bd2f-f5561413b156@googlegroups.com> Hi, I'm using a custom pickler that replaces any un-pickleable objects (such as sockets or files) with a string representation of them, based on the code from Shane Hathaway here: http://stackoverflow.com/questions/4080688/python-pickling-a-dict-with-some-unpicklable-items It works most of the time, but when I try to unpickle a Django HttpResponse, I get the following error: UnpicklingError: NEWOBJ class argument isn't a type object I have no clue what the error actually means. If it pickles okay, why should it not be able to unpickle? Any ideas? thanks for the help, imran Here is my code: from cPickle import Pickler, Unpickler, UnpicklingError class FilteredObject: def __init__(self, about): self.about = about def __repr__(self): return 'FilteredObject(%s)' % repr(self.about) class MyPickler(object): def __init__(self, file, protocol=2): pickler = Pickler(file, protocol) pickler.persistent_id = self.persistent_id self.dump = pickler.dump self.clear_memo = pickler.clear_memo def persistent_id(self, obj): if not hasattr(obj, '__getstate__') and not isinstance(obj, (basestring, bool, int, long, float, complex, tuple, list, set, dict)): return ["filtered:%s" % str(obj)] else: return None class MyUnpickler(object): def __init__(self, file): unpickler = Unpickler(file) unpickler.persistent_load = self.persistent_load self.load = unpickler.load self.noload = unpickler.noload def persistent_load(self, obj_id): if obj_id[0].startswith('filtered:'): return FilteredObject(obj_id[0][9:]) else: raise UnpicklingError('Invalid persistent id') ###### serialize to file f = open('test.txt','wb') p = MyPickler(f) p.dump(data) f.close() ###### unserialize from file f = open('test.txt','rb') pickled_data = f.read() f.seek(0) u = MyUnpickler(f) data = u.load() From rosuav at gmail.com Mon Jul 8 01:57:30 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 8 Jul 2013 15:57:30 +1000 Subject: UnpicklingError: NEWOBJ class argument isn't a type object In-Reply-To: <18dd6d34-5afa-4324-bd2f-f5561413b156@googlegroups.com> References: <18dd6d34-5afa-4324-bd2f-f5561413b156@googlegroups.com> Message-ID: On Mon, Jul 8, 2013 at 3:27 PM, skunkwerk wrote: > I'm using a custom pickler that replaces any un-pickleable objects (such as sockets or files) with a string representation of them... > > If it pickles okay, why should it not be able to unpickle? Any ideas? Generally, the reason something won't pickle is because it won't be able to be unpickled. So arbitrarily creating a string might allow the pickle operation to continue, but might well prevent unpickling still. I don't know, you'd have to play around with it. ChrisA From dieter at handshake.de Mon Jul 8 02:53:42 2013 From: dieter at handshake.de (dieter) Date: Mon, 08 Jul 2013 08:53:42 +0200 Subject: UnpicklingError: NEWOBJ class argument isn't a type object References: <18dd6d34-5afa-4324-bd2f-f5561413b156@googlegroups.com> Message-ID: <87hag5wnih.fsf@handshake.de> skunkwerk writes: > Hi, > I'm using a custom pickler that replaces any un-pickleable objects (such as sockets or files) with a string representation of them, based on the code from Shane Hathaway here: > http://stackoverflow.com/questions/4080688/python-pickling-a-dict-with-some-unpicklable-items > > It works most of the time, but when I try to unpickle a Django HttpResponse, I get the following error: > UnpicklingError: NEWOBJ class argument isn't a type object > > I have no clue what the error actually means. The pickling protocol uses a form of bytecode which is executed during the unpickling to reconstruct the python objects based on their state found in the pickle alongside the bytecode. "NEWOBJ" is executed in response to such a bytecode operation. It expects to get a type as a parameter but in your case, it gets something else. > If it pickles okay, why should it not be able to unpickle? Any ideas? It is by principle impossible for the pickler to garantee that an unpickler will later succeed: the pickler does not know which classes/types are available for the unpickler. In your special case, the pickler could probably detect that unpickling will fail - but when an aim cannot be achieved completely this may provide motivation to abandon it as a whole - and not put much effort into a partial achievement. I have seen many cases where pickling succeeded but unpickling failed and in principle the pickler could have already predicted the failure (under the assumption that the unpickler sees the same classes/types as the pickler). If it is important for you to get Django HttpResponses successfully unpickled then you likely need to guide their pickling process better. Maybe (as an alternative), you can extract the relevant information from the "HttpResponse" and pickle that instead of the response itself? From __peter__ at web.de Mon Jul 8 03:45:55 2013 From: __peter__ at web.de (Peter Otten) Date: Mon, 08 Jul 2013 09:45:55 +0200 Subject: UnpicklingError: NEWOBJ class argument isn't a type object References: <18dd6d34-5afa-4324-bd2f-f5561413b156@googlegroups.com> Message-ID: skunkwerk wrote: > Hi, > I'm using a custom pickler that replaces any un-pickleable objects (such > as sockets or files) with a string representation of them, based on the > code from Shane Hathaway here: > http://stackoverflow.com/questions/4080688/python-pickling-a-dict-with- some-unpicklable-items > > It works most of the time, but when I try to unpickle a Django > HttpResponse, I get the following error: UnpicklingError: NEWOBJ class > argument isn't a type object > > I have no clue what the error actually means. If it pickles okay, why > should it not be able to unpickle? Any ideas? A simple way to provoke the error is to rebind the name referring to the class of the pickled object: >>> import cPickle >>> class A(object): pass ... >>> p = cPickle.dumps(A(), -1) >>> cPickle.loads(p) <__main__.A object at 0x7fce7bb58c50> >>> A = 42 >>> cPickle.loads(p) Traceback (most recent call last): File "", line 1, in cPickle.UnpicklingError: NEWOBJ class argument isn't a type object You may be doing something to that effect. From skunkwerk at gmail.com Mon Jul 8 19:38:17 2013 From: skunkwerk at gmail.com (skunkwerk) Date: Mon, 8 Jul 2013 16:38:17 -0700 (PDT) Subject: UnpicklingError: NEWOBJ class argument isn't a type object In-Reply-To: References: <18dd6d34-5afa-4324-bd2f-f5561413b156@googlegroups.com> Message-ID: On Monday, July 8, 2013 12:45:55 AM UTC-7, Peter Otten wrote: > skunkwerk wrote: > > > > > Hi, > > > I'm using a custom pickler that replaces any un-pickleable objects (such > > > as sockets or files) with a string representation of them, based on the > > > code from Shane Hathaway here: > > > http://stackoverflow.com/questions/4080688/python-pickling-a-dict-with- > > some-unpicklable-items > > > > > > It works most of the time, but when I try to unpickle a Django > > > HttpResponse, I get the following error: UnpicklingError: NEWOBJ class > > > argument isn't a type object > > > > > > I have no clue what the error actually means. If it pickles okay, why > > > should it not be able to unpickle? Any ideas? > > > > A simple way to provoke the error is to rebind the name referring to the > > class of the pickled object: > > > > >>> import cPickle > > >>> class A(object): pass > > ... > > >>> p = cPickle.dumps(A(), -1) > > >>> cPickle.loads(p) > > <__main__.A object at 0x7fce7bb58c50> > > >>> A = 42 > > >>> cPickle.loads(p) > > Traceback (most recent call last): > > File "", line 1, in > > cPickle.UnpicklingError: NEWOBJ class argument isn't a type object > > > > You may be doing something to that effect. Hey Peter, I tried unpickling even from another file with no other code in it, but came up with the same error - so I don't think it's a rebinding issue. But I got the error to disappear when I removed the "hasattr(obj, '__getstate__')" from this line of code in the persistent_id function: if not hasattr(obj, '__getstate__') and isinstance(obj,(basestring, bool, int, long, float, complex, tuple, list, set, dict)): return ["filtered:%s" % type(obj)] When I do that, I get a few more FilteredObjects in the result, for things like: I figured these classes must have __getstate__ methods which leads to them being pickled without a persistent_id (it turns out they actually have __repr__ methods). So these classes get pickled fine, but run into problems when trying to unpickle them. I understand why ImportErrors would happen if the necessary modules haven't been loaded, but this NEWOBJ error is still kind of mystifying. I guess I just won't pickle any classes for now, if unpickling them is going to be dicey. thanks for the help guys, imran From __peter__ at web.de Tue Jul 9 03:24:41 2013 From: __peter__ at web.de (Peter Otten) Date: Tue, 09 Jul 2013 09:24:41 +0200 Subject: UnpicklingError: NEWOBJ class argument isn't a type object References: <18dd6d34-5afa-4324-bd2f-f5561413b156@googlegroups.com> Message-ID: skunkwerk wrote: > On Monday, July 8, 2013 12:45:55 AM UTC-7, Peter Otten wrote: >> skunkwerk wrote: >> >> >> >> > Hi, >> >> > I'm using a custom pickler that replaces any un-pickleable objects >> > (such >> >> > as sockets or files) with a string representation of them, based on >> > the >> >> > code from Shane Hathaway here: >> >> > http://stackoverflow.com/questions/4080688/python-pickling-a-dict-with- >> >> some-unpicklable-items >> >> > >> >> > It works most of the time, but when I try to unpickle a Django >> >> > HttpResponse, I get the following error: UnpicklingError: NEWOBJ class >> >> > argument isn't a type object >> >> > >> >> > I have no clue what the error actually means. If it pickles okay, why >> >> > should it not be able to unpickle? Any ideas? >> >> >> >> A simple way to provoke the error is to rebind the name referring to the >> >> class of the pickled object: >> >> >> >> >>> import cPickle >> >> >>> class A(object): pass >> >> ... >> >> >>> p = cPickle.dumps(A(), -1) >> >> >>> cPickle.loads(p) >> >> <__main__.A object at 0x7fce7bb58c50> >> >> >>> A = 42 >> >> >>> cPickle.loads(p) >> >> Traceback (most recent call last): >> >> File "", line 1, in >> >> cPickle.UnpicklingError: NEWOBJ class argument isn't a type object >> >> >> >> You may be doing something to that effect. > > Hey Peter, > I tried unpickling even from another file with no other code in it, but > came up with the same error - so I don't think it's a rebinding issue. > > But I got the error to disappear when I removed the "hasattr(obj, > '__getstate__')" from this line of code in the persistent_id function: if > not hasattr(obj, '__getstate__') and isinstance(obj,(basestring, bool, > int, long, float, complex, tuple, list, set, dict)): > return ["filtered:%s" % type(obj)] > > When I do that, I get a few more FilteredObjects in the result, for things > like: > > > I figured these classes must have __getstate__ methods which leads to them > being pickled without a persistent_id (it turns out they actually have > __repr__ methods). > > So these classes get pickled fine, but run into problems when trying to > unpickle them. I understand why ImportErrors would happen if the > necessary modules haven't been loaded, but this NEWOBJ error is still kind > of mystifying. I guess I just won't pickle any classes for now, if > unpickling them is going to be dicey. > > thanks for the help guys, > imran Maybe you can find the problem by temporarily switching from cPickle to the pickle module which produces a slightly more helpful traceback: >>> import cPickle, pickle >>> class A(object): pass ... >>> p = cPickle.dumps(A(), -1) >>> A = 42 >>> cPickle.loads(p) Traceback (most recent call last): File "", line 1, in cPickle.UnpicklingError: NEWOBJ class argument isn't a type object >>> pickle.loads(p) Traceback (most recent call last): File "", line 1, in File "/usr/lib/python2.7/pickle.py", line 1382, in loads return Unpickler(file).load() File "/usr/lib/python2.7/pickle.py", line 858, in load dispatch[key](self) File "/usr/lib/python2.7/pickle.py", line 1083, in load_newobj obj = cls.__new__(cls, *args) TypeError: int.__new__(X): X is not a type object (int) From dtbulmer2 at gmail.com Mon Jul 8 04:48:55 2013 From: dtbulmer2 at gmail.com (dtbulmer2 at gmail.com) Date: Mon, 8 Jul 2013 01:48:55 -0700 (PDT) Subject: Tutorials on Jinja In-Reply-To: <2e068b45-0635-4efb-8d7f-39229acb72ba@j9g2000vbp.googlegroups.com> References: <2e068b45-0635-4efb-8d7f-39229acb72ba@j9g2000vbp.googlegroups.com> Message-ID: On Wednesday, June 24, 2009 11:46:55 AM UTC-7, Saurabh wrote: > Hi All, > > I am trying to move my application on a MVC architecture and plan to > use Jinja for the same. Can anyone provide me with few quick links > that might help me to get started with Jinja? > > Thanks, > Saby this is a site that uses jinja2 https://github.com/davidnuon/davidnuon.com From asimjalis at gmail.com Mon Jul 15 13:32:15 2013 From: asimjalis at gmail.com (asimjalis at gmail.com) Date: Mon, 15 Jul 2013 10:32:15 -0700 (PDT) Subject: Tutorials on Jinja In-Reply-To: <2e068b45-0635-4efb-8d7f-39229acb72ba@j9g2000vbp.googlegroups.com> References: <2e068b45-0635-4efb-8d7f-39229acb72ba@j9g2000vbp.googlegroups.com> Message-ID: On Wednesday, June 24, 2009 11:46:55 AM UTC-7, Saurabh wrote: > Hi All, > > I am trying to move my application on a MVC architecture and plan to > use Jinja for the same. Can anyone provide me with few quick links > that might help me to get started with Jinja? > > Thanks, > Saby The documentation at http://jinja.pocoo.org/docs/templates is pretty good. Also this link contains an end-to-end example which might be helpful. https://gist.github.com/warren-runk/1317933 - Asim From sandile.mnukwa at gmail.com Mon Jul 8 04:53:06 2013 From: sandile.mnukwa at gmail.com (Sanza101) Date: Mon, 8 Jul 2013 01:53:06 -0700 (PDT) Subject: Newbie. Need help Message-ID: <7aa32d52-8f91-4ffc-8e0d-3be769cd0e33@googlegroups.com> I just started using Python recently, and i need help with the following: Please assist. 1. Create another function that generates a random number (You will have to import the relevant library to do this) 2. Create a function that is called from the main function, that accepts a number as a parameter and determines if the number is even or odd 3. Now enhance your script to generate 10 random even numbers and write them to a file So far what i have done is: import random def main(): pass if __name__ == '__main__': main() for evenNumber in range (0, 20, 2): print random.randrange(0, 101, 2) f = open("C:\Users\Documents\Myproj.txt", "w"); print f f = open("C:\Users\Documents\Myproj.txt", "a"); print f value = random.randrange(0, 101, 2, ) value1 = random.randrange(0, 201, 2, ) value2 = random.randrange(0, 301, 2, ) myString = str(value) myString1 = str(value1) myString2 = str(value2) print f.write(myString) print f.write(myString1) print f.write(myString2) f.close() From joshua.landau.ws at gmail.com Mon Jul 8 06:28:55 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Mon, 8 Jul 2013 11:28:55 +0100 Subject: Newbie. Need help In-Reply-To: <7aa32d52-8f91-4ffc-8e0d-3be769cd0e33@googlegroups.com> References: <7aa32d52-8f91-4ffc-8e0d-3be769cd0e33@googlegroups.com> Message-ID: On 8 July 2013 09:53, Sanza101 wrote: > I just started using Python recently, and i need help with the following: Please assist. Rather than saying you want help with "Please assist", why don't you ask a question? I find when people start their post with "I need help, please help" they forget that (a) what else would we try to do and (b) to state what they need help with. > 1. Create another function that generates a random number (You will have to import the relevant library to do this) > 2. Create a function that is called from the main function, that accepts a number as a parameter and determines if the number is even or odd > 3. Now enhance your script to generate 10 random even numbers and write them to a file It's normally nice to state when something's a homework. It puts you on much better, more honest terms than if you don't. We won't blank you if you tell us or anything. > So far what i have done is: > import random > def main(): > pass > > if __name__ == '__main__': > main() > for evenNumber in range (0, 20, 2): > print random.randrange(0, 101, 2) > > f = open("C:\Users\Documents\Myproj.txt", "w"); > print f > f = open("C:\Users\Documents\Myproj.txt", "a"); > print f > > value = random.randrange(0, 101, 2, ) > value1 = random.randrange(0, 201, 2, ) > value2 = random.randrange(0, 301, 2, ) > > myString = str(value) > myString1 = str(value1) > myString2 = str(value2) > > print f.write(myString) > print f.write(myString1) > print f.write(myString2) > f.close() OK... so? What do you need help with? I'll guide you through what you've done, and why it doesn't do what you want. > import random So far so good. > def main(): > pass This does.. nothing. You probably want to put everything inside your main function, so it is called only if "__name__ == '__main__'". This is done so that is people "import yourmodule" then they won't immediately have lots of things happening that aren't useful -- like putting numbers in files, but they can still use your functions that you write. > if __name__ == '__main__': > main() That's fine. > for evenNumber in range (0, 20, 2): > print random.randrange(0, 101, 2) You do "for evenNumber in ..." but never use "evenNumber". you should write "for i in range(10):" which is a much better way of saying "do this 10 times". Additionally, no-one in python seems to use thisWayOfNamingThings. Whilst it's not _wrong_, per se, it's not standard either. You'll be happier in the long run if you learn to use this_way_of_naming_things. You were asked to "Create another function that generates a random number". I'm not really sure what this means, but it sounds to me like you want to do: def generate_random_number(): ... which I would just write: "generate_random_number = random.randrange". Yes, just use what you already have. > f = open("C:\Users\Documents\Myproj.txt", "w"); Oh no! What is this ";" on the end? DESTROY IT NOW. > print f This just prints "". Why did you do this? Also, the recommended way to open a file and use it is like this: with open("C:\Users\Documents\Myproj.txt", "w") as f: print f This makes sure you don't forget to close it, and it handles some stuff really cleverly and well. You don't need to know what it all means yet, but just know that for "open" it does The Right Thing?. > f = open("C:\Users\Documents\Myproj.txt", "a"); > print f Exactly the same comments. > value = random.randrange(0, 101, 2, ) > value1 = random.randrange(0, 201, 2, ) > value2 = random.randrange(0, 301, 2, ) Oh.. kay. Why do you have a trailing ", "? Just do: > value = random.randrange(0, 101, 2) > value1 = random.randrange(0, 201, 2) > value2 = random.randrange(0, 301, 2) Also, why are they "value", "value1" and "value2"? Those names make no sense. You could always put them in a tuple (like a list of things): random_numbers = ( random.randrange(0, 101, 2), random.randrange(0, 201, 2), random.randrange(0, 301, 2) ) and then you can index them like "random_numbers[0]" instead of "value", "random_numbers[1]" instead of "value1" and "random_numbers[2]" instead of "value2". I don't know. > myString = str(value) > myString1 = str(value1) > myString2 = str(value2) Urm... "myString"? What does that even mean? If it were me, I'd convert when I need to: > print f.write(str(value)) > print f.write(str(value1)) > print f.write(str(value2)) Also, what do you think the "print" does here? It doesn't print the value. You have no reason for the print, and should remove it. > f.close() If you use "with open(...) as f:" you do not need to close f, btw. So back to the tasks: > 1. Create another function that generates a random number (You will have to import the relevant library to do this) As before, this is a stupid question. If you know what it means, do explain. > 2. Create a function that is called from the main function, that accepts a number as a parameter and determines if the number is even or odd There are only four - Create a function How do you do this? This is easy, so do tell. - that is called from the main function So you want to run the function inside main(); how do you do this? - that accepts a number as a parameter Do you know how to make functions take parameters? - determines if the number is even or odd How do you think you would do this? Hint: http://stackoverflow.com/questions/12754680/modulo-operator-in-python > 3. Now enhance your script to generate 10 random even numbers and write them to a file You know how to generate 10 random numbers. You know how, given a number to write it to a file. So, in your loop, write each number to a file. From joshua.landau.ws at gmail.com Mon Jul 8 09:02:04 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Mon, 8 Jul 2013 14:02:04 +0100 Subject: Newbie. Need help In-Reply-To: References: <7aa32d52-8f91-4ffc-8e0d-3be769cd0e33@googlegroups.com> Message-ID: On 8 July 2013 13:05, Sandile Mnukwa wrote: > Hi Joshua, Hello. You replied off-list (to me only, not to Python-list). I imagine this was a mistake, so I'm posting to Python-list again. If this wasn't a mistake, then I apologize and suggest telling people when you mean to reply off-list. Also, you top-posted, which isn't the right thing to do. This means you wrote your reply above the copy of the text you were replying to -- you should write below. You should also delete all of the parts that you aren't responding to, so people know what you are talking about. > Thanks for the help so far. > Just been panicking about learning the language... > > I have been given a task to do. I am a new to programming and Python. > My task is to : > -Create a function that is called from the main function, that accepts a number as a parameter and determines if the number is even or odd. > the next one is, You have done: def main(): pass So you know how to define a *main function*. This should take a parameter. Do you know how to make a function accept a parameter? Once you have that parameter, you need to check whether it is even or odd. If you don't know how to do this, check the link I gave in my last post or try a Google search for "python check number even odd". Knowing how to look for answers to simple (and later complicated) problems is important to being a good programmer. > -To create another function that generates a random number, and basically when i print this function, it should give me a list of random number separated by a "," commma or in a list. OK, that's good. You know how to generate random numbers (random.randrange), so now you want to define a function. def generate_numbers(): ... You want to *return* a *list* of *random numbers*: list_of_random_numbers = [] for i in range(10): random_number = # You fill this in list_of_random_numbers.append(random_number) then you want to *return* the *list_of_random_numbers*. Do you know how to return values? > -And lastly to enhance my script to generate 10 random EVEN numbers and write them to a .txt file. (This task is the most important of them all) You know how to generate 10 random even numbers. I know you know this because you have done: > for i in range(10): > print random.randrange(0, 101, 2) You should assign each random number to a name (using "="). *Inside* your loop, you want to do SOME_FILE.write(str(RANDOM_NUMBER)) > What I have done so far. > > import random > > if __name__ == '__main__': > main() > for i in range(10): > print random.randrange(0, 101, 2) When I said to put this in the main function, I meant the *function*, not here. This works, but it isn't what people normally do. See how you call "main()" inside here? Thus, when you have the > for i in range(10): > print random.randrange(0, 101, 2) inside "main()" it will get run anyway. Does this make sense? > with open ("C:\Users\Kenz09\Documents\Myproj.txt", "w") as f: > print f When I said use "with", I mean use "with" *everywhere*, not just here. So you should use this same pattern in the other places you open files. You will need to indent the *whole* of the code that requires use of the file "f" when you do this. > f = open("C:\Users\Kenz09\Documents\Myproj.txt", "a"); > print f > > value = ( > random.randrange (0, 101, 2), > random.randrange(0, 201, 2), > random.randrange(0, 301, 2) > ) > > random_numbers[0] > random_numbers[1] > random_numbers[2] > > print f.write(str(value)) > print f.write(str(value1)) > print f.write(str(value2)) > f.close() Remember that I'm telling you this so that you know what to do, not so you can take the code I give you. I will not give you complete solutions -- if you ask what "5 + 5" is I will show you how to work out "4 + 4", and then you can apply that to the original problem. From alister.ware at ntlworld.com Mon Jul 8 07:51:15 2013 From: alister.ware at ntlworld.com (Alister) Date: Mon, 08 Jul 2013 11:51:15 GMT Subject: Newbie. Need help References: <7aa32d52-8f91-4ffc-8e0d-3be769cd0e33@googlegroups.com> Message-ID: On Mon, 08 Jul 2013 01:53:06 -0700, Sanza101 wrote: > I just started using Python recently, and i need help with the > following: Please assist. > > 1. Create another function that generates a random number (You will have > to import the relevant library to do this) > 2. Create a function that is called from the main function, that accepts > a number as a parameter and determines if the number is even or odd 3. > Now enhance your script to generate 10 random even numbers and write > them to a file > > So far what i have done is: > import random def main(): > pass > > if __name__ == '__main__': > main() > for evenNumber in range (0, 20, 2): > print random.randrange(0, 101, 2) > > f = open("C:\Users\Documents\Myproj.txt", "w"); > print f f = open("C:\Users\Documents\Myproj.txt", "a"); > print f > > value = random.randrange(0, 101, 2, ) > value1 = random.randrange(0, 201, 2, ) > value2 = random.randrange(0, 301, 2, ) > > myString = str(value) > myString1 = str(value1) > myString2 = str(value2) > > print f.write(myString) > print f.write(myString1) > print f.write(myString2) > f.close() As this is course work I wont give you answers just pointers to help. from the wording of the question your teacher is probably more interested in you understanding how to define functions. so from the design requirements you will need something like def get_random(): #your code foes here def is_even(number): #you code here you then need to loop through the code writing to file each time you get an even result. a for loop is probably not the best type of loop for this task. once you have some more code I am sure we will help further if you get stuck with any specifics. -- The computer is to the information industry roughly what the central power station is to the electrical industry. -- Peter Drucker From sandile.mnukwa at gmail.com Mon Jul 8 08:01:37 2013 From: sandile.mnukwa at gmail.com (Kenz09) Date: Mon, 8 Jul 2013 05:01:37 -0700 (PDT) Subject: How do I write a script to generate 10 random EVEN numbers and write them to a .txt file? Message-ID: <25284710-bf3f-4e10-bac2-0ba19099f653@googlegroups.com> Hi, I have been given a task to do. I am a new to programming and Python. My task is to : -Create a function that is called from the main function, that accepts a number as a parameter and determines if the number is even or odd. the next one is, -To create another function that generates a random number, and basically when i print this function, it should give me a list of random number separated by a "," commma or in a list. -And lastly to enhance my script to generate 10 random EVEN numbers and write them to a .txt file. This what I have done so far. import random if __name__ == '__main__': main() for i in range(10): print random.randrange(0, 101, 2) with open ("C:\Users\Kenz09\Documents\Myproj.txt", "w") as f: print f f = open("C:\Users\Kenz09\Documents\Myproj.txt", "a"); print f value = ( random.randrange (0, 101, 2), random.randrange(0, 201, 2), random.randrange(0, 301, 2) ) random_numbers[0] random_numbers[1] random_numbers[2] print f.write(str(value)) print f.write(str(value1)) print f.write(str(value2)) f.close() From davea at davea.name Mon Jul 8 08:27:06 2013 From: davea at davea.name (Dave Angel) Date: Mon, 08 Jul 2013 08:27:06 -0400 Subject: How do I write a script to generate 10 random EVEN numbers and write them to a .txt file? In-Reply-To: <25284710-bf3f-4e10-bac2-0ba19099f653@googlegroups.com> References: <25284710-bf3f-4e10-bac2-0ba19099f653@googlegroups.com> Message-ID: On 07/08/2013 08:01 AM, Kenz09 wrote: > Hi, I have been given a task to do. I am a new to programming and Python. > My task is to : > -Create a function that is called from the main function, that accepts a number as a parameter and determines if the number is even or odd. > the next one is, > > -To create another function that generates a random number, and basically when i print this function, it should give me a list of random number separated by a "," commma or in a list. > > -And lastly to enhance my script to generate 10 random EVEN numbers and write them to a .txt file. > One of your classmates has already posted the question. However, you win the prize for a better subject line. Or are you the same student, changing your name and wasting our time by starting a new thread. > This what I have done so far. Where's your main function? > > import random > > if __name__ == '__main__': > main() > for i in range(10): > print random.randrange(0, 101, 2) Why is everything at top level, and not inside the function(s) ? > > with open ("C:\Users\Kenz09\Documents\Myproj.txt", "w") as f: > print f > f = open("C:\Users\Kenz09\Documents\Myproj.txt", "a"); That semicolon comes from another language. Very seldom any need for it in Python. > print f > > value = ( > random.randrange (0, 101, 2), > random.randrange(0, 201, 2), > random.randrange(0, 301, 2) > ) The reason the instructor told you to do 10 is probably to discourage you from typing separate statements for all 10. You're undoubtedly supposed to use a loop. > > random_numbers[0] > random_numbers[1] > random_numbers[2] When you asigned it, you called it value, but now you expect it to have another name? > > print f.write(str(value)) > print f.write(str(value1)) > print f.write(str(value2)) > f.close() > Why not start by doing the first statement of the assignment, and if you can't make it work, or don't understand some part of the question, ask again? Show your work, not just a bunch of random statements, most of which won't execute. If I were wording that first question, I'd say something like: """ Create a function that accepts a integer as a parameter and determines if the number is even or odd, returning a boolean value accordingly. Call that function multiple times from the main function, with a variety of values, indicating for each whether it's even or odd. """ -- DaveA From joshua.landau.ws at gmail.com Mon Jul 8 09:10:50 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Mon, 8 Jul 2013 14:10:50 +0100 Subject: How do I write a script to generate 10 random EVEN numbers and write them to a .txt file? In-Reply-To: References: <25284710-bf3f-4e10-bac2-0ba19099f653@googlegroups.com> Message-ID: On 8 July 2013 13:27, Dave Angel wrote: > One of your classmates has already posted the question. However, you win > the prize for a better subject line. Or are you the same student, changing > your name and wasting our time by starting a new thread. Considering the body of the question and his GMail user icon thingy are identical, I think that's a rigged question. From davea at davea.name Mon Jul 8 16:17:15 2013 From: davea at davea.name (Dave Angel) Date: Mon, 08 Jul 2013 16:17:15 -0400 Subject: How do I write a script to generate 10 random EVEN numbers and write them to a .txt file? In-Reply-To: References: <25284710-bf3f-4e10-bac2-0ba19099f653@googlegroups.com> Message-ID: On 07/08/2013 09:10 AM, Joshua Landau wrote: > On 8 July 2013 13:27, Dave Angel wrote: >> One of your classmates has already posted the question. However, you win >> the prize for a better subject line. Or are you the same student, changing >> your name and wasting our time by starting a new thread. > > Considering the body of the question and his GMail user icon thingy > are identical, I think that's a rigged question. > I didn't notice that the actual gmail address was the same; somehow when I saw it prefixed by the Sanza101 and Kenz09 it looked different. However, the body was entirely different, thoroughly rephrased. For example, first question in each, respectively, was: Create another function that generates a random number (You will have to import the relevant library to do this) Create a function that is called from the main function, that accepts a number as a parameter and determines if the number is even or odd. the next one is, -- DaveA From benedict.verheyen at gmail.com Mon Jul 8 08:46:42 2013 From: benedict.verheyen at gmail.com (Benedict Verheyen) Date: Mon, 8 Jul 2013 12:46:42 +0000 (UTC) Subject: ssl handshake operation timed out on Python 3.3.2 Message-ID: Hi, for a project, I need to post data to some aspx pages. The aspx pages are hosted by another company. I develop on a virtual Debian Wheezy (Virtual box) running on Windows. I couldn't get the code to run either on Windows nor Linux. On my testserver (also a Debian Linux Wheezy) however, the code works with Python 2.7.3 but not with Python 3.3.2. Another difference is that the testserver has a direct ip, so nothing in between the machine and the internet. I checked the firewall, the traffic passes without a problem. The error I get is that the SSL handshake operation timed out. I tried some code that specifies the ssl protocol but the results are the same. I'm stumped. Does anyone have an idea as to why this code works on python 2.7 and not on python 3.3? To summarize, on my dev machine, it doesn't work with either Python 2.7.3 or 3.3.2, on my test machine, it works with Python 2.7.3 but not with Python 3.3.2 An excerpt of the code: ... data = {'_UserName_': username, '_Password_': password} headers = {'Cache-control': 'private','Connection':'Keep-Alive'} r = requests.post("https://url.aspx", data=data, headers=headers, timeout=8) Any ideas would be welcome, Cheers, Benedict -- Benedict Verheyen Debian, Python and Django user GnuPG Public Key 0x712CBB8D From solipsis at pitrou.net Tue Jul 9 06:08:01 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 9 Jul 2013 10:08:01 +0000 (UTC) Subject: ssl handshake operation timed out on Python 3.3.2 References: Message-ID: Benedict Verheyen gmail.com> writes: > > Hi, > > for a project, I need to post data to some aspx pages. > The aspx pages are hosted by another company. > I develop on a virtual Debian Wheezy (Virtual box) running on Windows. > I couldn't get the code to run either on Windows nor Linux. > > On my testserver (also a Debian Linux Wheezy) however, the code works with > Python 2.7.3 but not with Python 3.3.2. Another difference is that the > testserver has a direct ip, so nothing in between the machine and the > internet. I checked the firewall, the traffic passes without a problem. > > The error I get is that the SSL handshake operation timed out. > I tried some code that specifies the ssl protocol but the results are the > same. This may be a IIS-specific problem. Take a look at http://stackoverflow.com/questions/16365483/iis-7-5-mercurial-setup-ignoring-maxallowedcontentlength http://bz.selenic.com/show_bug.cgi?id=3905 http://bugs.python.org/issue17948 Otherwise, dumping network traffic with Wireshark could give you some hints at to what is different between the SSL handshakes in the two setups. Regards Antoine. From benedict.verheyen at gmail.com Tue Jul 9 08:13:31 2013 From: benedict.verheyen at gmail.com (Benedict Verheyen) Date: Tue, 9 Jul 2013 12:13:31 +0000 (UTC) Subject: ssl handshake operation timed out on Python 3.3.2 References: Message-ID: Op Tue, 09 Jul 2013 10:08:01 +0000, schreef Antoine Pitrou: > This may be a IIS-specific problem. > Take a look at > http://stackoverflow.com/questions/16365483/iis-7-5-mercurial-setup- ignoring-maxallowedcontentlength > http://bz.selenic.com/show_bug.cgi?id=3905 > http://bugs.python.org/issue17948 > > Otherwise, dumping network traffic with Wireshark could give you some > hints at to what is different between the SSL handshakes in the two > setups. > > Regards > > Antoine. Hi Antoine, thanks, it seems to be the problem metioned in the links. Aaarggh. IIS... Cheers, Benedict -- Benedict Verheyen Debian, Python and Django user GnuPG Public Key 0x712CBB8D From davide.dalmasso at gmail.com Mon Jul 8 10:44:11 2013 From: davide.dalmasso at gmail.com (davide.dalmasso at gmail.com) Date: Mon, 8 Jul 2013 07:44:11 -0700 (PDT) Subject: ipython Message-ID: Hi, I work with Python 3.3. I downloaded an IPython executable version from http://www.lfd.uci.edu/~gohlke/pythonlibs/ I installed it but no shortcut appears in my start menu. How can I launch it or alternatively is there some other free source of executable file for Windows 7? Many Thanks From steve+comp.lang.python at pearwood.info Mon Jul 8 10:57:14 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 08 Jul 2013 14:57:14 GMT Subject: ipython References: Message-ID: <51dad34a$0$9505$c3e8da3$5496439d@news.astraweb.com> On Mon, 08 Jul 2013 07:44:11 -0700, davide.dalmasso wrote: > Hi, I work with Python 3.3. > I downloaded an IPython executable version from > http://www.lfd.uci.edu/~gohlke/pythonlibs/ I installed it but no > shortcut appears in my start menu. How can I launch it or alternatively > is there some other free source of executable file for Windows 7? Many > Thanks That's more of a Windows 7 question than a Python question. I don't know how to add a command to the Windows start menu. How do you know ipython installed successfully? When you installed it, did you get any errors? Try going to the Start menu, selecting the Run command, and typing ipython [enter]. -- Steven From wanderer at dialup4less.com Mon Jul 8 12:49:08 2013 From: wanderer at dialup4less.com (Wanderer) Date: Mon, 8 Jul 2013 09:49:08 -0700 (PDT) Subject: ipython In-Reply-To: References: Message-ID: <14240ad4-3b52-4acc-b257-62ae599bb58d@googlegroups.com> On Monday, July 8, 2013 10:44:11 AM UTC-4, davide.... at gmail.com wrote: > Hi, I work with Python 3.3. > > I downloaded an IPython executable version from http://www.lfd.uci.edu/~gohlke/pythonlibs/ > > I installed it but no shortcut appears in my start menu. > > How can I launch it or alternatively is there some other free source of executable file for Windows 7? > > Many Thanks You must right click and 'run as administrator' to get the start menu shortcuts. Otherwise, ipython.exe is in the Scripts subdirectory of your Python directory. ie C:\Python27\Scripts From wuwei23 at gmail.com Tue Jul 9 01:29:04 2013 From: wuwei23 at gmail.com (alex23) Date: Tue, 09 Jul 2013 15:29:04 +1000 Subject: ipython In-Reply-To: References: Message-ID: On 9/07/2013 12:44 AM, davide.dalmasso at gmail.com wrote: > Hi, I work with Python 3.3. > I downloaded an IPython executable version from http://www.lfd.uci.edu/~gohlke/pythonlibs/ > I installed it but no shortcut appears in my start menu. > How can I launch it or alternatively is there some other free source of executable file for Windows 7? The IPython installer actually mentions this when it finishes installing: "Distribute (setuptools) is required to create Start Menu items. Re-run this installer after installing distribute to get Start Menu items." You can grab the setup for distribute from: http://python-distribute.org/distribute_setup.py If your Python 3.3 installation is part of your PATH, you simply do: python3 distribute_setup.py install When it's finished, run the IPython installer again and you'll find it now creates a start menu shortcut for you. From davide.dalmasso at gmail.com Tue Jul 9 03:27:20 2013 From: davide.dalmasso at gmail.com (davide.dalmasso at gmail.com) Date: Tue, 9 Jul 2013 00:27:20 -0700 (PDT) Subject: ipython In-Reply-To: References: Message-ID: I did not see the massage when installation finisched. Sorry! Many Thanks!!! > > The IPython installer actually mentions this when it finishes > > installing: > > > > "Distribute (setuptools) is required to create Start Menu items. > > Re-run this installer after installing distribute to get Start > > Menu items." > > > > You can grab the setup for distribute from: > > > > http://python-distribute.org/distribute_setup.py > > > > If your Python 3.3 installation is part of your PATH, you simply do: > > > > python3 distribute_setup.py install > > > > When it's finished, run the IPython installer again and you'll find > > it now creates a start menu shortcut for you. From skip at pobox.com Mon Jul 8 16:43:39 2013 From: skip at pobox.com (Skip Montanaro) Date: Mon, 8 Jul 2013 15:43:39 -0500 Subject: homework + obfuscation ... this might work ... Message-ID: I have an idea. Take the threads where students ask the list to do their homework for them (but don't have the cojones to admit that's what they are doing), and merge them with the obfuscated Python idea. A group of people could come up with the solution off-list, then answer the poster's original question (no fair asking them to revise their poorly specified requirements) with a very obfuscated answer on-list. I think it would be interesting to be a fly on the wall when they explain how their programs work to their professors, especially if they have little easter eggs embedded in them. :-) Perhaps we could start with the 10 random even numbers? I'll start: #!/usr/bin/env python import random output = [] while len(output) < 10: output.append(random.randrange(0, 1000, 2)) print 1, output output = [random.randrange(0, 500) * 2 for _ in range(10)] print 2, output output = [] while len(output) < 10: r = int(round(random.random() * 500)) if r % 2: r *= 2 output.append(r) print 3, output getting-tired-of-homework-questions-in-my-old-age-ly, y'rs, Skip From ian.g.kelly at gmail.com Mon Jul 8 16:57:02 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 8 Jul 2013 14:57:02 -0600 Subject: homework + obfuscation ... this might work ... In-Reply-To: References: Message-ID: On Mon, Jul 8, 2013 at 2:43 PM, Skip Montanaro wrote: > I have an idea. Take the threads where students ask the list to do > their homework for them (but don't have the cojones to admit that's > what they are doing), and merge them with the obfuscated Python idea. > A group of people could come up with the solution off-list, then > answer the poster's original question (no fair asking them to revise > their poorly specified requirements) with a very obfuscated answer > on-list. I think it would be interesting to be a fly on the wall when > they explain how their programs work to their professors, especially > if they have little easter eggs embedded in them. You assume that the professor (or more likely, TA) will take the time to ask them to explain the program and not just grade them down for the extra work they had to do. From skip at pobox.com Mon Jul 8 17:08:38 2013 From: skip at pobox.com (Skip Montanaro) Date: Mon, 8 Jul 2013 16:08:38 -0500 Subject: homework + obfuscation ... this might work ... In-Reply-To: References: Message-ID: > You assume that the professor (or more likely, TA) will take the time > to ask them to explain the program and not just grade them down for > the extra work they had to do. Well, that would be fine too. :-) Skip From joshua.landau.ws at gmail.com Mon Jul 8 17:35:48 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Mon, 8 Jul 2013 22:35:48 +0100 Subject: homework + obfuscation ... this might work ... In-Reply-To: References: Message-ID: On 8 July 2013 21:43, Skip Montanaro wrote: > I have an idea. Take the threads where students ask the list to do > their homework for them (but don't have the cojones to admit that's > what they are doing), and merge them with the obfuscated Python idea. > A group of people could come up with the solution off-list, then > answer the poster's original question (no fair asking them to revise > their poorly specified requirements) with a very obfuscated answer > on-list. I think it would be interesting to be a fly on the wall when > they explain how their programs work to their professors, especially > if they have little easter eggs embedded in them. > > :-) > > Perhaps we could start with the 10 random even numbers? I'll start: You're doing it wrong: def getRandomNumbers() -> [int]: return [4]*10; # chosen by fair dice roll # guaranteed to be random #!/too/famous/to_need/a.reference From cmpython at gmail.com Mon Jul 8 16:52:40 2013 From: cmpython at gmail.com (CM) Date: Mon, 8 Jul 2013 13:52:40 -0700 (PDT) Subject: make sublists of a list broken at nth certain list items Message-ID: <9d0cd072-3cf7-4156-8e84-884faeef7048@googlegroups.com> I'm looking for a Pythonic way to do the following: I have data in the form of a long list of tuples. I would like to break that list into four sub-lists. The break points would be based on the nth occasion of a particular tuple. (The list represents behavioral data trials; the particular tuple represents the break between trials; I want to collect 20 trials at a time, so every 20th break between trials, start a new sublist). So say I have this data: data_list = [(0.0, 1.0), (1.0, 24.0), (24.0, 9.0), (9.0, 17.0), (17.0, 5.0), (5.0, 0.0), (5.0, 0.0), (5.0, 24.0), (24.0, 13.0), (13.0, 0.0), (13.0, 21.0), (21.0, 0.0), (21.0, 0.0), (21.0, 23.0), (23.0, 24.0), (24.0, 10.0), (10.0, 18.0), (18.0, 4.0), (4.0, 22.0), (22.0, 1.0), (1.0, 0.0), (1.0, 24.0), (24.0, 6.0), (6.0, 14.0), (14.0, 5.0), (5.0, 0.0), (5.0, 0.0), (5.0, 0.0), (5.0, 0.0), (5.0, 0.0), (5.0, 0.0), (5.0, 0.0), (5.0, 0.0), (5.0, 0.0), (5.0, 24.0), (24.0, 6.0), (6.0, 14.0), (14.0, 4.0), (4.0, 0.0), (4.0, 22.0), (22.0, 1.0), (1.0, 0.0), (1.0, 24.0), (24.0, 9.0), (9.0, 17.0), (17.0, 4.0), (4.0, 0.0), (4.0, 22.0), (22.0, 1.0), (1.0, 0.0), (1.0, 0.0), (1.0, 24.0), (24.0, 12.0), (12.0, 4.0), (4.0, 0.0), (4.0, 22.0)] #rest of data truncated... I'd like to break the list into sublists at the 20th, 40th, and 60th occasions of any tuple that begins with 1.0--so for example, (1.0, 0.0). This will produce four sub-lists, for trial 1-20, 21-40, 41-60, and 61-80. What I have, just to get the break points within the data_list, and which is not working is: trial_break_indexes_list = [] #needed to see where the sublists start trial_count = 0 #keep count of which trial we're on trial_break_indexes_list = [] #holds the index of the transitions_list for trials 1-20, 21-40, 41-60, and 61-80 trial_count = 0 for tup in data_list: if tup[0] == 1.0: #Therefore the start of a new trial #We have a match! Therefore get the index in the data_list data_list_index = data_list.index(tup) trial_count += 1 #update the trial count. if trial_count % 20 == 0: #this will match on 0, 20, 40, 60, 80 trial_break_indexes_list.append(data_list_index) print 'This is trial_break_indexes_list: ', trial_break_indexes_list Unfortunately, the final output here is: >>> This is trial_break_indexes_list: [1, 20, 20, 20, 20, 1, 20, 1] I sense there is a way more elegant/simpler/Pythonic way to approach this, let alone one that is actually correct, but I don't know of it. Suggestions appreciated! Thanks. From fabiosantosart at gmail.com Mon Jul 8 17:13:35 2013 From: fabiosantosart at gmail.com (=?ISO-8859-1?Q?F=E1bio_Santos?=) Date: Mon, 8 Jul 2013 22:13:35 +0100 Subject: make sublists of a list broken at nth certain list items In-Reply-To: <9d0cd072-3cf7-4156-8e84-884faeef7048@googlegroups.com> References: <9d0cd072-3cf7-4156-8e84-884faeef7048@googlegroups.com> Message-ID: You don't want to use index() to figure out the index of the tuples. It is slower, and will not find the item you want if there is more than one of the same. For example, [1, 4, 4, 4].index(4) will always be 1, no matter how many times you loop through it. Instead, use enumerate() to keep track of the index. Replace your loop by: for index, tup in enumerate(data_list): This should fix your problem. After you have the correct indices, look into list slicing syntax. On 8 Jul 2013 21:59, "CM" wrote: > I'm looking for a Pythonic way to do the following: > > I have data in the form of a long list of tuples. I would like to break > that list into four sub-lists. The break points would be based on the nth > occasion of a particular tuple. (The list represents behavioral data > trials; the particular tuple represents the break between trials; I want to > collect 20 trials at a time, so every 20th break between trials, start a > new sublist). > > So say I have this data: > > data_list = [(0.0, 1.0), (1.0, 24.0), (24.0, 9.0), (9.0, 17.0), (17.0, > 5.0), (5.0, 0.0), (5.0, 0.0), (5.0, 24.0), (24.0, 13.0), (13.0, 0.0), > (13.0, 21.0), (21.0, 0.0), (21.0, 0.0), (21.0, 23.0), (23.0, 24.0), (24.0, > 10.0), (10.0, 18.0), (18.0, 4.0), (4.0, 22.0), (22.0, 1.0), (1.0, 0.0), > (1.0, 24.0), (24.0, 6.0), (6.0, 14.0), (14.0, 5.0), (5.0, 0.0), (5.0, 0.0), > (5.0, 0.0), (5.0, 0.0), (5.0, 0.0), (5.0, 0.0), (5.0, 0.0), (5.0, 0.0), > (5.0, 0.0), (5.0, 24.0), (24.0, 6.0), (6.0, 14.0), (14.0, 4.0), (4.0, 0.0), > (4.0, 22.0), (22.0, 1.0), (1.0, 0.0), (1.0, 24.0), (24.0, 9.0), (9.0, > 17.0), (17.0, 4.0), (4.0, 0.0), (4.0, 22.0), (22.0, 1.0), (1.0, 0.0), (1.0, > 0.0), (1.0, 24.0), (24.0, 12.0), (12.0, 4.0), (4.0, 0.0), (4.0, 22.0)] > #rest of data truncated... > > I'd like to break the list into sublists at the 20th, 40th, and 60th > occasions of any tuple that begins with 1.0--so for example, (1.0, 0.0). > This will produce four sub-lists, for trial 1-20, 21-40, 41-60, and 61-80. > > What I have, just to get the break points within the data_list, and which > is not working is: > > trial_break_indexes_list = [] #needed to see where the sublists start > trial_count = 0 #keep count of which trial we're on > > trial_break_indexes_list = [] #holds the index of the transitions_list > for trials 1-20, 21-40, 41-60, and 61-80 > trial_count = 0 > > for tup in data_list: > if tup[0] == 1.0: #Therefore the start of a new trial > > #We have a match! Therefore get the index in the data_list > data_list_index = data_list.index(tup) > > trial_count += 1 #update the trial count. > > if trial_count % 20 == 0: #this will match on 0, 20, 40, 60, 80 > trial_break_indexes_list.append(data_list_index) > > print 'This is trial_break_indexes_list: ', trial_break_indexes_list > > Unfortunately, the final output here is: > > >>> > This is trial_break_indexes_list: [1, 20, 20, 20, 20, 1, 20, 1] > > I sense there is a way more elegant/simpler/Pythonic way to approach this, > let alone one that is actually correct, but I don't know of it. > Suggestions appreciated! > > Thanks. > -- > http://mail.python.org/mailman/listinfo/python-list > -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshua.landau.ws at gmail.com Mon Jul 8 17:24:09 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Mon, 8 Jul 2013 22:24:09 +0100 Subject: make sublists of a list broken at nth certain list items In-Reply-To: <9d0cd072-3cf7-4156-8e84-884faeef7048@googlegroups.com> References: <9d0cd072-3cf7-4156-8e84-884faeef7048@googlegroups.com> Message-ID: On 8 July 2013 21:52, CM wrote: > I'm looking for a Pythonic way to do the following: > > I have data in the form of a long list of tuples. I would like to break that list into four sub-lists. The break points would be based on the nth occasion of a particular tuple. (The list represents behavioral data trials; the particular tuple represents the break between trials; I want to collect 20 trials at a time, so every 20th break between trials, start a new sublist). I would do this like so: from collections import deque # Fast and hacky -- just how I like it exhaust_iterable = deque(maxlen=0).extend def chunk_of(data, *, length): count = 0 for datum in data: count += datum[0] == 1 yield datum if count == 60: break def chunked(data): data = iter(data) while True: chunk = chunk_of(data, length=20) yield chunk exhaust_iterable(chunk) You use "chunked(data)" and iterate over the 'chunks' in that. If you go to the next chunk before finishing the one you're on the previous chunk will be lost, so convert it to a permanent form first. Looking at you code: > for tup in data_list: > if tup[0] == 1.0: #Therefore the start of a new trial > > #We have a match! Therefore get the index in the data_list > data_list_index = data_list.index(tup) This is no good (ninja'd by F?bio). The proper way to keep an index is by: for index, tup in enumerate(data_list): > trial_count += 1 #update the trial count. > > if trial_count % 20 == 0: #this will match on 0, 20, 40, 60, 80 > trial_break_indexes_list.append(data_list_index) > > print 'This is trial_break_indexes_list: ', trial_break_indexes_list ... > I sense there is a way more elegant/simpler/Pythonic way to approach this, let alone one that is actually correct, but I don't know of it. Suggestions appreciated! Yup. From joshua.landau.ws at gmail.com Mon Jul 8 17:27:04 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Mon, 8 Jul 2013 22:27:04 +0100 Subject: make sublists of a list broken at nth certain list items In-Reply-To: References: <9d0cd072-3cf7-4156-8e84-884faeef7048@googlegroups.com> Message-ID: On 8 July 2013 22:24, Joshua Landau wrote: > if count == 60: Obviously this should be: if count == length: From ajetrumpet at gmail.com Mon Jul 8 21:45:16 2013 From: ajetrumpet at gmail.com (ajetrumpet at gmail.com) Date: Mon, 8 Jul 2013 18:45:16 -0700 (PDT) Subject: the general development using Python Message-ID: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> all, I am unhappy with the general Python documentation and tutorials. I have worked with Python very little and I'm well aware of the fact that it is a lower-level language that integrates with the shell. I came from a VB legacy background and I've already "un-learned" everything that I need to (I know, that language stinks, and isn't OOP or even useful!). I have to get back into writing Python but I'm lacking one thing ... a general understanding of how to write applications that can be deployed (either in .exe format or in other formats). So my issue is basically to understand how to go about writing programs and compiling them so they can be deployed to less tech-savvy people. Here's what I think I have to do, in a general sense: => Pick a GUI and just run through the tutorials to learn the interfaces as fast as possible. This is all fine and dandy, but more than likely when I do this the people that I am sending solutions to will, if not receiving a simple .exe file, receive the package from me and say to themselves "what in the world do I do with this!?" Is there anyway you guys would suggest that I fix this or help them deal with complex languages like Python and programs written with it? thanks guys. From joshua.landau.ws at gmail.com Mon Jul 8 21:59:37 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Tue, 9 Jul 2013 02:59:37 +0100 Subject: the general development using Python In-Reply-To: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> Message-ID: On 9 July 2013 02:45, wrote: > all, > > I am unhappy with the general Python documentation and tutorials. I have worked with Python very little and I'm well aware of the fact that it is a lower-level language that integrates with the shell. > > I came from a VB legacy background and I've already "un-learned" everything that I need to (I know, that language stinks, and isn't OOP or even useful!). > > I have to get back into writing Python but I'm lacking one thing ... a general understanding of how to write applications that can be deployed (either in .exe format or in other formats). > > So my issue is basically to understand how to go about writing programs and compiling them so they can be deployed to less tech-savvy people. Here's what I think I have to do, in a general sense: > > => Pick a GUI and just run through the tutorials to learn the interfaces as fast as possible. > > This is all fine and dandy, but more than likely when I do this the people that I am sending solutions to will, if not receiving a simple .exe file, receive the package from me and say to themselves "what in the world do I do with this!?" > > Is there anyway you guys would suggest that I fix this or help them deal with complex languages like Python and programs written with it? You cannot compile Python in any meaningful way that does what you want. There are projects that "bundle" the CPython interpreter with your project, but this makes those files really big. I suggest just making sure that Python is installed on their end - it's a one-time thing anyway. You don't expect to be able to run Javascript without a Javascript interpreter (such as a browser) so why would you expect differently for Python? From joshua.landau.ws at gmail.com Tue Jul 9 17:13:17 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Tue, 9 Jul 2013 22:13:17 +0100 Subject: the general development using Python In-Reply-To: References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> Message-ID: On 9 July 2013 03:08, Adam Evanovich wrote: > Joshua, > > Why did you send me an email reply instead of replying in the google groups? Apologies, although it's not quite that simple. I access this list the way it was originally intended -- through EMail. I replied "to all", which default to both the list (by CC) and the person, and I forgot to drop you from the reply. If you use Google Groups, it's also best to read http://wiki.python.org/moin/GoogleGroupsPython first. Secondly, please don't top post. It seems reasonable to post back this to the list, so I have. > I just want to make sure that it really is as simple as: > > 1) Installing Python on users machine. > 2) Running the deployment on their end to parse the DOM via the > application/tool that I write. > 3) Waiting for the output so they can see it. > > Is it really any more complicated than that (other than writing the > tool/app, obviously)? As other's have responded, that's it. > Can you tell me quickly what kind of deployment > options Python has? What you can just do is send a .zip/.tar.gz/.tar.xz or other archive format with all of the files they need to have. Normally I find it nicest to just run like that (I personally reserve installing for the things that benefit from it). If you want installation as an option, then AFAIK the only option in wide usage is http://docs.python.org/3/distutils/index.html. There may be others, but you'd need to ask the experts about that. > Can you wrap source code/libs/apps into an EXE and just > send that to the end user? Or is it more complicated for them? Urm.. yes. But don't. That's the "nuclear" option and isn't a good one. If you have a *really genuinely good reason* (you probably don't) to do this, there are ways. From cmpython at gmail.com Tue Jul 9 19:35:34 2013 From: cmpython at gmail.com (CM) Date: Tue, 9 Jul 2013 16:35:34 -0700 (PDT) Subject: the general development using Python In-Reply-To: References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> Message-ID: On Tuesday, July 9, 2013 5:13:17 PM UTC-4, Joshua Landau wrote: > On 9 July 2013 03:08, Adam Evanovich wrote: > > Can you wrap source code/libs/apps into an EXE and just > > send that to the end user? Or is it more complicated for them? > > Urm.. yes. But don't. That's the "nuclear" option and isn't a good > one. If you have a *really genuinely good reason* (you probably don't) > to do this, there are ways. I still think you are overstating it somewhat. Have a website on which you distribute your software to end users (and maybe even--gasp--charge them for it)? *That's* a good reason. And that's one of the top ways that users get software. Also, many programs rely on 2-3 dependencies, and sometimes that is asking a lot of the end user to install. (I know, I know, it shouldn't be...and with things like pip it really shouldn't be, but you know how it goes). I completely agree with you in Ideal World thinking, but in the gnarly one we actually have, .exe files *often* have their place. From joshua at landau.ws Tue Jul 9 20:14:44 2013 From: joshua at landau.ws (Joshua Landau) Date: Wed, 10 Jul 2013 01:14:44 +0100 Subject: the general development using Python In-Reply-To: References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> Message-ID: On 10 July 2013 00:35, CM wrote: > On Tuesday, July 9, 2013 5:13:17 PM UTC-4, Joshua Landau wrote: >> On 9 July 2013 03:08, Adam Evanovich wrote: >> > Can you wrap source code/libs/apps into an EXE and just >> > send that to the end user? Or is it more complicated for them? > >> Urm.. yes. But don't. That's the "nuclear" option and isn't a good >> one. If you have a *really genuinely good reason* (you probably don't) >> to do this, there are ways. > > I still think you are overstating it somewhat. Have a website on which you distribute your software to end users (and maybe even--gasp--charge them for it)? *That's* a good reason. Not really. It'd be a good reason if it disqualifies the other options, but it doesn't. Just give them an archive. If you're worried about keeping your code "safe" then: 1) You're going about it the wrong way. Like, seriously wrongly. 2) It's not going to be totally secure even if you do it the right way. The most safeyest way you can do with Python AFAIK?? is a long winded process to basically just "compile it with Cython". Note that it still requires CPython (the normal python interpreter) to be installed -- you get lots of .so files instead of a .exe. There are other silly things you can do as well. > And that's one of the top ways that users get software. Pah, I much prefer it over here on Linux :P. > Also, many programs rely on 2-3 dependencies, and sometimes that is asking a lot of the end user to install. (I know, I know, it shouldn't be...and with things like pip it really shouldn't be, but you know how it goes). But why do they need to install it at all? If you're not installing the .py file, then just include those dependencies in the archive -- .py files are tiny. If you are installing the .py with a setup.py (like with the link I included), then just install them at the same time. > I completely agree with you in Ideal World thinking, but in the gnarly one we actually have, .exe files *often* have their place. Yeah, but not for Python :P. For Python .exe files are a rarity and should be kept that way. ? I'm not even sure that Cython doesn't keep a copy of the original code for crash reports and other debugging stuff... ? Theres also Nuitka, which I found right now, so I'm not sure if it's any good for this or not. I haven't tried, really. From cmpython at gmail.com Tue Jul 9 22:16:53 2013 From: cmpython at gmail.com (CM) Date: Tue, 9 Jul 2013 19:16:53 -0700 (PDT) Subject: the general development using Python In-Reply-To: References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> Message-ID: <884377a7-7c95-4f2f-a52f-2e47f98b98d0@googlegroups.com> On Tuesday, July 9, 2013 8:14:44 PM UTC-4, Joshua Landau wrote: > > I still think you are overstating it somewhat. Have a website on which you distribute your software to end users (and maybe even--gasp--charge them for it)? *That's* a good reason. > Not really. It'd be a good reason if it disqualifies the other > options, but it doesn't. Just give them an archive. > If you're worried about keeping your code "safe" then: That's not what I was thinking in terms of, although it's fine to note that since people on this list occasionally think just that. What I was thinking of was that if you are going to sell software, you want to make it as easy as possible, and that includes not making the potential customer have to install anything, or even agree to allow you to "explicitly" install a runtime on their computer. If the potential customer just sees, clicks, and installs, that should be the most they ought to have to do. > > Also, many programs rely on 2-3 dependencies, and sometimes that is asking a lot of the end user to install. (I know, I know, it shouldn't be...and with things like pip it really shouldn't be, but you know how it goes). > > > But why do they need to install it at all? If you're not installing > the .py file, then just include those dependencies in the archive -- > .py files are tiny. If you are installing the .py with a setup.py > (like with the link I included), then just install them at the same > time. Maybe. I'll have to think about it. I'm referring to libaries as dependencies. So for example, though .py files are small, wxPython, for example, isn't tiny, nor are other libraries one might use. > Yeah, but not for Python :P. For Python .exe files are a rarity and > should be kept that way. That there is a significant interest in creating exe files suggest that not everyone feels that way. From rosuav at gmail.com Tue Jul 9 22:53:40 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 10 Jul 2013 12:53:40 +1000 Subject: the general development using Python In-Reply-To: <884377a7-7c95-4f2f-a52f-2e47f98b98d0@googlegroups.com> References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> <884377a7-7c95-4f2f-a52f-2e47f98b98d0@googlegroups.com> Message-ID: On Wed, Jul 10, 2013 at 12:16 PM, CM wrote: > On Tuesday, July 9, 2013 8:14:44 PM UTC-4, Joshua Landau wrote: >> Yeah, but not for Python :P. For Python .exe files are a rarity and >> should be kept that way. > > That there is a significant interest in creating exe files suggest that not everyone feels that way. No; there can be interest in something that should be a rarity. I have interest in turning a MUD client around backwards and having it bind and listen instead of connecting, but I know that that should be extremely rare. Useful, but rare. There are a number of wrong reasons for wanting to turn Python scripts into exe files, and a number of better reasons that place different demands on the structure. For instance, some people aren't trying to conceal their code, just to make a simple deployment system (as per this thread). In that case, some form of self-extracting zip file might be best. Others try to prevent their code from being stolen, or tampered with. That's approximately impossible, but in any case, that will obviously NOT be just a sfx. And then there are those who want to compile to binary for speed (and not all of them are even correct in what they're looking for). So there'll always be multiple solutions to what you think is one problem ("create an exe file from a Python application"), suggesting that it might not be one problem at all. ChrisA From joshua at landau.ws Wed Jul 10 00:12:16 2013 From: joshua at landau.ws (Joshua Landau) Date: Wed, 10 Jul 2013 05:12:16 +0100 Subject: the general development using Python In-Reply-To: <884377a7-7c95-4f2f-a52f-2e47f98b98d0@googlegroups.com> References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> <884377a7-7c95-4f2f-a52f-2e47f98b98d0@googlegroups.com> Message-ID: On , CM wrote: > What I was thinking of was that if you are going to sell software, you want to make it as easy as possible, and that includes not making the potential customer have to install anything, or even agree to allow you to "explicitly" install a runtime on their computer. If the potential customer just sees, clicks, and installs, that should be the most they ought to have to do. I don't really get what you are saying. Do you, or do you not, want it installed? > Also, many programs rely on 2-3 dependencies, and sometimes that is asking a lot of the end user to install. (I know, I know, it shouldn't be...and with things like pip it really shouldn't be, but you know how it goes). > >> I responded (to some parts): >> But why do they need to install it at all? If you're not installing >> the .py file, then just include those dependencies in the archive -- >> .py files are tiny. If you are installing the .py with a setup.py >> (like with the link I included), then just install them at the same >> time. > > Maybe. I'll have to think about it. I'm referring to libaries as dependencies. So for example, though .py files are small, wxPython, for example, isn't tiny, nor are other libraries one might use. Please excuse the fact I haven't done anything serious on Windows in years so I'm not really sure what I'm saying. How does Windows deal with dependencies? It's going to have to be fetched at one point anyway, so that's either at download-time, install-time or run-time. The first lets you just add it to the archive, the second lets you deal with it through a good standard distribution manager thing, the third is potentially crazy. Hence, wutz za probem bruv? From cmpython at gmail.com Wed Jul 10 00:49:07 2013 From: cmpython at gmail.com (CM) Date: Tue, 9 Jul 2013 21:49:07 -0700 (PDT) Subject: the general development using Python In-Reply-To: References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> <884377a7-7c95-4f2f-a52f-2e47f98b98d0@googlegroups.com> Message-ID: <3cca113a-90cc-445e-b609-2ed757212c4a@googlegroups.com> On Wednesday, July 10, 2013 12:12:16 AM UTC-4, Joshua Landau wrote: > On , CM wrote: > > > What I was thinking of was that if you are going to sell software, you want to make it as easy as possible, and that includes not making the potential customer have to install anything, or even agree to allow you to "explicitly" install a runtime on their computer. If the potential customer just sees, clicks, and installs, that should be the most they ought to have to do. > I don't really get what you are saying. Do you, or do you not, want it > installed? I'm just saying that sometimes one goes to download new software and are met with a statement such as: "Installing Eclipse is relatively easy, but does involve a few steps and software from at least two different sources. Eclipse is a Java-based application and, as such, requires a Java runtime environment (JRE) in order to run. ...Regardless of your operating system, you will need to install some Java virtual machine (JVM). You may either install a Java Runtime Environment (JRE), or a Java Development Kit (JDK), depending on what you want to do with Eclipse." This is not always the type of thing you want your customers to encounter. Can all the installation of the runtimes be done with an installer that is itself an .exe, like with PyInstaller? If so, that's probably fine. > > Maybe. I'll have to think about it. I'm referring to libaries as dependencies. So for example, though .py files are small, wxPython, for example, isn't tiny, nor are other libraries one might use. > > Please excuse the fact I haven't done anything serious on Windows in > years so I'm not really sure what I'm saying. How does Windows deal > with dependencies? > > It's going to have to be fetched at one point anyway, so that's either > at download-time, install-time or run-time. The first lets you just > add it to the archive, the second lets you deal with it through a good > standard distribution manager thing, the third is potentially crazy. > Hence, wutz za probem bruv? I'm good with the first way, and I'm fine with Linux's package manager/whatever doing it the second. To simplify everything: sales require massive simplicity for (some) end users. You can get 1-2 clicks out of them before they drop dead as buyers. Furthermore, and I haven't mentioned this yet, an .exe file on Windows has the look of authenticity, whereas a .py file (which is a text file, really) doesn't, which might also matter to customer perceptions. This is all psychology. The ease of deployment side is up for grabs, but yes, potentially a hassle for cross platform deployment. I'm open to the idea of using an installer .exe to set up the user's computer with Python and all the libraries he needs to get going. I just haven't done that so far. From joshua at landau.ws Wed Jul 10 01:49:15 2013 From: joshua at landau.ws (Joshua Landau) Date: Wed, 10 Jul 2013 06:49:15 +0100 Subject: the general development using Python In-Reply-To: <3cca113a-90cc-445e-b609-2ed757212c4a@googlegroups.com> References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> <884377a7-7c95-4f2f-a52f-2e47f98b98d0@googlegroups.com> <3cca113a-90cc-445e-b609-2ed757212c4a@googlegroups.com> Message-ID: On 10 July 2013 05:49, CM wrote: > On Wednesday, July 10, 2013 12:12:16 AM UTC-4, Joshua Landau wrote: >> On , CM wrote: >> >> > What I was thinking of was that if you are going to sell software, you want to make it as easy as possible, and that includes not making the potential customer have to install anything, or even agree to allow you to "explicitly" install a runtime on their computer. If the potential customer just sees, clicks, and installs, that should be the most they ought to have to do. > >> I don't really get what you are saying. Do you, or do you not, want it >> installed? > > I'm just saying that sometimes one goes to download new software and are met with a statement such as: > > "Installing Eclipse is relatively easy, but does involve a few steps and software from at least two different sources. Eclipse is a Java-based application and, as such, requires a Java runtime environment (JRE) in order to run. ...Regardless of your operating system, you will need to install some Java virtual machine (JVM). You may either install a Java Runtime Environment (JRE), or a Java Development Kit (JDK), depending on what you want to do with Eclipse." > > This is not always the type of thing you want your customers to encounter. > > Can all the installation of the runtimes be done with an installer that is itself an .exe, like with PyInstaller? If so, that's probably fine. I don't get what PyInstaller has to do with that, it seems to do something other than what you said. The rest of my na?ve just-thought-of-it-now approach is below. >> > Maybe. I'll have to think about it. I'm referring to libaries as dependencies. So for example, though .py files are small, wxPython, for example, isn't tiny, nor are other libraries one might use. >> >> Please excuse the fact I haven't done anything serious on Windows in >> years so I'm not really sure what I'm saying. How does Windows deal >> with dependencies? >> >> It's going to have to be fetched at one point anyway, so that's either >> at download-time, install-time or run-time. The first lets you just >> add it to the archive, the second lets you deal with it through a good >> standard distribution manager thing, the third is potentially crazy. >> Hence, wutz za probem bruv? > > I'm good with the first way, and I'm fine with Linux's package manager/whatever doing it the second. For the second I meant something like setuptools, distribute, distutils and co. > To simplify everything: sales require massive simplicity for (some) end users. You can get 1-2 clicks out of them before they drop dead as buyers. I was mainly talking in the context of the original post, where it seems something slightly different was meant. If you're deploying to customers, you'd want to offer them an installer. At least, I think you would. That's different from packing Python into a .exe file and pretending it's good-to-go. If they don't want it installed, again the best thing to do is an archive with some "executable" (possibly a batch file or something -- you Windows people would know better what you need) that just runs "python main_file.py". Then get them to extract + click. That's 2 operations, and a lot faster than some silly install process. That does require them to install Python, but you can just add Python's installer in there. That makes 3 operations, but you can always make the "launcher" run the installer if it's not found. >Furthermore, and I haven't mentioned this yet, an .exe file on Windows has the look of authenticity, whereas a .py file (which is a text file, really) doesn't, which might also matter to customer perceptions. This is all psychology. Unfortunately I cannot argue with that stupidity. It's true but shameful. > The ease of deployment side is up for grabs, but yes, potentially a hassle for cross platform deployment. > > I'm open to the idea of using an installer .exe to set up the user's computer with Python and all the libraries he needs to get going. I just haven't done that so far. There are a lot of ways of making an installer, and my current few Googles have shown that distutils comes with a way of making .msi files?, and there's also http://wix.tramontana.co.hu/. Some random internet guy reccomended https://code.google.com/p/omaha/, but I've no idea if it's appropriate. There's also http://nsis.sourceforge.net/Main_Page. Again, I'm no Windows user so I'm talking by guessing. These are saner solutions because they focus on installing rather than pretending that a .exe file with a packaged Python is anything like a compiled C source. ? Just run "python setup.py bdist_msi" with a working setup.py and you get a free .msi! I don't know what the .msi assumes, and I have no way of testing as of now. From ian.g.kelly at gmail.com Wed Jul 10 18:10:10 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 10 Jul 2013 16:10:10 -0600 Subject: the general development using Python In-Reply-To: <3cca113a-90cc-445e-b609-2ed757212c4a@googlegroups.com> References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> <884377a7-7c95-4f2f-a52f-2e47f98b98d0@googlegroups.com> <3cca113a-90cc-445e-b609-2ed757212c4a@googlegroups.com> Message-ID: On Tue, Jul 9, 2013 at 10:49 PM, CM wrote: > Can all the installation of the runtimes be done with an installer that is itself an .exe, like with PyInstaller? If so, that's probably fine. It should be noted that PyInstaller is confusingly named. It actually creates standalone executables, not installers. From cmpython at gmail.com Wed Jul 10 19:18:58 2013 From: cmpython at gmail.com (CM) Date: Wed, 10 Jul 2013 16:18:58 -0700 (PDT) Subject: the general development using Python In-Reply-To: References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> <884377a7-7c95-4f2f-a52f-2e47f98b98d0@googlegroups.com> <3cca113a-90cc-445e-b609-2ed757212c4a@googlegroups.com> Message-ID: > I was mainly talking in the context of the original post, where it > seems something slightly different was meant. If you're deploying to > customers, you'd want to offer them an installer. At least, I think > you would. That's different from packing Python into a .exe file and > pretending it's good-to-go. It depends. In some sense, probably an .exe is the very simplest thing you can provide a user: they download and move it to where they want, the click on it, it runs. It *is* good-to-go, isn't it? (btw, I know at least one person I've read on a forum who just won't use any installer--he feels it can put stuff on his computer where he doesn't have good easy control of it.) Sometimes an installer might be preferable, especially if there are data or image files or tutorials or something that go with the .exe (that weren't, for some reason, bundled inside it). > If they don't want it installed, again the best thing to do is an > archive with some "executable" (possibly a batch file or something -- > you Windows people would know better what you need) that just runs > "python main_file.py". Then get them to extract + click. That's 2 > operations, and a lot faster than some silly install process. But that's pretty much what the .exe that py2exe makes does anyway. It just kind of hides it all inside the .exe file. > There are a lot of ways of making an installer, and my current few > Googles have shown that distutils comes with a way of making .msi > files?, and there's also http://wix.tramontana.co.hu/. Some random > internet guy reccomended https://code.google.com/p/omaha/, but I've no > idea if it's appropriate. There's also > http://nsis.sourceforge.net/Main_Page. Again, I'm no Windows user so > I'm talking by guessing. And InnoSetup, which I've used to good effect. > These are saner solutions because they focus on installing rather than > pretending that a .exe file with a packaged Python is anything like a > compiled C source. I don't think most informed users of, say, py2exe think that. I think they see it as "freezing" the application for single-file portability. The fact that people will refer to it as "compiling it to an exe" is unfortunate, yes. Again, for anyone selling software, just make as few steps as possible for the user. Using py2exe (which is easy to do) to freeze a lot of .py scripts into one easily deployed app passes that test. So does any simple method you mentioned, I'm sure. It all gets us to the same place, right? From joshua at landau.ws Wed Jul 10 19:57:11 2013 From: joshua at landau.ws (Joshua Landau) Date: Thu, 11 Jul 2013 00:57:11 +0100 Subject: the general development using Python In-Reply-To: References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> <884377a7-7c95-4f2f-a52f-2e47f98b98d0@googlegroups.com> <3cca113a-90cc-445e-b609-2ed757212c4a@googlegroups.com> Message-ID: On 11 July 2013 00:18, CM wrote: > >> I was mainly talking in the context of the original post, where it >> seems something slightly different was meant. If you're deploying to >> customers, you'd want to offer them an installer. At least, I think >> you would. That's different from packing Python into a .exe file and >> pretending it's good-to-go. > > It depends. In some sense, probably an .exe is the very simplest thing you can provide a user: they download and move it to where they want, the click on it, it runs. It *is* good-to-go, isn't it? (btw, I know at least one person I've read on a forum who just won't use any installer--he feels it can put stuff on his computer where he doesn't have good easy control of it.) Yeah, but why keep shipping the Python interpreter? If you choose the installer route, you don't have to keep shipping it -- it's only downloaded if you need it. If not, then you don't download it again. > Sometimes an installer might be preferable, especially if there are data or image files or tutorials or something that go with the .exe (that weren't, for some reason, bundled inside it). > >> If they don't want it installed, again the best thing to do is an >> archive with some "executable" (possibly a batch file or something -- >> you Windows people would know better what you need) that just runs >> "python main_file.py". Then get them to extract + click. That's 2 >> operations, and a lot faster than some silly install process. > > But that's pretty much what the .exe that py2exe makes does anyway. It just kind of hides it all inside the .exe file. The problem is that it hides it all inside the .exe file. And I'm not suggesting putting Python inside the archive -- just the runtime dependencies. >> There are a lot of ways of making an installer . >> These are saner solutions because they focus on installing rather than >> pretending that a .exe file with a packaged Python is anything like a >> compiled C source. > > I don't think most informed users of, say, py2exe think that. I think they see it as "freezing" the application for single-file portability. The fact that people will refer to it as "compiling it to an exe" is unfortunate, yes. Fair enough. > Again, for anyone selling software, just make as few steps as possible for the user. Using py2exe (which is easy to do) to freeze a lot of .py scripts into one easily deployed app passes that test. So does any simple method you mentioned, I'm sure. It all gets us to the same place, right? But I still think my way is better. Perhaps I'm just too pragmatic about these things. This was triggered by the OP's original request -- he sounds very much like he doesn't actually want this -- and it seems to have stuck. I don't know if being pragmatic is a bad thing, though. In the end, something like this is best solved with a bit of A/B testing?. Sit a couple of your audience down, give them the best implementation of each strategy (starting from the website) for some trivialised module and see what they think. If it turns out that being pragmatic does have compromises, I'm not going to argue with the data. I'd still be grumpy, though. ? This of course assumes you care enough to do so. From cmpython at gmail.com Thu Jul 11 14:37:52 2013 From: cmpython at gmail.com (CM) Date: Thu, 11 Jul 2013 11:37:52 -0700 (PDT) Subject: the general development using Python In-Reply-To: References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> <884377a7-7c95-4f2f-a52f-2e47f98b98d0@googlegroups.com> <3cca113a-90cc-445e-b609-2ed757212c4a@googlegroups.com> Message-ID: <3d01aba0-99b8-4b95-8350-173e436cb6e2@googlegroups.com> On Wednesday, July 10, 2013 7:57:11 PM UTC-4, Joshua Landau wrote: > Yeah, but why keep shipping the Python interpreter? If you choose the > installer route, you don't have to keep shipping it -- it's only > downloaded if you need it. If not, then you don't download it again. I admit that not necessarily shipping the Python interpreter is much more efficient if your client base is expected to be a lot of people who already have Python and a lot of the common libraries (like wxPython). I have a fair number of Python 3rd party libraries already on my computer, so for me, it is preferable to just download the .py files the developer has made. For some of the things I have thought about doing, though, the majority of the clients would never even have heard of Python other than Monty or the constrictor. > But I still think my way is better. Perhaps I'm just too pragmatic > about these things. This was triggered by the OP's original request -- > he sounds very much like he doesn't actually want this -- and it seems > to have stuck. I don't know if being pragmatic is a bad thing, though. I'll definitely think about the business angle of doing it the way you suggest. For example, for consulting work where the end user is a technical worker who just need a script to automate some processes, and you expect to be providing a number of scripts over time in this way, a good way to install and setup a Python + GUI toolkit + other likely necessary libraries would be great. For more "product sales to the mass populous", I'm less sure. > In the end, something like this is best solved with a bit of A/B > testing?. Sit a couple of your audience down, give them the best > implementation of each strategy (starting from the website) for some > trivialised module and see what they think. If it turns out that being > pragmatic does have compromises, I'm not going to argue with the data. > I'd still be grumpy, though. Makes sense. I'm dubious about the full validity of A/B testing data without large data sets, but yes, at least one would be getting a feel for it. I can also see what other projects/businesses are doing. Thanks for the input! From rosuav at gmail.com Mon Jul 8 23:07:07 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 9 Jul 2013 13:07:07 +1000 Subject: the general development using Python In-Reply-To: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> Message-ID: On Tue, Jul 9, 2013 at 11:45 AM, wrote: > I have to get back into writing Python but I'm lacking one thing ... a general understanding of how to write applications that can be deployed (either in .exe format or in other formats). That's one last thing you need to un-learn, then :) You distribute Python applications simply as they are - as a .py file (or a collection of .py files), and your users run them. It's really that simple! In fact, deploying to .exe or equivalent would restrict your users to those running a compatible OS (same OS, same word/pointer size (32-bit or 64-bit), possibly other restrictions too), whereas deploying the .py files just requires that they have a compatible Python interpreter installed. Target the three most popular desktop platforms all at once, no Linux/Windows/Mac OS versioning. Target the lesser-known platforms like OS/2 with the same script. And completely eliminate the "compile step", which might take a long time with large projects. (Okay, your code does still get compiled, but the interpreter manages all that for you. All you need to know is that the .pyc files don't need to be distributed.) Python - like most other modern high level languages - is designed to save you the hassle of working with the details. This is another of those hassle-savings. :) ChrisA From joel.goldstick at gmail.com Mon Jul 8 22:13:59 2013 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Mon, 8 Jul 2013 22:13:59 -0400 Subject: the general development using Python In-Reply-To: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> Message-ID: On Mon, Jul 8, 2013 at 9:45 PM, wrote: > all, > > I am unhappy with the general Python documentation and tutorials. I have > worked with Python very little and I'm well aware of the fact that it is a > lower-level language that integrates with the shell. > > I came from a VB legacy background and I've already "un-learned" > everything that I need to (I know, that language stinks, and isn't OOP or > even useful!). > > I have to get back into writing Python but I'm lacking one thing ... a > general understanding of how to write applications that can be deployed > (either in .exe format or in other formats). > > So my issue is basically to understand how to go about writing programs > and compiling them so they can be deployed to less tech-savvy people. > Here's what I think I have to do, in a general sense: > > => Pick a GUI and just run through the tutorials to learn the interfaces > as fast as possible. > > This is all fine and dandy, but more than likely when I do this the people > that I am sending solutions to will, if not receiving a simple .exe file, > receive the package from me and say to themselves "what in the world do I > do with this!?" > > Is there anyway you guys would suggest that I fix this or help them deal > with complex languages like Python and programs written with it? > > thanks guys. > -- > http://mail.python.org/mailman/listinfo/python-list > Why do you want to use python? It isn't a language that can be packaged as an executable. Who are these people who you make software for who need to have a single file? -- Joel Goldstick http://joelgoldstick.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From cmpython at gmail.com Tue Jul 9 00:46:44 2013 From: cmpython at gmail.com (CM) Date: Mon, 8 Jul 2013 21:46:44 -0700 (PDT) Subject: the general development using Python In-Reply-To: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> Message-ID: <33fe3c46-9c2e-481a-8aa9-8411e1c32d02@googlegroups.com> On Monday, July 8, 2013 9:45:16 PM UTC-4, ajetr... at gmail.com wrote: > all, > > > > I am unhappy with the general Python documentation and tutorials. OK. Do you mean the official Python.org docs? Which tutorials? There's a ton out there. > I have worked with Python very little and I'm well aware of the fact that it is a lower-level language that integrates with the shell. I thought it was a high level language. Integrates with the shell? Isn't it just simplest to think of it as a programming language and that's what you need to know? > I came from a VB legacy background and I've already "un-learned" everything that I need to (I know, that language stinks, and isn't OOP or even useful!). On that last point, I think a quick Google search of job postings suggests otherwise. > I have to get back into writing Python but I'm lacking one thing ... I'm guessing it is probably more than *one* thing. But moving along... > So my issue is basically to understand how to go about writing programs and compiling them so they can be deployed to less tech-savvy people. Here's what I think I have to do, in a general sense: > > => Pick a GUI and just run through the tutorials to learn the interfaces as fast as possible. Yes. > This is all fine and dandy, but more than likely when I do this the people > that I am sending solutions to will, if not receiving a simple .exe file, > receive the package from me and say to themselves "what in the world do I do > with this!?" Yes. If they are not Python users, that's right. > Is there anyway you guys would suggest that I fix this or help them deal with > complex languages like Python and programs written with it? Again, "complex language"? It's a programming language, that's it. Anyway, yes: read the first sentence after "Overview" here: https://us.pycon.org/2012/schedule/presentation/393/ The other respondents to your post have a good philosophical point, that it is kind of unfortunate to bundle up a Python program and the whole interpreter when you can just send a much smaller .py file, but in reality, there are a number of cases where doing it is preferred. First, your case with completely unPython-savvy users. Second, if you have a lot of dependencies and it would make it necessary for end users to install all of them for your program to work. In the end, I'm a fan of them. Couple of responses to others in that regard: > There are projects that "bundle" the CPython interpreter with your > project, but this makes those files really big. Maybe 5-20 MB. That's a lot bigger than a few hundred K, but it's not that important to keep size down, really. > Target the three most popular desktop platforms all at once, no > Linux/Windows/Mac OS versioning. Ehhh... There are differences, in, e.g., wxPython between the three platforms, and you can either do different versions or, more aptly, just fix these differences in your code with conditional statements ("if this is Win, do this, else do that"). From rosuav at gmail.com Tue Jul 9 01:03:14 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 9 Jul 2013 15:03:14 +1000 Subject: the general development using Python In-Reply-To: <33fe3c46-9c2e-481a-8aa9-8411e1c32d02@googlegroups.com> References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> <33fe3c46-9c2e-481a-8aa9-8411e1c32d02@googlegroups.com> Message-ID: On Tue, Jul 9, 2013 at 2:46 PM, CM wrote: >> Target the three most popular desktop platforms all at once, no >> Linux/Windows/Mac OS versioning. > > Ehhh... There are differences, in, e.g., wxPython between the three platforms, and you can either do different versions or, more aptly, just fix these differences in your code with conditional statements ("if this is Win, do this, else do that"). Please watch your citations, you quoted several different people without any hint as to who said what :) Yes, there are a few differences. But a *lot* less than there are differences between a Linux executable and a Windows one, or between 32-bit and 64-bit binaries, or between Red Hat and Debian packages, etc, etc, etc. Differences in windowing systems or newlines or path separators will need to be dealt with regardless of the app, but there are a whole pile of additional differences when you distribute binary executables. ChrisA From ian.g.kelly at gmail.com Tue Jul 9 04:41:36 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Tue, 9 Jul 2013 02:41:36 -0600 Subject: the general development using Python In-Reply-To: <33fe3c46-9c2e-481a-8aa9-8411e1c32d02@googlegroups.com> References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> <33fe3c46-9c2e-481a-8aa9-8411e1c32d02@googlegroups.com> Message-ID: On Mon, Jul 8, 2013 at 10:46 PM, CM wrote: >> There are projects that "bundle" the CPython interpreter with your >> project, but this makes those files really big. > > Maybe 5-20 MB. That's a lot bigger than a few hundred K, but it's not that important to keep size down, really. Funny story: at my workplace we solve the distribution problem by placing both the Python interpreter and applications on a network share. I have inherited a program that nonetheless is stored on the network share as a pyInstaller bundle because it is (perceived to be) faster to transfer a single .exe over the network during start-up than all the individual files on demand. From rosuav at gmail.com Tue Jul 9 09:59:31 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 9 Jul 2013 23:59:31 +1000 Subject: the general development using Python In-Reply-To: References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> <33fe3c46-9c2e-481a-8aa9-8411e1c32d02@googlegroups.com> Message-ID: On Tue, Jul 9, 2013 at 6:41 PM, Ian Kelly wrote: > On Mon, Jul 8, 2013 at 10:46 PM, CM wrote: >>> There are projects that "bundle" the CPython interpreter with your >>> project, but this makes those files really big. >> >> Maybe 5-20 MB. That's a lot bigger than a few hundred K, but it's not that important to keep size down, really. > > Funny story: at my workplace we solve the distribution problem by > placing both the Python interpreter and applications on a network > share. I have inherited a program that nonetheless is stored on the > network share as a pyInstaller bundle because it is (perceived to be) > faster to transfer a single .exe over the network during start-up than > all the individual files on demand. It might actually be faster, if the startup cost of a transfer is high - which it very often can be. But it would likely also be faster to transfer a single .zip or .tar.gz for the same benefit; or, depending on requirements, use 'git pull' or scp. ChrisA From joshua.landau.ws at gmail.com Tue Jul 9 17:21:22 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Tue, 9 Jul 2013 22:21:22 +0100 Subject: the general development using Python In-Reply-To: <33fe3c46-9c2e-481a-8aa9-8411e1c32d02@googlegroups.com> References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> <33fe3c46-9c2e-481a-8aa9-8411e1c32d02@googlegroups.com> Message-ID: On 9 July 2013 05:46, CM wrote: *I said*: >> There are projects that "bundle" the CPython interpreter with your >> project, but this makes those files really big. > > Maybe 5-20 MB. That's a lot bigger than a few hundred K, but it's not that important to keep size down, really. Fair enough. It's not something I'd EMail to a friend, though. *Chris Angelico said*: >> Target the three most popular desktop platforms all at once, no >> Linux/Windows/Mac OS versioning. > > Ehhh... There are differences, in, e.g., wxPython between the three platforms, and you can either do different versions or, more aptly, just fix these differences in your code with conditional statements ("if this is Win, do this, else do that"). I agree with Chris -- it doesn't take much to make a package (depending on what you're doing) work on both Windows and Linux. It takes a hell of a lot to make a .exe file work on both. From cmpython at gmail.com Tue Jul 9 19:29:39 2013 From: cmpython at gmail.com (CM) Date: Tue, 9 Jul 2013 16:29:39 -0700 (PDT) Subject: the general development using Python In-Reply-To: References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> <33fe3c46-9c2e-481a-8aa9-8411e1c32d02@googlegroups.com> Message-ID: On Tuesday, July 9, 2013 1:03:14 AM UTC-4, Chris Angelico wrote: > On Tue, Jul 9, 2013 at 2:46 PM, CM wrote: > > >> Target the three most popular desktop platforms all at once, no > > >> Linux/Windows/Mac OS versioning. > > > Ehhh... There are differences, in, e.g., wxPython between the three platforms, and you can either do different versions or, more aptly, just fix these differences in your code with conditional statements ("if this is Win, do this, else do that"). > > > > Please watch your citations, you quoted several different people > without any hint as to who said what :) Uhp, sorry. I was just trimming for space, but good point. > Yes, there are a few differences. But a *lot* less than there are > differences between a Linux executable and a Windows one, or between > 32-bit and 64-bit binaries, or between Red Hat and Debian packages, > etc, etc, etc. Differences in windowing systems or newlines or pat > separators will need to be dealt with regardless of the app, but there > are a whole pile of additional differences when you distribute binary > executables. Agreed. That said, some of whether it is worth the trouble will depend on whether the OP (or whoever) is planning to deploy for all these platforms. If it is just the big three, then it will be some (significant) work at first to get the executables made, but then, with some luck, less work to maintain the various platform versions. But sure, I am not at all disputing the idea that if you have, say, a Python program with wxPython, and the user is easily willing to install (or already has) Python and wxPython, to just send the .py file. It would be ludicrous to not do that. From cmpython at gmail.com Tue Jul 9 19:38:36 2013 From: cmpython at gmail.com (CM) Date: Tue, 9 Jul 2013 16:38:36 -0700 (PDT) Subject: the general development using Python In-Reply-To: References: <51f7f573-e4a6-4349-ae50-21de7780ed8c@googlegroups.com> <33fe3c46-9c2e-481a-8aa9-8411e1c32d02@googlegroups.com> Message-ID: On Tuesday, July 9, 2013 5:21:22 PM UTC-4, Joshua Landau wrote: > On 9 July 2013 05:46, CM wrote: > > Maybe 5-20 MB. That's a lot bigger than a few hundred K, but it's not that important to keep size down, really. > Fair enough. It's not something I'd EMail to a friend, though. Again, agreed. I can't even try to email that; Gmail won't let me. I've had to use Dropbox to transfer files to a client. At this point, I probably *should* have had them install Python and wxPython, but then again, I suspected that operation could have (somehow) gone awkwardly, and I know that packaging up my program for each new version takes about five minutes and one click of a button, then drag the .exe to Dropbox, so maybe not. From saadharana at gmail.com Tue Jul 9 00:50:58 2013 From: saadharana at gmail.com (saadharana) Date: Mon, 8 Jul 2013 21:50:58 -0700 (PDT) Subject: Networked Printer Issues Message-ID: <1373345458893-5024177.post@n6.nabble.com> Hey All, I just built a new PC and networked the printer (just like w/the old setup) but it doesn't seem to work.. I remember when I did it last time it took all of 1 minute to setup but this time the troubleshooting is going on days of my spare time. I'd really like to just get this working. ----- used computers in chennai -- View this message in context: http://python.6.x6.nabble.com/Networked-Printer-Issues-tp5024177.html Sent from the Python - python-list mailing list archive at Nabble.com. From saadharana at gmail.com Tue Jul 9 00:52:19 2013 From: saadharana at gmail.com (saadharana) Date: Mon, 8 Jul 2013 21:52:19 -0700 (PDT) Subject: looking for a new router Message-ID: <1373345539380-5024178.post@n6.nabble.com> Hey i'm looking for a new router. I have no set budget. Only US stores. I have cable internet and few laptops connected to it so it needs to have a strong wireless internet signal. Also i do gaming as well on wireless internet and download many large files. Thank you for the help. ----- used computers in chennai -- View this message in context: http://python.6.x6.nabble.com/looking-for-a-new-router-tp5024178.html Sent from the Python - python-list mailing list archive at Nabble.com. From rosuav at gmail.com Tue Jul 9 00:57:37 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 9 Jul 2013 14:57:37 +1000 Subject: looking for a new router In-Reply-To: <1373345539380-5024178.post@n6.nabble.com> References: <1373345539380-5024178.post@n6.nabble.com> Message-ID: On Tue, Jul 9, 2013 at 2:52 PM, saadharana wrote: > Hey i'm looking for a new router. I have no set budget. Only US stores. I > have cable internet and few laptops connected to it so it needs to have a > strong wireless internet signal. Also i do gaming as well on wireless > internet and download many large files. Thank you for the help. I recommend you go to a small local store that has friendly people and real service, tell them what you're needing, and support local business with your custom. That'll be more helpful to you than asking on a mailing list that's about Python. :) ChrisA From dotancohen at gmail.com Tue Jul 9 01:27:32 2013 From: dotancohen at gmail.com (Dotan Cohen) Date: Tue, 9 Jul 2013 08:27:32 +0300 Subject: looking for a new router In-Reply-To: References: <1373345539380-5024178.post@n6.nabble.com> Message-ID: On Tue, Jul 9, 2013 at 7:57 AM, Chris Angelico wrote: > I recommend you go to a small local store that has friendly people and > real service, tell them what you're needing, and support local > business with your custom. That'll be more helpful to you than asking > on a mailing list that's about Python. :) > > ChrisA Chris, the account that you replied to is just a spam account for his link at the bottom. Helpfully, it appears that Nabble has removed the link and left only the text. -- Dotan Cohen http://gibberish.co.il http://what-is-what.com From rosuav at gmail.com Tue Jul 9 01:28:50 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 9 Jul 2013 15:28:50 +1000 Subject: looking for a new router In-Reply-To: References: <1373345539380-5024178.post@n6.nabble.com> Message-ID: On Tue, Jul 9, 2013 at 3:27 PM, Dotan Cohen wrote: > On Tue, Jul 9, 2013 at 7:57 AM, Chris Angelico wrote: >> I recommend you go to a small local store that has friendly people and >> real service, tell them what you're needing, and support local >> business with your custom. That'll be more helpful to you than asking >> on a mailing list that's about Python. :) >> >> ChrisA > > Chris, the account that you replied to is just a spam account for his > link at the bottom. Helpfully, it appears that Nabble has removed the > link and left only the text. Yes, I guessed that when the great spew of them started coming through. This was the first, so I thought it a legitimate (but mis-directed) post. ChrisA From bruce.kumita at gmail.com Tue Jul 9 01:29:35 2013 From: bruce.kumita at gmail.com (Kumita Bruce) Date: Tue, 9 Jul 2013 13:29:35 +0800 Subject: looking for a new router In-Reply-To: References: <1373345539380-5024178.post@n6.nabble.com> Message-ID: Agree. Sir, this mailing list is for Python discussion. :) On Tue, Jul 9, 2013 at 12:57 PM, Chris Angelico wrote: > On Tue, Jul 9, 2013 at 2:52 PM, saadharana wrote: > > Hey i'm looking for a new router. I have no set budget. Only US stores. I > > have cable internet and few laptops connected to it so it needs to have a > > strong wireless internet signal. Also i do gaming as well on wireless > > internet and download many large files. Thank you for the help. > > I recommend you go to a small local store that has friendly people and > real service, tell them what you're needing, and support local > business with your custom. That'll be more helpful to you than asking > on a mailing list that's about Python. :) > > ChrisA > -- > http://mail.python.org/mailman/listinfo/python-list > -- Cheers, Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From davea at davea.name Tue Jul 9 17:14:36 2013 From: davea at davea.name (Dave Angel) Date: Tue, 09 Jul 2013 17:14:36 -0400 Subject: looking for a new router In-Reply-To: References: <1373345539380-5024178.post@n6.nabble.com> Message-ID: On 07/09/2013 01:29 AM, Kumita Bruce wrote: > Agree. > > Sir, this mailing list is for Python discussion. :) > Save your breath. saadharana and saishreemathi are spambots, and are undoubtedly not "listening." -- DaveA From llanitedave at veawb.coop Tue Jul 9 19:30:09 2013 From: llanitedave at veawb.coop (llanitedave) Date: Tue, 9 Jul 2013 16:30:09 -0700 (PDT) Subject: looking for a new router In-Reply-To: References: <1373345539380-5024178.post@n6.nabble.com> Message-ID: On Tuesday, July 9, 2013 2:14:36 PM UTC-7, Dave Angel wrote: > On 07/09/2013 01:29 AM, Kumita Bruce wrote: > > > Agree. > > > > > > Sir, this mailing list is for Python discussion. :) > > > > > > > Save your breath. saadharana and saishreemathi are spambots, and are > > undoubtedly not "listening." > > > > > > > > > > -- > > DaveA But hey, the plunge router recommendations are great! From skip at pobox.com Tue Jul 9 22:29:22 2013 From: skip at pobox.com (Skip Montanaro) Date: Tue, 9 Jul 2013 21:29:22 -0500 Subject: looking for a new router In-Reply-To: References: <1373345539380-5024178.post@n6.nabble.com> Message-ID: > But hey, the plunge router recommendations are great! Yeah, based on Grant's research, I found a used PC 690-series base on eBay for my wife for her birthday. I think she will be psyched. Lots easier to make keyhole slots with a plunge base. :-) Skip From steve+comp.lang.python at pearwood.info Tue Jul 9 02:49:23 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 09 Jul 2013 06:49:23 GMT Subject: looking for a new router References: Message-ID: <51dbb273$0$6512$c3e8da3$5496439d@news.astraweb.com> On Mon, 08 Jul 2013 21:52:19 -0700, saadharana wrote: > Hey i'm looking for a new router. I recommend this one: http://www.bunnings.com.au/products_product_1350w-aeg-12-router-rt1350e_P6230066.aspx Helpfully-as-ever-ly yrs, -- Steven From rosuav at gmail.com Tue Jul 9 03:09:59 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 9 Jul 2013 17:09:59 +1000 Subject: looking for a new router In-Reply-To: <51dbb273$0$6512$c3e8da3$5496439d@news.astraweb.com> References: <51dbb273$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, Jul 9, 2013 at 4:49 PM, Steven D'Aprano wrote: > On Mon, 08 Jul 2013 21:52:19 -0700, saadharana wrote: > >> Hey i'm looking for a new router. > > I recommend this one: > > http://www.bunnings.com.au/products_product_1350w-aeg-12-router-rt1350e_P6230066.aspx > > > Helpfully-as-ever-ly yrs, Not very helpful actually, Steven. He said "Only US stores". Postage on that would be horrendous! Less-helpfully-than-Steven-ly yrs, ChrisA From invalid at invalid.invalid Tue Jul 9 10:18:12 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Tue, 9 Jul 2013 14:18:12 +0000 (UTC) Subject: looking for a new router References: <51dbb273$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2013-07-09, Steven D'Aprano wrote: > On Mon, 08 Jul 2013 21:52:19 -0700, saadharana wrote: > >> Hey i'm looking for a new router. > > I recommend this one: > > http://www.bunnings.com.au/products_product_1350w-aeg-12-router-rt1350e_P6230066.aspx I just got a Bosch combo model (both plunge and fixed base), and I'm very happy with it: http://www.boschtools.com/Products/Tools/Pages/BoschProductDetail.aspx?pid=1617EVSPK It did take me a few days to find the extra 1/4" collet that came with it since it had worked it's way inside the molded case lid. -- Grant Edwards grant.b.edwards Yow! The PINK SOCKS were at ORIGINALLY from 1952!! gmail.com But they went to MARS around 1953!! From skip at pobox.com Tue Jul 9 10:35:38 2013 From: skip at pobox.com (Skip Montanaro) Date: Tue, 9 Jul 2013 09:35:38 -0500 Subject: looking for a new router In-Reply-To: References: <51dbb273$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: As long as we are wandering off-topic. (What defines "on-topic" for spam?) > I just got a Bosch combo model (both plunge and fixed base), and I'm > very happy with it: > > http://www.boschtools.com/Products/Tools/Pages/BoschProductDetail.aspx?pid=1617EVSPK I have a 1604. Any idea if the plunge base is available separately and will fit my motor? Skip From invalid at invalid.invalid Tue Jul 9 10:48:03 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Tue, 9 Jul 2013 14:48:03 +0000 (UTC) Subject: looking for a new router References: <51dbb273$0$6512$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 2013-07-09, Skip Montanaro wrote: > As long as we are wandering off-topic. (What defines "on-topic" for spam?) > >> I just got a Bosch combo model (both plunge and fixed base), and I'm >> very happy with it: >> >> http://www.boschtools.com/Products/Tools/Pages/BoschProductDetail.aspx?pid=1617EVSPK > > I have a 1604. Any idea if the plunge base is available separately > and will fit my motor? I don't think so. Google gives me the impression that Bosch didn't make a plunge base for the 1604. These threads says a Porter-Cable 693 plunge base will work: http://www.routerforums.com/introductions/41838-bosch-1604-question.html http://forums.finewoodworking.com/fine-woodworking-knots/power-tools-and-machinery/pc-6931-plunge-base -- Grant Edwards grant.b.edwards Yow! Where's the Coke at machine? Tell me a joke!! gmail.com From saadharana at gmail.com Tue Jul 9 00:54:24 2013 From: saadharana at gmail.com (saadharana) Date: Mon, 8 Jul 2013 21:54:24 -0700 (PDT) Subject: Strange networking problems Message-ID: <1373345664845-5024179.post@n6.nabble.com> I have two computers connected through a router, one via cable and running XP, the other via wireless running win 7. This network was set up a while back and then not used for a long time, but at one point it was working, now there seem to be some sort of problems. ----- used computers in chennai -- View this message in context: http://python.6.x6.nabble.com/Strange-networking-problems-tp5024179.html Sent from the Python - python-list mailing list archive at Nabble.com. From saadharana at gmail.com Tue Jul 9 00:55:47 2013 From: saadharana at gmail.com (saadharana) Date: Mon, 8 Jul 2013 21:55:47 -0700 (PDT) Subject: crack a router passcode Message-ID: <1373345747399-5024180.post@n6.nabble.com> I need to crack my router passcode to see what firmware it's running. There's a passcode set but I don't remember it and it's not written down anywhere. ----- used computers in chennai -- View this message in context: http://python.6.x6.nabble.com/crack-a-router-passcode-tp5024180.html Sent from the Python - python-list mailing list archive at Nabble.com. From neilc at norwich.edu Tue Jul 9 08:08:59 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 9 Jul 2013 12:08:59 GMT Subject: crack a router passcode References: Message-ID: On 2013-07-09, saadharana wrote: > I need to crack my router passcode to see what firmware it's > running. There's a passcode set but I don't remember it and > it's not written down anywhere. No you don't. If it's your router and you forgot the password just reset it to factory defaults and reconfigure. -- Neil Cerutti From nikos at superhost.gr Tue Jul 9 09:23:42 2013 From: nikos at superhost.gr (Ferrous Cranus) Date: Tue, 09 Jul 2013 16:23:42 +0300 Subject: crack a router passcode In-Reply-To: References: Message-ID: ???? 9/7/2013 3:08 ??, ?/? Neil Cerutti ??????: > On 2013-07-09, saadharana wrote: >> I need to crack my router passcode to see what firmware it's >> running. There's a passcode set but I don't remember it and >> it's not written down anywhere. > > No you don't. If it's your router and you forgot the password > just reset it to factory defaults and reconfigure. Of course he can do what as you have said, but lets assume it was he router and had no reset button. Could python somehow brute force http://192.168.1.1/login.php giving user and pass trying to guess the password? Could it be able to pass values to the input boxes of router's web login interface? -- What is now proved was at first only imagined! From rosuav at gmail.com Tue Jul 9 09:32:53 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 9 Jul 2013 23:32:53 +1000 Subject: crack a router passcode In-Reply-To: References: Message-ID: On Tue, Jul 9, 2013 at 11:23 PM, Ferrous Cranus wrote: > Could python somehow brute force http://192.168.1.1/login.php giving user > and pass trying to guess the password? > > Could it be able to pass values to the input boxes of router's web login > interface? It certainly could. It's just simple HTTP requests, which Python handles admirably. But this request was sent by a spambot and doesn't need a response. ChrisA From neilc at norwich.edu Tue Jul 9 10:15:29 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 9 Jul 2013 14:15:29 GMT Subject: crack a router passcode References: Message-ID: On 2013-07-09, Chris Angelico wrote: > On Tue, Jul 9, 2013 at 11:23 PM, Ferrous Cranus wrote: >> Could python somehow brute force http://192.168.1.1/login.php giving user >> and pass trying to guess the password? >> >> Could it be able to pass values to the input boxes of router's web login >> interface? > > It certainly could. It's just simple HTTP requests, which Python > handles admirably. But this request was sent by a spambot and doesn't > need a response. FRANK DREBBIN Yes... I know that. Now. -- Neil Cerutti From nikos at superhost.gr Tue Jul 9 10:26:52 2013 From: nikos at superhost.gr (Ferrous Cranus) Date: Tue, 09 Jul 2013 17:26:52 +0300 Subject: crack a router passcode In-Reply-To: References: Message-ID: ???? 9/7/2013 4:32 ??, ?/? Chris Angelico ??????: > On Tue, Jul 9, 2013 at 11:23 PM, Ferrous Cranus wrote: >> Could python somehow brute force http://192.168.1.1/login.php giving user >> and pass trying to guess the password? >> >> Could it be able to pass values to the input boxes of router's web login >> interface? > > It certainly could. It's just simple HTTP requests, which Python > handles admirably. But this request was sent by a spambot and doesn't > need a response. > > ChrisA Seems a real person that responds back, why do you say its a spambot? How is is able to reply if it it one? -- What is now proved was at first only imagined! From davea at davea.name Tue Jul 9 10:46:36 2013 From: davea at davea.name (Dave Angel) Date: Tue, 09 Jul 2013 10:46:36 -0400 Subject: crack a router passcode In-Reply-To: References: Message-ID: On 07/09/2013 10:26 AM, Ferrous Cranus wrote: > ???? 9/7/2013 4:32 ??, ?/? Chris Angelico ??????: >> On Tue, Jul 9, 2013 at 11:23 PM, Ferrous Cranus >> wrote: >>> Could python somehow brute force http://192.168.1.1/login.php giving >>> user >>> and pass trying to guess the password? >>> >>> Could it be able to pass values to the input boxes of router's web login >>> interface? >> >> It certainly could. It's just simple HTTP requests, which Python >> handles admirably. But this request was sent by a spambot and doesn't >> need a response. >> >> ChrisA > > Seems a real person that responds back, why do you say its a spambot? > How is is able to reply if it it one? > Certainly spambots can reply to messages, and some even seem fairly credible while spouting nonsense. But this OP is nothing of the sort. Two aliases recently, saishreemathi saadharana neither of which has done ANY replies that I can see on comp.lang.python -- DaveA From nikos at superhost.gr Tue Jul 9 12:06:42 2013 From: nikos at superhost.gr (Ferrous Cranus) Date: Tue, 09 Jul 2013 19:06:42 +0300 Subject: crack a router passcode In-Reply-To: References: Message-ID: ???? 9/7/2013 5:46 ??, ?/? Dave Angel ??????: > On 07/09/2013 10:26 AM, Ferrous Cranus wrote: >> ???? 9/7/2013 4:32 ??, ?/? Chris Angelico ??????: >>> On Tue, Jul 9, 2013 at 11:23 PM, Ferrous Cranus >>> wrote: >>>> Could python somehow brute force http://192.168.1.1/login.php giving >>>> user >>>> and pass trying to guess the password? >>>> >>>> Could it be able to pass values to the input boxes of router's web >>>> login >>>> interface? >>> >>> It certainly could. It's just simple HTTP requests, which Python >>> handles admirably. But this request was sent by a spambot and doesn't >>> need a response. >>> >>> ChrisA >> >> Seems a real person that responds back, why do you say its a spambot? >> How is is able to reply if it it one? >> > > Certainly spambots can reply to messages, and some even seem fairly > credible while spouting nonsense. > > But this OP is nothing of the sort. Two aliases recently, > > saishreemathi > saadharana > > neither of which has done ANY replies that I can see on comp.lang.python > > What is the reason of a spambot? Spam a usenet forum to gain what? -- What is now proved was at first only imagined! From davea at davea.name Tue Jul 9 16:59:18 2013 From: davea at davea.name (Dave Angel) Date: Tue, 09 Jul 2013 16:59:18 -0400 Subject: crack a router passcode In-Reply-To: References: Message-ID: On 07/09/2013 12:06 PM, Ferrous Cranus wrote: >> > What is the reason of a spambot? Spam a usenet forum to gain what? > Spam is unsolicited advertising. A bot is a robot, or other automated device. So Spambots on a usenet newsgroup send apparently innocent questions that also contain links to their websites, or the trash they're pushing. Somebody here claimed that the trash got stripped from the message before it went out to us. All I know is it was obvious that there were about 8 spam messages, and so I ignored them. They were from one email address (then two), and in rapid succession, and none of them asked anything relevant to this newsgroup. To others it wasn't so obvious, and numerous people have wasted time responding to them. -- DaveA From rosuav at gmail.com Tue Jul 9 22:44:23 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 10 Jul 2013 12:44:23 +1000 Subject: crack a router passcode In-Reply-To: References: Message-ID: On Wed, Jul 10, 2013 at 6:59 AM, Dave Angel wrote: > On 07/09/2013 12:06 PM, Ferrous Cranus wrote: > > > >>> >> What is the reason of a spambot? Spam a usenet forum to gain what? >> > > Spam is unsolicited advertising. A bot is a robot, or other automated > device. So Spambots on a usenet newsgroup send apparently innocent > questions that also contain links to their websites, or the trash they're > pushing. Somebody here claimed that the trash got stripped from the message > before it went out to us. Yeah. The bottom of the original message had these words: "used computers in chennai". The most likely reason for that is that there was an around that, which got stripped at some point (maybe the post got converted from HTML to plain text somewhere). So these postings are being approximately useless, and will hopefully eventually stop. ChrisA From wuwei23 at gmail.com Tue Jul 9 23:18:03 2013 From: wuwei23 at gmail.com (alex23) Date: Wed, 10 Jul 2013 13:18:03 +1000 Subject: crack a router passcode In-Reply-To: References: Message-ID: On 10/07/2013 6:59 AM, Dave Angel wrote: > On 07/09/2013 12:06 PM, Ferrous Cranus wrote: >> What is the reason of a spambot? Spam a usenet forum to gain what? > > All I know is it was obvious that there were about 8 spam messages, and > so I ignored them. They were from one email address (then two), and in > rapid succession, and none of them asked anything relevant to this > newsgroup. To others it wasn't so obvious, and numerous people have > wasted time responding to them. Oh, the irony. From wrw at mac.com Tue Jul 9 09:58:34 2013 From: wrw at mac.com (William Ray Wing) Date: Tue, 09 Jul 2013 09:58:34 -0400 Subject: crack a router passcode In-Reply-To: <1373345747399-5024180.post@n6.nabble.com> References: <1373345747399-5024180.post@n6.nabble.com> Message-ID: On Jul 9, 2013, at 12:55 AM, saadharana wrote: > I need to crack my router passcode to see what firmware it's running. There's > a passcode set but I don't remember it and it's not written down anywhere. > > > This question really isn't appropriate for a python-list, BUT - every router I'm familiar with has a physical reset sequence (typically involving holding the reset button while cycling the power) that will reset everything, including the password to the original factory settings. If all else fails, I'd suggest trying that. -Bill From saadharana at gmail.com Tue Jul 9 00:58:37 2013 From: saadharana at gmail.com (saadharana) Date: Mon, 8 Jul 2013 21:58:37 -0700 (PDT) Subject: Need to build a server computer Message-ID: <1373345917503-5024182.post@n6.nabble.com> Hey, I'm in a bit of a dilemma and I need help. I need to host 2 or 3 Minecraft servers for a fellow gamer and I. However, I am not sure how much memory or what general kind of processor to get. Obviously, I won't need a lot of graphics power or hard drive space for a basic server, but I'm still stuck. It will most likely be running Windows 7 x64, and the max player count will not be more than 20 on each server. Any suggestions? ----- used computers in chennai -- View this message in context: http://python.6.x6.nabble.com/Need-to-build-a-server-computer-tp5024182.html Sent from the Python - python-list mailing list archive at Nabble.com. From saadharana at gmail.com Tue Jul 9 01:06:34 2013 From: saadharana at gmail.com (saadharana) Date: Mon, 8 Jul 2013 22:06:34 -0700 (PDT) Subject: RAM slots problem Message-ID: <1373346394596-5024183.post@n6.nabble.com> I've got some annoying problem with RAM. I was depth cleaning my case, everything regular, it wasn't my first time. And when I put it all together and powered it on, it wasn't working, just beeps fast. But how that happend when I put all back in like it was before?? Later I realised that problem was in position of RAM sticks. I still can't understand what happend, but now computer won't work on every RAM position, and especially not like it was before. ----- used computers in chennai -- View this message in context: http://python.6.x6.nabble.com/RAM-slots-problem-tp5024183.html Sent from the Python - python-list mailing list archive at Nabble.com. From gary.herron at islandtraining.com Tue Jul 9 02:59:42 2013 From: gary.herron at islandtraining.com (Gary Herron) Date: Mon, 08 Jul 2013 23:59:42 -0700 Subject: RAM slots problem In-Reply-To: <1373346394596-5024183.post@n6.nabble.com> References: <1373346394596-5024183.post@n6.nabble.com> Message-ID: <51DBB4DE.2040504@islandtraining.com> On 07/08/2013 10:06 PM, saadharana wrote: > I've got some annoying problem with RAM. I was depth cleaning my case, > everything regular, it wasn't my first time. And when I put it all together > and powered it on, it wasn't working, just beeps fast. But how that happend > when I put all back in like it was before?? Later I realised that problem > was in position of RAM sticks. I still can't understand what happend, but > now computer won't work on every RAM position, and especially not like it > was before. > > > > ----- > used computers in chennai > -- > View this message in context: http://python.6.x6.nabble.com/RAM-slots-problem-tp5024183.html > Sent from the Python - python-list mailing list archive at Nabble.com. This is a PYTHON list -- for discussing issues concerning Python. You'd best ask for HARDWARE help elsewhere. Good luck. From rosuav at gmail.com Tue Jul 9 03:16:11 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 9 Jul 2013 17:16:11 +1000 Subject: RAM slots problem In-Reply-To: <51DBB4DE.2040504@islandtraining.com> References: <1373346394596-5024183.post@n6.nabble.com> <51DBB4DE.2040504@islandtraining.com> Message-ID: On Tue, Jul 9, 2013 at 4:59 PM, Gary Herron wrote: > On 07/08/2013 10:06 PM, saadharana wrote: >> >> I've got some annoying problem with RAM. I was depth cleaning my case, >> everything regular, it wasn't my first time. And when I put it all >> together >> and powered it on, it wasn't working, just beeps fast. But how that >> happend >> when I put all back in like it was before?? Later I realised that problem >> was in position of RAM sticks. I still can't understand what happend, but >> now computer won't work on every RAM position, and especially not like it >> was before. >> >> >> >> ----- >> used computers in chennai >> -- >> View this message in context: >> http://python.6.x6.nabble.com/RAM-slots-problem-tp5024183.html >> Sent from the Python - python-list mailing list archive at Nabble.com. > > > This is a PYTHON list -- for discussing issues concerning Python. You'd best > ask for HARDWARE help elsewhere. As we've all been discovering, these are being sent by a bot. I think the original requests look like they've been lifted from elsewhere in Usenet, but in any case, it doesn't appear that anyone's looking for real help here. ChrisA From saadharana at gmail.com Tue Jul 9 01:09:50 2013 From: saadharana at gmail.com (saadharana) Date: Mon, 8 Jul 2013 22:09:50 -0700 (PDT) Subject: Re-using my SSD drive? Message-ID: <1373346590115-5024184.post@n6.nabble.com> Hi guys, So I heard that once your SSD is installed with windows with a new motherboard, it will be stuck with it forever? So does that mean SSDs and HDDs are pretty much not-recyclable? Thanks in advance ----- used computers in chennai -- View this message in context: http://python.6.x6.nabble.com/Re-using-my-SSD-drive-tp5024184.html Sent from the Python - python-list mailing list archive at Nabble.com. From saadharana at gmail.com Tue Jul 9 01:11:52 2013 From: saadharana at gmail.com (saadharana) Date: Mon, 8 Jul 2013 22:11:52 -0700 (PDT) Subject: How to Clone a Hard Drive Message-ID: <1373346712552-5024185.post@n6.nabble.com> I just recently realized that my laptop's hard drive was going down the road to failure, so I had to clone all the data from the old drive onto a new drive from Newegg. Since I have all the steps fresh in my mind, I figured I'd write a guide on how to clone a hard drive. ----- used computers in chennai -- View this message in context: http://python.6.x6.nabble.com/How-to-Clone-a-Hard-Drive-tp5024185.html Sent from the Python - python-list mailing list archive at Nabble.com. From saadharana at gmail.com Tue Jul 9 01:13:33 2013 From: saadharana at gmail.com (saadharana) Date: Mon, 8 Jul 2013 22:13:33 -0700 (PDT) Subject: USB wont format? Message-ID: <1373346813681-5024186.post@n6.nabble.com> Hi guys; i have here a Scandisk Usb that i formatted it to f32 and tried to use on Linux did not work now i'm trying to use it on windows and it says i have to format and i go format it and it wont format . ----- used computers in chennai -- View this message in context: http://python.6.x6.nabble.com/USB-wont-format-tp5024186.html Sent from the Python - python-list mailing list archive at Nabble.com. From jldunn2000 at gmail.com Tue Jul 9 05:39:21 2013 From: jldunn2000 at gmail.com (loial) Date: Tue, 9 Jul 2013 02:39:21 -0700 (PDT) Subject: How to clean up socket connection to printer Message-ID: I have a socket application that is connecting to a HP printer via port 9100. Occassionally I get a "Connection reset by peer" error which I am trapping and exiting the script with an error message. That works Ok, the issue I have is that the next time I run the script I get "Connection refused" from the printer, which suggests that the printer still thinks the port is is busy, though nothing is printing. I suspect that in some way my socket connection has not been closed correctly? When I get the "Connection rest by peer" error, I attempt to close the port as follows : try: sock.close() del sock except Exception, e: pass Is there anything else I should do in these circumstances to ensure that my socket connection is closed OK? From ulrich.eckhardt at dominolaser.com Tue Jul 9 10:05:08 2013 From: ulrich.eckhardt at dominolaser.com (Ulrich Eckhardt) Date: Tue, 09 Jul 2013 16:05:08 +0200 Subject: How to clean up socket connection to printer In-Reply-To: References: Message-ID: Am 09.07.2013 11:39, schrieb loial: > I have a socket application that is connecting to a HP printer via port 9100. > > Occassionally I get a "Connection reset by peer" error which I am > trapping and exiting the script with an error message. Strange. Why does the remote terminate the connection? > That works Ok, the issue I have is that the next time I run the > script I get "Connection refused" from the printer, which > suggests that the printer still thinks the port is is busy, > though nothing is printing. I suspect that in some way my socket > connection has not been closed correctly? I'm assuming you are using TCP. Getting a "connection refused" rather means that there is no server process that is listening on that port. It sounds a bit as if the printer was kind-of rebooting itself, which first resets the existing connection and then, after a rebooting, opens the port again for connections. Question here: 1. Does the printer accept connections again after some time? 2. Does the printer accept connections if you close and re-open the Python interpreter? 3. Is there actually a limit to the number of concurrent connections? In other words, what happens when you try to create a second connection without closing the first? > When I get the "Connection rest by peer" error, I attempt to close > the port as follows : [...] This is useless, the connection is already closed at that point. Your description suggests that it is a remote problem. I still wouldn't rule out that it is somehow caused by your code though, but without seeing that, it's impossible to tell. Good luck! Uli From jldunn2000 at gmail.com Thu Jul 11 05:28:04 2013 From: jldunn2000 at gmail.com (loial) Date: Thu, 11 Jul 2013 02:28:04 -0700 (PDT) Subject: How to clean up socket connection to printer In-Reply-To: References: Message-ID: Replies to questions : 1. Does the printer accept connections again after some time? Yes, bit seems to vary how long that takes 2. Does the printer accept connections if you close and re-open the Python interpreter? Not after a Connection reset error. The script exits after trapping the "Connection reset by peer" error and it is only when a new instance of the script is kicked off that the "Connection refused" issue is encountered. 3. Is there actually a limit to the number of concurrent connections? In other words, what happens when you try to create a second connection without closing the first? I get the Connction refused error in that scenerio too, but as my script exits after detecting the "Connection reset by peer error" there is only ever one instance of my script running(and therefore one attempt to connect) at a time, Which is why I am wondering whether the connection is closed properly by my code when the script exits afer the "Connection reset by peer" error. Or is it the printer not cleaning up the connection.? From rosuav at gmail.com Thu Jul 11 06:13:42 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 11 Jul 2013 20:13:42 +1000 Subject: How to clean up socket connection to printer In-Reply-To: References: Message-ID: On Thu, Jul 11, 2013 at 7:28 PM, loial wrote: > Replies to questions : > > 1. Does the printer accept connections again after some time? > > Yes, bit seems to vary how long that takes > > 2. Does the printer accept connections if you close and re-open the > Python interpreter? > > Not after a Connection reset error. The script exits after trapping the "Connection reset by peer" error and it is only when a new instance of the script is kicked off that the "Connection refused" issue is encountered. > What's the network between you and the printer like? Are you, for instance, going through an el cheapo NAT router? I've had some routers that have a tendency to just randomly dump their NAT mappings (or so it seems), which results in all manner of fun. The "connection reset" error most likely means that you thought you had a connection but the printer (or a router along the way, shouldn't happen but see above) thought it didn't. You won't actually get the error until you try to send some more data [1], which can result in the variance in time. In fact, if the printer is indeed rebooting (as per Ulrich's suggestion), you could get immediate success (reset, reconnect, succeed), or you could get a delay of anything up to the time it takes to reboot (if you "got in" straight after it went down). What's the longest you've ever seen it take from conn reset to able to connect again? It's almost [2] certainly not that you're failing to properly close the connection, though. I'd be inclined to simply code in a retry loop (NOT instant, chances are you'll get some fast failures and you don't want to spin; a progressive back-off delay is usually appropriate here) and treat the failures as uninteresting. ChrisA [1] Unless you're using TCP keepalive, which you probably aren't. [2] Insane stuff can be done, but hardware is presumed sane until proven otherwise. From jldunn2000 at gmail.com Mon Jul 15 07:22:42 2013 From: jldunn2000 at gmail.com (loial) Date: Mon, 15 Jul 2013 04:22:42 -0700 (PDT) Subject: How to clean up socket connection to printer In-Reply-To: References: Message-ID: <7eb6c167-dad1-49b2-a0cf-1c990ecdbe2a@googlegroups.com> Well, I certainly suspect the customers network connection to the printer which is over a WAN across half of Europe, but proving that is the problem is another matter. I can replicate a "Connection reset by peer" error on own printer by pulling the network cable out of the printer. And again I thereafter get the issue of "Connection refused" for what seems a variable amount of time. But at least I am now reassured that the "Connection Refused" is not due to something my script has not cleaned up. Thanks From saishreemathi at gmail.com Tue Jul 9 08:01:52 2013 From: saishreemathi at gmail.com (saishreemathi) Date: Tue, 9 Jul 2013 05:01:52 -0700 (PDT) Subject: iPhone 5 bluetooth pairing problem Message-ID: <1373371312886-5024212.post@n6.nabble.com> My problem is basically that my iPhone 5 wont pair with any bluetooth devices at all. It won't even see other bluetooth devices and when it actually does, well, it ONLY sees my iMac, it wont pair at all. ----- no credit check mobile phone contracts -- View this message in context: http://python.6.x6.nabble.com/iPhone-5-bluetooth-pairing-problem-tp5024212.html Sent from the Python - python-list mailing list archive at Nabble.com. From saishreemathi at gmail.com Tue Jul 9 08:02:58 2013 From: saishreemathi at gmail.com (saishreemathi) Date: Tue, 9 Jul 2013 05:02:58 -0700 (PDT) Subject: iPhone 4 USB problem Message-ID: <1373371378436-5024213.post@n6.nabble.com> Hello everybody ! I have a problem with my iphone 4. When I connect it to my pc it does not connect as a usb. I mean I can still open it with itunes but not read the files in my computer. So I cant download my photos and videos ! I've tried almost everything I've found by googling but it doesnt work! So can you pleas help , cause this girl is getting very annoyed ----- no credit check mobile phone contracts -- View this message in context: http://python.6.x6.nabble.com/iPhone-4-USB-problem-tp5024213.html Sent from the Python - python-list mailing list archive at Nabble.com. From saishreemathi at gmail.com Tue Jul 9 08:04:06 2013 From: saishreemathi at gmail.com (saishreemathi) Date: Tue, 9 Jul 2013 05:04:06 -0700 (PDT) Subject: iPhone 4s: legal unlock, but was still SIM-locked? Message-ID: <1373371446873-5024215.post@n6.nabble.com> I have a 4s that we used while in Germany. It was Tmobile, and the policy was that once we'd had it for 2 years it would be eligible to be "unlocked" for use elsewhere. I brought it back to the states and am finally attempting to unlock it. (It's unusable to us as a phone right now, we use Verizon, which uses no Sim cards, so it's all a moot point..) I don't need it unlocked, but figured I better do it in case I lose the paperwork! I had already wiped it clean so my son could use it as a glorified iPod, and thus, there was no screen lock code on it. I thought THIS was the code they were saying to "make sure to de-activate" before resetting" ----- no credit check mobile phone contracts -- View this message in context: http://python.6.x6.nabble.com/iPhone-4s-legal-unlock-but-was-still-SIM-locked-tp5024215.html Sent from the Python - python-list mailing list archive at Nabble.com. From alex.hanga at gmail.com Tue Jul 9 09:30:05 2013 From: alex.hanga at gmail.com (alex.hanga at gmail.com) Date: Tue, 9 Jul 2013 06:30:05 -0700 (PDT) Subject: Reading File Into 2D List Message-ID: <3b5b02ee-90a3-4a19-af96-b83b4a1039a2@googlegroups.com> Hello! I'm new here and fairly new to Python. I am attempting to read a data file into python and adding it to a 2D list to make it easy to use further down the line. My data file is just 7 numbers in a row seperated by commas and each bulk of data is seperated by the sign @ to indicate that the data from this point on should be stored into a new part of the 2D list. 1,1,1,1,1,1,1 2,2,2,2,2,2,2 @ 3,3,3,3,3,3,3 4,4,4,4,4,4,4 After reading the file, the way I imagine the data to be shown would be in this manner: data[0][0] = (1,1,1,1,1,1,1) data[0][1] = (2,2,2,2,2,2,2) data[1][0] = (3,3,3,3,3,3,3) data[1][1] = (4,4,4,4,4,4,4) This way it will be easy to loop across the data when I use it in Blender. My code looks like this; ------------------------------ object_data = [] object_data.append([]) for rows in data.splitlines(): elems = rows.split(',') if elems[0] != "@": object_data[i].append((float(elems[0]),float(elems[1]), float(elems[2]),float(elems[3]),float(elems[4]), float(elems[5]),int(elems[6]))) else: **start on object_data[1][j] and loop over it all again** ------------------------------- I could really use some help as to how I would force my code to not start on object_data[0] on the next iteration in the for loop, but rather on object_data[1]. I'm an avid Matlab user so I imagined this to be done simply by setting i=i+1 in the else part, however this does not work as it complains about the list being out of bounds. Any help is really appreciated! From davea at davea.name Tue Jul 9 10:24:06 2013 From: davea at davea.name (Dave Angel) Date: Tue, 09 Jul 2013 10:24:06 -0400 Subject: Reading File Into 2D List In-Reply-To: <3b5b02ee-90a3-4a19-af96-b83b4a1039a2@googlegroups.com> References: <3b5b02ee-90a3-4a19-af96-b83b4a1039a2@googlegroups.com> Message-ID: On 07/09/2013 09:30 AM, alex.hanga at gmail.com wrote: > Hello! > > I'm new here and fairly new to Python. I am attempting to read a data file into python and adding it to a 2D list to make it easy to use further down the line. > > My data file is just 7 numbers in a row seperated by commas and each bulk of data is seperated by the sign @ to indicate that the data from this point on should be stored into a new part of the 2D list. > > 1,1,1,1,1,1,1 > 2,2,2,2,2,2,2 > @ > 3,3,3,3,3,3,3 > 4,4,4,4,4,4,4 > Perhaps you mean: data = """\ 1,1,1,1,1,1,1 2,2,2,2,2,2,2 @ 3,3,3,3,3,3,3 4,4,4,4,4,4,4 """ > After reading the file, the way I imagine the data to be shown would be in this manner: > > data[0][0] = (1,1,1,1,1,1,1) > data[0][1] = (2,2,2,2,2,2,2) > data[1][0] = (3,3,3,3,3,3,3) > data[1][1] = (4,4,4,4,4,4,4) perhaps you mean: object_data[0][0] == (1,1,1,1,1,1) etc. > > This way it will be easy to loop across the data when I use it in Blender. > > My code looks like this; > > ------------------------------ > object_data = [] > object_data.append([]) You omitted i = 0 > > for rows in data.splitlines(): So exactly what is data? It's not what you say above. Did you get it by doing something like myfile.read() ? > elems = rows.split(',') > if elems[0] != "@": > object_data[i].append((float(elems[0]),float(elems[1]), > float(elems[2]),float(elems[3]),float(elems[4]), > float(elems[5]),int(elems[6]))) > else: > > **start on object_data[1][j] and loop over it all again** > ------------------------------- > > I could really use some help as to how I would force my code to not start on object_data[0] on the next iteration in the for loop, but rather on object_data[1]. I'm an avid Matlab user so I imagined this to be done simply by setting i=i+1 in the else part, however this does not work as it complains about the list being out of bounds. > > Any help is really appreciated! > Replace te **start line with something like: object_data.append([]) i += 1 This assumes a few missing lines, which must have been there or you would have already had runtime errors. For example, you'll need i=0 before the loop. Another comment about the append with all those calls to float(). When you see a line like that, you want to seriously consider making it a loop, or a comprehension, or something. Given that elem is a list of strings, you could convert it to a list of float, then convert that to a tuple (if you really wanted that). At that point, you just append it. All this could be done in one line if you like, something like (untested): object_data[i].append(tuple(map(float, row.split(",")))) -- DaveA From alex.hanga at gmail.com Tue Jul 9 11:07:47 2013 From: alex.hanga at gmail.com (alex.hanga at gmail.com) Date: Tue, 9 Jul 2013 08:07:47 -0700 (PDT) Subject: Reading File Into 2D List In-Reply-To: References: <3b5b02ee-90a3-4a19-af96-b83b4a1039a2@googlegroups.com> Message-ID: > Replace te **start line with something like: > > > > object_data.append([]) > > i += 1 > > > > This assumes a few missing lines, which must have been there or you > > would have already had runtime errors. For example, you'll need i=0 > > before the loop. > > > > Another comment about the append with all those calls to float(). When > > you see a line like that, you want to seriously consider making it a > > loop, or a comprehension, or something. > > > > Given that elem is a list of strings, you could convert it to a list of > > float, then convert that to a tuple (if you really wanted that). At > > that point, you just append it. All this could be done in one line if > > you like, something like (untested): > > > > object_data[i].append(tuple(map(float, row.split(",")))) > > > > > > > > -- > > DaveA Yes, indeed there are a few lines missing. Just the read file and the i=0 pretty much. The help you gave me solved my problem, thank you very much! I know the calls to float() are sub par and quite stupidly made but I'm not that good at python scripting (yet) so I try to keep my functions rather easy to read so I understand what is happening. A bit later I will take the time to make my code more sophisticated, but as of now I just want it to work, hehe. Again, thank you so much! From lo0446 at my.bristol.ac.uk Tue Jul 9 13:03:41 2013 From: lo0446 at my.bristol.ac.uk (L O'Shea) Date: Tue, 9 Jul 2013 10:03:41 -0700 (PDT) Subject: Help with 'self' and 'set_usage' Message-ID: Hi all, I'm interning and have been given the job of extending a program that has been written by someone else. I've never used Python before so it's a bit of a struggle but I've got to say I'm loving the language so far. In on of the scripts there is def set_usage(self,s): self.usage_str = s Could anyone shed some light on this? I can't find mention of this anywhere in any Python documentation or anywhere else in the code where usage_str might be defined. Can you just create variables in that object by writing self.name = "david" self.hobby = "fishing"?? If this is any help it is used later in the program (or in other scripts) like prog.set_usage("""%prog [options] ....""") Thanks for your help, I have spent a good while googling python docs, code snippets and reading through a Python book I have on my desk to no avail. Cheers in advance. From ethan at stoneleaf.us Tue Jul 9 13:19:26 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 09 Jul 2013 10:19:26 -0700 Subject: Help with 'self' and 'set_usage' In-Reply-To: References: Message-ID: <51DC461E.7070905@stoneleaf.us> On 07/09/2013 10:03 AM, L O'Shea wrote: > > Hi all, Howdy! > I'm interning and have been given the job of extending a program that has been written by someone else. I've never used Python before so it's a bit of a struggle but I've got to say I'm loving the language so far. Excellent way to start a question! :) > In one of the scripts there is > > def set_usage(self,s): > self.usage_str = s Careful of whitespace when posting (I fixed that one for you). > Can you just create variables in that object by writing self.name = "david" self.hobby = "fishing"?? Yup, you sure can. -- ~Ethan~ From antoon.pardon at rece.vub.ac.be Tue Jul 9 13:24:54 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Tue, 09 Jul 2013 19:24:54 +0200 Subject: Help with 'self' and 'set_usage' In-Reply-To: References: Message-ID: <51DC4766.3060703@rece.vub.ac.be> Op 09-07-13 19:03, L O'Shea schreef: > Hi all, > I'm interning and have been given the job of extending a program that > has been written by someone else. I've never used Python before so it's > a bit of a struggle but I've got to say I'm loving the language so far. > > In on of the scripts there is > > def set_usage(self,s): > self.usage_str = s > > Could anyone shed some light on this? I can't find mention of this > anywhere in any Python documentation or anywhere else in the code > where usage_str might be defined. Can you just create variables in > that object by writing self.name = "david" self.hobby = "fishing"?? Yes you can. -- Antoon Pardon From robotsondrugs at gmail.com Tue Jul 9 13:32:06 2013 From: robotsondrugs at gmail.com (Andrew Berg) Date: Tue, 09 Jul 2013 12:32:06 -0500 Subject: Help with 'self' and 'set_usage' In-Reply-To: References: Message-ID: <51DC4916.6090404@gmail.com> On 2013.07.09 12:03, L O'Shea wrote: > Could anyone shed some light on this? I can't find mention of this anywhere in any Python documentation or anywhere else in the code where usage_str might be defined. In Python, you don't declare or initialize variables before using them. In the example you gave, that is where usage_str is defined. You simply assign an object to a name or attribute (which may or may not have existed previously). You can then change it to anything else you want; Python is dynamically typed, so the object you assign to it can be of any type. Note: a class with __slots__ defined is the exception to this, but that's a bit of an advanced topic. BTW, you can play with objects in the interactive interpreter. It's a great way to quickly learn how certain things work. -- CPython 3.3.2 | Windows NT 6.2.9200 / FreeBSD 9.1 From lo0446 at my.bristol.ac.uk Wed Jul 10 04:23:47 2013 From: lo0446 at my.bristol.ac.uk (L O'Shea) Date: Wed, 10 Jul 2013 01:23:47 -0700 (PDT) Subject: Help with 'self' and 'set_usage' In-Reply-To: References: Message-ID: <729a285e-497f-4a04-a141-438ebc96997f@googlegroups.com> On Tuesday, 9 July 2013 18:03:41 UTC+1, L O'Shea wrote: > Hi all, > > I'm interning and have been given the job of extending a program that has been written by someone else. I've never used Python before so it's a bit of a struggle but I've got to say I'm loving the language so far. > > > > In on of the scripts there is > > > > def set_usage(self,s): > > self.usage_str = s > > > > Could anyone shed some light on this? I can't find mention of this anywhere in any Python documentation or anywhere else in the code where usage_str might be defined. Can you just create variables in that object by writing self.name = "david" self.hobby = "fishing"?? > > > > If this is any help it is used later in the program (or in other scripts) like > > > > prog.set_usage("""%prog [options] ....""") > > > > Thanks for your help, I have spent a good while googling python docs, code snippets and reading through a Python book I have on my desk to no avail. > > > > Cheers in advance. Thanks for the advice everyone that was really helpful! Gotta love the USA/UK time difference I can post a question here at the end of the day and it be answered by morning - Not that it took any of you very long! Stay tuned for more Python questions :) From fsaldan1 at gmail.com Tue Jul 9 14:09:07 2013 From: fsaldan1 at gmail.com (fsaldan1 at gmail.com) Date: Tue, 9 Jul 2013 11:09:07 -0700 (PDT) Subject: multiprocessing Message-ID: <5edd8f66-2097-4395-8a8c-eba6b83a0f37@googlegroups.com> I am a beginner with Python, coming from R, and I am having problems with parallelization with the multiprocessing module. I know that other people have asked similar questions but the answers were mostly over my head. Here is my problem: I tried to execute code in parallel in two ways: 1) In a plain xyz.py file without calling main() 2) In a xyz.py file that calls main Under 1) I was able to run parallel processes but: a) The whole script runs from the beginning up to the line where p1.start() or p2.start() is called. That is, if I had 10 processes p1, p2, ..., p10 the whole file would be run from the beginning up to the line where the command pX.start() is called. Maybe it has to be that way so that these processes get the environment they need, but I doubt it. b) I was not able to extract a value from the function called. I was able only to use print(). I tried to create a Queue object to get the return values but then I get error messages: > from multiprocessing import * > > print('\nRunning ' + __name__ + "\n") > > from multiprocessing import Process, Queue, freeze_support > freeze_support() # it does not make any difference to run this command or not > > queue1 = Queue() # create a queue object > > > def multiply(a, b, que): > que.put(a * b) > > p = Process(target=multiply, args=(5, 4, queue1)) > p.start() > p.join() Traceback (most recent call last): File "", line 1, in File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\multiprocessing\forking.py", line 350, in main prepare(preparation_data) File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\multiprocessing\forking.py", line 457, in prepare '__parents_main__', file, path_name, etc File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\imp.py", line 175, in load_module return load_source(name, filename, file) File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\imp.py", line 114, in load_source _LoadSourceCompatibility(name, pathname, file).load_module(name) File "", line 586, in _check_name_wrapper File "", line 1024, in load_module File "", line 1005, in load_module File "", line 562, in module_for_loader_wrapper File "", line 870, in _load_module File "", line 313, in _call_with_frames_removed File "C:\Files\Programs\Wush\Python\parallel_test_2.py", line 15, in p.start() File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\multiprocessing\process.py", line 111, in start self._popen = Popen(self) File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\multiprocessing\forking.py", line 216, in __init__ cmd = ' '.join('"%s"' % x for x in get_command_line()) File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\multiprocessing\forking.py", line 328, in get_command_line is not going to be frozen to produce a Windows executable.''') RuntimeError: Attempt to start a new process before the current process has finished its bootstrapping phase. This probably means that you are on Windows and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() ... The "freeze_support()" line can be omitted if the program is not going to be frozen to produce a Windows executable. >>> Under 2) I get problems with pickling. See below from multiprocessing import * def main(): print('\nRunning ' + __name__ + "\n") freeze_support() def f(name): print('hello', name) p = Process(target=f, args=('bob',)) p.start() p.join() if __name__ == '__main__': main() Running __main__ Traceback (most recent call last): File "", line 1, in File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 562, in runfile execfile(filename, namespace) File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 41, in execfile exec(compile(open(filename).read(), filename, 'exec'), namespace) File "C:\Files\Programs\Wush\Python\parallel_test.py", line 18, in main() File "C:\Files\Programs\Wush\Python\parallel_test.py", line 14, in main p.start() File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\multiprocessing\process.py", line 111, in start self._popen = Popen(self) File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\multiprocessing\forking.py", line 248, in __init__ dump(process_obj, to_child, HIGHEST_PROTOCOL) File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\multiprocessing\forking.py", line 166, in dump ForkingPickler(file, protocol).dump(obj) _pickle.PicklingError: Can't pickle : attribute lookup builtins.function failed >>> From ian.g.kelly at gmail.com Tue Jul 9 14:55:32 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Tue, 9 Jul 2013 12:55:32 -0600 Subject: multiprocessing In-Reply-To: <5edd8f66-2097-4395-8a8c-eba6b83a0f37@googlegroups.com> References: <5edd8f66-2097-4395-8a8c-eba6b83a0f37@googlegroups.com> Message-ID: On Tue, Jul 9, 2013 at 12:09 PM, wrote: > I am a beginner with Python, coming from R, and I am having problems with parallelization with the multiprocessing module. I know that other people have asked similar questions but the answers were mostly over my head. > > Here is my problem: I tried to execute code in parallel in two ways: > > 1) In a plain xyz.py file without calling main() > 2) In a xyz.py file that calls main > > Under 1) I was able to run parallel processes but: > > a) The whole script runs from the beginning up to the line where p1.start() or p2.start() is called. That is, if I had 10 processes p1, p2, ..., p10 the whole file would be run from the beginning up to the line where the command pX.start() is called. Maybe it has to be that way so that these processes get the environment they need, but I doubt it. See the multiprocessing programming guidelines at: http://docs.python.org/3/library/multiprocessing.html#multiprocessing-programming In particular, read the section titled "Safe importing of main module" under "17.2.3.2 Windows". The child process needs to import the main module, which means that anything that isn't protected by "if __name__ == '__main__'" is going to get executed in the child process. This also appears to be the cause of the error that you pasted. > Under 2) I get problems with pickling. See below > > > from multiprocessing import * > > > def main(): > > print('\nRunning ' + __name__ + "\n") > > freeze_support() > > def f(name): > print('hello', name) > > p = Process(target=f, args=('bob',)) > p.start() > p.join() > > if __name__ == '__main__': > main() Read the section "More picklability" under the above documentation link. I suspect the problem here is that the target function f that you're using is a closure, and when it tries to find it in the child process it can't, because main() hasn't been called there and so the function isn't defined. Try moving it to the module namespace. From russ.pobox at gmail.com Tue Jul 9 18:01:40 2013 From: russ.pobox at gmail.com (Russel Walker) Date: Tue, 9 Jul 2013 15:01:40 -0700 (PDT) Subject: Recursive class | can you modify self directly? Message-ID: <61751485-a298-403e-8b44-be7cf2504f0e@googlegroups.com> Sorry for the vague title. Probably best to just show you the code that explains it better. This is a simplified example of what I want to do: # THIS DOESN'T WORK from random import choice class Expr(object): """ Expr(expr, op, val) -> an expression object. """ def __init__(self, expr, op='', val=''): self.expr = expr # can be another instance of Expression. self.op = op self.val = val def __str__(self): return ("%s %s %s" % (self.expr, self.op, self.val)).strip() def expand(self): self = Expr(self, choice('+-*/'), choice('12345')) Then I tried messing around with Expr.__new__() and Expr.__init__() but that didn't work. The only way I've got it to work is by doing the expanding part outside of the object, by a method of some other class. But I want the Expr class to be responsible for itself. How can I do this and why doesn't the above work? From ian.g.kelly at gmail.com Tue Jul 9 18:18:00 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Tue, 9 Jul 2013 16:18:00 -0600 Subject: Recursive class | can you modify self directly? In-Reply-To: <61751485-a298-403e-8b44-be7cf2504f0e@googlegroups.com> References: <61751485-a298-403e-8b44-be7cf2504f0e@googlegroups.com> Message-ID: On Tue, Jul 9, 2013 at 4:01 PM, Russel Walker wrote: > Sorry for the vague title. Probably best to just show you the code that explains it better. > > This is a simplified example of what I want to do: > > > # THIS DOESN'T WORK > from random import choice > > class Expr(object): > """ > Expr(expr, op, val) -> an expression object. > """ > > def __init__(self, expr, op='', val=''): > self.expr = expr # can be another instance of Expression. > self.op = op > self.val = val > > def __str__(self): > return ("%s %s %s" % (self.expr, self.op, self.val)).strip() > > def expand(self): > self = Expr(self, choice('+-*/'), choice('12345')) "self" is just a local binding of the object to the name self. You can rebind the name like this as with any other local variable, but that's all it does. It doesn't modify the object in any way, and no other bindings of the same object are affected. If you actually want to modify the current object, you would need to do something like: def expand(self): import copy self.expr = Expr(self.expr, self.op, self.val) self.op = choice('+-*/') self.val = choice('12345') From ian.g.kelly at gmail.com Tue Jul 9 18:20:47 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Tue, 9 Jul 2013 16:20:47 -0600 Subject: Recursive class | can you modify self directly? In-Reply-To: References: <61751485-a298-403e-8b44-be7cf2504f0e@googlegroups.com> Message-ID: On Tue, Jul 9, 2013 at 4:18 PM, Ian Kelly wrote: > If you actually want to modify the current object, you would need to > do something like: > > def expand(self): > import copy > self.expr = Expr(self.expr, self.op, self.val) > self.op = choice('+-*/') > self.val = choice('12345') Minus the "import copy". I modified the code before sending and forgot to remove that. From russ.pobox at gmail.com Wed Jul 10 05:00:54 2013 From: russ.pobox at gmail.com (Russel Walker) Date: Wed, 10 Jul 2013 02:00:54 -0700 (PDT) Subject: Recursive class | can you modify self directly? In-Reply-To: References: <61751485-a298-403e-8b44-be7cf2504f0e@googlegroups.com> Message-ID: On Wednesday, July 10, 2013 12:20:47 AM UTC+2, Ian wrote: > On Tue, Jul 9, 2013 at 4:18 PM, Ian Kelly wrote: > > > If you actually want to modify the current object, you would need to > > > do something like: > > > > > > def expand(self): > > > import copy > > > self.expr = Expr(self.expr, self.op, self.val) > > > self.op = choice('+-*/') > > > self.val = choice('12345') > > > > Minus the "import copy". I modified the code before sending and > > forgot to remove that. Yes! This is exactly it. Thanks :) From davea at davea.name Tue Jul 9 18:30:29 2013 From: davea at davea.name (Dave Angel) Date: Tue, 09 Jul 2013 18:30:29 -0400 Subject: Recursive class | can you modify self directly? In-Reply-To: <61751485-a298-403e-8b44-be7cf2504f0e@googlegroups.com> References: <61751485-a298-403e-8b44-be7cf2504f0e@googlegroups.com> Message-ID: On 07/09/2013 06:01 PM, Russel Walker wrote: > Sorry for the vague title. Probably best to just show you the code that explains it better. > > This is a simplified example of what I want to do: > > > # THIS DOESN'T WORK > from random import choice > > class Expr(object): > """ > Expr(expr, op, val) -> an expression object. > """ > > def __init__(self, expr, op='', val=''): > self.expr = expr # can be another instance of Expression. > self.op = op > self.val = val > > def __str__(self): > return ("%s %s %s" % (self.expr, self.op, self.val)).strip() > > def expand(self): > self = Expr(self, choice('+-*/'), choice('12345')) > > > Then I tried messing around with Expr.__new__() and Expr.__init__() but that didn't work. > > The only way I've got it to work is by doing the expanding part outside of the object, by a method of some other class. But I want the Expr class to be responsible for itself. How can I do this and why doesn't the above work? > That line in method expand() indicates that you have a fundamental misunderstanding of the way that names and objects interact in Python. Names are bound to objects by the assignment operator, but the object itself is not changed. If you want to change the object that self is bound to, then change it by mutating it. You don't change an existing object by binding a name to a new object, whether of the same or different type. The original object remains unchanged. The fact that the name is 'self' is totally irrelevant. a = 45 b = a a = 12 This does NOT change b. No idea what you really wanted, but perhaps it's something like: def expand(self): self.op = choice("+-*/") self.val = choice("12345") No idea what expr is supposed to represent, so I didn't try to manipulate it here. -- DaveA From ethan at stoneleaf.us Tue Jul 9 18:19:00 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 09 Jul 2013 15:19:00 -0700 Subject: Recursive class | can you modify self directly? In-Reply-To: <61751485-a298-403e-8b44-be7cf2504f0e@googlegroups.com> References: <61751485-a298-403e-8b44-be7cf2504f0e@googlegroups.com> Message-ID: <51DC8C54.3040800@stoneleaf.us> On 07/09/2013 03:01 PM, Russel Walker wrote: > > This is a simplified example of what I want to do: > > > # THIS DOESN'T WORK > from random import choice > > class Expr(object): > """ > Expr(expr, op, val) -> an expression object. > """ > > def __init__(self, expr, op='', val=''): > self.expr = expr # can be another instance of Expression. > self.op = op > self.val = val > > def __str__(self): > return ("%s %s %s" % (self.expr, self.op, self.val)).strip() > > def expand(self): > self = Expr(self, choice('+-*/'), choice('12345')) `self` is just a name. In `expand()` you are rebinding the name `self` away from the object and to a new Expr instance. If you want to change `self` the original object you have to do something like: def expand(self): self.op = choice('+-*/') self.val = choice('12345') -- ~Ethan~ From russ.pobox at gmail.com Wed Jul 10 04:58:49 2013 From: russ.pobox at gmail.com (Russel Walker) Date: Wed, 10 Jul 2013 01:58:49 -0700 (PDT) Subject: Recursive class | can you modify self directly? In-Reply-To: <61751485-a298-403e-8b44-be7cf2504f0e@googlegroups.com> References: <61751485-a298-403e-8b44-be7cf2504f0e@googlegroups.com> Message-ID: <42591572-a09b-4362-8da2-d083c6424b4a@googlegroups.com> I didn't do a good job of explaining it cos I didn't want it to be a TLDR; but I could've added a little more. To clarify: Expr is just a way to represent simple arithmetic expressions for a calculator. Because the expression has to be modified and built over time, and evaluated from left to right and not by operator precedence, I thought it would be simpler to do it this way. Say the calculation in the calculator is currently at: 1 + 2 * 3 Now lets expand it out to see how it gets to this point. dial 1: (1) >>> x = Expr(1) # the base case dial +: ((1) + ) >>> x = Expr(x, '+') dial 2: ((1) + 2) >>> x.val = 2 dial *: (((1) + 2) + ) >>> x = Expr(x, '+') dial 3: (((1) + 2) + 3) >>> x.val = 3 I think it's called 'binary tree' but not entirely sure if that's correct. So I think understand what you guys are saying now. There is the name x and the class instance (the object) which exists somewhere in memory that x points to. self is just another name that points to the same object (not self in general but the argument passed to the self parameter when a method is called). However if the code inside the method reassigns self to some other object, it doesn't change the fact that x still refers to the original object. So self is just a local variable (an argument). The name self has no relevance to to the name x other than the fact that they point to the same object. So reassigning self has no effect on x. But modifying the object that self points to, does affect x because it points to the same object. Is this correct? So when you call x.somemethod() it's not passing x as the self argument, it's actually passing the object that x points to as the self argument. And that object has know knowledge of the fact that x points to it, or does it? From tjreedy at udel.edu Wed Jul 10 15:33:25 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 10 Jul 2013 15:33:25 -0400 Subject: Recursive class | can you modify self directly? In-Reply-To: <42591572-a09b-4362-8da2-d083c6424b4a@googlegroups.com> References: <61751485-a298-403e-8b44-be7cf2504f0e@googlegroups.com> <42591572-a09b-4362-8da2-d083c6424b4a@googlegroups.com> Message-ID: On 7/10/2013 4:58 AM, Russel Walker wrote: > There is the name x and the class instance (the object) which exists > somewhere in memory that x points to. self is just another name that > points to the same object (not self in general but the argument > passed to the self parameter when a method is called). However if the > code inside the method reassigns self to some other object, it > doesn't change the fact that x still refers to the original object. > So self is just a local variable (an argument). Yes, parameters *names* are the initial local names of the function. Calling the first parameter of instancemethods 'self' is a convention, not a requirement. > The name self has no > relevance to to the name x other than the fact that they point to the > same object. So reassigning self has no effect on x. But modifying > the object that self points to, does affect x because it points to > the same object. Is this correct? Yes. Multiple names for one objects are 'aliases'. Being able to modify a object with multiple names in different namespaces is both a boon and bug bait. > So when you call x.somemethod() it's not passing x as the self Python does not pass'x': it is 'call-by-object', not 'call-by-name'. > argument, it's actually passing the object that x points to as the > self argument. And that object has know knowledge of the fact that x > points to it, or does it? Some objects (modules, classes, functions) have definition names (.__name__ attributes) that are used in their representations (as in tracebacks). But they have no knowledge of their namespace names. -- Terry Jan Reedy From russ.pobox at gmail.com Wed Jul 10 16:09:27 2013 From: russ.pobox at gmail.com (Russel Walker) Date: Wed, 10 Jul 2013 13:09:27 -0700 (PDT) Subject: Recursive class | can you modify self directly? In-Reply-To: References: <61751485-a298-403e-8b44-be7cf2504f0e@googlegroups.com> <42591572-a09b-4362-8da2-d083c6424b4a@googlegroups.com> Message-ID: On Wednesday, July 10, 2013 9:33:25 PM UTC+2, Terry Reedy wrote: > On 7/10/2013 4:58 AM, Russel Walker wrote: > > > > > There is the name x and the class instance (the object) which exists > > > somewhere in memory that x points to. self is just another name that > > > points to the same object (not self in general but the argument > > > passed to the self parameter when a method is called). However if the > > > code inside the method reassigns self to some other object, it > > > doesn't change the fact that x still refers to the original object. > > > So self is just a local variable (an argument). > > > > Yes, parameters *names* are the initial local names of the function. > > Calling the first parameter of instancemethods 'self' is a convention, > > not a requirement. > > > > > The name self has no > > > relevance to to the name x other than the fact that they point to the > > > same object. So reassigning self has no effect on x. But modifying > > > the object that self points to, does affect x because it points to > > > the same object. Is this correct? > > > > Yes. Multiple names for one objects are 'aliases'. Being able to modify > > a object with multiple names in different namespaces is both a boon and > > bug bait. > > > > > So when you call x.somemethod() it's not passing x as the self > > > > Python does not pass'x': it is 'call-by-object', not 'call-by-name'. > > > > > argument, it's actually passing the object that x points to as the > > > self argument. And that object has know knowledge of the fact that x > > > points to it, or does it? > > > > Some objects (modules, classes, functions) have definition names > > (.__name__ attributes) that are used in their representations (as in > > tracebacks). But they have no knowledge of their namespace names. > > > > > > > > -- > > Terry Jan Reedy Thanks Terry, I did read all that twice just to make sure it sunk in and it was really clear and helpful. From russ.pobox at gmail.com Wed Jul 10 18:50:50 2013 From: russ.pobox at gmail.com (Russel Walker) Date: Wed, 10 Jul 2013 15:50:50 -0700 (PDT) Subject: Recursive class | can you modify self directly? In-Reply-To: <61751485-a298-403e-8b44-be7cf2504f0e@googlegroups.com> References: <61751485-a298-403e-8b44-be7cf2504f0e@googlegroups.com> Message-ID: I've been mucking around with this silly class pretty much the whole day and my eyes are about closing now so this is the solution for now I think. Please feel free to drop any suggestions. I think I mostly just ended up shaving off allot of extraneous responsibility for the class, that and inheriting from list. I wanted to try it and it seems to fit (I think). Maybe tomorrow I'll try it inheriting from instead since that is immutable. Should've probably thought of that before right this moment, as I had quite a time with "maximum recursion depth exceeded". class LRExpression(list): """ Construct for 'left to right' algebraic expressions. """ def __init__(self, *args): list.__init__(self, args[:1]) for x in args[1:]: self.append(x) def __str__(self): return '(%s)' % ' '.join(map(str, self)) def evaluate(self): return eval(str(self)) def append(self, x): if len(self) < 3: list.append(self, x) else: oldself = LRExpression(*self) self.__init__(oldself) self.append(x) def test(): a = LRExpression(1) a.append('+') a.append(2) a.append('*') a.append(3) try: print 'repr:', repr(a) print 'str:', a print "should be 9:", a.evaluate() except Exception as er: print er From ian.g.kelly at gmail.com Wed Jul 10 19:39:04 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 10 Jul 2013 17:39:04 -0600 Subject: Recursive class | can you modify self directly? In-Reply-To: References: <61751485-a298-403e-8b44-be7cf2504f0e@googlegroups.com> Message-ID: On Wed, Jul 10, 2013 at 4:50 PM, Russel Walker wrote: > def append(self, x): > if len(self) < 3: > list.append(self, x) > else: > oldself = LRExpression(*self) > self.__init__(oldself) > self.append(x) It's probably not wise to be re-calling __init__ after the class has been initialized, although it seems to work. I would instead use slice assignment to replace the list contents: self[:] = [oldself] That said, the problem with inheriting from list is that it allows your class to be used in a lot of ways lists can be used that are probably not appropriate. For example, continuing on from your test function: >>> len(a) 3 Why 3? Because that's the length of the list. But the caller would probably expect the length to have something to do with the size of the expression, not the list. >>> '*' in a # Is the '*' operator used in a? True >>> '+' in a # What about the '+' operator? False Again, this happens because the "in" operator is only looking at the list itself, not at sublists. But an expression container would probably be expected to return True for both of those. These cases can of course be easily overridden in the subclass, but what about something harder, like slicing? How should that be expected to work with an LRExpression? This isn't to say that you shouldn't necessarily use a list if it helps you implement the behavior you want, but having the LRExpression *be* a list is probably wrong. Finally, based on what you're doing here, I think you would be better off using something like the OOP Builder pattern. There will be two distinct classes: the "LRExpression" class that composes a tree structure, and an "ExpressionBuilder" class that does the work of actual assembly. It could look something like this: class LRExpression(object): def __init__(self, expr, op, value): self.expr = expr self.op = op self.value = value def __str__(self): return '(%s %s %s)' % (self.expr, self.op, self.value) def evaluate(self): # Subject to the usual warning that eval() should # never be used with untrusted input return eval(str(self)) class ExpressionBuilder(object): def __init__(self): self.parts = [] def append(self, part): self.parts.append(part) def build(self): expr = self.parts[0] for i in xrange(1, len(self.parts), 2): op, value = self.parts[i:i+2] expr = LRExpression(expr, op, value) return expr def test(): a = ExpressionBuilder() a.append(1) a.append('+') a.append(2) a.append('*') a.append(3) expr = a.build() print expr print expr.evaluate() >>> test() ((1 + 2) * 3) 9 From dashley at gmail.com Tue Jul 9 21:29:35 2013 From: dashley at gmail.com (David T. Ashley) Date: Tue, 09 Jul 2013 21:29:35 -0400 Subject: Best Scripting Language for Embedded Work? Message-ID: We develop embedded software for 32-bit micros using Windows as the development platform. We are seeking a general purpose scripting language to automate certain tasks, like cleaning out certain directories of certain types of files in preparation for ZIP'ing, generating certain source files automatically, etc. Selection criteria: a)Should be able to compile the script interpreter as a monolithic executable (no .DLL dependencies, etc.) for easy versioning and distribution of the script interpreter. (Note that I'm not asking that the script be a single executable, just the interpreter. To run a script you'd need both the script and the interpreter. The script would be a text file, and the interpreter would be a single .EXE.) b)Should be extensible, in that one could add commands or library functions to the script interpreter in C (for efficiency), and the whole script interpreter could again consist of a single executable with no other dependencies. (Note that I'm not asking that the script be a single executable, just the interpreter. To run a script you'd need both the script and the interpreter. The script would be a text file, and the interpreter would be a single .EXE.) c)Should be able to spawn compilers and capture the output, do file I/O, and all the other minor expected stuff. d)Graphical capability would be nice. I know that Tcl/Tk would do all of the above, but what about Python? Any other alternatives? Thanks, Dave Ashley From davea at davea.name Tue Jul 9 21:44:48 2013 From: davea at davea.name (Dave Angel) Date: Tue, 09 Jul 2013 21:44:48 -0400 Subject: Best Scripting Language for Embedded Work? In-Reply-To: References: Message-ID: On 07/09/2013 09:29 PM, David T. Ashley wrote: > We develop embedded software for 32-bit micros using Windows as the > development platform. > > We are seeking a general purpose scripting language to automate > certain tasks, like cleaning out certain directories of certain types > of files in preparation for ZIP'ing, generating certain source files > automatically, etc. > > Selection criteria: > > a)Should be able to compile the script interpreter as a monolithic > executable (no .DLL dependencies, etc.) for easy versioning and > distribution of the script interpreter. Oh, I thought you were going to run this on Windows. You're just developing it on Windows, and you want to cross-compile to target some other platform? Which? > (Note that I'm not asking > that the script be a single executable, just the interpreter. To run > a script you'd need both the script and the interpreter. The script > would be a text file, and the interpreter would be a single .EXE.) If you're also constraining your "program" to a single text file, you don't want Python. It uses modules, imported from your script to do much of the work. > > b)Should be extensible, in that one could add commands or library > functions to the script interpreter in C (for efficiency), and the > whole script interpreter could again consist of a single executable > with no other dependencies. (Note that I'm not asking that the script > be a single executable, just the interpreter. To run a script you'd > need both the script and the interpreter. The script would be a text > file, and the interpreter would be a single .EXE.) And that's supposed to HELP efficiency?? > > c)Should be able to spawn compilers and capture the output, do file > I/O, and all the other minor expected stuff. > > d)Graphical capability would be nice. > > I know that Tcl/Tk would do all of the above, I doubt it. > but what about Python? > Any other alternatives? -- DaveA From alain at dpt-info.u-strasbg.fr Wed Jul 10 02:41:41 2013 From: alain at dpt-info.u-strasbg.fr (Alain Ketterlin) Date: Wed, 10 Jul 2013 08:41:41 +0200 Subject: Best Scripting Language for Embedded Work? References: Message-ID: <8761wic3x6.fsf@dpt-info.u-strasbg.fr> David T. Ashley writes: > We develop embedded software for 32-bit micros using Windows as the > development platform. I'll mostly ignore the "Windows" qualifier. If you're stuck with Windows CE or similar, then ask them what they suggest. If you're developing on Windows and deploy on something else (typically, some form of linux), then, well, think again. > We are seeking a general purpose scripting language to automate > certain tasks, like cleaning out certain directories of certain types > of files in preparation for ZIP'ing, generating certain source files > automatically, etc. That's exactly what "shells" are for. They are basically command interpreters providing a nice interface to various system utilities. Busybox is a variant where utilities are actually compiled in, and where you can compile your own stuff in as well. See http://en.wikipedia.org/wiki/Busybox > a)Should be able to compile the script interpreter as a monolithic > executable (no .DLL dependencies, etc.) for easy versioning and > distribution of the script interpreter. A fairly common requirement for shells. > b)Should be extensible, in that one could add commands or library > functions to the script interpreter in C (for efficiency), and the > whole script interpreter could again consist of a single executable > with no other dependencies. It looks like busybox is able to do this (the faq is fairly precise in implementation details---I've never done this but it looks kind of trivial). > c)Should be able to spawn compilers and capture the output, do file > I/O, and all the other minor expected stuff. Trivial for any shell, and probably for any scripting language. > d)Graphical capability would be nice. That's a large can of worms. I don't know any small, self-contained interpreter that includes a full-featured GUI framework. These things usually end up in shared libs, which you explicitely want to avoid... > I know that Tcl/Tk would do all of the above, but what about Python? > Any other alternatives? Most scripting languages have evolved way beyond "scripting" tasks, and usually rely on a fairly extensive set of libraries (either shared libraries or collection of modules). I haven't looked at tcl/tk since I know python, but they can probable be classified in the same category. Of course, expect python supporters to... support python (which, btw, interfaces with tk). Another contender is lua, which has a good reputation regarding embeddability. The answer really depends on your use case. I think you will be better off if you keep the GUI aspect separated from the rest. Here is stackoverflow entry discussing the use guis for (unix) shell scripts: http://stackoverflow.com/questions/928019/how-to-make-a-gui-for-bash-scripts (via google "linux shell script gui"). -- Alain. From auriocus at gmx.de Wed Jul 10 03:03:54 2013 From: auriocus at gmx.de (Christian Gollwitzer) Date: Wed, 10 Jul 2013 09:03:54 +0200 Subject: Best Scripting Language for Embedded Work? In-Reply-To: References: Message-ID: Hi David, you have multi-posted this to comp.lang.tcl. Please don't do that - use crossposting and a proper follow-up (as I did now) Am 10.07.13 03:29, schrieb David T. Ashley: > We develop embedded software for 32-bit micros using Windows as the > development platform. Robert's answer made me hesitate - what exactly is your platform? Are you writing the scripts for the embedded platform, or for Windows, or does the embedded controller run Windows RT or something like this? > We are seeking a general purpose scripting language to automate > certain tasks, like cleaning out certain directories of certain types > of files in preparation for ZIP'ing, generating certain source files > automatically, etc. > > Selection criteria: > > a)Should be able to compile the script interpreter as a monolithic > executable (no .DLL dependencies, etc.) for easy versioning and > distribution of the script interpreter. (Note that I'm not asking > that the script be a single executable, just the interpreter. To run > a script you'd need both the script and the interpreter. The script > would be a text file, and the interpreter would be a single .EXE.) You are referring to tclkits. Yes, it's indeed possible to compile Tcl into a statically linked binary, and C extension packages can be statically linked, too. But tclkits are cheating: There are some files like the standard library (i.e. init.tcl, the clock command etc., unicode encondings...) which are packed into a database and attached to the tclkit. The tclkit then opens itself via the file system to read these files. I don't know if this is possible in a typical embedded system. If you are really talking about Windows, no issue. Python has similar capabilities, look for pyinstaller or py2exe. > b)Should be extensible, in that one could add commands or library > functions to the script interpreter in C (for efficiency), and the > whole script interpreter could again consist of a single executable > with no other dependencies. (Note that I'm not asking that the script > be a single executable, just the interpreter. To run a script you'd > need both the script and the interpreter. The script would be a text > file, and the interpreter would be a single .EXE.) That is possible in Tcl using static packages. In Python I don't know, but I think it should be possible. > c)Should be able to spawn compilers and capture the output, do file > I/O, and all the other minor expected stuff. no real issue > d)Graphical capability would be nice. For GUI Python relies on either Tcl/Tk, wxwidgets, QT, GTK... I think it is possible to assemble these into a packaged binary, too. However it will be a rather large thing in the end. > I know that Tcl/Tk would do all of the above, but what about Python? > Any other alternatives? I think Tcl/Tk is a really good match, especially if you are trying to do GUI, which is very easy there. Also most scripting stuff is available out of the box. Another option might be Lua http://www.lua.org/ Very compact (a static binary is about ~200K), clean synatx, relatively fast. OTOH, the standard library is of course not so extensive as for Tcl or Python. Bash is also an option, because it is some kind of standard for scripting. But on Windows it seems alien and you'd need something like MobaXTerm to get it into a single file. Christian From stefan_ml at behnel.de Wed Jul 10 03:15:49 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 10 Jul 2013 09:15:49 +0200 Subject: Best Scripting Language for Embedded Work? In-Reply-To: References: Message-ID: Christian Gollwitzer, 10.07.2013 09:03: > http://www.lua.org/ > > Very compact (a static binary is about ~200K), clean synatx, relatively > fast. OTOH, the standard library is of course not so extensive as for Tcl > or Python. "not so extensive" is a rather bold understatement. ISTM that most projects that use Lua build their own eco-system around it, or integrate Lua into their existing eco-system, with the obvious drawbacks for code reuse across different projects and user bases. Stefan From auriocus at gmx.de Fri Jul 12 02:52:27 2013 From: auriocus at gmx.de (Christian Gollwitzer) Date: Fri, 12 Jul 2013 08:52:27 +0200 Subject: Best Scripting Language for Embedded Work? In-Reply-To: <44mut85vjpefr1o1g8l2vr9mmaqnd7q101@4ax.com> References: <44mut85vjpefr1o1g8l2vr9mmaqnd7q101@4ax.com> Message-ID: Hi David, Am 12.07.13 03:18, schrieb David T. Ashley: > On Wed, 10 Jul 2013 09:03:54 +0200, Christian Gollwitzer > wrote: > >> Robert's answer made me hesitate - what exactly is your platform? Are >> you writing the scripts for the embedded platform, or for Windows, or >> does the embedded controller run Windows RT or something like this? > > Writing the scripts to run on Windows 7, but the scripts assist in > building the software for an embedded system. Because the embedded > system may be safety-related, it makes sense to be careful about how > the script interpreter is versioned, and a single monolithic > executable makes this easier. I see. In your situation, I'd also keep the sources to the interpreter in your backup. Who knows if you might need them to get it running on some future Windows system. >> >> Python has similar capabilities, look for pyinstaller or py2exe. > > Sorry, I miscommunicated. I don't need to embed the script in the > .EXE. I just want the interpreter to be a single .EXE, so it is > easier to ensure that every software developer has the same version of > the interpreter or that we are using the correct earlier version if an > older product needs to rebuilt in the future. I understand. But nothing prevents you to wrap a simple script loader main script; something like execfile(argv[1]) would do it in Python. pyinstaller has a single-file wrapping mode, where you get the analogue of a starpack in Tcl. But I still think Tcl/Tk is a much better fit. For one thing, pyinstaller makes huge executables. I did this on Linux to wrap a program with matplotlib and Tkinter; I ended up with ~100MB in size. That's because pyinstaller pulled in a lot of system libraries, among them QT, wxWidgets etc. Without them, the program did not run despite it used only Tkinter. In contrast, a starpack wish with just Tcl&Tk is around 2MB in size. Second, Tcl/Tk is much nicer to use interactively. Consider an interactive environment, which woud not include the "ls" command. In Tcl, you can define proc ls {args} { if {[length $args] == 0} { set args * } puts [join [glob -nocomplain {*}$args] \n] } and then, the following commands all work as expected ls ls *.jpg ls *.txt *.jpg The analogue in Python would look like this import glob def ls(*args): for pat in args if len(args)>0 else ('*'): print '\n'.join(glob.glob(pat)) but using it is ugly and inconvenient compared to the shellish way in Tcl: ls() ls('*.txt') ls('*.jpg', '*.txt') Of course this is a contrived example, because all interactive environments already provide ls. But I usually extend my Tcl with similar commands and then just use tkcon to enter them on-the-fly. I also think that for your use case, you will not need to write C extensions. A standard starpack gives you many things out of the box, like looking into ZIP files. Even creating them is just two pages of code. I would suggest that you put together a couple of packages, put them in a lib/ folder, have your main.tcl something like lappend auto_path [file join [file dirname [info script]] lib] package require tkcon tkcon show and bake that together using sdx into a starpack. Then you have a single interpreter, with an interactive console, and you can extend this by Tcl modules and still have a single executable. For example, you can get some inspiration from kbs.tcl, which contains a rather complete make system. It usually builds tclkits, but you can rip the build system off and write your own dependencies with it. Make a package out of it, and then your scripts would just package require make, to get things done which are best describe by a dependency graph. Christian From invalid at invalid.invalid Wed Jul 10 10:15:04 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Wed, 10 Jul 2013 14:15:04 +0000 (UTC) Subject: Best Scripting Language for Embedded Work? References: Message-ID: On 2013-07-10, David T Ashley wrote: > We develop embedded software for 32-bit micros using Windows as the > development platform. > > We are seeking a general purpose scripting language to automate > certain tasks, like cleaning out certain directories of certain types > of files in preparation for ZIP'ing, generating certain source files > automatically, etc. That's what a command shell is for. Since windows doesn't come with a real one, I'd recommend installing Cygwin and then using bash, make, etc. Better yet. Abandon windows and switch to an OS environment intended for software development rather than generating time-wasting power-point presentations. -- Grant Edwards grant.b.edwards Yow! Look! A ladder! at Maybe it leads to heaven, gmail.com or a sandwich! From jhibschman at gmail.com Wed Jul 10 15:38:51 2013 From: jhibschman at gmail.com (Johann Hibschman) Date: Wed, 10 Jul 2013 14:38:51 -0500 Subject: Best Scripting Language for Embedded Work? References: Message-ID: David T. Ashley writes: > We develop embedded software for 32-bit micros using Windows as the > development platform. ... > I know that Tcl/Tk would do all of the above, but what about Python? > Any other alternatives? Given that list, I'd say just use Tcl and be done. You could force the square peg of python into that round hole, but I doubt it'd be worth the effort. Cheers, Johann From dashley at gmail.com Thu Jul 11 21:19:40 2013 From: dashley at gmail.com (David T. Ashley) Date: Thu, 11 Jul 2013 21:19:40 -0400 Subject: Best Scripting Language for Embedded Work? References: Message-ID: On Wed, 10 Jul 2013 14:38:51 -0500, Johann Hibschman wrote: >David T. Ashley writes: > >> We develop embedded software for 32-bit micros using Windows as the >> development platform. >... >> I know that Tcl/Tk would do all of the above, but what about Python? >> Any other alternatives? > >Given that list, I'd say just use Tcl and be done. You could force the >square peg of python into that round hole, but I doubt it'd be worth the >effort. I tend to agree with you. I'm not sure how go monolithic with Python. DTA From stefan_ml at behnel.de Fri Jul 12 00:17:47 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 12 Jul 2013 06:17:47 +0200 Subject: Best Scripting Language for Embedded Work? In-Reply-To: References: Message-ID: David T. Ashley, 12.07.2013 03:19: > On Wed, 10 Jul 2013 14:38:51 -0500, Johann Hibschman wrote: > >> David T. Ashley writes: >> >>> We develop embedded software for 32-bit micros using Windows as the >>> development platform. >> ... >>> I know that Tcl/Tk would do all of the above, but what about Python? >>> Any other alternatives? >> >> Given that list, I'd say just use Tcl and be done. You could force the >> square peg of python into that round hole, but I doubt it'd be worth the >> effort. > > I tend to agree with you. I'm not sure how go monolithic with Python. PyRun? py2exe? Just to name two. Stefan From dashley at gmail.com Thu Jul 11 21:14:35 2013 From: dashley at gmail.com (David T. Ashley) Date: Thu, 11 Jul 2013 21:14:35 -0400 Subject: Best Scripting Language for Embedded Work? References: Message-ID: <80mut8lo1uufb2po2tui1bqlpv0apj6n2k@4ax.com> On Tue, 09 Jul 2013 21:44:48 -0400, Dave Angel wrote: >On 07/09/2013 09:29 PM, David T. Ashley wrote: >> We develop embedded software for 32-bit micros using Windows as the >> development platform. >> >> We are seeking a general purpose scripting language to automate >> certain tasks, like cleaning out certain directories of certain types >> of files in preparation for ZIP'ing, generating certain source files >> automatically, etc. >> >> Selection criteria: >> >> a)Should be able to compile the script interpreter as a monolithic >> executable (no .DLL dependencies, etc.) for easy versioning and >> distribution of the script interpreter. > >Oh, I thought you were going to run this on Windows. You're just >developing it on Windows, and you want to cross-compile to target some >other platform? Which? Sorry, I wasn't very complete. The scripts would run on Windows 7 only, but because they may generate part of the source code for the embedded system, we have to be careful about versioning the interpreter, and a monolithic executable makes that simpler. >> (Note that I'm not asking >> that the script be a single executable, just the interpreter. To run >> a script you'd need both the script and the interpreter. The script >> would be a text file, and the interpreter would be a single .EXE.) > >If you're also constraining your "program" to a single text file, you >don't want Python. It uses modules, imported from your script to do >much of the work. > >> >> b)Should be extensible, in that one could add commands or library >> functions to the script interpreter in C (for efficiency), and the >> whole script interpreter could again consist of a single executable >> with no other dependencies. (Note that I'm not asking that the script >> be a single executable, just the interpreter. To run a script you'd >> need both the script and the interpreter. The script would be a text >> file, and the interpreter would be a single .EXE.) > >And that's supposed to HELP efficiency?? Did I ever claim I wanted efficiency? >> c)Should be able to spawn compilers and capture the output, do file >> I/O, and all the other minor expected stuff. >> >> d)Graphical capability would be nice. >> >> I know that Tcl/Tk would do all of the above, I was able to do this with Tcl/Tk years ago. >I doubt it. > >> but what about Python? >> Any other alternatives? From stefan_ml at behnel.de Wed Jul 10 02:55:46 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 10 Jul 2013 08:55:46 +0200 Subject: Babel i18n package has new maintainers Message-ID: Hi, I've been looking for a Python package for formatting international dates, numbers and monetary values in a web context for a couple of days now, but the only thing that I could find that looked suitable at first sight was the Babel package, of which the last release dates back two years, with stale localisation data and missing Py3 support. http://pythonhosted.org/Babel https://pypi.python.org/pypi/Babel Now, after searching a bit more, I found that Armin Ronacher and Alex Morega have recently taken over that package. https://groups.google.com/forum/#!msg/python-babel/WImO1u-Q8GA/j62ABuuwTpAJ They're now developing the code on github and porting it to Py3.3. https://github.com/mitsuhiko/babel Since it took me quite a while to figure this out, I thought I'd post this here to make others aware of it. I think it's a very important tool to have, and I'm sure Armin and Alex can use some help and testing by others to get a new release out. There's always stuff like reading through and updating the documentation, finding bugs, testing with your existing code base, etc., that everyone can help with. Stefan From steve at pearwood.info Wed Jul 10 04:41:58 2013 From: steve at pearwood.info (Steven D'Aprano) Date: 10 Jul 2013 08:41:58 GMT Subject: Babel i18n package has new maintainers References: Message-ID: <51dd1e56$0$11094$c3e8da3@news.astraweb.com> On Wed, 10 Jul 2013 08:55:46 +0200, Stefan Behnel wrote: > Hi, > > I've been looking for a Python package for formatting international > dates, numbers and monetary values [...] > https://github.com/mitsuhiko/babel > > Since it took me quite a while to figure this out, I thought I'd post > this here to make others aware of it. Thanks Stefan, it's good to see informative posts about Python packages. I personally don't need it, but I'll keep it in mind for the future. -- Steven From matsp999 at aim.com Wed Jul 10 03:55:05 2013 From: matsp999 at aim.com (Mats Peterson) Date: Wed, 10 Jul 2013 07:55:05 +0000 (UTC) Subject: =?UTF-8?Q?Stack=20Overflow=20modera?= =?UTF-8?Q?tor=20=E2=80=9Canimuson=E2=80=9D?= Message-ID: A moderator who calls himself ?animuson? on Stack Overflow doesn?t want to face the truth. He has deleted all my postings regarding Python regular expression matching being extremely slow compared to Perl. Additionally my account has been suspended for 7 days. Such a dickwad. Mats -- Mats Peterson http://alicja.homelinux.com/~mats/ From rosuav at gmail.com Wed Jul 10 04:26:19 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 10 Jul 2013 18:26:19 +1000 Subject: =?windows-1252?Q?Re=3A_Stack_Overflow_moderator_=93animuson=94?= In-Reply-To: References: Message-ID: On Wed, Jul 10, 2013 at 5:55 PM, Mats Peterson wrote: > A moderator who calls himself ?animuson? on Stack Overflow doesn?t > want to face the truth. He has deleted all my postings regarding Python > regular expression matching being extremely slow compared to Perl. > Additionally my account has been suspended for 7 days. Such a dickwad. And this matters... how? What are we supposed to do about it? We are not the Python Secret Underground, which emphatically does not exist. ChrisA From steve at pearwood.info Wed Jul 10 04:37:35 2013 From: steve at pearwood.info (Steven D'Aprano) Date: 10 Jul 2013 08:37:35 GMT Subject: Stack Overflow moderator =?iso-8859-13?b?tGFuaW11c29uoQ==?= References: Message-ID: <51dd1d4f$0$11094$c3e8da3@news.astraweb.com> On Wed, 10 Jul 2013 18:26:19 +1000, Chris Angelico wrote: > On Wed, Jul 10, 2013 at 5:55 PM, Mats Peterson wrote: >> A moderator who calls himself ?animuson? on Stack Overflow doesn?t want >> to face the truth. He has deleted all my postings regarding Python >> regular expression matching being extremely slow compared to Perl. >> Additionally my account has been suspended for 7 days. Such a dickwad. > > And this matters... how? What are we supposed to do about it? We are not > the Python Secret Underground, which emphatically does not exist. Of course not, because if it did From matsp999 at aim.com Wed Jul 10 04:33:03 2013 From: matsp999 at aim.com (Mats Peterson) Date: Wed, 10 Jul 2013 08:33:03 +0000 (UTC) Subject: =?UTF-8?Q?=20Stack=20Overflow=20moder?= =?UTF-8?Q?ator=20=E2=80=9Canimuson=E2=80=9D?= References: <51dd1d4f$0$11094$c3e8da3@news.astraweb.com> Message-ID: Steven D'Aprano wrote: > On Wed, 10 Jul 2013 18:26:19 +1000, Chris Angelico wrote: > >> On Wed, Jul 10, 2013 at 5:55 PM, Mats Peterson wrote: >>> A moderator who calls himself ?animuson? on Stack Overflow doesn?t want >>> to face the truth. He has deleted all my postings regarding Python >>> regular expression matching being extremely slow compared to Perl. >>> Additionally my account has been suspended for 7 days. Such a dickwad. >> >> And this matters... how? What are we supposed to do about it? We are not >> the Python Secret Underground, which emphatically does not exist. > > Of course not, because if it did Because if it did what? :p -- Mats Peterson http://alicja.homelinux.com/~mats/ From rosuav at gmail.com Wed Jul 10 04:50:32 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 10 Jul 2013 18:50:32 +1000 Subject: =?windows-1252?Q?Re=3A_Stack_Overflow_moderator_=93animuson=94?= In-Reply-To: References: <51dd1d4f$0$11094$c3e8da3@news.astraweb.com> Message-ID: On Wed, Jul 10, 2013 at 6:33 PM, Mats Peterson wrote: > Steven D'Aprano wrote: >> On Wed, 10 Jul 2013 18:26:19 +1000, Chris Angelico wrote: >> >>> On Wed, Jul 10, 2013 at 5:55 PM, Mats Peterson wrote: >>>> A moderator who calls himself ?animuson? on Stack Overflow doesn?t want >>>> to face the truth. He has deleted all my postings regarding Python >>>> regular expression matching being extremely slow compared to Perl. >>>> Additionally my account has been suspended for 7 days. Such a dickwad. >>> >>> And this matters... how? What are we supposed to do about it? We are not >>> the Python Secret Underground, which emphatically does not exist. >> >> Of course not, because if it did > > Because if it did what? :p Then they would have full control of this list and what gets pos From matsp999 at aim.com Wed Jul 10 04:46:44 2013 From: matsp999 at aim.com (Mats Peterson) Date: Wed, 10 Jul 2013 08:46:44 +0000 (UTC) Subject: =?UTF-8?Q?=20Stack=20Overflow=20moder?= =?UTF-8?Q?ator=20=E2=80=9Canimuson=E2=80=9D?= References: <51dd1d4f$0$11094$c3e8da3@news.astraweb.com> Message-ID: Chris Angelico wrote: > On Wed, Jul 10, 2013 at 6:33 PM, Mats Peterson wrote: >> Steven D'Aprano wrote: >>> On Wed, 10 Jul 2013 18:26:19 +1000, Chris Angelico wrote: >>> >>>> On Wed, Jul 10, 2013 at 5:55 PM, Mats Peterson wrote: >>>>> A moderator who calls himself ?animuson? on Stack Overflow doesn?t want >>>>> to face the truth. He has deleted all my postings regarding Python >>>>> regular expression matching being extremely slow compared to Perl. >>>>> Additionally my account has been suspended for 7 days. Such a dickwad. >>>> >>>> And this matters... how? What are we supposed to do about it? We are not >>>> the Python Secret Underground, which emphatically does not exist. >>> >>> Of course not, because if it did >> >> Because if it did what? :p > > Then they would have full control of this list and what gets pos Ahhh.... so this is pos, right? Telling the truth? Interesting. -- Mats Peterson http://alicja.homelinux.com/~mats/ From ian.g.kelly at gmail.com Wed Jul 10 05:12:04 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 10 Jul 2013 03:12:04 -0600 Subject: =?windows-1252?Q?Re=3A_Stack_Overflow_moderator_=93animuson=94?= In-Reply-To: References: <51dd1d4f$0$11094$c3e8da3@news.astraweb.com> Message-ID: On Wed, Jul 10, 2013 at 2:46 AM, Mats Peterson wrote: >> Then they would have full control of this list and what gets pos > > Ahhh.... so this is pos, right? Telling the truth? Interesting. I don't know what you mean by that, but since the joke appears to have flown over your head, I'll explain it. Steven's "pos" was clearly meant to be the word "posted", before his sentence got cut off by the Python Secret Underground. From joshua at landau.ws Wed Jul 10 05:52:12 2013 From: joshua at landau.ws (Joshua Landau) Date: Wed, 10 Jul 2013 10:52:12 +0100 Subject: =?UTF-8?Q?Re=3A_Stack_Overflow_moderator_=E2=80=9Canimuson=E2=80=9D?= In-Reply-To: References: <51dd1d4f$0$11094$c3e8da3@news.astraweb.com> Message-ID: On 10 July 2013 10:12, Ian Kelly wrote: > On Wed, Jul 10, 2013 at 2:46 AM, Mats Peterson wrote: >>> Then they would have full control of this list and what gets pos >> >> Ahhh.... so this is pos, right? Telling the truth? Interesting. > > I don't know what you mean by that, but since the joke appears to have > flown over your head, I'll explain it. Steven's "pos" was clearly > mea What? I don't understand. From robert.kern at gmail.com Wed Jul 10 07:08:18 2013 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 10 Jul 2013 12:08:18 +0100 Subject: Stack Overflow moderator =?UTF-8?B?4oCcYW5pbXVzb27igJ0=?= In-Reply-To: References: <51dd1d4f$0$11094$c3e8da3@news.astraweb.com> Message-ID: On 2013-07-10 10:52, Joshua Landau wrote: > On 10 July 2013 10:12, Ian Kelly wrote: >> On Wed, Jul 10, 2013 at 2:46 AM, Mats Peterson wrote: >>>> Then they would have full control of this list and what gets pos >>> >>> Ahhh.... so this is pos, right? Telling the truth? Interesting. >> >> I don't know what you mean by that, but since the joke appears to have >> flown over your head, I'll explain it. Steven's "pos" was clearly >> mea > > What? I don't understand. Look, it's perfectly obvi From joshua at landau.ws Wed Jul 10 07:15:08 2013 From: joshua at landau.ws (Joshua Landau) Date: Wed, 10 Jul 2013 12:15:08 +0100 Subject: =?UTF-8?Q?Re=3A_Stack_Overflow_moderator_=E2=80=9Canimuson=E2=80=9D?= In-Reply-To: References: <51dd1d4f$0$11094$c3e8da3@news.astraweb.com> Message-ID: Google Groups is writing about your recently sent mail to "Joshua Landau". Unfortunately this address has been discontinued from usage for the foreseeable future. The sent message is displayed below: On 10 July 2013 12:08, Robert Kern wrote: > > On 2013-07-10 10:52, Joshua Landau wrote: >> >> >> On 10 July 2013 10:12, Ian Kelly wrote: >>> >>> >>> >>> >>> On Wed, Jul 10, 2013 at 2:46 AM, Mats Peterson wrote: >>>>> >>>>> >>>>> Then they would have full control of this list and what gets pos >>>>> >>>> >>>> >>>> >>>> >>>> Ahhh.... so this is pos, right? Telling the truth? Interesting. >>>> >>> >>> >>> >>> >>> I don't know what you mean by that, but since the joke appears to have >>> >>> flown over your head, I'll explain it. Steven's "pos" was clearly >>> >>> mea >>> >> >> >> >> >> What? I don't understand. >> > > > > > Look, it's perfectly obvi > > > From skip at pobox.com Wed Jul 10 07:52:34 2013 From: skip at pobox.com (Skip Montanaro) Date: Wed, 10 Jul 2013 06:52:34 -0500 Subject: =?UTF-8?Q?Re=3A_Stack_Overflow_moderator_=E2=80=9Canimuson=E2=80=9D?= In-Reply-To: References: <51dd1d4f$0$11094$c3e8da3@news.astraweb.com> Message-ID: > ... meant to be the word "posted", before his sentence got cut off by the > Python Secret Underground. Argh! That which shall not be named! Please, for the sake of all that is right, please only use the initials, PS From matsp999 at aim.com Wed Jul 10 05:13:15 2013 From: matsp999 at aim.com (Mats Peterson) Date: Wed, 10 Jul 2013 09:13:15 +0000 (UTC) Subject: =?UTF-8?Q?=20Stack=20Overflow=20moder?= =?UTF-8?Q?ator=20=E2=80=9Canimuson=E2=80=9D?= References: <51dd1d4f$0$11094$c3e8da3@news.astraweb.com> Message-ID: Ian Kelly wrote: > On Wed, Jul 10, 2013 at 2:46 AM, Mats Peterson wrote: >>> Then they would have full control of this list and what gets pos >> >> Ahhh.... so this is pos, right? Telling the truth? Interesting. > > I don't know what you mean by that, but since the joke appears to have > flown over your head, I'll explain it. Steven's "pos" was clearly > meant to be the word "posted", before his sentence got cut off by the > Python Secret Underground. Being snotty, are we? -- Mats Peterson http://alicja.homelinux.com/~mats/ From steve+comp.lang.python at pearwood.info Wed Jul 10 11:01:26 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 10 Jul 2013 15:01:26 GMT Subject: =?UTF-8?Q?=20Stack=20Overflow=20moder?= ator =?iso-8859-13?b?tGFuaW11c29uoQ==?= References: <51dd1d4f$0$11094$c3e8da3@news.astraweb.com> Message-ID: <51dd7746$0$9505$c3e8da3$5496439d@news.astraweb.com> On Wed, 10 Jul 2013 08:46:44 +0000, Mats Peterson wrote: > Chris Angelico wrote: >> On Wed, Jul 10, 2013 at 6:33 PM, Mats Peterson >> wrote: >>> Steven D'Aprano wrote: >>>> On Wed, 10 Jul 2013 18:26:19 +1000, Chris Angelico wrote: [...] >>>>> And this matters... how? What are we supposed to do about it? We are >>>>> not the Python Secret Underground, which emphatically does not >>>>> exist. >>>> >>>> Of course not, because if it did >>> >>> Because if it did what? :p >> >> Then they would have full control of this list and what gets pos > > Ahhh.... so this is pos, right? Telling the truth? Interesting. Mats, I fear you have misunderstood. If the Python Secret Underground existed, which it most certainly does not, it would absolutely not have the power to censor people's emails or cut them off in the middle of -- Steven From cmpython at gmail.com Thu Jul 11 03:36:50 2013 From: cmpython at gmail.com (CM) Date: Thu, 11 Jul 2013 00:36:50 -0700 (PDT) Subject: =?windows-1252?Q?Re=3A_Stack_Overflow_moder_ator_=93animuson=94?= In-Reply-To: <51dd7746$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <51dd1d4f$0$11094$c3e8da3@news.astraweb.com> <51dd7746$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Wednesday, July 10, 2013 11:01:26 AM UTC-4, Steven D'Aprano wrote: > Mats, I fear you have misunderstood. If the Python Secret Underground > existed, which it most certainly does not, it would absolutely not have > the power to censor people's emails or cut them off in the middle of > *That's* the Python Secret Underground's special power? That's it?! Cutting people off in the middle of an email? I mean how lame can From matsp999 at aim.com Wed Jul 10 04:50:28 2013 From: matsp999 at aim.com (Mats Peterson) Date: Wed, 10 Jul 2013 08:50:28 +0000 (UTC) Subject: =?UTF-8?Q?=20Stack=20Overflow=20moder?= =?UTF-8?Q?ator=20=E2=80=9Canimuson=E2=80=9D?= References: <51dd1d4f$0$11094$c3e8da3@news.astraweb.com> Message-ID: Chris Angelico wrote: > On Wed, Jul 10, 2013 at 6:33 PM, Mats Peterson wrote: >> Steven D'Aprano wrote: >>> On Wed, 10 Jul 2013 18:26:19 +1000, Chris Angelico wrote: >>> >>>> On Wed, Jul 10, 2013 at 5:55 PM, Mats Peterson wrote: >>>>> A moderator who calls himself ?animuson? on Stack Overflow doesn?t want >>>>> to face the truth. He has deleted all my postings regarding Python >>>>> regular expression matching being extremely slow compared to Perl. >>>>> Additionally my account has been suspended for 7 days. Such a dickwad. >>>> >>>> And this matters... how? What are we supposed to do about it? We are not >>>> the Python Secret Underground, which emphatically does not exist. >>> >>> Of course not, because if it did >> >> Because if it did what? :p > > Then they would have full control of this list and what gets pos Have you ever compared the regular expression performance between Perl and Python? If not, keep quiet. Mats -- Mats Peterson http://alicja.homelinux.com/~mats/ From rosuav at gmail.com Wed Jul 10 05:10:29 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 10 Jul 2013 19:10:29 +1000 Subject: =?windows-1252?Q?Re=3A_Stack_Overflow_moderator_=93animuson=94?= In-Reply-To: References: <51dd1d4f$0$11094$c3e8da3@news.astraweb.com> Message-ID: On Wed, Jul 10, 2013 at 6:50 PM, Mats Peterson wrote: > Have you ever compared the regular expression performance between Perl > and Python? If not, keep quiet. I think I can see why you were suspended. You and jmf should have a lot of fun together, I think. ChrisA From matsp999 at aim.com Wed Jul 10 04:32:25 2013 From: matsp999 at aim.com (Mats Peterson) Date: Wed, 10 Jul 2013 08:32:25 +0000 (UTC) Subject: =?UTF-8?Q?=20Stack=20Overflow=20moder?= =?UTF-8?Q?ator=20=E2=80=9Canimuson=E2=80=9D?= References: Message-ID: Chris Angelico wrote: > On Wed, Jul 10, 2013 at 5:55 PM, Mats Peterson wrote: >> A moderator who calls himself ?animuson? on Stack Overflow doesn?t >> want to face the truth. He has deleted all my postings regarding Python >> regular expression matching being extremely slow compared to Perl. >> Additionally my account has been suspended for 7 days. Such a dickwad. > > And this matters... how? What are we supposed to do about it? We are > not the Python Secret Underground, which emphatically does not exist. > > ChrisA You aren?t supposed to do squat about it. I just wanted to mention it to the Python evangelists, who need to see the light. Mats -- Mats Peterson http://alicja.homelinux.com/~mats/ From rosuav at gmail.com Wed Jul 10 04:46:35 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 10 Jul 2013 18:46:35 +1000 Subject: =?windows-1252?Q?Re=3A_Stack_Overflow_moderator_=93animuson=94?= In-Reply-To: References: Message-ID: On Wed, Jul 10, 2013 at 6:32 PM, Mats Peterson wrote: > Chris Angelico wrote: >> On Wed, Jul 10, 2013 at 5:55 PM, Mats Peterson wrote: >>> A moderator who calls himself ?animuson? on Stack Overflow doesn?t >>> want to face the truth. He has deleted all my postings regarding Python >>> regular expression matching being extremely slow compared to Perl. >>> Additionally my account has been suspended for 7 days. Such a dickwad. >> >> And this matters... how? What are we supposed to do about it? We are >> not the Python Secret Underground, which emphatically does not exist. >> >> ChrisA > > You aren?t supposed to do squat about it. I just wanted to mention it > to the Python evangelists, who need to see the light. Your post hits me squarely between the eyes, leaving an "Uhh?"-shaped hole. ChrisA From matsp999 at aim.com Wed Jul 10 04:43:18 2013 From: matsp999 at aim.com (Mats Peterson) Date: Wed, 10 Jul 2013 08:43:18 +0000 (UTC) Subject: =?UTF-8?Q?=20Stack=20Overflow=20moder?= =?UTF-8?Q?ator=20=E2=80=9Canimuson=E2=80=9D?= References: Message-ID: Chris Angelico wrote: > On Wed, Jul 10, 2013 at 6:32 PM, Mats Peterson wrote: >> Chris Angelico wrote: >>> On Wed, Jul 10, 2013 at 5:55 PM, Mats Peterson wrote: >>>> A moderator who calls himself ?animuson? on Stack Overflow doesn?t >>>> want to face the truth. He has deleted all my postings regarding Python >>>> regular expression matching being extremely slow compared to Perl. >>>> Additionally my account has been suspended for 7 days. Such a dickwad. >>> >>> And this matters... how? What are we supposed to do about it? We are >>> not the Python Secret Underground, which emphatically does not exist. >>> >>> ChrisA >> >> You aren?t supposed to do squat about it. I just wanted to mention it >> to the Python evangelists, who need to see the light. > > Your post hits me squarely between the eyes, leaving an "Uhh?"-shaped hole. > > ChrisA I fear you don?t even know what a regular expression is. Then this will of course not affect you. Mats -- Mats Peterson http://alicja.homelinux.com/~mats/ From torriem at gmail.com Thu Jul 11 00:20:42 2013 From: torriem at gmail.com (Michael Torrie) Date: Wed, 10 Jul 2013 22:20:42 -0600 Subject: Stack Overflow moderator =?UTF-8?B?4oCcYW5pbXVzb27igJ0=?= In-Reply-To: References: Message-ID: <51DE329A.3070609@gmail.com> On 07/10/2013 02:43 AM, Mats Peterson wrote: > I fear you don?t even know what a regular expression is. Then this will > of course not affect you. Hmm, and your stack exchange posts had a similar tone, hmm? I for one have never ready any of your posts on this forum before, so it looks like you've come here solely to troll. Perhaps your time would be better spent contributing a faster regex engine to the Python standard library. From ian.g.kelly at gmail.com Wed Jul 10 05:00:48 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 10 Jul 2013 03:00:48 -0600 Subject: =?windows-1252?Q?Re=3A_Stack_Overflow_moderator_=93animuson=94?= In-Reply-To: References: Message-ID: On Wed, Jul 10, 2013 at 2:32 AM, Mats Peterson wrote: > Chris Angelico wrote: >> On Wed, Jul 10, 2013 at 5:55 PM, Mats Peterson wrote: >>> A moderator who calls himself ?animuson? on Stack Overflow doesn?t >>> want to face the truth. He has deleted all my postings regarding Python >>> regular expression matching being extremely slow compared to Perl. >>> Additionally my account has been suspended for 7 days. Such a dickwad. >> >> And this matters... how? What are we supposed to do about it? We are >> not the Python Secret Underground, which emphatically does not exist. >> >> ChrisA > > You aren?t supposed to do squat about it. I just wanted to mention it > to the Python evangelists, who need to see the light. It's well known that regular expressions are slow. I wouldn't know how they compare to Perl, which I don't use. From matsp999 at aim.com Wed Jul 10 04:42:39 2013 From: matsp999 at aim.com (Mats Peterson) Date: Wed, 10 Jul 2013 08:42:39 +0000 (UTC) Subject: =?UTF-8?Q?=20Stack=20Overflow=20moder?= =?UTF-8?Q?ator=20=E2=80=9Canimuson=E2=80=9D?= References: Message-ID: Chris Angelico wrote: > On Wed, Jul 10, 2013 at 5:55 PM, Mats Peterson wrote: >> A moderator who calls himself ?animuson? on Stack Overflow doesn?t >> want to face the truth. He has deleted all my postings regarding Python >> regular expression matching being extremely slow compared to Perl. >> Additionally my account has been suspended for 7 days. Such a dickwad. > > And this matters... how? What are we supposed to do about it? We are > not the Python Secret Underground, which emphatically does not exist. > > ChrisA I fear you don?t even know what a regular expression is. Then this of course won?t affect you whatsoever. Mats -- Mats Peterson http://alicja.homelinux.com/~mats/ From rosuav at gmail.com Wed Jul 10 05:02:48 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 10 Jul 2013 19:02:48 +1000 Subject: =?windows-1252?Q?Re=3A_Stack_Overflow_moderator_=93animuson=94?= In-Reply-To: References: Message-ID: On Wed, Jul 10, 2013 at 6:42 PM, Mats Peterson wrote: > Chris Angelico wrote: >> On Wed, Jul 10, 2013 at 5:55 PM, Mats Peterson wrote: >>> A moderator who calls himself ?animuson? on Stack Overflow doesn?t >>> want to face the truth. He has deleted all my postings regarding Python >>> regular expression matching being extremely slow compared to Perl. >>> Additionally my account has been suspended for 7 days. Such a dickwad. >> >> And this matters... how? What are we supposed to do about it? We are >> not the Python Secret Underground, which emphatically does not exist. >> >> ChrisA > > I fear you don?t even know what a regular expression is. Then this of > course won?t affect you whatsoever. I know what regular expressions are. I've used them in Perl, PHP, JavaScript, Python, C++, Pike, and numerous text editors (which may have been backed by one of the above languages, or may have been something else). Doesn't change the fact that I have no idea what the significance is of your post. ChrisA From ian.g.kelly at gmail.com Wed Jul 10 05:06:39 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 10 Jul 2013 03:06:39 -0600 Subject: =?windows-1252?Q?Re=3A_Stack_Overflow_moderator_=93animuson=94?= In-Reply-To: References: Message-ID: On Wed, Jul 10, 2013 at 2:42 AM, Mats Peterson wrote: > Chris Angelico wrote: >> On Wed, Jul 10, 2013 at 5:55 PM, Mats Peterson wrote: >>> A moderator who calls himself ?animuson? on Stack Overflow doesn?t >>> want to face the truth. He has deleted all my postings regarding Python >>> regular expression matching being extremely slow compared to Perl. >>> Additionally my account has been suspended for 7 days. Such a dickwad. >> >> And this matters... how? What are we supposed to do about it? We are >> not the Python Secret Underground, which emphatically does not exist. >> >> ChrisA > > I fear you don?t even know what a regular expression is. Then this of > course won?t affect you whatsoever. Troll. A quick search of my Gmail archives turns up 18 threads in which Chris has used the phrase "regular expression". From matsp999 at aim.com Wed Jul 10 05:01:08 2013 From: matsp999 at aim.com (Mats Peterson) Date: Wed, 10 Jul 2013 09:01:08 +0000 (UTC) Subject: =?UTF-8?Q?=20Stack=20Overflow=20moder?= =?UTF-8?Q?ator=20=E2=80=9Canimuson=E2=80=9D?= References: Message-ID: Chris Angelico wrote: > On Wed, Jul 10, 2013 at 6:42 PM, Mats Peterson wrote: >> Chris Angelico wrote: >>> On Wed, Jul 10, 2013 at 5:55 PM, Mats Peterson wrote: >>>> A moderator who calls himself ?animuson? on Stack Overflow doesn?t >>>> want to face the truth. He has deleted all my postings regarding Python >>>> regular expression matching being extremely slow compared to Perl. >>>> Additionally my account has been suspended for 7 days. Such a dickwad. >>> >>> And this matters... how? What are we supposed to do about it? We are >>> not the Python Secret Underground, which emphatically does not exist. >>> >>> ChrisA >> >> I fear you don?t even know what a regular expression is. Then this of >> course won?t affect you whatsoever. > > I know what regular expressions are. I've used them in Perl, PHP, > JavaScript, Python, C++, Pike, and numerous text editors (which may > have been backed by one of the above languages, or may have been > something else). Doesn't change the fact that I have no idea what the > significance is of your post. > > ChrisA You do? And you haven't noticed the inferior performance of regular expressions in Python compared to Perl? Then you obviously haven't used them a lot. Mats -- Mats Peterson http://alicja.homelinux.com/~mats/ From rosuav at gmail.com Wed Jul 10 05:39:05 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 10 Jul 2013 19:39:05 +1000 Subject: =?windows-1252?Q?Re=3A_Stack_Overflow_moderator_=93animuson=94?= In-Reply-To: References: Message-ID: On Wed, Jul 10, 2013 at 7:01 PM, Mats Peterson wrote: > Chris Angelico wrote: >> I know what regular expressions are. I've used them in Perl, PHP, >> JavaScript, Python, C++, Pike, and numerous text editors (which may >> have been backed by one of the above languages, or may have been >> something else). Doesn't change the fact that I have no idea what the >> significance is of your post. >> >> ChrisA > > You do? And you haven't noticed the inferior performance of regular > expressions in Python compared to Perl? Then you obviously haven't > used them a lot. That would be correct. Why have I not used them all that much? Because Python has way better ways of doing many things. Regexps are notoriously hard to debug, largely because a nonmatching regex can't give much information about _where_ it failed to match, and when I parse strings, it's more often with (s)scanf notation instead - stuff like this (Pike example as Python doesn't, afaik, have scanf support): > data="Hello, world! I am number 42."; > sscanf(data,"Hello, %s! I am number %d.",foo,x); (3) Result: 2 > foo; (4) Result: "world" > x; (5) Result: 42 Or a more complicated example: sscanf(Stdio.File("/proc/meminfo")->read(),"%{%s: %d%*s\n%}",array data); mapping meminfo=(mapping)data; That builds up a mapping (Pike terminology for what Python calls a dict) with the important information out of /proc/meminfo, something like this: ([ "MemTotal": 2026144, "MemFree": 627652, "Buffers": 183572, "Cached": 380724, ..... etc etc ]) So, no. I haven't figured out that Perl's regular expressions outperform Python's or Pike's or SciTE's, because I simply don't need them all that much. With sscanf, I can at least get a partial match, which tells me where to look for the problem. ChrisA From matsp999 at aim.com Wed Jul 10 08:22:11 2013 From: matsp999 at aim.com (Mats Peterson) Date: Wed, 10 Jul 2013 12:22:11 +0000 (UTC) Subject: =?UTF-8?Q?=20Stack=20Overflow=20moder?= =?UTF-8?Q?ator=20=E2=80=9Canimuson=E2=80=9D?= References: Message-ID: Chris Angelico wrote: > On Wed, Jul 10, 2013 at 7:01 PM, Mats Peterson wrote: >> Chris Angelico wrote: >>> I know what regular expressions are. I've used them in Perl, PHP, >>> JavaScript, Python, C++, Pike, and numerous text editors (which may >>> have been backed by one of the above languages, or may have been >>> something else). Doesn't change the fact that I have no idea what the >>> significance is of your post. >>> >>> ChrisA >> >> You do? And you haven't noticed the inferior performance of regular >> expressions in Python compared to Perl? Then you obviously haven't >> used them a lot. > > That would be correct. Why have I not used them all that much? Because > Python has way better ways of doing many things. Regexps are > notoriously hard to debug, largely because a nonmatching regex can't > give much information about _where_ it failed to match, and when I > parse strings, it's more often with (s)scanf notation instead - stuff > like this (Pike example as Python doesn't, afaik, have scanf support): > >> data="Hello, world! I am number 42."; >> sscanf(data,"Hello, %s! I am number %d.",foo,x); > (3) Result: 2 >> foo; > (4) Result: "world" >> x; > (5) Result: 42 > > Or a more complicated example: > > sscanf(Stdio.File("/proc/meminfo")->read(),"%{%s: %d%*s\n%}",array data); > mapping meminfo=(mapping)data; > > That builds up a mapping (Pike terminology for what Python calls a > dict) with the important information out of /proc/meminfo, something > like this: > > ([ > "MemTotal": 2026144, > "MemFree": 627652, > "Buffers": 183572, > "Cached": 380724, > ..... etc etc > ]) > > So, no. I haven't figured out that Perl's regular expressions > outperform Python's or Pike's or SciTE's, because I simply don't need > them all that much. With sscanf, I can at least get a partial match, > which tells me where to look for the problem. > > ChrisA You're showing by these examples what regular expressions mean to you. Mats -- Mats Peterson http://alicja.homelinux.com/~mats/ From torriem at gmail.com Thu Jul 11 00:26:04 2013 From: torriem at gmail.com (Michael Torrie) Date: Wed, 10 Jul 2013 22:26:04 -0600 Subject: Stack Overflow moderator =?windows-1252?Q?=93animuson=94?= In-Reply-To: References: Message-ID: <51DE33DC.3080600@gmail.com> On 07/10/2013 06:22 AM, Mats Peterson wrote: > You're showing by these examples what regular expressions mean to you. Chris is showing no such thing. And you are simply trolling. What do you want us to do, fall down and worship you and admit that Python is a horrible language and we should all use Perl? What is your point? Are you looking for an admission of Python's slowness compared to Perl? If so, then yes it's slower. For what I need it for, it does not matter. Maybe you could write a better regex engine for python. Then we'd have the best of both worlds. From matsp999 at aim.com Wed Jul 10 05:03:24 2013 From: matsp999 at aim.com (Mats Peterson) Date: Wed, 10 Jul 2013 09:03:24 +0000 (UTC) Subject: =?UTF-8?Q?=20Stack=20Overflow=20moder?= =?UTF-8?Q?ator=20=E2=80=9Canimuson=E2=80=9D?= References: Message-ID: Ian Kelly wrote: > On Wed, Jul 10, 2013 at 2:42 AM, Mats Peterson wrote: >> Chris Angelico wrote: >>> On Wed, Jul 10, 2013 at 5:55 PM, Mats Peterson wrote: >>>> A moderator who calls himself ?animuson? on Stack Overflow doesn?t >>>> want to face the truth. He has deleted all my postings regarding Python >>>> regular expression matching being extremely slow compared to Perl. >>>> Additionally my account has been suspended for 7 days. Such a dickwad. >>> >>> And this matters... how? What are we supposed to do about it? We are >>> not the Python Secret Underground, which emphatically does not exist. >>> >>> ChrisA >> >> I fear you don?t even know what a regular expression is. Then this of >> course won?t affect you whatsoever. > > Troll. A quick search of my Gmail archives turns up 18 threads in > which Chris has used the phrase "regular expression". Not a troll. It's just hard to convince Python users that their beloved language would have inferior regular expression performance to Perl. Mats -- Mats Peterson http://alicja.homelinux.com/~mats/ From antoon.pardon at rece.vub.ac.be Wed Jul 10 07:14:41 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Wed, 10 Jul 2013 13:14:41 +0200 Subject: Stack Overflow moderator =?UTF-8?B?4oCcYW5pbXVzb27igJ0=?= In-Reply-To: References: Message-ID: <51DD4221.8000501@rece.vub.ac.be> Op 10-07-13 11:03, Mats Peterson schreef: > Not a troll. It's just hard to convince Python users that their beloved > language would have inferior regular expression performance to Perl. All right, you have convinced me. Now what? Why should I care? -- Antoon Pardon From joshua at landau.ws Wed Jul 10 07:19:52 2013 From: joshua at landau.ws (Joshua Landau) Date: Wed, 10 Jul 2013 12:19:52 +0100 Subject: =?UTF-8?Q?Re=3A_Stack_Overflow_moderator_=E2=80=9Canimuson=E2=80=9D?= In-Reply-To: <51DD4221.8000501@rece.vub.ac.be> References: <51DD4221.8000501@rece.vub.ac.be> Message-ID: On 10 July 2013 12:14, Antoon Pardon wrote: > Op 10-07-13 11:03, Mats Peterson schreef: >> Not a troll. It's just hard to convince Python users that their beloved >> language would have inferior regular expression performance to Perl. > > All right, you have convinced me. Now what? Why should I care? Isn't it obvious? Regex-based macros! From matsp999 at aim.com Wed Jul 10 08:01:35 2013 From: matsp999 at aim.com (Mats Peterson) Date: Wed, 10 Jul 2013 12:01:35 +0000 (UTC) Subject: =?UTF-8?Q?=20Stack=20Overflow=20moder?= =?UTF-8?Q?ator=20=E2=80=9Canimuson=E2=80=9D?= References: Message-ID: Antoon Pardon wrote: > Op 10-07-13 11:03, Mats Peterson schreef: >> Not a troll. It's just hard to convince Python users that their beloved >> language would have inferior regular expression performance to Perl. > > All right, you have convinced me. Now what? Why should I care? > Right. Why should you. And who cares about you? Mats -- Mats Peterson http://alicja.homelinux.com/~mats/ From joshua at landau.ws Wed Jul 10 08:13:41 2013 From: joshua at landau.ws (Joshua Landau) Date: Wed, 10 Jul 2013 13:13:41 +0100 Subject: =?UTF-8?Q?Re=3A_Stack_Overflow_moderator_=E2=80=9Canimuson=E2=80=9D?= In-Reply-To: References: Message-ID: On 10 July 2013 13:01, Mats Peterson wrote: > Antoon Pardon wrote: >> Op 10-07-13 11:03, Mats Peterson schreef: >>> Not a troll. It's just hard to convince Python users that their beloved >>> language would have inferior regular expression performance to Perl. >> >> All right, you have convinced me. Now what? Why should I care? >> > > Right. Why should you. And who cares about you? Not the Python Se From steve+comp.lang.python at pearwood.info Wed Jul 10 12:03:18 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 10 Jul 2013 16:03:18 GMT Subject: StackOverflowmoderator =?iso-8859-13?b?tGFuaW11c29uoQ==?= References: Message-ID: <51dd85c6$0$9505$c3e8da3$5496439d@news.astraweb.com> On Wed, 10 Jul 2013 09:03:24 +0000, Mats Peterson wrote: > Not a troll. It's just hard to convince Python users that their beloved > language would have inferior regular expression performance to Perl. I can't speak for others, but I've known for many years that Python's regex implementation was slower than Perl's. So what? Do you have a patch? -- Steven From steve at pearwood.info Wed Jul 10 05:00:23 2013 From: steve at pearwood.info (Steven D'Aprano) Date: 10 Jul 2013 09:00:23 GMT Subject: Stack Overflow moderator =?iso-8859-13?b?tGFuaW11c29uoQ==?= References: Message-ID: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> On Wed, 10 Jul 2013 07:55:05 +0000, Mats Peterson wrote: > A moderator who calls himself ?animuson? on Stack Overflow doesn?t want > to face the truth. He has deleted all my postings regarding Python > regular expression matching being extremely slow compared to Perl. That's by design. We don't want to make the same mistake as Perl, where every problem is solved by a regular expression: http://neilk.net/blog/2000/06/01/abigails-regex-to-test-for-prime-numbers/ so we deliberately make regexes as slow as possible so that programmers will look for a better way to solve their problem. If you check the source code for the re engine, you'll find that for certain regexes, it busy-waits for anything up to 30 seconds at a time, deliberately wasting cycles. The same with Unicode. We hate French people, you see, and so in an effort to drive everyone back to ASCII-only text, Python 3.3 introduces some memory optimizations that ensures that Unicode strings work correctly and are up to four times smaller than they used to be. You should get together with jmfauth, who has discovered our dastardly plot and keeps posting benchmarks showing how on carefully contrived micro- benchmarks using a beta version of Python 3.3, non-ASCII string operations can be marginally slower than in 3.2. > Additionally my account has been suspended for 7 days. Such a dickwad. I cannot imagine why he would have done that. -- Steven From square.steve at gmail.com Wed Jul 10 07:18:37 2013 From: square.steve at gmail.com (Steve Simmons) Date: Wed, 10 Jul 2013 12:18:37 +0100 Subject: =?UTF-8?Q?Re=3A_Stack_Overflow_moderator_=E2=80=9Canimuson=E2=80=9D?= In-Reply-To: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> Message-ID: <6c8f5f84-6b3a-4491-8234-3b80fbbbda19@email.android.com> Steven D'Aprano wrote: >On Wed, 10 Jul 2013 07:55:05 +0000, Mats Peterson wrote: > >> A moderator who calls himself ?animuson? on Stack Overflow doesn?t >want >> to face the truth. He has deleted all my postings regarding Python >> regular expression matching being extremely slow compared to Perl. > >That's by design. We don't want to make the same mistake as Perl, where > >every problem is solved by a regular expression: > >http://neilk.net/blog/2000/06/01/abigails-regex-to-test-for-prime-numbers/ > >so we deliberately make regexes as slow as possible so that programmers > >will look for a better way to solve their problem. If you check the >source code for the re engine, you'll find that for certain regexes, it > >busy-waits for anything up to 30 seconds at a time, deliberately >wasting >cycles. > >The same with Unicode. We hate French people, you see, and so in an >effort to drive everyone back to ASCII-only text, Python 3.3 introduces > >some memory optimizations that ensures that Unicode strings work >correctly and are up to four times smaller than they used to be. You >should get together with jmfauth, who has discovered our dastardly plot > >and keeps posting benchmarks showing how on carefully contrived micro- >benchmarks using a beta version of Python 3.3, non-ASCII string >operations can be marginally slower than in 3.2. > > >> Additionally my account has been suspended for 7 days. Such a >dickwad. > >I cannot imagine why he would have done that. > > >-- >Steven >-- >http://mail.python.org/mailman/listinfo/python-list :-) Thank you. Sent from a Galaxy far far away -------------- next part -------------- An HTML attachment was scrubbed... URL: From matsp999 at aim.com Wed Jul 10 08:13:14 2013 From: matsp999 at aim.com (Mats Peterson) Date: Wed, 10 Jul 2013 12:13:14 +0000 (UTC) Subject: =?UTF-8?Q?=20Stack=20Overflow=20moder?= =?UTF-8?Q?ator=20=E2=80=9Canimuson=E2=80=9D?= References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> Message-ID: Steven D'Aprano wrote: > On Wed, 10 Jul 2013 07:55:05 +0000, Mats Peterson wrote: > >> A moderator who calls himself ?animuson? on Stack Overflow doesn?t want >> to face the truth. He has deleted all my postings regarding Python >> regular expression matching being extremely slow compared to Perl. > > That's by design. We don't want to make the same mistake as Perl, where > every problem is solved by a regular expression: > > http://neilk.net/blog/2000/06/01/abigails-regex-to-test-for-prime-numbers/ > > so we deliberately make regexes as slow as possible so that programmers > will look for a better way to solve their problem. If you check the > source code for the re engine, you'll find that for certain regexes, it > busy-waits for anything up to 30 seconds at a time, deliberately wasting > cycles. > > The same with Unicode. We hate French people, you see, and so in an > effort to drive everyone back to ASCII-only text, Python 3.3 introduces > some memory optimizations that ensures that Unicode strings work > correctly and are up to four times smaller than they used to be. You > should get together with jmfauth, who has discovered our dastardly plot > and keeps posting benchmarks showing how on carefully contrived micro- > benchmarks using a beta version of Python 3.3, non-ASCII string > operations can be marginally slower than in 3.2. > > >> Additionally my account has been suspended for 7 days. Such a dickwad. > > I cannot imagine why he would have done that. > > You're obviously trying hard to be funny. It fails miserably. Mats -- Mats Peterson http://alicja.homelinux.com/~mats/ From rosuav at gmail.com Wed Jul 10 08:22:51 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 10 Jul 2013 22:22:51 +1000 Subject: =?windows-1252?Q?Re=3A_Stack_Overflow_moderator_=93animuson=94?= In-Reply-To: References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> Message-ID: On Wed, Jul 10, 2013 at 10:13 PM, Mats Peterson wrote: > Steven D'Aprano wrote: >> On Wed, 10 Jul 2013 07:55:05 +0000, Mats Peterson wrote: >> >>> A moderator who calls himself ?animuson? on Stack Overflow doesn?t want >>> to face the truth. He has deleted all my postings regarding Python >>> regular expression matching being extremely slow compared to Perl. >> >> That's by design. We don't want to make the same mistake as Perl, where >> every problem is solved by a regular expression: >> >> http://neilk.net/blog/2000/06/01/abigails-regex-to-test-for-prime-numbers/ >> >> so we deliberately make regexes as slow as possible so that programmers >> will look for a better way to solve their problem. If you check the >> source code for the re engine, you'll find that for certain regexes, it >> busy-waits for anything up to 30 seconds at a time, deliberately wasting >> cycles. >> >> The same with Unicode. We hate French people, you see, and so in an >> effort to drive everyone back to ASCII-only text, Python 3.3 introduces >> some memory optimizations that ensures that Unicode strings work >> correctly and are up to four times smaller than they used to be. You >> should get together with jmfauth, who has discovered our dastardly plot >> and keeps posting benchmarks showing how on carefully contrived micro- >> benchmarks using a beta version of Python 3.3, non-ASCII string >> operations can be marginally slower than in 3.2. >> >> >>> Additionally my account has been suspended for 7 days. Such a dickwad. >> >> I cannot imagine why he would have done that. >> >> > > You're obviously trying hard to be funny. It fails miserably. Either that or it's funny only to other Australians. ChrisA From pscott209 at gmail.com Wed Jul 10 08:27:53 2013 From: pscott209 at gmail.com (Paul Scott) Date: Wed, 10 Jul 2013 14:27:53 +0200 Subject: Stack Overflow moderator =?windows-1252?Q?=93animuson=94?= In-Reply-To: References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> Message-ID: <51DD5349.9080901@gmail.com> On 10/07/2013 14:22, Chris Angelico wrote: > Either that or it's funny only to other Australians. ChrisA As a South African, I found it funny too, but then again, we often get confused. From skip at pobox.com Wed Jul 10 08:35:10 2013 From: skip at pobox.com (Skip Montanaro) Date: Wed, 10 Jul 2013 07:35:10 -0500 Subject: =?UTF-8?Q?Re=3A_Stack_Overflow_moderator_=E2=80=9Canimuson=E2=80=9D?= In-Reply-To: References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> Message-ID: > Either that or it's funny only to other Australians. Or the Dutch. S From joshua at landau.ws Wed Jul 10 08:39:15 2013 From: joshua at landau.ws (Joshua Landau) Date: Wed, 10 Jul 2013 13:39:15 +0100 Subject: =?UTF-8?Q?Re=3A_Stack_Overflow_moderator_=E2=80=9Canimuson=E2=80=9D?= In-Reply-To: References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> Message-ID: On 10 July 2013 13:35, Skip Montanaro wrote: >> Either that or it's funny only to other Australians. > > Or the Dutch. Or us Brits. From skip at pobox.com Wed Jul 10 09:12:02 2013 From: skip at pobox.com (Skip Montanaro) Date: Wed, 10 Jul 2013 08:12:02 -0500 Subject: =?UTF-8?Q?Re=3A_Stack_Overflow_moderator_=E2=80=9Canimuson=E2=80=9D?= In-Reply-To: References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> Message-ID: On Wed, Jul 10, 2013 at 7:39 AM, Joshua Landau wrote: > On 10 July 2013 13:35, Skip Montanaro wrote: >>> Either that or it's funny only to other Australians. >> >> Or the Dutch. > > Or us Brits. Hells bells... It appears everyone found it funny except the trolls. S From memilanuk at gmail.com Wed Jul 10 10:34:10 2013 From: memilanuk at gmail.com (memilanuk) Date: Wed, 10 Jul 2013 07:34:10 -0700 Subject: Stack Overflow moderator =?windows-1252?Q?=93animuson=94?= In-Reply-To: References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> Message-ID: On 07/10/2013 05:39 AM, Joshua Landau wrote: > On 10 July 2013 13:35, Skip Montanaro wrote: >>> Either that or it's funny only to other Australians. >> >> Or the Dutch. > > Or us Brits. > Or the Yanks... Normally I kill-file threads like this pretty early on, but I have to admit - I'm enjoying watching y'all play with the troll this time ;) From wuwei23 at gmail.com Wed Jul 10 22:04:48 2013 From: wuwei23 at gmail.com (alex23) Date: Thu, 11 Jul 2013 12:04:48 +1000 Subject: Stack Overflow moderator =?UTF-8?B?4oCcYW5pbXVzb27igJ0=?= In-Reply-To: References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> Message-ID: On 10/07/2013 10:13 PM, Mats Peterson wrote: > You're obviously trying hard to be funny. It fails miserably. It's obvious that you are quite familiar with miserableness. Also obvious is that animuson did the world of StackOverflow quite the favour. If only e moderated this list... From joshua at landau.ws Wed Jul 10 11:54:02 2013 From: joshua at landau.ws (Joshua Landau) Date: Wed, 10 Jul 2013 16:54:02 +0100 Subject: =?UTF-8?Q?Re=3A_Stack_Overflow_moderator_=E2=80=9Canimuson=E2=80=9D?= In-Reply-To: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> Message-ID: On 10 July 2013 10:00, Steven D'Aprano wrote: > On Wed, 10 Jul 2013 07:55:05 +0000, Mats Peterson wrote: > >> A moderator who calls himself ?animuson? on Stack Overflow doesn?t want >> to face the truth. He has deleted all my postings regarding Python >> regular expression matching being extremely slow compared to Perl. > > That's by design. We don't want to make the same mistake as Perl, where > every problem is solved by a regular expression: > > http://neilk.net/blog/2000/06/01/abigails-regex-to-test-for-prime-numbers/ > > so we deliberately make regexes as slow as possible so that programmers > will look for a better way to solve their problem. If you check the > source code for the re engine, you'll find that for certain regexes, it > busy-waits for anything up to 30 seconds at a time, deliberately wasting > cycles. I hate to sound like this but do you realise that this is exactly what you're arguing for when saying that sum() shouldn't use "+="? (There is no spite in the above sentence, but it sounds like there is. There is however no way obvious to me to remove it without changing the sentence's meaning.) > The same with Unicode. We hate French people, And for good damn reason too. They're ruining our language, ? mon avis. > you see, and so in an > effort to drive everyone back to ASCII-only text, Python 3.3 introduces > some memory optimizations that ensures that Unicode strings work > correctly and are up to four times smaller than they used to be. You > should get together with jmfauth, who has discovered our dastardly plot > and keeps posting benchmarks showing how on carefully contrived micro- > benchmarks using a beta version of Python 3.3, non-ASCII string > operations can be marginally slower than in 3.2. From ethan at stoneleaf.us Wed Jul 10 12:18:27 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 10 Jul 2013 09:18:27 -0700 Subject: Stack Overflow moderator =?UTF-8?B?4oCcYW5pbXVzb27igJ0=?= In-Reply-To: References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> Message-ID: <51DD8953.60709@stoneleaf.us> On 07/10/2013 08:54 AM, Joshua Landau wrote: > On 10 July 2013 10:00, Steven D'Aprano wrote: >> On Wed, 10 Jul 2013 07:55:05 +0000, Mats Peterson wrote: >> >>> A moderator who calls himself ?animuson? on Stack Overflow doesn?t want >>> to face the truth. He has deleted all my postings regarding Python >>> regular expression matching being extremely slow compared to Perl. >> >> That's by design. We don't want to make the same mistake as Perl, where >> every problem is solved by a regular expression: >> >> http://neilk.net/blog/2000/06/01/abigails-regex-to-test-for-prime-numbers/ >> >> so we deliberately make regexes as slow as possible so that programmers >> will look for a better way to solve their problem. If you check the >> source code for the re engine, you'll find that for certain regexes, it >> busy-waits for anything up to 30 seconds at a time, deliberately wasting >> cycles. > > I hate to sound like this but do you realise that this is exactly what > you're arguing for when saying that sum() shouldn't use "+="? my_obj = SomeKoolClass() my_obj.modify_in_some_kool_way() new_result = sum([SKC1, SKC2, SKC3], my_obj) Guess what? You've just changed my_obj. -- ~Ethan~ From joel.goldstick at gmail.com Wed Jul 10 12:32:32 2013 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Wed, 10 Jul 2013 12:32:32 -0400 Subject: =?UTF-8?Q?Re=3A_Stack_Overflow_moderator_=E2=80=9Canimuson=E2=80=9D?= In-Reply-To: <51DD8953.60709@stoneleaf.us> References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> <51DD8953.60709@stoneleaf.us> Message-ID: call your mom On Wed, Jul 10, 2013 at 12:18 PM, Ethan Furman wrote: > On 07/10/2013 08:54 AM, Joshua Landau wrote: > >> On 10 July 2013 10:00, Steven D'Aprano wrote: >> >>> On Wed, 10 Jul 2013 07:55:05 +0000, Mats Peterson wrote: >>> >>> A moderator who calls himself ?animuson? on Stack Overflow doesn?t want >>>> to face the truth. He has deleted all my postings regarding Python >>>> regular expression matching being extremely slow compared to Perl. >>>> >>> >>> That's by design. We don't want to make the same mistake as Perl, where >>> every problem is solved by a regular expression: >>> >>> http://neilk.net/blog/2000/06/**01/abigails-regex-to-test-for-** >>> prime-numbers/ >>> >>> so we deliberately make regexes as slow as possible so that programmers >>> will look for a better way to solve their problem. If you check the >>> source code for the re engine, you'll find that for certain regexes, it >>> busy-waits for anything up to 30 seconds at a time, deliberately wasting >>> cycles. >>> >> >> I hate to sound like this but do you realise that this is exactly what >> you're arguing for when saying that sum() shouldn't use "+="? >> > > my_obj = SomeKoolClass() > my_obj.modify_in_some_kool_**way() > new_result = sum([SKC1, SKC2, SKC3], my_obj) > > Guess what? You've just changed my_obj. > > -- > ~Ethan~ > -- > http://mail.python.org/**mailman/listinfo/python-list > -- Joel Goldstick http://joelgoldstick.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshua at landau.ws Wed Jul 10 13:55:29 2013 From: joshua at landau.ws (Joshua Landau) Date: Wed, 10 Jul 2013 18:55:29 +0100 Subject: =?UTF-8?Q?Re=3A_Stack_Overflow_moderator_=E2=80=9Canimuson=E2=80=9D?= In-Reply-To: <51DD8953.60709@stoneleaf.us> References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> <51DD8953.60709@stoneleaf.us> Message-ID: On 10 July 2013 17:18, Ethan Furman wrote: > On 07/10/2013 08:54 AM, Joshua Landau wrote: >> >> On 10 July 2013 10:00, Steven D'Aprano wrote: >>> >>> On Wed, 10 Jul 2013 07:55:05 +0000, Mats Peterson wrote: >>> >>>> A moderator who calls himself ?animuson? on Stack Overflow doesn?t want >>>> to face the truth. He has deleted all my postings regarding Python >>>> regular expression matching being extremely slow compared to Perl. >>> >>> >>> That's by design. We don't want to make the same mistake as Perl, where >>> every problem is solved by a regular expression: >>> >>> >>> http://neilk.net/blog/2000/06/01/abigails-regex-to-test-for-prime-numbers/ >>> >>> so we deliberately make regexes as slow as possible so that programmers >>> will look for a better way to solve their problem. If you check the >>> source code for the re engine, you'll find that for certain regexes, it >>> busy-waits for anything up to 30 seconds at a time, deliberately wasting >>> cycles. >> >> >> I hate to sound like this but do you realise that this is exactly what >> you're arguing for when saying that sum() shouldn't use "+="? > > > my_obj = SomeKoolClass() > my_obj.modify_in_some_kool_way() > new_result = sum([SKC1, SKC2, SKC3], my_obj) > > Guess what? You've just changed my_obj. You're extrapolating too quickly. The first "+" is a copying "+", the rest add in-place to the new object. Thus, no unexpected side effect. From solipsis at pitrou.net Thu Jul 18 04:54:19 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 18 Jul 2013 08:54:19 +0000 (UTC) Subject: Stack Overflow moderator =?utf-8?b?4oCcYW5pbXVzb27igJ0=?= References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> Message-ID: Joshua Landau landau.ws> writes: > > > The same with Unicode. We hate French people, > > And for good damn reason too. They're ruining our language, ? mon avis. We do! Regards Antoine. From steve+comp.lang.python at pearwood.info Wed Jul 10 13:15:20 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 10 Jul 2013 17:15:20 GMT Subject: Stack Overflow moderator =?iso-8859-13?b?tGFuaW11c29uoQ==?= References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> Message-ID: <51dd96a8$0$9505$c3e8da3$5496439d@news.astraweb.com> On Wed, 10 Jul 2013 16:54:02 +0100, Joshua Landau wrote: > On 10 July 2013 10:00, Steven D'Aprano wrote: >> On Wed, 10 Jul 2013 07:55:05 +0000, Mats Peterson wrote: >> >>> A moderator who calls himself ?animuson? on Stack Overflow doesn?t >>> want to face the truth. He has deleted all my postings regarding >>> Python regular expression matching being extremely slow compared to >>> Perl. >> >> That's by design. We don't want to make the same mistake as Perl, where >> every problem is solved by a regular expression: >> >> http://neilk.net/blog/2000/06/01/abigails-regex-to-test-for-prime- numbers/ >> >> so we deliberately make regexes as slow as possible so that programmers >> will look for a better way to solve their problem. If you check the >> source code for the re engine, you'll find that for certain regexes, it >> busy-waits for anything up to 30 seconds at a time, deliberately >> wasting cycles. > > I hate to sound like this but do you realise that this is exactly what > you're arguing for when saying that sum() shouldn't use "+="? You're referencing an off-list conversation, which will probably confuse most others reading this. I don't agree with that. Apart from one throw-away comment where I said that sometimes it is handy to have a trivial example of an O(N**2) algorithm for teaching purposes, I have never made any suggestion that having sum(lists) be slow was a good thing in and of itself. My argument has always been that there are costs as well as benefits to changing sum of lists to use += instead of + and I'm not convinced that the benefits outweigh those costs. Quite frankly, looking at the pure-Python version of sum that Sergey has posted, I *really* hope he is a better C programmer than Python programmer, because his pure-Python version is so full of bugs it is ridiculous. But now I'm also referring to posts off-list :-) -- Steven From joshua at landau.ws Wed Jul 10 13:53:34 2013 From: joshua at landau.ws (Joshua Landau) Date: Wed, 10 Jul 2013 18:53:34 +0100 Subject: =?UTF-8?Q?Re=3A_Stack_Overflow_moderator_=E2=80=9Canimuson=E2=80=9D?= In-Reply-To: <51dd96a8$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> <51dd96a8$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 10 July 2013 18:15, Steven D'Aprano wrote: > On Wed, 10 Jul 2013 16:54:02 +0100, Joshua Landau wrote: > >> On 10 July 2013 10:00, Steven D'Aprano wrote: >>> On Wed, 10 Jul 2013 07:55:05 +0000, Mats Peterson wrote: >>> >>>> A moderator who calls himself ?animuson? on Stack Overflow doesn?t >>>> want to face the truth. He has deleted all my postings regarding >>>> Python regular expression matching being extremely slow compared to >>>> Perl. >>> >>> That's by design. We don't want to make the same mistake as Perl, where >>> every problem is solved by a regular expression: >>> >>> http://neilk.net/blog/2000/06/01/abigails-regex-to-test-for-prime- > numbers/ >>> >>> so we deliberately make regexes as slow as possible so that programmers >>> will look for a better way to solve their problem. If you check the >>> source code for the re engine, you'll find that for certain regexes, it >>> busy-waits for anything up to 30 seconds at a time, deliberately >>> wasting cycles. >> >> I hate to sound like this but do you realise that this is exactly what >> you're arguing for when saying that sum() shouldn't use "+="? > > You're referencing an off-list conversation, which will probably confuse > most others reading this. > > I don't agree with that. Apart from one throw-away comment where I said > that sometimes it is handy to have a trivial example of an O(N**2) > algorithm for teaching purposes, I have never made any suggestion that > having sum(lists) be slow was a good thing in and of itself. I might be misattributing posts then. Or... YOU'RE IN DENIAL! Who wins? You decide! From steve+comp.lang.python at pearwood.info Wed Jul 10 19:28:10 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 10 Jul 2013 23:28:10 GMT Subject: Stack Overflow moderator =?iso-8859-13?b?tGFuaW11c29uoQ==?= References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> <51dd96a8$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51ddee0a$0$9505$c3e8da3$5496439d@news.astraweb.com> On Wed, 10 Jul 2013 18:53:34 +0100, Joshua Landau wrote: > I might be misattributing posts then. Or... YOU'RE IN DENIAL! Ranting Rick? Is that you? :-) > Who wins? You decide! Ah, definitely not RR :-) -- Steven From wxjmfauth at gmail.com Fri Jul 19 14:54:56 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Fri, 19 Jul 2013 11:54:56 -0700 (PDT) Subject: =?windows-1252?Q?Re=3A_Stack_Overflow_moderator_=93animuson=94?= In-Reply-To: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> Message-ID: <713bdf49-10ee-4694-b7c1-54beef22a922@googlegroups.com> Le mercredi 10 juillet 2013 11:00:23 UTC+2, Steven D'Aprano a ?crit?: > On Wed, 10 Jul 2013 07:55:05 +0000, Mats Peterson wrote: > > > > > A moderator who calls himself ?animuson? on Stack Overflow doesn?t want > > > to face the truth. He has deleted all my postings regarding Python > > > regular expression matching being extremely slow compared to Perl. > > > > That's by design. We don't want to make the same mistake as Perl, where > > every problem is solved by a regular expression: > > > > http://neilk.net/blog/2000/06/01/abigails-regex-to-test-for-prime-numbers/ > > > > so we deliberately make regexes as slow as possible so that programmers > > will look for a better way to solve their problem. If you check the > > source code for the re engine, you'll find that for certain regexes, it > > busy-waits for anything up to 30 seconds at a time, deliberately wasting > > cycles. > > > > The same with Unicode. We hate French people, you see, and so in an > > effort to drive everyone back to ASCII-only text, Python 3.3 introduces > > some memory optimizations that ensures that Unicode strings work > > correctly and are up to four times smaller than they used to be. You > > should get together with jmfauth, who has discovered our dastardly plot > > and keeps posting benchmarks showing how on carefully contrived micro- > > benchmarks using a beta version of Python 3.3, non-ASCII string > > operations can be marginally slower than in 3.2. > > > > > > > Additionally my account has been suspended for 7 days. Such a dickwad. > > > > I cannot imagine why he would have done that. > > > > > > -- > > Steven This Flexible String Representation is a dream case study. Attempting to optimize a subset of character is a non sense. If you are a non-ascii user, such a mechanism is irrelevant, because per definition you do not need it. Not only it useless, it is penalizing, just by the fact of its existence. [*] Conversely (or identically), if you are an ascii user, same situation, it is irrelevant, useless and penalizing. Practically, and today, all coding schemes we have (including the endorsed Unicode utf transformers) work with a unique set of of encoded code points. If you wish to take the problem from the other side, it is because one can only work properly with a unique set of code points that so many coding schemes exist! Question: does this FSR use internally three coding schemes because it splits Unicode in three groups or does it split Unicode in three subsets to have the joyce to use three coding schemes? About "micro benchmarks". What to say, they appear practivally every time you use non ascii. And do not forget memory. The ?uro just become expensive. >>> sys.getsizeof('$') 26 >>> sys.getsizeof('?') 40 I do not know. When an ?uro char need 14 bytes more that a dollar, I belong to those who thing there is a problem somewhere. This FSR is a royal gift for those who wish to teach Unicode and the coding of characters. jmf From rosuav at gmail.com Fri Jul 19 18:35:52 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 20 Jul 2013 08:35:52 +1000 Subject: =?windows-1252?Q?Re=3A_Stack_Overflow_moderator_=93animuson=94?= In-Reply-To: <713bdf49-10ee-4694-b7c1-54beef22a922@googlegroups.com> References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> <713bdf49-10ee-4694-b7c1-54beef22a922@googlegroups.com> Message-ID: On Sat, Jul 20, 2013 at 4:54 AM, wrote: > And do not forget memory. The ?uro just become expensive. > >>>> sys.getsizeof(' > ) > 26 >>>> sys.getsizeof('?') > 40 > > I do not know. When an ?uro char need 14 bytes more that > a dollar, I belong to those who thing there is a problem > somewhere. Oh, I totally agree. But it's not just the Euro symbol that's expensive. Look how much I have to pay for a couple of square brackets! >>> sys.getsizeof((1)) 14 >>> sys.getsizeof([1]) 40 ChrisA From davea at davea.name Fri Jul 19 19:14:06 2013 From: davea at davea.name (Dave Angel) Date: Fri, 19 Jul 2013 19:14:06 -0400 Subject: Stack Overflow moderator =?windows-1252?Q?=93animuson=94?= In-Reply-To: References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> <713bdf49-10ee-4694-b7c1-54beef22a922@googlegroups.com> Message-ID: On 07/19/2013 06:35 PM, Chris Angelico wrote: > On Sat, Jul 20, 2013 at 4:54 AM, wrote: >> And do not forget memory. The ?uro just become expensive. >> >>>>> sys.getsizeof(' >> ) >> 26 >>>>> sys.getsizeof('?') >> 40 >> >> I do not know. When an ?uro char need 14 bytes more that >> a dollar, I belong to those who thing there is a problem >> somewhere. > > Oh, I totally agree. But it's not just the Euro symbol that's > expensive. Look how much I have to pay for a couple of square > brackets! > >>>> sys.getsizeof((1)) > 14 >>>> sys.getsizeof([1]) > 40 > And look how much a comma costs us (on 64bit 3.3): >>> sys.getsizeof((1)) 28 >>> sys.getsizeof((1,)) 64 >>> sys.getsizeof([1]) 80 ;-) -- DaveA From joshua at landau.ws Sat Jul 20 06:27:01 2013 From: joshua at landau.ws (Joshua Landau) Date: Sat, 20 Jul 2013 11:27:01 +0100 Subject: =?UTF-8?Q?Re=3A_Stack_Overflow_moderator_=E2=80=9Canimuson=E2=80=9D?= In-Reply-To: References: <51dd22a6$0$11094$c3e8da3@news.astraweb.com> <713bdf49-10ee-4694-b7c1-54beef22a922@googlegroups.com> Message-ID: On 19 July 2013 23:35, Chris Angelico wrote: > On Sat, Jul 20, 2013 at 4:54 AM, wrote: >> And do not forget memory. The ?uro just become expensive. >> >>>>> sys.getsizeof(' >> ) >> 26 >>>>> sys.getsizeof('?') >> 40 >> >> I do not know. When an ?uro char need 14 bytes more that >> a dollar, I belong to those who thing there is a problem >> somewhere. > > Oh, I totally agree. But it's not just the Euro symbol that's > expensive. Look how much I have to pay for a couple of square > brackets! > >>>> sys.getsizeof((1)) > 14 >>>> sys.getsizeof([1]) > 40 But when you do it generically, square brackets save you space! >>> sys.getsizeof((int)) 392 >>> sys.getsizeof([int]) 80 :D From joshua at landau.ws Wed Jul 10 07:12:10 2013 From: joshua at landau.ws (Joshua Landau) Date: Wed, 10 Jul 2013 12:12:10 +0100 Subject: =?UTF-8?Q?Re=3A_Stack_Overflow_moderator_=E2=80=9Canimuson=E2=80=9D?= In-Reply-To: References: Message-ID: On 10 July 2013 08:55, Mats Peterson wrote: > . [anumuson from Stack Overflow] has deleted all > my postings regarding Python regular expression matching being > extremely slow compared to Perl. Additionally my account has been > suspended for 7 days. . Whilst I don't normally respond to trolls, I'm actually curious. Do you have any non-trivial, properly benchmarked real-world examples that this affects, remembering to use full Unicode support in Perl (as Python has it by default)? Remember to try on both major CPython versions, and PyPy -- all of which are in large-scale usage. Remember not just to use the builtin re module, as most people also use https://pypi.python.org/pypi/regex and https://code.google.com/p/re2/ when they are appropriate, so pathological cases for re aren't actually a concern anyone cares about. If you actually can satisfy these basic standards for a comparison (as I'm sure any competent person with so much bravo could) I'd be willing to converse with you. I'd like to see these results where Python compares as "extremely slow". Note that, by your own wording, a 30% drop is irrelevant. From benedict.verheyen at gmail.com Wed Jul 10 07:56:03 2013 From: benedict.verheyen at gmail.com (Benedict Verheyen) Date: Wed, 10 Jul 2013 11:56:03 +0000 (UTC) Subject: Stack Overflow moderator =?iso-8859-13?b?tGFuaW11c29uoQ==?= References: Message-ID: Op Wed, 10 Jul 2013 12:12:10 +0100, schreef Joshua Landau: > > Do you have any non-trivial, properly benchmarked real-world examples > that this affects, remembering to use full Unicode support in Perl (as > Python has it by default)? > Indeed, as Joshua says, instead of going through all the effort to talk about it, why don't you show us that Python is slower in regexs than perl. That's at least relevant bandwdith. Next, if Python is slower, there might be ways to improve the performance. Last, if there is a SPU (Secret Python Underground), I want to j -- Benedict Verheyen Debian, Python and Django user GnuPG Public Key 0x712CBB8D From antoon.pardon at rece.vub.ac.be Thu Jul 11 05:36:41 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Thu, 11 Jul 2013 11:36:41 +0200 Subject: Stack Overflow moderator =?windows-1252?Q?=93animuson=94?= In-Reply-To: References: Message-ID: <51DE7CA9.9000405@rece.vub.ac.be> Op 10-07-13 13:56, Benedict Verheyen schreef: > Op Wed, 10 Jul 2013 12:12:10 +0100, schreef Joshua Landau: > > >> >> Do you have any non-trivial, properly benchmarked real-world examples >> that this affects, remembering to use full Unicode support in Perl (as >> Python has it by default)? >> > > Indeed, as Joshua says, instead of going through all the effort to talk > about it, why don't you show us that Python is slower in regexs than perl. > That's at least relevant bandwdith. > > Next, if Python is slower, there might be ways to improve the performance. > > Last, if there is a SPU (Secret Python Underground), I want to j > Heretic, it is the From matsp999 at aim.com Wed Jul 10 08:06:06 2013 From: matsp999 at aim.com (Mats Peterson) Date: Wed, 10 Jul 2013 12:06:06 +0000 (UTC) Subject: =?UTF-8?Q?=20Stack=20Overflow=20moder?= =?UTF-8?Q?ator=20=E2=80=9Canimuson=E2=80=9D?= References: Message-ID: Joshua Landau wrote: > On 10 July 2013 08:55, Mats Peterson wrote: >> . [anumuson from Stack Overflow] has deleted all >> my postings regarding Python regular expression matching being >> extremely slow compared to Perl. Additionally my account has been >> suspended for 7 days. . > > Whilst I don't normally respond to trolls, I'm actually curious. > > Do you have any non-trivial, properly benchmarked real-world examples > that this affects, remembering to use full Unicode support in Perl (as > Python has it by default)? > > Remember to try on both major CPython versions, and PyPy -- all of > which are in large-scale usage. Remember not just to use the builtin > re module, as most people also use https://pypi.python.org/pypi/regex > and https://code.google.com/p/re2/ when they are appropriate, so > pathological cases for re aren't actually a concern anyone cares > about. > > If you actually can satisfy these basic standards for a comparison (as > I'm sure any competent person with so much bravo could) I'd be willing > to converse with you. I'd like to see these results where Python compares > as "extremely slow". Note that, by your own wording, a 30% drop is irrelevant. I haven't provided a "real-world" example, since I expect you Python Einsteins to be able do an A/B test between Python and Perl yourselves (provided you know Perl, of course, which I'm afraid is not always the case). And why would I use any "custom" version of Python, when I don't have to do that with Perl? Mats -- Mats Peterson http://alicja.homelinux.com/~mats/ From benedict.verheyen at gmail.com Wed Jul 10 09:53:40 2013 From: benedict.verheyen at gmail.com (Benedict Verheyen) Date: Wed, 10 Jul 2013 13:53:40 +0000 (UTC) Subject: Stack Overflow moderator =?iso-8859-13?b?tGFuaW11c29uoQ==?= References: Message-ID: Op Wed, 10 Jul 2013 12:06:06 +0000, schreef Mats Peterson: > I haven't provided a "real-world" example, since I expect you Python > Einsteins to be able do an A/B test between Python and Perl yourselves > (provided you know Perl, of course, which I'm afraid is not always the > case). I don't know perl so I can't compare. > And why would I use any "custom" version of Python, when I don't > have to do that with Perl? If you're able to do that with Perl, and Perl is faster that Python, why would you want to bother with Python? Seems like you already have a fast alternative you like. -- Benedict Verheyen Debian, Python and Django user GnuPG Public Key 0x712CBB8D From ian.g.kelly at gmail.com Wed Jul 10 12:02:50 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 10 Jul 2013 10:02:50 -0600 Subject: =?windows-1252?Q?Re=3A_Stack_Overflow_moderator_=93animuson=94?= In-Reply-To: References: Message-ID: On Wed, Jul 10, 2013 at 6:06 AM, Mats Peterson wrote: > Joshua Landau wrote: >> If you actually can satisfy these basic standards for a comparison (as >> I'm sure any competent person with so much bravo could) I'd be willing >> to converse with you. I'd like to see these results where Python compares >> as "extremely slow". Note that, by your own wording, a 30% drop is irrelevant. > > I haven't provided a "real-world" example, since I expect you Python > Einsteins to be able do an A/B test between Python and Perl yourselves > (provided you know Perl, of course, which I'm afraid is not always the > case). And why would I use any "custom" version of Python, when I don't > have to do that with Perl? Well, I'm certainly not about to do the comparison myself. I only have so much time, and presently I am not sufficiently motivated about this topic to invest any time in it. A suggestion: *you* brought up the subject, so if you want us to pay attention to your claims, then *you* had better be prepared to get your hands dirty and do the actual work to support it. From wuwei23 at gmail.com Wed Jul 10 22:08:28 2013 From: wuwei23 at gmail.com (alex23) Date: Thu, 11 Jul 2013 12:08:28 +1000 Subject: Stack Overflow moderator =?UTF-8?B?4oCcYW5pbXVzb27igJ0=?= In-Reply-To: References: Message-ID: On 10/07/2013 11:53 PM, Benedict Verheyen wrote: > Op Wed, 10 Jul 2013 12:06:06 +0000, schreef Mats Peterson: >> And why would I use any "custom" version of Python, when I don't >> have to do that with Perl? > > If you're able to do that with Perl, and Perl is faster that Python, > why would you want to bother with Python? The OP has no interest in using Python, this is simply a perfect example of the effect of Dunbar's number. From torriem at gmail.com Thu Jul 11 00:38:22 2013 From: torriem at gmail.com (Michael Torrie) Date: Wed, 10 Jul 2013 22:38:22 -0600 Subject: =?windows-1252?Q?Re=3A_Stack_Overflow_bans_Mats_Peters?= =?windows-1252?Q?on_=28was_Re=3A_Stack_Overflow_moderator_=93?= =?windows-1252?Q?animuson=94=29?= In-Reply-To: References: Message-ID: <51DE36BE.2020401@gmail.com> On 07/10/2013 06:06 AM, Mats Peterson wrote: > I haven't provided a "real-world" example, since I expect you Python > Einsteins to be able do an A/B test between Python and Perl yourselves > (provided you know Perl, of course, which I'm afraid is not always the > case). And why would I use any "custom" version of Python, when I don't > have to do that with Perl? Oh wow. You really do sound like Ranting Rick. Always wanting someone else to do the work and carry the burden of proof. Since you don't use Python, don't like Python, and don't promote Python, why are you here on this list? Are you evangelizing Perl? Apologies to the list; I hope I'm done. From tjreedy at udel.edu Wed Jul 10 14:52:58 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 10 Jul 2013 14:52:58 -0400 Subject: Stack Overflow bans Mats Peterson (was Re: ....) In-Reply-To: References: Message-ID: On 7/10/2013 3:55 AM, Mats Peterson wrote: > A moderator who calls himself ?animuson? on Stack Overflow doesn?t > want to face the truth. He has deleted all my postings regarding Python > regular expression matching being extremely slow compared to Perl. > Additionally my account has been suspended for 7 days. Such a dickwad. Your opinion of "animuson" is off-topic for this list. StackOverflow is explicitly for technical questions with technical answers. I believe opinions, languages comparisions, and flamebaits in general are explicitly banned and subject to removal and possible banning. So are subterfuges like phrasing banned topics as questions. Given your behavior here, I am 90+% sure animuson's action was appropriate. -- Terry Jan Reedy From jake.angulo at gmail.com Fri Jul 19 03:33:01 2013 From: jake.angulo at gmail.com (Jake Angulo) Date: Fri, 19 Jul 2013 17:33:01 +1000 Subject: =?windows-1252?Q?Re=3A_Stack_Overflow_moderator_=93animuson=94?= In-Reply-To: References: Message-ID: On Wed, Jul 10, 2013 at 5:55 PM, Mats Peterson wrote: > A moderator who calls himself ?animuson? on Stack Overflow doesn?t > want to face the truth. He has deleted all my postings regarding Python > regular expression matching being extremely slow compared to Perl. > Additionally my account has been suspended for 7 days. Such a dickwad. > > The OP meant to post: A moderator who calls himself MatsDtroll on Stack Overflow doesn?t want to face the truth. He has deleted all my postings regarding Perl regular expression matching being extremely slow compared to Bash regex. Additionally my account has been suspended for 7 days. Such a dickwad. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kw at codebykevin.com Fri Jul 19 20:47:31 2013 From: kw at codebykevin.com (Kevin Walzer) Date: Fri, 19 Jul 2013 20:47:31 -0400 Subject: Stack Overflow moderator =?windows-1252?Q?=93animuson=94?= In-Reply-To: References: Message-ID: On 7/10/13 3:55 AM, Mats Peterson wrote: > A moderator who calls himself ???animuson??? on Stack Overflow doesn???t > want to face the truth. He has deleted all my postings regarding Python > regular expression matching being extremely slow compared to Perl. > Additionally my account has been suspended for 7 days. Such a dickwad. > > Mats > And we should care because...? -- Kevin Walzer Code by Kevin/Mobile Code by Kevin http://www.codebykevin.com http://www.wtmobilesoftware.com From oswaldclem at gmail.com Wed Jul 10 06:44:41 2013 From: oswaldclem at gmail.com (oswaldclem) Date: Wed, 10 Jul 2013 03:44:41 -0700 (PDT) Subject: WiFi problems with iPhone 4 Message-ID: <1373453081557-5024342.post@n6.nabble.com> Hi, I have Been looking into it and I can't find anything. My son's iPhone 4 ( iOS 6), according to him, is on and off from our wifi all by itself. Basically, when h? goes on YouTube before 8 am it is on 3G, even if we have wifi on. And the signal is strong enough. Is there a setting That he could have used, unknowingly or not, That can do this? ----- used iphone -- View this message in context: http://python.6.x6.nabble.com/WiFi-problems-with-iPhone-4-tp5024342.html Sent from the Python - python-list mailing list archive at Nabble.com. From oswaldclem at gmail.com Wed Jul 10 06:47:05 2013 From: oswaldclem at gmail.com (oswaldclem) Date: Wed, 10 Jul 2013 03:47:05 -0700 (PDT) Subject: iTunes not detecting iPhone Message-ID: <1373453225073-5024343.post@n6.nabble.com> I just received a used iPhone I purchased on Amazon. I took the phone out of the box and turned it on. The Apple logo appeared, then a "connect to iTunes" screen appeared, with the text "No SIM card installed" over the connect to iTunes graphic. I installed my SIM (which I know works, as I just pulled it from my other iPhone), but have a very weak signal... but that's not my issue right now. After putting in my SIM, I connect my cable to my PC and opened iTunes. Then I connected the new iPhone to the cable and waited for iTunes to detect it. Nothing. I tried shutting down iTunes and restarting, shutting down and restarting the phone... still nothing. ----- used iphone -- View this message in context: http://python.6.x6.nabble.com/iTunes-not-detecting-iPhone-tp5024343.html Sent from the Python - python-list mailing list archive at Nabble.com. From oswaldclem at gmail.com Wed Jul 10 06:48:50 2013 From: oswaldclem at gmail.com (oswaldclem) Date: Wed, 10 Jul 2013 03:48:50 -0700 (PDT) Subject: Iphone 3gs Problem Message-ID: <1373453330664-5024344.post@n6.nabble.com> I have a problem with my iPhone 3gs, It was Jailbreaked with cydia and then one day it stoped working, all it says is connect to itunes, and when I connect it to Itunes it say restore iphone, when I do that it get stuck on waiting for iphone, it have been stuck on that for 6 hours without doing anything so I dont know what to do! I realy need help please! ----- used iphone -- View this message in context: http://python.6.x6.nabble.com/Iphone-3gs-Problem-tp5024344.html Sent from the Python - python-list mailing list archive at Nabble.com. From oswaldclem at gmail.com Wed Jul 10 06:49:56 2013 From: oswaldclem at gmail.com (oswaldclem) Date: Wed, 10 Jul 2013 03:49:56 -0700 (PDT) Subject: Buying a used iPhone 5 Message-ID: <1373453396183-5024345.post@n6.nabble.com> I'm planning on buying a used ATT iPhone 5 off of craigslist, and i've been reading on how some people sell their iPhones to people and later on blacklisting it, screwing the buyer over, or how people are mistakenly buying already blacklisted iPhones. I was wondering if there's a way I can prevent this? I was thinking of meeting at an ATT store to make sure it's not blacklisted and to make sure it's not connected to any account that could black list it. Would that be a good idea? ----- used iphone -- View this message in context: http://python.6.x6.nabble.com/Buying-a-used-iPhone-5-tp5024345.html Sent from the Python - python-list mailing list archive at Nabble.com. From rodrick.brown at gmail.com Wed Jul 10 10:10:30 2013 From: rodrick.brown at gmail.com (Rodrick Brown) Date: Wed, 10 Jul 2013 10:10:30 -0400 Subject: Buying a used iPhone 5 In-Reply-To: <1373453396183-5024345.post@n6.nabble.com> References: <1373453396183-5024345.post@n6.nabble.com> Message-ID: Die On Wed, Jul 10, 2013 at 6:49 AM, oswaldclem wrote: > I'm planning on buying a used ATT iPhone 5 off of craigslist, and i've been > reading on how some people sell their iPhones to people and later on > blacklisting it, screwing the buyer over, or how people are mistakenly > buying already blacklisted iPhones. I was wondering if there's a way I can > prevent this? I was thinking of meeting at an ATT store to make sure it's > not blacklisted and to make sure it's not connected to any account that > could black list it. Would that be a good idea? > > > > ----- > used iphone > -- > View this message in context: > http://python.6.x6.nabble.com/Buying-a-used-iPhone-5-tp5024345.html > Sent from the Python - python-list mailing list archive at Nabble.com. > -- > http://mail.python.org/mailman/listinfo/python-list > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fabiosantosart at gmail.com Wed Jul 10 18:30:57 2013 From: fabiosantosart at gmail.com (=?ISO-8859-1?Q?F=E1bio_Santos?=) Date: Wed, 10 Jul 2013 23:30:57 +0100 Subject: Buying a used iPhone 5 In-Reply-To: References: <1373453396183-5024345.post@n6.nabble.com> Message-ID: On 10 Jul 2013 15:15, "Rodrick Brown" wrote: > > Die > > > On Wed, Jul 10, 2013 at 6:49 AM, oswaldclem wrote: >> >> I'm planning on buying a used ATT iPhone 5 off of craigslist, and i've been >> reading on how some people sell their iPhones to people and later on >> blacklisting it, screwing the buyer over, or how people are mistakenly >> buying already blacklisted iPhones. I was wondering if there's a way I can >> prevent this? I was thinking of meeting at an ATT store to make sure it's >> not blacklisted and to make sure it's not connected to any account that >> could black list it. Would that be a good idea? >> Dice -------------- next part -------------- An HTML attachment was scrubbed... URL: From oswaldclem at gmail.com Wed Jul 10 06:42:58 2013 From: oswaldclem at gmail.com (oswaldclem) Date: Wed, 10 Jul 2013 03:42:58 -0700 (PDT) Subject: Turn off Genius on iPhone Message-ID: <1373452977996-5024341.post@n6.nabble.com> How do I turn off the genius button on the iPhone ? I keep hitting it whenever I want to scrub to the middle of through a song. It's pretty annoying. ----- used iphone -- View this message in context: http://python.6.x6.nabble.com/Turn-off-Genius-on-iPhone-tp5024341.html Sent from the Python - python-list mailing list archive at Nabble.com. From rosuav at gmail.com Wed Jul 10 10:00:59 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 11 Jul 2013 00:00:59 +1000 Subject: Prime number generator Message-ID: And now for something completely different. I knocked together a prime number generator, just for the fun of it, that works like a Sieve of Eratosthenes but unbounded. It keeps track of all known primes and the "next composite" that it will produce - for instance, after yielding 13, the prime map will be {2: 20, 3: 18, 5: 20, 7: 21, 11: 22, 13: 26}, each one mapped to the first multiple greater than 13. Notable in the algorithm is an entire lack of division, or even multiplication. Everything is done with addition. So, a few questions. Firstly, is there a stdlib way to find the key with the lowest corresponding value? In the above map, it would return 3, because 18 is the lowest value in the list. I want to do this with a single pass over the dictionary. Secondly, can the "while i References: Message-ID: <15167633-b6e7-46cc-a043-8dfe8aaad11e@googlegroups.com> On Wednesday, July 10, 2013 4:00:59 PM UTC+2, Chris Angelico wrote: [...] > So, a few questions. Firstly, is there a stdlib way to find the key > with the lowest corresponding value? In the above map, it would return > 3, because 18 is the lowest value in the list. I want to do this with > a single pass over the dictionary. In [1]: prime = {2: 20, 3: 18, 5: 20, 7: 21, 11: 22, 13: 26} In [2]: smallest_key = min(prime.iteritems(), key=lambda k_v: k_v[1])[0] In [3]: smallest_key Out[3]: 3 Still trying to figure out your algorithm ... From rosuav at gmail.com Wed Jul 10 11:12:19 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 11 Jul 2013 01:12:19 +1000 Subject: Prime number generator In-Reply-To: <15167633-b6e7-46cc-a043-8dfe8aaad11e@googlegroups.com> References: <15167633-b6e7-46cc-a043-8dfe8aaad11e@googlegroups.com> Message-ID: On Thu, Jul 11, 2013 at 12:35 AM, Bas wrote: > On Wednesday, July 10, 2013 4:00:59 PM UTC+2, Chris Angelico wrote: > [...] >> So, a few questions. Firstly, is there a stdlib way to find the key >> with the lowest corresponding value? In the above map, it would return >> 3, because 18 is the lowest value in the list. I want to do this with >> a single pass over the dictionary. > > In [1]: prime = {2: 20, 3: 18, 5: 20, 7: 21, 11: 22, 13: 26} > > In [2]: smallest_key = min(prime.iteritems(), key=lambda k_v: k_v[1])[0] > > In [3]: smallest_key > Out[3]: 3 Well, that does answer the question. Unfortunately the use of lambda there has a severe performance cost (roughly doubles the total run time, when I ask for the thousandth prime), without majorly improving readability. I'll bear it in mind if there's a way to make that work on either readability or performance, but otherwise, I'll stick with the explicit loop. Thanks anyway! > Still trying to figure out your algorithm ... It's pretty simple. (That's a bad start, I know!) Like the Sieve of Eratosthenes, it locates prime numbers, then deems every multiple of them to be composite. Unlike the classic sieve, it does the "deem" part in parallel. Instead of marking all the multiples of 2 first, then picking three and marking all the multiples of 3, then 5, etc, this function records the fact that it's up to (say) 42 in marking multiples of 2, and then when it comes to check if 43 is prime or not, it moves to the next multiple of 2. This requires memory to store the previously-known primes, similarly to other methods, but needs no multiplication or division. ChrisA From blswinkels at gmail.com Wed Jul 10 11:47:36 2013 From: blswinkels at gmail.com (bas) Date: Wed, 10 Jul 2013 08:47:36 -0700 (PDT) Subject: Prime number generator In-Reply-To: References: <15167633-b6e7-46cc-a043-8dfe8aaad11e@googlegroups.com> Message-ID: On Wednesday, July 10, 2013 5:12:19 PM UTC+2, Chris Angelico wrote: > Well, that does answer the question. Unfortunately the use of lambda > there has a severe performance cost [ ...] If you care about speed, you might want to check the heapq module. Removing the smallest item and inserting a new item in a heap both cost O(log(N)) time, while finding the minimum in a dictionary requires iterating over the whole dictionary, which cost O(N) time. (untested) #before loop from heapq import * primes = [(2,2)] #heap of tuples (multiple, prime). start with 1 item, so no need for heapify #during loop smallest, prm = heappop(primes) heappush(primes, (smallest+prm, prm)) #when new prime found heappush(primes, (i+i, i)) > > Still trying to figure out your algorithm ... > It's pretty simple. [...] I understand what you are trying, but it is not immediately clear to me that it works correctly if for example a smallest factor appears twice in the list. I don't have time for it now, but it for sure can be simplified. The while loop, for example, can be replaced by an if, since it will never execute more than once (since if i is prime > 2, i+1 will be divisible by 2) From rosuav at gmail.com Wed Jul 10 12:15:06 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 11 Jul 2013 02:15:06 +1000 Subject: Prime number generator In-Reply-To: References: <15167633-b6e7-46cc-a043-8dfe8aaad11e@googlegroups.com> Message-ID: On Thu, Jul 11, 2013 at 1:47 AM, bas wrote: > On Wednesday, July 10, 2013 5:12:19 PM UTC+2, Chris Angelico wrote: >> Well, that does answer the question. Unfortunately the use of lambda >> there has a severe performance cost [ ...] > If you care about speed, you might want to check the heapq module. Removing the smallest item and inserting a new item in a heap both cost O(log(N)) time, while finding the minimum in a dictionary requires iterating over the whole dictionary, which cost O(N) time. Ehh, speed isn't the ultimate. I was just trying to avoid something that worked out ridiculously slow (a Python function call IS quite slow). I haven't profiled the code to find out where the bulk of the time is spent, but switching in the lambda-based version doubled total run time, so I didn't like it :) > (untested) > #before loop > from heapq import * > primes = [(2,2)] #heap of tuples (multiple, prime). start with 1 item, so no need for heapify > > #during loop > smallest, prm = heappop(primes) > heappush(primes, (smallest+prm, prm)) > > #when new prime found > heappush(primes, (i+i, i)) Ahh, that's the bit I should have thought of! Of course. My original thought experiment had involved basically a really long list, like the classic Sieve but getting longer as time moves on, with composites replaced by None and primes with their next-markers, which I then collapsed to a dict. Always I was thinking in terms of indexing with the prime to get its next composite. Here's the code involving heapq: # -- start -- def primes(): """Generate an infinite series of prime numbers.""" from heapq import heappush,heappop i=2 yield 2 prime=[(2,2)] # Heap while True: smallest, prm = heappop(prime) heappush(prime, (smallest+prm, prm)) while i> > Still trying to figure out your algorithm ... >> It's pretty simple. [...] > I understand what you are trying, but it is not immediately clear to me that it works correctly if for example a smallest factor appears twice in the list. I don't have time for it now, but it for sure can be simplified. The while loop, for example, can be replaced by an if, since it will never execute more than once (since if i is prime > 2, i+1 will be divisible by 2) Ah, good point. Again, I originally was starting from 1, so the while loop was necessary to catch 2 and 3 in the first run. When I changed it to start at 2 and explicitly yield it first, I didn't notice to change that. It works correctly with the smallest multiple appearing twice because it won't yield primes until the smallest value is higher than the current next-prime. So when it's just yielded 11, for instance, both the 2 and 3 slots hold 12; advancing one of those does nothing, advancing the other allows the bottom loop to notice that 13 is now lower than the minimum value (which will then be 14). ChrisA From joshua at landau.ws Wed Jul 10 13:47:11 2013 From: joshua at landau.ws (Joshua Landau) Date: Wed, 10 Jul 2013 18:47:11 +0100 Subject: Prime number generator In-Reply-To: References: <15167633-b6e7-46cc-a043-8dfe8aaad11e@googlegroups.com> Message-ID: On 10 July 2013 17:15, Chris Angelico wrote: > On Thu, Jul 11, 2013 at 1:47 AM, bas wrote: >> On Wednesday, July 10, 2013 5:12:19 PM UTC+2, Chris Angelico wrote: >>> Well, that does answer the question. Unfortunately the use of lambda >>> there has a severe performance cost [ ...] >> If you care about speed, you might want to check the heapq module. Removing the smallest item and inserting a new item in a heap both cost O(log(N)) time, while finding the minimum in a dictionary requires iterating over the whole dictionary, which cost O(N) time. Actually, because it's a list under the hood I'd imagine push and pop still take O(n) time :/. > Ehh, speed isn't the ultimate. I was just trying to avoid something > that worked out ridiculously slow (a Python function call IS quite > slow). I haven't profiled the code to find out where the bulk of the > time is spent, but switching in the lambda-based version doubled total > run time, so I didn't like it :) > >> (untested) >> #before loop >> from heapq import * >> primes = [(2,2)] #heap of tuples (multiple, prime). start with 1 item, so no need for heapify >> >> #during loop >> smallest, prm = heappop(primes) >> heappush(primes, (smallest+prm, prm)) >> >> #when new prime found >> heappush(primes, (i+i, i)) > > Ahh, that's the bit I should have thought of! Of course. > > My original thought experiment had involved basically a really long > list, like the classic Sieve but getting longer as time moves on, with > composites replaced by None and primes with their next-markers, which > I then collapsed to a dict. Always I was thinking in terms of indexing > with the prime to get its next composite. Here's the code involving > heapq: > > # -- start -- > def primes(): > """Generate an infinite series of prime numbers.""" > from heapq import heappush,heappop > i=2 > yield 2 > prime=[(2,2)] # Heap > while True: > smallest, prm = heappop(prime) > heappush(prime, (smallest+prm, prm)) > while i yield i > heappush(prime, (i+i, i)) > i+=1 > if i==smallest: i+=1 > > gen=primes() > print([next(gen) for i in range(10)]) > for i in range(1000): > next(gen) # Star Trek? > print("The next prime number is:",next(gen)) > # -- end -- > > And that's significantly shorter, clearer, AND faster than the original. Thanks! AFAICT, that's exactly my code but using a less-efficient storage medium and a *much* more efficient sorting mechanism. It'd be interesting what could happen if heapq didn't reject blists -- they have better efficiency for changing list sizes (which this code does a lot of). Thanks for the heads-up on heapq, by the way -- it's new to me and a blimmin' good idea. PS: It's faster to use heapreplace(...) than heappop(...);heappush(...) but it only saves a few %. From ian.g.kelly at gmail.com Wed Jul 10 14:56:16 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 10 Jul 2013 12:56:16 -0600 Subject: Prime number generator In-Reply-To: References: <15167633-b6e7-46cc-a043-8dfe8aaad11e@googlegroups.com> Message-ID: On Wed, Jul 10, 2013 at 11:47 AM, Joshua Landau wrote: >>> If you care about speed, you might want to check the heapq module. Removing the smallest item and inserting a new item in a heap both cost O(log(N)) time, while finding the minimum in a dictionary requires iterating over the whole dictionary, which cost O(N) time. > > Actually, because it's a list under the hood I'd imagine push and pop > still take O(n) time :/. It shouldn't. You can implement push by appending the new item and then getting it into the right place by performing O(log n) swaps. Likewise for pop, you can update the heap with O(log n) swaps and then removing the tail element. Appending or removing the tail element of a list is amortized O(1). > PS: It's faster to use heapreplace(...) than > heappop(...);heappush(...) but it only saves a few %. The problem with heapreplace here is that the value to be pushed is dependent on the value returned by pop. From joshua at landau.ws Wed Jul 10 15:06:44 2013 From: joshua at landau.ws (Joshua Landau) Date: Wed, 10 Jul 2013 20:06:44 +0100 Subject: Prime number generator In-Reply-To: References: <15167633-b6e7-46cc-a043-8dfe8aaad11e@googlegroups.com> Message-ID: On 10 July 2013 19:56, Ian Kelly wrote: > On Wed, Jul 10, 2013 at 11:47 AM, Joshua Landau wrote: >>>> If you care about speed, you might want to check the heapq module. Removing the smallest item and inserting a new item in a heap both cost O(log(N)) time, while finding the minimum in a dictionary requires iterating over the whole dictionary, which cost O(N) time. >> >> Actually, because it's a list under the hood I'd imagine push and pop >> still take O(n) time :/. > > It shouldn't. You can implement push by appending the new item and > then getting it into the right place by performing O(log n) swaps. > Likewise for pop, you can update the heap with O(log n) swaps and then > removing the tail element. Appending or removing the tail element of > a list is amortized O(1). Genius. Bas replied off-list that it won't matter too much anyway as they're over-allocated, but this makes tons of sense. >> PS: It's faster to use heapreplace(...) than >> heappop(...);heappush(...) but it only saves a few %. > > The problem with heapreplace here is that the value to be pushed > is dependent on the value returned by pop. That's fine because indexing is much faster than pop. The "few %" was a real statistic from working, timed code. From bryanjugglercryptographer at yahoo.com Wed Jul 31 00:57:24 2013 From: bryanjugglercryptographer at yahoo.com (bryanjugglercryptographer at yahoo.com) Date: Tue, 30 Jul 2013 21:57:24 -0700 (PDT) Subject: Prime number generator In-Reply-To: References: <15167633-b6e7-46cc-a043-8dfe8aaad11e@googlegroups.com> Message-ID: Chris Angelico wrote: > Bas wrote: > > Still trying to figure out your algorithm ... > > It's pretty simple. (That's a bad start, I know!) Like the Sieve of > Eratosthenes, it locates prime numbers, then deems every multiple of > them to be composite. Unlike the classic sieve, it does the "deem" > part in parallel. Instead of marking all the multiples of 2 first, > then picking three and marking all the multiples of 3, then 5, etc, > this function records the fact that it's up to (say) 42 in marking > multiples of 2, and then when it comes to check if 43 is prime or not, > it moves to the next multiple of 2. This requires memory to store the > previously-known primes, similarly to other methods, but needs no > multiplication or division. Knuth points to the method, using a priority queue, in exercise 15 of section 5.2.3 of /Sorting and Searching/, and credits it to "B. A. Chartres". -- -Bryan From steve+comp.lang.python at pearwood.info Wed Jul 10 12:01:36 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 10 Jul 2013 16:01:36 GMT Subject: Prime number generator References: Message-ID: <51dd8560$0$9505$c3e8da3$5496439d@news.astraweb.com> On Thu, 11 Jul 2013 00:00:59 +1000, Chris Angelico wrote: > And now for something completely different. > > I knocked together a prime number generator, just for the fun of it, > that works like a Sieve of Eratosthenes but unbounded. [...] > So, a few questions. Firstly, is there a stdlib way to find the key with > the lowest corresponding value? In the above map, it would return 3, > because 18 is the lowest value in the list. Not from a dict, but with a list you would probably use the bisect module. > I want to do this with a > single pass over the dictionary. Secondly, can the "while i i+=1" loop become a for...range? It's almost asking for it, but not > quite there. Thirdly, is there any sort of half-sane benchmark that I > can compare this code to? And finally, whose wheel did I reinvent here? > What name would this algorithm have? I can't directly answer that question, but I can make a shameless plug for this: https://pypi.python.org/pypi/pyprimes If your prime generator performs better than, or worse than, all of those in the above module, I may have to steal it from you :-) -- Steven From rosuav at gmail.com Wed Jul 10 12:52:31 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 11 Jul 2013 02:52:31 +1000 Subject: Prime number generator In-Reply-To: <51dd8560$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <51dd8560$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, Jul 11, 2013 at 2:01 AM, Steven D'Aprano wrote: > On Thu, 11 Jul 2013 00:00:59 +1000, Chris Angelico wrote: >> Thirdly, is there any sort of half-sane benchmark that I >> can compare this code to? And finally, whose wheel did I reinvent here? >> What name would this algorithm have? > > I can't directly answer that question, but I can make a shameless plug > for this: > > https://pypi.python.org/pypi/pyprimes > > If your prime generator performs better than, or worse than, all of those > in the above module, I may have to steal it from you :-) Ha, that's what I was hoping for. My algorithm outperforms several of yours! Look! Rosuav: 0.10868923284942639 awful_primes: 16.55546780386893 naive_primes1: 2.6105180965737276 naive_primes2: 1.358270674066116 trial_division: 0.06926075800136644 turner: 0.5736550315752424 croft: 0.007141969160883832 sieve: 0.01786707528470899 cookbook: 0.014790147909859996 wheel: 0.015050236831779529 Okay, so I outperform only algorithms listed as toys... :| Here's similar figures, going to a higher cutoff (I kept it low for the sake of awful_primes) and using only the algos designed for real use: Rosuav: 2.1318494389650082 croft: 0.11628042111497416 sieve: 0.26868582459502566 cookbook: 0.21551174800149164 wheel: 0.4761577239565362 I didn't seriously think that this would outperform mathematically proven and efficient algorithms, it was just a toy. But at least now I know: It's roughly 20 times slower than a leading algo. And that's after the bas-suggested improvement of using a heap. But hey. If you want the code, you're welcome to it - same with anyone else. Here's the heap version as used in the above timings. MIT license. def primes(): """Generate an infinite series of prime numbers.""" from heapq import heappush,heappop i=2 yield 2 prime=[(2,2)] # Heap while True: smallest, prm = heappop(prime) heappush(prime, (smallest+prm, prm)) if i Message-ID: <51f79c4c$0$1672$e4fe514c@dreader35.news.xs4all.nl> In article , Chris Angelico wrote: >And now for something completely different. > >I knocked together a prime number generator, just for the fun of it, >that works like a Sieve of Eratosthenes but unbounded. It keeps track >of all known primes and the "next composite" that it will produce - >for instance, after yielding 13, the prime map will be {2: 20, 3: 18, >5: 20, 7: 21, 11: 22, 13: 26}, each one mapped to the first multiple >greater than 13. > >Notable in the algorithm is an entire lack of division, or even >multiplication. Everything is done with addition. > >So, a few questions. Firstly, is there a stdlib way to find the key >with the lowest corresponding value? In the above map, it would return >3, because 18 is the lowest value in the list. I want to do this with >a single pass over the dictionary. Secondly, can the "while >iit, but not quite there. Thirdly, is there any sort of half-sane >benchmark that I can compare this code to? And finally, whose wheel >did I reinvent here? What name would this algorithm have? Notice that all values from i on are possibly present. So you are better off with a list indexed by forthcoming i's and each item containing a list of primes. What you do then, more or less, is keep track of all dividers of primes to be. This promises to be reasonable efficient. I've done a similar thing in Forth. I've also done a slightly different but related parallel program on a multi-processor Forth machine where each processor takes care of one prime. There is an unpleasant fact about this kind of generators. If you want to go unbounded, you've no choice but remember all primes. If you go bounded, you need to remember 168 up til 1M, say sqrt(limit)/log(limit). This dramatic difference (and the lack of processors) leads one quickly to decide for some upper bound. > >Code tested on Python 3.3, would probably run fine on pretty much any >Python that supports yield, though I don't have a Py2.2 to test from >__future__ import generators on! I had problems with the print statement on a 2 version, fixed easy enough. > >ChrisA > ># -- start -- >def primes(): > """Generate an infinite series of prime numbers.""" > i=2 > yield 2 > prime={2:2} # Map a prime number to its next composite (but bootstrap with 2:2) > while True: > # Find the smallest value in prime[] and its key. > # Is there a standard library way to do this?? > # (If two values are equal smallest, either can be returned.) > prm=None > for p,val in prime.items(): > if prm is None or val prm,smallest=p,val > prime[prm]+=prm > while i yield i > prime[i]=i+i > i+=1 > if i==smallest: i+=1 > >gen=primes() >for i in range(30): > print(next(gen),end="\t") # Star Trek? >print() ># -- end -- -- Albert van der Horst, UTRECHT,THE NETHERLANDS Economic growth -- being exponential -- ultimately falters. albert at spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst From ian.g.kelly at gmail.com Tue Jul 30 13:33:46 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Tue, 30 Jul 2013 11:33:46 -0600 Subject: Prime number generator In-Reply-To: <51f79c4c$0$1672$e4fe514c@dreader35.news.xs4all.nl> References: <51f79c4c$0$1672$e4fe514c@dreader35.news.xs4all.nl> Message-ID: On Tue, Jul 30, 2013 at 4:58 AM, Albert van der Horst wrote: > Notice that all values from i on are possibly present. > So you are better off with a list indexed by forthcoming i's and > each item containing a list of primes. What you do then, more or less, > is keep track of all dividers of primes to be. > This promises to be reasonable efficient. > I've done a similar thing in Forth. What do you do when you get up into the billions? It seems inefficient to keep a mostly empty billion-element list hanging around. > There is an unpleasant fact about this kind of generators. > If you want to go unbounded, you've no choice but remember all > primes. If you go bounded, you need to remember 168 up til 1M, > say sqrt(limit)/log(limit). This dramatic difference (and the lack > of processors) leads one quickly to decide for some upper bound. Actually, take a look at the generator that I posted in this thread, which avoids this. It is unbounded but gets away with only remembering sqrt(N) primes by recursively regenerating the primes already seen as needed. Thus it only uses (2 * N**(1/2) + 4 * N**(1/4) 8 * N**(1/8) + ...) / log(N) extra memory, which I believe is O(sqrt(N)). From joshua at landau.ws Wed Jul 10 11:43:32 2013 From: joshua at landau.ws (Joshua Landau) Date: Wed, 10 Jul 2013 16:43:32 +0100 Subject: Prime number generator In-Reply-To: References: Message-ID: On 10 July 2013 15:00, Chris Angelico wrote: > And now for something completely different. > > I knocked together a prime number generator, just for the fun of it, > that works like a Sieve of Eratosthenes but unbounded. It keeps track > of all known primes and the "next composite" that it will produce - > for instance, after yielding 13, the prime map will be {2: 20, 3: 18, > 5: 20, 7: 21, 11: 22, 13: 26}, each one mapped to the first multiple > greater than 13. > > Notable in the algorithm is an entire lack of division, or even > multiplication. Everything is done with addition. > > So, a few questions. Firstly, is there a stdlib way to find the key > with the lowest corresponding value? Of course there is. > In the above map, it would return > 3, because 18 is the lowest value in the list. I want to do this with > a single pass over the dictionary. Secondly, can the "while > i It's almost asking for > it, but not quite there. Thirdly, is there any sort of half-sane > benchmark that I can compare this code to? Of course there is. I have no clue what, though. > And finally, whose wheel > did I reinvent here? Mine :P. That just proves the algorithm is older still, though. You also did it much more neatly and much more slowly, though. The trick is to avoid that repeated effort of finding the minimum in the dict by just keeping a sorted list. I've mocked that up just now: # Faster code # from blist import sortedlist def generate_primes(): primes = sortedlist() primes.add([4, 2]) yield 2 i = 3 while True: next_nonprime, prime_factor = primes.pop(0) for prime in range(i, next_nonprime): yield prime primes.add([prime*2, prime]) i = next_nonprime + 1 primes.add([next_nonprime + prime_factor, prime_factor]) # No longer code # > What name would this algorithm have? I'll call it "the flamingo". > Code tested on Python 3.3, would probably run fine on pretty much any > Python that supports yield, though I don't have a Py2.2 to test from > __future__ import generators on! > > ChrisA > > # -- start -- def generate_primes(): """Generate an infinite series of prime numbers.""" i = 2 yield 2 primes = {2:2} # Map a prime number to its next composite (but bootstrap with 2:2) while True: prm = min(primes, key=primes.__getitem__) smallest = primes[prm] primes[prm] += prm for prm in range(i, smallest): yield prm primes[i] = 2*i i = smallest + 1 gen=generate_primes() for i in range(30): print(next(gen),end="\t") # Star Trek? print() > # -- end -- From rosuav at gmail.com Wed Jul 10 12:03:14 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 11 Jul 2013 02:03:14 +1000 Subject: Prime number generator In-Reply-To: References: Message-ID: On Thu, Jul 11, 2013 at 1:43 AM, Joshua Landau wrote: >> So, a few questions. Firstly, is there... > Of course there is. > >> Secondly, can the... > Of course it can. > >> Thirdly, is there... > Of course there is. I have no clue what, though. Heh, I guess I was asking for that kind of response :) > The trick > is to avoid that repeated effort of finding the minimum in the dict by > just keeping a sorted list. > > I've mocked that up just now: > > # Faster code # > from blist import sortedlist Hmm, blist isn't part of the standard library, so I'd have to hunt that down. Presumably it's on PyPI. >> What name would this algorithm have? > I'll call it "the flamingo". Guess that's as good a name as any! > def generate_primes(): > """Generate an infinite series of prime numbers.""" > i = 2 > yield 2 > > primes = {2:2} # Map a prime number to its next composite (but > bootstrap with 2:2) > while True: > prm = min(primes, key=primes.__getitem__) > smallest = primes[prm] > > primes[prm] += prm > > for prm in range(i, smallest): > yield prm > primes[i] = 2*i > > i = smallest + 1 > > gen=generate_primes() > for i in range(30): > print(next(gen),end="\t") # Star Trek? > print() And once again, a version that's slower than the original. This one does at least have the advantage of readability, though, and it's only a little slower, so I'd say this is the current winner. I would still replace 2*i with i+i for the sake of boasting "no multiplication", though :) In terms of the "one pass" criterion I put on the find-smallest algorithm, I had been seeking to avoid anything that involved constant lookups into primes[]. But this solution does look very clean and simple. I think. ChrisA From ian.g.kelly at gmail.com Wed Jul 10 12:16:32 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 10 Jul 2013 10:16:32 -0600 Subject: Prime number generator In-Reply-To: References: Message-ID: On Wed, Jul 10, 2013 at 8:00 AM, Chris Angelico wrote: > And now for something completely different. > > I knocked together a prime number generator, just for the fun of it, > that works like a Sieve of Eratosthenes but unbounded. It keeps track > of all known primes and the "next composite" that it will produce - > for instance, after yielding 13, the prime map will be {2: 20, 3: 18, > 5: 20, 7: 21, 11: 22, 13: 26}, each one mapped to the first multiple > greater than 13. Cool! I have a similar generator, and instead of mapping primes to next composites, it maps next composites to lists of primes. It works using increment-by-2 and checking the dictionary rather than searching for the min of the dictionary. You could though adapt that data structure and just use min(prime) to find the next composite (although as somebody else noted, a heap would probably be more efficient). The other interesting thing about my sieve is that it's a recursive generator. I'll dig it up later and share it. From ian.g.kelly at gmail.com Wed Jul 10 12:54:35 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 10 Jul 2013 10:54:35 -0600 Subject: Prime number generator In-Reply-To: References: Message-ID: On Wed, Jul 10, 2013 at 10:16 AM, Ian Kelly wrote: > The other interesting thing about my sieve is that it's a recursive > generator. I'll dig it up later and share it. As promised. Apologies for the excessive commenting. As noted, this implementation is a recursive generator, which is done so that the primes in the sieve can go only up to the square root of the current prime, rather than tossing in every prime seen so far. from collections import defaultdict from itertools import count, islice def primes(): yield 2 # Set up the data structures. multiples implements the sieve as a dict # that marks upcoming composite numbers and records the prime factors # that disqualified them. multiples = defaultdict(list) # subgen is a recursive sub-generator that supplies prime factors to be fed # into the sieve. islice is used to skip 2 (since we only consider odd # numbers) and 3 (since we can't get it from subgen before we've yielded it # below). subgen = islice(primes(), 2, None) # Initialize the sieve with the first prime factor q. q = 3 # When we reach threshold (the square of the last prime added to the sieve), # it's time to add another prime to the sieve. By construction, threshold # is always an odd composite. threshold = q * q multiples[threshold].append(q + q) # The main loop tries every odd integer, starting with 3. for p in count(3, 2): if p in multiples: # p is composite. Discard the entry from multiples and mark the # next odd multiple of each prime factor that disqualified p. # Because the multiples dict actually stores the doubles of the # prime factors, they need only be added once to get to the next # odd multiple. for q in multiples[p]: multiples[p + q].append(q) del multiples[p] if p == threshold: # All primes up to the previous q ** 2 have been generated. # Get the next prime factor q and add it to the sieve. q = subgen.next() threshold = q * q multiples[threshold].append(q + q) else: # p is prime. Yay! yield p From rosuav at gmail.com Wed Jul 10 13:14:30 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 11 Jul 2013 03:14:30 +1000 Subject: Prime number generator In-Reply-To: References: Message-ID: On Thu, Jul 11, 2013 at 2:54 AM, Ian Kelly wrote: > As promised. Apologies for the excessive commenting. As noted, this > implementation is a recursive generator, which is done so that the > primes in the sieve can go only up to the square root of the current > prime, rather than tossing in every prime seen so far. > Nice. Comes in at 0.23369310904039478 in my oh-so-scientific benchmarking, so it's comparable to several of Steven's algorithms. Runs on Python 3.3 with just the tiny tweak of converting iter.next() to next(iter). ChrisA From g.rodola at gmail.com Wed Jul 10 11:46:13 2013 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Wed, 10 Jul 2013 17:46:13 +0200 Subject: ANN: psutil 1.0.0 released Message-ID: Hi there folks, I'm pleased to announce the 1.0.0 release of psutil: http://code.google.com/p/psutil/ === About === psutil is a module providing an interface for retrieving information on all running processes and system utilization (CPU, memory, disks, network, users) in a portable way by using Python. It currently supports Linux, Windows, OSX, FreeBSD and Sun Solaris, both 32-bit and 64-bit, with Python versions from 2.4 to 3.3 by using a single code base. === New features === This new release finally includes support for Sun Solaris systems! I wish to thank Justin Venus who first wrote a prototype in Cython which I then converted in C (full story here: https://code.google.com/p/psutil/issues/detail?id=18). Complete list of bugfixes and enhancements is here: https://psutil.googlecode.com/hg/HISTORY === Compatitility notes === There's a couple of things you may want to know in terms of backward compatibility: * Process.get_connections() 'status' field is no longer a string but a constant object (psutil.CONN_*) * Process.get_connections() 'local_address' and 'remote_address' fields have been renamed to 'laddr' and 'raddr' * psutil.network_io_counters() has been renamed to psutil.net_io_counters() There are still aliases for the old names. They will issue a DeprecationWarning though. === Links === * Home page: http://code.google.com/p/psutil * Source tarball: http://psutil.googlecode.com/files/psutil-1.0.0.tar.gz * API Reference: http://code.google.com/p/psutil/wiki/Documentation Please try out this new release and let me know if you experience any problem by filing issues on the bug tracker. Thanks in advance and hooray for the new Solaris support! ;-) All the best, --- Giampaolo Rodola' http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ http://code.google.com/p/pysendfile/ From steve+comp.lang.python at pearwood.info Wed Jul 10 12:06:20 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 10 Jul 2013 16:06:20 GMT Subject: ANN: psutil 1.0.0 released References: Message-ID: <51dd867c$0$9505$c3e8da3$5496439d@news.astraweb.com> On Wed, 10 Jul 2013 17:46:13 +0200, Giampaolo Rodola' wrote: > Hi there folks, > I'm pleased to announce the 1.0.0 release of psutil: > http://code.google.com/p/psutil/ Congratulations on the 1.0.0 release! -- Steven From jurko.gospodnetic at pke.hr Wed Jul 10 12:07:15 2013 From: jurko.gospodnetic at pke.hr (=?UTF-8?B?SnVya28gR29zcG9kbmV0acSH?=) Date: Wed, 10 Jul 2013 18:07:15 +0200 Subject: ANN: psutil 1.0.0 released In-Reply-To: References: Message-ID: <51DD86B3.6090608@pke.hr> Hi. > I'm pleased to announce the 1.0.0 release of psutil: > http://code.google.com/p/psutil/ Great! :-) Btw. any change you can put up a prebuilt installer for a 64-bit built with Python 3.3? You have one for Python 3.2 (http://code.google.com/p/psutil/downloads/list), but the version for Python 3.3 is not there. Best regards, Jurko Gospodneti? From g.rodola at gmail.com Thu Jul 11 11:02:01 2013 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Thu, 11 Jul 2013 17:02:01 +0200 Subject: ANN: psutil 1.0.0 released In-Reply-To: <51DD86B3.6090608@pke.hr> References: <51DD86B3.6090608@pke.hr> Message-ID: > Congratulations on the 1.0.0 release! Thanks a lot. =) > Btw. any change you can put up a prebuilt installer for a 64-bit built > with Python 3.3? You have one for Python 3.2 > (http://code.google.com/p/psutil/downloads/list), but the version for Python > 3.3 is not there. Unfortunately I'm having troubles with Visual Studio 64-bit I still haven't managed to fix. Hope I will get around that soon. --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ http://code.google.com/p/pysendfile/ From dihedral88888 at gmail.com Thu Jul 11 11:31:35 2013 From: dihedral88888 at gmail.com (88888 Dihedral) Date: Thu, 11 Jul 2013 08:31:35 -0700 (PDT) Subject: ANN: psutil 1.0.0 released In-Reply-To: References: <51DD86B3.6090608@pke.hr> Message-ID: <6e27e03c-d6db-487b-b2f4-c0d051ff7190@googlegroups.com> Giampaolo Rodola'? 2013?7?11????UTC+8??11?02?01???? > > Congratulations on the 1.0.0 release! > > > > Thanks a lot. =) > > > > > Btw. any change you can put up a prebuilt installer for a 64-bit built > > > with Python 3.3? You have one for Python 3.2 > > > (http://code.google.com/p/psutil/downloads/list), but the version for Python > > > 3.3 is not there. > > > > Unfortunately I'm having troubles with Visual Studio 64-bit I still > > haven't managed to fix. > > Hope I will get around that soon. > > > > --- Giampaolo > > > > http://code.google.com/p/pyftpdlib/ > > http://code.google.com/p/psutil/ > > http://code.google.com/p/pysendfile/ Do you plan to install 8 or 16 G BYTES DRAM? From dotancohen at gmail.com Thu Jul 11 12:15:12 2013 From: dotancohen at gmail.com (Dotan Cohen) Date: Thu, 11 Jul 2013 19:15:12 +0300 Subject: ANN: psutil 1.0.0 released In-Reply-To: References: Message-ID: Thanks, this looks really nice. I was duplicating some of this for my CLI-based webserver control panel: https://github.com/dotancohen/burton As soon as I integrate psutil into Burton I'll add it to the README and such. How would you like me to mention attribution exactly? -- Dotan Cohen http://gibberish.co.il http://what-is-what.com From doug.farrell at gmail.com Wed Jul 10 16:44:51 2013 From: doug.farrell at gmail.com (writeson) Date: Wed, 10 Jul 2013 13:44:51 -0700 (PDT) Subject: Twisted and argparse Message-ID: <795693de-495d-4257-81ca-3bba3d6a98aa@googlegroups.com> Hi all, I'm trying to write an Twisted program that uses the Application object (and will run with twistd) and I'd like to parse command line arguments. The Twisted documentation shows how to use a Twisted thing called usage.Options. However to me this looks a lot like the older Python module getopts. Is there a way to use the argparse module with a Twisted Application? Thanks! Doug From cndefend-stuff at yahoo.co.uk Wed Jul 10 12:43:15 2013 From: cndefend-stuff at yahoo.co.uk (Chris Nash) Date: Wed, 10 Jul 2013 16:43:15 +0000 (UTC) Subject: python for loop References: <0rc5t4l6lqethggahj6auupb5vj67j78gk@4ax.com> <72i5t4tgfo2h4gd6ggcs02flkca85kgoku@4ax.com> <0dbc9665-3ee8-4d62-8e9e-75c6d700076e@f1g2000prb.googlegroups.com> Message-ID: The first item in a sequence is at index zero because it is that far away from the beginning. The second item is one away from the beginning. That is the reason for zero-based indexing. From joshua at landau.ws Wed Jul 10 23:15:37 2013 From: joshua at landau.ws (Joshua Landau) Date: Thu, 11 Jul 2013 04:15:37 +0100 Subject: Documenting builtin methods Message-ID: I have this innocent and simple code: from collections import deque exhaust_iter = deque(maxlen=0).extend exhaust_iter.__doc__ = "Exhaust an iterator efficiently without caching any of its yielded values." Obviously it does not work. Is there a way to get it to work simply and without creating a new scope (which would be a rather inefficient a way to set documentation, and would hamper introspection)? How about dropping the "simply" requirement? From dreamingforward at gmail.com Wed Jul 10 23:29:14 2013 From: dreamingforward at gmail.com (Mark Janssen) Date: Wed, 10 Jul 2013 20:29:14 -0700 Subject: Documenting builtin methods In-Reply-To: References: Message-ID: > I have this innocent and simple code: > > from collections import deque > exhaust_iter = deque(maxlen=0).extend > exhaust_iter.__doc__ = "Exhaust an iterator efficiently without > caching any of its yielded values." > > Obviously it does not work. Is there a way to get it to work simply > and without creating a new scope (which would be a rather inefficient > a way to set documentation, and would hamper introspection)? > > How about dropping the "simply" requirement? I think the canonical way to specialize a class (even if it's only docstrings or method re-names) is to extend it with a new class. markj From ben+python at benfinney.id.au Wed Jul 10 23:57:06 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 11 Jul 2013 13:57:06 +1000 Subject: Documenting builtin methods References: Message-ID: <7wk3kxsq99.fsf@benfinney.id.au> Joshua Landau writes: > I have this innocent and simple code: > > from collections import deque > exhaust_iter = deque(maxlen=0).extend > exhaust_iter.__doc__ = "Exhaust an iterator efficiently without > caching any of its yielded values." > > Obviously it does not work. Right. It raises a SyntaxError because you've got a simple string literal that isn't closed before the end of the line. > Is there a way to get it to work simply What behaviour would you expect from that? We can't guess what you mean by ?get it to work? unless we know what you're expecting and how the observed behaviour is different. A good way to do this would be to show some actual code that demonstrates the surprising behaviour (you may need to post using a service that doesn't munge your text), along with the traceback if any. Then, show a session that behaves the way you'd expect it to behave, and we can explain either how to achieve that or why it can't. -- \ Q: ?I've heard that Linux causes cancer...? Torvalds: ?That's a | `\ filthy lie. Besides, it was only in rats and has not been | _o__) reproduced in humans.? ?Linus Torvalds | Ben Finney From joshua at landau.ws Thu Jul 11 00:13:50 2013 From: joshua at landau.ws (Joshua Landau) Date: Thu, 11 Jul 2013 05:13:50 +0100 Subject: Documenting builtin methods In-Reply-To: <7wk3kxsq99.fsf@benfinney.id.au> References: <7wk3kxsq99.fsf@benfinney.id.au> Message-ID: On 11 July 2013 04:57, Ben Finney wrote: > Joshua Landau writes: > >> I have this innocent and simple code: >> >> from collections import deque >> exhaust_iter = deque(maxlen=0).extend >> exhaust_iter.__doc__ = "Exhaust an iterator efficiently without >> caching any of its yielded values." >> >> Obviously it does not work. > > Right. It raises a SyntaxError because you've got a simple string > literal that isn't closed before the end of the line. Is removing one newline such a big deal to you really? >> Is there a way to get it to work simply > > What behaviour would you expect from that? We can't guess what you mean > by ?get it to work? unless we know what you're expecting and how the > observed behaviour is different. Obviously I want to set .__doc__. What else could you possibly glean from me trying to set .__doc__? Are you that far away from an interpreter that you can't see that it doesn't work? > A good way to do this would be to show some actual code that > demonstrates the surprising behaviour That's what I did. > (you may need to post using a > service that doesn't munge your text), along with the traceback if any. I imagine you could've run it yourself, seeing as it's a small, reproducible test sample. > Then, show a session that behaves the way you'd expect it to behave, and > we can explain either how to achieve that or why it can't. You know, I'd like it to not crash. And do what it would normally do if it didn't crash. I'm not sure why you've given this response, as my question was fairly explanatory given that you spend the 10 seconds to run the code if you couldn't see the crash by eye. It's a whole lot shorter than the time you spent on the comment. There are a lot of people who post beasts of code to this list that isn't remotely runnable and with cryptic cropped errors that you have no obvious way of getting a duplicate of. I did not do that. For convenience, and because despite my tone of surprise I hold no grudge or disdain, here is the relevant part of the traceback: AttributeError: attribute '__doc__' of 'builtin_function_or_method' objects is not writable If you're still unhappy about the text-wrapping: from collections import deque exhaust_iter = deque(maxlen=0).extend exhaust_iter.__doc__ = "Exhaust an iterator efficiently" From joshua at landau.ws Thu Jul 11 00:20:43 2013 From: joshua at landau.ws (Joshua Landau) Date: Thu, 11 Jul 2013 05:20:43 +0100 Subject: Documenting builtin methods In-Reply-To: References: <7wk3kxsq99.fsf@benfinney.id.au> Message-ID: On 11 July 2013 05:13, Joshua Landau wrote: > Ah, I get it. It is easy to misread my post as "I have this exhaust_iter" and it's obvious it doesn't work because why else would I post here what do I do HALP! Yeah, sorry -- it wasn't meant to come across that way. From ben+python at benfinney.id.au Thu Jul 11 04:26:21 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 11 Jul 2013 18:26:21 +1000 Subject: Documenting builtin methods References: <7wk3kxsq99.fsf@benfinney.id.au> Message-ID: <7wvc4hjydu.fsf@benfinney.id.au> Joshua Landau writes: > On 11 July 2013 05:13, Joshua Landau wrote: > > > > Ah, I get it. It is easy to misread my post as "I have this > exhaust_iter" and it's obvious it doesn't work because why else would > I post here what do I do HALP! Right. Just because you think there's one obvious interpretation of your cry for help, doesn't mean there's *only* one obvious interpretation. It's best to be explicit: describe what behaviour you're seeing, and what behaviour you're expecting to see. -- \ ?We must respect the other fellow's religion, but only in the | `\ sense and to the extent that we respect his theory that his | _o__) wife is beautiful and his children smart.? ?Henry L. Mencken | Ben Finney From wuwei23 at gmail.com Thu Jul 11 01:35:56 2013 From: wuwei23 at gmail.com (alex23) Date: Thu, 11 Jul 2013 15:35:56 +1000 Subject: Documenting builtin methods In-Reply-To: References: Message-ID: On 11/07/2013 1:15 PM, Joshua Landau wrote: > I have this innocent and simple code: > > from collections import deque > exhaust_iter = deque(maxlen=0).extend > exhaust_iter.__doc__ = "Exhaust an iterator efficiently without > caching any of its yielded values." > > Obviously it does not work. Is there a way to get it to work simply > and without creating a new scope (which would be a rather inefficient > a way to set documentation, and would hamper introspection)? I would just go with the most obvious approach: def exhaust_iter(iter): """ Exhaust an iterator efficiently without caching any of its yielded values """ deque(maxlen=0).extend(iter) It's not going to be that inefficient unless you're calling it in a long inner loop. From steve at pearwood.info Thu Jul 11 02:06:36 2013 From: steve at pearwood.info (Steven D'Aprano) Date: 11 Jul 2013 06:06:36 GMT Subject: Documenting builtin methods References: Message-ID: <51de4b6c$0$11094$c3e8da3@news.astraweb.com> On Thu, 11 Jul 2013 04:15:37 +0100, Joshua Landau wrote: > I have this innocent and simple code: > > from collections import deque > exhaust_iter = deque(maxlen=0).extend At this point, exhaust_iter is another name for the bound instance method "extend" of one specific deque instance. Other implementations may do otherwise[1], but CPython optimizes built-in methods and functions. E.g. they have no __dict__ so you can't add attributes to them. When you look up exhaust_iter.__doc__, you are actually looking up (type(exhaust_iter)).__doc__, which is a descriptor: py> type(exhaust_iter).__doc__ py> type(type(exhaust_iter).__doc__) Confused yet? Don't worry, you will be... So, calling exhaust_iter.__doc__: 1) looks up '__doc__' on the class "builtin_function_or_method", not the instance; 2) which looks up '__doc__' on the class __dict__: py> type(exhaust_iter).__dict__['__doc__'] 3) This is a descriptor with __get__ and __set__ methods. Because the actual method is written in C, you can't access it's internals except via the API: even the class __dict__ is not really a dict, it's a wrapper around a dict: py> type(type(exhaust_iter).__dict__) Anyway, we have a descriptor that returns the doc string: py> descriptor = type(exhaust_iter).__doc__ py> descriptor.__get__(exhaust_iter) 'Extend the right side of the deque with elements from the iterable' My guess is that it is fetching this from some private C member, which you can't get to from Python except via the descriptor. And you can't set it: py> descriptor.__set__(exhaust_iter, '') Traceback (most recent call last): File "", line 1, in AttributeError: attribute '__doc__' of 'builtin_function_or_method' objects is not writable which is probably because if you could write to it, it would change the docstring for *every* deque. And that would be bad. If this were a pure-Python method, you could probably bypass the descriptor, but it's a C-level built-in. I think you're out of luck. I think the right solution here is the trivial: def exhaust(it): """Doc string here.""" deque(maxlen=0).extend(it) which will be fast enough for all but the tightest inner loops. But if you really care about optimizing this: def factory(): eatit = deque(maxlen=0).extend def exhaust_iter(it): """Doc string goes here""" eatit(it) return exhaust_iter exhaust_it = factory() del factory which will be about as efficient as you can get while still having a custom docstring. But really, I'm having trouble understanding what sort of application would have "run an iterator to exhaustion without doing anything with the values" as the performance bottleneck :-) > exhaust_iter.__doc__ = "Exhaust an iterator efficiently [...]" > > Obviously it does not work. Even if it did work, it would not do what you hope. Because __doc__ is a dunder attribute (double leading and trailing underscores), help() currently looks it up on the class, not the instance: class Spam: "Spam spam spam" x = Spam() help(x) => displays "Spam spam spam" x.__doc__ = "Yummy spam" help(x) => still displays "Spam spam spam" > Is there a way to get it to work simply and > without creating a new scope (which would be a rather inefficient a way > to set documentation, and would hamper introspection)? > > How about dropping the "simply" requirement? I don't believe so. [1] IronPython and Jython both currently do the same thing as CPython, so even if this is not explicitly language-defined behaviour, it looks like it may be de facto standard behaviour. -- Steven From rosuav at gmail.com Thu Jul 11 03:06:39 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 11 Jul 2013 17:06:39 +1000 Subject: Documenting builtin methods In-Reply-To: <51de4b6c$0$11094$c3e8da3@news.astraweb.com> References: <51de4b6c$0$11094$c3e8da3@news.astraweb.com> Message-ID: On Thu, Jul 11, 2013 at 4:06 PM, Steven D'Aprano wrote: > I think the right solution here is the trivial: > > def exhaust(it): > """Doc string here.""" > deque(maxlen=0).extend(it) > > > which will be fast enough for all but the tightest inner loops. But if > you really care about optimizing this: > > > def factory(): > eatit = deque(maxlen=0).extend > def exhaust_iter(it): > """Doc string goes here""" > eatit(it) > return exhaust_iter > > exhaust_it = factory() > del factory > > > which will be about as efficient as you can get while still having a > custom docstring. Surely no reason to go for the factory function: def exhaust(it,eatit=deque(maxlen=0).extend): eatit(it) ChrisA From steve at pearwood.info Thu Jul 11 03:15:08 2013 From: steve at pearwood.info (Steven D'Aprano) Date: 11 Jul 2013 07:15:08 GMT Subject: Documenting builtin methods References: <51de4b6c$0$11094$c3e8da3@news.astraweb.com> Message-ID: <51de5b7b$0$29862$c3e8da3$5496439d@news.astraweb.com> On Thu, 11 Jul 2013 17:06:39 +1000, Chris Angelico wrote: > On Thu, Jul 11, 2013 at 4:06 PM, Steven D'Aprano > wrote: >> I think the right solution here is the trivial: >> >> def exhaust(it): >> """Doc string here.""" >> deque(maxlen=0).extend(it) >> >> >> which will be fast enough for all but the tightest inner loops. But if >> you really care about optimizing this: >> >> >> def factory(): >> eatit = deque(maxlen=0).extend >> def exhaust_iter(it): >> """Doc string goes here""" >> eatit(it) >> return exhaust_iter >> >> exhaust_it = factory() >> del factory >> >> >> which will be about as efficient as you can get while still having a >> custom docstring. > > Surely no reason to go for the factory function: > > def exhaust(it,eatit=deque(maxlen=0).extend): > eatit(it) Now you have the function accept a second argument, which is public, just to hold a purely internal reference to something that you don't want the caller to replace. -- Steven From rosuav at gmail.com Thu Jul 11 03:23:06 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 11 Jul 2013 17:23:06 +1000 Subject: Documenting builtin methods In-Reply-To: <51de5b7b$0$29862$c3e8da3$5496439d@news.astraweb.com> References: <51de4b6c$0$11094$c3e8da3@news.astraweb.com> <51de5b7b$0$29862$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, Jul 11, 2013 at 5:15 PM, Steven D'Aprano wrote: > On Thu, 11 Jul 2013 17:06:39 +1000, Chris Angelico wrote: > >> On Thu, Jul 11, 2013 at 4:06 PM, Steven D'Aprano >> wrote: >>> I think the right solution here is the trivial: >>> >>> def exhaust(it): >>> """Doc string here.""" >>> deque(maxlen=0).extend(it) >>> >>> >>> which will be fast enough for all but the tightest inner loops. But if >>> you really care about optimizing this: >>> >>> >>> def factory(): >>> eatit = deque(maxlen=0).extend >>> def exhaust_iter(it): >>> """Doc string goes here""" >>> eatit(it) >>> return exhaust_iter >>> >>> exhaust_it = factory() >>> del factory >>> >>> >>> which will be about as efficient as you can get while still having a >>> custom docstring. >> >> Surely no reason to go for the factory function: >> >> def exhaust(it,eatit=deque(maxlen=0).extend): >> eatit(it) > > Now you have the function accept a second argument, which is public, just > to hold a purely internal reference to something that you don't want the > caller to replace. True, but doesn't that happen fairly often with default args? Usually it's in the "int=int" notation to snapshot for performance. ChrisA From joshua at landau.ws Thu Jul 11 19:11:11 2013 From: joshua at landau.ws (Joshua Landau) Date: Fri, 12 Jul 2013 00:11:11 +0100 Subject: Documenting builtin methods In-Reply-To: <51de4b6c$0$11094$c3e8da3@news.astraweb.com> References: <51de4b6c$0$11094$c3e8da3@news.astraweb.com> Message-ID: On 11 July 2013 07:06, Steven D'Aprano wrote: > > But really, I'm having trouble understanding what sort of application > would have "run an iterator to exhaustion without doing anything with the > values" as the performance bottleneck :-) Definitely not this one. Heck, there's even no real reason something as appropriately-named as "exhaust_iter" needs documentation. Largely I was asking because I'd felt I'd missed something more obvious; it seems there was not. I'm also doing some more functools stuff than usual -- this method also applies to functions generated with, say, functools.partial I had guessed. Only it does not, as you show below -- and functools.partial objects allow you to ineffectively set .__doc__ anyway. I also feel that: def factory(): eatit = deque(maxlen=0).extend def exhaust_iter(it): """Doc string goes here""" eatit(it) return exhaust_iter exhaust_it = factory() del factory is a very unobvious way to change a docstring and hides what I'm doing very effectively. Chris Angelico's method is a fair bit better in this regard, but I'm not sure it's worth it in this case. One recommendation with Chris's method is to make it keyword-only (with "*") which should keep the interface a touch cleaner. >> exhaust_iter.__doc__ = "Exhaust an iterator efficiently [...]" >> >> Obviously it does not work. > > Even if it did work, it would not do what you hope. Because __doc__ is a > dunder attribute (double leading and trailing underscores), help() > currently looks it up on the class, not the instance: I'd not considered that, and it seems to have doomed me from the start. From wuwei23 at gmail.com Thu Jul 11 23:43:05 2013 From: wuwei23 at gmail.com (alex23) Date: Fri, 12 Jul 2013 13:43:05 +1000 Subject: Documenting builtin methods In-Reply-To: References: <51de4b6c$0$11094$c3e8da3@news.astraweb.com> Message-ID: On 12/07/2013 9:11 AM, Joshua Landau wrote: > I also feel that: > > def factory(): > eatit = deque(maxlen=0).extend > def exhaust_iter(it): > """Doc string goes here""" > eatit(it) > return exhaust_iter > > exhaust_it = factory() > del factory > > is a very unobvious way to change a docstring and hides what I'm doing > very effectively. My last post seems to have been eaten by either Thunderbird or the EternalSeptember servers, but it contained an erroneous claim that the straight function version performed as well as the factory one. However, in the interim a co-worker has come up with a slightly faster variant: from functools import partial from collections import deque class exhaust_it(partial): """custom doc string""" exhaust_it = exhaust_it(deque(maxlen=0).extend) Shadowing the class name with the partial instance will ensure it has the same name when accessed via help(), and it's a simple way to avoid needing to clean up the namespace, as well. From joshua at landau.ws Fri Jul 12 03:09:57 2013 From: joshua at landau.ws (Joshua Landau) Date: Fri, 12 Jul 2013 08:09:57 +0100 Subject: Documenting builtin methods In-Reply-To: References: <51de4b6c$0$11094$c3e8da3@news.astraweb.com> Message-ID: On 12 July 2013 04:43, alex23 wrote: > > My last post seems to have been eaten by either Thunderbird or the > EternalSeptember servers, but it contained an erroneous claim that the > straight function version performed as well as the factory one. However, in > the interim a co-worker has come up with a slightly faster variant: > > from functools import partial > from collections import deque > > class exhaust_it(partial): > """custom doc string""" > > exhaust_it = exhaust_it(deque(maxlen=0).extend) > > Shadowing the class name with the partial instance will ensure it has the > same name when accessed via help(), and it's a simple way to avoid needing > to clean up the namespace, as well. That's beautiful. You could even trivially make a wrapper function: def wrap_docstring(function, docstring, *, name=None): class Wrapper(partial): pass Wrapper.__name__ = function.__name__ if name is None else name Wrapper.__doc__ = docstring return Wrapper(function) which is no slower. You get great introspection through the "func" attribute, too :). Also: >>> times = time_raw(), time_function(), time_factory(), time_argument_hack(), time_partial() >>> [round(time/times[0], 1) for time in times] [1.0, 16.8, 3.1, 3.0, 1.8] This times almost purely the constant overhead by calling exhaust_iterabe on an empty iterable. So your friend wins the premature optimisation test, too. From jsf80238 at gmail.com Thu Jul 11 00:57:09 2013 From: jsf80238 at gmail.com (Jason Friedman) Date: Wed, 10 Jul 2013 22:57:09 -0600 Subject: Concurrent writes to the same file Message-ID: Other than using a database, what are my options for allowing two processes to edit the same file at the same time? When I say same time, I can accept delays. I considered lock files, but I cannot conceive of how I avoid race conditions. -------------- next part -------------- An HTML attachment was scrubbed... URL: From davea at davea.name Thu Jul 11 01:34:48 2013 From: davea at davea.name (Dave Angel) Date: Thu, 11 Jul 2013 01:34:48 -0400 Subject: Concurrent writes to the same file In-Reply-To: References: Message-ID: On 07/11/2013 12:57 AM, Jason Friedman wrote: > Other than using a database, what are my options for allowing two processes > to edit the same file at the same time? When I say same time, I can accept > delays. I considered lock files, but I cannot conceive of how I avoid race > conditions. > In general, no. That's what a database is for. Now, you presumably have some reason to avoid database, but in its stead you have to specify some other limitations. To start with, what do you mean by "the same time"? If each process can modify the entire file, then there's no point in one process reading the file at all until it has the lock. So the mechanism would be 1) wait till you can acquire the lock 2) open the file, read it, modify it, flush and close 3) release the lock To come up with an appropriate lock, it'd be nice to start by specifying the entire environment. Which Python, which OS? Are the two processes on the same CPU, what's the file system, and is it locally mounted? -- DaveA From ndbecker2 at gmail.com Thu Jul 11 11:58:57 2013 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 11 Jul 2013 11:58:57 -0400 Subject: Concurrent writes to the same file References: Message-ID: Dave Angel wrote: > On 07/11/2013 12:57 AM, Jason Friedman wrote: >> Other than using a database, what are my options for allowing two processes >> to edit the same file at the same time? When I say same time, I can accept >> delays. I considered lock files, but I cannot conceive of how I avoid race >> conditions. >> > > In general, no. That's what a database is for. > > Now, you presumably have some reason to avoid database, but in its stead > you have to specify some other limitations. To start with, what do you > mean by "the same time"? If each process can modify the entire file, > then there's no point in one process reading the file at all until it > has the lock. So the mechanism would be > 1) wait till you can acquire the lock > 2) open the file, read it, modify it, flush and close > 3) release the lock > > To come up with an appropriate lock, it'd be nice to start by specifying > the entire environment. Which Python, which OS? Are the two processes > on the same CPU, what's the file system, and is it locally mounted? > > > > You can use a seperate server process to do file I/O, taking multiple inputs from clients. Like syslogd. From cs at zip.com.au Thu Jul 11 02:31:58 2013 From: cs at zip.com.au (Cameron Simpson) Date: Thu, 11 Jul 2013 16:31:58 +1000 Subject: Concurrent writes to the same file In-Reply-To: References: Message-ID: <20130711063158.GA31552@cskk.homeip.net> On 10Jul2013 22:57, Jason Friedman wrote: | Other than using a database, what are my options for allowing two processes | to edit the same file at the same time? When I say same time, I can accept | delays. I considered lock files, but I cannot conceive of how I avoid race | conditions. Sure. (Assuming UNIX. Windows probably excludes both processes having the file open at the one time, but that's not a show stopper.) You need to use a lock file to ensure only one process accesses the file at a time. While a process holds the lock, access the file. There are two basic approaches here: take lock open file for write, modify the file, close it release lock both processes have the file open for update take lock modify file, flush buffers release lock The code I use to take a lockfile is my "lockfile" context manager: https://bitbucket.org/cameron_simpson/css/src/374f650025f156554a986fb3fd472003d2a2519a/lib/python/cs/fileutils.py?at=default#cl-408 An example caller goes: with lockfile(self.csvpath): backup = "%s.bak-%s" % (self.csvpath, datetime.datetime.now().isoformat()) copyfile(self.csvpath, backup) write_csv_file(self.csvpath, self.nodedb.nodedata()) if not self.keep_backups: os.remove(backup) as shown here: https://bitbucket.org/cameron_simpson/css/src/374f650025f156554a986fb3fd472003d2a2519a/lib/python/cs/nodedb/csvdb.py?at=default#cl-171 Simplify as necessary; you just want: with lockfile(mainfile): modify(mainfile) if you have no wish for backups in case of a failed modify. If both processes need to see each other's changes then there's some more logic needed to monitor the file for changes. I've got such a scheme for CSV files in beta at present for a personal project. It won't suit all use cases; mine is well defined. Cheers, -- Cameron Simpson Is it true, Sen. Bedfellow, that your wife rides with bikers? - Milo Bloom From jsf80238 at gmail.com Fri Jul 12 00:48:08 2013 From: jsf80238 at gmail.com (Jason Friedman) Date: Thu, 11 Jul 2013 22:48:08 -0600 Subject: Concurrent writes to the same file In-Reply-To: <20130711063158.GA31552@cskk.homeip.net> References: <20130711063158.GA31552@cskk.homeip.net> Message-ID: > > https://bitbucket.org/cameron_simpson/css/src/374f650025f156554a986fb3fd472003d2a2519a/lib/python/cs/fileutils.py?at=default#cl-408 > > That looks like it will do the trick for me, thank you Cameron. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nobody at nowhere.com Thu Jul 11 04:30:35 2013 From: nobody at nowhere.com (Nobody) Date: Thu, 11 Jul 2013 09:30:35 +0100 Subject: Concurrent writes to the same file References: Message-ID: On Wed, 10 Jul 2013 22:57:09 -0600, Jason Friedman wrote: > Other than using a database, what are my options for allowing two processes > to edit the same file at the same time? When I say same time, I can accept > delays. What do you mean by "edit"? Overwriting bytes and appending bytes are simple enough, but inserting or deleting bytes such that subsequent bytes change their offsets is harder. > I considered lock files, Well, you shouldn't have, unless you're targeting a platform which doesn't support file locks (are there any left?). > but I cannot conceive of how I avoid race conditions. By using locks. E.g. fcntl.lockf() or msvcrt.locking(). From fletcherbenjiaa at gmail.com Thu Jul 11 02:42:14 2013 From: fletcherbenjiaa at gmail.com (fletcherbenjiaa) Date: Wed, 10 Jul 2013 23:42:14 -0700 (PDT) Subject: Wedding Planning Shortcuts Message-ID: <1373524934807-5024452.post@n6.nabble.com> A wedding is truly a labor of love for most engaged couples, and it's natural to feel a bit wary of the wedding planning process. However, it doesn't have to be so intimidating or cumbersome. Sure there are lots of details in even the smallest wedding, but most brides have months (and sometimes years) until the big day. And even though procrastination can sneak in here and there, you have time to get done what you need to plan the wedding of your dreams. ----- wedding planners uk -- View this message in context: http://python.6.x6.nabble.com/Wedding-Planning-Shortcuts-tp5024452.html Sent from the Python - python-list mailing list archive at Nabble.com. From alister.ware at ntlworld.com Thu Jul 11 11:57:38 2013 From: alister.ware at ntlworld.com (Alister) Date: Thu, 11 Jul 2013 15:57:38 GMT Subject: Wedding Planning Shortcuts References: Message-ID: On Wed, 10 Jul 2013 23:42:14 -0700, fletcherbenjiaa wrote: > A wedding is truly a labor of love for most engaged couples, and it's > natural to feel a bit wary of the wedding planning process. However, it > doesn't have to be so intimidating or cumbersome. Sure there are lots of > details in even the smallest wedding, but most brides have months (and > sometimes years) until the big day. And even though procrastination can > sneak in here and there, you have time to get done what you need to plan > the wedding of your dreams. > > > > > ----- > wedding planners uk so where can i download this python module? -- Send some filthy mail. From fletcherbenjiaa at gmail.com Thu Jul 11 02:44:27 2013 From: fletcherbenjiaa at gmail.com (fletcherbenjiaa) Date: Wed, 10 Jul 2013 23:44:27 -0700 (PDT) Subject: Good Books for Wedding Planners Message-ID: <1373525067857-5024454.post@n6.nabble.com> Wedding planners may seek guide from a wide range of good planning books essential in planning the perfect wedding. There are a number of reasons why one must be backed up by these books. But perhaps the biggest reason is that these good books can help wedding planners stay organized and be on top of things (even though there may be some issues or lost boutonnieres that will crop up!). ----- wedding planners uk -- View this message in context: http://python.6.x6.nabble.com/Good-Books-for-Wedding-Planners-tp5024454.html Sent from the Python - python-list mailing list archive at Nabble.com. From steve at pearwood.info Thu Jul 11 04:02:06 2013 From: steve at pearwood.info (Steven D'Aprano) Date: 11 Jul 2013 08:02:06 GMT Subject: Mypy Message-ID: <51de667e$0$29866$c3e8da3$5496439d@news.astraweb.com> Things are certainly heating up in the alternate Python compiler field. Mypy is a new, experimental, implementation of Python 3 with optional static typing and aiming for efficient compilation to machine code. http://www.mypy-lang.org/index.html -- Steven From fabiosantosart at gmail.com Thu Jul 11 04:16:50 2013 From: fabiosantosart at gmail.com (=?ISO-8859-1?Q?F=E1bio_Santos?=) Date: Thu, 11 Jul 2013 09:16:50 +0100 Subject: Mypy In-Reply-To: <51de667e$0$29866$c3e8da3$5496439d@news.astraweb.com> References: <51de667e$0$29866$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 11 Jul 2013 09:08, "Steven D'Aprano" wrote: > > Things are certainly heating up in the alternate Python compiler field. > Mypy is a new, experimental, implementation of Python 3 with optional > static typing and aiming for efficient compilation to machine code. > > http://www.mypy-lang.org/index.html > Guido tweeted that yesterday. It seems interesting. Although I'm not comfortable using a subset of the language. They seem to want to kill the GIL. This could get much more popular when they do. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Thu Jul 11 13:55:32 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 11 Jul 2013 19:55:32 +0200 Subject: Mypy In-Reply-To: References: <51de667e$0$29866$c3e8da3$5496439d@news.astraweb.com> Message-ID: F?bio Santos, 11.07.2013 10:16: > Guido tweeted that yesterday. It seems interesting. Although I'm not > comfortable using a subset of the language. > > They seem to want to kill the GIL. This could get much more popular when > they do. Then don't forget to also take a look at Cython. Stefan From anthra.norell at bluewin.ch Thu Jul 11 04:59:16 2013 From: anthra.norell at bluewin.ch (F.R.) Date: Thu, 11 Jul 2013 10:59:16 +0200 Subject: ElementTree: can't figure out a mismached-tag error Message-ID: <51DE73E4.6040007@bluewin.ch> Hi all, I haven't been able to get up to speed with XML. I do examples from the tutorials and experiment with variations. Time and time again I fail with errors messages I can't make sense of. Here's the latest one. The url is "http://finance.yahoo.com/q?s=XIDEQ&ql=0". Ubuntu 12.04 LTS, Python 2.7.3 (default, Aug 1 2012, 05:16:07) [GCC 4.6.3] >>> import xml.etree.ElementTree as ET >>> tree = ET.parse('q?s=XIDEQ') # output of wget http://finance.yahoo.com/q?s=XIDEQ&ql=0 Traceback (most recent call last): File "", line 1, in tree = ET.parse('q?s=XIDEQ') File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1183, in parse tree.parse(source, parser) File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 656, in parse parser.feed(data) File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1643, in feed self._raiseerror(v) File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1507, in _raiseerror raise err ParseError: mismatched tag: line 9, column 2 Below first nine lines. The line numbers and the following space are hand-edited in. Three dots stand for sections cut out to fit long lines. Line 6 is a bunch of "meta" statements, all of which I show on a separate line each in order to preserve the angled brackets. On all lines the angled brackets have been preserved. The mismatched character is the slash of the closing tag . What could be wrong with it? And if it is, what about fault tolerance? 1 2 3 4 XIDEQ: Summary for EXIDE TECH NEW- Yahoo! Finance 5 8 9 ^ Mismatch! Thanks for suggestions Frederic From fabiosantosart at gmail.com Thu Jul 11 05:08:04 2013 From: fabiosantosart at gmail.com (=?ISO-8859-1?Q?F=E1bio_Santos?=) Date: Thu, 11 Jul 2013 10:08:04 +0100 Subject: ElementTree: can't figure out a mismached-tag error In-Reply-To: <51DE73E4.6040007@bluewin.ch> References: <51DE73E4.6040007@bluewin.ch> Message-ID: On 11 Jul 2013 10:04, "F.R." wrote: > > Hi all, > > I haven't been able to get up to speed with XML. I do examples from the tutorials and experiment with variations. Time and time again I fail with errors messages I can't make sense of. Here's the latest one. The url is " http://finance.yahoo.com/q?s=XIDEQ&ql=0". Ubuntu 12.04 LTS, Python 2.7.3 (default, Aug 1 2012, 05:16:07) [GCC 4.6.3] > > >>> import xml.etree.ElementTree as ET > >>> tree = ET.parse('q?s=XIDEQ') # output of wget http://finance.yahoo.com/q?s=XIDEQ&ql=0 > Traceback (most recent call last): > File "", line 1, in > tree = ET.parse('q?s=XIDEQ') > File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1183, in parse > tree.parse(source, parser) > File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 656, in parse > parser.feed(data) > File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1643, in feed > self._raiseerror(v) > File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1507, in _raiseerror > raise err > ParseError: mismatched tag: line 9, column 2 > > Below first nine lines. The line numbers and the following space are hand-edited in. Three dots stand for sections cut out to fit long lines. Line 6 is a bunch of "meta" statements, all of which I show on a separate line each in order to preserve the angled brackets. On all lines the angled brackets have been preserved. The mismatched character is the slash of the closing tag . What could be wrong with it? And if it is, what about fault tolerance? > > 1 > 2 > 3 > 4 XIDEQ: Summary for EXIDE TECH NEW- Yahoo! Finance > 5 > > > > > > > > > 8 > 9 > ^ > Mismatch! > > Thanks for suggestions > > Frederic That is not XML. It is HTML. You get a mismatched tag because the tag doesn't need closing in HTML, but in XML every single tag needs closing. Use an HTML parser. I strongly recommend BeautifulSoup but I think etree has an HTML parser too. I am not sure.. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fronagzen at gmail.com Thu Jul 11 05:19:58 2013 From: fronagzen at gmail.com (fronagzen at gmail.com) Date: Thu, 11 Jul 2013 02:19:58 -0700 (PDT) Subject: ElementTree: can't figure out a mismached-tag error In-Reply-To: References: <51DE73E4.6040007@bluewin.ch> Message-ID: <3906a61a-1a52-48b8-b9df-fb30c5498e99@googlegroups.com> Actually, I don't think etree has a HTML parser. And I would counter-recommend lxml if speed is an issue: BeautifulSoup takes a looooong time to parse a large document. On Thursday, July 11, 2013 5:08:04 PM UTC+8, F?bio Santos wrote: > On 11 Jul 2013 10:04, "F.R." wrote: > > > > > > Hi all, > > > > > > I haven't been able to get up to speed with XML. I do examples from the tutorials and experiment with variations. Time and time again I fail with errors messages I can't make sense of. Here's the latest one. The url is "http://finance.yahoo.com/q?s=XIDEQ&ql=0". Ubuntu 12.04 LTS, Python 2.7.3 (default, Aug ?1 2012, 05:16:07) [GCC 4.6.3] > > > > > > > >>> import xml.etree.ElementTree as ET > > > >>> tree = ET.parse('q?s=XIDEQ') ?# output of wget http://finance.yahoo.com/q?s=XIDEQ&ql=0 > > > Traceback (most recent call last): > > > ? File "", line 1, in > > > ? ? tree = ET.parse('q?s=XIDEQ') > > > ? File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1183, in parse > > > ? ? tree.parse(source, parser) > > > ? File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 656, in parse > > > ? ? parser.feed(data) > > > ? File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1643, in feed > > > ? ? self._raiseerror(v) > > > ? File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1507, in _raiseerror > > > ? ? raise err > > > ParseError: mismatched tag: line 9, column 2 > > > > > > Below first nine lines. The line numbers and the following space are hand-edited in. Three dots stand for sections cut out to fit long lines. Line 6 is a bunch of "meta" statements, all of which I show on a separate line each in order to preserve the angled brackets. On all lines the angled brackets have been preserved. The mismatched character is the slash of the closing tag . What could be wrong with it? And if it is, what about fault tolerance? > > > > > > > 1 > > > 2 > > > 3 > > > 4 XIDEQ: Summary for EXIDE TECH NEW- Yahoo! Finance > > > 5 > > > ? > > > ? > > > ? > > > ? > > > ? > > > ? > > > ? > > > ? > > > 8 > > > 9 > > > ? ?^ > > > ? ? Mismatch! > > > > > > Thanks for suggestions > > > > > > Frederic > > That is not XML. It is HTML. You get a mismatched tag because the tag doesn't need closing in HTML, but in XML every single tag needs closing. > > Use an HTML parser. I strongly recommend? BeautifulSoup but I think etree has an HTML parser too. I am not sure.. From fabiosantosart at gmail.com Thu Jul 11 06:27:02 2013 From: fabiosantosart at gmail.com (=?ISO-8859-1?Q?F=E1bio_Santos?=) Date: Thu, 11 Jul 2013 11:27:02 +0100 Subject: ElementTree: can't figure out a mismached-tag error In-Reply-To: <3906a61a-1a52-48b8-b9df-fb30c5498e99@googlegroups.com> References: <51DE73E4.6040007@bluewin.ch> <3906a61a-1a52-48b8-b9df-fb30c5498e99@googlegroups.com> Message-ID: On 11 Jul 2013 10:24, wrote: > > Actually, I don't think etree has a HTML parser. And I would counter-recommend lxml if speed is an issue: BeautifulSoup takes a looooong time to parse a large document. > > On Thursday, July 11, 2013 5:08:04 PM UTC+8, F?bio Santos wrote: > > > > Use an HTML parser. I strongly recommend BeautifulSoup but I think etree has an HTML parser too. I am not sure.. I meant lxml. My apologies. -------------- next part -------------- An HTML attachment was scrubbed... URL: From anthra.norell at bluewin.ch Thu Jul 11 08:25:13 2013 From: anthra.norell at bluewin.ch (F.R.) Date: Thu, 11 Jul 2013 14:25:13 +0200 Subject: ElementTree: can't figure out a mismached-tag error In-Reply-To: <51DE73E4.6040007@bluewin.ch> References: <51DE73E4.6040007@bluewin.ch> Message-ID: <51DEA429.6040606@bluewin.ch> On 07/11/2013 10:59 AM, F.R. wrote: > Hi all, > > I haven't been able to get up to speed with XML. I do examples from > the tutorials and experiment with variations. Time and time again I > fail with errors messages I can't make sense of. Here's the latest > one. The url is "http://finance.yahoo.com/q?s=XIDEQ&ql=0". Ubuntu > 12.04 LTS, Python 2.7.3 (default, Aug 1 2012, 05:16:07) [GCC 4.6.3] > > >>> import xml.etree.ElementTree as ET > >>> tree = ET.parse('q?s=XIDEQ') # output of wget > http://finance.yahoo.com/q?s=XIDEQ&ql=0 > Traceback (most recent call last): > File "", line 1, in > tree = ET.parse('q?s=XIDEQ') > File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1183, in parse > tree.parse(source, parser) > File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 656, in parse > parser.feed(data) > File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1643, in feed > self._raiseerror(v) > File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1507, in > _raiseerror > raise err > ParseError: mismatched tag: line 9, column 2 > > Below first nine lines. The line numbers and the following space are > hand-edited in. Three dots stand for sections cut out to fit long > lines. Line 6 is a bunch of "meta" statements, all of which I show on > a separate line each in order to preserve the angled brackets. On all > lines the angled brackets have been preserved. The mismatched > character is the slash of the closing tag . What could be wrong > with it? And if it is, what about fault tolerance? > > 1 > 2 > 3 > 4 XIDEQ: Summary for EXIDE TECH NEW- Yahoo! Finance > 5 > > > > > > content="http://l.yimg.com/a/p/fi/31/09/00.jpg"> > > href="http://finance.yahoo.com/q?s=XIDEQ"> > 8 > 9 > ^ > Mismatch! > > Thanks for suggestions > > Frederic > Thank you all! I was a little apprehensive it could be a silly mistake. And so it was. I have BeautifulSoup somewhere. Having had no urgent need for it I remember shirking the learning curve. lxml seems to be a package with these components (from help (lxml)): PACKAGE CONTENTS ElementInclude _elementpath builder cssselect doctestcompare etree html (package) isoschematron (package) objectify pyclasslookup sax usedoctest I would start with "from lxml import html" and see what comes out. Break time now. Thanks again! Frederic From fronagzen at gmail.com Thu Jul 11 08:49:56 2013 From: fronagzen at gmail.com (fronagzen at gmail.com) Date: Thu, 11 Jul 2013 05:49:56 -0700 (PDT) Subject: ElementTree: can't figure out a mismached-tag error In-Reply-To: References: <51DE73E4.6040007@bluewin.ch> Message-ID: On Thursday, July 11, 2013 8:25:13 PM UTC+8, F.R. wrote: > On 07/11/2013 10:59 AM, F.R. wrote: > > > Hi all, > > > > > > I haven't been able to get up to speed with XML. I do examples from > > > the tutorials and experiment with variations. Time and time again I > > > fail with errors messages I can't make sense of. Here's the latest > > > one. The url is "http://finance.yahoo.com/q?s=XIDEQ&ql=0". Ubuntu > > > 12.04 LTS, Python 2.7.3 (default, Aug 1 2012, 05:16:07) [GCC 4.6.3] > > > > > > >>> import xml.etree.ElementTree as ET > > > >>> tree = ET.parse('q?s=XIDEQ') # output of wget > > > http://finance.yahoo.com/q?s=XIDEQ&ql=0 > > > Traceback (most recent call last): > > > File "", line 1, in > > > tree = ET.parse('q?s=XIDEQ') > > > File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1183, in parse > > > tree.parse(source, parser) > > > File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 656, in parse > > > parser.feed(data) > > > File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1643, in feed > > > self._raiseerror(v) > > > File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1507, in > > > _raiseerror > > > raise err > > > ParseError: mismatched tag: line 9, column 2 > > > > > > Below first nine lines. The line numbers and the following space are > > > hand-edited in. Three dots stand for sections cut out to fit long > > > lines. Line 6 is a bunch of "meta" statements, all of which I show on > > > a separate line each in order to preserve the angled brackets. On all > > > lines the angled brackets have been preserved. The mismatched > > > character is the slash of the closing tag . What could be wrong > > > with it? And if it is, what about fault tolerance? > > > > > > 1 > > > 2 > > > 3 > > > 4 XIDEQ: Summary for EXIDE TECH NEW- Yahoo! Finance > > > 5 > > > > > > > > > > > > > > > > > > > > content="http://l.yimg.com/a/p/fi/31/09/00.jpg"> > > > > > > > > href="http://finance.yahoo.com/q?s=XIDEQ"> > > > 8 > > > 9 > > > ^ > > > Mismatch! > > > > > > Thanks for suggestions > > > > > > Frederic > > > > > Thank you all! > > > > I was a little apprehensive it could be a silly mistake. And so it was. > > I have BeautifulSoup somewhere. Having had no urgent need for it I > > remember shirking the learning curve. > > > > lxml seems to be a package with these components (from help (lxml)): > > > > PACKAGE CONTENTS > > ElementInclude > > _elementpath > > builder > > cssselect > > doctestcompare > > etree > > html (package) > > isoschematron (package) > > objectify > > pyclasslookup > > sax > > usedoctest > > > > I would start with "from lxml import html" and see what comes out. > > > > Break time now. Thanks again! > > > > Frederic from lxml.html import parse from lxml.etree import ElementTree root = parse(target_url).getroot() This'll get you the root node of the element tree parsed from the URL. The lxml html parser, conveniently enough, can combine in the actual web page access. If you want to control things like socket timeout, though, you'll have to use urllib to request the URL and then feed that to the parser. From fronagzen at gmail.com Thu Jul 11 05:09:23 2013 From: fronagzen at gmail.com (fronagzen at gmail.com) Date: Thu, 11 Jul 2013 02:09:23 -0700 (PDT) Subject: Kivy for Python 3.3 Message-ID: <3814d9c3-c926-4024-81ab-a968c7f86324@googlegroups.com> Hello, first time poster here, and general newbie to Python. I'm looking to write a program in Python, (and have in fact written most of it by now,) and am trying to put together a GUI for it. Kivy looks very nice, particularly with the fact that it's supposed to be compatible with most platforms (including Android, which I would like to be able to use my program on in addition to running it on my desktop) with minimal hassle. However, its current iteration is Python 2.7 only, and I've only learned Python 3.3. I suppose I could learn Python 2.7, but I don't really want to. (Nor rewrite my program.) I could wait for the next release of Kivy, but who knows when that will be? In any case, it's apparently possible to pull and compile the development version of Kivy for 3.3: https://groups.google.com/forum/#!topic/kivy-dev/pRp_02jaJww/discussion, however, I haven't succeeded in getting it to work. I'm on a Windows machine, and after a good deal of wrangling, I'm getting: C:\Users\[SNIP]\Desktop\kivy-py3>python setup.py build_ext --inplace --compiler=mi ngw32 [INFO ] Kivy v1.8.0-dev Windows platform detected, force GLEW usage. running build_ext Build configuration is: * use_mesagl = False * use_x11 = False * use_rpi = False * use_opengl_es2 = True * use_opengl_debug = False * use_sdl = False * use_ios = False * use_glew = True Generate config.h Generate config.pxi cythoning kivy\properties.pyx to kivy\properties.c building 'kivy.properties' extension C:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IC:\Python33\include -IC:\Pytho n33\include -c kivy\properties.c -o build\temp.win32-3.3\Release\kivy\properties .o writing build\temp.win32-3.3\Release\kivy\properties.def C:\MinGW\bin\gcc.exe -mno-cygwin -shared -s build\temp.win32-3.3\Release\kivy\pr operties.o build\temp.win32-3.3\Release\kivy\properties.def -LC:\Python33\libs - LC:\Python33\PCbuild -lm -lpython33 -lmsvcr100 -o C:\Users\Lung\Desktop\kivy-py3 \kivy\properties.pyd C:\MinGW\bin\..\lib\gcc\mingw32\3.4.5\..\..\..\..\mingw32\bin\ld.exe: cannot fin d -lmsvcr100 collect2: ld returned 1 exit status error: command 'gcc' failed with exit status 1 I also tried to compile it with an Ubuntu virtual machine, but I failed hard on that since I'm not a native Linux user. Would anyone be able to help, either by suggesting how I can fix the above error, or, perhaps, by helping me compile Kivy? (I did post this on the Kivy group, but I've not gotten any response there... The group's a bit dead.) From ian at feete.org Thu Jul 11 08:25:11 2013 From: ian at feete.org (Ian Foote) Date: Thu, 11 Jul 2013 13:25:11 +0100 Subject: Kivy for Python 3.3 In-Reply-To: <3814d9c3-c926-4024-81ab-a968c7f86324@googlegroups.com> References: <3814d9c3-c926-4024-81ab-a968c7f86324@googlegroups.com> Message-ID: <51DEA427.6080801@feete.org> On 11/07/13 10:09, fronagzen at gmail.com wrote: > Hello, first time poster here, and general newbie to Python. > > I'm looking to write a program in Python, (and have in fact written most of it by now,) and am trying to put together a GUI for it. Kivy looks very nice, particularly with the fact that it's supposed to be compatible with most platforms (including Android, which I would like to be able to use my program on in addition to running it on my desktop) with minimal hassle. However, its current iteration is Python 2.7 only, and I've only learned Python 3.3. > > I suppose I could learn Python 2.7, but I don't really want to. (Nor rewrite my program.) I could wait for the next release of Kivy, but who knows when that will be? > > In any case, it's apparently possible to pull and compile the development version of Kivy for 3.3: https://groups.google.com/forum/#!topic/kivy-dev/pRp_02jaJww/discussion, however, I haven't succeeded in getting it to work. I'm on a Windows machine, and after a good deal of wrangling, I'm getting: > > C:\Users\[SNIP]\Desktop\kivy-py3>python setup.py build_ext --inplace --compiler=mi > ngw32 > [INFO ] Kivy v1.8.0-dev > Windows platform detected, force GLEW usage. > running build_ext > Build configuration is: > * use_mesagl = False > * use_x11 = False > * use_rpi = False > * use_opengl_es2 = True > * use_opengl_debug = False > * use_sdl = False > * use_ios = False > * use_glew = True > Generate config.h > Generate config.pxi > cythoning kivy\properties.pyx to kivy\properties.c > building 'kivy.properties' extension > C:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IC:\Python33\include -IC:\Pytho > n33\include -c kivy\properties.c -o build\temp.win32-3.3\Release\kivy\properties > .o > writing build\temp.win32-3.3\Release\kivy\properties.def > C:\MinGW\bin\gcc.exe -mno-cygwin -shared -s build\temp.win32-3.3\Release\kivy\pr > operties.o build\temp.win32-3.3\Release\kivy\properties.def -LC:\Python33\libs - > LC:\Python33\PCbuild -lm -lpython33 -lmsvcr100 -o C:\Users\Lung\Desktop\kivy-py3 > \kivy\properties.pyd > C:\MinGW\bin\..\lib\gcc\mingw32\3.4.5\..\..\..\..\mingw32\bin\ld.exe: cannot fin > d -lmsvcr100 > collect2: ld returned 1 exit status > error: command 'gcc' failed with exit status 1 > > I also tried to compile it with an Ubuntu virtual machine, but I failed hard on that since I'm not a native Linux user. Would anyone be able to help, either by suggesting how I can fix the above error, or, perhaps, by helping me compile Kivy? > > (I did post this on the Kivy group, but I've not gotten any response there... The group's a bit dead.) > Hi Have you tried asking on the kivy irc channel? (#kivy on freenode, or http://webchat.freenode.net/?nick=kvuser.&channels=kivy&uio=d4) The channel is fairly active, though you might have to wait a bit for a response. I'm not a Windows user myself, so unfortunately I can't help with the issue directly. Regards, Ian F From paul at subsignal.org Thu Jul 11 09:17:07 2013 From: paul at subsignal.org (=?UTF-8?B?UGF1bCBLw7ZsbGU=?=) Date: Thu, 11 Jul 2013 15:17:07 +0200 Subject: Kivy for Python 3.3 In-Reply-To: <3814d9c3-c926-4024-81ab-a968c7f86324@googlegroups.com> References: <3814d9c3-c926-4024-81ab-a968c7f86324@googlegroups.com> Message-ID: Am 11.07.2013 11:09, schrieb fronagzen at gmail.com: > Hello, first time poster here, and general newbie to Python. > > I'm looking to write a program in Python, (and have in fact written > most of it by now,) and am trying to put together a GUI for it. Kivy > looks very nice, particularly with the fact that it's supposed to be > compatible with most platforms (including Android, which I would like > to be able to use my program on in addition to running it on my > desktop) with minimal hassle. However, its current iteration is > Python 2.7 only, and I've only learned Python 3.3. [ snipp ] > C:\MinGW\bin\..\lib\gcc\mingw32\3.4.5\..\..\..\..\mingw32\bin\ld.exe: > cannot fin d -lmsvcr100 collect2: ld returned 1 exit status error: > command 'gcc' failed with exit status 1 It might help copying msvcr100.dll (from MS VC++ Redistributable Package) to c:\python33\libs (or wherever your python install is). See http://bugs.python.org/issue15315 msg191106 hth Paul From ulrich.eckhardt at dominolaser.com Thu Jul 11 08:49:53 2013 From: ulrich.eckhardt at dominolaser.com (Ulrich Eckhardt) Date: Thu, 11 Jul 2013 14:49:53 +0200 Subject: Kivy for Python 3.3 In-Reply-To: <3814d9c3-c926-4024-81ab-a968c7f86324@googlegroups.com> References: <3814d9c3-c926-4024-81ab-a968c7f86324@googlegroups.com> Message-ID: Welcome to Python! Am 11.07.2013 11:09, schrieb fronagzen at gmail.com: > I'm looking to write a program in Python, (and have in fact written > most of it by now,) and am trying to put together a GUI for it. Kivy > looks very nice, particularly with the fact that it's supposed to be > compatible with most platforms (including Android, which I would like > to be able to use my program on in addition to running it on my > desktop) with minimal hassle. However, its current iteration is > Python 2.7 only, and I've only learned Python 3.3. Last I looked, which was half a year ago, there was some Python 3 porting of Kivy underway, as you found yourself. If I were you, I would get on IRC (I think it was #kivy on irc.freenode.net) and try to contact the people there about the state of the Python 3 port. Just have some patience, there aren't hundreds of people (yet), so getting an answer could take some time. > C:\MinGW\bin\..\lib\gcc\mingw32\3.4.5\..\..\..\..\mingw32\bin\ld.exe: > cannot fin d -lmsvcr100 collect2: ld returned 1 exit status error: > command 'gcc' failed with exit status 1 'msvcr100' is the C runtime of MS Visual C++, I'm not sure if it is required for building Python modules on MS Windows. Just removing it from the commandline (or makefile) should tell you already. Alternatively, ask The Internet(R), http://bugs.python.org/issue15315 could be an answer. ;) Good luck! Uli From fronagzen at gmail.com Thu Jul 11 10:26:19 2013 From: fronagzen at gmail.com (fronagzen at gmail.com) Date: Thu, 11 Jul 2013 07:26:19 -0700 (PDT) Subject: Kivy for Python 3.3 In-Reply-To: References: <3814d9c3-c926-4024-81ab-a968c7f86324@googlegroups.com> Message-ID: On Thursday, July 11, 2013 9:17:07 PM UTC+8, Paul K?lle wrote: > Am 11.07.2013 11:09, schrieb fronagzen at gmail.com: > > > Hello, first time poster here, and general newbie to Python. > > > > > > I'm looking to write a program in Python, (and have in fact written > > > most of it by now,) and am trying to put together a GUI for it. Kivy > > > looks very nice, particularly with the fact that it's supposed to be > > > compatible with most platforms (including Android, which I would like > > > to be able to use my program on in addition to running it on my > > > desktop) with minimal hassle. However, its current iteration is > > > Python 2.7 only, and I've only learned Python 3.3. > > [ snipp ] > > > C:\MinGW\bin\..\lib\gcc\mingw32\3.4.5\..\..\..\..\mingw32\bin\ld.exe: > > > cannot fin d -lmsvcr100 collect2: ld returned 1 exit status error: > > > command 'gcc' failed with exit status 1 > > > > It might help copying msvcr100.dll (from MS VC++ Redistributable > > Package) to c:\python33\libs (or wherever your python install is). See > > http://bugs.python.org/issue15315 msg191106 > > > > hth > > Paul Thanks for the response, and I swear that I did try googling the error, but didn't find anything. Oh well. Anyway, it seems that adding msvcr100.dll to my python3.3\libs has done something. I'm getting a different file not found message now: C:\Users\[SNIP]\Desktop\kivy-py3>python setup.py build_ext --inplace [INFO ] Kivy v1.8.0-dev Windows platform detected, force GLEW usage. running build_ext Build configpython setup.py build_ext --inplace --compiler =mingw32 [INFO ] Kivy v1.8.0-dev Windows platform detected, force GLEW usage. running build_ext Build configuration is: * use_glew = True * use_ios = False * use_opengl_debug = False * use_opengl_es2 = True * use_rpi = False * use_mesagl = False * use_x11 = False * use_sdl = False Generate config.h Generate config.pxi skipping 'kivy\graphics\vertex_instructions.c' Cython extension (up-to-date) building 'kivy.graphics.vertex_instructions' extension C:\MinGW\bin\gcc.exe -mdll -O -Wall -IC:\Python33\include -IC:\Python33\include -c kivy\graphics\vertex_instructions.c -o build\temp.win32-3.3\Release\kivy\grap hics\vertex_instructions.o In file included from kivy\graphics\vertex_instructions.c:314:0: kivy\graphics\/gl_redirect.h:8:22: fatal error: GL/glew.h: No such file or direc tory compilation terminated. error: command 'gcc' failed with exit status 1 Working on the reasoning that sticking the missing file into the python3.3\libs file worked, I tried adding the glew files (glew32.dll, glew32.lib, GL\glew and GL\wglew.h) there, however, it doesn't seem to have made a difference. And this I googled this before asking. Doesn't seem to be much. From paul at subsignal.org Fri Jul 12 06:14:46 2013 From: paul at subsignal.org (=?UTF-8?B?UGF1bCBLw7ZsbGU=?=) Date: Fri, 12 Jul 2013 12:14:46 +0200 Subject: Kivy for Python 3.3 In-Reply-To: References: <3814d9c3-c926-4024-81ab-a968c7f86324@googlegroups.com> Message-ID: Am 11.07.2013 16:26, schrieb fronagzen at gmail.com: [scnipp] > C:\MinGW\bin\gcc.exe -mdll -O -Wall -IC:\Python33\include > -IC:\Python33\include -c kivy\graphics\vertex_instructions.c -o > build\temp.win32-3.3\Release\kivy\grap hics\vertex_instructions.o In > file included from kivy\graphics\vertex_instructions.c:314:0: > kivy\graphics\/gl_redirect.h:8:22: fatal error: GL/glew.h: No such > file or direc tory compilation terminated. error: command 'gcc' > failed with exit status 1 > > Working on the reasoning that sticking the missing file into the > python3.3\libs file worked, I tried adding the glew files > (glew32.dll, glew32.lib, GL\glew and GL\wglew.h) there, however, it > doesn't seem to have made a difference. And this I googled this > before asking. Doesn't seem to be much. Hi, I can feel your pain beeing dragged into compile-unknown-source-on-windows from an innocent looking python script ;) This time it's not the linker complaining but GCC (the compiler) itself not being able to find the header file GL/glew.h. Headers are searched in the so called "include path" which is specified in setup.py. This link http://glew.sourceforge.net/install.html suggests glew.h is part of VisualStudio, and has some other interesting information. hth Paul From fronagzen at gmail.com Fri Jul 12 07:58:05 2013 From: fronagzen at gmail.com (fronagzen at gmail.com) Date: Fri, 12 Jul 2013 04:58:05 -0700 (PDT) Subject: Kivy for Python 3.3 In-Reply-To: References: <3814d9c3-c926-4024-81ab-a968c7f86324@googlegroups.com> Message-ID: <7c1c0fb6-6806-4355-a77a-0f64070314ab@googlegroups.com> On Friday, July 12, 2013 6:14:46 PM UTC+8, Paul K?lle wrote: > Am 11.07.2013 16:26, schrieb fronagzen at gmail.com: > > [scnipp] > > > C:\MinGW\bin\gcc.exe -mdll -O -Wall -IC:\Python33\include > > > -IC:\Python33\include -c kivy\graphics\vertex_instructions.c -o > > > build\temp.win32-3.3\Release\kivy\grap hics\vertex_instructions.o In > > > file included from kivy\graphics\vertex_instructions.c:314:0: > > > kivy\graphics\/gl_redirect.h:8:22: fatal error: GL/glew.h: No such > > > file or direc tory compilation terminated. error: command 'gcc' > > > failed with exit status 1 > > > > > > Working on the reasoning that sticking the missing file into the > > > python3.3\libs file worked, I tried adding the glew files > > > (glew32.dll, glew32.lib, GL\glew and GL\wglew.h) there, however, it > > > doesn't seem to have made a difference. And this I googled this > > > before asking. Doesn't seem to be much. > > > > Hi, > > > > I can feel your pain beeing dragged into > > compile-unknown-source-on-windows from an innocent looking python script > > ;) This time it's not the linker complaining but GCC (the compiler) > > itself not being able to find the header file GL/glew.h. Headers are > > searched in the so called "include path" which is specified in setup.py. > > This link http://glew.sourceforge.net/install.html suggests glew.h is > > part of VisualStudio, and has some other interesting information. > > > > hth > > Paul Thanks for the response. After a bit of tinkering, I've finally managed to compile the blasted thing! I just straight up dumped Glew into the appropriate files for VS10, and used the VS10 compiler instead of mingw32- and it worked! From ulrich.eckhardt at dominolaser.com Thu Jul 11 09:05:59 2013 From: ulrich.eckhardt at dominolaser.com (Ulrich Eckhardt) Date: Thu, 11 Jul 2013 15:05:59 +0200 Subject: Callable or not callable, that is the question! Message-ID: Hello! I just stumbled over a case where Python (2.7 and 3.3 on MS Windows) fail to detect that an object is a function, using the callable() builtin function. Investigating, I found out that the object was indeed not callable, but in a way that was very unexpected to me: class X: @staticmethod def example(): pass test1 = example test2 = [example,] X.example() # OK X.test1() # OK X.test2[0]() # TypeError: 'staticmethod' object is not callable Bug or feature? Thanks! Uli From __peter__ at web.de Thu Jul 11 10:11:47 2013 From: __peter__ at web.de (Peter Otten) Date: Thu, 11 Jul 2013 16:11:47 +0200 Subject: Callable or not callable, that is the question! References: Message-ID: Ulrich Eckhardt wrote: > Hello! > > I just stumbled over a case where Python (2.7 and 3.3 on MS Windows) > fail to detect that an object is a function, using the callable() > builtin function. Investigating, I found out that the object was indeed > not callable, but in a way that was very unexpected to me: > > class X: > @staticmethod > def example(): > pass > test1 = example > test2 = [example,] > > X.example() # OK > X.test1() # OK > X.test2[0]() # TypeError: 'staticmethod' object is not callable Slightly modified example: >>> @staticmethod ... def example(): return 42 ... >>> class X: ... example = example ... >>> X.example() 42 >>> example() Traceback (most recent call last): File "", line 1, in TypeError: 'staticmethod' object is not callable When you access an attribute its __get__() method is implicitly called. This is part of the descriptor protocol: >>> X.example >>> example >>> example.__get__(X) While it would be easy to make staticmethod callable >>> class Staticmethod(staticmethod): ... def __call__(self, *args, **kw): ... return self.__func__(*args, **kw) ... >>> @Staticmethod ... def foo(): return "bar" ... >>> foo() 'bar' >>> X.baz = foo >>> X.baz() 'bar' I see no clear advantage over the current situation. > Bug or feature? No bug. Missing feature if you come up with a convincing use-case. From jason.swails at gmail.com Thu Jul 11 10:28:34 2013 From: jason.swails at gmail.com (Jason Swails) Date: Thu, 11 Jul 2013 10:28:34 -0400 Subject: Callable or not callable, that is the question! In-Reply-To: References: Message-ID: On Thu, Jul 11, 2013 at 9:05 AM, Ulrich Eckhardt < ulrich.eckhardt at dominolaser.com> wrote: > Hello! > > I just stumbled over a case where Python (2.7 and 3.3 on MS Windows) fail > to detect that an object is a function, using the callable() builtin > function. Investigating, I found out that the object was indeed not > callable, but in a way that was very unexpected to me: > > class X: > @staticmethod > def example(): > pass > test1 = example > test2 = [example,] > > X.example() # OK > X.test1() # OK > X.test2[0]() # TypeError: 'staticmethod' object is not callable > Interestingly, you can actually use this approach to 'fake' staticmethod before staticmethod was even introduced. By accessing example from inside the test2 class attribute list, there is no instance bound to that method (even if an instance was used to access it). Using Python 3.3: Python 3.3.2 (default, Jun 3 2013, 08:29:09) [GCC 4.5.4] on linux Type "help", "copyright", "credits" or "license" for more information. >>> class X: ... def example(): pass ... test = example, ... >>> X.test[0]() >>> Using Python 2.0 (pre-staticmethod): Python 2.0.1 (#1, Aug 28 2012, 20:25:41) [GCC 4.5.3] on linux3 Type "copyright", "credits" or "license" for more information. >>> class X: ... def example(): pass ... test = example, ... >>> X.test[0]() >>> staticmethod Traceback (most recent call last): File "", line 1, in ? NameError: There is no variable named 'staticmethod' Once you change test into an instance attribute, you get back to the expected behavior Python 3.3.2 (default, Jun 3 2013, 08:29:09) [GCC 4.5.4] on linux Type "help", "copyright", "credits" or "license" for more information. >>> class X: ... def example(self): pass ... test = example, ... >>> inst = X() >>> inst.example() >>> inst.test[0]() Traceback (most recent call last): File "", line 1, in TypeError: example() missing 1 required positional argument: 'self' >>> inst.test = inst.example, >>> inst.test[0]() >>> All the best, Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve+comp.lang.python at pearwood.info Thu Jul 11 22:12:07 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 12 Jul 2013 02:12:07 GMT Subject: Callable or not callable, that is the question! References: Message-ID: <51df65f7$0$9505$c3e8da3$5496439d@news.astraweb.com> On Thu, 11 Jul 2013 15:05:59 +0200, Ulrich Eckhardt wrote: > Hello! > > I just stumbled over a case where Python (2.7 and 3.3 on MS Windows) > fail to detect that an object is a function, using the callable() > builtin function. Investigating, I found out that the object was indeed > not callable, but in a way that was very unexpected to me: [...] > X.test2[0]() # TypeError: 'staticmethod' object is not callable > > > Bug or feature? In my opinion, a bug. I thought I had actually submitted it to the bug tracker, but apparently I was a shameful slacker and did not. However there was a discussion in this thread: http://mail.python.org/pipermail/python-dev/2011-March/109090.html Here's a simpler demonstration of the issue: assert callable(staticmethod(lambda: None)) -- Steven From ian.g.kelly at gmail.com Fri Jul 12 03:11:02 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Fri, 12 Jul 2013 01:11:02 -0600 Subject: Callable or not callable, that is the question! In-Reply-To: <51df65f7$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <51df65f7$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, Jul 11, 2013 at 8:12 PM, Steven D'Aprano wrote: > On Thu, 11 Jul 2013 15:05:59 +0200, Ulrich Eckhardt wrote: > >> Hello! >> >> I just stumbled over a case where Python (2.7 and 3.3 on MS Windows) >> fail to detect that an object is a function, using the callable() >> builtin function. Investigating, I found out that the object was indeed >> not callable, but in a way that was very unexpected to me: > [...] >> X.test2[0]() # TypeError: 'staticmethod' object is not callable >> >> >> Bug or feature? > > In my opinion, a bug. I thought I had actually submitted it to the bug > tracker, but apparently I was a shameful slacker and did not. However > there was a discussion in this thread: > > http://mail.python.org/pipermail/python-dev/2011-March/109090.html > > > Here's a simpler demonstration of the issue: > > assert callable(staticmethod(lambda: None)) If staticmethod is going to be callable then classmethod should be callable also. From ulrich.eckhardt at dominolaser.com Fri Jul 12 02:41:05 2013 From: ulrich.eckhardt at dominolaser.com (Ulrich Eckhardt) Date: Fri, 12 Jul 2013 08:41:05 +0200 Subject: Callable or not callable, that is the question! In-Reply-To: References: Message-ID: <11l4ba-u4l.ln1@satorlaser.homedns.org> Am 11.07.2013 16:11, schrieb Peter Otten: > Ulrich Eckhardt wrote: >> Bug or feature? > > No bug. Missing feature if you come up with a convincing use-case. class Parser: def _handle_bool(input): # ... pass types = {'bool': _handle_bool, 'boolean': _handle_bool,} def parse(self, line): t,s,v = line.partition(':') handler = types[t] return handler(v) I want a utility function that really behaves just like a function. I'd prefer to nest it inside the class that uses it, to make the code easier to understand. Since I don't want the implicit self-binding either, I would use staticmethod to make this clear, too. Since I can live without any of this, it's not a big issue. What is to me a big issue though is the fact that Python behaves unexpectedly and reading Steven's response and the link there, it seems I'm not alone. Greetings! Uli From duncan.booth at invalid.invalid Fri Jul 12 03:36:30 2013 From: duncan.booth at invalid.invalid (Duncan Booth) Date: 12 Jul 2013 07:36:30 GMT Subject: Callable or not callable, that is the question! References: <11l4ba-u4l.ln1@satorlaser.homedns.org> Message-ID: Ulrich Eckhardt wrote: > Am 11.07.2013 16:11, schrieb Peter Otten: >> Ulrich Eckhardt wrote: >>> Bug or feature? >> >> No bug. Missing feature if you come up with a convincing use-case. > > class Parser: > def _handle_bool(input): > # ... > pass > > types = {'bool': _handle_bool, > 'boolean': _handle_bool,} > > def parse(self, line): > t,s,v = line.partition(':') > handler = types[t] > return handler(v) > > I want a utility function that really behaves just like a function. I'd > prefer to nest it inside the class that uses it, to make the code easier > to understand. Since I don't want the implicit self-binding either, I > would use staticmethod to make this clear, too. But the example you gave works just fine as written! You are only using your utility function as a function so there's no need for it to be a staticmethod or indeed any other sort of method. To be a convincing use-case you would have to show a situation where something had to be both a static method and a utility method rather than just one or the other and also where you couldn't just have both. If you can persuade me that you need _handle_bool as both a static method and a utility function, you probably also need to explain why you can't just use both: class Parser: def _handle_bool(input): ... handle_bool = staticmethod(_handle_bool) -- Duncan Booth http://kupuguy.blogspot.com From steve+comp.lang.python at pearwood.info Fri Jul 12 05:49:14 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 12 Jul 2013 09:49:14 GMT Subject: Callable or not callable, that is the question! References: <11l4ba-u4l.ln1@satorlaser.homedns.org> Message-ID: <51dfd11a$0$9505$c3e8da3$5496439d@news.astraweb.com> On Fri, 12 Jul 2013 07:36:30 +0000, Duncan Booth wrote: > To be a convincing use-case you would have to show a situation where > something had to be both a static method and a utility method rather > than just one or the other and also where you couldn't just have both. I have a class where I have a function that needs to be called both while the class is being constructed, and afterwards: class Example: @staticmethod def do_stuff(arg): ... do_stuff(23) # This doesn't work. Example.do_stuff(42) I have work-arounds: inside the class, I call do_stuff.__func__ instead of do_stuff, but really, that's ugly and distracting and merely a work- around for the fact that staticmethods aren't callable. To make them callable is trivially easy: they just need a __call__ method that calls __func__ for you. > If you can persuade me that you need _handle_bool as both a static > method and a utility function, you probably also need to explain why you > can't just use both: > > class Parser: > def _handle_bool(input): ... > handle_bool = staticmethod(_handle_bool) That's extremely inelegant. Why have two functions for something which is conceptually one? -- Steven From __peter__ at web.de Fri Jul 12 06:36:55 2013 From: __peter__ at web.de (Peter Otten) Date: Fri, 12 Jul 2013 12:36:55 +0200 Subject: Callable or not callable, that is the question! References: <11l4ba-u4l.ln1@satorlaser.homedns.org> <51dfd11a$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: Steven D'Aprano wrote: > On Fri, 12 Jul 2013 07:36:30 +0000, Duncan Booth wrote: > >> To be a convincing use-case you would have to show a situation where >> something had to be both a static method and a utility method rather >> than just one or the other and also where you couldn't just have both. > > I have a class where I have a function that needs to be called both while > the class is being constructed, and afterwards: > > class Example: > @staticmethod > def do_stuff(arg): > ... > > do_stuff(23) # This doesn't work. > > Example.do_stuff(42) That is a bit too abstract for my taste to count as a use-case. Also, as given the example will work in Python 3 when you remove the @staticmethod ;) That said I can see that the error comes as a surprise and I would be fine with callable staticmethod objects. From zughumancapital at yahoo.com.br Thu Jul 11 10:27:43 2013 From: zughumancapital at yahoo.com.br (zughumancapital) Date: Thu, 11 Jul 2013 14:27:43 -0000 Subject: =?iso-8859-1?q?Oportunidade:_Estagi=E1rio_(Wanna-be-developer)_-_RJ?= Message-ID: Arpex Capital seleciona para uma de suas startups: Estagi?rio (Wanna-be-developer) Objetivo geral da Posi??o: Se divertir programando, resolvendo problemas e aprendendo coisas novas. Pr?-requisitos: Conhecimento de Ruby on Rails ou Python ou Node.js ou Angular.js, conhecimento UNIX e vontade de aprender o que n?o souber e for necess?rio. Tamb?m ? importante que tenha github ou alguma forma de conferir projetos antigos j? realizados para que possamos avaliar. Se n?o usar Windows (ou seja, usar Mac ou Linux), ? um mega plus. Ultra plus: usar Vim, Emacs ou SublimeText como editores de texto e/ou estar disposto a mudar para um melhor. Deveres: Ser obcecado em entender e resolver problemas: procuramos algu?m que durma mal se tiver um problema a ser resolvido, e que acorde pensando nele. Ter MUITA vontade de aprender e crescer, assim como gostar de novas tecnologias. N?O procuramos algu?m que esteja preso a tecnologias espec?ficas e/ou pouco aberto a mudan?as. N?o existe "bala de prata" no nosso mundo. Forma??o: N?o importa. Se souber programar e for autodidata, t? valendo. Principais Compet?ncias a serem observadas: Capacidade de aprender r?pido. Buscamos os melhores desenvolvedores do Brasil. N?o importa a linguagem. Queremos os nerds, n?o estamos preocupados e nem nos interessamos naqueles que querem parecer que s?o os melhores, queremos os que s?o os melhores de fato. Amamos os que amam programar e que programam desde a adolesc?ncia ou inf?ncia. Estamos aqui para ter o melhor time do Brasil! Bolsa Aux?lio + Vale Refei??o + Vale Transporte Local de Trabalho: Centro/RJ Os interessados dever?o enviar o CV para kgarcia at arpexcapital.com.br, mencionando no assunto Estagi?rio (Wanna-be-developer). From russ.pobox at gmail.com Thu Jul 11 10:52:14 2013 From: russ.pobox at gmail.com (Russel Walker) Date: Thu, 11 Jul 2013 07:52:14 -0700 (PDT) Subject: xslice idea | a generator slice Message-ID: <4a81b6a6-023e-4d47-9bd0-bbc0516caf6b@googlegroups.com> Just some dribble, nothing major. I like using slices but I also noticed that a slice expression returns a new sequence. I sometimes find myself using them in a for loop like this: seq = range(10) for even in seq[::2]: print even (That's just for an example) But wouldn't it be a bit of a waste if the slice expression returns a whole new list just when all you want to do in this case is iterate over it once? I suppose you could get around that will a little more code like: seq = range(10) for x in xrange(0, len(seq), 2): print seq[x] But it would be nice there was a way to iterate over a 'slice' of sorts, that would act as a generator. So I tried my handle at a little class called xslice (the x from xrange to indicate its a generator (although I'm not sure if that's technically correct)). With it, instead of the above example you can do: seq = range(10) for even in xslice(seq, 0, len(seq), 2): print even It would be more or less like using xrange. I know the syntax is not that much of an improvement over just using exrange and retrieving items by index, but it's the concept I'm looking at. Do you think this or something like be a builtin in the future? From russ.pobox at gmail.com Thu Jul 11 10:54:35 2013 From: russ.pobox at gmail.com (Russel Walker) Date: Thu, 11 Jul 2013 07:54:35 -0700 (PDT) Subject: xslice idea | a generator slice In-Reply-To: <4a81b6a6-023e-4d47-9bd0-bbc0516caf6b@googlegroups.com> References: <4a81b6a6-023e-4d47-9bd0-bbc0516caf6b@googlegroups.com> Message-ID: ...oh and here is the class I made for it. class xslice(object): ''' xslice(seq, start, stop, step) -> generator slice ''' def __init__(self, seq, *stop): if len(stop) > 3: raise TypeError("xslice takes at most 4 arguments") elif len(stop) < 0: raise TypeError("xslice requires atleast 2 arguments") else: start, stop, step = (((0,) + stop[:2])[-2:] + # start, stop (stop[2:] + (1,))[:1]) # step stop = min(stop, len(seq)) self._ind = iter(xrange(start, stop, step)) self._seq = seq def __iter__(self): return self def next(self): return self._seq[self._ind.next()] Although now that I think about it, it probably should've just been a simple generator function. From oscar.j.benjamin at gmail.com Thu Jul 11 11:14:46 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 11 Jul 2013 16:14:46 +0100 Subject: xslice idea | a generator slice In-Reply-To: References: <4a81b6a6-023e-4d47-9bd0-bbc0516caf6b@googlegroups.com> Message-ID: On 11 July 2013 15:54, Russel Walker wrote: > ...oh and here is the class I made for it. > > class xslice(object): > ''' > xslice(seq, start, stop, step) -> generator slice > ''' > > def __init__(self, seq, *stop): Wouldn't it be better if it has the same signature(s) as itertools.islice? > if len(stop) > 3: > raise TypeError("xslice takes at most 4 arguments") > elif len(stop) < 0: How would len(stop) be negative? > raise TypeError("xslice requires atleast 2 arguments") > else: > start, stop, step = (((0,) + stop[:2])[-2:] + # start, stop > (stop[2:] + (1,))[:1]) # step > stop = min(stop, len(seq)) > self._ind = iter(xrange(start, stop, step)) > self._seq = seq > > def __iter__(self): > return self > > def next(self): > return self._seq[self._ind.next()] > > > > Although now that I think about it, it probably should've just been a simple generator function. Or you can use itertools.imap: def xslice(sequence, start_or_stop, *args): indices = xrange(*slice(start_or_stop, *args).indices(len(sequence))) return imap(sequence.__getitem__, indices) Oscar From russ.pobox at gmail.com Thu Jul 11 12:21:16 2013 From: russ.pobox at gmail.com (Russel Walker) Date: Thu, 11 Jul 2013 09:21:16 -0700 (PDT) Subject: xslice idea | a generator slice In-Reply-To: References: <4a81b6a6-023e-4d47-9bd0-bbc0516caf6b@googlegroups.com> Message-ID: <56736add-f372-4245-9f50-076b3d44bb00@googlegroups.com> > > def __init__(self, seq, *stop): > > > > Wouldn't it be better if it has the same signature(s) as itertools.islice? That's actually what I was going for, except I was modeling it after range, but that was the only way I knew to implement it. > > if len(stop) > 3: > > > raise TypeError("xslice takes at most 4 arguments") > > > elif len(stop) < 0: > > > > How would len(stop) be negative? Yes, that should've been len(stop) < 1. > Or you can use itertools.imap: > > > > def xslice(sequence, start_or_stop, *args): > > indices = xrange(*slice(start_or_stop, *args).indices(len(sequence))) > > return imap(sequence.__getitem__, indices) > > > > > > Oscar I like the way you did that with imap, although I still like my parameter implementation. To confess, this is the second time I've made the mistake of trying to implement generator like functionality of a builtin when there already is on in itertools. Need to start studying that module abit more I think. I'm looking at the docs now and I see there are actually a couple of isomethings(). From oscar.j.benjamin at gmail.com Thu Jul 11 12:34:33 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 11 Jul 2013 17:34:33 +0100 Subject: xslice idea | a generator slice In-Reply-To: <56736add-f372-4245-9f50-076b3d44bb00@googlegroups.com> References: <4a81b6a6-023e-4d47-9bd0-bbc0516caf6b@googlegroups.com> <56736add-f372-4245-9f50-076b3d44bb00@googlegroups.com> Message-ID: On 11 July 2013 17:21, Russel Walker wrote: > To confess, this is the second time I've made the mistake of trying to implement generator like functionality of a builtin when there already is on in itertools. Need to start studying that module abit more I think. I'm looking at the docs now and I see there are actually a couple of isomethings(). Your xslice (or mine) would still be better than islice when the step size is large; islice has to iterate over all the skipped elements which could be wasteful if the input is indexable. Also islice doesn't support negative values for start, stop or step which xslice does. From ian.g.kelly at gmail.com Thu Jul 11 14:00:17 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Thu, 11 Jul 2013 12:00:17 -0600 Subject: xslice idea | a generator slice In-Reply-To: References: <4a81b6a6-023e-4d47-9bd0-bbc0516caf6b@googlegroups.com> <56736add-f372-4245-9f50-076b3d44bb00@googlegroups.com> Message-ID: On Thu, Jul 11, 2013 at 10:34 AM, Oscar Benjamin wrote: > On 11 July 2013 17:21, Russel Walker wrote: >> To confess, this is the second time I've made the mistake of trying to implement generator like functionality of a builtin when there already is on in itertools. Need to start studying that module abit more I think. I'm looking at the docs now and I see there are actually a couple of isomethings(). > > Your xslice (or mine) would still be better than islice when the step > size is large; islice has to iterate over all the skipped elements > which could be wasteful if the input is indexable. Also islice doesn't > support negative values for start, stop or step which xslice does. Ah, that's an interesting point. Of course the corollary to that is that xslice requires a sequence, not just an iterable. From fabiosantosart at gmail.com Thu Jul 11 15:58:49 2013 From: fabiosantosart at gmail.com (=?ISO-8859-1?Q?F=E1bio_Santos?=) Date: Thu, 11 Jul 2013 20:58:49 +0100 Subject: xslice idea | a generator slice In-Reply-To: References: <4a81b6a6-023e-4d47-9bd0-bbc0516caf6b@googlegroups.com> <56736add-f372-4245-9f50-076b3d44bb00@googlegroups.com> Message-ID: On 11 Jul 2013 17:38, "Oscar Benjamin" wrote: > > On 11 July 2013 17:21, Russel Walker wrote: > > To confess, this is the second time I've made the mistake of trying to implement generator like functionality of a builtin when there already is on in itertools. Need to start studying that module abit more I think. I'm looking at the docs now and I see there are actually a couple of isomethings(). > > Your xslice (or mine) would still be better than islice when the step > size is large; islice has to iterate over all the skipped elements > which could be wasteful if the input is indexable. Also islice doesn't > support negative values for start, stop or step which xslice does. Isn't all of itertools implemented in C? If we are not using a python-level and not-so-fast __getitem__ I would wager the C version is a lot faster. And if the input is indexable could I assume that it is not too large to have around in memory and thus any speed increase in looping over it would be tiny? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ian.g.kelly at gmail.com Thu Jul 11 17:02:39 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Thu, 11 Jul 2013 15:02:39 -0600 Subject: xslice idea | a generator slice In-Reply-To: References: <4a81b6a6-023e-4d47-9bd0-bbc0516caf6b@googlegroups.com> <56736add-f372-4245-9f50-076b3d44bb00@googlegroups.com> Message-ID: On Thu, Jul 11, 2013 at 1:58 PM, F?bio Santos wrote: > Isn't all of itertools implemented in C? If we are not using a python-level > and not-so-fast __getitem__ I would wager the C version is a lot faster. > > And if the input is indexable could I assume that it is not too large to > have around in memory and thus any speed increase in looping over it would > be tiny? Oscar's version which is an imap of an xrange is also effectively implemented in C. Only the setup is in Python. From ian.g.kelly at gmail.com Thu Jul 11 12:16:38 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Thu, 11 Jul 2013 10:16:38 -0600 Subject: xslice idea | a generator slice In-Reply-To: <4a81b6a6-023e-4d47-9bd0-bbc0516caf6b@googlegroups.com> References: <4a81b6a6-023e-4d47-9bd0-bbc0516caf6b@googlegroups.com> Message-ID: On Thu, Jul 11, 2013 at 8:52 AM, Russel Walker wrote: > Just some dribble, nothing major. > > I like using slices but I also noticed that a slice expression returns a new sequence. > > I sometimes find myself using them in a for loop like this: > > > seq = range(10) > for even in seq[::2]: > print even > > > (That's just for an example) But wouldn't it be a bit of a waste if the slice expression returns a whole new list just when all you want to do in this case is iterate over it once? > > I suppose you could get around that will a little more code like: > > > seq = range(10) > for x in xrange(0, len(seq), 2): > print seq[x] > > > But it would be nice there was a way to iterate over a 'slice' of sorts, that would act as a generator. That would be the itertools.islice function. From metaliobovinus at gmail.com Thu Jul 11 11:32:34 2013 From: metaliobovinus at gmail.com (Metallicow) Date: Thu, 11 Jul 2013 08:32:34 -0700 (PDT) Subject: How do I get the OS System Font Directory(Cross-Platform) in python? Message-ID: <3d97de86-f690-40ad-aba8-972120af7ff3@googlegroups.com> How do I get the OS System Font Directory(Cross-Platform) in python? Need a simple script for Windows, Linux, Mac, etc.. Or using wxPython. I can't seem to find anything that works, and I don't want to hard-code paths. From kwpolska at gmail.com Thu Jul 11 11:48:21 2013 From: kwpolska at gmail.com (=?UTF-8?B?Q2hyaXMg4oCcS3dwb2xza2HigJ0gV2Fycmljaw==?=) Date: Thu, 11 Jul 2013 17:48:21 +0200 Subject: How do I get the OS System Font Directory(Cross-Platform) in python? In-Reply-To: <3d97de86-f690-40ad-aba8-972120af7ff3@googlegroups.com> References: <3d97de86-f690-40ad-aba8-972120af7ff3@googlegroups.com> Message-ID: On Thu, Jul 11, 2013 at 5:32 PM, Metallicow wrote: > How do I get the OS System Font Directory(Cross-Platform) in python? > > Need a simple script for > Windows, Linux, Mac, etc.. > > Or using wxPython. > > I can't seem to find anything that works, and I don't want to hard-code paths. > -- > http://mail.python.org/mailman/listinfo/python-list Why do you want the paths to the fonts? You do not need them 99.9% of the time. So, what *exactly* are you trying to accomplish? -- Kwpolska | GPG KEY: 5EAAEA16 stop html mail | always bottom-post http://asciiribbon.org | http://caliburn.nl/topposting.html From metaliobovinus at gmail.com Thu Jul 11 11:54:17 2013 From: metaliobovinus at gmail.com (Metallicow) Date: Thu, 11 Jul 2013 08:54:17 -0700 (PDT) Subject: How do I get the OS System Font Directory(Cross-Platform) in python? In-Reply-To: <3d97de86-f690-40ad-aba8-972120af7ff3@googlegroups.com> References: <3d97de86-f690-40ad-aba8-972120af7ff3@googlegroups.com> Message-ID: For a portable font install tool. Finding if a particular font exists, useful when testing apps in virtual environent, rendering text from a font, Font file manipulations, etc.. From kwpolska at gmail.com Thu Jul 11 12:12:43 2013 From: kwpolska at gmail.com (Chris =?utf-8?B?4oCcS3dwb2xza2HigJ0=?= Warrick) Date: Thu, 11 Jul 2013 18:12:43 +0200 Subject: How do I get the OS System Font Directory(Cross-Platform) in python? In-Reply-To: References: <3d97de86-f690-40ad-aba8-972120af7ff3@googlegroups.com> Message-ID: <20130711161240.GA5092@kwpolska-lin> On Thu, Jul 11, 2013 at 08:54:17AM -0700, Metallicow wrote: > For a portable font install tool. > > Finding if a particular font exists, > useful when testing apps in virtual environent, > rendering text from a font, > Font file manipulations, > etc.. > -- > http://mail.python.org/mailman/listinfo/python-list Then do a nice little `if` checking the user?s OS. http://docs.python.org/2/library/sys.html#sys.platform hints what to use, and you need to do: if sys.platform.startswith('win') or sys.platform.startswith('cygwin'): FONTDIRS = [os.path.join(os.environ['WINDIR'], 'Fonts') elif sys.platform.startswith('darwin'): # [see below and devise it yourself] else: # linux, *bsd and everything else # [see below, commit suicide and develop this bit] Windows: The easiest of those three, all the fonts are in %WINDIR%/Fonts. In Python: os.path.join(os.environ['WINDIR'], 'Fonts') Mac OS X: http://support.apple.com/kb/ht2435 (and not all of them are guaranteed to exist). `os.expanduser()` may be useful for that first one. Linux, and everything else running that fancy open stuff (eg. *BSD): Hardcore. You just need to parse a nice tiny 155-line XML file. It?s /etc/fonts/fonts.conf, and it has some tags. Some of them may have a `prefix="xdg"` attribute which makes you prepend the value with $XDG_DATA_HOME (~/.local/share if that is empty). You also need `os.expanduser()`. Oh: and you need to go recursively through them, as subdirectories are also checked for fonts (and it probably goes further) -- Kwpolska | GPG KEY: 5EAAEA16 stop html mail | always bottom-post http://asciiribbon.org | http://caliburn.nl/topposting.html -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 490 bytes Desc: not available URL: From metaliobovinus at gmail.com Thu Jul 11 13:19:50 2013 From: metaliobovinus at gmail.com (Metallicow) Date: Thu, 11 Jul 2013 10:19:50 -0700 (PDT) Subject: How do I get the OS System Font Directory(Cross-Platform) in python? In-Reply-To: <3d97de86-f690-40ad-aba8-972120af7ff3@googlegroups.com> References: <3d97de86-f690-40ad-aba8-972120af7ff3@googlegroups.com> Message-ID: @ Chris ?Kwpolska? Warrick Thanks, that is a start anyway. a Pure-Python way was what I was wanting, not win32api stuff. "C:\Windows\Fonts" The windows path proves valid. Works on XP, Vista, 7. Not sure about win8...? Don't have a mac handy, but the link should be enough to help write some code before testing. I don't own a mac, but IIRC macs come with python installed, so a trip to the apple shop might be in order. As far as linux goes, basically Ubuntu/Kubuntu(dirivs), Mandriva/Mageia, Mint, and some of the more popular distros is all I am really looking for. Checking for these is VirtualBox shouldn't be too hard with your instructions and platform.linux_distribution() Thanks again. Here is the 'Open Font License 1.1 (OFL 1.1)' Fonts I will be using. http://sourceforge.net/projects/sourcecodepro.adobe/ http://sourceforge.net/projects/sourcesans.adobe/ Also found these links that might help with pathing. http://www.fontation.com/feature/ http://www.yearbooks.biz/?event=FAQ.Detail&faq=3 From christian at python.org Thu Jul 11 21:27:04 2013 From: christian at python.org (Christian Heimes) Date: Fri, 12 Jul 2013 03:27:04 +0200 Subject: How do I get the OS System Font Directory(Cross-Platform) in python? In-Reply-To: References: <3d97de86-f690-40ad-aba8-972120af7ff3@googlegroups.com> Message-ID: Am 11.07.2013 19:19, schrieb Metallicow: > @ Chris ?Kwpolska? Warrick > Thanks, that is a start anyway. > a Pure-Python way was what I was wanting, not win32api stuff. > > "C:\Windows\Fonts" > The windows path proves valid. Works on XP, Vista, 7. Not sure about win8....? That's the wrong way to do it. You have to use the proper Windows API to get to the font directory. It's SHGetKnownFolderPath() with FOLDERID_Font or SHGetFolderPath() with CSIDL_FONTS. See http://bugs.python.org/issue1763 Christian From metaliobovinus at gmail.com Fri Jul 12 00:24:00 2013 From: metaliobovinus at gmail.com (Metallicow) Date: Thu, 11 Jul 2013 21:24:00 -0700 (PDT) Subject: How do I get the OS System Font Directory(Cross-Platform) in python? In-Reply-To: References: <3d97de86-f690-40ad-aba8-972120af7ff3@googlegroups.com> Message-ID: <9218c1f3-f107-4942-90d6-26c017f1e4e7@googlegroups.com> On Thursday, July 11, 2013 8:27:04 PM UTC-5, Christian Heimes wrote: > Am 11.07.2013 19:19, schrieb Metallicow: > > > @ Chris ???Kwpolska??? Warrick > > > Thanks, that is a start anyway. > > > a Pure-Python way was what I was wanting, not win32api stuff. > > > > > > "C:\Windows\Fonts" > > > The windows path proves valid. Works on XP, Vista, 7. Not sure about win8....? > > > > That's the wrong way to do it. You have to use the proper Windows API to > > get to the font directory. It's SHGetKnownFolderPath() with > > FOLDERID_Font or SHGetFolderPath() with CSIDL_FONTS. > > > > See http://bugs.python.org/issue1763 > > > > Christian typo'd instead of copy/pasted. What I meant was... Chris's code... >>> FONTDIRS = os.path.join(os.environ['WINDIR'], 'Fonts') 'C:\\WINDOWS\\Fonts' returned valid as returned string. Forgot to add >>> part. Is there any way to edit posts? Don't see an edit linky anywhere. Is there something wrong with using os.environ...? win32api stuff is another dependency. Is it really needed? From rosuav at gmail.com Fri Jul 12 00:58:49 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 12 Jul 2013 14:58:49 +1000 Subject: How do I get the OS System Font Directory(Cross-Platform) in python? In-Reply-To: <9218c1f3-f107-4942-90d6-26c017f1e4e7@googlegroups.com> References: <3d97de86-f690-40ad-aba8-972120af7ff3@googlegroups.com> <9218c1f3-f107-4942-90d6-26c017f1e4e7@googlegroups.com> Message-ID: On Fri, Jul 12, 2013 at 2:24 PM, Metallicow wrote: > On Thursday, July 11, 2013 8:27:04 PM UTC-5, Christian Heimes wrote: >> Am 11.07.2013 19:19, schrieb Metallicow: >> >> > @ Chris ???Kwpolska??? Warrick >> >> > Thanks, that is a start anyway. >> >> > a Pure-Python way was what I was wanting, not win32api stuff. >> >> > >> >> > "C:\Windows\Fonts" >> >> > The windows path proves valid. Works on XP, Vista, 7. Not sure about win8....? >> >> >> >> That's the wrong way to do it. You have to use the proper Windows API to >> >> get to the font directory. It's SHGetKnownFolderPath() with >> >> FOLDERID_Font or SHGetFolderPath() with CSIDL_FONTS. >> >> >> >> See http://bugs.python.org/issue1763 >> >> >> >> Christian > > typo'd instead of copy/pasted. What I meant was... Chris's code... > >>>> FONTDIRS = os.path.join(os.environ['WINDIR'], 'Fonts') > 'C:\\WINDOWS\\Fonts' > > returned valid as returned string. Forgot to add >>> part. > Is there any way to edit posts? Don't see an edit linky anywhere. > Is there something wrong with using os.environ...? > win32api stuff is another dependency. Is it really needed? No, you fundamentally cannot edit posts. This is a Usenet newsgroup and a mailing list; once your post is sent, it gets carried by lots of different servers separately. Just send a followup, same as you did there. Since you seem to be using Google Groups, please read this to learn how to avoid antagonizing a significant proportion of the list's best responders: http://wiki.python.org/moin/GoogleGroupsPython ChrisA From steve+comp.lang.python at pearwood.info Fri Jul 12 00:55:35 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 12 Jul 2013 04:55:35 GMT Subject: How do I get the OS System Font Directory(Cross-Platform) in python? References: <3d97de86-f690-40ad-aba8-972120af7ff3@googlegroups.com> <9218c1f3-f107-4942-90d6-26c017f1e4e7@googlegroups.com> Message-ID: <51df8c46$0$9505$c3e8da3$5496439d@news.astraweb.com> On Thu, 11 Jul 2013 21:24:00 -0700, Metallicow wrote: > Forgot to add >>> part. Is there any way to edit posts? Not unless thousands of people give you access to their computer so you can edit the emails in their inboxes. When you send a post to a public mailing list, its out their on fifty thousand computers and a dozen public archives. No, you can't change your mind and edit what you've already said. > Don't see an edit linky anywhere. Is there something > wrong with using os.environ...? win32api stuff is another dependency. Is > it really needed? If you want to do it the right way, yes. In principle, any Windows machine could set the font directory to any valid directory. The only way to be sure you have got the right one is to ask Windows for the font directory, and that requires win32api. If you intend for this to be usable everywhere, my approach would be this: 1) You need a separate function for each platform you intend to support. At the very least, you should support Windows (2000, XP, Vista, 7 and 8), OS-X (Mac), Linux. You should also strongly consider supporting Android, FreeBSD, OpenBSD, and Solaris. If you intend running under older versions of Python, you should also consider supporting Windows CE. You might even like to support iOS. 2) Some of these (Windows, probably Mac and iOS) will require a call to the OS to find the current directory. That means a win32api or equivalent call. 3) If win32api is not available, you might like to fall back on a heuristic to guess the right directory. But remember that is only a little bit better than hard-coding a directory. 4) For Linux and Free- and OpenBSD, there is no standard font directory. You may be able to make a call to the GUI toolkit to ask what it thinks the font directory (or directories!) is, but there are many different toolkits, and they can all be active at the same time. E.g. I might run a Gnome app and a KDE app both under XFCE, plus OpenOffice, and all three might disagree as to what fonts I have available. The standard way to locate fonts in Linux is to look at two files, /etc/fonts/fonts.conf and /etc/fonts/local.conf, which list the locations you should check. Either, or both, files may be missing. Standard locations include: /usr/share/fonts /usr/local/share/fonts /home//.fonts /usr/X11R6/lib/X11/fonts /usr/share/X11/fonts/Type1 /usr/share/X11/fonts/OTF And of course, applications like OpenOffice might keep their own, private, font directory, just because. 5) Now that you have a function for each OS you support, you can create a master function that detects the OS and calls the appropriate one. Now at last you have a simple function get_font_directory() that works everywhere. Now that you see how complex it is to write this "simple" function, are you surprised that one doesn't yet exist? :-) -- Steven From nobody at nowhere.com Thu Jul 11 13:47:01 2013 From: nobody at nowhere.com (Nobody) Date: Thu, 11 Jul 2013 18:47:01 +0100 Subject: How do I get the OS System Font Directory(Cross-Platform) in python? References: <3d97de86-f690-40ad-aba8-972120af7ff3@googlegroups.com> Message-ID: On Thu, 11 Jul 2013 08:32:34 -0700, Metallicow wrote: > How do I get the OS System Font Directory(Cross-Platform) in python? What makes you think the system *has* a system font directory? In the traditional X11 model, the only program which needs fonts is the X server, and that can be configured to get its fonts from a font server rather than from a local directory. Even if it doesn't use a font server, the font directory will be on the system running the X server, not the one running the client. From metaliobovinus at gmail.com Thu Jul 11 14:25:55 2013 From: metaliobovinus at gmail.com (Metallicow) Date: Thu, 11 Jul 2013 11:25:55 -0700 (PDT) Subject: How do I get the OS System Font Directory(Cross-Platform) in python? In-Reply-To: References: <3d97de86-f690-40ad-aba8-972120af7ff3@googlegroups.com> Message-ID: <5768a87e-1544-45d8-bd4b-bd8e78c711ae@googlegroups.com> On Thursday, July 11, 2013 12:47:01 PM UTC-5, Nobody wrote: > > What makes you think the system *has* a system font directory? Way back when I was kid, I remember a computer that had two colors and 1 built-in font and no mouse. Heck the keyboard was even attached in front a tube screen box. Wow... those were the days. And to think I hated that thing... If the OS doesn't *have* a dedicated system fonts dir that is accessable by the user, then I am not that much interested in dealing with it. Rendering a font from a other location will eventually be worked in, but that isn't the problem at hand. From timr at probo.com Sat Jul 13 01:36:45 2013 From: timr at probo.com (Tim Roberts) Date: Fri, 12 Jul 2013 22:36:45 -0700 Subject: How do I get the OS System Font Directory(Cross-Platform) in python? References: <3d97de86-f690-40ad-aba8-972120af7ff3@googlegroups.com> <5768a87e-1544-45d8-bd4b-bd8e78c711ae@googlegroups.com> Message-ID: Metallicow wrote: > >If the OS doesn't *have* a dedicated system fonts dir that is accessable >by the user, then I am not that much interested in dealing with it. Really? Because Windows is the ONLY one of the major operating systems that actually has a dedicated system fonts directory. Linux doesn't even have a dedicated windowing system. -- Tim Roberts, timr at probo.com Providenza & Boekelheide, Inc. From metaliobovinus at gmail.com Sat Jul 13 02:38:18 2013 From: metaliobovinus at gmail.com (Metallicow) Date: Fri, 12 Jul 2013 23:38:18 -0700 (PDT) Subject: How do I get the OS System Font Directory(Cross-Platform) in python? In-Reply-To: References: <3d97de86-f690-40ad-aba8-972120af7ff3@googlegroups.com> <5768a87e-1544-45d8-bd4b-bd8e78c711ae@googlegroups.com> Message-ID: On Saturday, July 13, 2013 12:36:45 AM UTC-5, Tim Roberts wrote: > Really? Because Windows is the ONLY one of the major operating systems > > that actually has a dedicated system fonts directory. Linux doesn't even > > have a dedicated windowing system. So... Is it expected to install duplicates to multiple locations with Mac and Linux...? From steve+comp.lang.python at pearwood.info Sat Jul 13 03:20:29 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 13 Jul 2013 07:20:29 GMT Subject: How do I get the OS System Font Directory(Cross-Platform) in python? References: <3d97de86-f690-40ad-aba8-972120af7ff3@googlegroups.com> <5768a87e-1544-45d8-bd4b-bd8e78c711ae@googlegroups.com> Message-ID: <51e0ffbd$0$9505$c3e8da3$5496439d@news.astraweb.com> On Fri, 12 Jul 2013 23:38:18 -0700, Metallicow wrote: > On Saturday, July 13, 2013 12:36:45 AM UTC-5, Tim Roberts wrote: >> Really? Because Windows is the ONLY one of the major operating systems >> >> that actually has a dedicated system fonts directory. Linux doesn't >> even >> >> have a dedicated windowing system. > > So... Is it expected to install duplicates to multiple locations with > Mac and Linux...? No. Mac does have dedicated font locations. There are five official locations, with distinct meanings: http://support.apple.com/kb/ht2435 (I do not know if OS-X ever localises directory names according to the user's language.) There is no need to install fonts in multiple locations. Linux does not expect to have fonts installed *at all*. A graphical interface is entirely optional for Linux, so Linux may have no fonts. There are two standard font systems available on Linux. The 15+ year old X Font Server ("xfs") technology is still in use, but many Linux distros are moving towards the newer Fontconfig system. Fontconfig allows you to define as many font directories as you want, although there are a few de- facto standard locations. For more information, see for example: http://www.centos.org/docs/5/html/Deployment_Guide-en-US/s1-x-fonts.html -- Steven From devyncjohnson at gmail.com Thu Jul 11 19:44:05 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Thu, 11 Jul 2013 19:44:05 -0400 Subject: RE Module Performance Message-ID: <51DF4345.5020606@Gmail.com> I recently saw an email in this mailing list about the RE module being made slower. I no long have that email. However, I have viewed the source for the RE module, but I did not see any code that would slow down the script for no valid reason. Can anyone explain what that user meant or if I missed that part of the module? Can the RE module be optimized in any way or at least the "re.sub" portion? Mahalo, Devyn Collier Johnson DevynCJohnson at Gmail.com From rosuav at gmail.com Fri Jul 12 03:37:04 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 12 Jul 2013 17:37:04 +1000 Subject: RE Module Performance In-Reply-To: <51DF4345.5020606@Gmail.com> References: <51DF4345.5020606@Gmail.com> Message-ID: On Fri, Jul 12, 2013 at 9:44 AM, Devyn Collier Johnson wrote: > I recently saw an email in this mailing list about the RE module being made > slower. I no long have that email. However, I have viewed the source for the > RE module, but I did not see any code that would slow down the script for no > valid reason. Can anyone explain what that user meant or if I missed that > part of the module? > > Can the RE module be optimized in any way or at least the "re.sub" portion? There was a post by Steven D'Aprano [1] in which he referred to it busy-waiting just to make regular expressions slower than the alternatives, but his tongue was firmly in his cheek at the time. As to real performance questions, there have been a variety of alternatives proposed, including I think the regex module [2] which is supposed to outperform 're' by quite a margin, but since I tend towards other solutions, I can't quote personal results or hard figures. If re.sub can be optimized and you can see a way to do so, post a patch to the bug tracker; if it improves performance and doesn't have any ridiculous costs to it, it'll probably be accepted. [1] http://mail.python.org/pipermail/python-list/2013-July/651818.html [2] https://pypi.python.org/pypi/regex ChrisA From wxjmfauth at gmail.com Fri Jul 12 05:23:19 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Fri, 12 Jul 2013 02:23:19 -0700 (PDT) Subject: RE Module Performance In-Reply-To: References: Message-ID: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> Le vendredi 12 juillet 2013 01:44:05 UTC+2, Devyn Collier Johnson a ?crit?: > I recently saw an email in this mailing list about the RE module being > > made slower. I no long have that email. However, I have viewed the > > source for the RE module, but I did not see any code that would slow > > down the script for no valid reason. Can anyone explain what that user > > meant or if I missed that part of the module? > > > > Can the RE module be optimized in any way or at least the "re.sub" portion? > > > > Mahalo, > > > > Devyn Collier Johnson > > DevynCJohnson at Gmail.com ---------- I would not care too much about the performance of re. With the new Flexible String Representation, you can use a logarithmic scale to compare re results. To be honest, there is improvment if you are an ascii user. Am I the only one who tested this? Probably. jmf From rosuav at gmail.com Fri Jul 12 05:27:37 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 12 Jul 2013 19:27:37 +1000 Subject: RE Module Performance In-Reply-To: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> Message-ID: On Fri, Jul 12, 2013 at 7:23 PM, wrote: > > I would not care too much about the performance > of re. > > With the new Flexible String Representation, you > can use a logarithmic scale to compare re results. > To be honest, there is improvment if you are an > ascii user. > > Am I the only one who tested this? Probably. Am I the only one who thinks that Dihedral posted under jmf's name? ChrisA From joshua at landau.ws Fri Jul 12 05:39:01 2013 From: joshua at landau.ws (Joshua Landau) Date: Fri, 12 Jul 2013 10:39:01 +0100 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> Message-ID: On 12 July 2013 10:27, Chris Angelico wrote: > On Fri, Jul 12, 2013 at 7:23 PM, wrote: >> >> I would not care too much about the performance >> of re. >> >> With the new Flexible String Representation, you >> can use a logarithmic scale to compare re results. >> To be honest, there is improvment if you are an >> ascii user. >> >> Am I the only one who tested this? Probably. > > Am I the only one who thinks that Dihedral posted under jmf's name? A bot posting as a troll to troll a different troll? Meta. From rosuav at gmail.com Fri Jul 12 05:40:52 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 12 Jul 2013 19:40:52 +1000 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> Message-ID: On Fri, Jul 12, 2013 at 7:39 PM, Joshua Landau wrote: > On 12 July 2013 10:27, Chris Angelico wrote: >> On Fri, Jul 12, 2013 at 7:23 PM, wrote: >>> >>> I would not care too much about the performance >>> of re. >>> >>> With the new Flexible String Representation, you >>> can use a logarithmic scale to compare re results. >>> To be honest, there is improvment if you are an >>> ascii user. >>> >>> Am I the only one who tested this? Probably. >> >> Am I the only one who thinks that Dihedral posted under jmf's name? > > A bot posting as a troll to troll a different troll? Meta. Yeah, it is. But the only other explanation is that jmf has become rather more incomprehensible than usual. Normally I can understand what he's complaining enough to refute it, but here I feel like I'm responding to Dihedral. ChrisA From devyncjohnson at gmail.com Fri Jul 12 06:45:57 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Fri, 12 Jul 2013 06:45:57 -0400 Subject: RE Module Performance In-Reply-To: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> Message-ID: <51DFDE65.5040001@Gmail.com> Could you explain what you mean? What and where is the new Flexible String Representation? Devyn Collier Johnson On 07/12/2013 05:23 AM, wxjmfauth at gmail.com wrote: > Le vendredi 12 juillet 2013 01:44:05 UTC+2, Devyn Collier Johnson a ?crit : >> I recently saw an email in this mailing list about the RE module being >> >> made slower. I no long have that email. However, I have viewed the >> >> source for the RE module, but I did not see any code that would slow >> >> down the script for no valid reason. Can anyone explain what that user >> >> meant or if I missed that part of the module? >> >> >> >> Can the RE module be optimized in any way or at least the "re.sub" portion? >> >> >> >> Mahalo, >> >> >> >> Devyn Collier Johnson >> >> DevynCJohnson at Gmail.com > ---------- > > I would not care too much about the performance > of re. > > With the new Flexible String Representation, you > can use a logarithmic scale to compare re results. > To be honest, there is improvment if you are an > ascii user. > > Am I the only one who tested this? Probably. > > jmf > > From joshua at landau.ws Fri Jul 12 11:59:21 2013 From: joshua at landau.ws (Joshua Landau) Date: Fri, 12 Jul 2013 16:59:21 +0100 Subject: RE Module Performance In-Reply-To: <51DFDE65.5040001@Gmail.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> Message-ID: On 12 July 2013 11:45, Devyn Collier Johnson wrote: > Could you explain what you mean? What and where is the new Flexible String > Representation? Do not worry. jmf is on about his old rant comparing broken previous versions of Python to newer ones which in some microbenchmarks are slower. I don't really get why he spends his time on it. If you're interested, the basic of it is that strings now use a variable number of bytes to encode their values depending on whether values outside of the ASCII range and some other range are used, as an optimisation. From __peter__ at web.de Fri Jul 12 12:15:11 2013 From: __peter__ at web.de (Peter Otten) Date: Fri, 12 Jul 2013 18:15:11 +0200 Subject: RE Module Performance References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> Message-ID: Joshua Landau wrote: > On 12 July 2013 11:45, Devyn Collier Johnson > wrote: >> Could you explain what you mean? What and where is the new Flexible >> String Representation? > > Do not worry. jmf is on about his old rant comparing broken previous > versions of Python to newer ones which in some microbenchmarks are > slower. I don't really get why he spends his time on it. > > If you're interested, the basic of it is that strings now use a > variable number of bytes to encode their values depending on whether > values outside of the ASCII range and some other range are used, as an > optimisation. See also From torriem at gmail.com Fri Jul 12 19:13:47 2013 From: torriem at gmail.com (Michael Torrie) Date: Fri, 12 Jul 2013 17:13:47 -0600 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> Message-ID: <51E08DAB.7030702@gmail.com> On 07/12/2013 09:59 AM, Joshua Landau wrote: > If you're interested, the basic of it is that strings now use a > variable number of bytes to encode their values depending on whether > values outside of the ASCII range and some other range are used, as an > optimisation. Variable number of bytes is a problematic way to saying it. UTF-8 is a variable-number-of-bytes encoding scheme where each character can be 1, 2, 4, or more bytes, depending on the unicode character. As you can imagine this sort of encoding scheme would be very slow to do slicing with (looking up a character at a certain position). Python uses fixed-width encoding schemes, so they preserve the O(n) lookup speeds, but python will use 1, 2, or 4 bytes per every character in the string, depending on what is needed. Just in case the OP might have misunderstood what you are saying. jmf sees the case where a string is promoted from one width to another, and thinks that the brief slowdown in string operations to accomplish this is a problem. In reality I have never seen anyone use the types of string operations his pseudo benchmarks use, and in general Python 3's string behavior is pretty fast. And apparently much more correct than if jmf's ideas of unicode were implemented. From wxjmfauth at gmail.com Wed Jul 24 09:40:55 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Wed, 24 Jul 2013 06:40:55 -0700 (PDT) Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> Message-ID: <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> Le samedi 13 juillet 2013 01:13:47 UTC+2, Michael Torrie a ?crit?: > On 07/12/2013 09:59 AM, Joshua Landau wrote: > > > If you're interested, the basic of it is that strings now use a > > > variable number of bytes to encode their values depending on whether > > > values outside of the ASCII range and some other range are used, as an > > > optimisation. > > > > Variable number of bytes is a problematic way to saying it. UTF-8 is a > > variable-number-of-bytes encoding scheme where each character can be 1, > > 2, 4, or more bytes, depending on the unicode character. As you can > > imagine this sort of encoding scheme would be very slow to do slicing > > with (looking up a character at a certain position). Python uses > > fixed-width encoding schemes, so they preserve the O(n) lookup speeds, > > but python will use 1, 2, or 4 bytes per every character in the string, > > depending on what is needed. Just in case the OP might have > > misunderstood what you are saying. > > > > jmf sees the case where a string is promoted from one width to another, > > and thinks that the brief slowdown in string operations to accomplish > > this is a problem. In reality I have never seen anyone use the types of > > string operations his pseudo benchmarks use, and in general Python 3's > > string behavior is pretty fast. And apparently much more correct than > > if jmf's ideas of unicode were implemented. ------ Sorry, you are not understanding Unicode. What is a Unicode Transformation Format (UTF), what is the goal of a UTF and why it is important for an implementation to work with a UTF. Short example. Writing an editor with something like the FSR is simply impossible (properly). jmf From rosuav at gmail.com Wed Jul 24 09:48:48 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 24 Jul 2013 23:48:48 +1000 Subject: RE Module Performance In-Reply-To: <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> Message-ID: On Wed, Jul 24, 2013 at 11:40 PM, wrote: > Short example. Writing an editor with something like the > FSR is simply impossible (properly). jmf, have you ever written an editor with *any* string representation? Are you speaking from any level of experience at all? ChrisA From dwightdhutto at gmail.com Wed Jul 24 10:17:02 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Wed, 24 Jul 2013 10:17:02 -0400 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> Message-ID: I've screwed up plenty of times in python, but can write code like a pro when I'm feeling better(on SSI and medicaid). An editor can be built simply, but it's preference that makes the difference. Some might have used tkinter, gtk. wxpython or other methods for the task. I think the main issue in responding is your library preference, or widget set preference. These can make you right with some in your response, or wrong with others that have a preferable gui library that coincides with one's personal cognitive structure that makes t On Wed, Jul 24, 2013 at 9:48 AM, Chris Angelico wrote: > On Wed, Jul 24, 2013 at 11:40 PM, wrote: > > Short example. Writing an editor with something like the > > FSR is simply impossible (properly). > > jmf, have you ever written an editor with *any* string representation? > Are you speaking from any level of experience at all? > > ChrisA > -- > http://mail.python.org/mailman/listinfo/python-list > -- Best Regards, David Hutto *CEO:* *http://www.hitwebdevelopment.com* -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwightdhutto at gmail.com Wed Jul 24 10:19:06 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Wed, 24 Jul 2013 10:19:06 -0400 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> Message-ID: I've screwed up plenty of times in python, but can write code like a pro when I'm feeling better(on SSI and medicaid). An editor can be built simply, but it's preference that makes the difference. Some might have used tkinter, gtk. wxpython or other methods for the task. I think the main issue in responding is your library preference, or widget set preference. These can make you right with some in your response, or wrong with others that have a preferable gui library that coincides with one's personal cognitive structure that makes it more usable in relation to how you learned a preferable gui kit. On Wed, Jul 24, 2013 at 10:17 AM, David Hutto wrote: > I've screwed up plenty of times in python, but can write code like a pro > when I'm feeling better(on SSI and medicaid). An editor can be built > simply, but it's preference that makes the difference. Some might have used > tkinter, gtk. wxpython or other methods for the task. > > I think the main issue in responding is your library preference, or widget > set preference. These can make you right with some in your response, or > wrong with others that have a preferable gui library that coincides with > one's personal cognitive structure that makes t > > > > > On Wed, Jul 24, 2013 at 9:48 AM, Chris Angelico wrote: > >> On Wed, Jul 24, 2013 at 11:40 PM, wrote: >> > Short example. Writing an editor with something like the >> > FSR is simply impossible (properly). >> >> jmf, have you ever written an editor with *any* string representation? >> Are you speaking from any level of experience at all? >> >> ChrisA >> -- >> http://mail.python.org/mailman/listinfo/python-list >> > > > > -- > Best Regards, > David Hutto > *CEO:* *http://www.hitwebdevelopment.com* > -- Best Regards, David Hutto *CEO:* *http://www.hitwebdevelopment.com* -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Jul 24 10:34:24 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 00:34:24 +1000 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> Message-ID: On Thu, Jul 25, 2013 at 12:17 AM, David Hutto wrote: > I've screwed up plenty of times in python, but can write code like a pro > when I'm feeling better(on SSI and medicaid). An editor can be built simply, > but it's preference that makes the difference. Some might have used tkinter, > gtk. wxpython or other methods for the task. > > I think the main issue in responding is your library preference, or widget > set preference. These can make you right with some in your response, or > wrong with others that have a preferable gui library that coincides with > one's personal cognitive structure that makes t jmf's point is more about writing the editor widget (Scintilla, as opposed to SciTE), which most people will never bother to do. I've written several text editors, always by embedding someone else's widget, and therefore not concerning myself with its internal string representation. Frankly, Python's strings are a *terrible* internal representation for an editor widget - not because of PEP 393, but simply because they are immutable, and every keypress would result in a rebuilding of the string. On the flip side, I could quite plausibly imagine using a list of strings; whenever text gets inserted, the string gets split at that point, and a new string created for the insert (which also means that an Undo operation simply removes one entire string). In this usage, the FSR is beneficial, as it's possible to have different strings at different widths. But mainly, I'm just wondering how many people here have any basis from which to argue the point he's trying to make. I doubt most of us have (a) implemented an editor widget, or (b) tested multiple different internal representations to learn the true pros and cons of each. And even if any of us had, that still wouldn't have any bearing on PEP 393, which is about applications, not editor widgets. As stated above, Python strings before AND after PEP 393 are poor choices for an editor, ergo arguing from that standpoint is pretty useless. Not that that bothers jmf... ChrisA From torriem at gmail.com Wed Jul 24 11:00:39 2013 From: torriem at gmail.com (Michael Torrie) Date: Wed, 24 Jul 2013 09:00:39 -0600 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> Message-ID: <51EFEC17.90303@gmail.com> On 07/24/2013 08:34 AM, Chris Angelico wrote: > Frankly, Python's strings are a *terrible* internal representation > for an editor widget - not because of PEP 393, but simply because > they are immutable, and every keypress would result in a rebuilding > of the string. On the flip side, I could quite plausibly imagine > using a list of strings; whenever text gets inserted, the string gets > split at that point, and a new string created for the insert (which > also means that an Undo operation simply removes one entire string). > In this usage, the FSR is beneficial, as it's possible to have > different strings at different widths. Very good point. Seems like this is exactly what is tripping up jmf in general. His pseudo benchmarks are bogus for this exact reason. No one uses python strings in this fashion. Editors certainly would not. But then again his argument in the past does not mention editors. But it makes me wonder if jmf is using python strings appropriately, or even realizes they are immutable. > But mainly, I'm just wondering how many people here have any basis > from which to argue the point he's trying to make. I doubt most of > us have (a) implemented an editor widget, or (b) tested multiple > different internal representations to learn the true pros and cons > of each. Maybe, but simply thinking logically, FSR and UCS-4 are equivalent in pros and cons, and the cons of using UCS-2 (the old narrow builds) are well known. UCS-2 simply cannot represent all of unicode correctly. This is in the PEP of course. His most recent argument that Python should use UTF as a representation is very strange to be honest. The cons of UTF are apparent and widely known. The main con is that UTF strings are O(n) for indexing a position within the string. From tjreedy at udel.edu Wed Jul 24 13:52:39 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 24 Jul 2013 13:52:39 -0400 Subject: RE Module Performance In-Reply-To: <51EFEC17.90303@gmail.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51EFEC17.90303@gmail.com> Message-ID: On 7/24/2013 11:00 AM, Michael Torrie wrote: > On 07/24/2013 08:34 AM, Chris Angelico wrote: >> Frankly, Python's strings are a *terrible* internal representation >> for an editor widget - not because of PEP 393, but simply because >> they are immutable, and every keypress would result in a rebuilding >> of the string. On the flip side, I could quite plausibly imagine >> using a list of strings; I used exactly this, a list of strings, for a Python-coded text-only mock editor to replace the tk Text widget in idle tests. It works fine for the purpose. For small test texts, the inefficiency of immutable strings is not relevant. Tk apparently uses a C-coded btree rather than a Python list. All details are hidden, unless one finds and reads the source ;-), but but it uses C arrays rather than Python strings. >> In this usage, the FSR is beneficial, as it's possible to have >> different strings at different widths. For my purpose, the mock Text works the same in 2.7 and 3.3+. > Maybe, but simply thinking logically, FSR and UCS-4 are equivalent in > pros and cons, They both have the pro that indexing is direct *and correct*. The cons are different. > and the cons of using UCS-2 (the old narrow builds) are > well known. UCS-2 simply cannot represent all of unicode correctly. Python's narrow builds, at least for several releases, were in between USC-2 and UTF-16 in that they used surrogates to represent all unicodes but did not correct indexing for the presence of astral chars. This is a nuisance for those who do use astral chars, such as emotes and CJK name chars, on an everyday basis. -- Terry Jan Reedy From rosuav at gmail.com Wed Jul 24 14:15:42 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 04:15:42 +1000 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51EFEC17.90303@gmail.com> Message-ID: On Thu, Jul 25, 2013 at 3:52 AM, Terry Reedy wrote: > On 7/24/2013 11:00 AM, Michael Torrie wrote: >> >> On 07/24/2013 08:34 AM, Chris Angelico wrote: >>> >>> Frankly, Python's strings are a *terrible* internal representation >>> for an editor widget - not because of PEP 393, but simply because >>> they are immutable, and every keypress would result in a rebuilding >>> of the string. On the flip side, I could quite plausibly imagine >>> using a list of strings; > > > I used exactly this, a list of strings, for a Python-coded text-only mock > editor to replace the tk Text widget in idle tests. It works fine for the > purpose. For small test texts, the inefficiency of immutable strings is not > relevant. > > Tk apparently uses a C-coded btree rather than a Python list. All details > are hidden, unless one finds and reads the source ;-), but but it uses C > arrays rather than Python strings. > > >>> In this usage, the FSR is beneficial, as it's possible to have >>> different strings at different widths. > > > For my purpose, the mock Text works the same in 2.7 and 3.3+. Thanks for that report! And yes, it's going to behave exactly the same way, because its underlying structure is an ordered list of ordered lists of Unicode codepoints, ergo 3.3/PEP 393 is merely a question of performance. But if you put your code onto a narrow build, you'll have issues as seen below. >> Maybe, but simply thinking logically, FSR and UCS-4 are equivalent in >> pros and cons, > > They both have the pro that indexing is direct *and correct*. The cons are > different. They're close enough, though. It's simply a performance tradeoff - use the memory all the time, or take a bit of overhead to give yourself the option of using less memory. The difference is negligible compared to... >> and the cons of using UCS-2 (the old narrow builds) are >> well known. UCS-2 simply cannot represent all of unicode correctly. > > Python's narrow builds, at least for several releases, were in between USC-2 > and UTF-16 in that they used surrogates to represent all unicodes but did > not correct indexing for the presence of astral chars. This is a nuisance > for those who do use astral chars, such as emotes and CJK name chars, on an > everyday basis. ... this. If nobody had ever thought of doing a multi-format string representation, I could well imagine the Python core devs debating whether the cost of UTF-32 strings is worth the correctness and consistency improvements... and most likely concluding that narrow builds get abolished. And if any other language (eg ECMAScript) decides to move from UTF-16 to UTF-32, I would wholeheartedly support the move, even if it broke code to do so. To my mind, exposing UTF-16 surrogates to the application is a bug to be fixed, not a feature to be maintained. But since we can get the best of both worlds with only a small amount of overhead, I really don't see why anyone should be objecting. ChrisA From tjreedy at udel.edu Wed Jul 24 18:09:09 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 24 Jul 2013 18:09:09 -0400 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51EFEC17.90303@gmail.com> Message-ID: On 7/24/2013 2:15 PM, Chris Angelico wrote: > On Thu, Jul 25, 2013 at 3:52 AM, Terry Reedy wrote: >> For my purpose, the mock Text works the same in 2.7 and 3.3+. > > Thanks for that report! And yes, it's going to behave exactly the same > way, because its underlying structure is an ordered list of ordered > lists of Unicode codepoints, ergo 3.3/PEP 393 is merely a question of > performance. But if you put your code onto a narrow build, you'll have > issues as seen below. I carefully said 'For my purpose', which is to replace the tk Text widget. Up to 8.5, Tk's text is something like Python's narrow-build unicode. If put astral chars into the toy editor, then yes, it would not work on narrow builds, but would on 3.3+. ... > If nobody had ever thought of doing a multi-format string > representation, I could well imagine the Python core devs debating > whether the cost of UTF-32 strings is worth the correctness and > consistency improvements... and most likely concluding that narrow > builds get abolished. And if any other language (eg ECMAScript) > decides to move from UTF-16 to UTF-32, I would wholeheartedly support > the move, even if it broke code to do so. Making a UTF-16 implementation correct requires converting abstract 'character' array indexes to concrete double byte array indexes. The simple O(n) method of scanning the string from the beginning for each index operation is too slow. When PEP393 was being discussed, I devised a much faster way to do the conversion. The key idea is to add an auxiliary array of the abstract indexes of the astral chars in the abstract array. This is easily created when the string is created and can be done afterward with one linear scan (which is how I experimented with Python code). The length of that array is the number of surrogate pairs in the concrete 16-bit codepoint array. Subtracting that number from the length of the concrete array gives the length of the abstract array. Given a target index of a character in the abstract array, use the auxiliary array to determine k, the number of astral characters that precede the target character. That can be done with either a O(k) linear scan or O(log k) binary search. Add 2 * k to the abstract index to get the corresponding index in the concrete array. When slicing a string with i0 and i1, slice the auxiliary array with k0 and k1 and adjusting the contained indexes downward to get the corresponding auxiliary array. > To my mind, exposing UTF-16 surrogates to the application is a bug > to be fixed, not a feature to be maintained. It is definitely not a feature, but a proper UTF-16 implementation would not expose them except to codecs, just as with the PEP 393 implementation. (In both cases, I am excluding the sys size function as 'exposing to the application'.) > But since we can get the best of both worlds with only > a small amount of overhead, I really don't see why anyone should be > objecting. I presume you are referring to the PEP 393 1-2-4 byte implementation. Given how well it has been optimized, I think it was the right choice for Python. But a language that now uses USC2 or defective UTF-16 on all platforms might find the auxiliary array an easier fix. -- Terry Jan Reedy From rosuav at gmail.com Wed Jul 24 18:19:21 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 08:19:21 +1000 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51EFEC17.90303@gmail.com> Message-ID: On Thu, Jul 25, 2013 at 8:09 AM, Terry Reedy wrote: > On 7/24/2013 2:15 PM, Chris Angelico wrote: >> To my mind, exposing UTF-16 surrogates to the application is a bug >> to be fixed, not a feature to be maintained. > > It is definitely not a feature, but a proper UTF-16 implementation would not > expose them except to codecs, just as with the PEP 393 implementation. (In > both cases, I am excluding the sys size function as 'exposing to the > application'.) > >> But since we can get the best of both worlds with only >> a small amount of overhead, I really don't see why anyone should be >> objecting. > > I presume you are referring to the PEP 393 1-2-4 byte implementation. Given > how well it has been optimized, I think it was the right choice for Python. > But a language that now uses USC2 or defective UTF-16 on all platforms might > find the auxiliary array an easier fix. > I'm referring here to objections like jmf's, and also to threads like this: http://mozilla.6506.n7.nabble.com/Flexible-String-Representation-full-Unicode-for-ES6-td267585.html According to the ECMAScript people, UTF-16 and exposing surrogates to the application is a critical feature to be maintained. I disagree. But it's not my language, so I'm stuck with it. (I ended up writing a little wrapper function in C that detects unpaired surrogates, but that still doesn't deal with the possibility that character indexing can create a new character that was never there to start with.) ChrisA From torriem at gmail.com Wed Jul 24 18:59:15 2013 From: torriem at gmail.com (Michael Torrie) Date: Wed, 24 Jul 2013 16:59:15 -0600 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51EFEC17.90303@gmail.com> Message-ID: <51F05C43.7000504@gmail.com> On 07/24/2013 04:19 PM, Chris Angelico wrote: > I'm referring here to objections like jmf's, and also to threads like this: > > http://mozilla.6506.n7.nabble.com/Flexible-String-Representation-full-Unicode-for-ES6-td267585.html > > According to the ECMAScript people, UTF-16 and exposing surrogates to > the application is a critical feature to be maintained. I disagree. > But it's not my language, so I'm stuck with it. (I ended up writing a > little wrapper function in C that detects unpaired surrogates, but > that still doesn't deal with the possibility that character indexing > can create a new character that was never there to start with.) This is starting to drift off topic here now, but after reading your comments on that post, and others objections, I don't fully understand why making strings simply "unicode" in javascript breaks compatibility with older scripts. What operations are performed on strings that making unicode an abstract type would break? Is it just in the input and output of text that must be decoded and encode? Why should a script care about the internal representation of unicode strings? Is it because the incorrect behavior of UTF-16 and the exposed surrogates (and subsequent incorrect indexing) are actually depended on by some scripts? From rosuav at gmail.com Wed Jul 24 19:24:35 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 09:24:35 +1000 Subject: RE Module Performance In-Reply-To: <51F05C43.7000504@gmail.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51EFEC17.90303@gmail.com> <51F05C43.7000504@gmail.com> Message-ID: On Thu, Jul 25, 2013 at 8:59 AM, Michael Torrie wrote: > I don't fully understand > why making strings simply "unicode" in javascript breaks compatibility > with older scripts. What operations are performed on strings that > making unicode an abstract type would break? Imagine this in JavaScript and Python (apart from the fact that JS doesn't do backslash escapes past 0x10000): a = "asdf\U00012345qwer"; b = a[[..10]; What will this do? It depends on whether UTF-16 is used or not. ChrisA From storchaka at gmail.com Thu Jul 25 01:49:22 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 25 Jul 2013 08:49:22 +0300 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51EFEC17.90303@gmail.com> Message-ID: 24.07.13 21:15, Chris Angelico ???????(??): > To my mind, exposing UTF-16 > surrogates to the application is a bug to be fixed, not a feature to > be maintained. Python 3 uses code points from U+DC80 to U+DCFF (which are in surrogates area) to represent undecodable bytes with surrogateescape error handler. From rosuav at gmail.com Thu Jul 25 01:58:34 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 15:58:34 +1000 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51EFEC17.90303@gmail.com> Message-ID: On Thu, Jul 25, 2013 at 3:49 PM, Serhiy Storchaka wrote: > 24.07.13 21:15, Chris Angelico ???????(??): > >> To my mind, exposing UTF-16 >> surrogates to the application is a bug to be fixed, not a feature to >> be maintained. > > > Python 3 uses code points from U+DC80 to U+DCFF (which are in surrogates > area) to represent undecodable bytes with surrogateescape error handler. That's a deliberate and conscious use of the codepoints; that's not what I'm talking about here. Suppose you read a UTF-8 stream of bytes from a file, and decode them into your language's standard string type. At this point, you should be working with a string of Unicode codepoints: "\22\341\210\264\360\222\215\205" --> "\x12\u1234\U00012345" The incoming byte stream has a length of 8, the resulting character stream has a length of 3. Now, if the language wants to use UTF-16 internally, it's free to do so: 0012 1234 d808 df45 When I referred to exposing surrogates to the application, this is what I'm talking about. If decoding the above byte stream results in a length 4 string where the last two are \xd808 and \xdf45, then it's exposing them. If it's a length 3 string where the last is \U00012345, then it's hiding them. To be honest, I don't imagine I'll ever see a language that stores strings in UTF-16 and then exposes them to the application as UTF-32; there's very little point. But such *is* possible, and if it's working closely with libraries that demand UTF-16, it might well make sense to do things that way. ChrisA From steve+comp.lang.python at pearwood.info Thu Jul 25 03:15:44 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 25 Jul 2013 07:15:44 GMT Subject: RE Module Performance References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51EFEC17.90303@gmail.com> Message-ID: <51f0d0a0$0$29971$c3e8da3$5496439d@news.astraweb.com> On Thu, 25 Jul 2013 04:15:42 +1000, Chris Angelico wrote: > If nobody had ever thought of doing a multi-format string > representation, I could well imagine the Python core devs debating > whether the cost of UTF-32 strings is worth the correctness and > consistency improvements... and most likely concluding that narrow > builds get abolished. And if any other language (eg ECMAScript) decides > to move from UTF-16 to UTF-32, I would wholeheartedly support the move, > even if it broke code to do so. Unfortunately, so long as most language designers are European-centric, there is going to be a lot of push-back against any attempt to fix (say) Javascript, or Java just for the sake of "a bunch of dead languages" in the SMPs. Thank goodness for emoji. Wait til the young kids start complaining that their emoticons and emoji are broken in Javascript, and eventually it will get fixed. It may take a decade, for the young kids to grow up and take over Javascript from the old-codgers, but it will happen. > To my mind, exposing UTF-16 surrogates > to the application is a bug to be fixed, not a feature to be maintained. This, times a thousand. It is *possible* to have non-buggy string routines using UTF-16, but the implementation is a lot more complex than most language developers can be bothered with. I'm not aware of any language that uses UTF-16 internally that doesn't give wrong results for surrogate pairs. -- Steven From rosuav at gmail.com Thu Jul 25 03:58:10 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 17:58:10 +1000 Subject: RE Module Performance In-Reply-To: <51f0d0a0$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51EFEC17.90303@gmail.com> <51f0d0a0$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, Jul 25, 2013 at 5:15 PM, Steven D'Aprano wrote: > On Thu, 25 Jul 2013 04:15:42 +1000, Chris Angelico wrote: > >> If nobody had ever thought of doing a multi-format string >> representation, I could well imagine the Python core devs debating >> whether the cost of UTF-32 strings is worth the correctness and >> consistency improvements... and most likely concluding that narrow >> builds get abolished. And if any other language (eg ECMAScript) decides >> to move from UTF-16 to UTF-32, I would wholeheartedly support the move, >> even if it broke code to do so. > > Unfortunately, so long as most language designers are European-centric, > there is going to be a lot of push-back against any attempt to fix (say) > Javascript, or Java just for the sake of "a bunch of dead languages" in > the SMPs. Thank goodness for emoji. Wait til the young kids start > complaining that their emoticons and emoji are broken in Javascript, and > eventually it will get fixed. It may take a decade, for the young kids to > grow up and take over Javascript from the old-codgers, but it will happen. I don't know that that'll happen like that. Emoticons aren't broken in Javascript - you can use them just fine. You only start seeing problems when you index into that string. People will start to wonder why, for instance, a "500 character maximum" field deducts two from the limit when an emoticon goes in. Example: Type here:

You have 500 characters left (self.value.length).
You have 500 characters left (self.textLength). I've included an attribute documented here[1] as the "codepoint length of the control's value", but in Chrome on Windows, it still counts UTF-16 code units. However, I very much doubt that this will result in language changes. People will just live with it. Chinese and Japanese users will complain, perhaps, and the developers will write it off as whinging, and just say "That's what the internet does". Maybe, if you're really lucky, they'll acknowledge that "that's what JavaScript does", but even then I doubt it'd result in language changes. >> To my mind, exposing UTF-16 surrogates >> to the application is a bug to be fixed, not a feature to be maintained. > > This, times a thousand. > > It is *possible* to have non-buggy string routines using UTF-16, but the > implementation is a lot more complex than most language developers can be > bothered with. I'm not aware of any language that uses UTF-16 internally > that doesn't give wrong results for surrogate pairs. The problem isn't the underlying representation, the problem is what gets exposed to the application. Once you've decided to expose codepoints to the app (abstracting over your UTF-16 underlying representation), the change to using UTF-32, or mimicking PEP 393, or some other structure, is purely internal and an optimization. So I doubt any language will use UTF-16 internally and UTF-32 to the app. It'd be needlessly complex. ChrisA [1] https://developer.mozilla.org/en-US/docs/Web/API/HTMLTextAreaElement From steve+comp.lang.python at pearwood.info Thu Jul 25 05:22:17 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 25 Jul 2013 09:22:17 GMT Subject: RE Module Performance References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51EFEC17.90303@gmail.com> <51f0d0a0$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51f0ee48$0$29971$c3e8da3$5496439d@news.astraweb.com> On Thu, 25 Jul 2013 17:58:10 +1000, Chris Angelico wrote: > On Thu, Jul 25, 2013 at 5:15 PM, Steven D'Aprano > wrote: >> On Thu, 25 Jul 2013 04:15:42 +1000, Chris Angelico wrote: >> >>> If nobody had ever thought of doing a multi-format string >>> representation, I could well imagine the Python core devs debating >>> whether the cost of UTF-32 strings is worth the correctness and >>> consistency improvements... and most likely concluding that narrow >>> builds get abolished. And if any other language (eg ECMAScript) >>> decides to move from UTF-16 to UTF-32, I would wholeheartedly support >>> the move, even if it broke code to do so. >> >> Unfortunately, so long as most language designers are European-centric, >> there is going to be a lot of push-back against any attempt to fix >> (say) Javascript, or Java just for the sake of "a bunch of dead >> languages" in the SMPs. Thank goodness for emoji. Wait til the young >> kids start complaining that their emoticons and emoji are broken in >> Javascript, and eventually it will get fixed. It may take a decade, for >> the young kids to grow up and take over Javascript from the >> old-codgers, but it will happen. > > I don't know that that'll happen like that. Emoticons aren't broken in > Javascript - you can use them just fine. You only start seeing problems > when you index into that string. People will start to wonder why, for > instance, a "500 character maximum" field deducts two from the limit > when an emoticon goes in. I get that. I meant *Javascript developers*, not end-users. The young kids today who become Javascript developers tomorrow will grow up in a world where they expect to be able to write band names like "???????" (yes, really, I didn't make that one up) and have it just work. Okay, all those characters are in the BMP, but emoji aren't, and I guarantee that even as we speak some new hipster band is trying to decide whether to name themselves "Smiling ?" or "Crying ?". :-) >> It is *possible* to have non-buggy string routines using UTF-16, but >> the implementation is a lot more complex than most language developers >> can be bothered with. I'm not aware of any language that uses UTF-16 >> internally that doesn't give wrong results for surrogate pairs. > > The problem isn't the underlying representation, the problem is what > gets exposed to the application. Once you've decided to expose > codepoints to the app (abstracting over your UTF-16 underlying > representation), the change to using UTF-32, or mimicking PEP 393, or > some other structure, is purely internal and an optimization. So I doubt > any language will use UTF-16 internally and UTF-32 to the app. It'd be > needlessly complex. To be honest, I don't understand what you are trying to say. What I'm trying to say is that it is possible to use UTF-16 internally, but *not* assume that every code point (character) is represented by a single 2-byte unit. For example, the len() of a UTF-16 string should not be calculated by counting the number of bytes and dividing by two. You actually need to walk the string, inspecting each double-byte: # calculate length count = 0 inside_surrogate = False for bb in buffer: # get two bytes at a time if is_lower_surrogate(bb): inside_surrogate = True continue if is_upper_surrogate(bb): if inside_surrogate: count += 1 inside_surrogate = False continue raise ValueError("missing lower surrogate") if inside_surrogate: break count += 1 if inside_surrogate: raise ValueError("missing upper surrogate") Given immutable strings, you could validate the string once, on creation, and from then on assume they are well-formed: # calculate length, assuming the string is well-formed: count = 0 skip = False for bb in buffer: # get two bytes at a time if skip: count += 1 skip = False continue if is_surrogate(bb): skip = True count += 1 String operations such as slicing become much more complex once you can no longer assume a 1:1 relationship between code points and code units, whether they are 1, 2 or 4 bytes. Most (all?) language developers don't handle that complexity, and push responsibility for it back onto the coder using the language. -- Steven From rosuav at gmail.com Thu Jul 25 06:07:41 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 20:07:41 +1000 Subject: RE Module Performance In-Reply-To: <51f0ee48$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51EFEC17.90303@gmail.com> <51f0d0a0$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f0ee48$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, Jul 25, 2013 at 7:22 PM, Steven D'Aprano wrote: > What I'm trying to say is that it is possible to use UTF-16 internally, > but *not* assume that every code point (character) is represented by a > single 2-byte unit. For example, the len() of a UTF-16 string should not > be calculated by counting the number of bytes and dividing by two. You > actually need to walk the string, inspecting each double-byte Anything's possible. But since underlying representations can be changed fairly easily (relative term of course - it's a lot of work, but it can be changed in a single release, no deprecation required or anything), there's very little reason to continue using UTF-16 underneath. May as well switch to UTF-32 for convenience, or PEP 393 for convenience and efficiency, or maybe some other system that's still mostly fixed-width. ChrisA From steve+comp.lang.python at pearwood.info Thu Jul 25 01:56:46 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 25 Jul 2013 05:56:46 GMT Subject: RE Module Performance References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> Message-ID: <51f0be1e$0$29971$c3e8da3$5496439d@news.astraweb.com> On Wed, 24 Jul 2013 09:00:39 -0600, Michael Torrie wrote about JMF: > His most recent argument that Python should use UTF as a representation > is very strange to be honest. He's not arguing for anything, he is just hating on anything that gives even the tiniest benefit to ASCII users. This isn't about Python 3.3. hurting non-ASCII users, because that is demonstrably untrue: they are *better off* in Python 3.3. This is about denying even a tiny benefit to ASCII users. In Python 3.3, non-ASCII users have these advantages compared to previous versions: - strings will usually take less memory, and aside from trivial changes to the object header, they never take more memory than a wide build would use; - consequently nearly all objects will take less memory (especially builtins and standard library objects, which are all ASCII), since objects contain dozens of internal strings (attribute and method names in __dict__, class name, etc.); - consequently whole-application benchmarks show most applications will use significantly less memory, which leads to faster speeds; - you cannot break surrogate pairs apart by accident, which you can do in narrow builds; - in previous versions, code which works when run in a wide build may fail in a narrow build, but that is no longer an issue since the distinction between wide and narrow builds is gone; - Latin1 users, which includes JMF himself, will likewise see memory savings, since Latin1 strings will take half the size of narrow builds and a quarter the size of wide builds. The cost of all these benefits is a small overhead when creating a string in the first place, and some purely internal added complication to the string implementation. I'm the first to argue against complication unless there is a corresponding benefit. This is a case where the benefit has proven itself doubly: Python 3.3's Unicode implementation is *more correct* than before, and it uses less memory to do so. > The cons of UTF are apparent and widely > known. The main con is that UTF strings are O(n) for indexing a > position within the string. Not so for UTF-32. -- Steven From ian.g.kelly at gmail.com Thu Jul 25 16:53:10 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Thu, 25 Jul 2013 15:53:10 -0500 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> Message-ID: On Wed, Jul 24, 2013 at 9:34 AM, Chris Angelico wrote: > On Thu, Jul 25, 2013 at 12:17 AM, David Hutto wrote: >> I've screwed up plenty of times in python, but can write code like a pro >> when I'm feeling better(on SSI and medicaid). An editor can be built simply, >> but it's preference that makes the difference. Some might have used tkinter, >> gtk. wxpython or other methods for the task. >> >> I think the main issue in responding is your library preference, or widget >> set preference. These can make you right with some in your response, or >> wrong with others that have a preferable gui library that coincides with >> one's personal cognitive structure that makes t > > jmf's point is more about writing the editor widget (Scintilla, as > opposed to SciTE), which most people will never bother to do. I've > written several text editors, always by embedding someone else's > widget, and therefore not concerning myself with its internal string > representation. Frankly, Python's strings are a *terrible* internal > representation for an editor widget - not because of PEP 393, but > simply because they are immutable, and every keypress would result in > a rebuilding of the string. On the flip side, I could quite plausibly > imagine using a list of strings; whenever text gets inserted, the > string gets split at that point, and a new string created for the > insert (which also means that an Undo operation simply removes one > entire string). In this usage, the FSR is beneficial, as it's possible > to have different strings at different widths. > > But mainly, I'm just wondering how many people here have any basis > from which to argue the point he's trying to make. I doubt most of us > have (a) implemented an editor widget, or (b) tested multiple > different internal representations to learn the true pros and cons of > each. And even if any of us had, that still wouldn't have any bearing > on PEP 393, which is about applications, not editor widgets. As stated > above, Python strings before AND after PEP 393 are poor choices for an > editor, ergo arguing from that standpoint is pretty useless. Not that > that bothers jmf... I think you've just motivated me to finally get around to writing the custom output widget for my MUD client. Of course that will be simpler than a standard rich text editor widget, since it will never receive input from the user and modifications will (typically) always come in the form of append operations. I intend to write it in pure Python (well, wxPython), however. From steve+comp.lang.python at pearwood.info Thu Jul 25 03:02:21 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 25 Jul 2013 07:02:21 GMT Subject: RE Module Performance References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> Message-ID: <51f0cd7c$0$29971$c3e8da3$5496439d@news.astraweb.com> On Thu, 25 Jul 2013 00:34:24 +1000, Chris Angelico wrote: > But mainly, I'm just wondering how many people here have any basis from > which to argue the point he's trying to make. I doubt most of us have > (a) implemented an editor widget, or (b) tested multiple different > internal representations to learn the true pros and cons of each. And > even if any of us had, that still wouldn't have any bearing on PEP 393, > which is about applications, not editor widgets. As stated above, Python > strings before AND after PEP 393 are poor choices for an editor, ergo > arguing from that standpoint is pretty useless. That's a misleading way to put it. Using immutable strings as editor buffers might be a bad way to implement all but the most trivial, low- performance (i.e. slow) editor, but the basic concept of PEP 393, picking an internal representation of the text based on its contents, is not. That's just normal. The only difference with PEP 393 is that the choice is made on the fly, at runtime, instead of decided in advance by the programmer. I expect that the PEP 393 concept of optimizing memory per string buffer would work well in an editor. However the internal buffer is arranged, you can safely assume that each chunk of text (word, sentence, paragraph, buffer...) will very rarely shift from "all Latin 1" to "all BMP" to "includes SMP chars". So, for example, entering a SMP character will need to immediately up-cast the chunk from 1-byte per char to 4-bytes per char, which is relatively pricey, but it's a one-off cost. Down-casting when the SMP character is deleted doesn't need to be done immediately, it can be performed when the application is idle. If the chunks are relatively small (say, a paragraph rather than multiple pages of text) then even that initial conversion will be invisible. A fast touch typist hits a key about every 0.1 of a second; if it takes a millisecond to convert the chunk, you wouldn't even notice the delay. You can copy and up-cast a lot of bytes in a millisecond. -- Steven From rosuav at gmail.com Thu Jul 25 03:39:43 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 17:39:43 +1000 Subject: RE Module Performance In-Reply-To: <51f0cd7c$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f0cd7c$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, Jul 25, 2013 at 5:02 PM, Steven D'Aprano wrote: > On Thu, 25 Jul 2013 00:34:24 +1000, Chris Angelico wrote: > >> But mainly, I'm just wondering how many people here have any basis from >> which to argue the point he's trying to make. I doubt most of us have >> (a) implemented an editor widget, or (b) tested multiple different >> internal representations to learn the true pros and cons of each. And >> even if any of us had, that still wouldn't have any bearing on PEP 393, >> which is about applications, not editor widgets. As stated above, Python >> strings before AND after PEP 393 are poor choices for an editor, ergo >> arguing from that standpoint is pretty useless. > > That's a misleading way to put it. Using immutable strings as editor > buffers might be a bad way to implement all but the most trivial, low- > performance (i.e. slow) editor, but the basic concept of PEP 393, picking > an internal representation of the text based on its contents, is not. > That's just normal. The only difference with PEP 393 is that the choice > is made on the fly, at runtime, instead of decided in advance by the > programmer. Maybe I worded it poorly, but my point was the same as you're saying here: that a Python string is a poor buffer for editing, regardless of PEP 393. It's not that PEP 393 makes Python strings worse for writing a text editor, it's that immutability does that. ChrisA From torriem at gmail.com Wed Jul 24 10:47:36 2013 From: torriem at gmail.com (Michael Torrie) Date: Wed, 24 Jul 2013 08:47:36 -0600 Subject: RE Module Performance In-Reply-To: <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> Message-ID: <51EFE908.6080503@gmail.com> On 07/24/2013 07:40 AM, wxjmfauth at gmail.com wrote: > Sorry, you are not understanding Unicode. What is a Unicode > Transformation Format (UTF), what is the goal of a UTF and > why it is important for an implementation to work with a UTF. Really? Enlighten me. Personally, I would never use UTF as a representation *in memory* for a unicode string if it were up to me. Why? Because UTF characters are not uniform in byte width so accessing positions within the string is terribly slow and has to always be done by starting at the beginning of the string. That's at minimum O(n) compared to FSR's O(1). Surely you understand this. Do you dispute this fact? UTF is a great choice for interchange, though, and indeed that's what it was designed for. Are you calling for UTF to be adopted as the internal, in-memory representation of unicode? Or would you simply settle for UCS-4? Please be clear here. What are you saying? > Short example. Writing an editor with something like the > FSR is simply impossible (properly). How? FSR is just an implementation detail. It could be UCS-4 and it would also work. From rosuav at gmail.com Wed Jul 24 10:56:27 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 00:56:27 +1000 Subject: RE Module Performance In-Reply-To: <51EFE908.6080503@gmail.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51EFE908.6080503@gmail.com> Message-ID: On Thu, Jul 25, 2013 at 12:47 AM, Michael Torrie wrote: > On 07/24/2013 07:40 AM, wxjmfauth at gmail.com wrote: >> Sorry, you are not understanding Unicode. What is a Unicode >> Transformation Format (UTF), what is the goal of a UTF and >> why it is important for an implementation to work with a UTF. > > Really? Enlighten me. > > Personally, I would never use UTF as a representation *in memory* for a > unicode string if it were up to me. Why? Because UTF characters are > not uniform in byte width so accessing positions within the string is > terribly slow and has to always be done by starting at the beginning of > the string. That's at minimum O(n) compared to FSR's O(1). Surely you > understand this. Do you dispute this fact? Take care here; UTF is a general term for Unicode Translation Formats, of which one (UTF-32) is fixed-width. Every other UTF-n is variable width, though, so your point still stands. UTF-32 is the basis for Python's FSR. ChrisA From wxjmfauth at gmail.com Thu Jul 25 05:27:42 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Thu, 25 Jul 2013 02:27:42 -0700 (PDT) Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> Message-ID: <0420de60-b9b5-4ac4-ba7b-ca5ac2ca65fe@googlegroups.com> Le mercredi 24 juillet 2013 16:47:36 UTC+2, Michael Torrie a ?crit?: > On 07/24/2013 07:40 AM, wxjmfauth at gmail.com wrote: > > > Sorry, you are not understanding Unicode. What is a Unicode > > > Transformation Format (UTF), what is the goal of a UTF and > > > why it is important for an implementation to work with a UTF. > > > > Really? Enlighten me. > > > > Personally, I would never use UTF as a representation *in memory* for a > > unicode string if it were up to me. Why? Because UTF characters are > > not uniform in byte width so accessing positions within the string is > > terribly slow and has to always be done by starting at the beginning of > > the string. That's at minimum O(n) compared to FSR's O(1). Surely you > > understand this. Do you dispute this fact? > > > > UTF is a great choice for interchange, though, and indeed that's what it > > was designed for. > > > > Are you calling for UTF to be adopted as the internal, in-memory > > representation of unicode? Or would you simply settle for UCS-4? > > Please be clear here. What are you saying? > > > > > Short example. Writing an editor with something like the > > > FSR is simply impossible (properly). > > > > How? FSR is just an implementation detail. It could be UCS-4 and it > > would also work. --------- A coding scheme works with a unique set of characters (the repertoire), and the implementation (the programming) works with a unique set of encoded code points. The critical step is the path {unique set of characters} <--> {unique set of encoded code points} Fact: there is no other way to do it properly (This is explaining why we have to live today with all these coding schemes or also explaining why so many coding schemes hadto be created). How to understand it? With a sheet of paper and a pencil. In the byte string world, this step is a no-op. In Unicode, it is exactly the purpose of a "utf" to achieve this step. "utf": a confusing name covering at the same time the process and the result of the process. A "utf chunk", a series of bits (not bytes), hold intrisically the information about the character it is representing. Other "exotic" coding schemes like iso6937 of "CID-fonts" are woking in the same way. "Unicode" with the help of "utf(s)" does not differ from the basic rule. ----- ucs-2: ucs-2 is a perfecly and correctly working coding scheme. ucs-2 is not different from the other coding schemes and does not behave differently (cp... or iso-... or ...). It only covers a smaller repertoire. ----- utf32: as a pointed many times. You are already using it (maybe without knowing it). Where? in fonts (OpenType technology), rendering engines, pdf files. Why? Because there is not other way to do it better. ------ The Unicode table (its constuction) is a problem per se. It is not a technical problem, a very important "linguistic aspect" of Unicode. See https://groups.google.com/forum/#!topic/comp.lang.python/XkTKE7U8CS0 ------ If you are not understanding my "editor" analogy. One other proposed exercise. Build/create a flexible iso-8859-X coding scheme. You will quickly understand where the bottleneck is. Two working ways: - stupidly with an editor and your fingers. - lazily with a sheet of paper and you head. ---- About my benchmarks: No offense. You are not understanding them, because you do not understand what this FSR does and the coding of characters. It's a little bit a devil's circle. Conceptually, this FSR is spending its time in solving the problem it creates itsself, with plenty of side effects. ----- There is a clear difference between FSR and ucs-4/utf32. ----- See also: http://www.unicode.org/reports/tr17/ (In my mind, quite "dry" and not easy to understand at a first reading). jmf From rosuav at gmail.com Thu Jul 25 06:14:46 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 20:14:46 +1000 Subject: RE Module Performance In-Reply-To: <0420de60-b9b5-4ac4-ba7b-ca5ac2ca65fe@googlegroups.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <0420de60-b9b5-4ac4-ba7b-ca5ac2ca65fe@googlegroups.com> Message-ID: On Thu, Jul 25, 2013 at 7:27 PM, wrote: > A coding scheme works with a unique set of characters (the repertoire), > and the implementation (the programming) works with a unique set > of encoded code points. The critical step is the path > {unique set of characters} <--> {unique set of encoded code points} That's called Unicode. It maps the character 'A' to the code point U+0041 and so on. Code points are integers. In fact, they are very well represented in Python that way (also in Pike, fwiw): >>> ord('A') 65 >>> chr(65) 'A' >>> chr(123456) '\U0001e240' >>> ord(_) 123456 > In the byte string world, this step is a no-op. > > In Unicode, it is exactly the purpose of a "utf" to achieve this > step. "utf": a confusing name covering at the same time the > process and the result of the process. > A "utf chunk", a series of bits (not bytes), hold intrisically > the information about the character it is representing. No, now you're looking at another level: how to store codepoints in memory. That demands that they be stored as bits and bytes, because PC memory works that way. > utf32: as a pointed many times. You are already using it (maybe > without knowing it). Where? in fonts (OpenType technology), > rendering engines, pdf files. Why? Because there is not other > way to do it better. And UTF-32 is an excellent system... as long as you're okay with spending four bytes for every character. > See https://groups.google.com/forum/#!topic/comp.lang.python/XkTKE7U8CS0 I refuse to click this link. Give us a link to the python-list at python.org archive, or gmane, or something else more suited to the audience. I'm not going to Google Groups just to figure out what you're saying. > If you are not understanding my "editor" analogy. One other > proposed exercise. Build/create a flexible iso-8859-X coding > scheme. You will quickly understand where the bottleneck > is. > Two working ways: > - stupidly with an editor and your fingers. > - lazily with a sheet of paper and you head. What has this to do with the editor? > There is a clear difference between FSR and ucs-4/utf32. Yes. Memory usage. PEP 393 strings might take up half or even a quarter of what they'd take up in fixed UTF-32. Other than that, there's no difference. ChrisA From wxjmfauth at gmail.com Thu Jul 25 15:07:45 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Thu, 25 Jul 2013 12:07:45 -0700 (PDT) Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <0420de60-b9b5-4ac4-ba7b-ca5ac2ca65fe@googlegroups.com> Message-ID: <741eaf38-6655-4763-8962-748408e7c2d8@googlegroups.com> Le jeudi 25 juillet 2013 12:14:46 UTC+2, Chris Angelico a ?crit?: > On Thu, Jul 25, 2013 at 7:27 PM, wrote: > > > A coding scheme works with a unique set of characters (the repertoire), > > > and the implementation (the programming) works with a unique set > > > of encoded code points. The critical step is the path > > > {unique set of characters} <--> {unique set of encoded code points} > > > > That's called Unicode. It maps the character 'A' to the code point > > U+0041 and so on. Code points are integers. In fact, they are very > > well represented in Python that way (also in Pike, fwiw): > > > > >>> ord('A') > > 65 > > >>> chr(65) > > 'A' > > >>> chr(123456) > > '\U0001e240' > > >>> ord(_) > > 123456 > > > > > In the byte string world, this step is a no-op. > > > > > > In Unicode, it is exactly the purpose of a "utf" to achieve this > > > step. "utf": a confusing name covering at the same time the > > > process and the result of the process. > > > A "utf chunk", a series of bits (not bytes), hold intrisically > > > the information about the character it is representing. > > > > No, now you're looking at another level: how to store codepoints in > > memory. That demands that they be stored as bits and bytes, because PC > > memory works that way. > > > > > utf32: as a pointed many times. You are already using it (maybe > > > without knowing it). Where? in fonts (OpenType technology), > > > rendering engines, pdf files. Why? Because there is not other > > > way to do it better. > > > > And UTF-32 is an excellent system... as long as you're okay with > > spending four bytes for every character. > > > > > See https://groups.google.com/forum/#!topic/comp.lang.python/XkTKE7U8CS0 > > > > I refuse to click this link. Give us a link to the > > python-list at python.org archive, or gmane, or something else more > > suited to the audience. I'm not going to Google Groups just to figure > > out what you're saying. > > > > > If you are not understanding my "editor" analogy. One other > > > proposed exercise. Build/create a flexible iso-8859-X coding > > > scheme. You will quickly understand where the bottleneck > > > is. > > > Two working ways: > > > - stupidly with an editor and your fingers. > > > - lazily with a sheet of paper and you head. > > > > What has this to do with the editor? > > > > > There is a clear difference between FSR and ucs-4/utf32. > > > > Yes. Memory usage. PEP 393 strings might take up half or even a > > quarter of what they'd take up in fixed UTF-32. Other than that, > > there's no difference. > > > > ChrisA -------- Let start with a simple string \textemdash or \texttendash >>> sys.getsizeof('?') 40 >>> sys.getsizeof('a') 26 jmf jmf From rosuav at gmail.com Thu Jul 25 15:18:44 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 26 Jul 2013 05:18:44 +1000 Subject: RE Module Performance In-Reply-To: <741eaf38-6655-4763-8962-748408e7c2d8@googlegroups.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <0420de60-b9b5-4ac4-ba7b-ca5ac2ca65fe@googlegroups.com> <741eaf38-6655-4763-8962-748408e7c2d8@googlegroups.com> Message-ID: On Fri, Jul 26, 2013 at 5:07 AM, wrote: > Let start with a simple string \textemdash or \texttendash > >>>> sys.getsizeof('?') > 40 >>>> sys.getsizeof('a') > 26 Most of the cost is in those two apostrophes, look: >>> sys.getsizeof('a') 26 >>> sys.getsizeof(a) 8 Okay, that's slightly unfair (bonus points: figure out what I did to make this work; there are at least two right answers) but still, look at what an empty string costs: >>> sys.getsizeof('') 25 Or look at the difference between one of these characters and two: >>> sys.getsizeof('aa')-sys.getsizeof('a') 1 >>> sys.getsizeof('??')-sys.getsizeof('?') 2 That's what the characters really cost. The overhead is fixed. It is, in fact, almost completely insignificant. The storage requirement for a non-ASCII, BMP-only string converges to two bytes per character. ChrisA From ramit.prasad at jpmorgan.com Thu Jul 25 15:30:51 2013 From: ramit.prasad at jpmorgan.com (Prasad, Ramit) Date: Thu, 25 Jul 2013 19:30:51 +0000 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <0420de60-b9b5-4ac4-ba7b-ca5ac2ca65fe@googlegroups.com> <741eaf38-6655-4763-8962-748408e7c2d8@googlegroups.com> Message-ID: <5B80DD153D7D744689F57F4FB69AF474185CBA13@SCACMX008.exchad.jpmchase.net> Chris Angelico wrote: > On Fri, Jul 26, 2013 at 5:07 AM, wrote: > > Let start with a simple string \textemdash or \texttendash > > > >>>> sys.getsizeof('-') > > 40 > >>>> sys.getsizeof('a') > > 26 > > Most of the cost is in those two apostrophes, look: > > >>> sys.getsizeof('a') > 26 > >>> sys.getsizeof(a) > 8 > > Okay, that's slightly unfair (bonus points: figure out what I did to > make this work; there are at least two right answers) but still, look > at what an empty string costs: I like bonus points. :) >>> a = None >>> sys.getsizeof(a) 8 Not sure what the other right answer is...booleans take 12 bytes (on 2.6) > > >>> sys.getsizeof('') > 25 > > Or look at the difference between one of these characters and two: > > >>> sys.getsizeof('aa')-sys.getsizeof('a') > 1 > >>> sys.getsizeof('--')-sys.getsizeof('-') > 2 > > That's what the characters really cost. The overhead is fixed. It is, > in fact, almost completely insignificant. The storage requirement for > a non-ASCII, BMP-only string converges to two bytes per character. > > ChrisA > -- > http://mail.python.org/mailman/listinfo/python-list Ramit This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email. From torriem at gmail.com Thu Jul 25 23:06:21 2013 From: torriem at gmail.com (Michael Torrie) Date: Thu, 25 Jul 2013 21:06:21 -0600 Subject: RE Module Performance In-Reply-To: <741eaf38-6655-4763-8962-748408e7c2d8@googlegroups.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <0420de60-b9b5-4ac4-ba7b-ca5ac2ca65fe@googlegroups.com> <741eaf38-6655-4763-8962-748408e7c2d8@googlegroups.com> Message-ID: <51F1E7AD.9090705@gmail.com> On 07/25/2013 01:07 PM, wxjmfauth at gmail.com wrote: > Let start with a simple string \textemdash or \texttendash > >>>> sys.getsizeof('?') > 40 >>>> sys.getsizeof('a') > 26 That's meaningless. You're comparing the overhead of a string object itself (a one-time cost anyway), not the overhead of storing the actual characters. This is the only meaningful comparison: >>>> sys.getsizeof('??') - sys.getsizeof('?') >>>> sys.getsizeof('aa') - sys.getsizeof('a') Actually I'm not even sure what your point is after all this time of railing against FSR. You have said in the past that Python penalizes users of character sets that require wider byte encodings, but what would you have us do? use 4-byte characters and penalize everyone equally? Use 2-byte characters that incorrectly expose surrogate pairs for some characters? Use UTF-8 in memory and do O(n) indexing? Are your programs (actual programs, not contrived benchmarks) actually slower because of FSR? Is FSR incorrect? If so, according to what part of the unicode standard? I'm not trying to troll, or feed the troll. I'm actually curious. I think perhaps you feel that many of us who don't use unicode often don't understand unicode because some of us don't understand you. If so, I'm not sure that's actually true. From jeremy at jeremysanders.net Thu Jul 25 09:36:25 2013 From: jeremy at jeremysanders.net (Jeremy Sanders) Date: Thu, 25 Jul 2013 14:36:25 +0100 Subject: RE Module Performance References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> Message-ID: wxjmfauth at gmail.com wrote: > Short example. Writing an editor with something like the > FSR is simply impossible (properly). http://www.gnu.org/software/emacs/manual/html_node/elisp/Text-Representations.html#Text-Representations "To conserve memory, Emacs does not hold fixed-length 22-bit numbers that are codepoints of text characters within buffers and strings. Rather, Emacs uses a variable-length internal representation of characters, that stores each character as a sequence of 1 to 5 8-bit bytes, depending on the magnitude of its codepoint[1]. For example, any ASCII character takes up only 1 byte, a Latin-1 character takes up 2 bytes, etc. We call this representation of text multibyte. ... [1] This internal representation is based on one of the encodings defined by the Unicode Standard, called UTF-8, for representing any Unicode codepoint, but Emacs extends UTF-8 to represent the additional codepoints it uses for raw 8- bit bytes and characters not unified with Unicode. " Jeremy From devyncjohnson at gmail.com Thu Jul 25 09:44:38 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Thu, 25 Jul 2013 09:44:38 -0400 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> Message-ID: <51F12BC6.8040804@Gmail.com> On 07/25/2013 09:36 AM, Jeremy Sanders wrote: > wxjmfauth at gmail.com wrote: > >> Short example. Writing an editor with something like the >> FSR is simply impossible (properly). > http://www.gnu.org/software/emacs/manual/html_node/elisp/Text-Representations.html#Text-Representations > > "To conserve memory, Emacs does not hold fixed-length 22-bit numbers that are > codepoints of text characters within buffers and strings. Rather, Emacs uses a > variable-length internal representation of characters, that stores each > character as a sequence of 1 to 5 8-bit bytes, depending on the magnitude of > its codepoint[1]. For example, any ASCII character takes up only 1 byte, a > Latin-1 character takes up 2 bytes, etc. We call this representation of text > multibyte. > > ... > > [1] This internal representation is based on one of the encodings defined by > the Unicode Standard, called UTF-8, for representing any Unicode codepoint, but > Emacs extends UTF-8 to represent the additional codepoints it uses for raw 8- > bit bytes and characters not unified with Unicode. > > " > > Jeremy > > Wow! The thread that I started has changed a lot and lived a long time. I look forward to its first birthday (^u^). Devyn Collier Johnson From steve+comp.lang.python at pearwood.info Thu Jul 25 11:26:13 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 25 Jul 2013 15:26:13 GMT Subject: RE Module Performance References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> Message-ID: <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> On Thu, 25 Jul 2013 14:36:25 +0100, Jeremy Sanders wrote: > wxjmfauth at gmail.com wrote: > >> Short example. Writing an editor with something like the FSR is simply >> impossible (properly). > > http://www.gnu.org/software/emacs/manual/html_node/elisp/Text- Representations.html#Text-Representations > > "To conserve memory, Emacs does not hold fixed-length 22-bit numbers > that are codepoints of text characters within buffers and strings. > Rather, Emacs uses a variable-length internal representation of > characters, that stores each character as a sequence of 1 to 5 8-bit > bytes, depending on the magnitude of its codepoint[1]. For example, any > ASCII character takes up only 1 byte, a Latin-1 character takes up 2 > bytes, etc. We call this representation of text multibyte. Well, you've just proven what Vim users have always suspected: Emacs doesn't really exist. > [1] This internal representation is based on one of the encodings > defined by the Unicode Standard, called UTF-8, for representing any > Unicode codepoint, but Emacs extends UTF-8 to represent the additional > codepoints it uses for raw 8- bit bytes and characters not unified with > Unicode. > " Do you know what those characters not unified with Unicode are? Is there a list somewhere? I've read all of the pages from here to no avail: http://www.gnu.org/software/emacs/manual/html_node/elisp/Non_002dASCII-Characters.html -- Steven From rosuav at gmail.com Thu Jul 25 11:36:07 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 26 Jul 2013 01:36:07 +1000 Subject: RE Module Performance In-Reply-To: <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Fri, Jul 26, 2013 at 1:26 AM, Steven D'Aprano wrote: > On Thu, 25 Jul 2013 14:36:25 +0100, Jeremy Sanders wrote: >> "To conserve memory, Emacs does not hold fixed-length 22-bit numbers >> that are codepoints of text characters within buffers and strings. >> Rather, Emacs uses a variable-length internal representation of >> characters, that stores each character as a sequence of 1 to 5 8-bit >> bytes, depending on the magnitude of its codepoint[1]. For example, any >> ASCII character takes up only 1 byte, a Latin-1 character takes up 2 >> bytes, etc. We call this representation of text multibyte. > > Well, you've just proven what Vim users have always suspected: Emacs > doesn't really exist. ... lolwut? ChrisA From steve+comp.lang.python at pearwood.info Thu Jul 25 13:18:59 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 25 Jul 2013 17:18:59 GMT Subject: RE Module Performance References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> On Fri, 26 Jul 2013 01:36:07 +1000, Chris Angelico wrote: > On Fri, Jul 26, 2013 at 1:26 AM, Steven D'Aprano > wrote: >> On Thu, 25 Jul 2013 14:36:25 +0100, Jeremy Sanders wrote: >>> "To conserve memory, Emacs does not hold fixed-length 22-bit numbers >>> that are codepoints of text characters within buffers and strings. >>> Rather, Emacs uses a variable-length internal representation of >>> characters, that stores each character as a sequence of 1 to 5 8-bit >>> bytes, depending on the magnitude of its codepoint[1]. For example, >>> any ASCII character takes up only 1 byte, a Latin-1 character takes up >>> 2 bytes, etc. We call this representation of text multibyte. >> >> Well, you've just proven what Vim users have always suspected: Emacs >> doesn't really exist. > > ... lolwut? JMF has explained that it is impossible, impossible I say!, to write an editor using a flexible string representation. Since Emacs uses such a flexible string representation, Emacs is impossible, and therefore Emacs doesn't exist. QED. -- Steven From rosuav at gmail.com Thu Jul 25 13:27:29 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 26 Jul 2013 03:27:29 +1000 Subject: RE Module Performance In-Reply-To: <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Fri, Jul 26, 2013 at 3:18 AM, Steven D'Aprano wrote: > On Fri, 26 Jul 2013 01:36:07 +1000, Chris Angelico wrote: > >> On Fri, Jul 26, 2013 at 1:26 AM, Steven D'Aprano >> wrote: >>> On Thu, 25 Jul 2013 14:36:25 +0100, Jeremy Sanders wrote: >>>> "To conserve memory, Emacs does not hold fixed-length 22-bit numbers >>>> that are codepoints of text characters within buffers and strings. >>>> Rather, Emacs uses a variable-length internal representation of >>>> characters, that stores each character as a sequence of 1 to 5 8-bit >>>> bytes, depending on the magnitude of its codepoint[1]. For example, >>>> any ASCII character takes up only 1 byte, a Latin-1 character takes up >>>> 2 bytes, etc. We call this representation of text multibyte. >>> >>> Well, you've just proven what Vim users have always suspected: Emacs >>> doesn't really exist. >> >> ... lolwut? > > > JMF has explained that it is impossible, impossible I say!, to write an > editor using a flexible string representation. Since Emacs uses such a > flexible string representation, Emacs is impossible, and therefore Emacs > doesn't exist. > > QED. Quad Error Demonstrated. I never got past the level of Canis Latinicus in debating class. ChrisA From ian.g.kelly at gmail.com Thu Jul 25 16:45:38 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Thu, 25 Jul 2013 15:45:38 -0500 Subject: RE Module Performance In-Reply-To: <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, Jul 25, 2013 at 12:18 PM, Steven D'Aprano wrote: > On Fri, 26 Jul 2013 01:36:07 +1000, Chris Angelico wrote: > >> On Fri, Jul 26, 2013 at 1:26 AM, Steven D'Aprano >> wrote: >>> On Thu, 25 Jul 2013 14:36:25 +0100, Jeremy Sanders wrote: >>>> "To conserve memory, Emacs does not hold fixed-length 22-bit numbers >>>> that are codepoints of text characters within buffers and strings. >>>> Rather, Emacs uses a variable-length internal representation of >>>> characters, that stores each character as a sequence of 1 to 5 8-bit >>>> bytes, depending on the magnitude of its codepoint[1]. For example, >>>> any ASCII character takes up only 1 byte, a Latin-1 character takes up >>>> 2 bytes, etc. We call this representation of text multibyte. >>> >>> Well, you've just proven what Vim users have always suspected: Emacs >>> doesn't really exist. >> >> ... lolwut? > > > JMF has explained that it is impossible, impossible I say!, to write an > editor using a flexible string representation. Since Emacs uses such a > flexible string representation, Emacs is impossible, and therefore Emacs > doesn't exist. > > QED. Except that the described representation used by Emacs is a variant of UTF-8, not an FSR. It doesn't have three different possible encodings for the letter 'a' depending on what other characters happen to be in the string. As I understand it, jfm would be perfectly happy if Python used UTF-8 (or presumably the Emacs variant) as its internal string representation. From steve+comp.lang.python at pearwood.info Thu Jul 25 22:48:17 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 26 Jul 2013 02:48:17 GMT Subject: RE Module Performance References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51f1e371$0$29971$c3e8da3$5496439d@news.astraweb.com> On Thu, 25 Jul 2013 15:45:38 -0500, Ian Kelly wrote: > On Thu, Jul 25, 2013 at 12:18 PM, Steven D'Aprano > wrote: >> On Fri, 26 Jul 2013 01:36:07 +1000, Chris Angelico wrote: >> >>> On Fri, Jul 26, 2013 at 1:26 AM, Steven D'Aprano >>> wrote: >>>> On Thu, 25 Jul 2013 14:36:25 +0100, Jeremy Sanders wrote: >>>>> "To conserve memory, Emacs does not hold fixed-length 22-bit numbers >>>>> that are codepoints of text characters within buffers and strings. >>>>> Rather, Emacs uses a variable-length internal representation of >>>>> characters, that stores each character as a sequence of 1 to 5 8-bit >>>>> bytes, depending on the magnitude of its codepoint[1]. For example, >>>>> any ASCII character takes up only 1 byte, a Latin-1 character takes >>>>> up 2 bytes, etc. We call this representation of text multibyte. >>>> >>>> Well, you've just proven what Vim users have always suspected: Emacs >>>> doesn't really exist. >>> >>> ... lolwut? >> >> >> JMF has explained that it is impossible, impossible I say!, to write an >> editor using a flexible string representation. Since Emacs uses such a >> flexible string representation, Emacs is impossible, and therefore >> Emacs doesn't exist. >> >> QED. > > Except that the described representation used by Emacs is a variant of > UTF-8, not an FSR. It doesn't have three different possible encodings > for the letter 'a' depending on what other characters happen to be in > the string. > > As I understand it, jfm would be perfectly happy if Python used UTF-8 > (or presumably the Emacs variant) as its internal string representation. UTF-8 uses a flexible representation on a character-by-character basis. When parsing UTF-8, one needs to look at EVERY character to decide how many bytes you need to read. In Python 3, the flexible representation is on a string-by-string basis: once Python has looked at the string header, it can tell whether the *entire* string takes 1, 2 or 4 bytes per character, and the string is then fixed-width. You can't do that with UTF-8. To put it in terms of pseudo-code: # Python 3.3 def parse_string(astring): # Decision gets made once per string. if astring uses 1 byte: count = 1 elif astring uses 2 bytes: count = 2 else: count = 4 while not done: char = convert(next(count bytes)) # UTF-8 def parse_string(astring): while not done: b = next(1 byte) # Decision gets made for every single char if uses 1 byte: char = convert(b) elif uses 2 bytes: char = convert(b, next(1 byte)) elif uses 3 bytes: char = convert(b, next(2 bytes)) else: char = convert(b, next(3 bytes)) So UTF-8 requires much more runtime overhead than Python 3.3, and Emac's variation can in fact require more bytes per character than either. (UTF-8 and Python 3.3 can require up to four bytes, Emacs up to five.) I'm not surprised that JMF would prefer UTF-8 -- he is completely out of his depth, and is a fine example of the Dunning-Kruger effect in action. He is so sure he is right based on so little evidence. One advantage of UTF-8 is that for some BMP characters, you can get away with only three bytes instead of four. For transmitting data over the wire, or storage on disk, that's potentially up to a 25% reduction in space, which is not to be sneezed at. (Although in practice it's usually much less than that, since the most common characters are encoded to 1 or 2 bytes, not 4). But that comes at the cost of much more runtime overhead, which in my opinion makes UTF-8 a second-class string representation compared to fixed-width representations. -- Steven From ian.g.kelly at gmail.com Thu Jul 25 23:20:45 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Thu, 25 Jul 2013 21:20:45 -0600 Subject: RE Module Performance In-Reply-To: <51f1e371$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f1e371$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, Jul 25, 2013 at 8:48 PM, Steven D'Aprano wrote: > UTF-8 uses a flexible representation on a character-by-character basis. > When parsing UTF-8, one needs to look at EVERY character to decide how > many bytes you need to read. In Python 3, the flexible representation is > on a string-by-string basis: once Python has looked at the string header, > it can tell whether the *entire* string takes 1, 2 or 4 bytes per > character, and the string is then fixed-width. You can't do that with > UTF-8. UTF-8 does not use a flexible representation. A codec that is encoding a string in UTF-8 and examining a particular character does not have any choice of how to encode that character; there is exactly one sequence of bits that is the UTF-8 encoding for the character. Further, for any given sequence of code points there is exactly one sequence of bytes that is the UTF-8 encoding of those code points. In contrast, with the FSR there are as many as three different sequences of bytes that encode a sequence of code points, with one of them (the shortest) being canonical. That's what makes it flexible. Anyway, my point was just that Emacs is not a counter-example to jmf's claim about implementing text editors, because UTF-8 is not what he (or anybody else) is referring to when speaking of the FSR or "something like the FSR". From wxjmfauth at gmail.com Fri Jul 26 09:36:49 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Fri, 26 Jul 2013 06:36:49 -0700 (PDT) Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f1e371$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <1ca6bb15-ce10-4a23-82fc-aa0af0f7ac97@googlegroups.com> Le vendredi 26 juillet 2013 05:20:45 UTC+2, Ian a ?crit?: > On Thu, Jul 25, 2013 at 8:48 PM, Steven D'Aprano > > wrote: > > > UTF-8 uses a flexible representation on a character-by-character basis. > > > When parsing UTF-8, one needs to look at EVERY character to decide how > > > many bytes you need to read. In Python 3, the flexible representation is > > > on a string-by-string basis: once Python has looked at the string header, > > > it can tell whether the *entire* string takes 1, 2 or 4 bytes per > > > character, and the string is then fixed-width. You can't do that with > > > UTF-8. > > > > UTF-8 does not use a flexible representation. A codec that is > > encoding a string in UTF-8 and examining a particular character does > > not have any choice of how to encode that character; there is exactly > > one sequence of bits that is the UTF-8 encoding for the character. > > Further, for any given sequence of code points there is exactly one > > sequence of bytes that is the UTF-8 encoding of those code points. In > > contrast, with the FSR there are as many as three different sequences > > of bytes that encode a sequence of code points, with one of them (the > > shortest) being canonical. That's what makes it flexible. > > > > Anyway, my point was just that Emacs is not a counter-example to jmf's > > claim about implementing text editors, because UTF-8 is not what he > > (or anybody else) is referring to when speaking of the FSR or > > "something like the FSR". -------- BTW, it is not necessary to use an endorsed Unicode coding scheme (utf*), a string literal would have been possible, but then one falls on memory issures. All these utf are following the basic coding scheme. I repeat again. A coding scheme works with a unique set of characters and its implementation works with a unique set of encoded code points (the utf's, in case of Unicode). And again, that why we live today with all these coding schemes, or, to take the problem from the other side, that's because one has to work with a unique set of encoded code points, that all these coding schemes had to be created. utf's have not been created by newbies ;-) jmf From wxjmfauth at gmail.com Fri Jul 26 11:46:58 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Fri, 26 Jul 2013 08:46:58 -0700 (PDT) Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f1e371$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: Le vendredi 26 juillet 2013 05:20:45 UTC+2, Ian a ?crit?: > On Thu, Jul 25, 2013 at 8:48 PM, Steven D'Aprano > > wrote: > > > UTF-8 uses a flexible representation on a character-by-character basis. > > > When parsing UTF-8, one needs to look at EVERY character to decide how > > > many bytes you need to read. In Python 3, the flexible representation is > > > on a string-by-string basis: once Python has looked at the string header, > > > it can tell whether the *entire* string takes 1, 2 or 4 bytes per > > > character, and the string is then fixed-width. You can't do that with > > > UTF-8. > > > > UTF-8 does not use a flexible representation. A codec that is > > encoding a string in UTF-8 and examining a particular character does > > not have any choice of how to encode that character; there is exactly > > one sequence of bits that is the UTF-8 encoding for the character. > > Further, for any given sequence of code points there is exactly one > > sequence of bytes that is the UTF-8 encoding of those code points. In > > contrast, with the FSR there are as many as three different sequences > > of bytes that encode a sequence of code points, with one of them (the > > shortest) being canonical. That's what makes it flexible. > > > > Anyway, my point was just that Emacs is not a counter-example to jmf's > > claim about implementing text editors, because UTF-8 is not what he > > (or anybody else) is referring to when speaking of the FSR or > > "something like the FSR". ----- Let's be clear. I'm perfectly understanding what is utf-8 and that's for that precise reason, I put the "editor" as an exemple on the table. This FSR is not *a* coding scheme. It is more a composite coding scheme. (And form there, all the problems). BTW, I'm pleased to read "sequence of bits" and not bytes. Again, utf transformers are producing sequence of bits, call Unicode Transformation Units, with lengths of 8/16/32 *bits*, from there the names utf8/16/32. UCS transformers are (were) producing bytes, from there the names ucs-2/4. jmf From steve+comp.lang.python at pearwood.info Sat Jul 27 02:28:56 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 27 Jul 2013 06:28:56 GMT Subject: RE Module Performance References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f1e371$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51f368a8$0$29971$c3e8da3$5496439d@news.astraweb.com> On Fri, 26 Jul 2013 08:46:58 -0700, wxjmfauth wrote: > BTW, I'm pleased to read "sequence of bits" and not bytes. Again, utf > transformers are producing sequence of bits, call Unicode Transformation > Units, with lengths of 8/16/32 *bits*, from there the names utf8/16/32. > UCS transformers are (were) producing bytes, from there the names > ucs-2/4. Not only does your distinction between bits and bytes make no practical difference on nearly all hardware in common use today[1], but the Unicode Consortium disagrees with you, and defines UTC in terms of bytes: "A Unicode transformation format (UTF) is an algorithmic mapping from every Unicode code point (except surrogate code points) to a unique byte sequence." http://www.unicode.org/faq/utf_bom.html#gen2 [1] There may still be some old supercomputers where a byte is more than 8 bits in use, but they're unlikely to support Unicode. -- Steven From steve+comp.lang.python at pearwood.info Fri Jul 26 23:37:20 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 27 Jul 2013 03:37:20 GMT Subject: RE Module Performance References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f1e371$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51f3406f$0$29971$c3e8da3$5496439d@news.astraweb.com> On Thu, 25 Jul 2013 21:20:45 -0600, Ian Kelly wrote: > On Thu, Jul 25, 2013 at 8:48 PM, Steven D'Aprano > wrote: >> UTF-8 uses a flexible representation on a character-by-character basis. >> When parsing UTF-8, one needs to look at EVERY character to decide how >> many bytes you need to read. In Python 3, the flexible representation >> is on a string-by-string basis: once Python has looked at the string >> header, it can tell whether the *entire* string takes 1, 2 or 4 bytes >> per character, and the string is then fixed-width. You can't do that >> with UTF-8. > > UTF-8 does not use a flexible representation. I disagree, and so does Jeremy Sanders who first pointed out the similarity between Emacs' UTF-8 and Python's FSR. I'll quote from the Emacs documentation again: "To conserve memory, Emacs does not hold fixed-length 22-bit numbers that are codepoints of text characters within buffers and strings. Rather, Emacs uses a variable-length internal representation of characters, that stores each character as a sequence of 1 to 5 8-bit bytes, depending on the magnitude of its codepoint. For example, any ASCII character takes up only 1 byte, a Latin-1 character takes up 2 bytes, etc." And the Python FSR: "To conserve memory, Python does not hold fixed-length 21-bit numbers that are codepoints of text characters within buffers and strings. Rather, Python uses a variable-length internal representation of characters, that stores each character as a sequence of 1 to 4 8-bit bytes, depending on the magnitude of the largest codepoint in the string. For example, any all-ASCII or all-Latin1 string takes up only 1 byte per character, an all- BMP string takes up 2 bytes per character, etc." See the similarity now? Both flexibly change the width used by code- points, UTF-8 based on the code-point itself regardless of the rest of the string, Python based on the largest code-point in the string. [...] > Anyway, my point was just that Emacs is not a counter-example to jmf's > claim about implementing text editors, because UTF-8 is not what he (or > anybody else) is referring to when speaking of the FSR or "something > like the FSR". Whether JMF can see the similarities between different implementations of strings or not is beside the point, those similarities do exist. As do the differences, of course, but in this case the differences are in favour of Python's FSR. Even if your string is entirely Latin1, a UTF-8 implementation *cannot know that*, and still has to walk the string byte- by-byte checking whether the current code point requires 1, 2, 3, or 4 bytes, while a FSR implementation can simply record the fact that the string is pure Latin1 at creation time, and then treat it as fixed-width from then on. JMF claims that FSR is "impossible" to use efficiently, and yet he supports encoding schemes which are *less* efficient. Go figure. He tells us he has no problem with any of the established UTF encodings, and yet the FSR internally uses UTF-16 and UTF-32. (Technically, it's UCS-2, not UTF-16, since there are no surrogate pairs. But the difference is insignificant.) Having watched this issue from Day One when JMF first complained about it, I believe this is entirely about denying any benefit to ASCII users. Had Python implemented a system identical to the current FSR except that it added a fourth category, "all ASCII", which used an eight-byte encoding scheme (thus making ASCII strings twice as expensive as strings including code points from the Supplementary Multilingual Planes), JMF would be the scheme's number one champion. I cannot see any other rational explanation for why JMF prefers broken, buggy Unicode implementations, or implementations which are equally expensive for all strings, over one which is demonstrably correct, demonstrably saves memory, and for realistic, non-contrived benchmarks, demonstrably faster, except that he wants to punish ASCII users more than he wants to support Unicode users. -- Steven From ian.g.kelly at gmail.com Sat Jul 27 00:12:36 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Fri, 26 Jul 2013 22:12:36 -0600 Subject: RE Module Performance In-Reply-To: <51f3406f$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f1e371$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f3406f$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Fri, Jul 26, 2013 at 9:37 PM, Steven D'Aprano wrote: > See the similarity now? Both flexibly change the width used by code- > points, UTF-8 based on the code-point itself regardless of the rest of > the string, Python based on the largest code-point in the string. No, I think we're just using the word "flexible" differently. In my view, simply being variable-width does not make an encoding "flexible" in the sense of the FSR. But I'm not going to keep repeating myself in order to argue about it. > Having watched this issue from Day One when JMF first complained about > it, I believe this is entirely about denying any benefit to ASCII users. > Had Python implemented a system identical to the current FSR except that > it added a fourth category, "all ASCII", which used an eight-byte > encoding scheme (thus making ASCII strings twice as expensive as strings > including code points from the Supplementary Multilingual Planes), JMF > would be the scheme's number one champion. I agree. In fact I made a similar observation back in December: http://mail.python.org/pipermail/python-list/2012-December/636942.html From steve+comp.lang.python at pearwood.info Sat Jul 27 01:04:03 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 27 Jul 2013 05:04:03 GMT Subject: RE Module Performance References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f1e371$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f3406f$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51f354c3$0$29971$c3e8da3$5496439d@news.astraweb.com> On Fri, 26 Jul 2013 22:12:36 -0600, Ian Kelly wrote: > On Fri, Jul 26, 2013 at 9:37 PM, Steven D'Aprano > wrote: >> See the similarity now? Both flexibly change the width used by code- >> points, UTF-8 based on the code-point itself regardless of the rest of >> the string, Python based on the largest code-point in the string. > > No, I think we're just using the word "flexible" differently. In my > view, simply being variable-width does not make an encoding "flexible" > in the sense of the FSR. But I'm not going to keep repeating myself in > order to argue about it. But I paid for the full half hour! http://en.wikipedia.org/wiki/The_Argument_Sketch -- Steven From wxjmfauth at gmail.com Fri Jul 26 09:19:42 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Fri, 26 Jul 2013 06:19:42 -0700 (PDT) Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <606b75ca-e1eb-4a69-a23d-6f0372004114@googlegroups.com> Le jeudi 25 juillet 2013 22:45:38 UTC+2, Ian a ?crit?: > On Thu, Jul 25, 2013 at 12:18 PM, Steven D'Aprano > > wrote: > > > On Fri, 26 Jul 2013 01:36:07 +1000, Chris Angelico wrote: > > > > > >> On Fri, Jul 26, 2013 at 1:26 AM, Steven D'Aprano > > >> wrote: > > >>> On Thu, 25 Jul 2013 14:36:25 +0100, Jeremy Sanders wrote: > > >>>> "To conserve memory, Emacs does not hold fixed-length 22-bit numbers > > >>>> that are codepoints of text characters within buffers and strings. > > >>>> Rather, Emacs uses a variable-length internal representation of > > >>>> characters, that stores each character as a sequence of 1 to 5 8-bit > > >>>> bytes, depending on the magnitude of its codepoint[1]. For example, > > >>>> any ASCII character takes up only 1 byte, a Latin-1 character takes up > > >>>> 2 bytes, etc. We call this representation of text multibyte. > > >>> > > >>> Well, you've just proven what Vim users have always suspected: Emacs > > >>> doesn't really exist. > > >> > > >> ... lolwut? > > > > > > > > > JMF has explained that it is impossible, impossible I say!, to write an > > > editor using a flexible string representation. Since Emacs uses such a > > > flexible string representation, Emacs is impossible, and therefore Emacs > > > doesn't exist. > > > > > > QED. > > > > Except that the described representation used by Emacs is a variant of > > UTF-8, not an FSR. It doesn't have three different possible encodings > > for the letter 'a' depending on what other characters happen to be in > > the string. > > > > As I understand it, jfm would be perfectly happy if Python used UTF-8 > > (or presumably the Emacs variant) as its internal string > > representation. ------ And emacs it probably working smoothly. Your comment summarized all this stuff very correctly and very shortly. utf8/16/32? I do not care. There are all working correctly, smoothly and efficiently. In fact, these utf's are already doing correctly, what this FSR is doing in a wrong way. My preference? utf32. Why? It is the most simple and consequently performing choice. I'm not a narrow minded ascii user. (I do not pretend to belong to those who are solving the quadrature of the circle, I pretend to belong to those who know, the quadrature of the circle is not solvable). Note: text processing tools or tools that have to process characters ? and the tools to build these tools ? are all moving to utf32, if not already done. There are technical reasons behind this, which are going beyond the pure raw unicode. There are however still 100% Unicode compliant. jmf From torriem at gmail.com Thu Jul 25 23:09:34 2013 From: torriem at gmail.com (Michael Torrie) Date: Thu, 25 Jul 2013 21:09:34 -0600 Subject: RE Module Performance In-Reply-To: <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51F1E86E.2080407@gmail.com> On 07/25/2013 11:18 AM, Steven D'Aprano wrote: > JMF has explained that it is impossible, impossible I say!, to write an > editor using a flexible string representation. Since Emacs uses such a > flexible string representation, Emacs is impossible, and therefore Emacs > doesn't exist. Now I'm even more confused. He once pointed to Go as an example of how unicode should be done in a language. yet Go uses UTF-8 I think. But I don't think UTF-8 is what JMF refers to as "flexible string representation." FSR does use 1,2 or 4 bytes per character, but each character in the string uses the same width. That's different from UTF-8 or UTF-16, which is variable width per character. From wxjmfauth at gmail.com Fri Jul 26 09:21:10 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Fri, 26 Jul 2013 06:21:10 -0700 (PDT) Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> Le vendredi 26 juillet 2013 05:09:34 UTC+2, Michael Torrie a ?crit?: > On 07/25/2013 11:18 AM, Steven D'Aprano wrote: > > > JMF has explained that it is impossible, impossible I say!, to write an > > > editor using a flexible string representation. Since Emacs uses such a > > > flexible string representation, Emacs is impossible, and therefore Emacs > > > doesn't exist. > > > > Now I'm even more confused. He once pointed to Go as an example of how > > unicode should be done in a language. yet Go uses UTF-8 I think. > > > > But I don't think UTF-8 is what JMF refers to as "flexible string > > representation." FSR does use 1,2 or 4 bytes per character, but each > > character in the string uses the same width. That's different from > > UTF-8 or UTF-16, which is variable width per character. ----- >>> sys.getsizeof('??') - sys.getsizeof('?') I have already explained / commented this. -------- Hint: To understand Unicode (and every coding scheme), you should understand "utf". The how and the *why*. jmf From antoon.pardon at rece.vub.ac.be Fri Jul 26 16:38:59 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Fri, 26 Jul 2013 22:38:59 +0200 Subject: RE Module Performance In-Reply-To: <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> Message-ID: <51F2DE63.8050408@rece.vub.ac.be> Op 26-07-13 15:21, wxjmfauth at gmail.com schreef: > > Hint: To understand Unicode (and every coding scheme), you should > understand "utf". The how and the *why*. No you don't. You are mixing the information with how the information is coded. utf is like base64, a way of coding the information that is usefull for storage or transfer. But once you have decode the byte stream, you no longer need any understanding of base64 to process your information. Likewise, once you have decode the bytestream into uniocde information you don't need knowledge of utf to process unicode strings. -- Antoon Pardon From torriem at gmail.com Fri Jul 26 22:05:03 2013 From: torriem at gmail.com (Michael Torrie) Date: Fri, 26 Jul 2013 20:05:03 -0600 Subject: RE Module Performance In-Reply-To: <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> Message-ID: <51F32ACF.4080603@gmail.com> On 07/26/2013 07:21 AM, wxjmfauth at gmail.com wrote: >>>> sys.getsizeof('??') - sys.getsizeof('?') > > I have already explained / commented this. Maybe it got lost in translation, but I don't understand your point with that. > Hint: To understand Unicode (and every coding scheme), you should > understand "utf". The how and the *why*. Hmm, so if python used utf-8 internally to represent unicode strings would not that punish *all* users (not just non-ascii users) since searching a string for a certain character position requires an O(n) operation? UTF-32 I could see (and indeed that's essentially what FSR uses when necessary does it not?), but not utf-8 or utf-16. From wxjmfauth at gmail.com Sat Jul 27 14:21:14 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Sat, 27 Jul 2013 11:21:14 -0700 (PDT) Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> Message-ID: Le samedi 27 juillet 2013 04:05:03 UTC+2, Michael Torrie a ?crit?: > On 07/26/2013 07:21 AM, wxjmfauth at gmail.com wrote: > > >>>> sys.getsizeof('??') - sys.getsizeof('?') > > > > > > I have already explained / commented this. > > > > Maybe it got lost in translation, but I don't understand your point with > > that. > > > > > Hint: To understand Unicode (and every coding scheme), you should > > > understand "utf". The how and the *why*. > > > > Hmm, so if python used utf-8 internally to represent unicode strings > > would not that punish *all* users (not just non-ascii users) since > > searching a string for a certain character position requires an O(n) > > operation? UTF-32 I could see (and indeed that's essentially what FSR > > uses when necessary does it not?), but not utf-8 or utf-16. ------ Did you read my previous link? Unicode Character Encoding Model. Did you understand it? Unicode only - No FSR (I skip some points and I still attempt to be still correct.) Unicode is a four-steps process. [ {unique set of characters} --> {unique set of code points, the "labels"} --> {unique set of encoded code points} ] --> implementation (bytes) First point to notice. "pure unicode", [...], is different from the "implementation". *This is a deliberate choice*. The critical step is the path {unique set of characters} ---> {unique set of encoded code points} in such a way so that the implementation can "work comfortably" with this *unique* set of encoded code points. Conceptualy, the implementation works with an unique set of "already prepared encoded code points". This is a very critical step. To explain it in a dirty way: in the above chain, this problem is "already" eliminated and solved. Like a byte/char coding schemes where this step is a no-op. Now, and if you wish this is a seperated/different problem. To create this unique set of encoded code points, "Unicode" uses these "utf(s)". I repeat again, a confusing name, for the process and the result of the process. (I neglect ucs). What are these? Chunks of bits, group of 8/16/32 bits, words. It is up to the implementation to convert these sequences of bits into bytes, ***if you wish to convert these in bytes!***. Suprise! Why not putting two of the 32-bits words in a 64-bits "machine"? (see golang / rune / int32). Back to utf. utfs are not only elements of a unique set of encoded code points. They have an interesting feature. Each "utf chunk" holds intrisically the character (in fact the code point) it is supposed to represent. In utf-32, the obvious case, it is just the code point. In utf-8, that's the first chunk which helps and utf-16 is a mixed case (utf-8 / utf-32). In other words, in an implementation using bytes, for any pointer position it is always possible to find the corresponding encoded code point and from this the corresponding character without any "programmed" information. See my editor example, how to find the char under the caret? In fact, a silly example, how can the caret can be positioned or moved, if the underlying corresponding encoded code point can not be dicerned! Next step and one another separated problem. Why all these utf versions? It is always the same story. Some prefer the universality (utf-32) and some prefer, well, some kind of conservatism. utf-8 is more complicated, it demands more work and logically, in an expected way, some performance regression. utf-8 is more suited to produce bytes, utf16/32 for internal processing. utf-8 had no choice to lose the indexing. And so on. Fact: all these coding schemes are working with a unique set of encoded code points (suprise again, it's like byte string!). The loss of performance of utf-8 is very minimal compared to the loss of performance one can get compare to a multiple coding scheme. This kind of work has been done, and if my informations are correct, even by the creators of utf-8. (There are sometimes good scientists). There are plenty of advantages in using utf instead of something else and advantages in other fields than just the pure coding. utf-16/32 schemes have the advantages to ditch ascii for ever. The ascii concept is no more existing. One should also understand that all this stuff has not been created from scratch. It was a balance between existing technologies. MS sticked with the idea, no more ascii, let's use ucs-2 and the *x world breaks the unicode adoption as possible. utf-8 is one of the compromise for the adoption of Unicode. Retrospectivly, a not so good compromise. Computer scientists are funny scientists. They do love to solve the problems they created themselves. ----- Quickly. sys.getsizeof() at the light of what I explained. 1) As this FSR works with multiple encoding, it has to keep track of the encoding. it puts is in the overhead of str class (overhead = real overhead + encoding). In such a absurd way, that a >>> sys.getsizeof('?') 40 needs 14 bytes more than a >>> sys.getsizeof('z') 26 You may vary the length of the str. The problem is still here. Not bad for a coding scheme. 2) Take a look at this. Get rid of the overhead. >>> sys.getsizeof('b'*1000000 + 'c') 1000026 >>> sys.getsizeof('b'*1000000 + '?') 2000040 What does it mean? It means that Python has to reencode a str every time it is necessary because it works with multiple codings. This FSR is not even a copy of the utf-8. >>> len(('b'*1000000 + '?').encode('utf-8')) 1000003 utf-8 or any (utf) never need and never spend their time in reencoding. 3) Unicode compliance. We know retrospectively, latin-1, is was a bad choice. Unusable for 17 European languages. Believe of not. 20 years of Unicode of incubation is not long enough to learn it. When discussing once with a French Python core dev, one with commit access, he did not know one can not use latin-1 for the French language! BTW, I proposed to the French devs, to test the FST with the set of characters, recognized by the "Imprimerie Nationale", some kind of the legal French authority regarding characters and typography. Never heared about it. Of course, I dit it. In short FSR = bad performance + bad memory mangement + non unicode compliance. Good point. FSR, nice tool for those who wish to teach Unicode. It is not every day, one has such an opportunity. --------- I'm practicaly no more programming, writing applications. I'm still active and observing since a decade and plus all this unicode world, languages (go, c#, Python, Ruby), text processing systems (esp. Unicode TeX engines) and font technology. Very, very interesting. jmf From ian.g.kelly at gmail.com Sat Jul 27 23:53:22 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Sat, 27 Jul 2013 21:53:22 -0600 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> Message-ID: On Sat, Jul 27, 2013 at 12:21 PM, wrote: > Back to utf. utfs are not only elements of a unique set of encoded > code points. They have an interesting feature. Each "utf chunk" > holds intrisically the character (in fact the code point) it is > supposed to represent. In utf-32, the obvious case, it is just > the code point. In utf-8, that's the first chunk which helps and > utf-16 is a mixed case (utf-8 / utf-32). In other words, in an > implementation using bytes, for any pointer position it is always > possible to find the corresponding encoded code point and from this > the corresponding character without any "programmed" information. See > my editor example, how to find the char under the caret? In fact, > a silly example, how can the caret can be positioned or moved, if > the underlying corresponding encoded code point can not be > dicerned! Yes, given a pointer location into a utf-8 or utf-16 string, it is easy to determine the identity of the code point at that location. But this is not often a useful operation, save for resynchronization in the case that the string data is corrupted. The caret of an editor does not conceptually correspond to a pointer location, but to a character index. Given a particular character index (e.g. 127504), an editor must be able to determine the identity and/or the memory location of the character at that index, and for UTF-8 and UTF-16 without an auxiliary data structure that is a O(n) operation. > 2) Take a look at this. Get rid of the overhead. > >>>> sys.getsizeof('b'*1000000 + 'c') > 1000026 >>>> sys.getsizeof('b'*1000000 + '?') > 2000040 > > What does it mean? It means that Python has to > reencode a str every time it is necessary because > it works with multiple codings. Large strings in practical usage do not need to be resized like this often. Python 3.3 has been in production use for months now, and you still have yet to produce any real-world application code that demonstrates a performance regression. If there is no real-world regression, then there is no problem. > 3) Unicode compliance. We know retrospectively, latin-1, > is was a bad choice. Unusable for 17 European languages. > Believe of not. 20 years of Unicode of incubation is not > long enough to learn it. When discussing once with a French > Python core dev, one with commit access, he did not know one > can not use latin-1 for the French language! Probably because for many French strings, one can. As far as I am aware, the only characters that are missing from Latin-1 are the Euro sign (an unfortunate victim of history), the ligature ? (I have no doubt that many users just type oe anyway), and the rare capital ? (the miniscule version is present in Latin-1). All French strings that are fortunate enough to be absent these characters can be represented in Latin-1 and so will have a 1-byte width in the FSR. From antoon.pardon at rece.vub.ac.be Sun Jul 28 04:45:25 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Sun, 28 Jul 2013 10:45:25 +0200 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> Message-ID: <51F4DA25.3030304@rece.vub.ac.be> Op 27-07-13 20:21, wxjmfauth at gmail.com schreef: > Quickly. sys.getsizeof() at the light of what I explained. > > 1) As this FSR works with multiple encoding, it has to keep > track of the encoding. it puts is in the overhead of str > class (overhead = real overhead + encoding). In such > a absurd way, that a > >>>> sys.getsizeof('?') > 40 > > needs 14 bytes more than a > >>>> sys.getsizeof('z') > 26 > > You may vary the length of the str. The problem is > still here. Not bad for a coding scheme. > > 2) Take a look at this. Get rid of the overhead. > >>>> sys.getsizeof('b'*1000000 + 'c') > 1000026 >>>> sys.getsizeof('b'*1000000 + '?') > 2000040 > > What does it mean? It means that Python has to > reencode a str every time it is necessary because > it works with multiple codings. So? The same effect can be seen with other datatypes. >>> nr = 32767 >>> sys.getsizeof(nr) 14 >>> nr += 1 >>> sys.getsizeof(nr) 16 > > This FSR is not even a copy of the utf-8. >>>> len(('b'*1000000 + '?').encode('utf-8')) > 1000003 Why should it be? Why should a unicode string be a copy of its utf-8 encoding? That makes as much sense as expecting that a number would be a copy of its string reprensentation. > > utf-8 or any (utf) never need and never spend their time > in reencoding. So? That python sometimes needs to do some kind of background processing is not a problem, whether it is garbage collection, allocating more memory, shufling around data blocks or reencoding a string, that doesn't matter. If you've got a real world example where one of those things noticeably slows your program down or makes the program behave faulty then you have something that is worthy of attention. Until then you are merely harboring a pet peeve. -- Antoon Pardon From joshua at landau.ws Sun Jul 28 14:19:01 2013 From: joshua at landau.ws (Joshua Landau) Date: Sun, 28 Jul 2013 19:19:01 +0100 Subject: RE Module Performance In-Reply-To: <51F4DA25.3030304@rece.vub.ac.be> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <51F4DA25.3030304@rece.vub.ac.be> Message-ID: On 28 July 2013 09:45, Antoon Pardon wrote: > Op 27-07-13 20:21, wxjmfauth at gmail.com schreef: > >> utf-8 or any (utf) never need and never spend their time >> in reencoding. >> > > So? That python sometimes needs to do some kind of background > processing is not a problem, whether it is garbage collection, > allocating more memory, shufling around data blocks or reencoding a > string, that doesn't matter. If you've got a real world example where > one of those things noticeably slows your program down or makes the > program behave faulty then you have something that is worthy of > attention. Somewhat off topic, but befitting of the triviality of this thread, do I understand correctly that you are saying garbage collection never causes any noticeable slowdown in real-world circumstances? That's not remotely true. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Jul 28 14:29:54 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 28 Jul 2013 19:29:54 +0100 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <51F4DA25.3030304@rece.vub.ac.be> Message-ID: On Sun, Jul 28, 2013 at 7:19 PM, Joshua Landau wrote: > On 28 July 2013 09:45, Antoon Pardon wrote: >> >> Op 27-07-13 20:21, wxjmfauth at gmail.com schreef: >>> >>> utf-8 or any (utf) never need and never spend their time >>> in reencoding. >> >> >> So? That python sometimes needs to do some kind of background >> processing is not a problem, whether it is garbage collection, >> allocating more memory, shufling around data blocks or reencoding a >> string, that doesn't matter. If you've got a real world example where >> one of those things noticeably slows your program down or makes the >> program behave faulty then you have something that is worthy of >> attention. > > > Somewhat off topic, but befitting of the triviality of this thread, do I > understand correctly that you are saying garbage collection never causes any > noticeable slowdown in real-world circumstances? That's not remotely true. If it's done properly, garbage collection shouldn't hurt the *overall* performance of the app; most of the issues with GC timing are when one operation gets unexpectedly delayed for a GC run (making performance measurement hard, and such). It should certainly never cause your program to behave faultily, though I have seen cases where the GC run appears to cause the program to crash - something like this: some_string = buggy_call() ... gc() ... print(some_string) The buggy call mucked up the reference count, so the gc run actually wiped the string from memory - resulting in a segfault on next usage. But the GC wasn't at fault, the original call was. (Which, btw, was quite a debugging search, especially since the function in question wasn't my code.) ChrisA From tjreedy at udel.edu Sun Jul 28 15:06:34 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 28 Jul 2013 15:06:34 -0400 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <51F4DA25.3030304@rece.vub.ac.be> Message-ID: On 7/28/2013 2:29 PM, Chris Angelico wrote: > On Sun, Jul 28, 2013 at 7:19 PM, Joshua Landau wrote: >> Somewhat off topic, but befitting of the triviality of this thread, do I >> understand correctly that you are saying garbage collection never causes any >> noticeable slowdown in real-world circumstances? That's not remotely true. > > If it's done properly, garbage collection shouldn't hurt the *overall* > performance of the app; There are situations, some discussed on this list, where doing gc 'right' means turning off the cycle garbage collector. As I remember, an example is creating a list of a million tuples, which otherwise triggers a lot of useless background bookkeeping. The cyclic gc is tuned for 'normal' use patterns. -- Terry Jan Reedy From joshua at landau.ws Sun Jul 28 18:14:53 2013 From: joshua at landau.ws (Joshua Landau) Date: Sun, 28 Jul 2013 23:14:53 +0100 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <51F4DA25.3030304@rece.vub.ac.be> Message-ID: On 28 July 2013 19:29, Chris Angelico wrote: > On Sun, Jul 28, 2013 at 7:19 PM, Joshua Landau wrote: > > On 28 July 2013 09:45, Antoon Pardon > wrote: > >> > >> Op 27-07-13 20:21, wxjmfauth at gmail.com schreef: > >>> > >>> utf-8 or any (utf) never need and never spend their time > >>> in reencoding. > >> > >> > >> So? That python sometimes needs to do some kind of background > >> processing is not a problem, whether it is garbage collection, > >> allocating more memory, shufling around data blocks or reencoding a > >> string, that doesn't matter. If you've got a real world example where > >> one of those things noticeably slows your program down or makes the > >> program behave faulty then you have something that is worthy of > >> attention. > > > > > > Somewhat off topic, but befitting of the triviality of this thread, do I > > understand correctly that you are saying garbage collection never causes > any > > noticeable slowdown in real-world circumstances? That's not remotely > true. > > If it's done properly, garbage collection shouldn't hurt the *overall* > performance of the app; most of the issues with GC timing are when one > operation gets unexpectedly delayed for a GC run (making performance > measurement hard, and such). It should certainly never cause your > program to behave faultily, though I have seen cases where the GC run > appears to cause the program to crash - something like this: > > some_string = buggy_call() > ... > gc() > ... > print(some_string) > > The buggy call mucked up the reference count, so the gc run actually > wiped the string from memory - resulting in a segfault on next usage. > But the GC wasn't at fault, the original call was. (Which, btw, was > quite a debugging search, especially since the function in question > wasn't my code.) > GC does have sometimes severe impact in memory-constrained environments, though. See http://sealedabstract.com/rants/why-mobile-web-apps-are-slow/, about half-way down, specifically http://sealedabstract.com/wp-content/uploads/2013/05/Screen-Shot-2013-05-14-at-10.15.29-PM.png . The best verification of these graphs I could find was https://blog.mozilla.org/nnethercote/category/garbage-collection/, although it's not immediately clear in Chrome's and Opera's case mainly due to none of the benchmarks pushing memory usage significantly. I also don't quite agree with the first post (sealedabstract) because I get by *fine* on 2GB memory, so I don't see why you can't on a phone. Maybe IOS is just really heavy. Nonetheless, the benchmarks aren't lying. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Jul 28 19:07:13 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 29 Jul 2013 00:07:13 +0100 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <51F4DA25.3030304@rece.vub.ac.be> Message-ID: On Sun, Jul 28, 2013 at 11:14 PM, Joshua Landau wrote: > GC does have sometimes severe impact in memory-constrained environments, > though. See http://sealedabstract.com/rants/why-mobile-web-apps-are-slow/, > about half-way down, specifically > http://sealedabstract.com/wp-content/uploads/2013/05/Screen-Shot-2013-05-14-at-10.15.29-PM.png. > > The best verification of these graphs I could find was > https://blog.mozilla.org/nnethercote/category/garbage-collection/, although > it's not immediately clear in Chrome's and Opera's case mainly due to none > of the benchmarks pushing memory usage significantly. > > I also don't quite agree with the first post (sealedabstract) because I get > by *fine* on 2GB memory, so I don't see why you can't on a phone. Maybe IOS > is just really heavy. Nonetheless, the benchmarks aren't lying. The ultimate in non-managed memory (the opposite of a GC) would have to be the assembly language programming I did in my earlier days, firing up DEBUG.EXE and writing a .COM file that lived inside a single 64KB segment for everything (256-byte Program Segment Prefix, then code, then initialized data, then uninitialized data and stack), crashing the computer with remarkable ease. Everything "higher level" than that (even malloc/free) has its conveniences and its costs, usually memory wastage. If you malloc random-sized blocks, free them at random, and ensure that your total allocated size stays below some limit, you'll still eventually run yourself out of memory. This is unsurprising. The only question is, how bad is the wastage and how much time gets devoted to it? ChrisA From antoon.pardon at rece.vub.ac.be Sun Jul 28 14:51:26 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Sun, 28 Jul 2013 20:51:26 +0200 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <51F4DA25.3030304@rece.vub.ac.be> Message-ID: <51F5682E.3090400@rece.vub.ac.be> Op 28-07-13 20:19, Joshua Landau schreef: > On 28 July 2013 09:45, Antoon Pardon > wrote: > > Op 27-07-13 20:21, wxjmfauth at gmail.com > schreef: > > utf-8 or any (utf) never need and never spend their time > in reencoding. > > > So? That python sometimes needs to do some kind of background > processing is not a problem, whether it is garbage collection, > allocating more memory, shufling around data blocks or reencoding a > string, that doesn't matter. If you've got a real world example where > one of those things noticeably slows your program down or makes the > program behave faulty then you have something that is worthy of > attention. > > > Somewhat off topic, but befitting of the triviality of this thread, do I > understand correctly that you are saying garbage collection never causes > any noticeable slowdown in real-world circumstances? That's not remotely > true. No that is not what I am saying. But if jmf would be complaining about garbage collection in an analog way as he is complaining about the FSR, he wouldn't be complaining about real-world circumstances but about theorectical possibilities and micro bench marks. In those circunstances the "garbage collection problem" wouldn't be worthy of attention much. -- Antoon Pardon From torriem at gmail.com Sun Jul 28 11:52:47 2013 From: torriem at gmail.com (Michael Torrie) Date: Sun, 28 Jul 2013 09:52:47 -0600 Subject: FSR and unicode compliance - was Re: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> Message-ID: <51F53E4F.8080104@gmail.com> On 07/27/2013 12:21 PM, wxjmfauth at gmail.com wrote: > Good point. FSR, nice tool for those who wish to teach > Unicode. It is not every day, one has such an opportunity. I had a long e-mail composed, but decided to chop it down, but still too long. so I ditched a lot of the context, which jmf also seems to do. Apologies. 1. FSR *is* UTF-32 so it is as unicode compliant as UTF-32, since UTF-32 is an official encoding. FSR only differs from UTF-32 in that the padding zeros are stripped off such that it is stored in the most compact form that can handle all the characters in string, which is always known at string creation time. Now you can argue many things, but to say FSR is not unicode compliant is quite a stretch! What unicode entities or characters cannot be stored in strings using FSR? What sequences of bytes in FSR result in invalid Unicode entities? 2. strings in Python *never change*. They are immutable. The + operator always copies strings character by character into a new string object, even if Python had used UTF-8 internally. If you're doing a lot of string concatenations, perhaps you're using the wrong data type. A byte buffer might be better for you, where you can stuff utf-8 sequences into it to your heart's content. 3. UTF-8 and UTF-16 encodings, being variable width encodings, mean that slicing a string would be very very slow, and that's unacceptable for the use cases of python strings. I'm assuming you understand big O notation, as you talk of experience in many languages over the years. FSR and UTF-32 both are O(1) for slicing and lookups. UTF-8, 16 and any variable-width encoding are always O(n). A lot slower! 4. Unicode is, well, unicode. You seem to hop all over the place from talking about code points to bytes to bits, using them all interchangeably. And now you seem to be claiming that a particular byte encoding standard is by definition unicode (UTF-8). Or at least that's how it sounds. And also claim FSR is not compliant with unicode standards, which appears to me to be completely false. Is my understanding of these things wrong? From rosuav at gmail.com Sun Jul 28 13:03:09 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 28 Jul 2013 18:03:09 +0100 Subject: FSR and unicode compliance - was Re: RE Module Performance In-Reply-To: <51F53E4F.8080104@gmail.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <51F53E4F.8080104@gmail.com> Message-ID: On Sun, Jul 28, 2013 at 4:52 PM, Michael Torrie wrote: > Is my understanding of these things wrong? No, your understanding of those matters is fine. There's just one area you seem to be misunderstanding; you appear to think that jmf actually cares about logical argument. I gave up on that theory a long time ago, and now I respond for the benefit of those reading, rather than jmf himself. I've also given up on trying to figure out what he actually wants; the nearest I can come up with is that he's King Gama-esque - that he just wants to complain. ChrisA From tjreedy at udel.edu Sun Jul 28 13:36:00 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 28 Jul 2013 13:36:00 -0400 Subject: FSR and unicode compliance - was Re: RE Module Performance In-Reply-To: <51F53E4F.8080104@gmail.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <51F53E4F.8080104@gmail.com> Message-ID: On 7/28/2013 11:52 AM, Michael Torrie wrote: > > 3. UTF-8 and UTF-16 encodings, being variable width encodings, mean that > slicing a string would be very very slow, Not necessarily so. See below. > and that's unacceptable for > the use cases of python strings. I'm assuming you understand big O > notation, as you talk of experience in many languages over the years. > FSR and UTF-32 both are O(1) for slicing and lookups. Slicing is at least O(m) where m is the length of the slice. > UTF-8, 16 and any variable-width encoding are always O(n).\ I posted about a week ago, in response to Chris A., a method by which lookup for UTF-16 can be made O(log2 k), or perhaps more accurately, O(1+log2(k+1)), where k is the number of non-BMP chars in the string. This uses an auxiliary array of k ints. An auxiliary array of n ints would make UFT-16 lookup O(1), but then one is using more space than with UFT-32. Similar comments apply to UTF-8. The unicode standard says that a single strings should use exactly one coding scheme. It does *not* say that all strings in an application must use the same scheme. I just rechecked a few days ago. It also does not say that an application cannot associate additional data with a string to make processing of the string easier. -- Terry Jan Reedy From rosuav at gmail.com Sun Jul 28 14:03:48 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 28 Jul 2013 19:03:48 +0100 Subject: FSR and unicode compliance - was Re: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <51F53E4F.8080104@gmail.com> Message-ID: On Sun, Jul 28, 2013 at 6:36 PM, Terry Reedy wrote: > I posted about a week ago, in response to Chris A., a method by which lookup > for UTF-16 can be made O(log2 k), or perhaps more accurately, > O(1+log2(k+1)), where k is the number of non-BMP chars in the string. > Which is an optimization choice that favours strings containing very few non-BMP characters. To justify the extra complexity of out-of-band storage, you would need to be working with almost exclusively the BMP. That would drastically improve jmf's microbenchmarks which do exactly that, but it would penalize strings that are almost exclusively higher-codepoint characters. Its quality, then, would be based on a major survey of string usage: are there enough strings with mostly-BMP-but-a-few-SMP? Bearing in mind that pure BMP is handled better by PEP 393, so this is only of value when there are actually those mixed strings. ChrisA From wxjmfauth at gmail.com Mon Jul 29 09:36:16 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Mon, 29 Jul 2013 06:36:16 -0700 (PDT) Subject: FSR and unicode compliance - was Re: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <51F53E4F.8080104@gmail.com> Message-ID: <11a47ae7-27af-46dd-ad32-fa851ead45d5@googlegroups.com> Le dimanche 28 juillet 2013 19:36:00 UTC+2, Terry Reedy a ?crit?: > On 7/28/2013 11:52 AM, Michael Torrie wrote: > > > > > > 3. UTF-8 and UTF-16 encodings, being variable width encodings, mean that > > > slicing a string would be very very slow, > > > > Not necessarily so. See below. > > > > > and that's unacceptable for > > > the use cases of python strings. I'm assuming you understand big O > > > notation, as you talk of experience in many languages over the years. > > > FSR and UTF-32 both are O(1) for slicing and lookups. > > > > Slicing is at least O(m) where m is the length of the slice. > > > > > UTF-8, 16 and any variable-width encoding are always O(n).\ > > > > I posted about a week ago, in response to Chris A., a method by which > > lookup for UTF-16 can be made O(log2 k), or perhaps more accurately, > > O(1+log2(k+1)), where k is the number of non-BMP chars in the string. > > > > This uses an auxiliary array of k ints. An auxiliary array of n ints > > would make UFT-16 lookup O(1), but then one is using more space than > > with UFT-32. Similar comments apply to UTF-8. > > > > The unicode standard says that a single strings should use exactly one > > coding scheme. It does *not* say that all strings in an application must > > use the same scheme. I just rechecked a few days ago. It also does not > > say that an application cannot associate additional data with a string > > to make processing of the string easier. > > > > -- > > Terry Jan Reedy To my knowledge, the Unicode doc always speak about the misc. utf* coding schemes in an "exclusive or" way. Having multiple encoded strings is one thing. Manipulating multiple encoded strings is something else. Maybe the mistake was to not emphasize the fact that one has to work with a unique set of encoded code points (utf-8 or utf-16 or utf-32) because it was considered, as to obvious one can not work properly with multiple coding schemes. You are also right in saying " ...application cannot associate additional data...". The doc does not specify it either. It is superfleous. jmf From wxjmfauth at gmail.com Sun Jul 28 14:13:29 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Sun, 28 Jul 2013 11:13:29 -0700 (PDT) Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> Message-ID: <4117e08f-941a-42d5-87b6-09e66f8c7b60@googlegroups.com> Le dimanche 28 juillet 2013 05:53:22 UTC+2, Ian a ?crit?: > On Sat, Jul 27, 2013 at 12:21 PM, wrote: > > > Back to utf. utfs are not only elements of a unique set of encoded > > > code points. They have an interesting feature. Each "utf chunk" > > > holds intrisically the character (in fact the code point) it is > > > supposed to represent. In utf-32, the obvious case, it is just > > > the code point. In utf-8, that's the first chunk which helps and > > > utf-16 is a mixed case (utf-8 / utf-32). In other words, in an > > > implementation using bytes, for any pointer position it is always > > > possible to find the corresponding encoded code point and from this > > > the corresponding character without any "programmed" information. See > > > my editor example, how to find the char under the caret? In fact, > > > a silly example, how can the caret can be positioned or moved, if > > > the underlying corresponding encoded code point can not be > > > dicerned! > > > > Yes, given a pointer location into a utf-8 or utf-16 string, it is > > easy to determine the identity of the code point at that location. > > But this is not often a useful operation, save for resynchronization > > in the case that the string data is corrupted. The caret of an editor > > does not conceptually correspond to a pointer location, but to a > > character index. Given a particular character index (e.g. 127504), an > > editor must be able to determine the identity and/or the memory > > location of the character at that index, and for UTF-8 and UTF-16 > > without an auxiliary data structure that is a O(n) operation. > > > > > 2) Take a look at this. Get rid of the overhead. > > > > > >>>> sys.getsizeof('b'*1000000 + 'c') > > > 1000026 > > >>>> sys.getsizeof('b'*1000000 + '?') > > > 2000040 > > > > > > What does it mean? It means that Python has to > > > reencode a str every time it is necessary because > > > it works with multiple codings. > > > > Large strings in practical usage do not need to be resized like this > > often. Python 3.3 has been in production use for months now, and you > > still have yet to produce any real-world application code that > > demonstrates a performance regression. If there is no real-world > > regression, then there is no problem. > > > > > 3) Unicode compliance. We know retrospectively, latin-1, > > > is was a bad choice. Unusable for 17 European languages. > > > Believe of not. 20 years of Unicode of incubation is not > > > long enough to learn it. When discussing once with a French > > > Python core dev, one with commit access, he did not know one > > > can not use latin-1 for the French language! > > > > Probably because for many French strings, one can. As far as I am > > aware, the only characters that are missing from Latin-1 are the Euro > > sign (an unfortunate victim of history), the ligature ? (I have no > > doubt that many users just type oe anyway), and the rare capital ? > > (the miniscule version is present in Latin-1). All French strings > > that are fortunate enough to be absent these characters can be > > represented in Latin-1 and so will have a 1-byte width in the FSR. ------ latin-1? that's not even truth. >>> sys.getsizeof('a') 26 >>> sys.getsizeof('?') 38 >>> sys.getsizeof('aa') 27 >>> sys.getsizeof('a?') 39 jmf From python at mrabarnett.plus.com Sun Jul 28 15:04:56 2013 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 28 Jul 2013 20:04:56 +0100 Subject: RE Module Performance In-Reply-To: <4117e08f-941a-42d5-87b6-09e66f8c7b60@googlegroups.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <4117e08f-941a-42d5-87b6-09e66f8c7b60@googlegroups.com> Message-ID: <51F56B58.5000100@mrabarnett.plus.com> On 28/07/2013 19:13, wxjmfauth at gmail.com wrote: > Le dimanche 28 juillet 2013 05:53:22 UTC+2, Ian a ?crit : >> On Sat, Jul 27, 2013 at 12:21 PM, wrote: >> >> > Back to utf. utfs are not only elements of a unique set of encoded >> >> > code points. They have an interesting feature. Each "utf chunk" >> >> > holds intrisically the character (in fact the code point) it is >> >> > supposed to represent. In utf-32, the obvious case, it is just >> >> > the code point. In utf-8, that's the first chunk which helps and >> >> > utf-16 is a mixed case (utf-8 / utf-32). In other words, in an >> >> > implementation using bytes, for any pointer position it is always >> >> > possible to find the corresponding encoded code point and from this >> >> > the corresponding character without any "programmed" information. See >> >> > my editor example, how to find the char under the caret? In fact, >> >> > a silly example, how can the caret can be positioned or moved, if >> >> > the underlying corresponding encoded code point can not be >> >> > dicerned! >> >> >> >> Yes, given a pointer location into a utf-8 or utf-16 string, it is >> >> easy to determine the identity of the code point at that location. >> >> But this is not often a useful operation, save for resynchronization >> >> in the case that the string data is corrupted. The caret of an editor >> >> does not conceptually correspond to a pointer location, but to a >> >> character index. Given a particular character index (e.g. 127504), an >> >> editor must be able to determine the identity and/or the memory >> >> location of the character at that index, and for UTF-8 and UTF-16 >> >> without an auxiliary data structure that is a O(n) operation. >> >> >> >> > 2) Take a look at this. Get rid of the overhead. >> >> > >> >> >>>> sys.getsizeof('b'*1000000 + 'c') >> >> > 1000026 >> >> >>>> sys.getsizeof('b'*1000000 + '?') >> >> > 2000040 >> >> > >> >> > What does it mean? It means that Python has to >> >> > reencode a str every time it is necessary because >> >> > it works with multiple codings. >> >> >> >> Large strings in practical usage do not need to be resized like this >> >> often. Python 3.3 has been in production use for months now, and you >> >> still have yet to produce any real-world application code that >> >> demonstrates a performance regression. If there is no real-world >> >> regression, then there is no problem. >> >> >> >> > 3) Unicode compliance. We know retrospectively, latin-1, >> >> > is was a bad choice. Unusable for 17 European languages. >> >> > Believe of not. 20 years of Unicode of incubation is not >> >> > long enough to learn it. When discussing once with a French >> >> > Python core dev, one with commit access, he did not know one >> >> > can not use latin-1 for the French language! >> >> >> >> Probably because for many French strings, one can. As far as I am >> >> aware, the only characters that are missing from Latin-1 are the Euro >> >> sign (an unfortunate victim of history), the ligature ? (I have no >> >> doubt that many users just type oe anyway), and the rare capital ? >> >> (the miniscule version is present in Latin-1). All French strings >> >> that are fortunate enough to be absent these characters can be >> >> represented in Latin-1 and so will have a 1-byte width in the FSR. > > ------ > > latin-1? that's not even truth. > >>>> sys.getsizeof('a') > 26 >>>> sys.getsizeof('?') > 38 >>>> sys.getsizeof('aa') > 27 >>>> sys.getsizeof('a?') > 39 > >>> sys.getsizeof('aa') - sys.getsizeof('a') 1 One byte per codepoint. >>> sys.getsizeof('??') - sys.getsizeof('?') 1 Also one byte per codepoint. >>> sys.getsizeof('?') - sys.getsizeof('a') 12 Clearly there's more going on here. FSR is an optimisation. You'll always be able to find some circumstances where an optimisation makes things worse, but what matters is the overall result. From wxjmfauth at gmail.com Sun Jul 28 15:30:58 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Sun, 28 Jul 2013 12:30:58 -0700 (PDT) Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <4117e08f-941a-42d5-87b6-09e66f8c7b60@googlegroups.com> Message-ID: <95b91473-b707-4288-860c-d02fda7af1ea@googlegroups.com> Le dimanche 28 juillet 2013 21:04:56 UTC+2, MRAB a ?crit?: > On 28/07/2013 19:13, wxjmfauth at gmail.com wrote: > > > Le dimanche 28 juillet 2013 05:53:22 UTC+2, Ian a ?crit : > > >> On Sat, Jul 27, 2013 at 12:21 PM, wrote: > > >> > > >> > Back to utf. utfs are not only elements of a unique set of encoded > > >> > > >> > code points. They have an interesting feature. Each "utf chunk" > > >> > > >> > holds intrisically the character (in fact the code point) it is > > >> > > >> > supposed to represent. In utf-32, the obvious case, it is just > > >> > > >> > the code point. In utf-8, that's the first chunk which helps and > > >> > > >> > utf-16 is a mixed case (utf-8 / utf-32). In other words, in an > > >> > > >> > implementation using bytes, for any pointer position it is always > > >> > > >> > possible to find the corresponding encoded code point and from this > > >> > > >> > the corresponding character without any "programmed" information. See > > >> > > >> > my editor example, how to find the char under the caret? In fact, > > >> > > >> > a silly example, how can the caret can be positioned or moved, if > > >> > > >> > the underlying corresponding encoded code point can not be > > >> > > >> > dicerned! > > >> > > >> > > >> > > >> Yes, given a pointer location into a utf-8 or utf-16 string, it is > > >> > > >> easy to determine the identity of the code point at that location. > > >> > > >> But this is not often a useful operation, save for resynchronization > > >> > > >> in the case that the string data is corrupted. The caret of an editor > > >> > > >> does not conceptually correspond to a pointer location, but to a > > >> > > >> character index. Given a particular character index (e.g. 127504), an > > >> > > >> editor must be able to determine the identity and/or the memory > > >> > > >> location of the character at that index, and for UTF-8 and UTF-16 > > >> > > >> without an auxiliary data structure that is a O(n) operation. > > >> > > >> > > >> > > >> > 2) Take a look at this. Get rid of the overhead. > > >> > > >> > > > >> > > >> >>>> sys.getsizeof('b'*1000000 + 'c') > > >> > > >> > 1000026 > > >> > > >> >>>> sys.getsizeof('b'*1000000 + '?') > > >> > > >> > 2000040 > > >> > > >> > > > >> > > >> > What does it mean? It means that Python has to > > >> > > >> > reencode a str every time it is necessary because > > >> > > >> > it works with multiple codings. > > >> > > >> > > >> > > >> Large strings in practical usage do not need to be resized like this > > >> > > >> often. Python 3.3 has been in production use for months now, and you > > >> > > >> still have yet to produce any real-world application code that > > >> > > >> demonstrates a performance regression. If there is no real-world > > >> > > >> regression, then there is no problem. > > >> > > >> > > >> > > >> > 3) Unicode compliance. We know retrospectively, latin-1, > > >> > > >> > is was a bad choice. Unusable for 17 European languages. > > >> > > >> > Believe of not. 20 years of Unicode of incubation is not > > >> > > >> > long enough to learn it. When discussing once with a French > > >> > > >> > Python core dev, one with commit access, he did not know one > > >> > > >> > can not use latin-1 for the French language! > > >> > > >> > > >> > > >> Probably because for many French strings, one can. As far as I am > > >> > > >> aware, the only characters that are missing from Latin-1 are the Euro > > >> > > >> sign (an unfortunate victim of history), the ligature ? (I have no > > >> > > >> doubt that many users just type oe anyway), and the rare capital ? > > >> > > >> (the miniscule version is present in Latin-1). All French strings > > >> > > >> that are fortunate enough to be absent these characters can be > > >> > > >> represented in Latin-1 and so will have a 1-byte width in the FSR. > > > > > > ------ > > > > > > latin-1? that's not even truth. > > > > > >>>> sys.getsizeof('a') > > > 26 > > >>>> sys.getsizeof('?') > > > 38 > > >>>> sys.getsizeof('aa') > > > 27 > > >>>> sys.getsizeof('a?') > > > 39 > > > > > > > >>> sys.getsizeof('aa') - sys.getsizeof('a') > > 1 > > > > One byte per codepoint. > > > > >>> sys.getsizeof('??') - sys.getsizeof('?') > > 1 > > > > Also one byte per codepoint. > > > > >>> sys.getsizeof('?') - sys.getsizeof('a') > > 12 > > > > Clearly there's more going on here. > > > > FSR is an optimisation. You'll always be able to find some > > circumstances where an optimisation makes things worse, but what > > matters is the overall result. ---- Yes, I know my examples are always wrong, never real examples. I can point long strings, I should point short strings. I point a short string (char), it is not long enough. Strings as dict keys, no the problem is in Python dict. Performance? no that's a memory issue. Memory? no, it's a question to keep perfomance. I am using this char, no you should not, it's no common. The nabla operator in TeX file, who is so stupid to use that char? Many time, I'm just mimicking 'BDFL' examples, just by replacing "his" ascii chars by non ascii char ;-) And so on. To be short, this is *never* the FSR, always something else. Suggestion. Start by solving all these "micro-benchmarks". all the memory cases. It a good start, no? jmf From antoon.pardon at rece.vub.ac.be Sun Jul 28 16:01:10 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Sun, 28 Jul 2013 22:01:10 +0200 Subject: RE Module Performance In-Reply-To: <95b91473-b707-4288-860c-d02fda7af1ea@googlegroups.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <4117e08f-941a-42d5-87b6-09e66f8c7b60@googlegroups.com> <95b91473-b707-4288-860c-d02fda7af1ea@googlegroups.com> Message-ID: <51F57886.90409@rece.vub.ac.be> Op 28-07-13 21:30, wxjmfauth at gmail.com schreef: > To be short, this is *never* the FSR, always something > else. > > Suggestion. Start by solving all these "micro-benchmarks". > all the memory cases. It a good start, no? > There is nothing to solve. Unicode doesn't force implementations to use the same size of memory for strings of the same length. So you pointing out examples of same length strings that don't use the same size of memory doesn't point at something that must be solved. -- Antoon Pardon From lele at metapensiero.it Sun Jul 28 16:45:37 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Sun, 28 Jul 2013 22:45:37 +0200 Subject: RE Module Performance References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <4117e08f-941a-42d5-87b6-09e66f8c7b60@googlegroups.com> <95b91473-b707-4288-860c-d02fda7af1ea@googlegroups.com> Message-ID: <87ppu29zwu.fsf@nautilus.nautilus> wxjmfauth at gmail.com writes: > Suggestion. Start by solving all these "micro-benchmarks". > all the memory cases. It a good start, no? Since you seem the only one who has this dramatic problem with such micro-benchmarks, that BTW have nothing to do with "unicode compliance", I'd suggest *you* should find a better implementation and propose it to the core devs. An even better suggestion, with due respect, is to get a life and find something more interesting to do, or at least better arguments :-) ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From wxjmfauth at gmail.com Tue Jul 30 10:01:02 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Tue, 30 Jul 2013 07:01:02 -0700 (PDT) Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> Message-ID: <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> Le dimanche 28 juillet 2013 05:53:22 UTC+2, Ian a ?crit?: > On Sat, Jul 27, 2013 at 12:21 PM, wrote: > > > Back to utf. utfs are not only elements of a unique set of encoded > > > code points. They have an interesting feature. Each "utf chunk" > > > holds intrisically the character (in fact the code point) it is > > > supposed to represent. In utf-32, the obvious case, it is just > > > the code point. In utf-8, that's the first chunk which helps and > > > utf-16 is a mixed case (utf-8 / utf-32). In other words, in an > > > implementation using bytes, for any pointer position it is always > > > possible to find the corresponding encoded code point and from this > > > the corresponding character without any "programmed" information. See > > > my editor example, how to find the char under the caret? In fact, > > > a silly example, how can the caret can be positioned or moved, if > > > the underlying corresponding encoded code point can not be > > > dicerned! > > > > Yes, given a pointer location into a utf-8 or utf-16 string, it is > > easy to determine the identity of the code point at that location. > > But this is not often a useful operation, save for resynchronization > > in the case that the string data is corrupted. The caret of an editor > > does not conceptually correspond to a pointer location, but to a > > character index. Given a particular character index (e.g. 127504), an > > editor must be able to determine the identity and/or the memory > > location of the character at that index, and for UTF-8 and UTF-16 > > without an auxiliary data structure that is a O(n) operation. > > ------ Same conceptual mistake as Steven's example with its buffers, the buffer does not know it holds characters. This is not the point to discuss. ----- I am pretty sure that once you have typed your 127504 ascii characters, you are very happy the buffer of your editor does not waste time in reencoding the buffer as soon as you enter an ?, the 125505th char. Sorry, I wanted to say z instead of euro, just to show that backspacing the last char and reentering a new char implies twice a reencoding. Somebody wrote "FSR" is just an optimization. Yes, but in case of an editor ? la FSR, this optimization take place everytime you enter a char. Your poor editor, in fact the FSR, is finally spending its time in optimizing and finally it optimizes nothing. (It is even worse). If you type correctly a z instead of an ?, it is not necessary to reencode the buffer. Problem, you do you know that you do not have to reencode? simple just check it, and by just checking it wastes time to test it you have to optimized or not and hurt a little bit more what is supposed to be an optimization. Do not confuse the process of optimisation and the result of optimization (funny, it's like the utf's). There is a trick to make the editor to know if it has to be "optimized". Just put some flag somewhere. Then you fall on the "Houston" syndrome. Houston, we got a problem, our buffer consumes much more bytes than expected. >>> sys.getsizeof('?') 40 >>> sys.getsizeof('a') 26 Now the good news. In an editor ? la FSR, the "composition" is not so important. You know, "practicality beats purity". The hard job is the text rendering engine and the handling of the font (even in a raw unicode editor). And as these tools are luckily not woking ? la FSR (probably because they understand the coding of the characters), your editor is still working not so badly. jmf From antoon.pardon at rece.vub.ac.be Tue Jul 30 10:38:09 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Tue, 30 Jul 2013 16:38:09 +0200 Subject: RE Module Performance In-Reply-To: <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> Message-ID: <51F7CFD1.1090403@rece.vub.ac.be> Op 30-07-13 16:01, wxjmfauth at gmail.com schreef: > > I am pretty sure that once you have typed your 127504 > ascii characters, you are very happy the buffer of your > editor does not waste time in reencoding the buffer as > soon as you enter an ?, the 125505th char. Sorry, I wanted > to say z instead of euro, just to show that backspacing the > last char and reentering a new char implies twice a reencoding. Using a single string as an editor buffer is a bad idea in python for the simple reason that strings are immutable. So adding characters would mean continuously copying the string buffer into a new string with the next character added. Copying 127504 characters into a new string will not make that much of a difference whether the octets are just copied to octets or are unpacked into 32 bit words. > Somebody wrote "FSR" is just an optimization. Yes, but in case > of an editor ? la FSR, this optimization take place everytime you > enter a char. Your poor editor, in fact the FSR, is finally > spending its time in optimizing and finally it optimizes nothing. > (It is even worse). Even if you would do it this way, it would *not* take place every time you enter a char. Once your buffer would contain a wide character, it would just need to convert the single character that is added after each keystroke. It would not need to convert the whole buffer after each key stroke. > If you type correctly a z instead of an ?, it is not necessary > to reencode the buffer. Problem, you do you know that you do > not have to reencode? simple just check it, and by just checking > it wastes time to test it you have to optimized or not and hurt > a little bit more what is supposed to be an optimization. Your scenario is totally unrealistic. First of all because of the immutable nature of python strings, second because you suggest that real time usage would result in frequent conversions which is highly unlikely. -- Antoon Pardon From python at mrabarnett.plus.com Tue Jul 30 12:13:40 2013 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 30 Jul 2013 17:13:40 +0100 Subject: RE Module Performance In-Reply-To: <51F7CFD1.1090403@rece.vub.ac.be> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> Message-ID: <51F7E634.2030200@mrabarnett.plus.com> On 30/07/2013 15:38, Antoon Pardon wrote: > Op 30-07-13 16:01, wxjmfauth at gmail.com schreef: >> >> I am pretty sure that once you have typed your 127504 ascii >> characters, you are very happy the buffer of your editor does not >> waste time in reencoding the buffer as soon as you enter an ?, the >> 125505th char. Sorry, I wanted to say z instead of euro, just to >> show that backspacing the last char and reentering a new char >> implies twice a reencoding. > > Using a single string as an editor buffer is a bad idea in python for > the simple reason that strings are immutable. Using a single string as an editor buffer is a bad idea in _any_ language because an insertion would require all the following characters to be moved. > So adding characters would mean continuously copying the string > buffer into a new string with the next character added. Copying > 127504 characters into a new string will not make that much of a > difference whether the octets are just copied to octets or are > unpacked into 32 bit words. > >> Somebody wrote "FSR" is just an optimization. Yes, but in case of >> an editor ? la FSR, this optimization take place everytime you >> enter a char. Your poor editor, in fact the FSR, is finally >> spending its time in optimizing and finally it optimizes nothing. >> (It is even worse). > > Even if you would do it this way, it would *not* take place every > time you enter a char. Once your buffer would contain a wide > character, it would just need to convert the single character that is > added after each keystroke. It would not need to convert the whole > buffer after each key stroke. > >> If you type correctly a z instead of an ?, it is not necessary to >> reencode the buffer. Problem, you do you know that you do not have >> to reencode? simple just check it, and by just checking it wastes >> time to test it you have to optimized or not and hurt a little bit >> more what is supposed to be an optimization. > > Your scenario is totally unrealistic. First of all because of the > immutable nature of python strings, second because you suggest that > real time usage would result in frequent conversions which is highly > unlikely. > What you would have is a list of mutable chunks. Inserting into a chunk would be fast, and a chunk would be split if it's already full. Also, small adjacent chunks would be joined together. Finally, a chunk could use FSR to reduce memory usage. From antoon.pardon at rece.vub.ac.be Tue Jul 30 12:39:29 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Tue, 30 Jul 2013 18:39:29 +0200 Subject: RE Module Performance In-Reply-To: <51F7E634.2030200@mrabarnett.plus.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> <51F7E634.2030200@mrabarnett.plus.com> Message-ID: <51F7EC41.5010704@rece.vub.ac.be> Op 30-07-13 18:13, MRAB schreef: > On 30/07/2013 15:38, Antoon Pardon wrote: >> Op 30-07-13 16:01, wxjmfauth at gmail.com schreef: >>> >>> I am pretty sure that once you have typed your 127504 ascii >>> characters, you are very happy the buffer of your editor does not >>> waste time in reencoding the buffer as soon as you enter an ?, the >>> 125505th char. Sorry, I wanted to say z instead of euro, just to >>> show that backspacing the last char and reentering a new char >>> implies twice a reencoding. >> >> Using a single string as an editor buffer is a bad idea in python for >> the simple reason that strings are immutable. > > Using a single string as an editor buffer is a bad idea in _any_ > language because an insertion would require all the following > characters to be moved. Not if you use a gap buffer. -- Antoon Pardon. From python at mrabarnett.plus.com Tue Jul 30 13:14:55 2013 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 30 Jul 2013 18:14:55 +0100 Subject: RE Module Performance In-Reply-To: <51F7EC41.5010704@rece.vub.ac.be> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> <51F7E634.2030200@mrabarnett.plus.com> <51F7EC41.5010704@rece.vub.ac.be> Message-ID: <51F7F48F.3070803@mrabarnett.plus.com> On 30/07/2013 17:39, Antoon Pardon wrote: > Op 30-07-13 18:13, MRAB schreef: >> On 30/07/2013 15:38, Antoon Pardon wrote: >>> Op 30-07-13 16:01, wxjmfauth at gmail.com schreef: >>>> >>>> I am pretty sure that once you have typed your 127504 ascii >>>> characters, you are very happy the buffer of your editor does not >>>> waste time in reencoding the buffer as soon as you enter an ?, the >>>> 125505th char. Sorry, I wanted to say z instead of euro, just to >>>> show that backspacing the last char and reentering a new char >>>> implies twice a reencoding. >>> >>> Using a single string as an editor buffer is a bad idea in python for >>> the simple reason that strings are immutable. >> >> Using a single string as an editor buffer is a bad idea in _any_ >> language because an insertion would require all the following >> characters to be moved. > > Not if you use a gap buffer. > The disadvantage there is that when you move the cursor you must move characters around. For example, what if the cursor was at the start and you wanted to move it to the end? Also, when the gap has been filled, you need to make a new one. From antoon.pardon at rece.vub.ac.be Tue Jul 30 14:19:02 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Tue, 30 Jul 2013 20:19:02 +0200 Subject: RE Module Performance In-Reply-To: <51F7F48F.3070803@mrabarnett.plus.com> References: <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> <51F7E634.2030200@mrabarnett.plus.com> <51F7EC41.5010704@rece.vub.ac.be> <51F7F48F.3070803@mrabarnett.plus.com> Message-ID: <51F80396.2020904@rece.vub.ac.be> Op 30-07-13 19:14, MRAB schreef: > On 30/07/2013 17:39, Antoon Pardon wrote: >> Op 30-07-13 18:13, MRAB schreef: >>> On 30/07/2013 15:38, Antoon Pardon wrote: >>>> Op 30-07-13 16:01, wxjmfauth at gmail.com schreef: >>>>> >>>>> I am pretty sure that once you have typed your 127504 ascii >>>>> characters, you are very happy the buffer of your editor does not >>>>> waste time in reencoding the buffer as soon as you enter an ?, the >>>>> 125505th char. Sorry, I wanted to say z instead of euro, just to >>>>> show that backspacing the last char and reentering a new char >>>>> implies twice a reencoding. >>>> >>>> Using a single string as an editor buffer is a bad idea in python for >>>> the simple reason that strings are immutable. >>> >>> Using a single string as an editor buffer is a bad idea in _any_ >>> language because an insertion would require all the following >>> characters to be moved. >> >> Not if you use a gap buffer. >> > The disadvantage there is that when you move the cursor you must move > characters around. For example, what if the cursor was at the start and > you wanted to move it to the end? Also, when the gap has been filled, > you need to make a new one. So? Why are you making this a point of discussion? I was not aware that the pro and cons of various editor buffer implemantations was relevant to the point I was trying to make. If you prefer an other data structure in the editor you are working on, I will not dissuade you. -- Antoon Pardon From torriem at gmail.com Tue Jul 30 23:30:09 2013 From: torriem at gmail.com (Michael Torrie) Date: Tue, 30 Jul 2013 21:30:09 -0600 Subject: RE Module Performance In-Reply-To: <51F80396.2020904@rece.vub.ac.be> References: <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> <51F7E634.2030200@mrabarnett.plus.com> <51F7EC41.5010704@rece.vub.ac.be> <51F7F48F.3070803@mrabarnett.plus.com> <51F80396.2020904@rece.vub.ac.be> Message-ID: <51F884C1.1030505@gmail.com> On 07/30/2013 12:19 PM, Antoon Pardon wrote: > So? Why are you making this a point of discussion? I was not aware that > the pro and cons of various editor buffer implemantations was relevant > to the point I was trying to make. I for one found it very interesting. In fact this thread caused me to wonder how one actually does create an efficient editor. Off the original topic true, but still very interesting. From antoon.pardon at rece.vub.ac.be Wed Jul 31 03:23:33 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Wed, 31 Jul 2013 09:23:33 +0200 Subject: RE Module Performance In-Reply-To: <51F884C1.1030505@gmail.com> References: <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> <51F7E634.2030200@mrabarnett.plus.com> <51F7EC41.5010704@rece.vub.ac.be> <51F7F48F.3070803@mrabarnett.plus.com> <51F80396.2020904@rece.vub.ac.be> <51F884C1.1030505@gmail.com> Message-ID: <51F8BB75.9000003@rece.vub.ac.be> Op 31-07-13 05:30, Michael Torrie schreef: > On 07/30/2013 12:19 PM, Antoon Pardon wrote: >> So? Why are you making this a point of discussion? I was not aware that >> the pro and cons of various editor buffer implemantations was relevant >> to the point I was trying to make. > > I for one found it very interesting. In fact this thread caused me to > wonder how one actually does create an efficient editor. Off the > original topic true, but still very interesting. > Yes, it can be interesting. But I really think if that is what you want to discuss, it deserves its own subject thread. -- Antoon Pardon From torriem at gmail.com Wed Jul 31 10:27:43 2013 From: torriem at gmail.com (Michael Torrie) Date: Wed, 31 Jul 2013 08:27:43 -0600 Subject: RE Module Performance In-Reply-To: <51F8BB75.9000003@rece.vub.ac.be> References: <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> <51F7E634.2030200@mrabarnett.plus.com> <51F7EC41.5010704@rece.vub.ac.be> <51F7F48F.3070803@mrabarnett.plus.com> <51F80396.2020904@rece.vub.ac.be> <51F884C1.1030505@gmail.com> <51F8BB75.9000003@rece.vub.ac.be> Message-ID: <51F91EDF.8070403@gmail.com> On 07/31/2013 01:23 AM, Antoon Pardon wrote: > Op 31-07-13 05:30, Michael Torrie schreef: >> On 07/30/2013 12:19 PM, Antoon Pardon wrote: >>> So? Why are you making this a point of discussion? I was not aware that >>> the pro and cons of various editor buffer implemantations was relevant >>> to the point I was trying to make. >> >> I for one found it very interesting. In fact this thread caused me to >> wonder how one actually does create an efficient editor. Off the >> original topic true, but still very interesting. >> > > Yes, it can be interesting. But I really think if that is what you want > to discuss, it deserves its own subject thread. Subject lines can and should be changed to reflect the ebbs and flows of the discussion. In fact this thread's subject should have been changed a long time ago since the original topic was RE module performance! From wxjmfauth at gmail.com Tue Jul 30 15:09:11 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Tue, 30 Jul 2013 12:09:11 -0700 (PDT) Subject: RE Module Performance In-Reply-To: References: <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> <51F7E634.2030200@mrabarnett.plus.com> <51F7EC41.5010704@rece.vub.ac.be> <51F7F48F.3070803@mrabarnett.plus.com> Message-ID: <39155ddf-437c-459e-ad7c-dd841810a592@googlegroups.com> Matable, immutable, copyint + xxx, bufferint, O(n) .... Yes, but conceptualy the reencoding happen sometime, somewhere. The internal "ucs-2" will never automagically be transformed into "ucs-4" (eg). >>> timeit.timeit("'a'*10000 +'?'") 7.087220684719967 >>> timeit.timeit("'a'*10000 +'z'") 1.5685214234430873 >>> timeit.timeit("z = 'a'*10000; z = z +'?'") 7.169538866162213 >>> timeit.timeit("z = 'a'*10000; z = z +'z'") 1.5815893830557286 >>> timeit.timeit("z = 'a'*10000; z += 'z'") 1.606955741596181 >>> timeit.timeit("z = 'a'*10000; z += '?'") 7.160483334521416 And do not forget, in a pure utf coding scheme, your char or a char will *never* be larger than 4 bytes. >>> sys.getsizeof('a') 26 >>> sys.getsizeof('\U000101000') 48 jmf From rosuav at gmail.com Tue Jul 30 16:04:18 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 30 Jul 2013 21:04:18 +0100 Subject: RE Module Performance In-Reply-To: <39155ddf-437c-459e-ad7c-dd841810a592@googlegroups.com> References: <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> <51F7E634.2030200@mrabarnett.plus.com> <51F7EC41.5010704@rece.vub.ac.be> <51F7F48F.3070803@mrabarnett.plus.com> <39155ddf-437c-459e-ad7c-dd841810a592@googlegroups.com> Message-ID: On Tue, Jul 30, 2013 at 8:09 PM, wrote: > Matable, immutable, copyint + xxx, bufferint, O(n) .... > Yes, but conceptualy the reencoding happen sometime, somewhere. > The internal "ucs-2" will never automagically be transformed > into "ucs-4" (eg). But probably not on the entire document. With even a brainless scheme like I posted code for, no more than 1024 bytes will need to be recoded at a time (except in some odd edge cases, and even then, no more than once for any given file). > And do not forget, in a pure utf coding scheme, your > char or a char will *never* be larger than 4 bytes. > >>>> sys.getsizeof('a') > 26 >>>> sys.getsizeof('\U000101000') > 48 Yeah, you have a few odd issues like, oh, I dunno, GC overhead, reference count, object class, and string length, all stored somewhere there. Honestly jmf, if you want raw assembly you know where to get it. ChrisA From torriem at gmail.com Tue Jul 30 23:54:01 2013 From: torriem at gmail.com (Michael Torrie) Date: Tue, 30 Jul 2013 21:54:01 -0600 Subject: RE Module Performance In-Reply-To: <39155ddf-437c-459e-ad7c-dd841810a592@googlegroups.com> References: <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> <51F7E634.2030200@mrabarnett.plus.com> <51F7EC41.5010704@rece.vub.ac.be> <51F7F48F.3070803@mrabarnett.plus.com> <39155ddf-437c-459e-ad7c-dd841810a592@googlegroups.com> Message-ID: <51F88A59.7030900@gmail.com> On 07/30/2013 01:09 PM, wxjmfauth at gmail.com wrote: > Matable, immutable, copyint + xxx, bufferint, O(n) .... > Yes, but conceptualy the reencoding happen sometime, somewhere. > The internal "ucs-2" will never automagically be transformed > into "ucs-4" (eg). So what major python project are you working on where you've found FSR in general to be a problem? Maybe we can help you work out a more appropriate data structure and algorithm to use. But if you're not developing something, and not developing in Python, perhaps you should withdraw and let us use our horrible FSR in peace, because it doesn't seem to bother the vast majority of python programmers, and does not bother some large python projects out there. In fact I think most of us welcome integrated, correct, full unicode. From steve+comp.lang.python at pearwood.info Wed Jul 31 01:45:18 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 31 Jul 2013 05:45:18 GMT Subject: RE Module Performance References: <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> <51F7E634.2030200@mrabarnett.plus.com> <51F7EC41.5010704@rece.vub.ac.be> <51F7F48F.3070803@mrabarnett.plus.com> <39155ddf-437c-459e-ad7c-dd841810a592@googlegroups.com> Message-ID: <51f8a46e$0$30000$c3e8da3$5496439d@news.astraweb.com> On Tue, 30 Jul 2013 12:09:11 -0700, wxjmfauth wrote: > And do not forget, in a pure utf coding scheme, your char or a char will > *never* be larger than 4 bytes. > >>>> sys.getsizeof('a') > 26 >>>> sys.getsizeof('\U000101000') > 48 Neither character above is larger than 4 bytes. You forgot to deduct the size of the object header. Python is a high-level object-oriented language, if you care about minimizing every possible byte, you should use a low-level language like C. Then you can give every character 21 bits, and be happy that you don't waste even one bit. -- Steven From rosuav at gmail.com Wed Jul 31 03:17:33 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 31 Jul 2013 08:17:33 +0100 Subject: RE Module Performance In-Reply-To: <51f8a46e$0$30000$c3e8da3$5496439d@news.astraweb.com> References: <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> <51F7E634.2030200@mrabarnett.plus.com> <51F7EC41.5010704@rece.vub.ac.be> <51F7F48F.3070803@mrabarnett.plus.com> <39155ddf-437c-459e-ad7c-dd841810a592@googlegroups.com> <51f8a46e$0$30000$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Wed, Jul 31, 2013 at 6:45 AM, Steven D'Aprano wrote: > if you care about minimizing every possible byte, you should > use a low-level language like C. Then you can give every character 21 > bits, and be happy that you don't waste even one bit. Could go better! Since not every character has been assigned, and some are specifically banned (eg U+FFFE and U+D800-U+DFFF), you could cut them out of your representation system and save memory! ChrisA From wxjmfauth at gmail.com Wed Jul 31 16:15:23 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Wed, 31 Jul 2013 13:15:23 -0700 (PDT) Subject: RE Module Performance In-Reply-To: <51f8a46e$0$30000$c3e8da3$5496439d@news.astraweb.com> References: <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> <51F7E634.2030200@mrabarnett.plus.com> <51F7EC41.5010704@rece.vub.ac.be> <51F7F48F.3070803@mrabarnett.plus.com> <39155ddf-437c-459e-ad7c-dd841810a592@googlegroups.com> <51f8a46e$0$30000$c3e8da3$5496439d@news.astraweb.com> Message-ID: <7a4be3ec-4665-4262-9cc6-286362fe2932@googlegroups.com> Le mercredi 31 juillet 2013 07:45:18 UTC+2, Steven D'Aprano a ?crit?: > On Tue, 30 Jul 2013 12:09:11 -0700, wxjmfauth wrote: > > > > > And do not forget, in a pure utf coding scheme, your char or a char will > > > *never* be larger than 4 bytes. > > > > > >>>> sys.getsizeof('a') > > > 26 > > >>>> sys.getsizeof('\U000101000') > > > 48 > > > > Neither character above is larger than 4 bytes. You forgot to deduct the > > size of the object header. Python is a high-level object-oriented > > language, if you care about minimizing every possible byte, you should > > use a low-level language like C. Then you can give every character 21 > > bits, and be happy that you don't waste even one bit. > > > > > > -- > > Steven ... char never consumes or requires more than 4 bytes ... jmf From rosuav at gmail.com Wed Jul 31 16:41:41 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 31 Jul 2013 21:41:41 +0100 Subject: RE Module Performance In-Reply-To: <7a4be3ec-4665-4262-9cc6-286362fe2932@googlegroups.com> References: <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> <51F7E634.2030200@mrabarnett.plus.com> <51F7EC41.5010704@rece.vub.ac.be> <51F7F48F.3070803@mrabarnett.plus.com> <39155ddf-437c-459e-ad7c-dd841810a592@googlegroups.com> <51f8a46e$0$30000$c3e8da3$5496439d@news.astraweb.com> <7a4be3ec-4665-4262-9cc6-286362fe2932@googlegroups.com> Message-ID: On Wed, Jul 31, 2013 at 9:15 PM, wrote: > ... char never consumes or requires more than 4 bytes ... > The integer 5 should be able to be stored in 3 bits. >>> sys.getsizeof(5) 14 Clearly Python is doing something really horribly wrong here. In fact, sys.getsizeof needs to be changed to return a float, to allow it to more properly reflect these important facts. ChrisA From antoon.pardon at rece.vub.ac.be Wed Jul 31 04:11:10 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Wed, 31 Jul 2013 10:11:10 +0200 Subject: RE Module Performance In-Reply-To: <39155ddf-437c-459e-ad7c-dd841810a592@googlegroups.com> References: <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> <51F7E634.2030200@mrabarnett.plus.com> <51F7EC41.5010704@rece.vub.ac.be> <51F7F48F.3070803@mrabarnett.plus.com> <39155ddf-437c-459e-ad7c-dd841810a592@googlegroups.com> Message-ID: <51F8C69E.3010102@rece.vub.ac.be> Op 30-07-13 21:09, wxjmfauth at gmail.com schreef: > Matable, immutable, copyint + xxx, bufferint, O(n) .... > Yes, but conceptualy the reencoding happen sometime, somewhere. Which is a far cry from your previous claim that it happened every time you enter a char. This of course make your case harder to argue. Because the impact of something that happens sometime, somewhere is vastly less than something that happens everytime you enter a char. > The internal "ucs-2" will never automagically be transformed > into "ucs-4" (eg). It will just start producing wrong results when someone starts using characters that don't fit into ucs-2. >>>> timeit.timeit("'a'*10000 +'?'") > 7.087220684719967 >>>> timeit.timeit("'a'*10000 +'z'") > 1.5685214234430873 >>>> timeit.timeit("z = 'a'*10000; z = z +'?'") > 7.169538866162213 >>>> timeit.timeit("z = 'a'*10000; z = z +'z'") > 1.5815893830557286 >>>> timeit.timeit("z = 'a'*10000; z += 'z'") > 1.606955741596181 >>>> timeit.timeit("z = 'a'*10000; z += '?'") > 7.160483334521416 > > > And do not forget, in a pure utf coding scheme, your > char or a char will *never* be larger than 4 bytes. > >>>> sys.getsizeof('a') > 26 >>>> sys.getsizeof('\U000101000') > 48 Nonsense. >>> sys.getsizeof('a'.encode('utf-8')) 18 From wxjmfauth at gmail.com Wed Jul 31 04:32:13 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Wed, 31 Jul 2013 01:32:13 -0700 (PDT) Subject: RE Module Performance In-Reply-To: References: <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> <51F7E634.2030200@mrabarnett.plus.com> <51F7EC41.5010704@rece.vub.ac.be> <51F7F48F.3070803@mrabarnett.plus.com> <39155ddf-437c-459e-ad7c-dd841810a592@googlegroups.com> Message-ID: <797da2f0-5f62-43b9-ab4d-c5eb8d6c64a2@googlegroups.com> FSR: === The 'a' in 'a?' and 'a\U0001d11e: >>> ['{:#010b}'.format(c) for c in 'a?'.encode('utf-16-be')] ['0b00000000', '0b01100001', '0b00100000', '0b10101100'] >>> ['{:#010b}'.format(c) for c in 'a\U0001d11e'.encode('utf-32-be')] ['0b00000000', '0b00000000', '0b00000000', '0b01100001', '0b00000000', '0b00000001', '0b11010001', '0b00011110'] Has to be done. sys.getsizeof('a?') 42 sys.getsizeof('a\U0001d11e') 48 sys.getsizeof('aa') 27 Unicode/utf* ============ i) ("primary key") Create and use a unique set of encoded code points. ii) ("secondary key") Depending of the wish, memory/performance: utf-8/16/32 Two advantages at the light of the above example: iii) The "a" has never to be reencoded. iv) An "a" size never exceeds 4 bytes. Hard job to solve/satisfy i), ii), iii) and iv) at the same time. Is is possible? ;-) The solution is in the problem. jmf From antoon.pardon at rece.vub.ac.be Wed Jul 31 04:59:20 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Wed, 31 Jul 2013 10:59:20 +0200 Subject: RE Module Performance In-Reply-To: <797da2f0-5f62-43b9-ab4d-c5eb8d6c64a2@googlegroups.com> References: <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> <51F7E634.2030200@mrabarnett.plus.com> <51F7EC41.5010704@rece.vub.ac.be> <51F7F48F.3070803@mrabarnett.plus.com> <39155ddf-437c-459e-ad7c-dd841810a592@googlegroups.com> <797da2f0-5f62-43b9-ab4d-c5eb8d6c64a2@googlegroups.com> Message-ID: <51F8D1E8.4020401@rece.vub.ac.be> Op 31-07-13 10:32, wxjmfauth at gmail.com schreef: > Unicode/utf* > ============ > > i) ("primary key") Create and use a unique set of encoded > code points. FSR does this. >>> st1 = 'a?' >>> st2 = 'aa' >>> ord(st1[0]) 97 >>> ord(st2[0]) 97 >>> > ii) ("secondary key") Depending of the wish, > memory/performance: utf-8/16/32 Whose wish? I don't know any language that allows the programmer choose the internal representation of its strings. If it is the designers choice FSR does this, if it is the programmers choice, I don't see why this is necessary for compliance. > Two advantages at the light of the above example: > iii) The "a" has never to be reencoded. FSR: check. Using a container with wider slots is not a re?ncoding. If such widening is encoding then your 'choice' between utf-8/16/32 implies that it will also have to reencode when it changes from utf-8 to utf-16 or utf-32. > iv) An "a" size never exceeds 4 bytes. FSR: check. > Hard job to solve/satisfy i), ii), iii) and iv) at the same time. > Is is possible? ;-) The solution is in the problem. Mayby you should use bytes or bytearrays if that is really what you want. -- Antoon Pardon From torriem at gmail.com Wed Jul 31 10:44:44 2013 From: torriem at gmail.com (Michael Torrie) Date: Wed, 31 Jul 2013 08:44:44 -0600 Subject: RE Module Performance In-Reply-To: <797da2f0-5f62-43b9-ab4d-c5eb8d6c64a2@googlegroups.com> References: <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> <51F7E634.2030200@mrabarnett.plus.com> <51F7EC41.5010704@rece.vub.ac.be> <51F7F48F.3070803@mrabarnett.plus.com> <39155ddf-437c-459e-ad7c-dd841810a592@googlegroups.com> <797da2f0-5f62-43b9-ab4d-c5eb8d6c64a2@googlegroups.com> Message-ID: <51F922DC.9090209@gmail.com> On 07/31/2013 02:32 AM, wxjmfauth at gmail.com wrote: > Unicode/utf* Why do you keep using the terms "utf" and "Unicode" interchangeably? From joshua at landau.ws Tue Jul 30 13:40:17 2013 From: joshua at landau.ws (Joshua Landau) Date: Tue, 30 Jul 2013 18:40:17 +0100 Subject: RE Module Performance In-Reply-To: <51F7EC41.5010704@rece.vub.ac.be> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> <51F7E634.2030200@mrabarnett.plus.com> <51F7EC41.5010704@rece.vub.ac.be> Message-ID: On 30 July 2013 17:39, Antoon Pardon wrote: > Op 30-07-13 18:13, MRAB schreef: > > On 30/07/2013 15:38, Antoon Pardon wrote: >> >>> Op 30-07-13 16:01, wxjmfauth at gmail.com schreef: >>> >>>> >>>> I am pretty sure that once you have typed your 127504 ascii >>>> characters, you are very happy the buffer of your editor does not >>>> waste time in reencoding the buffer as soon as you enter an ?, the >>>> 125505th char. Sorry, I wanted to say z instead of euro, just to >>>> show that backspacing the last char and reentering a new char >>>> implies twice a reencoding. >>>> >>> >>> Using a single string as an editor buffer is a bad idea in python for >>> the simple reason that strings are immutable. >>> >> >> Using a single string as an editor buffer is a bad idea in _any_ >> language because an insertion would require all the following >> characters to be moved. >> > > Not if you use a gap buffer. Additionally, who says a language couldn't use, say, B-Trees for all of its list-like types, including strings? -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue Jul 30 17:05:54 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 30 Jul 2013 17:05:54 -0400 Subject: RE Module Performance In-Reply-To: References: <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> <51F7E634.2030200@mrabarnett.plus.com> <51F7EC41.5010704@rece.vub.ac.be> Message-ID: On 7/30/2013 1:40 PM, Joshua Landau wrote: > Additionally, who says a language couldn't use, say, B-Trees for all of > its list-like types, including strings? Tk apparently uses a B-tree in its text widget. -- Terry Jan Reedy From nhodgson at iinet.net.au Tue Jul 30 23:09:54 2013 From: nhodgson at iinet.net.au (Neil Hodgson) Date: Wed, 31 Jul 2013 13:09:54 +1000 Subject: RE Module Performance In-Reply-To: References: <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> <51F7CFD1.1090403@rece.vub.ac.be> <51F7E634.2030200@mrabarnett.plus.com> <51F7EC41.5010704@rece.vub.ac.be> Message-ID: MRAB: > The disadvantage there is that when you move the cursor you must move > characters around. For example, what if the cursor was at the start and > you wanted to move it to the end? Also, when the gap has been filled, > you need to make a new one. The normal technique is to only move the gap when text is added or removed, not when the cursor moves. Code that reads the contents, such as for display, handles the gap by checking the requested position and using a different offset when the position is after the gap. Gap buffers work well because changes are generally close to the previous change, so require moving only a relatively small amount of text. Even an occasional move of the whole contents won't cause too much trouble for interactivity with current processors moving multiple megabytes per millisecond. Neil From rosuav at gmail.com Tue Jul 30 10:45:57 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 30 Jul 2013 15:45:57 +0100 Subject: RE Module Performance In-Reply-To: <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> Message-ID: On Tue, Jul 30, 2013 at 3:01 PM, wrote: > I am pretty sure that once you have typed your 127504 > ascii characters, you are very happy the buffer of your > editor does not waste time in reencoding the buffer as > soon as you enter an ?, the 125505th char. Sorry, I wanted > to say z instead of euro, just to show that backspacing the > last char and reentering a new char implies twice a reencoding. You're still thinking that the editor's buffer is a Python string. As I've shown earlier, this is a really bad idea, and that has nothing to do with FSR/PEP 393. An immutable string is *horribly* inefficient at this; if you want to keep concatenating onto a string, the recommended method is a list of strings that gets join()d at the end, and the same technique works well here. Here's a little demo class that could make the basis for such a system: class EditorBuffer: def __init__(self,fn): self.fn=fn self.buffer=[open(fn).read()] def insert(self,pos,char): if pos==0: # Special case: insertion at beginning of buffer if len(self.buffer[0])>1024: self.buffer.insert(0,char) else: self.buffer[0]=char+self.buffer[0] return for idx,part in enumerate(self.buffer): l=len(part) if pos>l: pos-=l continue if pos1024: self.buffer[idx:idx+1]=self.buffer[idx],char else: self.buffer[idx]+=char return raise ValueError("Cannot insert past end of buffer") def __str__(self): return ''.join(self.buffer) def save(self): open(fn,"w").write(str(self)) It guarantees that inserts will never need to resize more than 1KB of text. As a real basis for an editor, it still sucks, but it's purely to prove this one point. ChrisA From timothy.c.delaney at gmail.com Tue Jul 30 13:27:03 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Wed, 31 Jul 2013 03:27:03 +1000 Subject: RE Module Performance In-Reply-To: <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <43ce1b65-9d6d-47dd-b209-9a3bbafc0b8c@googlegroups.com> Message-ID: On 31 July 2013 00:01, wrote: > > I am pretty sure that once you have typed your 127504 > ascii characters, you are very happy the buffer of your > editor does not waste time in reencoding the buffer as > soon as you enter an ?, the 125505th char. Sorry, I wanted > to say z instead of euro, just to show that backspacing the > last char and reentering a new char implies twice a reencoding. > And here we come to the root of your complete misunderstanding and mischaracterisation of the FSR. You don't appear to understand that strings in Python are immutable and that to add a character to an existing string requires copying the entire string + new character. In your hypothetical situation above, you have already performed 127504 copy + new character operations before you ever get to a single widening operation. The overhead of the copy + new character repeated 127504 times dwarfs the overhead of a single widening operation. Given your misunderstanding, it's no surprise that you are focused on microbenchmarks that demonstrate that copying entire strings and adding a character can be slower in some situations than others. When the only use case you have is implementing the buffer of an editor using an immutable string I can fully understand why you would be concerned about the performance of adding and removing individual characters. However, in that case *you're focused on the wrong problem*. Until you can demonstrate an understanding that doing the above in any language which has immutable strings is completely insane you will have no credibility and the only interest anyone will pay to your posts is refuting your FUD so that people new to the language are not driven off by you. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From wxjmfauth at gmail.com Sun Jul 28 15:23:04 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Sun, 28 Jul 2013 12:23:04 -0700 (PDT) Subject: FSR and unicode compliance - was Re: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> Message-ID: Le dimanche 28 juillet 2013 17:52:47 UTC+2, Michael Torrie a ?crit?: > On 07/27/2013 12:21 PM, wxjmfauth at gmail.com wrote: > > > Good point. FSR, nice tool for those who wish to teach > > > Unicode. It is not every day, one has such an opportunity. > > > > I had a long e-mail composed, but decided to chop it down, but still too > > long. so I ditched a lot of the context, which jmf also seems to do. > > Apologies. > > > > 1. FSR *is* UTF-32 so it is as unicode compliant as UTF-32, since UTF-32 > > is an official encoding. FSR only differs from UTF-32 in that the > > padding zeros are stripped off such that it is stored in the most > > compact form that can handle all the characters in string, which is > > always known at string creation time. Now you can argue many things, > > but to say FSR is not unicode compliant is quite a stretch! What > > unicode entities or characters cannot be stored in strings using FSR? > > What sequences of bytes in FSR result in invalid Unicode entities? > > > > 2. strings in Python *never change*. They are immutable. The + > > operator always copies strings character by character into a new string > > object, even if Python had used UTF-8 internally. If you're doing a lot > > of string concatenations, perhaps you're using the wrong data type. A > > byte buffer might be better for you, where you can stuff utf-8 sequences > > into it to your heart's content. > > > > 3. UTF-8 and UTF-16 encodings, being variable width encodings, mean that > > slicing a string would be very very slow, and that's unacceptable for > > the use cases of python strings. I'm assuming you understand big O > > notation, as you talk of experience in many languages over the years. > > FSR and UTF-32 both are O(1) for slicing and lookups. UTF-8, 16 and any > > variable-width encoding are always O(n). A lot slower! > > > > 4. Unicode is, well, unicode. You seem to hop all over the place from > > talking about code points to bytes to bits, using them all > > interchangeably. And now you seem to be claiming that a particular byte > > encoding standard is by definition unicode (UTF-8). Or at least that's > > how it sounds. And also claim FSR is not compliant with unicode > > standards, which appears to me to be completely false. > > > > Is my understanding of these things wrong? ------ Compare these (a BDFL exemple, where I'using a non-ascii char) Py 3.2 (narrow build) >>> timeit.timeit("a = 'hundred'; 'x' in a") 0.09897159682121348 >>> timeit.timeit("a = 'hundre?'; 'x' in a") 0.09079501961732461 >>> sys.getsizeof('d') 32 >>> sys.getsizeof('?') 32 >>> sys.getsizeof('dd') 34 >>> sys.getsizeof('d?') 34 Py3.3 >>> timeit.timeit("a = 'hundred'; 'x' in a") 0.12183182740848858 >>> timeit.timeit("a = 'hundre?'; 'x' in a") 0.2365732969632326 >>> sys.getsizeof('d') 26 >>> sys.getsizeof('?') 40 >>> sys.getsizeof('dd') 27 >>> sys.getsizeof('d?') 42 Tell me which one seems to be more "unicode compliant"? The goal of Unicode is to handle every char "equaly". Now, the problem: memory. Do not forget that ? la "FSR" mechanism for a non-ascii user is *irrelevant*. As soon as one uses one single non-ascii, your ascii feature is lost. (That why we have all these dedicated coding schemes, utfs included). >>> sys.getsizeof('abc' * 1000 + 'z') 3026 >>> sys.getsizeof('abc' * 1000 + '\U00010010') 12044 A bit secret. The larger a repertoire of characters is, the more bits you needs. Secret #2. You can not escape from this. jmf From python at mrabarnett.plus.com Sun Jul 28 15:44:17 2013 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 28 Jul 2013 20:44:17 +0100 Subject: FSR and unicode compliance - was Re: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> Message-ID: <51F57491.6040008@mrabarnett.plus.com> On 28/07/2013 20:23, wxjmfauth at gmail.com wrote: [snip] > > Compare these (a BDFL exemple, where I'using a non-ascii char) > > Py 3.2 (narrow build) > Why are you using a narrow build of Python 3.2? It doesn't treat all codepoints equally (those outside the BMP can't be stored in one code unit) and, therefore, it isn't "Unicode compliant"! >>>> timeit.timeit("a = 'hundred'; 'x' in a") > 0.09897159682121348 >>>> timeit.timeit("a = 'hundre?'; 'x' in a") > 0.09079501961732461 >>>> sys.getsizeof('d') > 32 >>>> sys.getsizeof('?') > 32 >>>> sys.getsizeof('dd') > 34 >>>> sys.getsizeof('d?') > 34 > > > Py3.3 > >>>> timeit.timeit("a = 'hundred'; 'x' in a") > 0.12183182740848858 >>>> timeit.timeit("a = 'hundre?'; 'x' in a") > 0.2365732969632326 >>>> sys.getsizeof('d') > 26 >>>> sys.getsizeof('?') > 40 >>>> sys.getsizeof('dd') > 27 >>>> sys.getsizeof('d?') > 42 > > Tell me which one seems to be more "unicode compliant"? > The goal of Unicode is to handle every char "equaly". > > Now, the problem: memory. Do not forget that ? la "FSR" > mechanism for a non-ascii user is *irrelevant*. As > soon as one uses one single non-ascii, your ascii feature > is lost. (That why we have all these dedicated coding > schemes, utfs included). > >>>> sys.getsizeof('abc' * 1000 + 'z') > 3026 >>>> sys.getsizeof('abc' * 1000 + '\U00010010') > 12044 > > A bit secret. The larger a repertoire of characters > is, the more bits you needs. > Secret #2. You can not escape from this. > > > jmf > From antoon.pardon at rece.vub.ac.be Sun Jul 28 15:55:50 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Sun, 28 Jul 2013 21:55:50 +0200 Subject: FSR and unicode compliance - was Re: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> Message-ID: <51F57746.3030206@rece.vub.ac.be> Op 28-07-13 21:23, wxjmfauth at gmail.com schreef: > Le dimanche 28 juillet 2013 17:52:47 UTC+2, Michael Torrie a ?crit : >> On 07/27/2013 12:21 PM, wxjmfauth at gmail.com wrote: >> >>> Good point. FSR, nice tool for those who wish to teach >> >>> Unicode. It is not every day, one has such an opportunity. >> >> >> >> I had a long e-mail composed, but decided to chop it down, but still too >> >> long. so I ditched a lot of the context, which jmf also seems to do. >> >> Apologies. >> >> >> >> 1. FSR *is* UTF-32 so it is as unicode compliant as UTF-32, since UTF-32 >> >> is an official encoding. FSR only differs from UTF-32 in that the >> >> padding zeros are stripped off such that it is stored in the most >> >> compact form that can handle all the characters in string, which is >> >> always known at string creation time. Now you can argue many things, >> >> but to say FSR is not unicode compliant is quite a stretch! What >> >> unicode entities or characters cannot be stored in strings using FSR? >> >> What sequences of bytes in FSR result in invalid Unicode entities? >> >> >> >> 2. strings in Python *never change*. They are immutable. The + >> >> operator always copies strings character by character into a new string >> >> object, even if Python had used UTF-8 internally. If you're doing a lot >> >> of string concatenations, perhaps you're using the wrong data type. A >> >> byte buffer might be better for you, where you can stuff utf-8 sequences >> >> into it to your heart's content. >> >> >> >> 3. UTF-8 and UTF-16 encodings, being variable width encodings, mean that >> >> slicing a string would be very very slow, and that's unacceptable for >> >> the use cases of python strings. I'm assuming you understand big O >> >> notation, as you talk of experience in many languages over the years. >> >> FSR and UTF-32 both are O(1) for slicing and lookups. UTF-8, 16 and any >> >> variable-width encoding are always O(n). A lot slower! >> >> >> >> 4. Unicode is, well, unicode. You seem to hop all over the place from >> >> talking about code points to bytes to bits, using them all >> >> interchangeably. And now you seem to be claiming that a particular byte >> >> encoding standard is by definition unicode (UTF-8). Or at least that's >> >> how it sounds. And also claim FSR is not compliant with unicode >> >> standards, which appears to me to be completely false. >> >> >> >> Is my understanding of these things wrong? > > ------ > > Compare these (a BDFL exemple, where I'using a non-ascii char) > > Py 3.2 (narrow build) > >>>> timeit.timeit("a = 'hundred'; 'x' in a") > 0.09897159682121348 >>>> timeit.timeit("a = 'hundre?'; 'x' in a") > 0.09079501961732461 >>>> sys.getsizeof('d') > 32 >>>> sys.getsizeof('?') > 32 >>>> sys.getsizeof('dd') > 34 >>>> sys.getsizeof('d?') > 34 > > > Py3.3 > >>>> timeit.timeit("a = 'hundred'; 'x' in a") > 0.12183182740848858 >>>> timeit.timeit("a = 'hundre?'; 'x' in a") > 0.2365732969632326 >>>> sys.getsizeof('d') > 26 >>>> sys.getsizeof('?') > 40 >>>> sys.getsizeof('dd') > 27 >>>> sys.getsizeof('d?') > 42 > > Tell me which one seems to be more "unicode compliant"? Cant tell, you give no relevant information on which one can decide this question. > The goal of Unicode is to handle every char "equaly". Not to this kind of detail, which is looking at irrelevant implementation details. > Now, the problem: memory. Do not forget that ? la "FSR" > mechanism for a non-ascii user is *irrelevant*. As > soon as one uses one single non-ascii, your ascii feature > is lost. (That why we have all these dedicated coding > schemes, utfs included). So? Why should that trouble me? As far as I understand whether I have an ascii string or not is totally irrelevant to the application programmer. Within the application I just process strings and let the programming environment keep track of these details in a transparant way unless you start looking at things like getsizeof, which gives you implementation details that are mostly irrelevant in deciding whether the behaviour is compliant or not. >>>> sys.getsizeof('abc' * 1000 + 'z') > 3026 >>>> sys.getsizeof('abc' * 1000 + '\U00010010') > 12044 > > A bit secret. The larger a repertoire of characters > is, the more bits you needs. > Secret #2. You can not escape from this. And totally unimportant for deciding complyance. -- Antoon Pardon From steve+comp.lang.python at pearwood.info Sun Jul 28 16:52:16 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 28 Jul 2013 20:52:16 GMT Subject: FSR and unicode compliance - was Re: RE Module Performance References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> Message-ID: <51f5847f$0$29971$c3e8da3$5496439d@news.astraweb.com> On Sun, 28 Jul 2013 12:23:04 -0700, wxjmfauth wrote: > Do not forget that ? la "FSR" mechanism for a non-ascii user is > *irrelevant*. You have been told repeatedly, Python's internals are *full* of ASCII- only strings. py> dir(list) ['__add__', '__class__', '__contains__', '__delattr__', '__delitem__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__iter__', '__le__', '__len__', '__lt__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', 'append', 'clear', 'copy', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort'] There's 45 ASCII-only strings right there, in only one built-in type, out of dozens. There are dozens, hundreds of ASCII-only strings in Python: builtin functions and classes, attributes, exceptions, internal attributes, variable names, and so on. You already know this, and yet you persist in repeating nonsense. -- Steven From wxjmfauth at gmail.com Mon Jul 29 07:43:46 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Mon, 29 Jul 2013 04:43:46 -0700 (PDT) Subject: FSR and unicode compliance - was Re: RE Module Performance In-Reply-To: <51f5847f$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <51f5847f$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: Le dimanche 28 juillet 2013 22:52:16 UTC+2, Steven D'Aprano a ?crit?: > On Sun, 28 Jul 2013 12:23:04 -0700, wxjmfauth wrote: > > > > > Do not forget that ? la "FSR" mechanism for a non-ascii user is > > > *irrelevant*. > > > > You have been told repeatedly, Python's internals are *full* of ASCII- > > only strings. > > > > py> dir(list) > > ['__add__', '__class__', '__contains__', '__delattr__', '__delitem__', > > '__dir__', '__doc__', '__eq__', '__format__', '__ge__', > > '__getattribute__', '__getitem__', '__gt__', '__hash__', '__iadd__', > > '__imul__', '__init__', '__iter__', '__le__', '__len__', '__lt__', > > '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', > > '__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__', > > '__sizeof__', '__str__', '__subclasshook__', 'append', 'clear', 'copy', > > 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort'] > > > > There's 45 ASCII-only strings right there, in only one built-in type, out > > of dozens. There are dozens, hundreds of ASCII-only strings in Python: > > builtin functions and classes, attributes, exceptions, internal > > attributes, variable names, and so on. > > > > You already know this, and yet you persist in repeating nonsense. > > > > > > -- > > Steven 3.2 >>> timeit.timeit("r = dir(list)") 22.300465007102908 3.3 >>> timeit.timeit("r = dir(list)") 27.13981129541519 For the record, I do not put your example to contradict you. I was expecting such a result even before testing. Now, if you do not understand why, you do not understand. There nothing wrong. jmf From rosuav at gmail.com Mon Jul 29 07:57:47 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 29 Jul 2013 12:57:47 +0100 Subject: FSR and unicode compliance - was Re: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <51f5847f$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Mon, Jul 29, 2013 at 12:43 PM, wrote: > Le dimanche 28 juillet 2013 22:52:16 UTC+2, Steven D'Aprano a ?crit : > 3.2 >>>> timeit.timeit("r = dir(list)") > 22.300465007102908 > > 3.3 >>>> timeit.timeit("r = dir(list)") > 27.13981129541519 3.2: >>> len(dir(list)) 42 3.3: >>> len(dir(list)) 45 Wonder if that might maybe have an impact on the timings. ChrisA From modelnine at modelnine.org Mon Jul 29 08:06:12 2013 From: modelnine at modelnine.org (Heiko Wundram) Date: Mon, 29 Jul 2013 14:06:12 +0200 Subject: FSR and unicode compliance - was Re: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <51f5847f$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51F65AB4.8080309@modelnine.org> Am 29.07.2013 13:43, schrieb wxjmfauth at gmail.com: > 3.2 >>>> timeit.timeit("r = dir(list)") > 22.300465007102908 > > 3.3 >>>> timeit.timeit("r = dir(list)") > 27.13981129541519 > > For the record, I do not put your example to contradict > you. I was expecting such a result even before testing. > > Now, if you do not understand why, you do not understand. > There nothing wrong. Please give a single *proof* (not your gut feeling) that this is related to the FSR, and not rather due to other side-effects such as changes in how dir() works or (as Chris pointed out) due to more members on the list type in 3.3. If you can't or won't give that proof, there's no sense in continuing the discussion. -- --- Heiko. From devyncjohnson at gmail.com Mon Jul 29 08:43:03 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Mon, 29 Jul 2013 08:43:03 -0400 Subject: FSR and unicode compliance - was Re: RE Module Performance In-Reply-To: <51F65AB4.8080309@modelnine.org> References: <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <51f5847f$0$29971$c3e8da3$5496439d@news.astraweb.com> <51F65AB4.8080309@modelnine.org> Message-ID: <51F66357.7050300@Gmail.com> On 07/29/2013 08:06 AM, Heiko Wundram wrote: > Am 29.07.2013 13:43, schrieb wxjmfauth at gmail.com: >> 3.2 >>>>> timeit.timeit("r = dir(list)") >> 22.300465007102908 >> >> 3.3 >>>>> timeit.timeit("r = dir(list)") >> 27.13981129541519 >> >> For the record, I do not put your example to contradict >> you. I was expecting such a result even before testing. >> >> Now, if you do not understand why, you do not understand. >> There nothing wrong. > > Please give a single *proof* (not your gut feeling) that this is > related to the FSR, and not rather due to other side-effects such as > changes in how dir() works or (as Chris pointed out) due to more > members on the list type in 3.3. If you can't or won't give that > proof, there's no sense in continuing the discussion. > Wow! The RE Module thread I created is evolving into Unicode topics. That thread grew up so fast! DCJ From wxjmfauth at gmail.com Mon Jul 29 08:56:29 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Mon, 29 Jul 2013 05:56:29 -0700 (PDT) Subject: FSR and unicode compliance - was Re: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <51f5847f$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: Le lundi 29 juillet 2013 13:57:47 UTC+2, Chris Angelico a ?crit?: > On Mon, Jul 29, 2013 at 12:43 PM, wrote: > > > Le dimanche 28 juillet 2013 22:52:16 UTC+2, Steven D'Aprano a ?crit : > > > 3.2 > > >>>> timeit.timeit("r = dir(list)") > > > 22.300465007102908 > > > > > > 3.3 > > >>>> timeit.timeit("r = dir(list)") > > > 27.13981129541519 > > > > 3.2: > > >>> len(dir(list)) > > 42 > > > > 3.3: > > >>> len(dir(list)) > > 45 > > > > Wonder if that might maybe have an impact on the timings. > > > > ChrisA Good point. I stupidely forgot this. jmf From wxjmfauth at gmail.com Mon Jul 29 10:20:54 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Mon, 29 Jul 2013 07:20:54 -0700 (PDT) Subject: FSR and unicode compliance - was Re: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <51f5847f$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <8babe5b2-779d-49c3-b7e8-addccbadd660@googlegroups.com> Le lundi 29 juillet 2013 13:57:47 UTC+2, Chris Angelico a ?crit?: > On Mon, Jul 29, 2013 at 12:43 PM, wrote: > > > Le dimanche 28 juillet 2013 22:52:16 UTC+2, Steven D'Aprano a ?crit : > > > 3.2 > > >>>> timeit.timeit("r = dir(list)") > > > 22.300465007102908 > > > > > > 3.3 > > >>>> timeit.timeit("r = dir(list)") > > > 27.13981129541519 > > > > 3.2: > > >>> len(dir(list)) > > 42 > > > > 3.3: > > >>> len(dir(list)) > > 45 > > > > Wonder if that might maybe have an impact on the timings. > > > > ChrisA -------- class C: a = 'abc' b = 'def' def aaa(self): pass def bbb(self): pass def ccc(self): pass if __name__ == '__main__': import timeit print(timeit.timeit("r = dir(C)", setup="from __main__ import C")) >c:\python32\pythonw -u "timitmod.py" 15.258061416225663 >Exit code: 0 >c:\Python33\pythonw -u "timitmod.py" 17.052203122286194 >Exit code: 0 jmf From rosuav at gmail.com Mon Jul 29 10:49:34 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 29 Jul 2013 15:49:34 +0100 Subject: FSR and unicode compliance - was Re: RE Module Performance In-Reply-To: <8babe5b2-779d-49c3-b7e8-addccbadd660@googlegroups.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <51f5847f$0$29971$c3e8da3$5496439d@news.astraweb.com> <8babe5b2-779d-49c3-b7e8-addccbadd660@googlegroups.com> Message-ID: On Mon, Jul 29, 2013 at 3:20 PM, wrote: >>c:\python32\pythonw -u "timitmod.py" > 15.258061416225663 >>Exit code: 0 >>c:\Python33\pythonw -u "timitmod.py" > 17.052203122286194 >>Exit code: 0 >>> len(dir(C)) Did you even think to check that before you posted timings? ChrisA From wxjmfauth at gmail.com Mon Jul 29 12:31:01 2013 From: wxjmfauth at gmail.com (wxjmfauth at gmail.com) Date: Mon, 29 Jul 2013 09:31:01 -0700 (PDT) Subject: FSR and unicode compliance - was Re: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <4f1067f6-bc99-42ad-9166-37fb228b90e8@googlegroups.com> <51f14395$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f15e03$0$29971$c3e8da3$5496439d@news.astraweb.com> <8203e802-9dc5-44c5-9547-6e1947ee224b@googlegroups.com> <51f5847f$0$29971$c3e8da3$5496439d@news.astraweb.com> <8babe5b2-779d-49c3-b7e8-addccbadd660@googlegroups.com> Message-ID: <6ebcd6c1-ea5e-492b-a3ea-0541fc1d63cb@googlegroups.com> Le lundi 29 juillet 2013 16:49:34 UTC+2, Chris Angelico a ?crit?: > On Mon, Jul 29, 2013 at 3:20 PM, wrote: > > >>c:\python32\pythonw -u "timitmod.py" > > > 15.258061416225663 > > >>Exit code: 0 > > >>c:\Python33\pythonw -u "timitmod.py" > > > 17.052203122286194 > > >>Exit code: 0 > > > > >>> len(dir(C)) > > > > Did you even think to check that before you posted timings? > > > > ChrisA Boum, no! the diff is one. I have however noticed, I can increase the number of attributes (ascii), the timing differences is very well marked. I do not draw conclusions. Such a factor for one unit.... jmf From rosuav at gmail.com Fri Jul 12 12:21:29 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 13 Jul 2013 02:21:29 +1000 Subject: RE Module Performance In-Reply-To: <51DFDE65.5040001@Gmail.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> Message-ID: On Fri, Jul 12, 2013 at 8:45 PM, Devyn Collier Johnson wrote: > Could you explain what you mean? What and where is the new Flexible String > Representation? (You're top-posting again. Please put your text underneath what you're responding to - it helps maintain flow and structure.) Python versions up to and including 3.2 came in two varieties: narrow builds (commonly found on Windows) and wide builds (commonly found on Linux). Narrow builds internally represented Unicode strings in UTF-16, while wide builds used UTF-32. This is a problem, because it means that taking a program from one to another actually changes its behaviour: Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15) [GCC 4.4.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> len(u"\U00012345") 1 Python 2.7.4 (default, Apr 6 2013, 19:54:46) [MSC v.1500 32 bit (Intel)] on win32 >>> len(u"\U00012345") 2 In fact, the narrow builds are flat-out buggy, because you can put something in as a single character that simply isn't a single character. You can then pull that out as two characters and make a huge mess of things: >>> s=u"\U00012345" >>> s[0] u'\ud808' >>> s[1] u'\udf45' *Any* string indexing will be broken if there is a single character >U+FFFF ahead of it in the string. Now, this problem is not unique to Python. Heaps of other languages have the same issue, the same buggy behaviour, the same compromises. What's special about Python is that it actually managed to come back from that problem. (Google's V8 JavaScript engine, for instance, is stuck with it, because the ECMAScript specification demands UTF-16. I asked on an ECMAScript list and was told "can't change that, it'd break code". So it's staying buggy.) There are a number of languages that take the Texan RAM-guzzling approach of storing all strings in UTF-32; Python (since version 3.3) is among a *very* small number of languages that store strings in multiple different ways according to their content. That's described in PEP 393 [1], titled "Flexible String Representation". It details a means whereby a Python string will be represented in, effectively, UTF-32 with some of the leading zero bytes elided. Or if you prefer, in either Latin-1, UCS-2, or UCS-4, whichever's the smallest it can fit in. The difference between a string stored one-byte-per-character and a string stored four-bytes-per-character is almost invisible to a Python script; you can find out by checking the string's memory usage, but otherwise you don't need to worry about it. Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600 32 bit (Intel)] on win32 >>> sys.getsizeof("asdfasdfasdfasd") 40 >>> sys.getsizeof("asdfasdfasdfasdf") 41 Adding another character adds another 1 byte. (There's quite a bit of overhead for small strings - GC headers and such - but it gets dwarfed by the actual content after a while.) >>> sys.getsizeof("\u1000sdfasdfasdfasd") 68 >>> sys.getsizeof("\u1000sdfasdfasdfasdf") 70 Two bytes to add another character. >>> sys.getsizeof("\U00010001sdfasdfasdfasd") 100 >>> sys.getsizeof("\U00010001sdfasdfasdfasdf") 104 Four bytes. It uses only what it needs. Strings in Python are immutable, so there's no need to worry about up-grading or down-grading a string; there are a few optimizations that can't be done, but they're fairly trivial. Look, I'll pull a jmf and find a microbenchmark that makes 3.3 look worse: 2.7.4: >>> timeit.repeat('a=u"A"*100; a+=u"\u1000"') [0.8175005482540385, 0.789617954237201, 0.8152240019332098] >>> timeit.repeat('a=u"A"*100; a+=u"a"') [0.8088905154146744, 0.8123691698246631, 0.8172558244134365] 3.3.0: >>> timeit.repeat('a=u"A"*100; a+=u"\u1000"') [0.9623714745976031, 0.970628669281723, 0.9696310564468149] >>> timeit.repeat('a=u"A"*100; a+=u"a"') [0.7017891938739922, 0.7024725209339522, 0.6989539173082449] See? It's clearly worse on the newer Python! But actually, this is an extremely unusual situation, and 3.3 outperforms 2.7 on the more common case (where the two strings are of the same width). Python's PEP 393 strings are following the same sort of model as the native string type in a semantically-similar but syntactically-different language, Pike. In Pike (also free software, like Python), the string type can be indexed character by character, and each character can be anything in the Unicode range; and just as in Python 3.3, memory usage goes up by just one byte if every character in the string fits inside 8 bits. So it's not as if this is an untested notion; Pike has been running like this for years (I don't know how long it's had this functionality, but it won't be more than 18 years as Unicode didn't have multiple planes until 1996), and performance has been *just fine* for all that time. Pike tends to be run on servers, so memory usage and computation speed translate fairly directly into TPS. And there are some sizeable commercial entities using and developing Pike, so if the flexible string representation had turned out to be a flop, someone would have put in the coding time to rewrite it by now. And yet, despite all these excellent reasons for moving to this way of doing strings, jmf still sees his microbenchmarks as more important, and so he jumps in on threads like this to whine about how Python 3.3 is somehow US-centric because it more efficiently handles the entire Unicode range. I'd really like to take some highlights from Python and Pike and start recommending that other languages take up the ideas, but to be honest, I hesitate to inflict jmf on them all. ECMAScript may have the right idea after all - stay with UTF-16 and avoid answering jmf's stupid objections every week. [1] http://www.python.org/dev/peps/pep-0393/ ChrisA From devyncjohnson at gmail.com Fri Jul 12 13:58:29 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Fri, 12 Jul 2013 13:58:29 -0400 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> Message-ID: <51E043C5.1010804@Gmail.com> On 07/12/2013 12:21 PM, Chris Angelico wrote: > On Fri, Jul 12, 2013 at 8:45 PM, Devyn Collier Johnson > wrote: >> Could you explain what you mean? What and where is the new Flexible String >> Representation? > (You're top-posting again. Please put your text underneath what you're > responding to - it helps maintain flow and structure.) > > Python versions up to and including 3.2 came in two varieties: narrow > builds (commonly found on Windows) and wide builds (commonly found on > Linux). Narrow builds internally represented Unicode strings in > UTF-16, while wide builds used UTF-32. This is a problem, because it > means that taking a program from one to another actually changes its > behaviour: > > Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15) > [GCC 4.4.1] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> len(u"\U00012345") > 1 > > Python 2.7.4 (default, Apr 6 2013, 19:54:46) [MSC v.1500 32 bit > (Intel)] on win32 >>>> len(u"\U00012345") > 2 > > In fact, the narrow builds are flat-out buggy, because you can put > something in as a single character that simply isn't a single > character. You can then pull that out as two characters and make a > huge mess of things: > >>>> s=u"\U00012345" >>>> s[0] > u'\ud808' >>>> s[1] > u'\udf45' > > *Any* string indexing will be broken if there is a single character >> U+FFFF ahead of it in the string. > Now, this problem is not unique to Python. Heaps of other languages > have the same issue, the same buggy behaviour, the same compromises. > What's special about Python is that it actually managed to come back > from that problem. (Google's V8 JavaScript engine, for instance, is > stuck with it, because the ECMAScript specification demands UTF-16. I > asked on an ECMAScript list and was told "can't change that, it'd > break code". So it's staying buggy.) > > There are a number of languages that take the Texan RAM-guzzling > approach of storing all strings in UTF-32; Python (since version 3.3) > is among a *very* small number of languages that store strings in > multiple different ways according to their content. That's described > in PEP 393 [1], titled "Flexible String Representation". It details a > means whereby a Python string will be represented in, effectively, > UTF-32 with some of the leading zero bytes elided. Or if you prefer, > in either Latin-1, UCS-2, or UCS-4, whichever's the smallest it can > fit in. The difference between a string stored one-byte-per-character > and a string stored four-bytes-per-character is almost invisible to a > Python script; you can find out by checking the string's memory usage, > but otherwise you don't need to worry about it. > > Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600 > 32 bit (Intel)] on win32 >>>> sys.getsizeof("asdfasdfasdfasd") > 40 >>>> sys.getsizeof("asdfasdfasdfasdf") > 41 > > Adding another character adds another 1 byte. (There's quite a bit of > overhead for small strings - GC headers and such - but it gets dwarfed > by the actual content after a while.) > >>>> sys.getsizeof("\u1000sdfasdfasdfasd") > 68 >>>> sys.getsizeof("\u1000sdfasdfasdfasdf") > 70 > > Two bytes to add another character. > >>>> sys.getsizeof("\U00010001sdfasdfasdfasd") > 100 >>>> sys.getsizeof("\U00010001sdfasdfasdfasdf") > 104 > > Four bytes. It uses only what it needs. > > Strings in Python are immutable, so there's no need to worry about > up-grading or down-grading a string; there are a few optimizations > that can't be done, but they're fairly trivial. Look, I'll pull a jmf > and find a microbenchmark that makes 3.3 look worse: > > 2.7.4: >>>> timeit.repeat('a=u"A"*100; a+=u"\u1000"') > [0.8175005482540385, 0.789617954237201, 0.8152240019332098] >>>> timeit.repeat('a=u"A"*100; a+=u"a"') > [0.8088905154146744, 0.8123691698246631, 0.8172558244134365] > > 3.3.0: >>>> timeit.repeat('a=u"A"*100; a+=u"\u1000"') > [0.9623714745976031, 0.970628669281723, 0.9696310564468149] >>>> timeit.repeat('a=u"A"*100; a+=u"a"') > [0.7017891938739922, 0.7024725209339522, 0.6989539173082449] > > See? It's clearly worse on the newer Python! But actually, this is an > extremely unusual situation, and 3.3 outperforms 2.7 on the more > common case (where the two strings are of the same width). > > Python's PEP 393 strings are following the same sort of model as the > native string type in a semantically-similar but > syntactically-different language, Pike. In Pike (also free software, > like Python), the string type can be indexed character by character, > and each character can be anything in the Unicode range; and just as > in Python 3.3, memory usage goes up by just one byte if every > character in the string fits inside 8 bits. So it's not as if this is > an untested notion; Pike has been running like this for years (I don't > know how long it's had this functionality, but it won't be more than > 18 years as Unicode didn't have multiple planes until 1996), and > performance has been *just fine* for all that time. Pike tends to be > run on servers, so memory usage and computation speed translate fairly > directly into TPS. And there are some sizeable commercial entities > using and developing Pike, so if the flexible string representation > had turned out to be a flop, someone would have put in the coding time > to rewrite it by now. > > And yet, despite all these excellent reasons for moving to this way of > doing strings, jmf still sees his microbenchmarks as more important, > and so he jumps in on threads like this to whine about how Python 3.3 > is somehow US-centric because it more efficiently handles the entire > Unicode range. I'd really like to take some highlights from Python and > Pike and start recommending that other languages take up the ideas, > but to be honest, I hesitate to inflict jmf on them all. ECMAScript > may have the right idea after all - stay with UTF-16 and avoid > answering jmf's stupid objections every week. > > [1] http://www.python.org/dev/peps/pep-0393/ > > ChrisA Thanks for the thorough response. I learned a lot. You should write articles on Python. I plan to spend some time optimizing the re.py module for Unix systems. I would love to amp up my programs that use that module. Devyn Collier Johnson From timothy.c.delaney at gmail.com Fri Jul 12 18:16:05 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Sat, 13 Jul 2013 08:16:05 +1000 Subject: RE Module Performance In-Reply-To: <51E043C5.1010804@Gmail.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <51E043C5.1010804@Gmail.com> Message-ID: On 13 July 2013 03:58, Devyn Collier Johnson wrote: > > Thanks for the thorough response. I learned a lot. You should write > articles on Python. > I plan to spend some time optimizing the re.py module for Unix systems. I > would love to amp up my programs that use that module. If you are finding that regular expressions are taking too much time, have a look at the https://pypi.python.org/pypi/re2/ and https://pypi.python.org/pypi/regex/2013-06-26 modules to see if they already give you enough of a speedup. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Fri Jul 12 19:16:06 2013 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 13 Jul 2013 00:16:06 +0100 Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <51E043C5.1010804@Gmail.com> Message-ID: <51E08E36.5030108@mrabarnett.plus.com> On 12/07/2013 23:16, Tim Delaney wrote: > On 13 July 2013 03:58, Devyn Collier Johnson > wrote: > > > Thanks for the thorough response. I learned a lot. You should write > articles on Python. > I plan to spend some time optimizing the re.py module for Unix > systems. I would love to amp up my programs that use that module. > > > If you are finding that regular expressions are taking too much time, > have a look at the https://pypi.python.org/pypi/re2/ and > https://pypi.python.org/pypi/regex/2013-06-26 modules to see if they > already give you enough of a speedup. > FYI, you're better off going to http://pypi.python.org/pypi/regex because that will take you to the latest version. From timothy.c.delaney at gmail.com Sat Jul 13 15:34:27 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Sun, 14 Jul 2013 05:34:27 +1000 Subject: RE Module Performance In-Reply-To: <51E08E36.5030108@mrabarnett.plus.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <51E043C5.1010804@Gmail.com> <51E08E36.5030108@mrabarnett.plus.com> Message-ID: On 13 July 2013 09:16, MRAB wrote: > On 12/07/2013 23:16, Tim Delaney wrote: > >> On 13 July 2013 03:58, Devyn Collier Johnson > >> wrote: >> >> >> Thanks for the thorough response. I learned a lot. You should write >> articles on Python. >> I plan to spend some time optimizing the re.py module for Unix >> systems. I would love to amp up my programs that use that module. >> >> >> If you are finding that regular expressions are taking too much time, >> have a look at the https://pypi.python.org/pypi/**re2/and >> https://pypi.python.org/pypi/**regex/2013-06-26modules to see if they >> already give you enough of a speedup. >> >> FYI, you're better off going to http://pypi.python.org/pypi/**regex > because that will take you to the latest version. Absolutely - what was I thinking? Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From devyncjohnson at gmail.com Tue Jul 16 06:30:33 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Tue, 16 Jul 2013 06:30:33 -0400 Subject: RE Module Performance In-Reply-To: <51E08E36.5030108@mrabarnett.plus.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <51E043C5.1010804@Gmail.com> <51E08E36.5030108@mrabarnett.plus.com> Message-ID: <51E520C9.6070407@Gmail.com> Am 07/12/2013 07:16 PM, schrieb MRAB: > On 12/07/2013 23:16, Tim Delaney wrote: >> On 13 July 2013 03:58, Devyn Collier Johnson > > wrote: >> >> >> Thanks for the thorough response. I learned a lot. You should write >> articles on Python. >> I plan to spend some time optimizing the re.py module for Unix >> systems. I would love to amp up my programs that use that module. >> >> >> If you are finding that regular expressions are taking too much time, >> have a look at the https://pypi.python.org/pypi/re2/ and >> https://pypi.python.org/pypi/regex/2013-06-26 modules to see if they >> already give you enough of a speedup. >> > FYI, you're better off going to http://pypi.python.org/pypi/regex > because that will take you to the latest version. Thank you everyone for the suggestions. I have not tried them yet. Devyn Collier Johnson From dihedral88888 at gmail.com Thu Jul 18 16:17:08 2013 From: dihedral88888 at gmail.com (88888 Dihedral) Date: Thu, 18 Jul 2013 13:17:08 -0700 (PDT) Subject: RE Module Performance In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <51E043C5.1010804@Gmail.com> <51E08E36.5030108@mrabarnett.plus.com> Message-ID: Devyn Collier Johnson? 2013?7?16????UTC+8??6?30?33???? > Am 07/12/2013 07:16 PM, schrieb MRAB: > > > On 12/07/2013 23:16, Tim Delaney wrote: > > >> On 13 July 2013 03:58, Devyn Collier Johnson > >> > wrote: > > >> > > >> > > >> Thanks for the thorough response. I learned a lot. You should write > > >> articles on Python. > > >> I plan to spend some time optimizing the re.py module for Unix > > >> systems. I would love to amp up my programs that use that module. > > >> > > >> > > >> If you are finding that regular expressions are taking too much time, > > >> have a look at the https://pypi.python.org/pypi/re2/ and > > >> https://pypi.python.org/pypi/regex/2013-06-26 modules to see if they > > >> already give you enough of a speedup. > > >> > > > FYI, you're better off going to http://pypi.python.org/pypi/regex > > > because that will take you to the latest version. > > Thank you everyone for the suggestions. I have not tried them yet. > > > > Devyn Collier Johnson I was thinking to decompose RE patterns into string matching formats of various strings in some formats. Anyway that involves some compiler techniques. From steve+comp.lang.python at pearwood.info Sat Jul 13 01:37:46 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 13 Jul 2013 05:37:46 GMT Subject: RE Module Performance References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> Message-ID: <51e0e7aa$0$9505$c3e8da3$5496439d@news.astraweb.com> On Fri, 12 Jul 2013 13:58:29 -0400, Devyn Collier Johnson wrote: > I plan to spend some time optimizing the re.py module for Unix systems. > I would love to amp up my programs that use that module. In my experience, often the best way to optimize a regex is to not use it at all. [steve at ando ~]$ python -m timeit -s "import re" \ > -s "data = 'a'*100+'b'" \ > "if re.search('b', data): pass" 100000 loops, best of 3: 2.77 usec per loop [steve at ando ~]$ python -m timeit -s "data = 'a'*100+'b'" \ > "if 'b' in data: pass" 1000000 loops, best of 3: 0.219 usec per loop In Python, we often use plain string operations instead of regex-based solutions for basic tasks. Regexes are a 10lb sledge hammer. Don't use them for cracking peanuts. -- Steven From dihedral88888 at gmail.com Sun Jul 14 14:17:06 2013 From: dihedral88888 at gmail.com (88888 Dihedral) Date: Sun, 14 Jul 2013 11:17:06 -0700 (PDT) Subject: RE Module Performance In-Reply-To: <51e0e7aa$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <51e0e7aa$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: <9a97b618-3e80-4149-9155-14fb210a0758@googlegroups.com> On Saturday, July 13, 2013 1:37:46 PM UTC+8, Steven D'Aprano wrote: > On Fri, 12 Jul 2013 13:58:29 -0400, Devyn Collier Johnson wrote: > > > > > I plan to spend some time optimizing the re.py module for Unix systems. > > > I would love to amp up my programs that use that module. > > > > In my experience, often the best way to optimize a regex is to not use it > > at all. > > > > [steve at ando ~]$ python -m timeit -s "import re" \ > > > -s "data = 'a'*100+'b'" \ > > > "if re.search('b', data): pass" > > 100000 loops, best of 3: 2.77 usec per loop > > > > [steve at ando ~]$ python -m timeit -s "data = 'a'*100+'b'" \ > > > "if 'b' in data: pass" > > 1000000 loops, best of 3: 0.219 usec per loop > > > > In Python, we often use plain string operations instead of regex-based > > solutions for basic tasks. Regexes are a 10lb sledge hammer. Don't use > > them for cracking peanuts. > > > > > > > > -- > > Steven OK, lets talk about the indexed search algorithms of a character streamor strig which can be buffered and indexed randomly for RW operations but faster in sequential block RW operations after some pre-processing. This was solved long time ago in the suffix array or suffix tree part and summarized in the famous BWT paper in 199X. Do we want volunteers to speed up search operations in the string module in Python? From devyncjohnson at gmail.com Mon Jul 15 06:06:06 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Mon, 15 Jul 2013 06:06:06 -0400 Subject: RE Module Performance In-Reply-To: <9a97b618-3e80-4149-9155-14fb210a0758@googlegroups.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <51e0e7aa$0$9505$c3e8da3$5496439d@news.astraweb.com> <9a97b618-3e80-4149-9155-14fb210a0758@googlegroups.com> Message-ID: <51E3C98E.5070604@Gmail.com> On 07/14/2013 02:17 PM, 88888 Dihedral wrote: > On Saturday, July 13, 2013 1:37:46 PM UTC+8, Steven D'Aprano wrote: >> On Fri, 12 Jul 2013 13:58:29 -0400, Devyn Collier Johnson wrote: >> >> >> >>> I plan to spend some time optimizing the re.py module for Unix systems. >>> I would love to amp up my programs that use that module. >> >> >> In my experience, often the best way to optimize a regex is to not use it >> >> at all. >> >> >> >> [steve at ando ~]$ python -m timeit -s "import re" \ >> >>> -s "data = 'a'*100+'b'" \ >>> "if re.search('b', data): pass" >> 100000 loops, best of 3: 2.77 usec per loop >> >> >> >> [steve at ando ~]$ python -m timeit -s "data = 'a'*100+'b'" \ >> >>> "if 'b' in data: pass" >> 1000000 loops, best of 3: 0.219 usec per loop >> >> >> >> In Python, we often use plain string operations instead of regex-based >> >> solutions for basic tasks. Regexes are a 10lb sledge hammer. Don't use >> >> them for cracking peanuts. >> >> >> >> >> >> >> >> -- >> >> Steven > OK, lets talk about the indexed search algorithms of > a character streamor strig which can be buffered and > indexed randomly for RW operations but faster in sequential > block RW operations after some pre-processing. > > This was solved long time ago in the suffix array or > suffix tree part and summarized in the famous BWT paper in 199X. > > Do we want volunteers to speed up > search operations in the string module in Python? It would be nice if someone could speed it up. From steve+comp.lang.python at pearwood.info Mon Jul 15 08:36:13 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 15 Jul 2013 12:36:13 GMT Subject: RE Module Performance References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <51e0e7aa$0$9505$c3e8da3$5496439d@news.astraweb.com> <9a97b618-3e80-4149-9155-14fb210a0758@googlegroups.com> Message-ID: <51e3ecbd$0$9505$c3e8da3$5496439d@news.astraweb.com> On Mon, 15 Jul 2013 06:06:06 -0400, Devyn Collier Johnson wrote: > On 07/14/2013 02:17 PM, 88888 Dihedral wrote: [...] >> Do we want volunteers to speed up >> search operations in the string module in Python? > > It would be nice if someone could speed it up. Devyn, 88888 Dihedral is our resident bot, not a human being. Nobody knows who controls it, and why they are running it, but we are pretty certain that it is a bot responding mechanically to keywords in people's posts. It's a very clever bot, but still a bot. About one post in four is meaningless jargon, the other three are relevant enough to fool people into thinking that maybe it is a human being. It had me fooled for a long time. -- Steven From devyncjohnson at gmail.com Mon Jul 15 08:52:48 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Mon, 15 Jul 2013 08:52:48 -0400 Subject: Dihedral In-Reply-To: <51e3ecbd$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <51e0e7aa$0$9505$c3e8da3$5496439d@news.astraweb.com> <9a97b618-3e80-4149-9155-14fb210a0758@googlegroups.com> <51e3ecbd$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51E3F0A0.9060001@Gmail.com> On 07/15/2013 08:36 AM, Steven D'Aprano wrote: > On Mon, 15 Jul 2013 06:06:06 -0400, Devyn Collier Johnson wrote: > >> On 07/14/2013 02:17 PM, 88888 Dihedral wrote: > [...] >>> Do we want volunteers to speed up >>> search operations in the string module in Python? >> It would be nice if someone could speed it up. > Devyn, > > 88888 Dihedral is our resident bot, not a human being. Nobody knows who > controls it, and why they are running it, but we are pretty certain that > it is a bot responding mechanically to keywords in people's posts. > > It's a very clever bot, but still a bot. About one post in four is > meaningless jargon, the other three are relevant enough to fool people > into thinking that maybe it is a human being. It had me fooled for a long > time. > > > Wow! Our mailing list has a pet bot. I bet other mailing lists are so jealous of us. Who ever created Dihedral is a genius! Artificial Intelligence developers put chatbots on mailing lists so that the program can learn. I use Python3 to program AI applications. If you see my Launchpad account, you will see my two AI projects - Neobot and Novabot. (https://launchpad.net/neobot Neo and Nova are still unstable) AI developers let their bots loose on the Internet to learn from people. Dihedral is learning from us. Dihedral only responses when it feels it has sufficient knowledge on the topic. Chatbots want to appear human. That is their goal. We should feel honored that Dihedral's botmaster feels that this mailinglist would benefit the development of Dihedral's knowledge. Devyn Collier Johnson From joel.goldstick at gmail.com Mon Jul 15 09:03:25 2013 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Mon, 15 Jul 2013 09:03:25 -0400 Subject: Dihedral In-Reply-To: <51E3F0A0.9060001@Gmail.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <51e0e7aa$0$9505$c3e8da3$5496439d@news.astraweb.com> <9a97b618-3e80-4149-9155-14fb210a0758@googlegroups.com> <51e3ecbd$0$9505$c3e8da3$5496439d@news.astraweb.com> <51E3F0A0.9060001@Gmail.com> Message-ID: On Mon, Jul 15, 2013 at 8:52 AM, Devyn Collier Johnson < devyncjohnson at gmail.com> wrote: > > On 07/15/2013 08:36 AM, Steven D'Aprano wrote: > >> On Mon, 15 Jul 2013 06:06:06 -0400, Devyn Collier Johnson wrote: >> >> On 07/14/2013 02:17 PM, 88888 Dihedral wrote: >>> >> [...] >> >>> Do we want volunteers to speed up >>>> search operations in the string module in Python? >>>> >>> It would be nice if someone could speed it up. >>> >> Devyn, >> >> 88888 Dihedral is our resident bot, not a human being. Nobody knows who >> controls it, and why they are running it, but we are pretty certain that >> it is a bot responding mechanically to keywords in people's posts. >> >> It's a very clever bot, but still a bot. About one post in four is >> meaningless jargon, the other three are relevant enough to fool people >> into thinking that maybe it is a human being. It had me fooled for a long >> time. >> >> >> >> Wow! Our mailing list has a pet bot. I bet other mailing lists are so > jealous of us. Who ever created Dihedral is a genius! > > Artificial Intelligence developers put chatbots on mailing lists so that > the program can learn. I use Python3 to program AI applications. If you see > my Launchpad account, you will see my two AI projects - Neobot and Novabot. > (https://launchpad.net/neobot Neo and Nova are still unstable) AI > developers let their bots loose on the Internet to learn from people. > Dihedral is learning from us. Dihedral only responses when it feels it has > sufficient knowledge on the topic. Chatbots want to appear human. That is > their goal. We should feel honored that Dihedral's botmaster feels that > this mailinglist would benefit the development of Dihedral's knowledge. > > Devyn Collier Johnson > -- > http://mail.python.org/**mailman/listinfo/python-list > I particularly enjoy the misspellings, that seem to be such a human quality on email messages! -- Joel Goldstick http://joelgoldstick.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From wayne at waynewerner.com Mon Jul 15 18:43:06 2013 From: wayne at waynewerner.com (Wayne Werner) Date: Mon, 15 Jul 2013 17:43:06 -0500 (CDT) Subject: Dihedral In-Reply-To: <51E3F0A0.9060001@Gmail.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <51e0e7aa$0$9505$c3e8da3$5496439d@news.astraweb.com> <9a97b618-3e80-4149-9155-14fb210a0758@googlegroups.com> <51e3ecbd$0$9505$c3e8da3$5496439d@news.astraweb.com> <51E3F0A0.9060001@Gmail.com> Message-ID: On Mon, 15 Jul 2013, Devyn Collier Johnson wrote: > > On 07/15/2013 08:36 AM, Steven D'Aprano wrote: >> On Mon, 15 Jul 2013 06:06:06 -0400, Devyn Collier Johnson wrote: >> >>> On 07/14/2013 02:17 PM, 88888 Dihedral wrote: >> [...] >>>> Do we want volunteers to speed up >>>> search operations in the string module in Python? >>> It would be nice if someone could speed it up. >> Devyn, >> >> 88888 Dihedral is our resident bot, not a human being. Nobody knows who >> controls it, and why they are running it, but we are pretty certain that >> it is a bot responding mechanically to keywords in people's posts. >> >> It's a very clever bot, but still a bot. About one post in four is >> meaningless jargon, the other three are relevant enough to fool people >> into thinking that maybe it is a human being. It had me fooled for a long >> time. >> >> >> > Wow! Our mailing list has a pet bot. I bet other mailing lists are so jealous > of us. Who ever created Dihedral is a genius! > > Artificial Intelligence developers put chatbots on mailing lists so that the > program can learn. I use Python3 to program AI applications. If you see my > Launchpad account, you will see my two AI projects - Neobot and Novabot. > (https://launchpad.net/neobot Neo and Nova are still unstable) AI developers > let their bots loose on the Internet to learn from people. Dihedral is > learning from us. Dihedral only responses when it feels it has sufficient > knowledge on the topic. Chatbots want to appear human. That is their goal. We > should feel honored that Dihedral's botmaster feels that this mailinglist > would benefit the development of Dihedral's knowledge. Are *you* a bot? ~_^ That post felt surprisingly like Dihedral... -W From fabiosantosart at gmail.com Mon Jul 15 18:54:02 2013 From: fabiosantosart at gmail.com (=?ISO-8859-1?Q?F=E1bio_Santos?=) Date: Mon, 15 Jul 2013 23:54:02 +0100 Subject: Dihedral In-Reply-To: <51E3F0A0.9060001@Gmail.com> References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <51e0e7aa$0$9505$c3e8da3$5496439d@news.astraweb.com> <9a97b618-3e80-4149-9155-14fb210a0758@googlegroups.com> <51e3ecbd$0$9505$c3e8da3$5496439d@news.astraweb.com> <51E3F0A0.9060001@Gmail.com> Message-ID: > On 07/15/2013 08:36 AM, Steven D'Aprano wrote: >> >> Devyn, >> >> 88888 Dihedral is our resident bot, not a human being. Nobody knows who >> controls it, and why they are running it, but we are pretty certain that >> it is a bot responding mechanically to keywords in people's posts. >> >> It's a very clever bot, but still a bot. About one post in four is >> meaningless jargon, the other three are relevant enough to fool people >> into thinking that maybe it is a human being. It had me fooled for a long >> time. >> Does this mean he passes the Turing test? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Jul 15 18:59:04 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 16 Jul 2013 08:59:04 +1000 Subject: Dihedral In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <51e0e7aa$0$9505$c3e8da3$5496439d@news.astraweb.com> <9a97b618-3e80-4149-9155-14fb210a0758@googlegroups.com> <51e3ecbd$0$9505$c3e8da3$5496439d@news.astraweb.com> <51E3F0A0.9060001@Gmail.com> Message-ID: On Tue, Jul 16, 2013 at 8:54 AM, F?bio Santos wrote: > >> On 07/15/2013 08:36 AM, Steven D'Aprano wrote: >>> >>> Devyn, >>> >>> 88888 Dihedral is our resident bot, not a human being. Nobody knows who >>> controls it, and why they are running it, but we are pretty certain that >>> it is a bot responding mechanically to keywords in people's posts. >>> >>> It's a very clever bot, but still a bot. About one post in four is >>> meaningless jargon, the other three are relevant enough to fool people >>> into thinking that maybe it is a human being. It had me fooled for a long >>> time. >>> > > Does this mean he passes the Turing test? Yes, absolutely. The original Turing test was defined in terms of five minutes of analysis, and Dihedral and jmf have clearly been indistinguishably human across that approximate period. ChrisA From timothy.c.delaney at gmail.com Tue Jul 16 02:06:46 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Tue, 16 Jul 2013 16:06:46 +1000 Subject: Dihedral In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <51e0e7aa$0$9505$c3e8da3$5496439d@news.astraweb.com> <9a97b618-3e80-4149-9155-14fb210a0758@googlegroups.com> <51e3ecbd$0$9505$c3e8da3$5496439d@news.astraweb.com> <51E3F0A0.9060001@Gmail.com> Message-ID: On 16 July 2013 08:59, Chris Angelico wrote: > On Tue, Jul 16, 2013 at 8:54 AM, F?bio Santos > wrote: > > > >> On 07/15/2013 08:36 AM, Steven D'Aprano wrote: > >>> > >>> Devyn, > >>> > >>> 88888 Dihedral is our resident bot, not a human being. Nobody knows who > >>> controls it, and why they are running it, but we are pretty certain > that > >>> it is a bot responding mechanically to keywords in people's posts. > >>> > >>> It's a very clever bot, but still a bot. About one post in four is > >>> meaningless jargon, the other three are relevant enough to fool people > >>> into thinking that maybe it is a human being. It had me fooled for a > long > >>> time. > >>> > > > > Does this mean he passes the Turing test? > > Yes, absolutely. The original Turing test was defined in terms of five > minutes of analysis, and Dihedral and jmf have clearly been > indistinguishably human across that approximate period. > The big difference between them is that the jmfbot does not appear to evolve its routines in response to external sources - it seems to be stuck in a closed feedback loop. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Wed Jul 24 14:08:18 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 24 Jul 2013 20:08:18 +0200 Subject: Dihedral In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <51e0e7aa$0$9505$c3e8da3$5496439d@news.astraweb.com> <9a97b618-3e80-4149-9155-14fb210a0758@googlegroups.com> <51e3ecbd$0$9505$c3e8da3$5496439d@news.astraweb.com> <51E3F0A0.9060001@Gmail.com> Message-ID: F?bio Santos, 16.07.2013 00:54: >> On 07/15/2013 08:36 AM, Steven D'Aprano wrote: >>> >>> Devyn, >>> >>> 88888 Dihedral is our resident bot, not a human being. Nobody knows who >>> controls it, and why they are running it, but we are pretty certain that >>> it is a bot responding mechanically to keywords in people's posts. >>> >>> It's a very clever bot, but still a bot. About one post in four is >>> meaningless jargon, the other three are relevant enough to fool people >>> into thinking that maybe it is a human being. It had me fooled for a long >>> time. > > Does this mean he passes the Turing test? I'd say that "it" appears more correct. Or is there any indication of a specific bot gender? (I sure might have missed it...) Note that being of a specific gender is definitely not required to pass the Turing test. I'm not even sure pretending to have one is required. Stefan From rosuav at gmail.com Wed Jul 24 14:23:45 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 04:23:45 +1000 Subject: Dihedral In-Reply-To: References: <571a6dfe-fd66-42cf-92fc-8b97cbe6e9e4@googlegroups.com> <51DFDE65.5040001@Gmail.com> <51e0e7aa$0$9505$c3e8da3$5496439d@news.astraweb.com> <9a97b618-3e80-4149-9155-14fb210a0758@googlegroups.com> <51e3ecbd$0$9505$c3e8da3$5496439d@news.astraweb.com> <51E3F0A0.9060001@Gmail.com> Message-ID: On Thu, Jul 25, 2013 at 4:08 AM, Stefan Behnel wrote: > F?bio Santos, 16.07.2013 00:54: >> Does this mean he passes the Turing test? > > I'd say that "it" appears more correct. Or is there any indication of a > specific bot gender? (I sure might have missed it...) > > Note that being of a specific gender is definitely not required to pass the > Turing test. I'm not even sure pretending to have one is required. Gender: [ ] Male [ ] Female [ ] Both [ ] Neither [ ] Robot [ ] Other In absence of specific information to the contrary, I'll usually refer to computers and programs by masculine pronouns; no particular reason for it, just the same sort of convention that has most ships and boats being "she". And you're quite right, gender and the pretense thereto are completely optional in a Turing test, even though the Turing test was derived from a similar concept involving gender. ChrisA From dreamingforward at gmail.com Thu Jul 11 20:23:04 2013 From: dreamingforward at gmail.com (Mark Janssen) Date: Thu, 11 Jul 2013 17:23:04 -0700 Subject: Casting classes WAS: Documenting builtin methods Message-ID: A user was wondering why they can't change a docstring in a module's class. This made me think: why not have a casting operator ("reciprocal"?) to transform a bonafide class into a mere carcass of a class which can then modified and reanimated back into its own type with the type function? Such that "type(reciprocal(myClass))==myClass"... reciprocal(myClass) returns a writeable, nonproxy class.__dict__ (perhaps also a list of its bases and name) Just a thought to consider... MarkJ Tacoma, Washington From christian at python.org Thu Jul 11 21:21:14 2013 From: christian at python.org (Christian Heimes) Date: Fri, 12 Jul 2013 03:21:14 +0200 Subject: Casting classes WAS: Documenting builtin methods In-Reply-To: References: Message-ID: Am 12.07.2013 02:23, schrieb Mark Janssen: > A user was wondering why they can't change a docstring in a module's class. For CPython builtin types (classes) and function have read-only doc strings for multiple reasons. Internally the doc strings are stored as constant C string literals. The __doc__ attribute is implemented as descriptor that turns the const char *tp_doc member into a Python string. const char* are ... constant. :) All types and builtins are shared across all subinterpreters of Python. The types and their doc strings are identical. Most people are not aware of the subinterpreter feature. Subinterpreters run in different threads of the same process but are isolated from each other. Mutable type and builtin doc strings would break the isolation. You could share information between subinterpreters by e.g. setting the doc string of the int type. We don't want that. Heap types (classes defined with Python code) are not shared. The same goes to modules, even the sys module. Christian From zlchen.ken at gmail.com Thu Jul 11 23:52:09 2013 From: zlchen.ken at gmail.com (Ziliang Chen) Date: Thu, 11 Jul 2013 20:52:09 -0700 (PDT) Subject: python2.6 epoll.modify failed to unregister events ? Message-ID: <3ad30fda-02f4-4ea0-aae8-e2fb3568163a@googlegroups.com> Hi Guys, I have encountered an epoll issues. On the server side, I use epoll.poll() to wait for events, when there is a socket which has EPOLLIN/EPOLLUP events, I first try to read the socket (I did this coz it says EPOLLIN ready, I might think it has some data in its recv queue). After reading 0 length data, the code thinks the peer shutdown its socket, so it notifies the epoll that it doesn't care any events for this socket anymore by calling epoll.modify(sock, 0). But this call seems failed to function, coz when I did another round of epoll.poll(), the socket is still ready for EPOLLUP. My understanding for epoll Linux interface is that it should work if we pass no events when calling epoll_ctl. Steps to reproduce this problem. Open up two consoles, one for client, one for server. Input the following code in the two consoles by the following order. --- Server --- >>> import socket >>> accept_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) >>> accept_socket.setblocking(False) >>> accept_socket.bind(('10.5.6.10', 9999)) >>> accept_socket.listen(socket.SOMAXCONN) >>> import select >>> ep = select.epoll() >>> ep.register(accept_socket.fileno(), select.EPOLLIN) client --- >>> import socket >>> st = socket.socket(socket.AF_INET, socket.SOCK_STREAM) >>> st.connect(('10.5.6.10', 9999)) Server ---- >>> ep.poll() [(4, 1)] >>> sock, peer_addr = accept_socket.accept() >>> ep.register(sock, select.EPOLLIN) Client --- >>> st.send('ahello') 6 Server --- >>> ep.poll() [(6, 1)] >>> sock.recv(1024) 'ahello' >>> sock.shutdown(socket.SHUT_WR) Client --- >>> st.recv(1024) '' >>> st.shutdown(socket.SHUT_WR) Server ---- >>> ep.poll() [(6, 17)] >>> sock.recv(1024) '' >>> ep.modify(sock, 0) <= I am saying, i don't care any events for the sock >>> ep.poll() <= but epoll still return the event for this sock [(6, 16)] >>> ep.poll() [(6, 16)] >>> ep.modify(sock, 0) >>> ep.poll() [(6, 16)] From devyncjohnson at gmail.com Thu Jul 11 19:59:17 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Thu, 11 Jul 2013 19:59:17 -0400 Subject: Question about mailing list rules Message-ID: <51DF46D5.5050605@Gmail.com> Am I allowed to ask questions like "Here is my code. How can I optimize it?" on this mailing list? Mahalo, Devyn Collier Johnson DevynCJohnson at Gmail.com From rosuav at gmail.com Fri Jul 12 03:26:21 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 12 Jul 2013 17:26:21 +1000 Subject: Question about mailing list rules In-Reply-To: <51DF46D5.5050605@Gmail.com> References: <51DF46D5.5050605@Gmail.com> Message-ID: On Fri, Jul 12, 2013 at 9:59 AM, Devyn Collier Johnson wrote: > Am I allowed to ask questions like "Here is my code. How can I optimize it?" > on this mailing list? Sure you can! And you'll get a large number of responses, not all of which are directly to do with your question. :) I assume the code in question _is_ written in Python, right? ChrisA From devyncjohnson at gmail.com Fri Jul 12 06:44:01 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Fri, 12 Jul 2013 06:44:01 -0400 Subject: Question about mailing list rules In-Reply-To: References: <51DF46D5.5050605@Gmail.com> Message-ID: <51DFDDF1.3040309@Gmail.com> I am going to love this mailing list even more. Really, only Python code? I wanted to ask Python users about Perl! (^u^) Devyn Collier Johnson On 07/12/2013 03:26 AM, Chris Angelico wrote: > On Fri, Jul 12, 2013 at 9:59 AM, Devyn Collier Johnson > wrote: >> Am I allowed to ask questions like "Here is my code. How can I optimize it?" >> on this mailing list? > Sure you can! And you'll get a large number of responses, not all of > which are directly to do with your question. :) > > I assume the code in question _is_ written in Python, right? > > ChrisA From rosuav at gmail.com Fri Jul 12 06:47:02 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 12 Jul 2013 20:47:02 +1000 Subject: Question about mailing list rules In-Reply-To: <51DFDDF1.3040309@Gmail.com> References: <51DF46D5.5050605@Gmail.com> <51DFDDF1.3040309@Gmail.com> Message-ID: On Fri, Jul 12, 2013 at 8:44 PM, Devyn Collier Johnson wrote: > I am going to love this mailing list even more. > > Really, only Python code? I wanted to ask Python users about Perl! (^u^) > > Devyn Collier Johnson Heh. You'd be surprised what comes up. If it's at least broadly related to Python (maybe you're comparing constructs in Python and Perl??), it'll probably be accepted. And even stuff that's completely off-topic does at times get discussed. One small thing, though. Please avoid top-posting; the convention on this list is to quote text above, and write yours underneath. Makes it easier to follow the flow of conversation. Thanks! ChrisA From kwpolska at gmail.com Fri Jul 12 07:11:43 2013 From: kwpolska at gmail.com (=?UTF-8?B?Q2hyaXMg4oCcS3dwb2xza2HigJ0gV2Fycmljaw==?=) Date: Fri, 12 Jul 2013 13:11:43 +0200 Subject: Question about mailing list rules In-Reply-To: References: <51DF46D5.5050605@Gmail.com> <51DFDDF1.3040309@Gmail.com> Message-ID: On Fri, Jul 12, 2013 at 12:47 PM, Chris Angelico wrote: > On Fri, Jul 12, 2013 at 8:44 PM, Devyn Collier Johnson > wrote: >> I am going to love this mailing list even more. >> >> Really, only Python code? I wanted to ask Python users about Perl! (^u^) >> >> Devyn Collier Johnson > > Heh. You'd be surprised what comes up. If it's at least broadly > related to Python (maybe you're comparing constructs in Python and > Perl??), it'll probably be accepted. And even stuff that's completely > off-topic does at times get discussed. > > One small thing, though. Please avoid top-posting; the convention on > this list is to quote text above, and write yours underneath. Makes it > easier to follow the flow of conversation. Thanks! > > ChrisA > -- > http://mail.python.org/mailman/listinfo/python-list One more thing Devyn should do is watch the ?To:? field and make sure it says python-list at python.org, because the above message was sent to Chris only, and that is not what should happen most of the time. Another option is to use Reply All, but it will make old Usenet hags angry, because they would get two copies. -- Kwpolska | GPG KEY: 5EAAEA16 stop html mail | always bottom-post http://asciiribbon.org | http://caliburn.nl/topposting.html From devyncjohnson at gmail.com Fri Jul 12 08:25:50 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Fri, 12 Jul 2013 08:25:50 -0400 Subject: Question about mailing list rules In-Reply-To: References: <51DF46D5.5050605@Gmail.com> <51DFDDF1.3040309@Gmail.com> Message-ID: <51DFF5CE.2060403@Gmail.com> On 07/12/2013 07:11 AM, Chris ?Kwpolska? Warrick wrote: > On Fri, Jul 12, 2013 at 12:47 PM, Chris Angelico wrote: >> On Fri, Jul 12, 2013 at 8:44 PM, Devyn Collier Johnson >> wrote: >>> I am going to love this mailing list even more. >>> >>> Really, only Python code? I wanted to ask Python users about Perl! (^u^) >>> >>> Devyn Collier Johnson >> Heh. You'd be surprised what comes up. If it's at least broadly >> related to Python (maybe you're comparing constructs in Python and >> Perl??), it'll probably be accepted. And even stuff that's completely >> off-topic does at times get discussed. >> >> One small thing, though. Please avoid top-posting; the convention on >> this list is to quote text above, and write yours underneath. Makes it >> easier to follow the flow of conversation. Thanks! >> >> ChrisA >> -- >> http://mail.python.org/mailman/listinfo/python-list > One more thing Devyn should do is watch the ?To:? field and make sure > it says python-list at python.org, because the above message was sent to > Chris only, and that is not what should happen most of the time. > Another option is to use Reply All, but it will make old Usenet hags > angry, because they would get two copies. > > -- > Kwpolska | GPG KEY: 5EAAEA16 > stop html mail | always bottom-post > http://asciiribbon.org | http://caliburn.nl/topposting.html Thank you and sorry about that. Kwpolska, I noticed your email shows "stop html mail" at the bottom. I have Thunderbird setup to use HTML mail. Are my emails coming up as plain text or HTML on this mailing list? Devyn Collier Johnson From kwpolska at gmail.com Fri Jul 12 08:34:00 2013 From: kwpolska at gmail.com (=?UTF-8?B?Q2hyaXMg4oCcS3dwb2xza2HigJ0gV2Fycmljaw==?=) Date: Fri, 12 Jul 2013 14:34:00 +0200 Subject: Question about mailing list rules In-Reply-To: <51DFF5CE.2060403@Gmail.com> References: <51DF46D5.5050605@Gmail.com> <51DFDDF1.3040309@Gmail.com> <51DFF5CE.2060403@Gmail.com> Message-ID: On Fri, Jul 12, 2013 at 2:25 PM, Devyn Collier Johnson wrote: > > On 07/12/2013 07:11 AM, Chris ?Kwpolska? Warrick wrote: >> >> On Fri, Jul 12, 2013 at 12:47 PM, Chris Angelico wrote: >>> >>> On Fri, Jul 12, 2013 at 8:44 PM, Devyn Collier Johnson >>> wrote: >>>> >>>> I am going to love this mailing list even more. >>>> >>>> Really, only Python code? I wanted to ask Python users about Perl! (^u^) >>>> >>>> Devyn Collier Johnson >>> >>> Heh. You'd be surprised what comes up. If it's at least broadly >>> related to Python (maybe you're comparing constructs in Python and >>> Perl??), it'll probably be accepted. And even stuff that's completely >>> off-topic does at times get discussed. >>> >>> One small thing, though. Please avoid top-posting; the convention on >>> this list is to quote text above, and write yours underneath. Makes it >>> easier to follow the flow of conversation. Thanks! >>> >>> ChrisA >>> -- >>> http://mail.python.org/mailman/listinfo/python-list >> >> One more thing Devyn should do is watch the ?To:? field and make sure >> it says python-list at python.org, because the above message was sent to >> Chris only, and that is not what should happen most of the time. >> Another option is to use Reply All, but it will make old Usenet hags >> angry, because they would get two copies. >> >> -- >> Kwpolska | GPG KEY: 5EAAEA16 >> stop html mail | always bottom-post >> http://asciiribbon.org | http://caliburn.nl/topposting.html > > Thank you and sorry about that. > > Kwpolska, I noticed your email shows "stop html mail" at the bottom. I have > Thunderbird setup to use HTML mail. Are my emails coming up as plain text or > HTML on this mailing list? > > Devyn Collier Johnson Apparently, it comes in plaintext. Mailman does not say that it removed HTML, so it should be fine. Either way, there should be a switch in the new message window to set what should be sent. -- Kwpolska | GPG KEY: 5EAAEA16 stop html mail | always bottom-post http://asciiribbon.org | http://caliburn.nl/topposting.html From davea at davea.name Fri Jul 12 10:58:26 2013 From: davea at davea.name (Dave Angel) Date: Fri, 12 Jul 2013 10:58:26 -0400 Subject: Question about mailing list rules In-Reply-To: References: <51DF46D5.5050605@Gmail.com> <51DFDDF1.3040309@Gmail.com> <51DFF5CE.2060403@Gmail.com> Message-ID: On 07/12/2013 08:34 AM, Chris ?Kwpolska? Warrick wrote: > On Fri, Jul 12, 2013 at 2:25 PM, Devyn Collier Johnson > wrote: >> >>> >>> One more thing Devyn should do is watch the ?To:? field and make sure >>> it says python-list at python.org, because the above message was sent to >>> Chris only, and that is not what should happen most of the time. >>> Another option is to use Reply All, but it will make old Usenet hags >>> angry, because they would get two copies. In Thunderbird, use Reply-List, rather than Reply-All, and it'll go to the right place. Still worth checking, though. >>> >>> -- >>> Kwpolska | GPG KEY: 5EAAEA16 >>> stop html mail | always bottom-post >>> http://asciiribbon.org | http://caliburn.nl/topposting.html >> >> Thank you and sorry about that. >> >> Kwpolska, I noticed your email shows "stop html mail" at the bottom. I have >> Thunderbird setup to use HTML mail. Are my emails coming up as plain text or >> HTML on this mailing list? >> >> Devyn Collier Johnson > > Apparently, it comes in plaintext. Mailman does not say that it > removed HTML, so it should be fine. Either way, there should be a > switch in the new message window to set what should be sent. > I only see one message from Devyn, and it's not the one quoted here. But the one I do see is in plain text. The safest way to tell (other than noticing that indentation randomly goes away, or that things are missing, or ...) is to view the source. In Thunderbird, select the message, then View->MessageSource. Then search in that for html. Frequently there's two versions of the message, one text and one html. What you want is a single version, text. -- DaveA. From andipersti at gmail.com Fri Jul 12 03:37:55 2013 From: andipersti at gmail.com (Andreas Perstinger) Date: Fri, 12 Jul 2013 09:37:55 +0200 Subject: Question about mailing list rules In-Reply-To: <51DF46D5.5050605@Gmail.com> References: <51DF46D5.5050605@Gmail.com> Message-ID: <51DFB253.4040009@gmail.com> On 12.07.2013 01:59, Devyn Collier Johnson wrote: > Am I allowed to ask questions like "Here is my code. How can I optimize > it?" on this mailing list? If it's written in Python, why not? But that doesn't mean you are guaranteed to get an answer :-). And please read http://sscce.org/ before posting your latest 10,000 line program :-) Bye, Andreas From roy at panix.com Fri Jul 12 09:04:40 2013 From: roy at panix.com (Roy Smith) Date: Fri, 12 Jul 2013 09:04:40 -0400 Subject: Question about mailing list rules References: <51DF46D5.5050605@Gmail.com> Message-ID: In article , Chris Angelico wrote: > On Fri, Jul 12, 2013 at 9:59 AM, Devyn Collier Johnson > wrote: > > Am I allowed to ask questions like "Here is my code. How can I optimize it?" > > on this mailing list? > > Sure you can! And you'll get a large number of responses, not all of > which are directly to do with your question. :) > > I assume the code in question _is_ written in Python, right? If not, then the answer to "how can I optimize it" is obvious, isn't it? From devyncjohnson at gmail.com Fri Jul 12 10:08:25 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Fri, 12 Jul 2013 10:08:25 -0400 Subject: Question about mailing list rules In-Reply-To: References: <51DF46D5.5050605@Gmail.com> Message-ID: <51E00DD9.6030602@Gmail.com> On 07/12/2013 09:04 AM, Roy Smith wrote: > In article , > Chris Angelico wrote: > >> On Fri, Jul 12, 2013 at 9:59 AM, Devyn Collier Johnson >> wrote: >>> Am I allowed to ask questions like "Here is my code. How can I optimize it?" >>> on this mailing list? >> Sure you can! And you'll get a large number of responses, not all of >> which are directly to do with your question. :) >> >> I assume the code in question _is_ written in Python, right? > If not, then the answer to "how can I optimize it" is obvious, isn't it? Yeah, I should have been more specific and said Python. From nikos at superhost.gr Fri Jul 12 10:18:20 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Ouc66z4zOu86xz4I=?=) Date: Fri, 12 Jul 2013 17:18:20 +0300 Subject: GeoIP2 for retrieving city and region ? Message-ID: Hello, iam still looking for a way to identify the city of my website visitors. The closet i have gone is to come up with the visitor's ISP city: try: gi = pygeoip.GeoIP('/usr/local/share/GeoLiteCity.dat') city = gi.time_zone_by_addr( os.environ['HTTP_CF_CONNECTING_IP'] ) host = socket.gethostbyaddr( os.environ['HTTP_CF_CONNECTING_IP'] )[0] except Exception as e: host = repr(e) But today i was searching again for this and found out about geoip2, maybe that would help more. >>> import geoip2 Traceback (most recent call last): File "", line 1, in ImportError: No module named 'geoip2' >>> client = geoip2.webservices.Client(42, 'abcdef123456') >>> omni = client.omni('24.24.24.24') >>> country = omni.country >>> print(country.iso_code) I cant even import the module even though my 'pip install geopip2' was successful There is definately i way to identify the users location based solely on its ip address as this site does it: http://www.geoiptool.com/ Google, MS, facebook and twitter are not the only ones that can do it? Perhaps this is being done by giving longitude and latitude? -- What is now proved was at first only imagined! From davea at davea.name Fri Jul 12 11:32:27 2013 From: davea at davea.name (Dave Angel) Date: Fri, 12 Jul 2013 11:32:27 -0400 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: On 07/12/2013 10:18 AM, ??????? wrote: > Hello, iam still looking for a way to identify the city of my website > visitors. > > > > I cant even import the module even though my 'pip install geopip2' wa > successful Either it wasn't successful, or it's not the package you thought. There are lots of things you might have downloaded, but since you give no details... > > There is definately i way to identify the users location based solely on > its ip address as this site does it: http://www.geoiptool.com/ > Sure, and as long as you don't mind it being 1000 miles off, you too can claim to do it too. When I go to that site, the little pin is in Kansas, which is 1100 miles from where I live on the east coast of the US. > Google, MS, facebook and twitter are not the only ones that can do it? > > Perhaps this is being done by giving longitude and latitude? Or by reading the mind of the programmer. I suggest you read that geoiptool site, in particular the page http://www.geoiptool.com/en/ip_info/ There is some misinformation, but notice carefully the part about dynamic IP addresses. Probably 99% of the individual users on the web (the ones using a browser) have dynamic IP addresses. The fixed ones are needed by servers, and especially for DNS use, where the name lookup wants to be stable for relatively log periods of time. -- DaveA From rosuav at gmail.com Fri Jul 12 12:41:25 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 13 Jul 2013 02:41:25 +1000 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: On Sat, Jul 13, 2013 at 1:32 AM, Dave Angel wrote: >> >> There is definately i way to identify the users location based solely on >> its ip address as this site does it: http://www.geoiptool.com/ >> > > Sure, and as long as you don't mind it being 1000 miles off, you too can > claim to do it too. When I go to that site, the little pin is in Kansas, > which is 1100 miles from where I live on the east coast of the US. I have two IPs at this house, not counting the ones I could get off mobile connections (which are valid probably anywhere in the state, maybe further afield). One of them is plotted fairly accurately (not more than a couple of kilometers wrong), but the other is listed at Elizabeth and Bourke in the CBD... which is half an hour's train journey away from me. And the one that was wrong was the one that's actually an official static IP address (as opposed to a "technically dynamic but hasn't changed for a couple of years" address). Obligatory XKCD link: http://xkcd.com/713/ ChrisA From nikos at superhost.gr Fri Jul 12 11:52:20 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Ouc66z4zOu86xz4I=?=) Date: Fri, 12 Jul 2013 18:52:20 +0300 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: ???? 12/7/2013 6:32 ??, ?/? Dave Angel ??????: > > I suggest you read that geoiptool site, in particular the page > > http://www.geoiptool.com/en/ip_info/ > > There is some misinformation, but notice carefully the part about > dynamic IP addresses. Probably 99% of the individual users on the web > (the ones using a browser) have dynamic IP addresses. The fixed ones > are needed by servers, and especially for DNS use, where the name lookup > wants to be stable for relatively log periods of time. I did, for me it gives exact city location and not the ISP's city location. I dont know whay for you ti just says Kansas, it shoudln't, since it susing longitute and latitude, it should have been accurate. -- What is now proved was at first only imagined! From joel.goldstick at gmail.com Fri Jul 12 12:24:32 2013 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Fri, 12 Jul 2013 12:24:32 -0400 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: On Fri, Jul 12, 2013 at 11:52 AM, ??????? wrote: > ???? 12/7/2013 6:32 ??, ?/? Dave Angel ??????: > > >> I suggest you read that geoiptool site, in particular the page >> >> http://www.geoiptool.com/en/**ip_info/ >> >> There is some misinformation, but notice carefully the part about >> dynamic IP addresses. Probably 99% of the individual users on the web >> (the ones using a browser) have dynamic IP addresses. The fixed ones >> are needed by servers, and especially for DNS use, where the name lookup >> wants to be stable for relatively log periods of time. >> > > > I did, for me it gives exact city location and not the ISP's city location. > > I dont know whay for you ti just says Kansas, it shoudln't, since it > susing longitute and latitude, it should have been accurate. > > > Nikos, this is the point where you (again) loose credibility on this list. You asked this question about how to find the location where someone is browsing your site from. You got several answers. The bottom line is that you can't know that location unless the browsing machine does its own geo location (think cell phones) or the user has provided his location via some form. So, now, a week or two later you come back with the same question, as if it hasn't already been answered -- but you give it a 'red herring' kind of twist. As I understand it, you found a new module that you think will do something that all parties answering this question before explained that this is not possible. And you switch up to say "why can't I load this thing?". Well, I don't why you can't load it, but I don't want to help because it doesn't help you solve your stated problem. Your stated problem is done. move on. > -- > What is now proved was at first only imagined! > -- > http://mail.python.org/**mailman/listinfo/python-list > -- Joel Goldstick http://joelgoldstick.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From nikos at superhost.gr Fri Jul 12 12:32:44 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Ouc66z4zOu86xz4I=?=) Date: Fri, 12 Jul 2013 19:32:44 +0300 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: I know i have asked before but hwta i get is ISP city not visitors precise city. GeoLiteCity.dat isnt accurate that's why it comes for free. i must somehow get access to GeoIPCity.dat which is the full version. And of course it can be done, i dont want to believe that it cant. When visiting http://www.geoiptool.com/en/__ip_info/ it pinpoints my _exact_ city of living, not the ISP's. It did not even ask me to allow a geop ip javascript to run it present sit instantly. So, it certainly is possible if only one can find the correct database to use. So, my question now is, if there is some way we can get an accurate Geo City database. -- What is now proved was at first only imagined! From invalid at invalid.invalid Fri Jul 12 12:38:44 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Fri, 12 Jul 2013 16:38:44 +0000 (UTC) Subject: GeoIP2 for retrieving city and region ? References: Message-ID: On 2013-07-12, ?????????????? wrote: > I know i have asked before but hwta i get is ISP city not visitors > precise city. You can't reliably do that. > GeoLiteCity.dat isnt accurate that's why it comes for free. i must > somehow get access to GeoIPCity.dat which is the full version. > > And of course it can be done, i dont want to believe that it cant. Believe what you want. > When visiting http://www.geoiptool.com/en/__ip_info/ it pinpoints my > _exact_ city of living, not the ISP's. It did not even ask me to > allow a geop ip javascript to run it present sit instantly. So you've reached your conclusion on a sample size of one? -- Grant Edwards grant.b.edwards Yow! I'm encased in the at lining of a pure pork gmail.com sausage!! From rosuav at gmail.com Fri Jul 12 12:47:38 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 13 Jul 2013 02:47:38 +1000 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: On Sat, Jul 13, 2013 at 2:38 AM, Grant Edwards wrote: > On 2013-07-12, ?????????????? wrote: >> When visiting http://www.geoiptool.com/en/__ip_info/ it pinpoints my >> _exact_ city of living, not the ISP's. It did not even ask me to >> allow a geop ip javascript to run it present sit instantly. > > So you've reached your conclusion on a sample size of one? This is Nikos. He doesn't read responses properly, doesn't do his research, and has (by his own admission) an iron head that doesn't let information cross it lightly. Yes, he reached his conclusion on a sample size of one. Oh, and just for laughs, I tried a few of my recent mobile IP addresses in the GeoIP lookup. All of them quoted Melbourne someplace, some in the CBD and some out in the suburbs, but all vastly wrong, and places I haven't been. But I'd never expect it to be accurate on those. ChrisA From joel.goldstick at gmail.com Fri Jul 12 19:42:42 2013 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Fri, 12 Jul 2013 19:42:42 -0400 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: <7v11u8p3hrs2jvj9jqmvct8k1cl0fjc9ct@4ax.com> References: <7v11u8p3hrs2jvj9jqmvct8k1cl0fjc9ct@4ax.com> Message-ID: On Fri, Jul 12, 2013 at 7:04 PM, Dennis Lee Bieber wrote: > On Sat, 13 Jul 2013 02:47:38 +1000, Chris Angelico > declaimed the following: > > > > >Oh, and just for laughs, I tried a few of my recent mobile IP > >addresses in the GeoIP lookup. All of them quoted Melbourne someplace, > >some in the CBD and some out in the suburbs, but all vastly wrong, and > >places I haven't been. But I'd never expect it to be accurate on > >those. > > > Well... the MaxMind demo of "my IP" did get the proper metropolitan > area... But they list the ISP as "AT&T"... My real ISP is Earthlink > (piggybacking on AT&T DSL service). > > The Lat/Long, however shows as > > 42.9634 -85.6681 > whereas a recent GPS readout shows > 42.9159 -85.5541 > > or 2m50s too far north, and 6m50s too far west. > > Same website, accessed from my Blackberry phone, gave a result of > "United States, NA" and location 38 -97 > -- > Wulfraed Dennis Lee Bieber AF6VN > wlfraed at ix.netcom.com HTTP://wlfraed.home.netcom.com/ > > -- > http://mail.python.org/mailman/listinfo/python-list > Speaking more from a political perspective, an important aspect of the internet is that if you are a publisher who doesn't require registration to read what you post, your readers are free to be more or less anonymous. This is a good thing in many ways. If you have a service that requires some sort of sign up, then you can require knowing things about them (location, etc). If you want to know where your readers are, you need to make arrangements with their ISPs to get that information, since in many cases the ISP has a physical link to your machine. But why should they provide that to you? After all you are not their customer. It might be fun to know, but the repercussions are serious. -- Joel Goldstick http://joelgoldstick.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Jul 12 20:36:28 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 13 Jul 2013 10:36:28 +1000 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: <7v11u8p3hrs2jvj9jqmvct8k1cl0fjc9ct@4ax.com> References: <7v11u8p3hrs2jvj9jqmvct8k1cl0fjc9ct@4ax.com> Message-ID: On Sat, Jul 13, 2013 at 9:04 AM, Dennis Lee Bieber wrote: > On Sat, 13 Jul 2013 02:47:38 +1000, Chris Angelico > declaimed the following: > >> >>Oh, and just for laughs, I tried a few of my recent mobile IP >>addresses in the GeoIP lookup. All of them quoted Melbourne someplace, >>some in the CBD and some out in the suburbs, but all vastly wrong, and >>places I haven't been. But I'd never expect it to be accurate on >>those. >> > Well... the MaxMind demo of "my IP" did get the proper metropolitan > area... But they list the ISP as "AT&T"... My real ISP is Earthlink > (piggybacking on AT&T DSL service). > > The Lat/Long, however shows as > > 42.9634 -85.6681 > whereas a recent GPS readout shows > 42.9159 -85.5541 > > or 2m50s too far north, and 6m50s too far west. > > Same website, accessed from my Blackberry phone, gave a result of > "United States, NA" and location 38 -97 When you try to place a visitor geographically by IP address, the only thing you can be absolutely 100% certain of is which RIR they're at (proxies aside - you're just testing the proxy rather than the ultimate origin). Country is also highly likely to be right, though not certain (I've never known it to be wrong, but I've never been able to confirm what happens with some of the small European countries - for all I know they could share ISPs and netblocks). Anything tighter than that is goign to be pretty hit-and-miss. But I have to say, it's improved a lot over the years. Back in the early 2000s - say, about 8 years ago, I think - I was playing with this sort of technology, and it placed me in Sydney. That's one state away, lots of rivalry separating us (friendly rivalry, of course; in a country that's doing its best to kill us all, we can't afford to really hate each other), and roughly 750-1000km wrong by distance (depending on how you measure - most people don't put an odometer on a crow). So at least now it gets within the same degree of latitude and longitude... most of the time. ChrisA From nikos at superhost.gr Sat Jul 13 01:48:24 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Ouc66z4zOu86xz4I=?=) Date: Sat, 13 Jul 2013 08:48:24 +0300 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: ???? 13/7/2013 2:04 ??, ?/? Dennis Lee Bieber ??????: > On Sat, 13 Jul 2013 02:47:38 +1000, Chris Angelico > declaimed the following: > >> >> Oh, and just for laughs, I tried a few of my recent mobile IP >> addresses in the GeoIP lookup. All of them quoted Melbourne someplace, >> some in the CBD and some out in the suburbs, but all vastly wrong, and >> places I haven't been. But I'd never expect it to be accurate on >> those. >> > Well... the MaxMind demo of "my IP" did get the proper metropolitan > area... But they list the ISP as "AT&T"... My real ISP is Earthlink > (piggybacking on AT&T DSL service). > > The Lat/Long, however shows as > > 42.9634 -85.6681 > whereas a recent GPS readout shows > 42.9159 -85.5541 > > or 2m50s too far north, and 6m50s too far west. > > Same website, accessed from my Blackberry phone, gave a result of > "United States, NA" and location 38 -97 > I have read all your answer very carefully but i still need some way of getting it done. All my Greek website visitors say they are from Europe/Athens which is the ISP's location and not user's homeland. Well it worked for me but as many other told me it wasn't accurate for them too. Please try this: http://www.maxmind.com/en/geoip_demo and tell me if maxmind's database can pippont you city's location. Thank you. -- What is now proved was at first only imagined! From rosuav at gmail.com Sat Jul 13 01:53:48 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 13 Jul 2013 15:53:48 +1000 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: On Sat, Jul 13, 2013 at 3:48 PM, ??????? wrote: > ???? 13/7/2013 2:04 ??, ?/? Dennis Lee Bieber ??????: >> >> On Sat, 13 Jul 2013 02:47:38 +1000, Chris Angelico >> declaimed the following: >> >>> >>> Oh, and just for laughs, I tried a few of my recent mobile IP >>> addresses in the GeoIP lookup. All of them quoted Melbourne someplace, >>> some in the CBD and some out in the suburbs, but all vastly wrong, and >>> places I haven't been. But I'd never expect it to be accurate on >>> those. >>> >> Well... the MaxMind demo of "my IP" did get the proper >> metropolitan >> area... But they list the ISP as "AT&T"... My real ISP is Earthlink >> (piggybacking on AT&T DSL service). >> >> The Lat/Long, however shows as >> >> 42.9634 -85.6681 >> whereas a recent GPS readout shows >> 42.9159 -85.5541 >> >> or 2m50s too far north, and 6m50s too far west. >> >> Same website, accessed from my Blackberry phone, gave a result of >> "United States, NA" and location 38 -97 >> > > > I have read all your answer very carefully but i still need some way of > getting it done. > > All my Greek website visitors say they are from Europe/Athens which is the > ISP's location and not user's homeland. > > Well it worked for me but as many other told me it wasn't accurate for them > too. > > Please try this: http://www.maxmind.com/en/geoip_demo > > and tell me if maxmind's database can pippont you city's location. Nikos, you keep asking for a way to do the impossible. We keep telling you that it is impossible. No alternative technique will do what cannot be done! I just tried that on my two IPs and it was quite wrong on both of them - further wrong than some of the others have been. Stop expecting magic. ChrisA From nikos at superhost.gr Sat Jul 13 02:07:26 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Ouc66z4zOu86xz4I=?=) Date: Sat, 13 Jul 2013 09:07:26 +0300 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: ???? 13/7/2013 8:53 ??, ?/? Chris Angelico ??????: > On Sat, Jul 13, 2013 at 3:48 PM, ??????? wrote: >> ???? 13/7/2013 2:04 ??, ?/? Dennis Lee Bieber ??????: >>> >>> On Sat, 13 Jul 2013 02:47:38 +1000, Chris Angelico >>> declaimed the following: >>> >>>> >>>> Oh, and just for laughs, I tried a few of my recent mobile IP >>>> addresses in the GeoIP lookup. All of them quoted Melbourne someplace, >>>> some in the CBD and some out in the suburbs, but all vastly wrong, and >>>> places I haven't been. But I'd never expect it to be accurate on >>>> those. >>>> >>> Well... the MaxMind demo of "my IP" did get the proper >>> metropolitan >>> area... But they list the ISP as "AT&T"... My real ISP is Earthlink >>> (piggybacking on AT&T DSL service). >>> >>> The Lat/Long, however shows as >>> >>> 42.9634 -85.6681 >>> whereas a recent GPS readout shows >>> 42.9159 -85.5541 >>> >>> or 2m50s too far north, and 6m50s too far west. >>> >>> Same website, accessed from my Blackberry phone, gave a result of >>> "United States, NA" and location 38 -97 >>> >> >> >> I have read all your answer very carefully but i still need some way of >> getting it done. >> >> All my Greek website visitors say they are from Europe/Athens which is the >> ISP's location and not user's homeland. >> >> Well it worked for me but as many other told me it wasn't accurate for them >> too. >> >> Please try this: http://www.maxmind.com/en/geoip_demo >> >> and tell me if maxmind's database can pippont you city's location. > > Nikos, you keep asking for a way to do the impossible. We keep telling > you that it is impossible. No alternative technique will do what > cannot be done! > > I just tried that on my two IPs and it was quite wrong on both of them > - further wrong than some of the others have been. > > Stop expecting magic. > > ChrisA > But it works for me, How can it be impossible and worked for me at the same time? Also i tried some other website that asked me to allow it to run a javascript on my browser and it pinpointed even my street! If it wasnt possbile then MaxMind would be seeling its GeoIP2 app for 1380$ per year. -- What is now proved was at first only imagined! From rosuav at gmail.com Sat Jul 13 02:22:57 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 13 Jul 2013 16:22:57 +1000 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: On Sat, Jul 13, 2013 at 4:07 PM, ??????? wrote: > But it works for me, How can it be impossible and worked for me at the same > time? If I roll ten six-sided dice, will they total 35? Maybe. Maybe they'll be close. But it's impossible to come up with a table for rolling those dice on that will guarantee you exactly 35 every time. ChrisA From nikos at superhost.gr Sat Jul 13 02:28:54 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Ouc66z4zOu86xz4I=?=) Date: Sat, 13 Jul 2013 09:28:54 +0300 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: ???? 13/7/2013 9:22 ??, ?/? Chris Angelico ??????: > On Sat, Jul 13, 2013 at 4:07 PM, ??????? wrote: >> But it works for me, How can it be impossible and worked for me at the same >> time? > > If I roll ten six-sided dice, will they total 35? Maybe. Maybe they'll > be close. But it's impossible to come up with a table for rolling > those dice on that will guarantee you exactly 35 every time. I just had some other friends of me, who live in different cities around Greece to test the link i gave to you in my previous post and for all of them it returned the correct city of their origin. Seems like GeoIP2 is doing a better job that its predecessor GeopIP. -- What is now proved was at first only imagined! From lele at metapensiero.it Sat Jul 13 04:12:23 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Sat, 13 Jul 2013 10:12:23 +0200 Subject: GeoIP2 for retrieving city and region ? References: Message-ID: <87wqoulvyw.fsf@nautilus.nautilus> ??????? writes: > But it works for me, How can it be impossible and worked for me at the > same time? Read the answers you got. What is *impossible* is *exactly and precisely* find the geographical location of your machine, just using an IP database like what you are doing, i.e. without your local machine cooperating in the process (for example, by providing the numbers taken from your local machine's GPS device). What you are using is just some kind of heuristic, it's not an exact science: each ISP is generally assigned a bunch of IPs which it manages in some way, reserving some of them as *static IP* (that is, the same IP is assigned to the same contractor, always), and assigning the other on demand, like a DHCP server would do in a intranet. Think to the latter: do you think it is possible to exactly locate in which room every wireless notebook that travels inside your big house is at any given time? Hint: no, you cannot, the best you can say is that each notebook may be within a ?sphere? of radius 50mt (just making up a number, it obviously depend on the wireless signal power, and eventually on the presence of wireless repeaters...). Maybe you did provide your exact street address when signed the contract with your ISP, but now ask yourself: would you be happy if your ISP gives that kind of information to whomever may ask for it (in the specific case, a geolocation service like maxmind.com)? ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From roy at panix.com Sat Jul 13 12:59:20 2013 From: roy at panix.com (Roy Smith) Date: Sat, 13 Jul 2013 12:59:20 -0400 Subject: GeoIP2 for retrieving city and region ? References: Message-ID: In article , ???????????????????? wrote: > But it works for me, How can it be impossible and worked for me at the > same time? > > Also i tried some other website that asked me to allow it to run a > javascript on my browser and it pinpointed even my street! > > If it wasnt possbile then MaxMind would be seeling its GeoIP2 app for > 1380$ per year. At Songza, we purchase the MaxMind database for doing geolocation based on IP address. This is how we enforce our content licenses which only allow us to stream music to the US and Canada. We also use it to target specific features (or advertising) to specific cities or other geographic areas within the US and Canada. That being said, we understand that it is only an approximate solution. It is useful, but not perfect. If you go to the MaxMind web site, you will see they have a range of products, at different prices, which promise various levels of accuracy and error rates. But none of them, for any amount of money, offers "resolution down to the street address and 0% error rate". From wayne at waynewerner.com Sat Jul 13 14:35:00 2013 From: wayne at waynewerner.com (Wayne Werner) Date: Sat, 13 Jul 2013 13:35:00 -0500 (CDT) Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: On Sat, 13 Jul 2013, ??????? wrote: > But it works for me, How can it be impossible and worked for me at the > same time? 2 + 2 = 4 2 + 6 = 8??? Why can't I make 2 and 6 equal 4? It worked for 2, so I know it's not impossible! I don't care what everyone says, I was able to make one case work so obviously I juat need to figure out how to make it work! Allegorically, W From ian.g.kelly at gmail.com Fri Jul 12 13:19:41 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Fri, 12 Jul 2013 11:19:41 -0600 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: On Fri, Jul 12, 2013 at 10:32 AM, ??????? wrote: > > I know i have asked before but hwta i get is ISP city not visitors precise > city. > > GeoLiteCity.dat isnt accurate that's why it comes for free. > i must somehow get access to GeoIPCity.dat which is the full version. > > And of course it can be done, i dont want to believe that it cant. > > When visiting http://www.geoiptool.com/en/__ip_info/ it pinpoints my _exact_ > city of living, not the ISP's. > It did not even ask me to allow a geop ip javascript to run it present sit > instantly. Try this: 1) Go to http://incloak.com (or any other free web proxy site). 2) Paste in the URL http://www.geoiptool.com and press Enter 3) See where it thinks you are now. When I tried it, it placed me on the wrong side of the Atlantic Ocean. From tjreedy at udel.edu Fri Jul 12 18:07:00 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 12 Jul 2013 18:07:00 -0400 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: On 7/12/2013 1:19 PM, Ian Kelly wrote: > Try this: > > 1) Go to http://incloak.com (or any other free web proxy site). > 2) Paste in the URL http://www.geoiptool.com and press Enter > 3) See where it thinks you are now. > > When I tried it, it placed me on the wrong side of the Atlantic Ocean. Me to. Thanks for the link. -- Terry Jan Reedy From python at mrabarnett.plus.com Fri Jul 12 18:07:53 2013 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 12 Jul 2013 23:07:53 +0100 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: <51E07E39.5070201@mrabarnett.plus.com> On 12/07/2013 17:32, ??????? wrote: > > I know i have asked before but hwta i get is ISP city not visitors > precise city. > > GeoLiteCity.dat isnt accurate that's why it comes for free. > i must somehow get access to GeoIPCity.dat which is the full version. > > And of course it can be done, i dont want to believe that it cant. > > When visiting http://www.geoiptool.com/en/__ip_info/ it pinpoints my > _exact_ city of living, not the ISP's. Have you considered that your ISP might be in the same city as you? According to geoiptool, my ISP is near Leeds, UK, but the important point is that _I'm not_. > It did not even ask me to allow a geop ip javascript to run it present > sit instantly. > > So, it certainly is possible if only one can find the correct database > to use. > > So, my question now is, if there is some way we can get an accurate Geo > City database. > From torriem at gmail.com Fri Jul 12 18:15:27 2013 From: torriem at gmail.com (Michael Torrie) Date: Fri, 12 Jul 2013 16:15:27 -0600 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: <51E07FFF.5070204@gmail.com> On 07/12/2013 10:32 AM, ??????? wrote: > So, my question now is, if there is some way we can get an accurate Geo > City database. As has been said pretty much by every other poster, there is no way to do get an accurate location database. Period. The databases that do exist were built by hand, and also guessed at based on routing information. The best you can really do is region, or country, and even that fails sometimes. If you want to know a visitor's city you should ask them using the new browser location apis available to javascript. http://diveintohtml5.info/geolocation.html Since IPs can be dynamic, sometimes even assigned across a region, there's no way to accurately map ip addresses to a city with the reliability that you seem to want. Google is pretty accurate because they've spent a lot of time building up their own database, and also convincing users to reveal their locations to them. Unless you do the same thing, you have to just get by with what others have provided for you. From nikos at superhost.gr Sat Jul 13 01:32:24 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Ouc66z4zOu86xz4I=?=) Date: Sat, 13 Jul 2013 08:32:24 +0300 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: ???? 13/7/2013 1:07 ??, ?/? MRAB ??????: > > Have you considered that your ISP might be in the same city as you? > > According to geoiptool, my ISP is near Leeds, UK, but the important > point is that _I'm not_. My ISP is in Athens and i live in Thessalon?ki and it returned back Thessalon?ki not Athens, which it was accurate for me. I dont know why it was not accurate for the other members here. And of course if you are using a proxy then the GeoIP tool has no way of telling the real location of the visitor but the proxy's location in stead. Also in another page which it asked me if i allow it to run a javascipt Geo app and i replied positively it gave me my exact city and street number too! So many sites can identify accurately the correct city and even street some times, so there must be a way. -- What is now proved was at first only imagined! From rosuav at gmail.com Sat Jul 13 13:03:20 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 14 Jul 2013 03:03:20 +1000 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: On Sun, Jul 14, 2013 at 2:54 AM, Dennis Lee Bieber wrote: > Are you paying for a fixed IP number? I suspect you are if you were > running a world-accessible server. > > Obviously a fixed IP will be tied to a fixed connection and thereby to > a fixed location which can be provided to a location database. And even that is no guarantee. I have two connections, one static IP, the other dynamic. The static one fails geolocation by a greater distance than the other. No, there's no way to be sure. ChrisA From roy at panix.com Sat Jul 13 13:09:05 2013 From: roy at panix.com (Roy Smith) Date: Sat, 13 Jul 2013 13:09:05 -0400 Subject: GeoIP2 for retrieving city and region ? References: Message-ID: In article , Dennis Lee Bieber wrote: > Obviously a fixed IP will be tied to a fixed connection and thereby to > a fixed location which can be provided to a location database. And even then, it can be wrong. When I worked for EMC, they (apparently, from what I could see) back-hauled internet connections for all their offices to Hopkinton, MA where they had one central ISP connection, via NAT. So, when I sat at my desk in White Plains, NY and went to Moviephone's web site, I was conveniently shown what all the theaters in the Hopkinton area were playing that day. Ditto for the weather. Not to mention that some IP addresses move. People use mobile devices in cars, busses, trains, planes, boats, etc. Not to mention somewhat more esoteric places like the ISS. From nikos at superhost.gr Sat Jul 13 13:43:14 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Ouc66z4zOu86xz4I=?=) Date: Sat, 13 Jul 2013 20:43:14 +0300 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: ???? 13/7/2013 7:54 ??, ?/? Dennis Lee Bieber ??????: > Are you paying for a fixed IP number? I suspect you are if you were > running a world-accessible server. > > Obviously a fixed IP will be tied to a fixed connection and thereby to > a fixed location which can be provided to a location database. > > But most of us have DHCP assigned IP numbers, which change everytime we > reboot our connection (or even when the DHCP lease expires -- which may be > daily). Same networking scheme for me too, dynamic that is. Every time the DHCP lease expires or i reboot the router i get a new ip address but every time the link i provided states accurately that my ip address is from Thessalon?ki and not Europe/Athens which is were my ISP location is. Not to mention that in facebook, no matter the way i'am joining, via smartphone, tablet, laptop it always pinpoints my exact location. But yes, i can understand your skepticism. An ip address can move anywhere while remaining connected to the same ISP, just like a networking device in the house, remains connected to the same router while changing rooms or even floors, or even buildings. But then how do you explain the fact that http://www.maxmind.com/en/geoip_demo pinpointed Thessalon?ki and not Athens and for 2 friends of mine that use the same ISP as me but live in different cities also accurately identified their locations too? I -- What is now proved was at first only imagined! From roy at panix.com Sat Jul 13 14:19:00 2013 From: roy at panix.com (Roy Smith) Date: Sat, 13 Jul 2013 14:19:00 -0400 Subject: GeoIP2 for retrieving city and region ? References: Message-ID: In article , ???????????????????? wrote: > But then how do you explain the fact that > http://www.maxmind.com/en/geoip_demo pinpointed Thessalon??ki and not > Athens and for 2 friends of mine that use the same ISP as me but live > in different cities also accurately identified their locations too? I just tried 24.136.109.105 on that demo. It comes up with: US Englewood Cliffs, New Jersey, United States, North America 07632 40.8915, -73.9471 Time Warner Cable Time Warner Cable rr.com 501 Not bad. Google street view shows that as a very pretty residential neighborhood in one of New York's fancier suburbs. Unfortunately, it happens to be in run-down industrial building in a factory district about 20 km away. There are lots of interesting (and superior) ways to do geolocation other than looking up IP addresses. Here's a few: 1) GPS. Obviously, if you're on a device that has a GPS receiver and you have access to that data, and you've got a good signal, nothing is going to beat GPS. Well, other than Glonass. And Galileo and IRNSS, whenever they become operational. And whatever the Chinese are calling theirs. 2) Cell (i.e. mobile) phone tower triangulation. The phone systems know where all the towers are and know which towers your phone is receiving signal from. Since they know the signal strengths from each of those towers, they can do a rough triangulation. It's kind of messy since signal propagation depends terrain and obstructions which aren't well mapped. But it's better than nothing. 3) WiFi triangulation. Right now, I can see four WiFi networks (Worb, J24, MusicWiFi, and jcglinksys). There are databases of WiFi network names and approximate locations (obtained by wardriving and other ways). If you can see enough networks, it's easy to look in the database and figure out where you must be. 4) Who knows what the future will bring. I suppose some day, inertial nav will become cheap enough that we'll all be walking around with INS in our phones. In general, mobile operating systems control direct access to all of these signals and only allow applications to get the location data when the user agrees to such access. From nikos at superhost.gr Sat Jul 13 14:23:07 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Ouc66z4zOu86xz4I=?=) Date: Sat, 13 Jul 2013 21:23:07 +0300 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: ???? 13/7/2013 9:19 ??, ?/? Roy Smith ??????: > In article , > ?????????? wrote: > >> But then how do you explain the fact that >> http://www.maxmind.com/en/geoip_demo pinpointed Thessalon?ki and not >> Athens and for 2 friends of mine that use the same ISP as me but live >> in different cities also accurately identified their locations too? > > I just tried 24.136.109.105 on that demo. It comes up with: > > US Englewood Cliffs, > New Jersey, > United States, > North America > 07632 40.8915, > -73.9471 > Time Warner Cable > Time Warner Cable > rr.com > 501 > > Not bad. Google street view shows that as a very pretty residential > neighborhood in one of New York's fancier suburbs. Unfortunately, it > happens to be in run-down industrial building in a factory district > about 20 km away. > > There are lots of interesting (and superior) ways to do geolocation > other than looking up IP addresses. Here's a few: > > 1) GPS. Obviously, if you're on a device that has a GPS receiver and > you have access to that data, and you've got a good signal, nothing is > going to beat GPS. Well, other than Glonass. And Galileo and IRNSS, > whenever they become operational. And whatever the Chinese are calling > theirs. > > 2) Cell (i.e. mobile) phone tower triangulation. The phone systems know > where all the towers are and know which towers your phone is receiving > signal from. Since they know the signal strengths from each of those > towers, they can do a rough triangulation. It's kind of messy since > signal propagation depends terrain and obstructions which aren't well > mapped. But it's better than nothing. > > 3) WiFi triangulation. Right now, I can see four WiFi networks (Worb, > J24, MusicWiFi, and jcglinksys). There are databases of WiFi network > names and approximate locations (obtained by wardriving and other ways). > If you can see enough networks, it's easy to look in the database and > figure out where you must be. > > 4) Who knows what the future will bring. I suppose some day, inertial > nav will become cheap enough that we'll all be walking around with INS > in our phones. > > In general, mobile operating systems control direct access to all of > these signals and only allow applications to get the location data when > the user agrees to such access. > Do you know a way of implementing anyone of these methods to a script? -- What is now proved was at first only imagined! From roy at panix.com Sat Jul 13 14:30:47 2013 From: roy at panix.com (Roy Smith) Date: Sat, 13 Jul 2013 14:30:47 -0400 Subject: GeoIP2 for retrieving city and region ? References: Message-ID: In article , ???????????????????? wrote: > > There are lots of interesting (and superior) ways to do geolocation > > other than looking up IP addresses. Here's a few: > . [...] > > In general, mobile operating systems control direct access to all of > > these signals and only allow applications to get the location data when > > the user agrees to such access. > > > Do you know a way of implementing anyone of these methods to a script? Unfortunately, no. I don't do front-end development. I am aware of the technologies, but do not know the details of how you access them on any particular device. I would start with the API docs for the device you are interested in and look for "location services". From torriem at gmail.com Sat Jul 13 18:57:42 2013 From: torriem at gmail.com (Michael Torrie) Date: Sat, 13 Jul 2013 16:57:42 -0600 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: <51E1DB66.6040707@gmail.com> On 07/13/2013 12:23 PM, ??????? wrote: > Do you know a way of implementing anyone of these methods to a script? Yes. Modern browsers all support a location API in the browser for javascript. See this: http://diveintohtml5.info/geolocation.html From python.list at tim.thechases.com Sun Jul 14 14:32:54 2013 From: python.list at tim.thechases.com (Tim Chase) Date: Sun, 14 Jul 2013 13:32:54 -0500 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: <51E1DB66.6040707@gmail.com> References: <51E1DB66.6040707@gmail.com> Message-ID: <20130714133254.18c4e008@bigbox.christie.dr> On 2013-07-13 16:57, Michael Torrie wrote: > On 07/13/2013 12:23 PM, ??????? wrote: > > Do you know a way of implementing anyone of these methods to a > > script? > > Yes. Modern browsers all support a location API in the browser for > javascript. And the good browsers give the user the option to disclose this information or not (and, as I mentioned elsewhere on this thread, even lie about where you are such as with the Geolocater plugin[1] for FF). Some of us value the modicum of privacy that we receive by not being locatable by IP address. -tkc [1] https://addons.mozilla.org/en-us/firefox/addon/geolocater/ From nikos at superhost.gr Tue Jul 16 15:43:35 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Ouc66z4zOu86xz4I=?=) Date: Tue, 16 Jul 2013 22:43:35 +0300 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: ???? 14/7/2013 1:57 ??, ?/? Michael Torrie ??????: > On 07/13/2013 12:23 PM, ??????? wrote: >> Do you know a way of implementing anyone of these methods to a script? > > Yes. Modern browsers all support a location API in the browser for > javascript. See this: > > http://diveintohtml5.info/geolocation.html > Lest say i embed inside my index.html the Javascript Geo Code. Is there a way to pass Javascript's outcome to my Python cgi script somehow? Can Javascript and Python Cgi somehow exchnage data? -- What is now proved was at first only imagined! From joel.goldstick at gmail.com Tue Jul 16 15:51:33 2013 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Tue, 16 Jul 2013 15:51:33 -0400 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: On Tue, Jul 16, 2013 at 3:43 PM, ??????? wrote: > ???? 14/7/2013 1:57 ??, ?/? Michael Torrie ??????: > >> On 07/13/2013 12:23 PM, ??????? wrote: >> >>> Do you know a way of implementing anyone of these methods to a script? >>> >> >> Yes. Modern browsers all support a location API in the browser for >> javascript. See this: >> >> http://diveintohtml5.info/**geolocation.html >> >> > Lest say i embed inside my index.html the Javascript Geo Code. > > Is there a way to pass Javascript's outcome to my Python cgi script > somehow? > > Can Javascript and Python Cgi somehow exchnage data? > > -- > What is now proved was at first only imagined! > -- > http://mail.python.org/**mailman/listinfo/python-list > Learn about ajax -- Joel Goldstick http://joelgoldstick.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Jul 16 20:58:36 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 17 Jul 2013 10:58:36 +1000 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: <22qbu8tkujd2s0e8ptvhmmt06nlfonh5p4@4ax.com> References: <22qbu8tkujd2s0e8ptvhmmt06nlfonh5p4@4ax.com> Message-ID: On Wed, Jul 17, 2013 at 10:51 AM, Dennis Lee Bieber wrote: > On Tue, 16 Jul 2013 22:43:35 +0300, ??????? declaimed > the following: > >> >>Lest say i embed inside my index.html the Javascript Geo Code. >> >>Is there a way to pass Javascript's outcome to my Python cgi script somehow? >> >>Can Javascript and Python Cgi somehow exchnage data? > > Using plain CGI is going to be painful -- since /everything/ is handled > a whole new page request. You would have to handle session cookies, etc. > > Your "index.html" would attempt to run the JavaScript (note that some > users may have JavaScript turned off -- how will you handle that), if it > gets information it would have to do a "GET index2.html?lat=xxx?long=yyy" > or something similar, which will result in a new page load on the user -- > hopefully with a cookie set so you know NOT to run the geolocation code on > subsequent pages. > > AJAX is a process to allow JavaScript on the browser to interact with a > server (using XML data structures), and likely use DOM operations in the > browser (and the last time I did something on this nature, I had to have > different code for Internet Explorer vs Firefox, and don't know of > Chrome/Opera share one of the others calls) to make changes on the web page > without doing a full submit/render operation AJAX changes the client end, but not the server (well, you might change the server's output format to make it easier, but it's not essential). So you *can* still use the CGI that you're familiar with. For reference, Firefox/Chrome/Opera/Safari are all pretty much identical for this sort of work; and the recent IEs (9, I think, and 10) are following them too. There are trivial differences, but for the basics, it's possible to support IE8+, Chrome, Firefox back as far as the Iceweasel from Debian Squeeze (I think that's Ff 3.5), and so on, all from the same code. No per-browser checks required. ChrisA From benjamin.kaplan at case.edu Sat Jul 13 14:17:15 2013 From: benjamin.kaplan at case.edu (Benjamin Kaplan) Date: Sat, 13 Jul 2013 11:17:15 -0700 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: On Sat, Jul 13, 2013 at 10:43 AM, ??????? wrote: > ???? 13/7/2013 7:54 ??, ?/? Dennis Lee Bieber ??????: >> >> Are you paying for a fixed IP number? I suspect you are if you >> were >> running a world-accessible server. >> >> Obviously a fixed IP will be tied to a fixed connection and >> thereby to >> a fixed location which can be provided to a location database. >> >> But most of us have DHCP assigned IP numbers, which change >> everytime we >> reboot our connection (or even when the DHCP lease expires -- which may be >> daily). > > > Same networking scheme for me too, dynamic that is. > > Every time the DHCP lease expires or i reboot the router i get a new ip > address but every time the link i provided states accurately that my ip > address is from Thessalon?ki and not Europe/Athens which is were my ISP > location is. > > Not to mention that in facebook, no matter the way i'am joining, via > smartphone, tablet, laptop it always pinpoints my exact location. > > But yes, i can understand your skepticism. > An ip address can move anywhere while remaining connected to the same ISP, > just like a networking device in the house, remains connected to the same > router while changing rooms or even floors, or even buildings. > > But then how do you explain the fact that > http://www.maxmind.com/en/geoip_demo > pinpointed Thessalon?ki and not Athens and for 2 friends of mine that use > the same ISP as me but live in different cities also accurately identified > their locations too? > It's not telling you where your ISP is headquartered. It's telling you where the servers that you're connecting to are. In your case, you're connecting to servers that your Athens-based ISP has in a Thessaloniki datacenter. The only way to get an accurate location is to use something other than IP- phones like to use a combination of their GPS, a map of the cell phone towers, and a map of wi-fi hotspots (this is one of the things that Google's StreetView cars log as they drive). From nikos at superhost.gr Sat Jul 13 14:28:25 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Ouc66z4zOu86xz4I=?=) Date: Sat, 13 Jul 2013 21:28:25 +0300 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: ???? 13/7/2013 9:21 ??, ?/? Dennis Lee Bieber ??????: > On Sat, 13 Jul 2013 20:43:14 +0300, ??????? declaimed > the following: > >> But then how do you explain the fact that >> http://www.maxmind.com/en/geoip_demo >> pinpointed Thessalon?ki and not Athens and for 2 friends of mine that >> use the same ISP as me but live in different cities also accurately >> identified their locations too? >> > You have encountered an ISP that reserves IP blocks for specific > suburbs/cities -- and can be tracked thereby. That is, all IPs in > xxx.yyy.zzz.??? may be held for just one neighborhood (granted, I've > defined an old class C network there, but modern network assignments are no > longer in A/B/C (8/16/24 bit network address, with 24/16/8 bit node > addresses) assignments. > > It all comes down to how the ISP allocates addresses. It may be that > (as in my case) the ISP is irrelevant -- the IP is issued by the phone > company that actually provides the DSL, and a block of IPs is allocated to > each "central office" of that phone company... And thereby, regardless of > the lease renewal, the IP is still identified as belonging to that central > office location (or whatever location the phone company supplies to the > location service for the IP block). > You are right and i can back this up. While FORTHnet ISP(my ISP) works well with maxmind which can pinpoint the visitor's exact location other ISPs like Cyta, OTEnet, HOL always says Europe/Athens. So it seems that all boil down to the way the ISP configure its blocks of ip addresses per city. All should do the same and then it would be an easy task to accurately identify a visitor by its ip address. -- What is now proved was at first only imagined! From rosuav at gmail.com Sat Jul 13 19:12:57 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 14 Jul 2013 09:12:57 +1000 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: On Sun, Jul 14, 2013 at 4:28 AM, ??????? wrote: > So it seems that all boil down to the way the ISP configure its blocks of ip > addresses per city. > > All should do the same and then it would be an easy task to accurately > identify a visitor by its ip address. So every ISP in the world needs to warp its business to your convenience. Are you at all thinking about what you're asking for, here? ChrisA From wayne at waynewerner.com Sat Jul 13 14:46:07 2013 From: wayne at waynewerner.com (Wayne Werner) Date: Sat, 13 Jul 2013 13:46:07 -0500 (CDT) Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: On Sat, 13 Jul 2013, ??????? wrote: > But then how do you explain the fact that > http://www.maxmind.com/en/geoip_demo > pinpointed Thessalon?ki and not Athens and for 2 friends of mine that > use the same ISP as me but live in different cities also accurately > identified their locations too? If you bothered doing something as simple as read the Wikipedia article on Geolocation, you could answer this question yourself: Evidently you, and your friends have things like cookies, or some other helps that identify your location, which is why your addresses are close. -W From nikos at superhost.gr Mon Jul 15 09:25:09 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Ouc66z4zOu86xz4I=?=) Date: Mon, 15 Jul 2013 16:25:09 +0300 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: ???? 13/7/2013 9:17 ??, ?/? Benjamin Kaplan ??????: > On Sat, Jul 13, 2013 at 10:43 AM, ??????? wrote: >> ???? 13/7/2013 7:54 ??, ?/? Dennis Lee Bieber ??????: >>> >>> Are you paying for a fixed IP number? I suspect you are if you >>> were >>> running a world-accessible server. >>> >>> Obviously a fixed IP will be tied to a fixed connection and >>> thereby to >>> a fixed location which can be provided to a location database. >>> >>> But most of us have DHCP assigned IP numbers, which change >>> everytime we >>> reboot our connection (or even when the DHCP lease expires -- which may be >>> daily). >> >> >> Same networking scheme for me too, dynamic that is. >> >> Every time the DHCP lease expires or i reboot the router i get a new ip >> address but every time the link i provided states accurately that my ip >> address is from Thessalon?ki and not Europe/Athens which is were my ISP >> location is. >> >> Not to mention that in facebook, no matter the way i'am joining, via >> smartphone, tablet, laptop it always pinpoints my exact location. >> >> But yes, i can understand your skepticism. >> An ip address can move anywhere while remaining connected to the same ISP, >> just like a networking device in the house, remains connected to the same >> router while changing rooms or even floors, or even buildings. >> >> But then how do you explain the fact that >> http://www.maxmind.com/en/geoip_demo >> pinpointed Thessalon?ki and not Athens and for 2 friends of mine that use >> the same ISP as me but live in different cities also accurately identified >> their locations too? >> > > It's not telling you where your ISP is headquartered. It's telling you > where the servers that you're connecting to are. In your case, you're > connecting to servers that your Athens-based ISP has in a Thessaloniki > datacenter. The only way to get an accurate location is to use > something other than IP- phones like to use a combination of their > GPS, a map of the cell phone towers, and a map of wi-fi hotspots (this > is one of the things that Google's StreetView cars log as they drive). Actually that happens only for my ISP(FORTHnet). For other ISPs all locations boil down just to Europe/Athens. This happens to be because my ISP's network scheme is to assign blcoks of ip addresses per city in Greek area. Same thing doesn't apply for others ISPs unfortunately here in Greece. I have no idea how to implement the solution you proposed. These are nice ideas we need to have a way of implement them within a script. I have no way of grasping a map of cell towers of a map of wi-fi hotspots. -- What is now proved was at first only imagined! From torriem at gmail.com Mon Jul 15 22:48:40 2013 From: torriem at gmail.com (Michael Torrie) Date: Mon, 15 Jul 2013 20:48:40 -0600 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: <51E4B488.5070906@gmail.com> On 07/15/2013 06:34 PM, Dennis Lee Bieber wrote: >> I have no idea how to implement the solution you proposed. >> These are nice ideas we need to have a way of implement them within a >> script. >> >> I have no way of grasping a map of cell towers of a map of wi-fi hotspots. >> > You don't... The phone company knows where their towers are, THEY do > the triangulation based on signal strength from cell phones on their > network, and they provide that position to the phone. The phone can then > use that data to respond to applications running ON the phone that request > location information using the phone's OS API (which is different for an > Android phone vs Blackberry vs Apple). I've posted a link to detailed information on this no less than three times, yet Nikos has not read any of it, sadly. From wuwei23 at gmail.com Tue Jul 16 20:07:30 2013 From: wuwei23 at gmail.com (alex23) Date: Wed, 17 Jul 2013 10:07:30 +1000 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: On 16/07/2013 12:48 PM, Michael Torrie wrote: > I've posted a link to detailed information on this no less than three > times, yet Nikos has not read any of it, sadly. Just a quick reminder for everyone: "Ferrous Cranus is utterly impervious to reason, persuasion and new ideas, and when engaged in battle he will not yield an inch in his position regardless of its hopelessness. Though his thrusts are decisively repulsed, his arguments crushed in every detail and his defenses demolished beyond repair he will remount the same attack again and again with only the slightest variation in tactics." http://www.politicsforum.org/images/flame_warriors/flame_62.php From nikos at superhost.gr Sun Jul 14 01:18:54 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Ouc66z4zOu86xz4I=?=) Date: Sun, 14 Jul 2013 08:18:54 +0300 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: Can we get the location serived from lat/long coordinates? Using a mapping service you can create a query that asks "what is the smallest region that contains points [(x1, y1), (x2, y2),...]" and use that as our geolocate answer? That should work, can you show me how such thing can be done please? -- What is now proved was at first only imagined! From rosuav at gmail.com Sun Jul 14 01:24:32 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 14 Jul 2013 15:24:32 +1000 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: On Sun, Jul 14, 2013 at 3:18 PM, ??????? wrote: > Can we get the location serived from lat/long coordinates? Yes, assuming you get accurate latitude and longitude, so you're back to square 1. ChrisA From nikos at superhost.gr Sun Jul 14 03:51:24 2013 From: nikos at superhost.gr (=?UTF-8?B?zp3Ouc66z4zOu86xz4I=?=) Date: Sun, 14 Jul 2013 10:51:24 +0300 Subject: GeoIP2 for retrieving city and region ? In-Reply-To: References: Message-ID: ???? 14/7/2013 8:24 ??, ?/? Chris Angelico ??????: > On Sun, Jul 14, 2013 at 3:18 PM, ??????? wrote: >> Can we get the location serived from lat/long coordinates? > > Yes, assuming you get accurate latitude and longitude, so you're back > to square 1. > > ChrisA > Dear Freelance, Thank you for your interest in MaxMind Web Services. We have set up a demo account which includes the following web service(s): GeoIP City Demo (1000 lookups available) Usage: http://geoip.maxmind.com/b?l=YOUR_LICENSE_KEY&i=24.24.24.24 Example scripts may be found at: http://dev.maxmind.com/geoip/web-services GeoIP City with ISP and Organization Demo (1000 lookups available) Usage: http://geoip.maxmind.com/f?l=YOUR_LICENSE_KEY&i=24.24.24.24 Example scripts may be found at: http://dev.maxmind.com/geoip/web-services Lets see if that would be of any help. Please try it too you can request a demo trial of maxminds Geo web services. -- What is now proved was at first only imagined! From lo0446 at my.bristol.ac.uk Fri Jul 12 10:22:59 2013 From: lo0446 at my.bristol.ac.uk (L O'Shea) Date: Fri, 12 Jul 2013 07:22:59 -0700 (PDT) Subject: Understanding other people's code Message-ID: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> Hi all, I've been asked to take over a project from someone else and to extend the functionality of this. The project is written in Python which I haven't had any real experience with (although I do really like it) so I've spent the last week or two settling in, trying to get my head around Python and the way in which this code works. The problem is the code was clearly written by someone who is exceptionally good and seems to inherit everything from everywhere else. It all seems very dynamic, nothing is written statically except in some configuration files. Basically the problem is I am new to the language and this was clearly written by someone who at the moment is far better at it than I am! I'm starting to get pretty worried about my lack of overall progress and so I wondered if anyone out there had some tips and techniques for understanding other peoples code. There has to be 10/15 different scripts with at least 10 functions in each file I would say. Literally any idea will help, pen and paper, printing off all the code and doing some sort of highlighting session - anything! I keep reading bits of code and thinking "well where the hell has that been defined and what does it mean" to find it was inherited from 3 modules up the chain. I really need to get a handle on how exactly all this slots together! Any techniques,tricks or methodologies that people find useful would be much appreciated. From rosuav at gmail.com Fri Jul 12 10:46:15 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 13 Jul 2013 00:46:15 +1000 Subject: Understanding other people's code In-Reply-To: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> References: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> Message-ID: On Sat, Jul 13, 2013 at 12:22 AM, L O'Shea wrote: > I'm starting to get pretty worried about my lack of overall progress and so I wondered if anyone out there had some tips and techniques for understanding other peoples code. There has to be 10/15 different scripts with at least 10 functions in each file I would say. The first thing I'd recommend is getting yourself familiar with the language itself, and (to some extent) the standard library. Then you'll know that any unrecognized wotzit must have come from your own project, so you'll be able to search up its definition. Then I'd tackle source files one at a time, and look at the very beginning. If the original coder was at all courteous, each file will start off with a block of 'import' statements, looking something like this: import re import itertools Or possibly like this: from itertools import cycle, islice Or, if you're unlucky, like this: from tkinter import * The first form is easy. You'll find references to "re.sub" or "itertools.takewhile"; the second form at least names what it's grabbing (so you'll find "cycle" or "islice" in the code), and the third just dumps a whole lot of stuff into your namespace. Actually, if the programmer's been really nice, there'll be a block comment or a docstring at the top of the file, which might even be up-to-date and accurate. But I'm guessing you already know to look for that. :) The other thing I would STRONGLY recommend: Keep the interactive interpreter handy. Any line of code you don't understand, paste it into the interpreter. Chances are it won't wipe out your entire hard drive :) But seriously, there is much to gain and nothing to lose by keeping IDLE or the command-line interpreter handy. ChrisA From __peter__ at web.de Fri Jul 12 11:21:39 2013 From: __peter__ at web.de (Peter Otten) Date: Fri, 12 Jul 2013 17:21:39 +0200 Subject: Understanding other people's code References: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> Message-ID: L O'Shea wrote: > Hi all, > I've been asked to take over a project from someone else and to extend the > functionality of this. The project is written in Python which I haven't > had any real experience with (although I do really like it) so I've spent > the last week or two settling in, trying to get my head around Python and > the way in which this code works. > > The problem is the code was clearly written by someone who is > exceptionally good and seems to inherit everything from everywhere else. > It all seems very dynamic, nothing is written statically except in some > configuration files. Basically the problem is I am new to the language and > this was clearly written by someone who at the moment is far better at it > than I am! > > I'm starting to get pretty worried about my lack of overall progress and > so I wondered if anyone out there had some tips and techniques for > understanding other peoples code. There has to be 10/15 different scripts > with at least 10 functions in each file I would say. That sounds like the project is well-organised and not too big. If you take one day per module you're there in two weeks... > Literally any idea will help, pen and paper, printing off all the code and > doing some sort of highlighting session - anything! I keep reading bits of > code and thinking "well where the hell has that been defined and what does > it mean" to find it was inherited from 3 modules up the chain. As you put it here, the project is too complex. So now we have a mixed message. Of course your impression may stem from lack of experience... > I really > need to get a handle on how exactly all this slots together! Any > techniques,tricks or methodologies that people find useful would be much > appreciated. Is there any documentation? Read that. Do the functions have docstrings? import the modules (start with the main entry point) in the interactive interpreter and use help(): >>> import some_module >>> help(some_module) Or use $ python -m pydoc -g and hit "open browser" (the project directory has to be in PYTHONPATH). See if you can talk to the author/previous maintainer. He may be willing to give you the big picture or hints for the parts where he did "clever" things. Try to improve your Python by writing unrelated scripts. Make little changes to the project (add print statements, invoke functions from your own driver script, make a local variable global for further inspection in the interactive interpreter using dir() -- whatever you can think of. The latter should of course be done in a test installation rather than the production environment. Rely on version control once you start making modifications for real -- but I think you knew that already... From esj at harvee.org Fri Jul 12 12:04:45 2013 From: esj at harvee.org (Eric S. Johansson) Date: Fri, 12 Jul 2013 12:04:45 -0400 Subject: Understanding other people's code In-Reply-To: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> References: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> Message-ID: On Fri, 12 Jul 2013 10:22:59 -0400, L O'Shea wrote: > Literally any idea will help, pen and paper, printing off all the code > and doing some sort of highlighting session - anything! I keep reading > bits of code and thinking "well where the hell has that been defined and > what does it mean" to find it was inherited from 3 modules up the chain. > I really need to get a handle on how exactly all this slots together! > Any techniques,tricks or methodologies that people find useful would be > much appreciated. glad to hear you're having a WTF moment (what's that function). Suggestion would be index cards, each containing notes on a class. truly understand what each parent class is in which methods are to be overloaded. Then look at one child and understand how it. Work your way breadth first down the inheritance tree. From tjreedy at udel.edu Fri Jul 12 18:11:54 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 12 Jul 2013 18:11:54 -0400 Subject: Understanding other people's code In-Reply-To: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> References: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> Message-ID: On 7/12/2013 10:22 AM, L O'Shea wrote: > Hi all, I've been asked to take over a project from someone else and > to extend the functionality of this. The project is written in Python > which I haven't had any real experience with (although I do really > like it) so I've spent the last week or two settling in, trying to > get my head around Python and the way in which this code works. If the functions are not documented in prose, is there a test suite that you can dive into? -- Terry Jan Reedy From joel.goldstick at gmail.com Fri Jul 12 19:49:31 2013 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Fri, 12 Jul 2013 19:49:31 -0400 Subject: Understanding other people's code In-Reply-To: References: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> Message-ID: On Fri, Jul 12, 2013 at 6:11 PM, Terry Reedy wrote: > On 7/12/2013 10:22 AM, L O'Shea wrote: > >> Hi all, I've been asked to take over a project from someone else and >> to extend the functionality of this. The project is written in Python >> which I haven't had any real experience with (although I do really >> like it) so I've spent the last week or two settling in, trying to >> get my head around Python and the way in which this code works. >> > > If the functions are not documented in prose, is there a test suite that > you can dive into? > > > -- > Terry Jan Reedy > > -- > http://mail.python.org/**mailman/listinfo/python-list > I'm very appreciative of pydoc. -- even for code I write myself!. Learn about it and redirect its output to files, so you can print out all of your modules. (well -- my suggestion!). For the functions and classes that are lacking docstrings, review them and see if you can figure out what they do. Add docstrings.. Not to disrespect this original coder in the slightest, but my work experience has been involved in reading and fixing or updating lots of other peoples code -- most less documented than would be nice. So my def of good code is code with good descriptive docstrings -- at the top level even before documenting the details. Its nice to know where the developer's head was at when the system was put together. -- Joel Goldstick http://joelgoldstick.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From cmpython at gmail.com Sun Jul 14 02:58:30 2013 From: cmpython at gmail.com (CM) Date: Sat, 13 Jul 2013 23:58:30 -0700 (PDT) Subject: Understanding other people's code In-Reply-To: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> References: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> Message-ID: > Basically the problem is I am new to the language and this was clearly > written by someone who at the moment is far better at it than I am! Sure, as a beginner, yes, but also it sounds like the programmer didn't document it much at all, and that doesn't help you. I bet s/he didn't always use very human readable names for objects/methods/classes, either, eh? > I'm starting to get pretty worried about my lack of overall progress and so I > wondered if anyone out there had some tips and techniques for understanding > other peoples code. There has to be 10/15 different scripts with at least 10 > functions in each file I would say. Unless the programmer was really super spaghetti coding, I would think that there would be some method to the madness, and that the 10-15 scripts each have some specific kind of purpose. The first thing, I'd think (and having not seen your codebase) would be to sketch out what those scripts do, and familiarize yourself with their names. Did the coder use this form for importing from modules? from client_utils import * If so, that's going to make your life much harder, because all of the names of the module will now be available to the script it was imported into, and yet they are not defined in that script. If s/he had written: import client_utils Than at least you would expect lines like this in the script you're looking at: customer_name = client_utils.GetClient() Or, if the naming is abstruse, at very least: cn = client_utils.GC() It's awful, but at least then you know that GC() is a function within the client_utils.py script and you don't have to go searching for it. If s/he did use "from module import *", then maybe it'd be worth it to re-do all the imports in the "import module" style, which will break everything, but then force you to go through all the errors and make the names like module.FunctionName() instead of just FunctionName(). Some of that depends on how big this project is, of course. > Literally any idea will help, pen and paper, printing off all the code and > doing some sort of highlighting session - anything! What tools are you using to work on this code? Do you have an IDE that has a "browse to" function that allows you to click on a name and see where in the code above it was defined? Or does it have UML or something like that? From lo0446 at my.bristol.ac.uk Mon Jul 15 06:02:30 2013 From: lo0446 at my.bristol.ac.uk (Azureaus) Date: Mon, 15 Jul 2013 03:02:30 -0700 (PDT) Subject: Understanding other people's code In-Reply-To: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> References: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> Message-ID: On Friday, 12 July 2013 15:22:59 UTC+1, Azureaus wrote: > Hi all, > > I've been asked to take over a project from someone else and to extend the functionality of this. The project is written in Python which I haven't had any real experience with (although I do really like it) so I've spent the last week or two settling in, trying to get my head around Python and the way in which this code works. > > > > The problem is the code was clearly written by someone who is exceptionally good and seems to inherit everything from everywhere else. It all seems very dynamic, nothing is written statically except in some configuration files. > > Basically the problem is I am new to the language and this was clearly written by someone who at the moment is far better at it than I am! > > > > I'm starting to get pretty worried about my lack of overall progress and so I wondered if anyone out there had some tips and techniques for understanding other peoples code. There has to be 10/15 different scripts with at least 10 functions in each file I would say. > > > > Literally any idea will help, pen and paper, printing off all the code and doing some sort of highlighting session - anything! I keep reading bits of code and thinking "well where the hell has that been defined and what does it mean" to find it was inherited from 3 modules up the chain. I really need to get a handle on how exactly all this slots together! Any techniques,tricks or methodologies that people find useful would be much appreciated. Thanks for all the suggestions, I'm afraid I didn't get a chance to view them over the weekend but I will get started with them this morning. I'm currently using sublime 2 for my text editor and tried to create a UML diagram using Pylint to try and get a map overview of what's going on. Unfortunately it seemed to map the classes into groups such as StringIO, ThreadPool, GrabOut etc.. rather than into the modules they belong go and how they fit together. Maybe this is just my inexperience showing through or I'm using the program wrong. If anyone has any 'mapping' programs they use to help them visualise program flow that would be a great bonus. To be fair to who programmed it, most functions are commented and I can't complain about the messiness of the code, It's actually very tidy. (I suppose Python forcing it's formatting is another reason it's an easily readable language!) Luckily not blanked import * were used otherwise I really would be up the creek without a paddle. Thanks! From cmpython at gmail.com Mon Jul 15 15:13:14 2013 From: cmpython at gmail.com (CM) Date: Mon, 15 Jul 2013 12:13:14 -0700 (PDT) Subject: Understanding other people's code In-Reply-To: References: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> Message-ID: On Monday, July 15, 2013 6:02:30 AM UTC-4, Azureaus wrote: > To be fair to who programmed it, most functions are commented and I can't > complain about the messiness of the code, It's actually very tidy. (I suppose > Python forcing it's formatting is another reason it's an easily readable > language!) Luckily not blanked import * were used otherwise I really would be > up the creek without a paddle. Oh, good! OK, so then what you can think in terms of, in terms of a simple strategy for getting clear without any fancy tools: Learn what each module is for. In my own application programming, I don't just put random classes and functions in any old module--the modules have some order to them. So, for example, one module may represent one panel in the application, or all the database stuff, or all the graphing stuff, or some other set of logic, or whatever. One might be the main GUI frame. Etc. So I'd get a notebook or file and make notes for yourself about what each module is for, and the name. Even tack a piece of paper above your workstation with the module names and a one line note about what they do, like: MODULES: Map_panel: Displays a panel with the map of the city, with a few buttons. Dbases: Has all utility functions relevant to the database. Utils: Has a collection of utility functions to format time, i18n, etc. Now, there's a cheat sheet. So, if you come across a line in your code like: pretty_time = Utils.GetPrettyTime(datetime) You can quickly look at Utils module and read more about that function. Does this approach make sense to at least clear the cobwebs? From jeanmichel at sequans.com Tue Jul 16 05:09:16 2013 From: jeanmichel at sequans.com (Jean-Michel Pichavant) Date: Tue, 16 Jul 2013 11:09:16 +0200 (CEST) Subject: Understanding other people's code In-Reply-To: Message-ID: <1683526321.10342703.1373965756785.JavaMail.root@sequans.com> ----- Original Message ----- > Thanks for all the suggestions, I'm afraid I didn't get a chance to > view them over the weekend but I will get started with them this > morning. I'm currently using sublime 2 for my text editor and tried > to create a UML diagram using Pylint to try and get a map overview > of what's going on. Unfortunately it seemed to map the classes into > groups such as StringIO, ThreadPool, GrabOut etc.. rather than into > the modules they belong go and how they fit together. Maybe this is > just my inexperience showing through or I'm using the program wrong. > If anyone has any 'mapping' programs they use to help them visualise > program flow that would be a great bonus. > > To be fair to who programmed it, most functions are commented and I > can't complain about the messiness of the code, It's actually very > tidy. (I suppose Python forcing it's formatting is another reason > it's an easily readable language!) Luckily not blanked import * were > used otherwise I really would be up the creek without a paddle. > Thanks! Do not hesitate to ask this list about specific part of the code, looks like you've already put some efforts understanding it so people will happily try to help you. Just make sure you can actually expose the code to the public. JM -- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. From albert at spenarnc.xs4all.nl Sat Jul 27 09:13:08 2013 From: albert at spenarnc.xs4all.nl (Albert van der Horst) Date: 27 Jul 2013 13:13:08 GMT Subject: Understanding other people's code References: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> Message-ID: <51f3c764$0$26864$e4fe514c@dreader37.news.xs4all.nl> In article , Azureaus wrote: >On Friday, 12 July 2013 15:22:59 UTC+1, Azureaus wrote: > >To be fair to who programmed it, most functions are commented and I >can't complain about the messiness of the code, It's actually very tidy. >(I suppose Python forcing it's formatting is another reason it's an >easily readable language!) Luckily not blanked import * were used >otherwise I really would be up the creek without a paddle. If the code is really tidy, it is possible to understand a function using only the *documentation* (not the code itself) of any function or data it uses. In oo you also need a context about what an object is supposed to do. The next step is to proof for yourself that the function exactly does what is promised in its own documentation. And you get nowhere without domain knowledge. If you're in railways and don't know the difference between a "normal" and an "English" whathaveyou, then you're lost, plain and simple. Don't treat the original comment as sacred. Any time it is unclear rewrite it. You may get it wrong, but that's wat source control systems are for. If at all possible, if you add a statement about a function, try to add a test that proves that statement. Anytime you come across something that is unsufficiently documented, you document it tentatively yourself, keeping in mind that what you write down may be wrong. This does no harm! Because you must keep in mind that everything written by the original programmer may be wrong, there is actually no difference! Now study the places where it is called and check whether it makes sense. This an infinite process. After one round of improvements you have to go through everything again. I've got pretty bad stuff under control this way. You'll find bugs this way. They may or may not let you fix them. There is however not much point in "working in" by reading through the code. Time is probably better spent by running and studying, maybe creating test cases. Trying to understand any substantial code body in detail is a waste of time. For example: I once had to change the call code of the gcc compiler to be able to use a 68000 assembler library (regarding which register contain what data passed to the function). There is absolutely no point in studying the gcc compiler. You must have an overview then zoom in on the relevant part. In the end maybe only a couple of lines need change. A couple of days, and a pretty hairy problem was solved. (The assembler library was totally undocumented. Nobody even tried to study it. ). There is an indication that the original programmer made it all very easy and maybe you go about it not quite the right way. If you have a tower of abstractions, then you must *not* go down all the way to find out "eactly" what happens. You must pick a level in the middle and understand it in terms of usage, then understand what is on top of that in terms of that usage. That is how good programmers build there programs. Once there is a certain level they don't think about what's underneath, but concentrate on how to use it. If it is done really well, each source module can be understood on its own. All this is of course general, not just for Python. >Thanks! -- Albert van der Horst, UTRECHT,THE NETHERLANDS Economic growth -- being exponential -- ultimately falters. albert at spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst From rosuav at gmail.com Sat Jul 27 09:41:10 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 27 Jul 2013 14:41:10 +0100 Subject: Understanding other people's code In-Reply-To: <51f3c764$0$26864$e4fe514c@dreader37.news.xs4all.nl> References: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> <51f3c764$0$26864$e4fe514c@dreader37.news.xs4all.nl> Message-ID: On Sat, Jul 27, 2013 at 2:13 PM, Albert van der Horst wrote: > If the code is really tidy, it is possible to understand a function > using only the *documentation* (not the code itself) of any function > or data it uses. I'd broaden that slightly to the function's signature, which consists of the declaration line and any associated comments (which in Python should be in the docstring). The docstring kinda violates this concept, but what I generally try to explain is that you should be able to understand a function without reading any of the indented content. ChrisA From asimjalis at gmail.com Mon Jul 15 22:10:46 2013 From: asimjalis at gmail.com (asimjalis at gmail.com) Date: Mon, 15 Jul 2013 19:10:46 -0700 (PDT) Subject: Understanding other people's code In-Reply-To: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> References: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> Message-ID: On Friday, July 12, 2013 7:22:59 AM UTC-7, Azureaus wrote: > Hi all, > I've been asked to take over a project from someone else and to extend the functionality of this. The project is written in Python which I haven't had any real experience with (although I do really like it) so I've spent the last week or two settling in, trying to get my head around Python and the way in which this code works. Here are some techniques I use in these situations. 1. Do a superficial scan of the code looking at names of classes, functions, variables, and speculate where the modification that I have to make will go. Chances are you don't need to understand the entire system to make your change. 2. Build some hypotheses about how the system works and use print statements or some other debugging technique to run the program and see if you get the result you expect. 3. Insert your code into a separate class and function and see if you can inject a call to your new code from the existing code so that it now works with the new functionality. If you have to understand the details of some code, one approach is to try to summarize blocks of code with a single comment to wrap your mind around it. Asim From chess at us.ibm.com Tue Jul 16 14:38:14 2013 From: chess at us.ibm.com (David M Chess) Date: Tue, 16 Jul 2013 14:38:14 -0400 Subject: Understanding other people's code In-Reply-To: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> References: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> Message-ID: > Literally any idea will help, pen and paper, printing off all the code and doing some sort of highlighting session - anything! > I keep reading bits of code and thinking "well where the hell has that been defined and what does it mean" to find it was inherited from 3 modules up the chain. > I really need to get a handle on how exactly all this slots together! Any techniques,tricks or methodologies that people find useful would be much appreciated. I'd highly recommend Eclipse with PyDev, unless you have some strong reason not to. That's what I use, and it saves pretty much all of those "what's this thing?" problems, as well as lots of others... DC -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwightdhutto at gmail.com Tue Jul 16 20:05:18 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Tue, 16 Jul 2013 20:05:18 -0400 Subject: Understanding other people's code In-Reply-To: References: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> Message-ID: Any program, to me, is just like speaking english. The class, or function name might not fully mesh with what your cognitive structure assumes it to be.read through the imports first, and see the classes and functions come alive with experience comes intuition of what it does, and the instances that can be utilized with it. The term RTFM, and google always comes to mind as well. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwightdhutto at gmail.com Fri Jul 19 22:57:51 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Fri, 19 Jul 2013 22:57:51 -0400 Subject: Understanding other people's code In-Reply-To: References: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> Message-ID: I forgot to mention idle. It can step through another's code and show you a step-by-step insructional, of what the code does. On Tue, Jul 16, 2013 at 8:05 PM, David Hutto wrote: > Any program, to me, is just like speaking english. The class, or function > name might not fully mesh with what your cognitive structure assumes it to > be.read through the imports first, and see the classes and functions come > alive with experience comes intuition of what it does, and the instances > that can be utilized with it. The term RTFM, and google always comes to > mind as well. > -- Best Regards, David Hutto *CEO:* *http://www.hitwebdevelopment.com* -------------- next part -------------- An HTML attachment was scrubbed... URL: From lo0446 at my.bristol.ac.uk Thu Jul 25 12:26:08 2013 From: lo0446 at my.bristol.ac.uk (Azureaus) Date: Thu, 25 Jul 2013 09:26:08 -0700 (PDT) Subject: Understanding other people's code In-Reply-To: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> References: <66c25416-eaa5-4ac1-a71d-2b2948dec2fb@googlegroups.com> Message-ID: On Friday, July 12, 2013 3:22:59 PM UTC+1, Azureaus wrote: > Hi all, > > I've been asked to take over a project from someone else and to extend the functionality of this. The project is written in Python which I haven't had any real experience with (although I do really like it) so I've spent the last week or two settling in, trying to get my head around Python and the way in which this code works. > > > > The problem is the code was clearly written by someone who is exceptionally good and seems to inherit everything from everywhere else. It all seems very dynamic, nothing is written statically except in some configuration files. > > Basically the problem is I am new to the language and this was clearly written by someone who at the moment is far better at it than I am! > > > > I'm starting to get pretty worried about my lack of overall progress and so I wondered if anyone out there had some tips and techniques for understanding other peoples code. There has to be 10/15 different scripts with at least 10 functions in each file I would say. > > > > Literally any idea will help, pen and paper, printing off all the code and doing some sort of highlighting session - anything! I keep reading bits of code and thinking "well where the hell has that been defined and what does it mean" to find it was inherited from 3 modules up the chain. I really need to get a handle on how exactly all this slots together! Any techniques,tricks or methodologies that people find useful would be much appreciated. Thank you to everyone who replied constructively, the various suggestions all helped a lot. I'd like to suggest to anyone who reads this in the future who is in a similar situation to do as David Chess suggested and install eclipse with pydev. Although I prefer to use Sublime to actually write code, Eclipse turned out to be invaluable in helping me jump around and understand the code especially how things were passed around) and for debugging things over the last few days. Success! Cheers everyone. From rosuav at gmail.com Fri Jul 12 10:52:32 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 13 Jul 2013 00:52:32 +1000 Subject: =?UTF-8?B?UmU6IFtQeXRob24taWRlYXNdIGZsb2F0KCfiiJ4nKT1mbG9hdCgnaW5mJyk=?= In-Reply-To: References: Message-ID: On Sat, Jul 13, 2013 at 12:43 AM, Gerald Britton wrote: > Man I don't know how you are doing this! I just tried: > > float('') and got > > Value error: could not convert string to float '' > > For that matter, I can't figure out how to type the greek letter for > pi in gmail! Guess I have some things to learn. > > So, if Python doesn't recognize the symbol for pi, why should it > recognize the one for infinity? Considering that Python can't represent ? in a float anyway, I wouldn't be too bothered. And what else? float('?') for twice that value? Not really necessary imho. ChrisA From rosuav at gmail.com Fri Jul 12 10:53:21 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 13 Jul 2013 00:53:21 +1000 Subject: =?UTF-8?B?UmU6IFtQeXRob24taWRlYXNdIGZsb2F0KCfiiJ4nKT1mbG9hdCgnaW5mJyk=?= In-Reply-To: References: Message-ID: On Sat, Jul 13, 2013 at 12:52 AM, Chris Angelico wrote: > On Sat, Jul 13, 2013 at 12:43 AM, Gerald Britton > wrote: >> Man I don't know how you are doing this! I just tried: >> >> float('') and got >> >> Value error: could not convert string to float '' >> >> For that matter, I can't figure out how to type the greek letter for >> pi in gmail! Guess I have some things to learn. >> >> So, if Python doesn't recognize the symbol for pi, why should it >> recognize the one for infinity? > > Considering that Python can't represent ? in a float anyway, I > wouldn't be too bothered. And what else? float('?') for twice that > value? Not really necessary imho. > > ChrisA Take no notice of me going insane over here. Nothing to see, move along, these are not the neurons you're looking for! ChrisA From rosuav at gmail.com Sat Jul 13 21:53:12 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 14 Jul 2013 11:53:12 +1000 Subject: =?UTF-8?B?UmU6IFtQeXRob24taWRlYXNdIGZsb2F0KCfiiJ4nKT1mbG9hdCgnaW5mJyk=?= In-Reply-To: References: <51E07F3B.5000405@mrabarnett.plus.com> <51E1FDC5.1030501@pearwood.info> Message-ID: On Sun, Jul 14, 2013 at 11:39 AM, Joshua Landau wrote: > On 14 July 2013 02:24, Steven D'Aprano wrote: >> and on-going costs: >> >> - that's one more thing for every user to learn; > > Doesn't apply here. Yes, it does; what happens to someone who reads someone else's Python code? To write code, you need to understand one way of spelling something. To read code, you need to understand _every_ way of spelling that thing. That may not be a particularly great cost in this case, but it is a cost. ChrisA From rosuav at gmail.com Sat Jul 13 21:53:55 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 14 Jul 2013 11:53:55 +1000 Subject: =?UTF-8?B?UmU6IFtQeXRob24taWRlYXNdIGZsb2F0KCfiiJ4nKT1mbG9hdCgnaW5mJyk=?= In-Reply-To: References: <51E07F3B.5000405@mrabarnett.plus.com> <51E1FDC5.1030501@pearwood.info> Message-ID: On Sun, Jul 14, 2013 at 11:53 AM, Chris Angelico wrote: > On Sun, Jul 14, 2013 at 11:39 AM, Joshua Landau wrote: >> On 14 July 2013 02:24, Steven D'Aprano wrote: >>> and on-going costs: >>> >>> - that's one more thing for every user to learn; >> >> Doesn't apply here. > > Yes, it does; what happens to someone who reads someone else's Python > code? To write code, you need to understand one way of spelling > something. To read code, you need to understand _every_ way of > spelling that thing. That may not be a particularly great cost in this > case, but it is a cost. Doh, I forgot which channel this was on again :( It feels like a python-list thread. ChrisA From steve+comp.lang.python at pearwood.info Sat Jul 13 23:04:30 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 14 Jul 2013 03:04:30 GMT Subject: [Python-ideas] =?iso-2022-kr?q?float=28'=0E!D=0F'=29=3Dfloat=28'inf'=29?= References: <51E07F3B.5000405@mrabarnett.plus.com> <51E1FDC5.1030501@pearwood.info> Message-ID: <51e2153e$0$9505$c3e8da3$5496439d@news.astraweb.com> On Sun, 14 Jul 2013 11:53:55 +1000, Chris Angelico wrote: > Doh, I forgot which channel this was on again :( It feels like a > python-list thread. Can't you just hit Reply-List or even Reply-All? -- Steven From rosuav at gmail.com Sat Jul 13 23:09:01 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 14 Jul 2013 13:09:01 +1000 Subject: =?UTF-8?B?UmU6IFtQeXRob24taWRlYXNdIGZsb2F0KCfiiJ4nKT1mbG9hdCgnaW5mJyk=?= In-Reply-To: <51e2153e$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <51E07F3B.5000405@mrabarnett.plus.com> <51E1FDC5.1030501@pearwood.info> <51e2153e$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sun, Jul 14, 2013 at 1:04 PM, Steven D'Aprano wrote: > On Sun, 14 Jul 2013 11:53:55 +1000, Chris Angelico wrote: > >> Doh, I forgot which channel this was on again :( It feels like a >> python-list thread. > > > Can't you just hit Reply-List or even Reply-All? I don't like Reply-All as it's too easy to send the double-ups, and Gmail doesn't have Reply-List. I should switch to Thunderbird or something at some point, but haven't gotten around to it yet. Incidents like this are a definite push, but my D&D campaign is demanding my attention right now, so I haven't made the move. ChrisA From storchaka at gmail.com Sun Jul 14 06:23:37 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 14 Jul 2013 13:23:37 +0300 Subject: [Python-ideas] =?UTF-8?B?ZmxvYXQoJ+KInicpPWZsb2F0KCdpbmYnKQ==?= In-Reply-To: References: <51E07F3B.5000405@mrabarnett.plus.com> <51E1FDC5.1030501@pearwood.info> <51e2153e$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: 14.07.13 06:09, Chris Angelico ???????(??): > Incidents like this are a definite push, but my D&D campaign is > demanding my attention right now, so I haven't made the move. Are you role-playing Chaos Mage [1]? [1] http://www.dandwiki.com/wiki/Chaos_Mage_(3.5e_Class) From rosuav at gmail.com Sun Jul 14 06:27:43 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 14 Jul 2013 20:27:43 +1000 Subject: =?UTF-8?B?UmU6IFtQeXRob24taWRlYXNdIGZsb2F0KCfiiJ4nKT1mbG9hdCgnaW5mJyk=?= In-Reply-To: References: <51E07F3B.5000405@mrabarnett.plus.com> <51E1FDC5.1030501@pearwood.info> <51e2153e$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sun, Jul 14, 2013 at 8:23 PM, Serhiy Storchaka wrote: > 14.07.13 06:09, Chris Angelico ???????(??): > >> Incidents like this are a definite push, but my D&D campaign is >> demanding my attention right now, so I haven't made the move. > > > Are you role-playing Chaos Mage [1]? > > [1] http://www.dandwiki.com/wiki/Chaos_Mage_(3.5e_Class) I should probably try that some time. Though in our D&D parties, I tend to land the role of Dungeon Master by default... still looking for people to DM some more campaigns. ChrisA From wayne at waynewerner.com Fri Jul 12 14:11:12 2013 From: wayne at waynewerner.com (Wayne Werner) Date: Fri, 12 Jul 2013 13:11:12 -0500 (CDT) Subject: UTF-EBCDIC encoding? Message-ID: Is anyone aware of a UTF-EBCDIC[1] decoder? While Python does have a few EBCDIC dialects in the codecs, it does not have the (relatively new?) UTF-EBCDIC one. Additionally, if anyone is aware of a Python tool that can unpack a mainframe PDS file, that would also be worthwhile. Thanks, Wayne [1]: https://en.wikipedia.org/wiki/UTF-EBCDIC From joel.goldstick at gmail.com Fri Jul 12 14:35:14 2013 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Fri, 12 Jul 2013 14:35:14 -0400 Subject: UTF-EBCDIC encoding? In-Reply-To: References: Message-ID: On Fri, Jul 12, 2013 at 2:11 PM, Wayne Werner wrote: > Is anyone aware of a UTF-EBCDIC[1] decoder? > > While Python does have a few EBCDIC dialects in the codecs, it does not > have the (relatively new?) UTF-EBCDIC one. > > Additionally, if anyone is aware of a Python tool that can unpack a > mainframe PDS file, that would also be worthwhile. > > > Thanks, > Wayne > > [1]: https://en.wikipedia.org/wiki/**UTF-EBCDIC > -- > http://mail.python.org/**mailman/listinfo/python-list > I can't help you. I'm astonished. Trying to imagine the work environment where this technology would be necessary -- Joel Goldstick http://joelgoldstick.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Fri Jul 12 15:12:19 2013 From: skip at pobox.com (Skip Montanaro) Date: Fri, 12 Jul 2013 14:12:19 -0500 Subject: UTF-EBCDIC encoding? In-Reply-To: References: Message-ID: > I can't help you. I'm astonished. Trying to imagine the work environment > where this technology would be necessary http://www.iseriespython.com/app/ispMain.py/Start?job=Home Skip From joel.goldstick at gmail.com Fri Jul 12 15:34:41 2013 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Fri, 12 Jul 2013 15:34:41 -0400 Subject: UTF-EBCDIC encoding? In-Reply-To: References: Message-ID: On Fri, Jul 12, 2013 at 3:12 PM, Skip Montanaro wrote: > > I can't help you. I'm astonished. Trying to imagine the work > environment > > where this technology would be necessary > > http://www.iseriespython.com/app/ispMain.py/Start?job=Home > > Skip > I remember the AS400 series.. although I never worked with one. What kind of business still use that stuff? Is it for large corporation accounting, MIS stuff? -- Joel Goldstick http://joelgoldstick.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevin.p.dwyer at gmail.com Mon Jul 15 16:47:00 2013 From: kevin.p.dwyer at gmail.com (Kev Dwyer) Date: Mon, 15 Jul 2013 21:47 +0100 Subject: UTF-EBCDIC encoding? References: Message-ID: Joel Goldstick wrote: > On Fri, Jul 12, 2013 at 3:12 PM, Skip Montanaro wrote: > >> > I can't help you. I'm astonished. Trying to imagine the work >> environment >> > where this technology would be necessary >> >> http://www.iseriespython.com/app/ispMain.py/Start?job=Home >> >> Skip >> > I remember the AS400 series.. although I never worked with one. What kind > of business still use that stuff? Is it for large corporation accounting, > MIS stuff? > > Some banks still run legacy systems on AS/400s, and I've seen them used for airline booking systems and retail POS. From wayne at waynewerner.com Mon Jul 15 18:48:52 2013 From: wayne at waynewerner.com (Wayne Werner) Date: Mon, 15 Jul 2013 17:48:52 -0500 (CDT) Subject: UTF-EBCDIC encoding? In-Reply-To: References: Message-ID: On Mon, 15 Jul 2013, Kev Dwyer wrote: > Joel Goldstick wrote: > >> On Fri, Jul 12, 2013 at 3:12 PM, Skip Montanaro wrote: >> >>>> I can't help you. I'm astonished. Trying to imagine the work >>> environment >>>> where this technology would be necessary >>> >>> http://www.iseriespython.com/app/ispMain.py/Start?job=Home >>> >>> Skip >>> >> I remember the AS400 series.. although I never worked with one. What kind >> of business still use that stuff? Is it for large corporation accounting, >> MIS stuff? >> >> > > Some banks still run legacy systems on AS/400s, and I've seen them used for > airline booking systems and retail POS. Sadly there are many larger corporations that have oodles of legacy code on the Mainframe. We run z/OS and IBM DB2 here. In my off time I fiddle around with Python in an attempt to make life more enjoyable - and one of these forays has led me to attempt to unpack some packed data. Which is also an interesting (if not terribly useful) project. -W From jmazzonelli at gmail.com Sat Jul 13 23:04:08 2013 From: jmazzonelli at gmail.com (Jorge Mazzonelli) Date: Sun, 14 Jul 2013 00:04:08 -0300 Subject: UTF-EBCDIC encoding? In-Reply-To: References: Message-ID: Hi, Regarding your second part of your mail, I have used in the past this page[1] as a base for code to process a mainframe PS file with packed decimals. May be it can help you a bit... Best regards, Jorge [1]: http://www.linuxtopia.org/online_books/programming_books/python_programming/python_ch34s05.html On Fri, Jul 12, 2013 at 3:11 PM, Wayne Werner wrote: > Is anyone aware of a UTF-EBCDIC[1] decoder? > > While Python does have a few EBCDIC dialects in the codecs, it does not > have the (relatively new?) UTF-EBCDIC one. > > Additionally, if anyone is aware of a Python tool that can unpack a > mainframe PDS file, that would also be worthwhile. > > > Thanks, > Wayne > > [1]: https://en.wikipedia.org/wiki/**UTF-EBCDIC > -- > http://mail.python.org/**mailman/listinfo/python-list > -- -- (\__/) (='.'=)This is Bunny. Copy and paste bunny into your (")_(")signature to help him gain world domination. -------------- next part -------------- An HTML attachment was scrubbed... URL: From o at owenmarshall.invalid Mon Jul 15 20:40:19 2013 From: o at owenmarshall.invalid (Owen Marshall) Date: 16 Jul 2013 00:40:19 GMT Subject: UTF-EBCDIC encoding? References: Message-ID: On 2013-07-12, Joel Goldstick wrote: > --047d7bdc8be492d67804e154c580 Content-Type: text/plain; charset=UTF-8 > > On Fri, Jul 12, 2013 at 2:11 PM, Wayne Werner > wrote: > >> Is anyone aware of a UTF-EBCDIC[1] decoder? >> >> While Python does have a few EBCDIC dialects in the codecs, it does >> not have the (relatively new?) UTF-EBCDIC one. >> >> Additionally, if anyone is aware of a Python tool that can unpack a >> mainframe PDS file, that would also be worthwhile. >> >> >> Thanks, Wayne >> >> [1]: >> https://en.wikipedia.org/wiki/**UTF-EBCDIC >> -- >> http://mail.python.org/**mailman/listinfo/python-list >> > > > I can't help you. I'm astonished. Trying to imagine the work > environment where this technology would be necessary > Ask any poor shmuck who has ever had to consume files created by the government -- especially stuff from the Social Security Administration -- and they'll tell horror stories about EBCDIC. Typically I've seen ``iconv'' used for this task, so I'd love to see a native solution... -- -owen From roy at panix.com Mon Jul 15 20:47:31 2013 From: roy at panix.com (Roy Smith) Date: Mon, 15 Jul 2013 20:47:31 -0400 Subject: UTF-EBCDIC encoding? References: Message-ID: In article , Owen Marshall wrote: > On 2013-07-12, Joel Goldstick wrote: > > --047d7bdc8be492d67804e154c580 Content-Type: text/plain; charset=UTF-8 > > > > On Fri, Jul 12, 2013 at 2:11 PM, Wayne Werner > > wrote: > > > >> Is anyone aware of a UTF-EBCDIC[1] decoder? > >> > >> While Python does have a few EBCDIC dialects in the codecs, it does > >> not have the (relatively new?) UTF-EBCDIC one. > >> > >> Additionally, if anyone is aware of a Python tool that can unpack a > >> mainframe PDS file, that would also be worthwhile. > >> > >> > >> Thanks, Wayne > >> > >> [1]: > >> https://en.wikipedia.org/wiki/**UTF-EBCDIC >> F-EBCDIC> > >> -- > >> http://mail.python.org/**mailman/listinfo/python-list >> g/mailman/listinfo/python-list> > >> > > > > > > I can't help you. I'm astonished. Trying to imagine the work > > environment where this technology would be necessary > > > > Ask any poor shmuck who has ever had to consume files created by the > government -- especially stuff from the Social Security Administration > -- and they'll tell horror stories about EBCDIC. > > Typically I've seen ``iconv'' used for this task, so I'd love to see a > native solution... Unix command-line: dd conv=ascii or perhaps dd conv=ascii,unblock cbs=80 ibs=80 if=/dev/cr of=/dev/lp (*) (*) My dd-fu has tarnished with age. Forgive me if I've munged the incantation somewhat. From peter at ifoley.id.au Sat Jul 13 02:43:55 2013 From: peter at ifoley.id.au (peter at ifoley.id.au) Date: Fri, 12 Jul 2013 23:43:55 -0700 (PDT) Subject: help with explaining how to split a list of tuples into parts Message-ID: Hi List, I am new to Python and wondering if there is a better python way to do something. As a learning exercise I decided to create a python bash script to wrap around the Python Crypt library (Version 2.7). My attempt is located here - https://gist.github.com/pjfoley/5989653 I am trying to wrap my head around list comprehensions, I have read the docs at http://docs.python.org/2/tutorial/datastructures.html#list-comprehensions and read various google results. I think my lack of knowledge is making it difficult to know what key word to search on. Essentially I have this list of tuples # Tuple == (Hash Method, Salt Length, Magic String, Hashed Password Length) supported_hashes=[('crypt',2,'',13), ('md5',8,'$1$',22), ('sha256',16,'$5$',43), ('sha512',16,'$6$',86)] This list contains the valid hash methods that the Crypt Library supports plus some lookup values I want to use in the code. I have managed to work out how to extract a list of just the first value of each tuple (line 16) which I use as part of the validation against the --hash argparse option. My Question. Looking at line 27, This line returns the tuple that mataches the hash type the user selects from the command line. Which I then split the seperate parts over lines 29 to 31. I am wondering if there is a more efficient way to do this such that I could do: salt_length, hash_type, expected_password_length = [x for x in supported_hashes if x[0] == args.hash] >From my limited understanding the first x is the return value from the function which meets the criteria. So could I do something like: ... = [(x[0][1], x[0][2], x[0][3]) for x in supported_hashes if x[0] == args.hash] I am happy to be pointed to some documentation which might help clarify what I need to do. Also if there is anything else that could be improved on with the code happy to be contacted off list. Thanks, Peter. From __peter__ at web.de Sat Jul 13 03:28:50 2013 From: __peter__ at web.de (Peter Otten) Date: Sat, 13 Jul 2013 09:28:50 +0200 Subject: help with explaining how to split a list of tuples into parts References: Message-ID: peter at ifoley.id.au wrote: > Hi List, > > I am new to Python and wondering if there is a better python way to do > something. As a learning exercise I decided to create a python bash > script to wrap around the Python Crypt library (Version 2.7). > > My attempt is located here - https://gist.github.com/pjfoley/5989653 > > I am trying to wrap my head around list comprehensions, I have read the > docs at > http://docs.python.org/2/tutorial/datastructures.html#list-comprehensions > and read various google results. I think my lack of knowledge is making > it difficult to know what key word to search on. > > Essentially I have this list of tuples > > # Tuple == (Hash Method, Salt Length, Magic String, Hashed Password > # Length) > supported_hashes=[('crypt',2,'',13), ('md5',8,'$1$',22), > ('sha256',16,'$5$',43), ('sha512',16,'$6$',86)] > > This list contains the valid hash methods that the Crypt Library supports > plus some lookup values I want to use in the code. > > I have managed to work out how to extract a list of just the first value > of each tuple (line 16) which I use as part of the validation against the > --hash argparse option. > > My Question. > > Looking at line 27, This line returns the tuple that mataches the hash > type the user selects from the command line. Which I then split the > seperate parts over lines 29 to 31. > > I am wondering if there is a more efficient way to do this such that I > could do: > > salt_length, hash_type, expected_password_length = [x for x in > supported_hashes if x[0] == args.hash] > > From my limited understanding the first x is the return value from the > function which meets the criteria. So could I do something like: > > ... = [(x[0][1], x[0][2], x[0][3]) for x in supported_hashes if x[0] == > args.hash] > > I am happy to be pointed to some documentation which might help clarify > what I need to do. > > Also if there is anything else that could be improved on with the code > happy to be contacted off list. Every time when you have to look up something you should think 'dict', and I expect that pretty that will happen automatically. Also, to split a tuple into its items you can "unpack" it: triple = (1, 2, 3) one, two, three = triple assert one == 1 and two == 2 and three == 3 So: supported_hashes = { "crypt": (2, "", 13), "md5": (8, "$1$", 22), ... } ... parser.add_argument( '--hash', default='sha512', choices=supported_hashes, # accept the keys help='Which Hash function to use') ... salt_length, hash_type, expected_password_length = supported_hashes[args.hash] ... From steve+comp.lang.python at pearwood.info Sat Jul 13 04:11:18 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 13 Jul 2013 08:11:18 GMT Subject: help with explaining how to split a list of tuples into parts References: Message-ID: <51e10ba5$0$9505$c3e8da3$5496439d@news.astraweb.com> On Fri, 12 Jul 2013 23:43:55 -0700, peter wrote: > Hi List, > > I am new to Python and wondering if there is a better python way to do > something. As a learning exercise I decided to create a python bash > script to wrap around the Python Crypt library (Version 2.7). A Python bash script? What does that mean? Python and bash are two different languages. > My attempt is located here - https://gist.github.com/pjfoley/5989653 A word of advice: don't assume that just because people are reading your posts, that they necessarily will follow links to view your code. There could be all sorts of reasons why they might not: - they may have email access, but are blocked from the web; - they may not have a browser that gets on with github; - they may be reading email via a smart phone, and not want to pay extra to go to a website; - too lazy, or too busy, to follow a link; - they don't want to get bogged down in trying to debug a large block of someone else's code. Or some other reason. For best results, you should try to simplify the problem as much as possible, bringing it down to the most trivial, easy example you can, small enough to include directly in the body of your email. That might be one line, or twenty lines. See also: http://sscce.org/ > I am trying to wrap my head around list comprehensions, I have read the > docs at > http://docs.python.org/2/tutorial/datastructures.html#list- comprehensions > and read various google results. I think my lack of knowledge is making > it difficult to know what key word to search on. I don't really think that list comps have anything to do with the problem at hand. You seem to have discovered a hammer (list comps) and are now trying to hammer everything with it. I don't think the list comp is the right tool for the job below. But, for what it is worth, a list comp is simply a short-cut for a for- loop, where the body of the loop is limited to a single expression. So this for-loop: result = [] for value in some_values: result.append(calculate(value)) can be re-written as this list comp: result = [calculate(value) for value in some_values] > Essentially I have this list of tuples > > # Tuple == (Hash Method, Salt Length, Magic String, Hashed Password > Length) > supported_hashes=[('crypt',2,'',13), ('md5',8,'$1$',22), > ('sha256',16,'$5$',43), ('sha512',16,'$6$',86)] > > This list contains the valid hash methods that the Crypt Library > supports plus some lookup values I want to use in the code. Consider using namedtuple to use named fields rather than just numbered fields. For example: from collections import namedtuple Record = namedtuple("Record", "method saltlen magic hashpwdlen") defines a new type called "Record", with four named fields. They you might do something like this: x = Record('crypt', 2, '', 13) print(x.saltlen) => prints 2 You might do this: supported_hashes = [ Record('crypt', 2, '', 13), Record('md5', 8, '$1$', 22), Record('sha256', 16, '$5$', 43), Record('sha512', 16, '$6$', 86), ] although I think a better plan would be to use a dict rather than a list, something like this: from collections import namedtuple CryptRecord = namedtuple("CryptRecord", "salt_length magic_string expected_password_length") supported_hashes = { 'crypt': CryptRecord(2, '', 13), 'md5': CryptRecord(8, '$1$', 22), 'sha256': CryptRecord(16, '$5$', 43), 'sha512': CryptRecord(16, '$6$', 86), } This will let you look crypt methods up by name: method = supported_hashes['md5'] print(method.magic_string) => prints '$1$' > I have managed to work out how to extract a list of just the first value > of each tuple (line 16) which I use as part of the validation against > the --hash argparse option. > > My Question. > > Looking at line 27, This line returns the tuple that mataches the hash > type the user selects from the command line. Which I then split the > seperate parts over lines 29 to 31. > > I am wondering if there is a more efficient way to do this such that I > could do: > > salt_length, hash_type, expected_password_length = [x for x in > supported_hashes if x[0] == args.hash] Have you tried it? What happens when you do so? What error message do you get? If you print the list comp, what do you get? Hint: on the left hand side, you have three names. On the right hand side, you have a list containing one item. That the list was created from a list comprehension is irrelevant. What happens when you do this? spam, ham, eggs = [(1, 2, 3)] # List of 1 item, a tuple. What happens when you extract the item out of the list? spam, ham, eggs = [(1, 2, 3)][0] > From my limited understanding the first x is the return value from the > function which meets the criteria. So could I do something like: No, not the first x. The *only* x, since you only have one x that matches the condition. Consider your list of supported_hashes. If you run this code: result = [] for x in supported_hashes: if x == arg.hash: result.append(x) what is the value of result? How many items does it have? If need be, call len(result) to see. You need to extract the first (only) item from the list. Then you will get a different error: *too many* items to unpack, instead of too few. So you need either an extra name on the left, which you ignore: spam, ham, eggs = [(1, 2, 3, 4)][0] # fails who_cares, spam, ham, eggs = [(1, 2, 3, 4)][0] or you need to reduce the number of items on the right: spam, ham, eggs = [(1, 2, 3, 4)][0][1:] Can you see why the last one works? The word you are looking for is "slicing", and you can test it like this: print( [100, 200, 300, 400, 500][1:] ) print( [100, 200, 300, 400, 500][2:4] ) print( [100, 200, 300, 400, 500][2:5] ) > ... = [(x[0][1], x[0][2], x[0][3]) for x in supported_hashes if x[0] == > args.hash] You don't need to manually split the x tuple into 3 pieces. Slicing is faster and simpler: [x[1:] for x in supported_hashes if x[0] == args.hash] But if you use my suggestion for a dictionary, you can just say: salt_length, hash_type, password_length = supported_hashes[args.hash] as a simple, fast lookup, no list comp needed. -- Steven From peter at ifoley.id.au Sat Jul 13 08:05:08 2013 From: peter at ifoley.id.au (peter at ifoley.id.au) Date: Sat, 13 Jul 2013 05:05:08 -0700 (PDT) Subject: help with explaining how to split a list of tuples into parts In-Reply-To: <51e10ba5$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <51e10ba5$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: <892e3baa-b214-4c57-a828-a51db0ff7595@googlegroups.com> On Saturday, 13 July 2013 18:11:18 UTC+10, Steven D'Aprano wrote: > On Fri, 12 Jul 2013 23:43:55 -0700, peter wrote: > > > > I am new to Python and wondering if there is a better python way to do > > A Python bash script? What does that mean? Python and bash are two > different languages. Sorry I was trying to give some context, my mistake I am glad it did not confuse the intent of my question and you were able to provide some really helpful suggestions below. > > > My attempt is located here - https://gist.github.com/pjfoley/5989653 > > A word of advice: don't assume that just because people are reading your > > posts, that they necessarily will follow links to view your code. There > > could be all sorts of reasons why they might not: > > * SNIP Justification * > > Or some other reason. For best results, you should try to simplify the > > problem as much as possible, bringing it down to the most trivial, easy > > example you can, small enough to include directly in the body of your > > email. That might be one line, or twenty lines. > In my defence I was trying to give some context for my problem domain. The bottom of the original post (which you provided some suggestions on) had the simplified question. I find that people are more likely to help if they don't think its just a *homework question* and that someone is genuinely trying to learn. As I mentioned in the very first line of my email I am just starting to learn Python and was not even sure what I was supposed to be looking for. Obviously I found the hammer straight away *BINGO*! However in my defence when I was searching I did not have the necessary knowledge to even start looking in the right place. For reference I started searching for slice's, splitting and separating. I will look into the techniques and keywords you suggested and try your various suggestions. Thanks for your pointers. Peter From roy at panix.com Sat Jul 13 10:50:38 2013 From: roy at panix.com (Roy Smith) Date: Sat, 13 Jul 2013 10:50:38 -0400 Subject: help with explaining how to split a list of tuples into parts References: <51e10ba5$0$9505$c3e8da3$5496439d@news.astraweb.com> <892e3baa-b214-4c57-a828-a51db0ff7595@googlegroups.com> Message-ID: In article <892e3baa-b214-4c57-a828-a51db0ff7595 at googlegroups.com>, peter at ifoley.id.au wrote: > In my defence I was trying to give some context for my problem domain. I think posting a link to the github page was perfectly fine. It wasn't a huge amount of code to look at, and the way github presents the code with line numbers (which you referred to) made it easy to understand what you were asking. From peter at ifoley.id.au Sat Jul 13 07:47:10 2013 From: peter at ifoley.id.au (peter at ifoley.id.au) Date: Sat, 13 Jul 2013 04:47:10 -0700 (PDT) Subject: help with explaining how to split a list of tuples into parts In-Reply-To: References: Message-ID: <2a457e73-cbb2-4f2f-9b84-5feaaab809dd@googlegroups.com> On Saturday, 13 July 2013 17:28:50 UTC+10, Peter Otten wrote: > > Every time when you have to look up something you should think 'dict', and I > > expect that pretty that will happen automatically. > > Also, to split a tuple into its items you can "unpack" it: > > > > triple = (1, 2, 3) > > one, two, three = triple > > assert one == 1 and two == 2 and three == 3 > > > > So: > > > > supported_hashes = { > > "crypt": (2, "", 13), > > "md5": (8, "$1$", 22), > > ... > > } > > ... > > parser.add_argument( > > '--hash', default='sha512', > > choices=supported_hashes, # accept the keys > > help='Which Hash function to use') > > ... > > salt_length, hash_type, expected_password_length = supported_hashes[args.hash] > > ... Hi Peter, Thanks for the pointers I will try your suggestion out and read some more. Peter. From roy at panix.com Sat Jul 13 10:33:27 2013 From: roy at panix.com (Roy Smith) Date: Sat, 13 Jul 2013 10:33:27 -0400 Subject: help with explaining how to split a list of tuples into parts References: Message-ID: In article , peter at ifoley.id.au wrote: > Hi List, > > I am new to Python and wondering if there is a better python way to do > something. As a learning exercise I decided to create a python bash script > to wrap around the Python Crypt library (Version 2.7). > > My attempt is located here - https://gist.github.com/pjfoley/5989653 This looks like it should work, but it's a kind of weird use of list comprehensions. Fundamentally, you're not trying to create a list, you're trying to select the one item which matches your key. A better data structure would be a dict: supported_hashes={'crypt': (2, '', 13), 'md5': (8, '$1$', 22), 'sha256': (16, '$5$', 43), 'sha512': (16, '$6$', 86), } then your selection logic becomes: try: crypt_tuple = supported_hashes[args.hash] except KeyError: print "unknown hash type" Another thing you might want to look into is named tuples (http://docs.python.org/2/library/collections.html). You could do something like: from collections import namedtuple HashInfo = namedtuple('HashInfo', ['salt_length', 'hash_type', 'expected_password_length']) supported_hashes={'crypt': HashInfo(2, '', 13), 'md5': HashInfo(8, '$1$', 22), 'sha256': HashInfo(16, '$5$', 43), 'sha512': HashInfo(16, '$6$', 86), } and now you can refer to the tuple elements by name instead of by numeric index. From simfake.mapping at gmail.com Sat Jul 13 04:53:04 2013 From: simfake.mapping at gmail.com (Simfake Fake) Date: Sat, 13 Jul 2013 18:23:04 +0930 Subject: Bluetooth Sockets Message-ID: Hi. I'm trying to connect to a bluetooth serial adaptor using python 3.x. However, in python 3.3.2 win x32, I get "AttributeError: module has no attribute AF_..." when trying to use socket.AF_BLUETOOTH, despite the docs http://docs.python.org/3.3/library/socket.html . The code I have is very similar to that found http://kevindoran1.blogspot.com.au/2013/04/bluetooth-programming-with-python-3.html I need to use python 3.x, as this is eventually going to run within Blender. It's going to run within blender so I can use blender to visualize and solve inverse-kinematic stuff for a robot arm. Regards Jak_o_Shadows -------------- next part -------------- An HTML attachment was scrubbed... URL: From simfake.mapping at gmail.com Tue Jul 16 01:52:27 2013 From: simfake.mapping at gmail.com (Simfake Fake) Date: Tue, 16 Jul 2013 15:22:27 +0930 Subject: Bluetooth Sockets In-Reply-To: References: Message-ID: Just bumping this, but has anybody have any personal experience with bluetooth in python 3? Perhaps my issue is that the windows version doesn't include it? On Sat, Jul 13, 2013 at 6:23 PM, Simfake Fake wrote: > Hi. I'm trying to connect to a bluetooth serial adaptor using python 3.x. > However, in python 3.3.2 win x32, I get "AttributeError: module has no > attribute AF_..." when trying to use socket.AF_BLUETOOTH, despite the docs > http://docs.python.org/3.3/library/socket.html . The code I have is very > similar to that found > http://kevindoran1.blogspot.com.au/2013/04/bluetooth-programming-with-python-3.html > I need to use python 3.x, as this is eventually going to run within > Blender. It's going to run within blender so I can use blender to visualize > and solve inverse-kinematic stuff for a robot arm. > > Regards > Jak_o_Shadows > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Jul 16 04:00:14 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 16 Jul 2013 18:00:14 +1000 Subject: Bluetooth Sockets In-Reply-To: References: Message-ID: On Tue, Jul 16, 2013 at 3:52 PM, Simfake Fake wrote: > Just bumping this, but has anybody have any personal experience with > bluetooth in python 3? Perhaps my issue is that the windows version doesn't > include it? I haven't worked with Bluetooth in Python, but my reading of the socket module docs is that AF_BLUETOOTH doesn't imply strong support anyway. I'd recommend looking for a dedicated bluetooth module; there seem to be some around. ChrisA From christian at python.org Tue Jul 16 06:08:59 2013 From: christian at python.org (Christian Heimes) Date: Tue, 16 Jul 2013 12:08:59 +0200 Subject: Bluetooth Sockets In-Reply-To: References: Message-ID: Am 13.07.2013 10:53, schrieb Simfake Fake: > Hi. I'm trying to connect to a bluetooth serial adaptor using python > 3.x. However, in python 3.3.2 win x32, I get "AttributeError: module has > no attribute AF_..." when trying to use socket.AF_BLUETOOTH, despite the > docs http://docs.python.org/3.3/library/socket.html . The code I have is > very similar to that > found http://kevindoran1.blogspot.com.au/2013/04/bluetooth-programming-with-python-3.html > I need to use python 3.x, as this is eventually going to run within > Blender.. It's going to run within blender so I can use blender to > visualize and solve inverse-kinematic stuff for a robot arm. AF_BLUETOOTH is only available on Linux and some BSD variants. Windows has different bluetooth stacks. From fronagzen at gmail.com Sat Jul 13 07:07:21 2013 From: fronagzen at gmail.com (fronagzen at gmail.com) Date: Sat, 13 Jul 2013 04:07:21 -0700 (PDT) Subject: Ideal way to separate GUI and logic? Message-ID: Well, I'm a newcome to Python, but I'm developing a program with a GUI in tkinter, and I'm wondering what is the best, 'most pythonic' way of doing this? I could, obviously, write a monolithic block of code. I can define the logic and the GUI as two separate classes and then call from those classes within a single script. Those are probably less than ideal. But how then do I separate out the logic and the GUI? Because the GUI needs to have commands defined for button presses and so forth, and those have to come from the logic section. Do I write the logic in a separate file and then import those to the GUI, and then run it there? Wouldn't it be better to have one file for the GUI, one for the logic, and then a third one that imports from both and then executes the app? How do I do that? (Examples would be nice, if possible.) From steve+comp.lang.python at pearwood.info Sat Jul 13 07:43:25 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 13 Jul 2013 11:43:25 GMT Subject: Ideal way to separate GUI and logic? References: Message-ID: <51e13d5d$0$9505$c3e8da3$5496439d@news.astraweb.com> On Sat, 13 Jul 2013 04:07:21 -0700, fronagzen wrote: > Well, I'm a newcome to Python, but I'm developing a program with a GUI > in tkinter, and I'm wondering what is the best, 'most pythonic' way of > doing this? > > I could, obviously, write a monolithic block of code. > > I can define the logic and the GUI as two separate classes and then call > from those classes within a single script. > > Those are probably less than ideal. > > But how then do I separate out the logic and the GUI? Because the GUI > needs to have commands defined for button presses and so forth, and > those have to come from the logic section. Do I write the logic in a > separate file and then import those to the GUI, and then run it there? > Wouldn't it be better to have one file for the GUI, one for the logic, > and then a third one that imports from both and then executes the app? > How do I do that? (Examples would be nice, if possible.) The important thing is that the application logic is separate from the user interface. It doesn't matter whether than means "two functions in the same module", or "two modules". Whether you decide to split your code into two modules depends on how complex your code is. For example, if you are writing a "Hello World" application, you would be nuts to split this into two files. === cut === def greet(name): return "Hello, %s!" % name def do_greet(name=None): if name is None: name = raw_input("What is your name? ").strip() print greet(name) do_greet() === cut === Only you know how big and complex your application is, so only you know whether or not you should separate the GUI interface from the internal logic. But, my guess is that for anything non-trivial, you probably should have one main module handling the UI, and a second (and possibly more) modules handling the implementation, plus at least one other module for testing. -- Steven From r.koebler at yahoo.de Sat Jul 13 09:56:20 2013 From: r.koebler at yahoo.de (Roland Koebler) Date: Sat, 13 Jul 2013 15:56:20 +0200 Subject: Ideal way to separate GUI and logic? In-Reply-To: References: Message-ID: <20130713135620.GA18186@localhost> Hi, > But how then do I separate out the logic and the GUI? I usually write a library (C library, Python module, ...) which contains the logic. Then, I write a GUI (in a separate file), which imports and uses the library. If I need another UI (e.g. GUI with an other toolkit, or a text-based or HTML5-based interface), I simply write another UI (in a separate file), and import+use the library again. That's the cleanest way to separate user-interface and logic in my opinion. (But keep in mind that it's not always obvious which parts belong to the library and which belong to the GUI, and you sometimes have to carefully think about it.) Oh, and yes, you can do nice things then, e.g. remote-GUIs by transparently tunneling all calls from the GUI to the library through RPC over a network (like I have done with a GTK+-GUI for Raspberry Pi; the GUI runs on the PC, uses JSON-RPC over TCP-sockets and calls functions on the RPi). regards, Roland From davecook at nowhere.net Sat Jul 13 13:40:49 2013 From: davecook at nowhere.net (Dave Cook) Date: 13 Jul 2013 17:40:49 GMT Subject: Ideal way to separate GUI and logic? References: Message-ID: <51e19121$0$1844$c3e8da3$f017e9df@news.astraweb.com> On 2013-07-13, fronagzen at gmail.com wrote: > I'm wondering what is the best, 'most pythonic' way I recommend PyPubsub: http://pubsub.sourceforge.net/ Dave Cook From wayne at waynewerner.com Sat Jul 13 15:03:24 2013 From: wayne at waynewerner.com (Wayne Werner) Date: Sat, 13 Jul 2013 14:03:24 -0500 (CDT) Subject: Ideal way to separate GUI and logic? In-Reply-To: References: Message-ID: On Sat, 13 Jul 2013, fronagzen at gmail.com wrote: > Well, I'm a newcome to Python, but I'm developing a program with a GUI in tkinter, and I'm wondering what is the best, 'most pythonic' way of doing this? > > I could, obviously, write a monolithic block of code. True, you could, but don't do that. You should investigate strategies like model view presenter, and test or behavior driven development. My recommendation echos the other advice you've been given. Basically you think of your application in terms of two problems or domains. One is the what that you want no do. What information do you need? The other problem is how do you get that information from the user and retun to them information than they need? Regardless of which side you have driving, UI or Presenter, having a design such as this allows for much cleaner code. HTH -W From fronagzen at gmail.com Sun Jul 14 20:24:40 2013 From: fronagzen at gmail.com (fronagzen at gmail.com) Date: Sun, 14 Jul 2013 17:24:40 -0700 (PDT) Subject: Ideal way to separate GUI and logic? In-Reply-To: References: Message-ID: <9a364049-f80a-49e0-89be-e8a2c645a1dd@googlegroups.com> Thanks for all the responses! So as a general idea, I should at the very least separate the GUI from the program logic by defining the logic as a function, correct? And the next level of separation is to define the logic as a class in one or more separate files, and then import it to the file with the GUI, correct? My next question is, to what degree should I 'slice' my logic into functions? How small or how large should one function be, as a rule of thumb? From fronagzen at gmail.com Sun Jul 14 20:25:32 2013 From: fronagzen at gmail.com (fronagzen at gmail.com) Date: Sun, 14 Jul 2013 17:25:32 -0700 (PDT) Subject: Ideal way to separate GUI and logic? In-Reply-To: References: Message-ID: Thanks for all the responses! So as a general idea, I should at the very least separate the GUI from the program logic by defining the logic as a function, correct? And the next level of separation is to define the logic as a class in one or more separate files, and then import it to the file with the GUI, correct? My next question is, to what degree should I 'slice' my logic into functions? How small or how large should one function be, as a rule of thumb? From joel.goldstick at gmail.com Sun Jul 14 21:02:00 2013 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Sun, 14 Jul 2013 21:02:00 -0400 Subject: Ideal way to separate GUI and logic? In-Reply-To: References: Message-ID: On Sun, Jul 14, 2013 at 8:25 PM, wrote: > Thanks for all the responses! > > So as a general idea, I should at the very least separate the GUI from the > program logic by defining the logic as a function, correct? And the next > level of separation is to define the logic as a class in one or more > separate files, and then import it to the file with the GUI, correct? > > My next question is, to what degree should I 'slice' my logic into > functions? How small or how large should one function be, as a rule of > thumb? > -- > http://mail.python.org/mailman/listinfo/python-list > Others may differ. I think you should just write the code. In actually doing that you will learn the pitfalls of how you have divided up your logic. Writing code isn't all theory. It takes practice, and since the days of The Mythical Man-Month, it has been well understood that you always end up throwing away the first system anyway. It has to be built to truly understand what you think you want to create, but in the learning, you realize that its easier and better to start more or less from scratch rather than try to fix the first concept. -- Joel Goldstick http://joelgoldstick.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy at panix.com Sun Jul 14 21:30:52 2013 From: roy at panix.com (Roy Smith) Date: Sun, 14 Jul 2013 21:30:52 -0400 Subject: Ideal way to separate GUI and logic? References: Message-ID: In article , Joel Goldstick wrote: > Writing code isn't all theory. It takes practice, and since the days > of The Mythical Man-Month, it has been well understood that you > always end up throwing away the first system anyway. If I may paraphrase Brooks, "Plan to throw the first one away, because it's going to suck. Then, the next one you write to replace it will also suck because it's going to suffer from Second System Effect" :-) BTW, anybody who enjoyed The Mythical Man-Month should also read Ed Yourdon's Death March. From steve at pearwood.info Sun Jul 14 21:44:19 2013 From: steve at pearwood.info (Steven D'Aprano) Date: 15 Jul 2013 01:44:19 GMT Subject: Ideal way to separate GUI and logic? References: Message-ID: <51e353f3$0$29866$c3e8da3$5496439d@news.astraweb.com> On Sun, 14 Jul 2013 17:25:32 -0700, fronagzen wrote: > My next question is, to what degree should I 'slice' my logic into > functions? How small or how large should one function be, as a rule of > thumb? I aim to keep my functions preferably below a dozen lines (excluding the doc string), and definitely below a page. But more important than size is functionality. Every function should do *one thing*. If that thing can be divided into two or more "sub-things" then they should be factored out into separate functions, which I then call. Possibly private, internal only functions. -- Steven From asimjalis at gmail.com Mon Jul 15 13:06:30 2013 From: asimjalis at gmail.com (asimjalis at gmail.com) Date: Mon, 15 Jul 2013 10:06:30 -0700 (PDT) Subject: Ideal way to separate GUI and logic? In-Reply-To: References: Message-ID: <22e50125-633a-4bae-a317-a83a748be225@googlegroups.com> fron... at gmail.com wrote: > So as a general idea, I should at the very least separate the GUI from the program logic by defining the logic as a function, correct? And the next level of separation is to define the logic as a class in one or more separate files, and then import it to the file with the GUI, correct? > > My next question is, to what degree should I 'slice' my logic into functions? How small or how large should one function be, as a rule of thumb? The way I do this is to write unit tests against the class and the functions (take a look at the unittest module). The functions methods (take a look at the unittest module). Each function should contain the smallest bit of testable logic. Another way to think about this is that each function should contain the smallest piece of logic that you can describe as one action. - Asim Jalis From fronagzen at gmail.com Mon Jul 15 20:25:57 2013 From: fronagzen at gmail.com (fronagzen at gmail.com) Date: Mon, 15 Jul 2013 17:25:57 -0700 (PDT) Subject: Ideal way to separate GUI and logic? In-Reply-To: <22e50125-633a-4bae-a317-a83a748be225@googlegroups.com> References: <22e50125-633a-4bae-a317-a83a748be225@googlegroups.com> Message-ID: <8152b57e-5ff3-4024-9de1-d0a20b5c14b6@googlegroups.com> On Tuesday, July 16, 2013 1:06:30 AM UTC+8, asim... at gmail.com wrote: > fron... at gmail.com wrote: > > > So as a general idea, I should at the very least separate the GUI from the program logic by defining the logic as a function, correct? And the next level of separation is to define the logic as a class in one or more separate files, and then import it to the file with the GUI, correct? > > > > > > My next question is, to what degree should I 'slice' my logic into functions? How small or how large should one function be, as a rule of thumb? > > > > The way I do this is to write unit tests against the class and the functions (take a look at the unittest module). The functions methods (take a look at the unittest module). Each function should contain the smallest bit of testable logic. > > > > Another way to think about this is that each function should contain the smallest piece of logic that you can describe as one action. > > > > - > > Asim Jalis Again, thanks for all the responses. I'm curious, though, what exactly is the rationale for making functions so small? (I've heard that the function calling of Python has relatively high overhead?) From o at owenmarshall.invalid Mon Jul 15 20:42:34 2013 From: o at owenmarshall.invalid (Owen Marshall) Date: 16 Jul 2013 00:42:34 GMT Subject: Ideal way to separate GUI and logic? References: <22e50125-633a-4bae-a317-a83a748be225@googlegroups.com> <8152b57e-5ff3-4024-9de1-d0a20b5c14b6@googlegroups.com> Message-ID: On 2013-07-16, fronagzen at gmail.com wrote: > On Tuesday, July 16, 2013 1:06:30 AM UTC+8, asim... at gmail.com wrote: >> fron... at gmail.com wrote: >> >> > So as a general idea, I should at the very least separate the GUI >> > from the program logic by defining the logic as a function, >> > correct? And the next level of separation is to define the logic as >> > a class in one or more separate files, and then import it to the >> > file with the GUI, correct? >> >> > >> >> > My next question is, to what degree should I 'slice' my logic into >> > functions? How small or how large should one function be, as a rule >> > of thumb? >> >> >> >> The way I do this is to write unit tests against the class and the >> functions (take a look at the unittest module). The functions methods >> (take a look at the unittest module). Each function should contain >> the smallest bit of testable logic. >> >> >> >> Another way to think about this is that each function should contain >> the smallest piece of logic that you can describe as one action. >> >> >> >> - >> >> Asim Jalis > > Again, thanks for all the responses. I'm curious, though, what exactly > is the rationale for making functions so small? (I've heard that the > function calling of Python has relatively high overhead?) Small functions are _always_ encouraged for every language. This is best practice for everything, not just Python. Has nothing to do with overhead. -- -owen From asimjalis at gmail.com Mon Jul 15 21:02:03 2013 From: asimjalis at gmail.com (Asim Jalis) Date: Mon, 15 Jul 2013 18:02:03 -0700 Subject: Ideal way to separate GUI and logic? In-Reply-To: <8152b57e-5ff3-4024-9de1-d0a20b5c14b6@googlegroups.com> References: <22e50125-633a-4bae-a317-a83a748be225@googlegroups.com> <8152b57e-5ff3-4024-9de1-d0a20b5c14b6@googlegroups.com> Message-ID: On Mon, Jul 15, 2013 at 5:25 PM, wrote: > Again, thanks for all the responses. I'm curious, though, what exactly is > the rationale for making functions so small? (I've heard that the function > calling of Python has relatively high overhead?) > There is a small overhead, but it makes the code easier to read and understand. You can look at the function name and get and idea of _what_ the function is doing instead of having to figure out _how_ it is doing it. Regarding optimization, after you have written your application if you see performance issues you can surgically optimize the spots that have the issues and leave most of the code untouched. To quote Don Knuth, "premature optimization is the root of all evil". Also the article at http://en.wikipedia.org/wiki/Program_optimization makes some good points. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Jul 15 23:18:56 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 16 Jul 2013 13:18:56 +1000 Subject: Ideal way to separate GUI and logic? In-Reply-To: <8152b57e-5ff3-4024-9de1-d0a20b5c14b6@googlegroups.com> References: <22e50125-633a-4bae-a317-a83a748be225@googlegroups.com> <8152b57e-5ff3-4024-9de1-d0a20b5c14b6@googlegroups.com> Message-ID: On Tue, Jul 16, 2013 at 10:25 AM, wrote: > Again, thanks for all the responses. I'm curious, though, what exactly is the rationale for making functions so small? (I've heard that the function calling of Python has relatively high overhead?) A function should be as long as it needs to be - neither longer nor shorter. If you can name a function appropriately, and it's doing exactly what its name suggests, it's the right length. This generally produces short functions rather than long ones, but if the right length for a function exceeds some arbitrary limit, let the function be longer. For instance, I have a single function that's a bit over a page in length, because it's one big switch block (this isn't in Python, obviously), doing one thing fairly cleanly. Larger than that would have to be called code smell, but there's certainly nothing wrong with having the odd function here or there that's over Steven's dozen-line estimate. There'll also be plenty of really short functions - even one-liners. The largest single function in any of my code, I think, is a gigantic double-nested switch block in PHP .In any decent language, that would be divided up not just into functions but into files, but PHP has some stupid restrictions on its include directive that make that impractical. So syntactically it's one massive function, but logically it's about seven or eight separate sub-blocks, and the code is laid out in those blocks. It's just that the technical nesting level never actually hits zero in between them :) Have to confess, though, I've had some fairly large functions in C++ (not as big as the aforementioned, but still fairly large - what's the next one down from gigantic, megantic? [1] would suggest so), which in some cases could be split if I felt like putting in the time to do it. ChrisA [1] http://gatherer.wizards.com/Pages/Card/Details.aspx?multiverseid=370794 From bv4bv4bv4 at gmail.com Sat Jul 13 16:45:29 2013 From: bv4bv4bv4 at gmail.com (bv4bv4bv4 at gmail.com) Date: Sat, 13 Jul 2013 13:45:29 -0700 (PDT) Subject: Is Jesus God or sent by God? Message-ID: <0647cf2e-d53a-49ad-8c34-985d2f6d5675@googlegroups.com> Is Jesus God or sent by God? The first of a two-part article discussing the true role of Jesus. Part 1: Discusses whether Jesus called himself God, Jesus referred to as Lord and the nature of Jesus. Jesus is a figure who is loved and revered by billions of people the world over. Yet there is so much confusion surrounding the status of this colossal personality. Muslims and Christians both hold Jesus in high regard but view him in very different ways. The questions raised in this article aim to get to the heart of the issues surrounding Jesus: Is Jesus God? Or was he sent by God? Who was the real historical Jesus? Some ambiguous verses of the Bible can be applied erroneously to show that Jesus is in some way divine. But if we look at the clear, direct verses of the Bible, we see again and again that Jesus is being referred to as an extraordinary human being and nothing more. What emerges, when we consider the historical and logical facts about Jesus? life, is conclusive proof not only that Jesus cannot be God, but that he never claimed to be either. What follows are five lines of reasoning which clarify this subject for us through the Bible itself and thereby allow us to discover the real Jesus. 1. Jesus Never Calls Himself God The Bible (in spite of being changed and adulterated over time) contains many verses in which Jesus speaks of God as a separate person to himself. Here are just a few of them: When a man addressed Jesus as ?Good Teacher?, he replied ?Why do you call me good? No one is good except the one God.?? [Mark 10:18] In another instance he says: ?I can?t do anything by myself. Whatever I hear, I judge, and my judgment is just. I don?t seek my own will but the will of the one who sent me.? [John 5:30] Jesus speaks of God as a separate being to himself: I?m going up to my Father and your Father, to my God and your God. [John 20:17] In this verse he affirms that he was sent by God: This is eternal life: to know you, the only true God, and Jesus Christ whom you sent. [John 17:3] If Jesus was God he would have told people to worship him, but he did the opposite and disapproved anyone worshipping him: And in vain they worship Me [Matthew 15:9] If Jesus claimed to be God than there should be hundreds of verses in the Bible which would have mentioned it. But there is not a single verse in the entire Bible in which Jesus says I am God, worship me. 2. Jesus as Son and Lord? Jesus is sometimes referred to as ?Lord? in the Bible and at other times as ?Son of God?. God is called the ?Father?, so putting these names together it could be claimed that Jesus is the son of God. But if we look at each of these titles in context we will find that they are symbolic and not to be taken literally. ?Son of God? is a term used in ancient Hebrew for a righteous person. God calls Israel his ?son?: This is what the LORD says: Israel is my oldest son.[Exodus 4:22]. Also, David is called the ?Son of God?: The LORD has said to Me, ?You are My Son, Today I have begotten You.? [Psalm 2:7]. In fact anyone who is righteous is referred to as God?s ?son?: All who are led by God?s Spirit are God?s sons and daughters. [Romans 8:14]. In the same way, when the word ?Father? is used to refer to God it shouldn?t be taken literally. Instead it?s a way of saying God is the creator, sustainer, cherisher etc. There are many verses for us to understand this symbolic meaning of the word ?Father?, for example: one God and Father of all. [Ephesians 4:6]. Jesus is sometimes called ?Lord? by the disciples. ?Lord? is a term used for God and also for people who are held in high esteem. There are many examples of the word ?Lord? being used for people in the Bible: So they (Joseph?s brothers) went up to Joseph?s steward and spoke to him at the entrance to the house. ?We beg your pardon, our lord,? they said. [Genesis 43:19-20]. Also, in other parts of the Bible, Jesus is even called a ?servant? of God by the disciples: the God of our fathers, has glorified his servant Jesus. [Acts 3:13]. This clearly shows that when ?Lord? is used to refer to Jesus, it is a title of respect not of divinity. 3. The Nature of Jesus The nature of Jesus was totally different to that of God. There are many parts of the Bible that highlight this difference in nature: God is All-Knowing but Jesus by his own admission was not All-Knowing. This can be seen in the following passage when Jesus says ?But nobody knows when that day or hour will come, not the heavenly angels and not the Son. Only the Father knows.? [Matthew 24:36] God is independent and he doesn?t need sleep, food or water. Jesus however ate, drank, slept and depended on God: As the living Father sent me, and I live because of the Father. [John 6:57]. Another sign of Jesus? dependence on God is that he prayed to God: Going a little farther, he (Jesus) fell with his face to the ground and prayed [Matthew 26:39]. This shows that Jesus himself sought help from God. God, being the one who answers prayers does not need to pray to anyone. Also, Jesus said: I am going to the Father, because the Father is greater than me. [John 14:28]. The Bible is clear that God is unseen and is not a man: for no one may see me and live. [Exodus 33:20], God is not a man [Numbers 23:19]. Jesus on the other hand was a man who was seen by thousands of people, so he could not be God. Furthermore, the Bible makes it clear that God is too great to be inside his creation: But how could God possibly live on earth with people? If heaven, even the highest heaven, can?t contain you [2 Chronicles 6:18]. According to this verse Jesus cannot be God living on the earth. Also the Bible calls Jesus a Prophet [Matthew 21:10-11], so how could Jesus be God and be God?s Prophet at the same time? That wouldn?t make sense. Additionally the Bible informs us that God does not change: I the Lord do not change. [Malachi 3:6:]. Jesus however went through many changes in his life such as age, height, weight etc. These are just some of the proofs within the Bible, which make it clear that the nature of Jesus and God is completely different. Some people may claim that Jesus had both a human and a divine nature. This is a claim that Jesus never made, and is in clear contradiction to the Bible which maintains that God has one nature. 4. The Message of Jesus The Prophets of the Old Testament such as Abraham, Noah and Jonah never preached that God is part of a Trinity, and did not believe in Jesus as their saviour. Their message was simple: there is one God and He alone deserves your worship. It doesn?t make sense that God sent Prophets for thousands of years with the same essential message, and then all of a sudden he says he is in a Trinity and that you must believe in Jesus to be saved. The truth is that Jesus preached the same message that the Prophets in the Old Testament preached. There is a passage in the Bible which really emphasizes his core message. A man came to Jesus and asked ?Which is the first commandment of all??Jesus answered, ?The first of all the commandments is Hear, O Israel, the Lord our God, the Lord is one.??[Mark 12:28-29]. So the greatest commandment, the most important belief according to Jesus is that God is one. If Jesus was God he would have said ?I am God, worship me?, but he didn?t. He merely repeated a verse from the Old Testament confirming that God is One. Some people claim that Jesus came to die for the sins of the world. But consider the following statement of Jesus: This is eternal life: to know you, the only true God, and Jesus Christ whom you sent. I have glorified you on earth by finishing the work you gave me to do.[John 17:3-4]. Jesus said this before he was caught and taken to be crucified. It is clear from this verse that Jesus did not come to die for the sins of the world, as he finished the work God gave him before he was taken to be crucified. Also Jesus said ?salvation is of the Jews? [John 4:22]. So according to this we don?t need to believe in the Trinity or that Jesus died for our sins to attain salvation since the Jews don?t have these beliefs. 5. The Early Christians Historically there were many sects in early Christianity who had a range of beliefs regarding Jesus[1]. Some believed Jesus was God, others believed Jesus was not God but partly divine, and yet others believed he was a human being and nothing more. Trinitarian Christianity which is the belief that God, Jesus and the Holy Spirit are one in three persons became the dominant sect of Christianity, once it was formalized as the state religion of the Roman Empire in the 4th Century. Christians who denied Jesus being God were persecuted by the Roman Authorities[2]. From this point onwards the Trinitarian belief became widespread amongst Christians. There were various movements in early Christianity which denied the Trinity, among the more well known of them is Adoptionism and Arianism. Dr Jerald Dirks who is an expert on early Christianity had this to say on the subject: Early Christianity was quite conflicted about the issue of the nature of Jesus. The various Adoptionist positions within early Christianity were numerous and at times dominate. One can even speculate that Arian and Nestorian Christianity might well be an extremely sizable source within Christianity today, if it were not for the fact that these two branches of Christianity, which were located primarily in the middle east and in North Africa were so similar to the Islamic teaching regarding the nature of Jesus that they quite naturally were absorbed into Islam at the beginning of the seventh century.?[3] Since there were so many sects in early Christianity, each with different beliefs about Jesus and with their own versions of the Bible, which one can we say was following the true teachings of Jesus? It doesn?t make sense that God sends countless Prophets like Noah, Abraham and Moses to tell people to believe in one God, and then suddenly sends a radically different message of the Trinity which contradicts his previous Prophets teachings. It is clear that the sect of Christianity who believed Jesus to be a human Prophet and nothing more, were following the true teachings of Jesus. This is because their concept of God is the same as that which was taught by the Prophets in the Old Testament. Jesus in Islam The Islamic belief about Jesus demystifies for us who the real Jesus was. Jesus in Islam was an extraordinary individual, chosen by God as a Prophet and sent to the Jewish people. He never preached that he himself was God or the actual son of God. He was miraculously born without a father, and he performed many amazing miracles such as healing the blind and the lepers and raising the dead ? all by God?s permission. Muslims believe that Jesus will return before the day of Judgement to bring justice and peace to the world. This Islamic belief about Jesus is similar to the belief of some of the early Christians. In the Quran, God addresses the Christians about Jesus in the following way: O People of the Book, do not commit excesses in your religion, and do not say anything about God except the truth: the Messiah, Jesus, son of Mary, was nothing more than a messenger of God, His word, directed to Mary and a spirit from Him. So believe in God and His Messengers and do not speak of a ?Trinity?? stop [this], that is better for you? God is only one God, He is far above having a son, everything in the heavens and earth belongs to Him and He is the best one to trust. [4:171] Islam is not just another religion. It is the same message preached by Moses, Jesus and Abraham. Islam literally means ?submission to God? and it teaches us to have a direct relationship with God. It reminds us that since God created us, no one should be worshipped except God alone. It also teaches that God is nothing like a human being or like anything that we can imagine. The concept of God is summarized in the Quran as: ?Say, He is God, the One. God, the Absolute. He does not give birth, nor was He born, and there is nothing like Him.? (Quran 112:1-4)[4] Becoming a Muslim is not turning your back to Jesus. Rather it?s going back to the original teachings of Jesus and obeying him. Footnotes: [1] John Evans, History of All Christian Sects and Denominations, ISBN: 0559228791 [2] C.N. Kolitsas, The Life and Times of Constantine the Great, ISBN: 1419660411 [3] Excerpt from ?Islamic Trajectories in Early Christianity? by Dr Jerald Dirks [4] God is not male or female, the word ?Him? when used for God does not refer to gender. http://www.islamreligion.com/articles/5232/ http://www.islamreligion.com/articles/5231/ thank you From leegold at operamail.com Sat Jul 13 16:49:46 2013 From: leegold at operamail.com (goldtech) Date: Sat, 13 Jul 2013 13:49:46 -0700 (PDT) Subject: How to read Mozrepl Javascript result into a Python Variable? Message-ID: Hi, With Mozrepl addon in Firefox and Python I do: >>> import telnetlib >>> tn = telnetlib.Telnet(r'127.0.0.1', 4242, 5) >>> tn.read_eager() '\nWelcome to MozRepl.\n\n - If you get stuck at the "' >>> tn.read_until("repl> ") ...snip... >>> tn.write(r'alert(window.content.location.href)'+"\n") and I get an alert box with the URL of the active tab. But how do I read that URL into a python variable? Something like tn.write(r';var zz = window.content.location.href'+ "\n") but that doesn't get it into python. I would be grateful for help. Thanks. From rosuav at gmail.com Sat Jul 13 19:26:13 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 14 Jul 2013 09:26:13 +1000 Subject: How to read Mozrepl Javascript result into a Python Variable? In-Reply-To: References: Message-ID: On Sun, Jul 14, 2013 at 6:49 AM, goldtech wrote: > Hi, > > With Mozrepl addon in Firefox and Python I do: > >>>> import telnetlib >>>> tn = telnetlib.Telnet(r'127.0.0.1', 4242, 5) >>>> tn.read_eager() > '\nWelcome to MozRepl.\n\n - If you get stuck at the "' >>>> tn.read_until("repl> ") > ...snip... >>>> tn.write(r'alert(window.content.location.href)'+"\n") > > and I get an alert box with the URL of the active tab. > > But how do I read that URL into a python variable? Something like tn.write(r';var zz = window.content.location.href'+ "\n") but that doesn't get it into python. > > I would be grateful for help. Thanks. This seems to be what you're after: https://github.com/bard/mozrepl/wiki/Tutorial By my reading of that, this should do it: tn.write('window.content.location.href\n') It seems to function somewhat like Python's own interactive mode - a bare expression gets sent to you. Very convenient. ChrisA From vek.m1234 at gmail.com Sat Jul 13 23:09:31 2013 From: vek.m1234 at gmail.com (vek.m1234 at gmail.com) Date: Sat, 13 Jul 2013 20:09:31 -0700 (PDT) Subject: Beazley 4E P.E.R, Page29: Unicode Message-ID: <51cbaddd-c29d-48a3-97ab-3beb1d944f1a@googlegroups.com> http://stackoverflow.com/questions/17632246/beazley-4e-p-e-r-page29-unicode "directly writing a raw UTF-8 encoded string such as 'Jalape\xc3\xb1o' simply produces a nine-character string U+004A, U+0061, U+006C, U+0061, U+0070, U+0065, U+00C3, U+00B1, U+006F, which is probably not what you intended.This is because in UTF-8, the multi- byte sequence \xc3\xb1 is supposed to represent the single character U+00F1, not the two characters U+00C3 and U+00B1." My original question was: Shouldn't this be 8 characters - not 9? He says: \xc3\xb1 is supposed to represent the single character. However after some interaction with fellow Pythonistas i'm even more confused. With reference to the above para: 1. What does he mean by "writing a raw UTF-8 encoded string"?? In Python2, once can do 'Jalape funny-n o'. This is a 'bytes' string where each glyph is 1 byte long when stored internally so each glyph is associated with an integer as per charset ASCII or Latin-1. If these charsets have a funny-n glyph then yay! else nay! There is no UTF-8 here!! or UTF-16!! These are plain bytes (8 bits). Unicode is a really big mapping table between glyphs and integers and are denoted as Uxxxx or Uxxxx-xxxx. UTF-8 UTF-16 are encodings to store those big integers in an efficient manner. So when DB says "writing a raw UTF-8 encoded string" - well the only way to do this is to use Python3 where the default string literals are stored in Unicode which then will use a UTF-8 UTF-16 internally to store the bytes in their respective structures; or, one could use u'Jalape' which is unicode in both languages (note the leading 'u'). 2. So assuming this is Python 3: 'Jalape \xYY \xZZ o' (spaces for readability) what DB is saying is that, the stupid-user would expect Jalapeno with a squiggly-n but instead he gets is: Jalape funny1 funny2 o (spaces for readability) -9 glyphs or 9 Unicode-points or 9-UTF8 characters. Correct? 3. Which leaves me wondering what he means by: "This is because in UTF-8, the multi- byte sequence \xc3\xb1 is supposed to represent the single character U+00F1, not the two characters U+00C3 and U+00B1" Could someone take the time to read carefully and clarify what DB is saying?? From tjreedy at udel.edu Sun Jul 14 03:08:22 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 14 Jul 2013 03:08:22 -0400 Subject: Beazley 4E P.E.R, Page29: Unicode In-Reply-To: <51cbaddd-c29d-48a3-97ab-3beb1d944f1a@googlegroups.com> References: <51cbaddd-c29d-48a3-97ab-3beb1d944f1a@googlegroups.com> Message-ID: On 7/13/2013 11:09 PM, vek.m1234 at gmail.com wrote: > http://stackoverflow.com/questions/17632246/beazley-4e-p-e-r-page29-unicode Is this David Beazley? (You referred to 'DB' later.) > "directly writing a raw UTF-8 encoded string such as > 'Jalape\xc3\xb1o' simply produces a nine-character string U+004A, > U+0061, U+006C, U+0061, U+0070, U+0065, U+00C3, U+00B1, U+006F, which > is probably not what you intended.This is because in UTF-8, the > multi- byte sequence \xc3\xb1 is supposed to represent the single > character U+00F1, not the two characters U+00C3 and U+00B1." > > My original question was: Shouldn't this be 8 characters - not 9? He > says: \xc3\xb1 is supposed to represent the single character. However > after some interaction with fellow Pythonistas i'm even more > confused. > > With reference to the above para: 1. What does he mean by "writing a > raw UTF-8 encoded string"?? As much respect as I have for DB, I think this is an impossible to parse confused statement, fueled by the Python 2 confusion between characters and bytes. I suggest forgetting it and the discussion that followed. Bytes as bytes can carry any digital information, just as modulated sine waves can carry any analog information. In both cases, one can regard them as either purely what they are or as encoding information in some other form. -- Terry Jan Reedy From joshua at landau.ws Sun Jul 14 03:13:59 2013 From: joshua at landau.ws (Joshua Landau) Date: Sun, 14 Jul 2013 08:13:59 +0100 Subject: Beazley 4E P.E.R, Page29: Unicode In-Reply-To: <51cbaddd-c29d-48a3-97ab-3beb1d944f1a@googlegroups.com> References: <51cbaddd-c29d-48a3-97ab-3beb1d944f1a@googlegroups.com> Message-ID: On 14 July 2013 04:09, wrote: > http://stackoverflow.com/questions/17632246/beazley-4e-p-e-r-page29-unicode > > "directly writing a raw UTF-8 encoded string such as 'Jalape\xc3\xb1o' simply produces a nine-character string U+004A, U+0061, U+006C, U+0061, U+0070, U+0065, U+00C3, U+00B1, U+006F, which is probably not what you intended.This is because in UTF-8, the multi- byte sequence \xc3\xb1 is supposed to represent the single character U+00F1, not the two characters U+00C3 and U+00B1." Correct. > My original question was: Shouldn't this be 8 characters - not 9? No, Python tends to be right on these things. > He says: \xc3\xb1 is supposed to represent the single character. However after some interaction with fellow Pythonistas i'm even more confused. You would be, given the way he said it. > With reference to the above para: > 1. What does he mean by "writing a raw UTF-8 encoded string"?? Well, that doesn't really mean much with no context like he gave it. > In Python2, once can do 'Jalape funny-n o'. This is a 'bytes' string where each glyph is 1 byte long when stored internally so each glyph is associated with an integer as per charset ASCII or Latin-1. If these charsets have a funny-n glyph then yay! else nay! There is no UTF-8 here!! or UTF-16!! These are plain bytes (8 bits). > > Unicode is a really big mapping table between glyphs and integers and are denoted as Uxxxx or Uxxxx-xxxx. *Waits for our resident unicode experts to explain why you're actually wrong* > UTF-8 UTF-16 are encodings to store those big integers in an efficient manner. So when DB says "writing a raw UTF-8 encoded string" - well the only way to do this is to use Python3 where the default string literals are stored in Unicode which then will use a UTF-8 UTF-16 internally to store the bytes in their respective structures; or, one could use u'Jalape' which is unicode in both languages (note the leading 'u'). Correct. > 2. So assuming this is Python 3: 'Jalape \xYY \xZZ o' (spaces for readability) what DB is saying is that, the stupid-user would expect Jalapeno with a squiggly-n but instead he gets is: Jalape funny1 funny2 o (spaces for readability) -9 glyphs or 9 Unicode-points or 9-UTF8 characters. Correct? I think so. > 3. Which leaves me wondering what he means by: > "This is because in UTF-8, the multi- byte sequence \xc3\xb1 is supposed to represent the single character U+00F1, not the two characters U+00C3 and U+00B1" He's mixed some things up, AFAICT. > Could someone take the time to read carefully and clarify what DB is saying?? Here's a simple explanation: you're both wrong (or you're both *almost* right): As of Python 3: >>> "\xc3\xb1" '??' >>> b"\xc3\xb1".decode() '?' "WHAT?!" you scream, "THAT'S WRONG!" But it's not. Let me explain. Python 3's strings want you to give each character separately (*winces in case I'm wrong*). Python is interpreting the "\xc3" as "\N{LATIN CAPITAL LETTER A WITH TILDE}" and "\xb1" as "\N{PLUS-MINUS SIGN}"?. This means that Python is given *two* characters. Python is basically doing this: number = int("c3", 16) # Convert from base16 chr(number) # Turn to the character from the Unicode mapping When you give Python *raw bytes*, you are saying that this is what the string looks like *when encoded* -- you are not giving Python Unicode, but *encoded Unicode*. This means that when you decode it (.decode()) it is free to convert multibyte sections to their relevant characters. To see how an *encoded string* is not the same as the string itself, see: >>> "Jalepe?o".encode("ASCII", errors="xmlcharrefreplace") b'Jalepeño' Those *represent* the same thing, but the first (according to Python) *is* the thing, the second needs to be *decoded*. Now, bringing this back to the original: >>> "\xc3\xb1".encode() b'\xc3\x83\xc2\xb1' You can see that the *encoded* bytes represent the *two* characters; the string you see above is *not the encoded one*. The encoding is *internal to Python*. I hope that helps; good luck. ? Note that I find the "\N{...}" form much easier to read, and recommend it. From vek.m1234 at gmail.com Sun Jul 14 04:10:57 2013 From: vek.m1234 at gmail.com (vek.m1234 at gmail.com) Date: Sun, 14 Jul 2013 01:10:57 -0700 (PDT) Subject: Beazley 4E P.E.R, Page29: Unicode In-Reply-To: References: <51cbaddd-c29d-48a3-97ab-3beb1d944f1a@googlegroups.com> Message-ID: <2bd60e4b-df82-4dc1-9db4-4f4d7722b9fa@googlegroups.com> thank you (both of you) I follow now :) From steve+comp.lang.python at pearwood.info Sun Jul 14 04:18:15 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 14 Jul 2013 08:18:15 GMT Subject: Beazley 4E P.E.R, Page29: Unicode References: <51cbaddd-c29d-48a3-97ab-3beb1d944f1a@googlegroups.com> Message-ID: <51e25ec5$0$9505$c3e8da3$5496439d@news.astraweb.com> On Sat, 13 Jul 2013 20:09:31 -0700, vek.m1234 wrote: > http://stackoverflow.com/questions/17632246/beazley-4e-p-e-r-page29- unicode > > "directly writing a raw UTF-8 encoded string such as 'Jalape\xc3\xb1o' > simply produces a nine-character string U+004A, U+0061, U+006C, U+0061, > U+0070, U+0065, U+00C3, U+00B1, U+006F, which is probably not what you > intended. This is because in UTF-8, the multi-byte sequence \xc3\xb1 is > supposed to represent the single character U+00F1, not the two > characters U+00C3 and U+00B1." This demonstrates confusion of the fundamental concepts, while still accidentally stumbling across the basic facts right. No wonder it is confusing you, it confuses me too! :-) Encoding does not generate a character string, it generates bytes. So the person you are quoting is causing confusion when he talks about an "encoded string", he should either make it clear he means a string of bytes, or not mention the word string at all. Either of these would work: ... a UTF-8 encoded byte-string b'Jalape\xc3\xb1o' ... UTF-8 encoded bytes b'Jalape\xc3\xb1o' For older versions of Python (2.5 or older), unfortunately the b'' notation does not work, and you have to leave out the b. Even better would be if Python did not conflate ASCII characters with bytes, and forced you to write byte strings like this: ... a UTF-8 encoded byte-string b'\x4a\x61\x6c\x61\x70\x65\xc3\xb1\x6f' thus keeping the distinction between ASCII characters and bytes clear. But that would break backwards compatibility *way* too much, and so Python continues to conflate ASCII characters with bytes, even in Python 3. But I digress. The important thing here is that bytes b'Jalape\xc3\xb1o' consists of nine hexadecimal values, as shown above. Seven of them represent the ASCII characters Jalape and o and two of them are not ASCII. Their meaning depends on what encoding you are using. (To be precise, even the meaning of the other seven bytes depends on the encoding. Fortunately, or unfortunately as the case may be, *most* but not all encodings use the same hex values for ASCII characters as ASCII itself does, so I will stop mentioning this and just pretend that character J always equals hex byte 4A. But now you know the truth.) Since we're using the UTF-8 encoding, the two bytes \xc3\xb1 represent the character ?, also known as LATIN SMALL LETTER N WITH TILDE. In other encodings, those two bytes will represent something different. So, I presume that the original person's *intention* was to get a Unicode text string 'Jalape?o'. If they were wise in the ways of Unicode, they would write one of these: 'Jalape\N{LATIN SMALL LETTER N WITH TILDE}o' 'Jalape\u00F1o' 'Jalape\U000000F1o' 'Jalape\xF1o' # hex 'Jalape\361o' # octal and be happy. (In Python 2, they would need to prefix all of these with u, to use Unicode strings instead of byte strings.) But alas they have been mislead by those who propagate myths, misunderstandings and misapprehensions about Unicode all over the Internet, and so they looked up ? somewhere, discovered that it has the double-byte hex value c3b1 in UTF-8, and thought they could write this: 'Jalape\xc3\xb1o' This does not do what they think it does. It creates a *text string*, a Unicode string, with NINE characters: J a l a p e ? ? o Why? Because character ? has ordinal value 195, which is c3 in hex, hence \xc3 is the character ?; likewise \xb1 is the character ? which has ordinal value 177 (b1 in hex). And so they have discovered the wickedness that is mojibake. http://en.wikipedia.org/wiki/Mojibake Instead, if they had started with a *byte-string*, and explicitly decoded it as UTF-8, they would have been fine: # I manually encoded 'Jalape?o' to get the bytes below: bytes = b'Jalape\xc3\xb1o' print(bytes.decode('utf-8')) > My original question was: Shouldn't this be 8 characters - not 9? He > says: \xc3\xb1 is supposed to represent the single character. However > after some interaction with fellow Pythonistas i'm even more confused. Depends on the context. \xc3\xb1 could mean the Unicode string '\xc3\xb1' (in Python 2, written u'\xc3\xb1') or it could mean the byte- string b'\xc3\xb1' (in Python 2.5 or older, written without the b). As a string, \xc3\xb1 means two characters, with ordinal values 0xC3 (or decimal 195) and 0xB1 (or decimal 177), namely '?' and '?'. As bytes, \xc3\xb1 represent two bytes (well, duh), which could mean nearly anything: - the 16-bit Big Endian integer 50097 - the 16-bit Little Endian integer 45507 - a 4x4 black and white bitmap - the character '?' (CJK UNIFIED IDEOGRAPH-7C3D) in Big5 encoded bytes - '?' (HANGUL SYLLABLE NWAES) in UTF-16 (Big Endian) encoded bytes - '?' in UTF-8 encoded bytes - the two characters '??' in Latin-1 encoded bytes - '??' in MacRoman encoded bytes - '??' in ISO-8859-7 encoded bytes and so forth. Without knowing the context, there is no way of telling what those two bytes represent, or whether they need to be taken together as a pair, or as two distinct things. > With reference to the above para: > 1. What does he mean by "writing a raw UTF-8 encoded string"?? He means he is confused. You don't get a text string by encoding, you get bytes (I will accept "byte-string"). The adjective "raw" doesn't really mean anything in this context. You have bytes that were encoded, or you have a string containing characters. Raw doesn't really mean anything except "hey, pay attention, this is low-level stuff" (for some definition of "low level"). > In Python2, once can do 'Jalape funny-n o'. Nothing funny about it to Spanish speakers. Personally, I have always considered "o" to be pretty funny. Say "woman" and "women" aloud -- in the first one, it sounds like "w-oo-man", in the second it sounds like "w-i-men". Now that's funny. But I digress. If you type 'Jalape?o' in Python 2 (with or without the b prefix), the result you get will depend on your terminal settings, but the chances are high that the terminal will internally represent the string as UTF-8, which gives you bytes b'Jalape\xc3\xb1o' which is *nine* bytes. When printed, your terminal will try to print each byte separately, giving: byte \x4a prints as J byte \x61 prints as a byte \x6c prints as l ... and so forth. If you are *unlucky* your terminal may even be smart enough to print the two bytes \xc3\xb1 as one character, giving you the ? you were hoping for. Why unlucky? Because you got the right result by accident. Next time you do the same thing, on a different terminal, or the same terminal set to a different encoding, you will get a completely different result, and think that Unicode is too messed up to use. Using Python 2.5, here I print the same string three times in a row, changing the terminal's encoding each time: py> print 'Jalape\xc3\xb1o' # terminal set to UTF-8 Jalape?o py> print 'Jalape\xc3\xb1o' # and ISO-8859-6 (Arabic) Jalape??o py> print 'Jalape\xc3\xb1o' # and ISO-8859-5 (Cyrillic) Jalape??o Which one is "right"? Answer: none of them. Not even the first, which by accident just happened to be what we were hoping for. Really, don't feel bad that you are confused. Between Python 2, and the terminal trying *really hard* to do the right thing, it is easy to get confused because something the right thing happens and sometimes it doesn't. > This is a 'bytes' string where each glyph is 1 byte long Nope. It's a string of characters. Glyphs don't come into it. Glyphs are the little pictures of letters that you see on the screen, or printed on paper. They could be bitmaps, or fancy vector graphics. They are unlikely to be one byte each -- more likely 200 bytes per glyph, based on a very rough calculation[1], but depending on whether it is a bitmap, a Postscript font, an OpenType font, or something else. > when stored internally so each glyph is > associated with an integer as per charset ASCII or Latin-1. If these > charsets have a funny-n glyph then yay! else nay! There is no UTF-8 > here!! or UTF-16!! These are plain bytes (8 bits). You're getting closer. But you are right: Python 2 "strings" are byte- strings, which means UTF-8 doesn't come into it. But your terminal might treat those bytes as UTF-8, and so accidentally do the "right" (wrong) thing. > Unicode is a really big mapping table between glyphs and integers and Not glyphs. Between abstract "characters" and integers, called Code Points. Unicode contains: - distinct letters, digits, characters - accented letters - accents on their own - symbols, emoticons - ligatures and variant forms of characters - chars required only for backwards-compatibility with older encodings - whitespace - control characters - code points reserved for private use, which can mean anything you like - code points reserved as "will never be used" - code points explicitly labelled "not a character" and possibly others I have forgotten. > are denoted as Uxxxx or Uxxxx-xxxx. The official Unicode notation is: U+xxxx U+xxxxx U+xxxxxx that is U+ followed by exactly four, five or six hex digits. The U is always uppercase. Unfortunately Python doesn't support that notation, and you have to use either four or eight hex digits, e.g.: \uFFFF \U0010FFFF For code points (ordinals) up to 255, you can also use hex or octal escapes, e.g. \xFF \3FF > UTF-8 UTF-16 are encodings to store > those big integers in an efficient manner. Almost correct. They're not necessarily efficient. Unicode code points are just abstract numbers that we give some meaning to. Code point 65 (U+0041, because hex 41 == decimal 65) means letter A, and so forth. Imagine these abstract code points floating in your head. How do you get the abstract concept of a code point into concrete form on a computer? The same way *everything* is put in a computer: as bytes, so we have to turn each abstract code point (a number) into a series of bytes. Unicode code points range from U+0000 to U+10FFFF, which means we could just use exactly three bytes, which take values from 000000 to 10FFFF in hexadecimal. Values outside of this range, say 110000, would be an error. For reasons of efficiency, it's faster and better to use *four* bytes, even though one of the four will always have the value zero. In a nutshell, that's the UTF-32 encoding: ever character uses exactly four bytes. E.g. code point U+0041 (character A) is hex bytes 00000041, or possible 41000000, depending on whether your computer is Big Endian or Little Endian. Since *most* text uses quite low ordinal values, that's awfully wasteful of memory. So UTF-16 uses just two bytes per character, and a weird scheme using so-called "surrogate pairs" for everything that won't fit into two bytes. It works, for some definition of "works", but is complicated, and you really want to avoid UTF-16 if you need code points above U+FFFF. UTF-8 uses a neat variable encoding where characters with low ordinal values get encoded as a single byte (better still: it is the same byte as ASCII uses, which means old software that assumes everything in the world is ASCII will keep working, well mostly working). Higher ordinals get encoded as two, three or four bytes[2]. Best of all, unlike most historical variable-width encodings, UTF-8 is self-synchronising. In legacy encodings, if a single byte gets corrupted, it can mangle *everything* from that point on. With UTF-8, a single corrupted byte will mangle only the single code-point containing it, everything following will be okay. > So when DB says "writing a > raw UTF-8 encoded string" - well the only way to do this is to use > Python3 where the default string literals are stored in Unicode which > then will use a UTF-8 UTF-16 internally to store the bytes in their > respective structures; or, one could use u'Jalape' which is unicode in > both languages (note the leading 'u'). Python never uses UTF-8 internally for storing strings in memory. Because it is a variable width encoding, you cannot index strings efficiently if they use UTF-8 for storage. Instead, Python uses one of three different systems: - Up to Python 3.3, you have a choice. When you compile the Python interpreter, you can choose whether it should use UTF-16 or UTF-32 for in- memory storage. This choice is called "narrow" or "wide" build. A narrow build uses less memory, but cannot handle code points above U+FFFF very well. A wide build uses more memory, but handles the complete range of code points perfectly. - Starting in Python 3.3, the choice of how to store the string in memory is no longer decided up front when you build the Python interpreter. Instead, Python automatically chooses the most efficient internal representation for each individual string. Strings which only use ASCII or Latin-1 characters use one byte per character; string which use code points up to U+FFFF use two bytes per character; and only strings which use code points above that use four bytes per character. > 2. So assuming this is Python 3: 'Jalape \xYY \xZZ o' (spaces for > readability) what DB is saying is that, the stupid-user would expect > Jalapeno with a squiggly-n but instead he gets is: Jalape funny1 funny2 > o (spaces for readability) -9 glyphs or 9 Unicode-points or 9-UTF8 > characters. Correct? Kind of. See above. > 3. Which leaves me wondering what he means by: "This is because in > UTF-8, the multi- byte sequence \xc3\xb1 is supposed to represent the > single character U+00F1, not the two characters U+00C3 and U+00B1" He means that the single code point U+00F1 (character ?, n with a tilde) is stored as the two bytes c3b1 (in hexadecimal) if you encode it using UTF-8. But if you stuff characters \xc3 \xb1 into a Unicode string (instead of bytes), then you get two Unicode characters U+00C3 and U+00B1. To put it another way, inside strings, Python treats the hex escape \xC3 as just a different way of writing the Unicode code point \u00C3 or \U000000C3. However, if you create a byte-string: b'Jalape\xc3\xb1o' by looking up a table of UTF-8 encodings, as presumably the original poster did, and then decode those bytes to a string, you will get what you expect. Using Python 2.5, where the b prefix is not needed: py> tasty = 'Jalape\xc3\xb1o' # actually bytes py> tasty.decode('utf-8') u'Jalape\xf1o' py> print tasty.decode('utf-8') # oops I forgot to reset my terminal Jalape??o py> print tasty.decode('utf-8') # terminal now set to UTF-8 Jalape?o > Could someone take the time to read carefully and clarify what DB is > saying?? Hope this helps. [1] Assume the font file is 100K in size, and it has glyphs for 256 characters. That works out to 195 bytes per glyph. [2] Technically, the UTF-8 scheme can handle 31-bit code points, up to the (hypothetical) code point U+7FFFFFFF, using up to six bytes per code point. But Unicode officially will never go past U+10FFFF, and so UTF-8 also will never go past four bytes per code point. -- Steven From vek.m1234 at gmail.com Sun Jul 14 05:39:09 2013 From: vek.m1234 at gmail.com (vek.m1234 at gmail.com) Date: Sun, 14 Jul 2013 02:39:09 -0700 (PDT) Subject: Beazley 4E P.E.R, Page29: Unicode In-Reply-To: <51e25ec5$0$9505$c3e8da3$5496439d@news.astraweb.com> References: <51cbaddd-c29d-48a3-97ab-3beb1d944f1a@googlegroups.com> <51e25ec5$0$9505$c3e8da3$5496439d@news.astraweb.com> Message-ID: Hello Steven, a 'thank you' sounds insufficient and largely disproportionate to to the time and energy you spent in drafting a thoroughly comprehensive answer to my question. I've cross posted both answers to stackoverflow (with some minor formatting changes). I'll try to do something nice on your account. From gialloporpora at gmail.com Sun Jul 14 07:28:15 2013 From: gialloporpora at gmail.com (gialloporpora) Date: Sun, 14 Jul 2013 13:28:15 +0200 Subject: How to internationalize a python script? (i18n) Message-ID: Hello, I am trying to internationalize a script. First I have tried with a little script to understand how it works, but unfortunately, it doesn't. I have followed instruction in this page: http://docs.python.org/2/library/i18n.html I have created my script, marked strings with the _() function, installed it in the main namespace, but when I try to load the locale file it said methat locale is unavailable: IOError: [Errno 2] No translation file found for domain: 'helloi18n' C:\dropbox\Public\helloi18n> I have created the pot file with pygettext, localized it with poedit and compiled the related .mo file. As reported in this page: http://docs.python.org/2/library/gettext.html ?Bind the domain to the locale directory localedir. More concretely, gettext will look for binary .mo files for the given domain using the path (on Unix): localedir/language/LC_MESSAGES/domain.mo, where languages is searched for in the environment variables LANGUAGE, LC_ALL, LC_MESSAGES, and LANG respectively.? I have put my .mo file in locale\it\LC_MESSAGES naming it helloi18n.mo here are all my files: https://dl.dropboxusercontent.com/u/4400966/helloi18n.tar.gz This is the code of the helloi18n.py file: > # Simple script to use internationalization (i18n) > import gettext > import os > > > LOCALE_DIR = os.path.join(os.path.abspath('.'), 'locale') > print LOCALE_DIR > print "---" > a=gettext.find('helloi18n', LOCALE_DIR, 'it') > print a > gettext.install('helloi18n', localedir=LOCALE_DIR, unicode=1) > gettext.textdomain ('helloi18n') > gettext.translation('helloi18n', LOCALE_DIR, 'it') > > > if __name__ == '__main__': > print _('Hello world!') > print (_('My first localized python script')) Somebody could help me? Sandro From gialloporpora at gmail.com Sun Jul 14 11:04:51 2013 From: gialloporpora at gmail.com (gialloporpora) Date: Sun, 14 Jul 2013 17:04:51 +0200 Subject: How to internationalize a python script? (i18n) In-Reply-To: References: Message-ID: Risposta al messaggio di gialloporpora : >>gettext.translation('helloi18n', LOCALE_DIR, 'it') Ok, I have, with a little help of my friend, found the issue. The language code must be passed as a list not as a string. Sorry. Sandro -- *Thunderbird come evitare il circolo vizioso ?Re: R:? negli oggetti delle mail * - http://bit.ly/19yMSsZ From rosuav at gmail.com Sun Jul 14 10:56:29 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 15 Jul 2013 00:56:29 +1000 Subject: Timing of string membership (was Re: hex dump w/ or w/out utf-8 chars) Message-ID: On Sun, Jul 14, 2013 at 11:44 PM, wrote: > Le dimanche 14 juillet 2013 12:44:12 UTC+2, Steven D'Aprano a ?crit : >> On Sun, 14 Jul 2013 01:20:33 -0700, wxjmfauth wrote: >> >> >> >> > For a very simple reason, the latin-1 block: considered and accepted >> >> > today as beeing a Unicode design mistake. >> >> >> >> Latin-1 (also known as ISO-8859-1) was based on DEC's "Multinational >> >> Character Set", which goes back to 1983. ISO-8859-1 was first published >> >> in 1985, and was in use on Commodore computers the same year. >> >> >> >> The concept of Unicode wasn't even started until 1987, and the first >> >> draft wasn't published until the end of 1990. Unicode wasn't considered >> >> ready for production use until 1991, six years after Latin-1 was already >> >> in use in people's computers. >> >> >> >> >> >> >> >> -- >> >> Steven > > ------ > > "Unicode" (in fact iso-14xxx) was not created in one > night (Deus ex machina). > > What's count today is this: > >>>> timeit.repeat("a = 'hundred'; 'x' in a") > [0.11785943134991479, 0.09850454944486256, 0.09761604599423179] >>>> timeit.repeat("a = 'hundre?'; 'x' in a") > [0.23955250303158593, 0.2195812612416752, 0.22133896997401692] >>>> >>>> >>>> sys.getsizeof('d') > 26 >>>> sys.getsizeof('?') > 40 >>>> sys.version > '3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:03:43) [MSC v.1600 32 bit (Intel)]' jmf has raised an interesting point. Some string membership operations do seem oddly slow. # Get ourselves a longish ASCII string with no duplicates - escape apostrophe and backslash for code later on >>> asciichars=''.join(chr(i) for i in range(32,128)).replace("\\",r"\\").replace("'",r"\'") >>> haystack=[ ("ASCII",asciichars+"\u0001"), ("BMP",asciichars+"\u1234"), ("SMP",asciichars+"\U00012345"), ] >>> needle=[ ("ASCII","\u0002"), ("BMP","\u1235"), ("SMP","\U00012346"), ] >>> useset=[ ("",""), (", as set","; a=set(a)"), ] >>> for time,desc in sorted((min(timeit.repeat("'%s' in a"%n,("a='%s'"%h)+s)),"%s in %s%s"%(nd,hd,sd)) for nd,n in needle for hd,h in haystack for sd,s in useset): print("%.10f %s"%(time,desc)) 0.1765129367 ASCII in ASCII, as set 0.1767096097 BMP in SMP, as set 0.1778647845 ASCII in BMP, as set 0.1785266004 BMP in BMP, as set 0.1789093307 SMP in SMP, as set 0.1790431465 SMP in BMP, as set 0.1796504863 BMP in ASCII, as set 0.1803854959 SMP in ASCII, as set 0.1810674262 ASCII in SMP, as set 0.1817367850 SMP in BMP 0.1884555160 SMP in ASCII 0.2132371572 BMP in ASCII 0.3137454621 ASCII in ASCII 0.4472624314 BMP in BMP 0.6672795006 SMP in SMP 0.7493052888 ASCII in BMP 0.9261783271 ASCII in SMP 0.9865787412 BMP in SMP (In separate testing I ascertained that it makes little difference whether the character is absent from the string or is the last character in it. Presumably the figures would be lower if the character is at the start of the string, but this is not germane to this discussion.) Set membership is faster than string membership, though marginally on something this short. If the needle is wider than the haystack, it obviously can't be present, so a false return comes back at the speed of a set check. Otherwise, an actual search must be done. Searching for characters in strings of the same width gets slower as the strings get larger in memory (unsurprising). What I'm seeing of the top-end results, though, is that the search for a narrower string in a wider one is quite significantly slower. I don't know of an actual proven use-case for this, but it seems likely to happen (eg you take user input and want to know if there are any HTML-sensitive characters in it, so you check ('<' in string or '&' in string), for instance). The question is, is it worth constructing an "expanded string" at the haystack's width prior to doing the search? ChrisA From tjreedy at udel.edu Mon Jul 15 00:18:46 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 15 Jul 2013 00:18:46 -0400 Subject: Timing of string membership (was Re: hex dump w/ or w/out utf-8 chars) In-Reply-To: References: Message-ID: On 7/14/2013 10:56 AM, Chris Angelico wrote: > On Sun, Jul 14, 2013 at 11:44 PM, wrote: >>>>> timeit.repeat("a = 'hundred'; 'x' in a") >> [0.11785943134991479, 0.09850454944486256, 0.09761604599423179] >>>>> timeit.repeat("a = 'hundre?'; 'x' in a") >> [0.23955250303158593, 0.2195812612416752, 0.22133896997401692] >>>>> sys.version >> '3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:03:43) [MSC v.1600 32 bit (Intel)]' As issue about finding stings in strings was opened last September and, as reported on this list, fixes were applied about last March. As I remember, some but not all of the optimizations were applied to 3.3. Perhaps some were applied too late for 3.3.1 (3.3.2 is 3.3.1 with some emergency patches to correct regressions). Python 3.4.0a2: >>> import timeit >>> timeit.repeat("a = 'hundred'; 'x' in a") [0.17396483610667152, 0.16277956641670813, 0.1627937074749941] >>> timeit.repeat("a = 'hundreo'; 'x' in a") [0.18441108179403187, 0.16277311071618783, 0.16270517215355085] The difference is gone, again, as previously reported. > jmf has raised an interesting point. Some string membership operations > do seem oddly slow. He raised it a year ago and action was taken. > > # Get ourselves a longish ASCII string with no duplicates - escape > apostrophe and backslash for code later on >>>> asciichars=''.join(chr(i) for i in range(32,128)).replace("\\",r"\\").replace("'",r"\'") >>>> haystack=[ > ("ASCII",asciichars+"\u0001"), > ("BMP",asciichars+"\u1234"), > ("SMP",asciichars+"\U00012345"), > ] >>>> needle=[ > ("ASCII","\u0002"), > ("BMP","\u1235"), > ("SMP","\U00012346"), > ] >>>> useset=[ > ("",""), > (", as set","; a=set(a)"), > ] >>>> for time,desc in sorted((min(timeit.repeat("'%s' in a"%n,("a='%s'"%h)+s)),"%s in %s%s"%(nd,hd,sd)) for nd,n in needle for hd,h in haystack for sd,s in useset): > print("%.10f %s"%(time,desc)) > > 0.1765129367 ASCII in ASCII, as set > 0.1767096097 BMP in SMP, as set > 0.1778647845 ASCII in BMP, as set > 0.1785266004 BMP in BMP, as set > 0.1789093307 SMP in SMP, as set > 0.1790431465 SMP in BMP, as set > 0.1796504863 BMP in ASCII, as set > 0.1803854959 SMP in ASCII, as set > 0.1810674262 ASCII in SMP, as set Much of this time is overhead; 'pass' would not run too much faster. > 0.1817367850 SMP in BMP > 0.1884555160 SMP in ASCII > 0.2132371572 BMP in ASCII For these, 3.3 does no searching because it knows from the internal char kind that the answer is No without looking. > 0.3137454621 ASCII in ASCII > 0.4472624314 BMP in BMP > 0.6672795006 SMP in SMP > 0.7493052888 ASCII in BMP > 0.9261783271 ASCII in SMP > 0.9865787412 BMP in SMP ... > Set membership is faster than string membership, though marginally on > something this short. If the needle is wider than the haystack, it > obviously can't be present, so a false return comes back at the speed > of a set check. Jim ignores these cases where 3.3+ uses the information about the max codepoint to do the operation much faster than in 3.2. > Otherwise, an actual search must be done. Searching > for characters in strings of the same width gets slower as the strings > get larger in memory (unsurprising). What I'm seeing of the top-end > results, though, is that the search for a narrower string in a wider > one is quite significantly slower. 50% longer is not bad, even > I don't know of an actual proven use-case for this, but it seems > likely to happen (eg you take user input and want to know if there are > any HTML-sensitive characters in it, so you check ('<' in string or > '&' in string), for instance). In my editing of code, I nearly always search for words or long names. The question is, is it worth > constructing an "expanded string" at the haystack's width prior to > doing the search? I would not make any assumptions about what Python does or does not do without checking the code. All I know is that Python uses a modified version of one of the pre-process and skip-forward algorithms (Boyer-Moore?, Knuth-Pratt?, I forget). These are designed to work efficiently with needles longer than 1 char, and indeed may work better with longer needles. Searching for an single char in n chars is O(n). Searching for a len m needle is potentially O(m*n) and the point of the fancy algorithms is make all searches as close to O(n) as possible. -- Terry Jan Reedy From rosuav at gmail.com Mon Jul 15 00:31:13 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 15 Jul 2013 14:31:13 +1000 Subject: Timing of string membership (was Re: hex dump w/ or w/out utf-8 chars) In-Reply-To: References: Message-ID: On Mon, Jul 15, 2013 at 2:18 PM, Terry Reedy wrote: > On 7/14/2013 10:56 AM, Chris Angelico wrote: > As issue about finding stings in strings was opened last September and, as > reported on this list, fixes were applied about last March. As I remember, > some but not all of the optimizations were applied to 3.3. Perhaps some were > applied too late for 3.3.1 (3.3.2 is 3.3.1 with some emergency patches to > correct regressions). D'oh. I knew there was something raised and solved regarding that, but I forgot to go check a 3.4 alpha to see if it exhibited the same. Whoops. My bad. Sorry! > Python 3.4.0a2: >>>> import timeit > >>>> timeit.repeat("a = 'hundred'; 'x' in a") > [0.17396483610667152, 0.16277956641670813, 0.1627937074749941] >>>> timeit.repeat("a = 'hundreo'; 'x' in a") > [0.18441108179403187, 0.16277311071618783, 0.16270517215355085] > > The difference is gone, again, as previously reported. Yep, that looks exactly like I would have hoped it would. >> 0.1765129367 ASCII in ASCII, as set > > Much of this time is overhead; 'pass' would not run too much faster. > >> 0.1817367850 SMP in BMP >> 0.1884555160 SMP in ASCII >> 0.2132371572 BMP in ASCII > > For these, 3.3 does no searching because it knows from the internal char > kind that the answer is No without looking. Yeah, I mainly included those results so I could say to jmf "Look, FSR allows some string membership operations to be, I kid you not, as fast as set operations!". >> 0.3137454621 ASCII in ASCII >> 0.4472624314 BMP in BMP >> 0.6672795006 SMP in SMP >> 0.7493052888 ASCII in BMP >> 0.9261783271 ASCII in SMP >> 0.9865787412 BMP in SMP > >> Otherwise, an actual search must be done. Searching >> for characters in strings of the same width gets slower as the strings >> get larger in memory (unsurprising). What I'm seeing of the top-end >> results, though, is that the search for a narrower string in a wider >> one is quite significantly slower. > > 50% longer is not bad, even Hard to give an estimate; my first tests were the ASCII in ASCII and ASCII in BMP, which then looked more like 2:1 time. However, rescaling the needle to BMP makes it more like the 50% you're quoting, so yes, it's not as much as I thought. In any case, the most important thing to note is: 3.4 has already fixed this, ergo jmf should shut up about it. And here I thought I could credit him with a second actually-useful report... ChrisA From jcasale at activenetwerx.com Sun Jul 14 13:10:34 2013 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Sun, 14 Jul 2013 17:10:34 +0000 Subject: List comp help Message-ID: I have a dict of lists. I need to create a list of 2 tuples, where each tuple is a key from the dict with one of the keys list items. my_dict = { 'key_a': ['val_a', 'val_b'], 'key_b': ['val_c'], 'key_c': [] } [(k, x) for k, v in my_dict.items() for x in v] This works, but I need to test for an empty v like the last key, and create one tuple ('key_c', None). Anyone know the trick to reorganize this to accept the test for an empty v and add the else? Thanks! jlc From rosuav at gmail.com Sun Jul 14 13:16:23 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 15 Jul 2013 03:16:23 +1000 Subject: List comp help In-Reply-To: References: Message-ID: On Mon, Jul 15, 2013 at 3:10 AM, Joseph L. Casale wrote: > I have a dict of lists. I need to create a list of 2 tuples, where each tuple is a key from > the dict with one of the keys list items. > > my_dict = { > 'key_a': ['val_a', 'val_b'], > 'key_b': ['val_c'], > 'key_c': [] > } > [(k, x) for k, v in my_dict.items() for x in v] > > This works, but I need to test for an empty v like the last key, and create one tuple ('key_c', None). > Anyone know the trick to reorganize this to accept the test for an empty v and add the else? Yeah, it's remarkably easy too! Try this: [(k, x) for k, v in my_dict.items() for x in v or [None]] An empty list counts as false, so the 'or' will then take the second option, and iterate over the one-item list with None in it. Have fun! ChrisA From jcasale at activenetwerx.com Sun Jul 14 14:09:45 2013 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Sun, 14 Jul 2013 18:09:45 +0000 Subject: List comp help In-Reply-To: References: Message-ID: <827285033f2148e6a92d67869ba94e74@exch.activenetwerx.com> > Yeah, it's remarkably easy too! Try this: > > [(k, x) for k, v in my_dict.items() for x in v or [None]] > > An empty list counts as false, so the 'or' will then take the second option, and iterate over the one-item list with > > None in it. Right, I overlooked that! Much appreciated, jlc From rurpy at yahoo.com Sun Jul 14 14:32:34 2013 From: rurpy at yahoo.com (rurpy at yahoo.com) Date: Sun, 14 Jul 2013 11:32:34 -0700 (PDT) Subject: List comp help In-Reply-To: References: Message-ID: On 07/14/2013 11:16 AM, Chris Angelico wrote: > On Mon, Jul 15, 2013 at 3:10 AM, Joseph L. Casale > wrote: >> I have a dict of lists. I need to create a list of 2 tuples, where each tuple is a key from >> the dict with one of the keys list items. >> >> my_dict = { >> 'key_a': ['val_a', 'val_b'], >> 'key_b': ['val_c'], >> 'key_c': [] >> } >> [(k, x) for k, v in my_dict.items() for x in v] >> >> This works, but I need to test for an empty v like the last key, and create one tuple ('key_c', None). >> Anyone know the trick to reorganize this to accept the test for an empty v and add the else? > > Yeah, it's remarkably easy too! Try this: > > [(k, x) for k, v in my_dict.items() for x in v or [None]] > > An empty list counts as false, so the 'or' will then take the second > option, and iterate over the one-item list with None in it. Or more simply: [(k, v or None) for k, v in my_dict.items()] This assumes that all the values in my_dict are lists, and not other false values like 0, which would also be replaced by None. From rurpy at yahoo.com Sun Jul 14 14:38:35 2013 From: rurpy at yahoo.com (rurpy at yahoo.com) Date: Sun, 14 Jul 2013 11:38:35 -0700 (PDT) Subject: List comp help In-Reply-To: References: Message-ID: On Sunday, July 14, 2013 12:32:34 PM UTC-6, ru... at yahoo.com wrote: > Or more simply: > [(k, v or None) for k, v in my_dict.items()] Too simply :-( Didn't read the op carefully enough. Sorry. From tjreedy at udel.edu Mon Jul 15 00:46:05 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 15 Jul 2013 00:46:05 -0400 Subject: List comp help In-Reply-To: References: Message-ID: On 7/14/2013 1:10 PM, Joseph L. Casale wrote: > I have a dict of lists. I need to create a list of 2 tuples, where each tuple is a key from > the dict with one of the keys list items. > > my_dict = { > 'key_a': ['val_a', 'val_b'], > 'key_b': ['val_c'], > 'key_c': [] > } > [(k, x) for k, v in my_dict.items() for x in v] The order of the tuples in not deterministic unless you sort, so if everything is hashable, a set may be better. > This works, but I need to test for an empty v like the last key, and create one tuple ('key_c', None). > Anyone know the trick to reorganize this to accept the test for an empty v and add the else? When posting code, it is a good idea to includes the expected or desired answer in code as well as text. pairs = {(k, x) for k, v in my_dict.items() for x in v or [None]} assert pairs == {('key_a', 'val_a'), ('key_a', 'val_b'), ('key_b', 'val_c'), ('key_c', None)} -- Terry Jan Reedy From gildororonar at mail-on.us Sun Jul 14 22:27:45 2013 From: gildororonar at mail-on.us (Gildor Oronar) Date: Mon, 15 Jul 2013 10:27:45 +0800 Subject: what thread-synch mech to use for clean exit from a thread Message-ID: A currency exchange thread updates exchange rate once a minute. If the thread faield to update currency rate for 5 hours, it should inform main() for a clean exit. This has to be done gracefully, because main() could be doing something delicate. I, a newbie, read all the thread sync tool, and wasn't sure which one to use. In fact I am not sure if there is a need of thread sync, because there is no racing cond. I thought of this naive way: class CurrencyExchange(): def __init__(in_case_callback): this.callback = in_case_callback def __run__(): while time.time() - self.rate_timestamp < 5*3600: ... # update exchange rate if success: self.rate_timestamp == time.time() time.sleep(60) this.callback() # rate not updated 5 hours, a crisis def main(): def callback() Go_On = False agio = CurrencyExchange(in_case = callback) agio.start() Go_On = True while Go_On: do_something_delicate(rate_supplied_by=agio) As you can see, if there is no update of currency rate for 5 hours, the CurrencyExchange object calls the callback, which prevents main() from doing the next delicate_thing, but do not interrupt the current delicate_thing. This seems OK, but doesn't look pythonic -- replacing callback() with a lambda doesn't help much, it still look naive. What is the professional way in this case? Thanks in advance! From zhangweiwu at realss.com Sun Jul 14 22:32:52 2013 From: zhangweiwu at realss.com (zhangweiwu at realss.com) Date: Sun, 14 Jul 2013 19:32:52 -0700 (PDT) Subject: what thread-synch mech to use for clean exit from a thread In-Reply-To: References: Message-ID: On Monday, July 15, 2013 10:27:45 AM UTC+8, Gildor Oronar wrote: > What is the professional way in this case? Hi. I am not a professional neither but I think a professional does this: class CurrencyExchange(): def __init__(in_case_callback): this.callback = in_case_callback def __run__(): while time.time() - self.rate_timestamp < 5*3600: ... # update exchange rate if success: self.rate_timestamp == time.time() time.sleep(60) def main(): agio = CurrencyExchange(in_case = callback) agio.start() while agio.is_alive(): do_something_delicate(rate_supplied_by=agio) Notice even if agio is no longer alive, it can still supply exchange rate for the last delicate_thing, only that it no longer updates punctually. This is semantic wrong, and I think it is the fault of python: how can something dead execute its method? In the API, thread.is_alive() should be renamed to thread.is_activate_and_on_his_own() From steve at pearwood.info Sun Jul 14 23:04:14 2013 From: steve at pearwood.info (Steven D'Aprano) Date: 15 Jul 2013 03:04:14 GMT Subject: what thread-synch mech to use for clean exit from a thread References: Message-ID: <51e366ae$0$29866$c3e8da3$5496439d@news.astraweb.com> On Mon, 15 Jul 2013 10:27:45 +0800, Gildor Oronar wrote: > A currency exchange thread updates exchange rate once a minute. If the > thread faield to update currency rate for 5 hours, it should inform > main() for a clean exit. This has to be done gracefully, because main() > could be doing something delicate. > > I, a newbie, read all the thread sync tool, and wasn't sure which one to > use. In fact I am not sure if there is a need of thread sync, because > there is no racing cond. I thought of this naive way: > > class CurrencyExchange(): > def __init__(in_case_callback): > this.callback = in_case_callback You need to declare the instance parameter, which is conventionally called "self" not "this". Also, your class needs to inherit from Thread, and critically it MUST call the superclass __init__. So: class CurrencyExchange(threading.Thread): def __init__(self, in_case_callback): super(CurrencyExchange, self).__init__() self.callback = in_case_callback But I'm not sure that a callback is the right approach here. See below. > def __run__(): Likewise, you need a "self" parameter. > while time.time() - self.rate_timestamp < 5*3600: > ... # update exchange rate > if success: > self.rate_timestamp == time.time() > time.sleep(60) > this.callback() # rate not updated 5 hours, a crisis I think that a cleaner way is to just set a flag on the thread instance. Initiate it with: self.updates_seen = True in the __init__ method, and then add this after the while loop: self.updates_seen = False > def main(): > def callback() > Go_On = False I don't believe this callback will work, because it will simply create a local variable call "Go_On", not change the non-local variable. In Python 3, you can use the nonlocal keyword to get what you want, but I think a better approach is with a flag on the thread. > agio = CurrencyExchange(in_case = callback) > agio.start() > > Go_On = True > while Go_On: > do_something_delicate(rate_supplied_by=agio) Change to: while agio.updates_seen: do_something_delicate... -- Steven From steve at pearwood.info Sun Jul 14 23:10:01 2013 From: steve at pearwood.info (Steven D'Aprano) Date: 15 Jul 2013 03:10:01 GMT Subject: what thread-synch mech to use for clean exit from a thread References: <51e366ae$0$29866$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51e36809$0$29866$c3e8da3$5496439d@news.astraweb.com> Oh, I forgot another comment... On Mon, 15 Jul 2013 03:04:14 +0000, Steven D'Aprano wrote: > On Mon, 15 Jul 2013 10:27:45 +0800, Gildor Oronar wrote: >> while time.time() - self.rate_timestamp < 5*3600: >> ... # update exchange rate >> if success: >> self.rate_timestamp == time.time() >> time.sleep(60) >> this.callback() # rate not updated 5 hours, a crisis > > I think that a cleaner way is to just set a flag on the thread instance. > Initiate it with: > > self.updates_seen = True > > in the __init__ method, and then add this after the while loop: > > self.updates_seen = False Sorry, I forgot to mention... I assume that the intention is that if the thread hasn't seen any updates for five hours, it should set the flag, and then *keep going*. Perhaps the rate will start updating again later. If the intention is to actually close the thread, then there's no reason for an extra flag. Just exit the run() method normally, the thread will die, and you can check the thread's status with the is_alive() method. -- Steven From python at mrabarnett.plus.com Mon Jul 15 08:09:07 2013 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 15 Jul 2013 13:09:07 +0100 Subject: what thread-synch mech to use for clean exit from a thread In-Reply-To: <51e366ae$0$29866$c3e8da3$5496439d@news.astraweb.com> References: <51e366ae$0$29866$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51E3E663.7030302@mrabarnett.plus.com> On 15/07/2013 04:04, Steven D'Aprano wrote: > On Mon, 15 Jul 2013 10:27:45 +0800, Gildor Oronar wrote: > >> A currency exchange thread updates exchange rate once a minute. If the >> thread faield to update currency rate for 5 hours, it should inform >> main() for a clean exit. This has to be done gracefully, because main() >> could be doing something delicate. >> >> I, a newbie, read all the thread sync tool, and wasn't sure which one to >> use. In fact I am not sure if there is a need of thread sync, because >> there is no racing cond. I thought of this naive way: >> >> class CurrencyExchange(): >> def __init__(in_case_callback): >> this.callback = in_case_callback > > You need to declare the instance parameter, which is conventionally > called "self" not "this". Also, your class needs to inherit from Thread, > and critically it MUST call the superclass __init__. > > So: > > class CurrencyExchange(threading.Thread): > def __init__(self, in_case_callback): > super(CurrencyExchange, self).__init__() > self.callback = in_case_callback > > But I'm not sure that a callback is the right approach here. See below. > > >> def __run__(): > > Likewise, you need a "self" parameter. > > >> while time.time() - self.rate_timestamp < 5*3600: >> ... # update exchange rate >> if success: The "==" in this line should, of course, be "=": >> self.rate_timestamp == time.time() >> time.sleep(60) >> this.callback() # rate not updated 5 hours, a crisis > > I think that a cleaner way is to just set a flag on the thread instance. > Initiate it with: > > self.updates_seen = True > > in the __init__ method, and then add this after the while loop: > > self.updates_seen = False > > > >> def main(): >> def callback() >> Go_On = False > > I don't believe this callback will work, because it will simply create a > local variable call "Go_On", not change the non-local variable. > > In Python 3, you can use the nonlocal keyword to get what you want, but I > think a better approach is with a flag on the thread. > >> agio = CurrencyExchange(in_case = callback) >> agio.start() >> >> Go_On = True >> while Go_On: >> do_something_delicate(rate_supplied_by=agio) > > Change to: > > while agio.updates_seen: > do_something_delicate... > > From thrinaxodon.fan.club512 at gmail.com Sun Jul 14 23:30:35 2013 From: thrinaxodon.fan.club512 at gmail.com (LOUZY) Date: Sun, 14 Jul 2013 23:30:35 -0400 Subject: SMITHSONIAN HAS IT'S LAST WORDS... Message-ID: ======================= A TOUCHY SUBJECT... ======================= > A WILY THRINAXODON SUED THE SMITHSONIAN FIVE HUNDRED DOLLARS FOR SUPPRESSION OF FREEDOM OF EXPRESSION. > "This is a blow to evolutionism," SAID RICHARD DAWKINS. > ONE WHOM THRINAXODON HAS HAD SEVERAL *long* RUNNING FEUDS OVER THE PAST 40 YEARS. > THE SMITHSONIAN IS BEING TORN DOWN. > THE SPECIMENS BURNED, BOOKS REWRITTEN, etc. > "This never happened with Ed Conrad," SAID BARACK OBAMA. > EVOLUTIONISTS ALL OVER THE WORLD GET SUNK IN TEARS AS ONE OF THE MAJOR CORPORATIONS GET SHUT DOWN... > ======================== > I KNOW, I KNOW...YOU NEED A RESOURCE FOR THIS SCIENCE. > WELL TYPE IN news://sci.bio.paleontology, news://sci.skeptic, news://dc.smithsonian, etc ON YOUR WEB BROWSER. > ======================= > http://thrinaxodon.wordpress.com/ > ======================= > THRINAXODON IS NOW ON TWITTER. From alphonse23 at gmail.com Mon Jul 15 05:23:40 2013 From: alphonse23 at gmail.com (alphonse23 at gmail.com) Date: Mon, 15 Jul 2013 02:23:40 -0700 (PDT) Subject: minfuds - coding puzzle Message-ID: <0d5e1ae8-30b9-42d3-bc3e-268bdd32f717@googlegroups.com> Would anybody be up to helping me solve this? It's one of the questions on Codility. According to them, it should only take 30 mins. Two non-empty zero-indexed arrays A and B, each consisting of N integers, are given. Four functions are defined based on these arrays: F(X,K) = A[K]*X + B[K] U(X) = max{ F(X,K) : 0 ? K < N } D(X) = min{ F(X,K) : 0 ? K < N } S(X) = U(X) ? D(X) Write a function: double solution(int A[], int B[], int N); that, given two arrays A and B consisting of N integers each, returns the minimum value of S(X) where X can be any real number. For example, given the following arrays A and B consisting of three elements each: A[0] = -1 B[0] = 3 A[1] = 1 B[1] = 0 A[2] = 0 B[2] = 2 the function should return 0.5 because: U(X) = ?1*X + 3 if X ? 1 U(X) = 0*X + 2 if 1 ? X ? 2 U(X) = 1*X + 0 if 2 ? X and: D(X) = 1*X + 0 if X ? 1.5 D(X) = ?1*X + 3 if 1.5 ? X so for X = 1.5, function S(X) is equal to 0.5 and this is the minimum value of this function. Assume that: N is an integer within the range [1..100,000]; each element of array A is an integer within the range [?1,000..1,000]; each element of array B is an integer within the range [?1,000..1,000]. Complexity: expected worst-case time complexity is O(N*log(N)); expected worst-case space complexity is O(N), beyond input storage (not counting the storage required for input arguments). Elements of input arrays can be modified. I'm not sure how to solve it. I feel like it's the equivalent of finding the minimum of a function on a graph. I'd know how to do that in algebra/calculus, but not with code. Also, I'm not sure why their example skipped A[2] = 0 and B[2] = 2 for D(X). This is as far as I got: def solution(A, B): # write your code here... min = 0 return min def S(A,B,x): return U(A,B,x) - D(A,B,x) def F(a,x,b): return a*x + b def U(A,B,x): max = F(A[0],x,B[0]) for i in range(1,len(A)): u = F(A[i],x,B[i]) if max < u: max = u return max def D(A,B,x): min = F(A[0],x,B[0]) for i in range(1,len(A)): d = F(A[i],x,B[i]) if min > d: min = d return min From jeanmichel at sequans.com Mon Jul 15 06:13:44 2013 From: jeanmichel at sequans.com (Jean-Michel Pichavant) Date: Mon, 15 Jul 2013 12:13:44 +0200 (CEST) Subject: Python - remote object protocols and security In-Reply-To: <59272663.8296927.1373882548643.JavaMail.root@sequans.com> Message-ID: <890384520.8302247.1373883224109.JavaMail.root@sequans.com> Hello everyone, I'd like to exchange some simple python objects over the internet. I initially planned to use Pyro, after reading http://pythonhosted.org/Pyro4/security.html I'm still puzzled. I don't mind encrypting data, if someone wants to sniff what I'm sending, he's welcome. What I think I need to care about, is malicious code injections. Because both client/server will be in python, would someone capable of executing code by changing one side python source ? How do I prevent this and still provide the source to everyone ? JM -- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeanmichel at sequans.com Mon Jul 15 06:20:06 2013 From: jeanmichel at sequans.com (Jean-Michel Pichavant) Date: Mon, 15 Jul 2013 12:20:06 +0200 (CEST) Subject: Python - remote object protocols and security In-Reply-To: <890384520.8302247.1373883224109.JavaMail.root@sequans.com> Message-ID: <900292775.8323132.1373883606595.JavaMail.root@sequans.com> In text format... sorry for my previous html post Hello everyone, I'd like to exchange some simple python objects over the internet. I initially planned to use Pyro, after reading http://pythonhosted.org/Pyro4/security.html I'm still puzzled. I don't mind encrypting data, if someone wants to sniff what I'm sending, he's welcome. What I think I need to care about, is malicious code injections. Because both client/server will be in python, would someone capable of executing code by changing one side python source ? How do I prevent this and still provide the source to everyone ? JM -- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. From davea at davea.name Mon Jul 15 07:17:42 2013 From: davea at davea.name (Dave Angel) Date: Mon, 15 Jul 2013 07:17:42 -0400 Subject: Python - remote object protocols and security In-Reply-To: <900292775.8323132.1373883606595.JavaMail.root@sequans.com> References: <890384520.8302247.1373883224109.JavaMail.root@sequans.com> <900292775.8323132.1373883606595.JavaMail.root@sequans.com> Message-ID: On 07/15/2013 06:20 AM, Jean-Michel Pichavant wrote: > In text format... sorry for my previous html post > > Hello everyone, > > I'd like to exchange some simple python objects over the internet. > I initially planned to use Pyro, after reading http://pythonhosted.org/Pyro4/security.html I'm still puzzled. > > I don't mind encrypting data, if someone wants to sniff what I'm sending, he's welcome. > I don't think the word you need there is "mind," but I get the idea. You don't care who reads the data being sent, but don't want anybody to be able to masquerade as somebody else, either by sending with invalid credentials or by intercepting and modifying legitimate traffic. What's needed for that is public/private key encryption, where the legitimate user uses his own private key to encode data, and you use their public key to decode it. Anybody can decode it, since the key is public, but anyone (especialy you) can be sure who sent it, since they presumably keep their private key private. > What I think I need to care about, is malicious code injections. Because both client/server will be in python, would someone capable of executing code by changing one side python source ? > Even if you have a friendly user sending data, you still need to guard against code injection because their system may have been compromised. And with code injection, you risk trashing not only their own data, but other people's and your own as well. In other words, encryption provides you no assurances. > How do I prevent this and still provide the source to everyone ? > Make sure your deserializing logic (on your own machine) is entirely under your control, and impervious to such attacks. In general, the more types that can be encoded, the more likely it's vulnerable. So authors of such libraries have two conflicting goals. I can't tell you if pyro, or any other particular one is safe. But if you decide to roll your own, which is implied when you say you'll be publishing your source, then keep it dirt-simple. Have a very specific and finite set of classes which you'll handle, and write rigorous code to make sure the types are limited to those. For example, you could restrict yourself to lists of strings, or lists of strings and floats, and you have only three decoders to write. Note that DOS attacks are possible whatever encoding scheme you have. Make sure that self-references within the data are well-defined (or impossible), and put limits on size per transaction, and transactions per minute per legitimate user. -- DaveA From jeanmichel at sequans.com Mon Jul 15 08:26:27 2013 From: jeanmichel at sequans.com (Jean-Michel Pichavant) Date: Mon, 15 Jul 2013 14:26:27 +0200 (CEST) Subject: Python - remote object protocols and security In-Reply-To: Message-ID: <142271145.8401649.1373891187398.JavaMail.root@sequans.com> ----- Original Message ----- > > I don't mind encrypting data, if someone wants to sniff what I'm > > sending, he's welcome. > > > > I don't think the word you need there is "mind," but I get the idea. You're right, I wanted to state actually the opposite, I don't want to encrypt data because I don't care if someone sniffs it. It's pretty meaningless and doesn't include credentials. Basically, I need to transfer numbers (int). Possibly dictionaries like {string: int} in order to structure things a little bit. I don't think I need a credential system neither, cause if someone is sending me crap with the wrong identity it will only mess my statistics, this is pretty harmless. > Even if you have a friendly user sending data, you still need to > guard > against code injection because their system may have been > compromised. That is definitively something I'm trying to avoid. > Make sure your deserializing logic (on your own machine) is entirely > under your control, and impervious to such attacks. In general, the > more types that can be encoded, the more likely it's vulnerable. So > authors of such libraries have two conflicting goals. If I understand correctly any available remote protocols are pretty much of the chart. Since I'm planning to send only int and strings I think I'll follow your advice of serializing/deserializing myself. > DaveA thanks, Jean-Michel -- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. From rosuav at gmail.com Mon Jul 15 08:30:52 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 15 Jul 2013 22:30:52 +1000 Subject: Python - remote object protocols and security In-Reply-To: <142271145.8401649.1373891187398.JavaMail.root@sequans.com> References: <142271145.8401649.1373891187398.JavaMail.root@sequans.com> Message-ID: On Mon, Jul 15, 2013 at 10:26 PM, Jean-Michel Pichavant wrote: > Basically, I need to transfer numbers (int). Possibly dictionaries like {string: int} in order to structure things a little bit. I strongly recommend JSON, then. It's a well-known system, it's compact, it's secure, and Python comes with a json module. ChrisA From davea at davea.name Mon Jul 15 08:45:48 2013 From: davea at davea.name (Dave Angel) Date: Mon, 15 Jul 2013 08:45:48 -0400 Subject: Python - remote object protocols and security In-Reply-To: References: <142271145.8401649.1373891187398.JavaMail.root@sequans.com> Message-ID: On 07/15/2013 08:30 AM, Chris Angelico wrote: > On Mon, Jul 15, 2013 at 10:26 PM, Jean-Michel Pichavant > wrote: >> Basically, I need to transfer numbers (int). Possibly dictionaries like {string: int} in order to structure things a little bit. > > I strongly recommend JSON, then. It's a well-known system, it's > compact, it's secure, and Python comes with a json module. > And presumably has been tested against injection attacks (implied by your use of 'secure.') JM: That's the flip side. If you CAN find some open-source that exactly meets your needs, it presumably has had lots of eyes on it to spot the little bugs that are likely to pop up in any new implementation. There's a vast grey area between 1) so simple it's safer to do it myself and 2) so complex the open-source version must have bugs, so I'd better do it myself. in between, you use the open-source code or library. But this is why I always start by trying to narrow the choice of what you *need*. ...the only secure system is one physically contained in a room with a padlock, and with a guard. And only if the guard is yourself... -- DaveA From rosuav at gmail.com Mon Jul 15 08:57:55 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 15 Jul 2013 22:57:55 +1000 Subject: Python - remote object protocols and security In-Reply-To: References: <142271145.8401649.1373891187398.JavaMail.root@sequans.com> Message-ID: On Mon, Jul 15, 2013 at 10:45 PM, Dave Angel wrote: > On 07/15/2013 08:30 AM, Chris Angelico wrote: >> >> On Mon, Jul 15, 2013 at 10:26 PM, Jean-Michel Pichavant >> wrote: >>> >>> Basically, I need to transfer numbers (int). Possibly dictionaries like >>> {string: int} in order to structure things a little bit. >> >> >> I strongly recommend JSON, then. It's a well-known system, it's >> compact, it's secure, and Python comes with a json module. >> > > And presumably has been tested against injection attacks (implied by your > use of 'secure.') Talking about the json module? I would expect it has, given that JSON is frequently used in untrusted contexts (unlike, for instance, pickle, which is specifically *not* for untrusted data). But even if it has some sort of exploit, that would be a bug to be fixed in the library; it would be an issue that affects many other users, and someone will likely report it and get it fixed in the next point release. But what I meant was that the protocol itself is designed with security restrictions in mind. It's designed not to fetch additional content from the network (as XML can), nor to retrieve named objects from the environment (as pickle can), etc, etc. That doesn't mean it's perfect, but it's a lot easier to make a secure protocol based on JSON than one based on pickle, simply because starting with the basics and adding safely is easier than starting with massive power and then protecting around issues. ChrisA From burak.arslan at arskom.com.tr Mon Jul 15 08:50:10 2013 From: burak.arslan at arskom.com.tr (Burak Arslan) Date: Mon, 15 Jul 2013 13:50:10 +0100 Subject: Python - remote object protocols and security In-Reply-To: References: <142271145.8401649.1373891187398.JavaMail.root@sequans.com> Message-ID: <51E3F002.1050103@arskom.com.tr> Hi, On 07/15/13 13:30, Chris Angelico wrote: > On Mon, Jul 15, 2013 at 10:26 PM, Jean-Michel Pichavant > wrote: >> Basically, I need to transfer numbers (int). Possibly dictionaries like {string: int} in order to structure things a little bit. > I strongly recommend JSON, then. It's a well-known system, it's > compact, it's secure, and Python comes with a json module. > Especially for numbers, MessagePack is more efficient. Its API is identical to Json, so it's almost a drop-in replacement. A project that I've been working on, Spyne, is designed to implement public RPC services. It supports both Json and MessagePack. Here's the json example: http://spyne.io/#inprot=JsonDocument&outprot=JsonDocument&s=rpc&tpt=WsgiApplication&validator=true If you choose to use MessagePack, you must HTTP POST the MessagePack document the same way you'd POST the json document. Best regards, Burak From irmen.NOSPAM at xs4all.nl Mon Jul 15 12:57:32 2013 From: irmen.NOSPAM at xs4all.nl (Irmen de Jong) Date: Mon, 15 Jul 2013 18:57:32 +0200 Subject: Python - remote object protocols and security In-Reply-To: References: <890384520.8302247.1373883224109.JavaMail.root@sequans.com> <900292775.8323132.1373883606595.JavaMail.root@sequans.com> Message-ID: <51e429fb$0$15994$e4fe514c@news.xs4all.nl> On 15-7-2013 13:17, Dave Angel wrote: > On 07/15/2013 06:20 AM, Jean-Michel Pichavant wrote: >> In text format... sorry for my previous html post >> >> Hello everyone, >> >> I'd like to exchange some simple python objects over the internet. >> I initially planned to use Pyro, after reading >> http://pythonhosted.org/Pyro4/security.html I'm still puzzled. Hi, Pyro's author here. I agree that this chapter of the manual can use some cleanup. Is there anything in particular that you are puzzled about at this time? >> >> I don't mind encrypting data, if someone wants to sniff what I'm sending, he's welcome. >> I don't quite understand what you're saying in this sentence: is it okay if someone eavesdrops on your unencrypted data stream? >> What I think I need to care about, is malicious code injections. Because both >> client/server will be in python, would someone capable of executing code by changing >> one side python source ? Pyro since version 4.20 uses a serialization format that is safe against arbitrary code execution: https://pypi.python.org/pypi/serpent That format only encodes and decodes Python literal expressions, and no arbitrary objects are instantiated. You can also tell Pyro to use JSON (or marshal even), both of which should be impervious to this type of attack/vulnerability as well. The problem is with older Pyro versions, which allowed only Pickle to be used as serialization format. It is pickle that causes the remote code execution vulnerability. So as long as you don't explicitly tell Pyro (4.20+) to use pickle (a configuration switch), you should be safe. > I can't tell you if pyro, or any other particular one is safe. Pyro should be, since version 4.20 and provided you don't tell it to use pickle. See above. > Note that DOS attacks are possible whatever encoding scheme you have. Make sure that > self-references within the data are well-defined (or impossible), and put limits on size > per transaction, and transactions per minute per legitimate user. Pyro doesn't provide anything by itself to protect against this. Cheers Irmen de Jong From irmen.NOSPAM at xs4all.nl Mon Jul 15 13:05:45 2013 From: irmen.NOSPAM at xs4all.nl (Irmen de Jong) Date: Mon, 15 Jul 2013 19:05:45 +0200 Subject: Python - remote object protocols and security In-Reply-To: <51e429fb$0$15994$e4fe514c@news.xs4all.nl> References: <890384520.8302247.1373883224109.JavaMail.root@sequans.com> <900292775.8323132.1373883606595.JavaMail.root@sequans.com> <51e429fb$0$15994$e4fe514c@news.xs4all.nl> Message-ID: <51e42be7$0$15884$e4fe514c@news.xs4all.nl> On 15-7-2013 18:57, Irmen de Jong wrote: >> Note that DOS attacks are possible whatever encoding scheme you have. Make sure that >> self-references within the data are well-defined (or impossible), and put limits on size >> per transaction, and transactions per minute per legitimate user. > > Pyro doesn't provide anything by itself to protect against this. I'm sorry to follow up on myself, but there is actually one thing: Pyro's choice of serializers (except pickle, again) don't allow self-references. So that type of DOS attack (infinite recursion) is ruled out. Irmen From jeanmichel at sequans.com Mon Jul 15 13:31:17 2013 From: jeanmichel at sequans.com (Jean-Michel Pichavant) Date: Mon, 15 Jul 2013 19:31:17 +0200 (CEST) Subject: Python - remote object protocols and security In-Reply-To: <51e429fb$0$15994$e4fe514c@news.xs4all.nl> Message-ID: <1597960330.8733256.1373909477065.JavaMail.root@sequans.com> ----- Original Message ----- > On 15-7-2013 13:17, Dave Angel wrote: > > On 07/15/2013 06:20 AM, Jean-Michel Pichavant wrote: > >> In text format... sorry for my previous html post > >> > >> Hello everyone, > >> > >> I'd like to exchange some simple python objects over the internet. > >> I initially planned to use Pyro, after reading > >> http://pythonhosted.org/Pyro4/security.html I'm still puzzled. > > Hi, Pyro's author here. > I agree that this chapter of the manual can use some cleanup. > Is there anything in particular that you are puzzled about at this > time? Nothing wrong with the manual, just my poor knowledge of security issues. > >> > >> I don't mind encrypting data, if someone wants to sniff what I'm > >> sending, he's welcome. > >> > > I don't quite understand what you're saying in this sentence: is it > okay if someone > eavesdrops on your unencrypted data stream? It's okay is someone eavesdrops, my English is as bad as my net code. > Pyro since version 4.20 uses a serialization format that is safe > against arbitrary code > > > Cheers > Irmen de Jong Thanks for the clarifications (and writing Pyro), I'll make sure I'll be using 4.20+. JM -- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. From rosuav at gmail.com Mon Jul 15 07:18:41 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 15 Jul 2013 21:18:41 +1000 Subject: Python - remote object protocols and security In-Reply-To: <890384520.8302247.1373883224109.JavaMail.root@sequans.com> References: <59272663.8296927.1373882548643.JavaMail.root@sequans.com> <890384520.8302247.1373883224109.JavaMail.root@sequans.com> Message-ID: On Mon, Jul 15, 2013 at 8:13 PM, Jean-Michel Pichavant wrote: > I'd like to exchange some simple python objects over the internet. > I initially planned to use Pyro, after reading > http://pythonhosted.org/Pyro4/security.html I'm still puzzled. > > I don't mind encrypting data, if someone wants to sniff what I'm sending, > he's welcome. > > What I think I need to care about, is malicious code injections. Because > both client/server will be in python, would someone capable of executing > code by changing one side python source ? > > How do I prevent this and still provide the source to everyone ? How complicated are the objects you want to transmit? If they're just strings, integers, floats, and lists or dictionaries of the above, then you could use JSON instead; that's much safer, but (and because) it's majorly restricted. Sometimes it's worth warping your data structure slightly (eg use a dict and global functions instead of a custom object with methods) to improve security. ChrisA From jeanmichel at sequans.com Mon Jul 15 08:41:12 2013 From: jeanmichel at sequans.com (Jean-Michel Pichavant) Date: Mon, 15 Jul 2013 14:41:12 +0200 (CEST) Subject: Python - remote object protocols and security In-Reply-To: Message-ID: <595253102.8424684.1373892072113.JavaMail.root@sequans.com> ----- Original Message ----- > > What I think I need to care about, is malicious code injections. > > Because > > both client/server will be in python, would someone capable of > > executing > > code by changing one side python source ? > > > > How do I prevent this and still provide the source to everyone ? > > How complicated are the objects you want to transmit? If they're just > strings, integers, floats, and lists or dictionaries of the above, > then you could use JSON instead; that's much safer, but (and because) > it's majorly restricted. Sometimes it's worth warping your data > structure slightly (eg use a dict and global functions instead of a > custom object with methods) to improve security. > > ChrisA In the end just strings and Int. Dave seems to agree with you and JSON is the way to go. However, I don't want to write net code, I'm lazy and most importantly I'm so bad at it. So how would I send Json strings from one machine to a remote ? If I'm using http://code.google.com/p/jsonrpclib/, would it still be a Json safe way of sending strings and int ? JM -- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. From rosuav at gmail.com Mon Jul 15 08:51:48 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 15 Jul 2013 22:51:48 +1000 Subject: Python - remote object protocols and security In-Reply-To: <595253102.8424684.1373892072113.JavaMail.root@sequans.com> References: <595253102.8424684.1373892072113.JavaMail.root@sequans.com> Message-ID: On Mon, Jul 15, 2013 at 10:41 PM, Jean-Michel Pichavant wrote: > ----- Original Message ----- >> > What I think I need to care about, is malicious code injections. >> > Because >> > both client/server will be in python, would someone capable of >> > executing >> > code by changing one side python source ? >> > >> > How do I prevent this and still provide the source to everyone ? >> >> How complicated are the objects you want to transmit? If they're just >> strings, integers, floats, and lists or dictionaries of the above, >> then you could use JSON instead; that's much safer, but (and because) >> it's majorly restricted. Sometimes it's worth warping your data >> structure slightly (eg use a dict and global functions instead of a >> custom object with methods) to improve security. >> >> ChrisA > > In the end just strings and Int. > Dave seems to agree with you and JSON is the way to go. > > However, I don't want to write net code, I'm lazy and most importantly I'm so bad at it. > So how would I send Json strings from one machine to a remote ? > If I'm using http://code.google.com/p/jsonrpclib/, would it still be a Json safe way of sending strings and int ? To send JSON-encoded data, you: 1) Encode your data in JSON format and some character encoding (eg UTF-8) 2) Transmit the resulting stream of bytes over the network 3) Decode UTF-8 and then JSON Python provides all this functionality: >>> data = {"English":"Hello, world","Russian":"??????, ???"} >>> json.dumps(data).encode() b'{"English": "Hello, world", "Russian": "\\u041f\\u0440\\u0438\\u0432\\u0435\\u0442, \\u043c\\u0438\\u0440"}' which happens to look very much like the original input, though this is more coincidence than design. Note that you could leave the non-ASCII characters as they are, and transmit them as UTF-8 sequences: >>> json.dumps(data,ensure_ascii=False).encode() b'{"English": "Hello, world", "Russian": "\xd0\x9f\xd1\x80\xd0\xb8\xd0\xb2\xd0\xb5\xd1\x82, \xd0\xbc\xd0\xb8\xd1\x80"}' Take your pick, based on what you want to do at the other end. The second form is (obviously) a lot more compact than the first. Decoding is just as easy: >>> data=b'{"English": "Hello, world", "Russian": "\xd0\x9f\xd1\x80\xd0\xb8\xd0\xb2\xd0\xb5\xd1\x82, \xd0\xbc\xd0\xb8\xd1\x80"}' >>> json.loads(data.decode()) {'English': 'Hello, world', 'Russian': '??????, ???'} So the only bit you still need is: How do you transmit this across the network? Since it's now all just bytes, that's easy enough to do, eg with TCP. But that depends on the rest of your system, and is a quite separate question - and quite probably one you already have the answer to. ChrisA From burak.arslan at arskom.com.tr Mon Jul 15 11:42:07 2013 From: burak.arslan at arskom.com.tr (Burak Arslan) Date: Mon, 15 Jul 2013 16:42:07 +0100 Subject: Python - remote object protocols and security In-Reply-To: References: <595253102.8424684.1373892072113.JavaMail.root@sequans.com> Message-ID: <51E4184F.3080607@arskom.com.tr> On 07/15/13 13:51, Chris Angelico wrote: > So the only bit you still need is: How do you transmit this across the > network? Since it's now all just bytes, that's easy enough to do, eg > with TCP. But that depends on the rest of your system, and is a quite > separate question - and quite probably one you already have the answer > to. For Json, you need to have a way of delimiting messages -- to my knowledge, Python's json library does not support parsing streams. You can send the json document in the body of a Http POST, or a ZeroMQ message, or in a UDP datagram (if you can guarantee it fits inside one) or in a simple TCP-based encapsulation mechanism that e.g. prepends the length of the message to the document. e.g. '\x00\x00\x00\x07{"a":1}' As MessagePack already does this, you can send MessagePack documents via an ordinary TCP socket and easily recover them on the other side of the pipe. >>> import msgpack; from StringIO import StringIO >>> s = StringIO(msgpack.dumps({"a":1}) + msgpack.dumps({"b":2})) >>> for doc in msgpack.Unpacker(s): ... print doc ... {'a': 1} {'b': 2} This won't work with Json: >>> import json; from StringIO import StringIO >>> s = StringIO(json.dumps({"a":1}) + json.dumps({"b":2})) >>> for doc in json.load(s): # or whatever ??? ... print doc ... Traceback (most recent call last): File "", line 1, in File "/usr/lib64/python2.7/json/__init__.py", line 290, in load **kw) File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads return _default_decoder.decode(s) File "/usr/lib64/python2.7/json/decoder.py", line 368, in decode raise ValueError(errmsg("Extra data", s, end, len(s))) ValueError: Extra data: line 1 column 9 - line 1 column 17 (char 8 - 16) Note that this is a limitation of python's Json parser, not Json itself. There seems to be a json.scanner module that *sounds* like it provides this functionality, but I couldn't find any documentation about it. Alternatively, PyYaml can also parse streams. yaml.{dump,load}_all() provide pickle-like unsafe (de)serialization support and yaml.safe_{dump,load}_all provide msgpack-like safe-but-limited stream parsing support. also; On 07/15/13 13:57, Chris Angelico wrote: > But what I meant was that the [Json] protocol itself is designed with > security restrictions in mind. It's designed not to fetch additional > content from the network (as XML can), Can you explain how parsing XML can fetch data from the network? Best, Burak From rosuav at gmail.com Mon Jul 15 11:53:21 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 16 Jul 2013 01:53:21 +1000 Subject: Python - remote object protocols and security In-Reply-To: <51E4184F.3080607@arskom.com.tr> References: <595253102.8424684.1373892072113.JavaMail.root@sequans.com> <51E4184F.3080607@arskom.com.tr> Message-ID: On Tue, Jul 16, 2013 at 1:42 AM, Burak Arslan wrote: > On 07/15/13 13:57, Chris Angelico wrote: >> But what I meant was that the [Json] protocol itself is designed with >> security restrictions in mind. It's designed not to fetch additional >> content from the network (as XML can), > > Can you explain how parsing XML can fetch data from the network? I haven't looked into the details, but there was one among a list of exploits that was being discussed a few months ago; it involved XML schemas, I think, and quite a few generic XML parsers could be tricked into fetching arbitrary documents. Whether this could be used for anything more serious than a document-viewed receipt or a denial of service (via latency) I don't know, but if nothing else, it's a vector that JSON simply doesn't have. ChrisA From burak.arslan at arskom.com.tr Mon Jul 15 12:31:33 2013 From: burak.arslan at arskom.com.tr (Burak Arslan) Date: Mon, 15 Jul 2013 17:31:33 +0100 Subject: Python - remote object protocols and security In-Reply-To: References: <595253102.8424684.1373892072113.JavaMail.root@sequans.com> <51E4184F.3080607@arskom.com.tr> Message-ID: <51E423E5.2030303@arskom.com.tr> On 07/15/13 16:53, Chris Angelico wrote: > I haven't looked into the details, but there was one among a list of > exploits that was being discussed a few months ago; it involved XML > schemas, I think, and quite a few generic XML parsers could be tricked > into fetching arbitrary documents. Whether this could be used for > anything more serious than a document-viewed receipt or a denial of > service (via latency) I don't know, but if nothing else, it's a vector > that JSON simply doesn't have. ChrisA I must have missed that exploit report, can you provide a link? Parsing arbitrary xml documents and parsing xml schema documents and applying xml schema semantics to these documents are two very different operations. Xml schemas are not "tricked" into fetching arbitrary documents, xs:include and xs:import fetch external documents, it's a well-known feature. If you don't want this, you should ship all of the schema documents together and generate the schemas in a way to not include any external references. So I'm surprised this was presented as a security exploit. Json schemas also have similar functionality: http://json-schema.org/latest/json-schema-core.html#anchor30 """ if canonical dereferencing is used, the implementation will dereference this URI, and fetch the content at this URI; """ So I don't understand how you're so sure of yourself, but to me, it seems like Json schemas have the same attack vectors. Best regards, Burak From rosuav at gmail.com Mon Jul 15 12:41:09 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 16 Jul 2013 02:41:09 +1000 Subject: Python - remote object protocols and security In-Reply-To: <51E423E5.2030303@arskom.com.tr> References: <595253102.8424684.1373892072113.JavaMail.root@sequans.com> <51E4184F.3080607@arskom.com.tr> <51E423E5.2030303@arskom.com.tr> Message-ID: On Tue, Jul 16, 2013 at 2:31 AM, Burak Arslan wrote: > On 07/15/13 16:53, Chris Angelico wrote: >> I haven't looked into the details, but there was one among a list of >> exploits that was being discussed a few months ago; it involved XML >> schemas, I think, and quite a few generic XML parsers could be tricked >> into fetching arbitrary documents. Whether this could be used for >> anything more serious than a document-viewed receipt or a denial of >> service (via latency) I don't know, but if nothing else, it's a vector >> that JSON simply doesn't have. ChrisA > > I must have missed that exploit report, can you provide a link? > > Parsing arbitrary xml documents and parsing xml schema documents and > applying xml schema semantics to these documents are two very different > operations. I don't remember all the details; it isn't something I took particular note of, as I don't work much with XML. It was something involving either a schema declaration or a DTD or something of the sort, where normally no external lookup is required but there's an HTTP URL in there and it's possible to force that to be resolved. > Xml schemas are not "tricked" into fetching arbitrary documents, > xs:include and xs:import fetch external documents, it's a well-known > feature. If you don't want this, you should ship all of the schema > documents together and generate the schemas in a way to not include any > external references. So I'm surprised this was presented as a security > exploit. It was something that parsing a basic XML document could trigger, and in an environment where you wouldn't normally expect extra HTTP requests to be going out, hence "tricked". > Json schemas also have similar functionality: > http://json-schema.org/latest/json-schema-core.html#anchor30 > > """ > if canonical dereferencing is used, the implementation will dereference > this URI, and fetch the content at this URI; > """ > > So I don't understand how you're so sure of yourself, but to me, it > seems like Json schemas have the same attack vectors. Yes, but normal JSON data doesn't include schema references. Normal XML data can and often does. ChrisA From philip.w.mcadams at intel.com Mon Jul 15 09:11:09 2013 From: philip.w.mcadams at intel.com (Mcadams, Philip W) Date: Mon, 15 Jul 2013 13:11:09 +0000 Subject: Question regarding building Python Windows installer In-Reply-To: <7B6560631CF08A4B8189A9CB669768F4228E86B7@FMSMSX101.amr.corp.intel.com> References: <7B6560631CF08A4B8189A9CB669768F4228E86B7@FMSMSX101.amr.corp.intel.com> Message-ID: <7B6560631CF08A4B8189A9CB669768F4228EB578@FMSMSX101.amr.corp.intel.com> I'm attempting to create a Python 64-bit Windows Installer. Following the instructions here: http://docs.python.org/2/distutils/builtdist.html I'm to navigate to my Python folder and user command: python setup.py build --plat-name=win-amd64 bdist_wininst I get error: COMPILED_WTH_PYDEBUG = ('-with-pydebug' in sysconfig.get_config_var("CONFIG_ARGS")) TypeError: argument of type 'NoneType' is not iterable I also have tried: setup.py build --plat-name=win-amd64 bdist_wininst and get error: File "setup.py", line 263 Print "%-*s %-*s %-*s" % (longest, e, longet, f, SyntaxError: invalid syntax I followed the instructions here: http://docs.python.org/devguide/setup.html to create a PC build for Windows which allows me to run a Python prompt. Now I need to create a Windows Installer to install this Python on a Windows Server 2008 R2 box. To explain why I'm attempting to do this instead of just using the Windows Installer provided by Python: I needed to modify a _ssl.c file in the Python source code to deal a Mercurial that I'm trying to resolve. Any help on why I'm hitting these errors would be appreciated. Thank you. Philip McAdams Systems Administrator - NVM Solutions Group Systems Engineering Apps & Infrastructure Desk: (916) 377-6156 Cell: (916) 534-0092 Pole: FM3-1-D7 -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Mon Jul 15 10:27:39 2013 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 15 Jul 2013 15:27:39 +0100 Subject: Question regarding building Python Windows installer In-Reply-To: <7B6560631CF08A4B8189A9CB669768F4228EB578@FMSMSX101.amr.corp.intel.com> References: <7B6560631CF08A4B8189A9CB669768F4228E86B7@FMSMSX101.amr.corp.intel.com> <7B6560631CF08A4B8189A9CB669768F4228EB578@FMSMSX101.amr.corp.intel.com> Message-ID: <51E406DB.7080608@mrabarnett.plus.com> On 15/07/2013 14:11, Mcadams, Philip W wrote: > I?m attempting to create a Python 64-bit Windows Installer. Following > the instructions here: http://docs.python.org/2/distutils/builtdist.html > I?m to navigate to my Python folder and user command: > > python setup.py build --plat-name=win-amd64 bdist_wininst > > I get error: COMPILED_WTH_PYDEBUG = (??with-pydebug? in > sysconfig.get_config_var(?CONFIG_ARGS?)) > > TypeError: argument of type ?NoneType? is not iterable > > I also have tried: > > setup.py build --plat-name=win-amd64 bdist_wininst > > and get error: > > File ?setup.py?, line 263 > Print ?%-*s %-*s %-*s? % (longest, e, longet, f, > > SyntaxError: invalid syntax > Does the line really start with "Print" (initial capital letter)? Also, are you using Python 2 or Python 3? From the link above it looks like Python 2. > I followed the instructions here: > http://docs.python.org/devguide/setup.html to create a PC build for > Windows which allows me to run a Python prompt. Now I need to create a > Windows Installer to install this Python on a Windows Server 2008 R2 box. > > To explain why I?m attempting to do this instead of just using the > Windows Installer provided by Python: > > I needed to modify a _ssl.c file in the Python source code to deal a > Mercurial that I?m trying to resolve. > > Any help on why I?m hitting these errors would be appreciated. > From zachary.ware+pylist at gmail.com Mon Jul 15 11:46:41 2013 From: zachary.ware+pylist at gmail.com (Zachary Ware) Date: Mon, 15 Jul 2013 10:46:41 -0500 Subject: Question regarding building Python Windows installer In-Reply-To: <7B6560631CF08A4B8189A9CB669768F4228EB578@FMSMSX101.amr.corp.intel.com> References: <7B6560631CF08A4B8189A9CB669768F4228E86B7@FMSMSX101.amr.corp.intel.com> <7B6560631CF08A4B8189A9CB669768F4228EB578@FMSMSX101.amr.corp.intel.com> Message-ID: On Mon, Jul 15, 2013 at 8:11 AM, Mcadams, Philip W wrote: > I?m attempting to create a Python 64-bit Windows Installer. Following the > instructions here: http://docs.python.org/2/distutils/builtdist.html I?m to > navigate to my Python folder and user command: > > > > python setup.py build --plat-name=win-amd64 bdist_wininst > > > > I get error: COMPILED_WTH_PYDEBUG = (??with-pydebug? in > sysconfig.get_config_var(?CONFIG_ARGS?)) > > TypeError: argument of type ?NoneType? is not iterable > > > > I also have tried: > > > > setup.py build --plat-name=win-amd64 bdist_wininst > > > > and get error: > > > > File ?setup.py?, line 263 > > Print ?%-*s %-*s %-*s? % (longest, e, longet, f, > > > > SyntaxError: invalid syntax > > > > I followed the instructions here: http://docs.python.org/devguide/setup.html > to create a PC build for Windows which allows me to run a Python prompt. > Now I need to create a Windows Installer to install this Python on a Windows > Server 2008 R2 box. > > > > To explain why I?m attempting to do this instead of just using the Windows > Installer provided by Python: > > > > I needed to modify a _ssl.c file in the Python source code to deal a > Mercurial that I?m trying to resolve. > > > > Any help on why I?m hitting these errors would be appreciated. > > > > Thank you. > > Most of your problem is that setup.py is just for the extension modules of the standard library, not the interpreter and all. The rest of your problem is that setup.py is really just not Windows-friendly, relying on some Makefile vars and Modules/Setup. Do you really need to install it? If you're still in the testing phase, would it be enough to just copy the source tree (with compiled interpreter) to the box you need it on? You can still use Mercurial with it, just stick the hg modules somewhere on PYTHONPATH. If you really do need an installer, I would suggest trying out Tools/buildbot/buildmsi.bat and see if you can get it to work for you. I will warn you, buildmsi.bat is not well maintained and it may take quite a bit of effort to make it work. HTH, -- Zach From philip.w.mcadams at intel.com Mon Jul 15 14:02:08 2013 From: philip.w.mcadams at intel.com (Mcadams, Philip W) Date: Mon, 15 Jul 2013 18:02:08 +0000 Subject: Question regarding building Python Windows installer In-Reply-To: References: <7B6560631CF08A4B8189A9CB669768F4228E86B7@FMSMSX101.amr.corp.intel.com> <7B6560631CF08A4B8189A9CB669768F4228EB578@FMSMSX101.amr.corp.intel.com> Message-ID: <7B6560631CF08A4B8189A9CB669768F4228EC7B2@FMSMSX101.amr.corp.intel.com> Thanks for the reply Zachery. We have decided to just use another solution. Out of curiosity though I wanted to clarification on your statement: just stick the hg modules somewhere on PYTHONPATH. Are you saying that I would just map hg modules i.e.: C:\Users\pwmcadam\Downloads\Python-2.7.4\Python-2.7.4\Modules to my environment variables in Windows. Wasn't exactly following your comment. Thank you. Philip McAdams Systems Administrator - NVM Solutions Group Systems Engineering Apps & Infrastructure Desk: (916) 377-6156 Cell: (916) 534-0092 Pole: FM3-1-D7 -----Original Message----- From: zachary.ware at gmail.com [mailto:zachary.ware at gmail.com] On Behalf Of Zachary Ware Sent: Monday, July 15, 2013 8:47 AM To: Mcadams, Philip W Cc: python-list at python.org Subject: Re: Question regarding building Python Windows installer On Mon, Jul 15, 2013 at 8:11 AM, Mcadams, Philip W wrote: > I?m attempting to create a Python 64-bit Windows Installer. Following > the instructions here: > http://docs.python.org/2/distutils/builtdist.html I?m to navigate to my Python folder and user command: > > > > python setup.py build --plat-name=win-amd64 bdist_wininst > > > > I get error: COMPILED_WTH_PYDEBUG = (??with-pydebug? in > sysconfig.get_config_var(?CONFIG_ARGS?)) > > TypeError: argument of type ?NoneType? is not iterable > > > > I also have tried: > > > > setup.py build --plat-name=win-amd64 bdist_wininst > > > > and get error: > > > > File ?setup.py?, line 263 > > Print ?%-*s %-*s %-*s? % (longest, e, longet, f, > > > > SyntaxError: invalid syntax > > > > I followed the instructions here: > http://docs.python.org/devguide/setup.html > to create a PC build for Windows which allows me to run a Python prompt. > Now I need to create a Windows Installer to install this Python on a > Windows Server 2008 R2 box. > > > > To explain why I?m attempting to do this instead of just using the > Windows Installer provided by Python: > > > > I needed to modify a _ssl.c file in the Python source code to deal a > Mercurial that I?m trying to resolve. > > > > Any help on why I?m hitting these errors would be appreciated. > > > > Thank you. > > Most of your problem is that setup.py is just for the extension modules of the standard library, not the interpreter and all. The rest of your problem is that setup.py is really just not Windows-friendly, relying on some Makefile vars and Modules/Setup. Do you really need to install it? If you're still in the testing phase, would it be enough to just copy the source tree (with compiled interpreter) to the box you need it on? You can still use Mercurial with it, just stick the hg modules somewhere on PYTHONPATH. If you really do need an installer, I would suggest trying out Tools/buildbot/buildmsi.bat and see if you can get it to work for you. I will warn you, buildmsi.bat is not well maintained and it may take quite a bit of effort to make it work. HTH, -- Zach From zachary.ware+pylist at gmail.com Mon Jul 15 15:06:39 2013 From: zachary.ware+pylist at gmail.com (Zachary Ware) Date: Mon, 15 Jul 2013 14:06:39 -0500 Subject: Question regarding building Python Windows installer In-Reply-To: <7B6560631CF08A4B8189A9CB669768F4228EC7B2@FMSMSX101.amr.corp.intel.com> References: <7B6560631CF08A4B8189A9CB669768F4228E86B7@FMSMSX101.amr.corp.intel.com> <7B6560631CF08A4B8189A9CB669768F4228EB578@FMSMSX101.amr.corp.intel.com> <7B6560631CF08A4B8189A9CB669768F4228EC7B2@FMSMSX101.amr.corp.intel.com> Message-ID: On Mon, Jul 15, 2013 at 1:02 PM, Mcadams, Philip W wrote: > Thanks for the reply Zachery. We have decided to just use another solution. Out of curiosity though I wanted to clarification on your statement: > > just stick the hg modules somewhere on PYTHONPATH. > > Are you saying that I would just map hg modules i.e.: C:\Users\pwmcadam\Downloads\Python-2.7.4\Python-2.7.4\Modules to my environment variables in Windows. > > Wasn't exactly following your comment. > Did I understand correctly that in your initial message you said you needed an installer to try to resolve a Mercurial issue? If not, ignore that whole sentence of mine :) Otherwise, I meant that you can just copy the mercurial and hgext directories from \Lib\site-packages to \Lib\site-packages, which would add that dir to the start of sys.path. I did somewhat misuse 'PYTHONPATH' in my earlier message. -- Zach From philip.w.mcadams at intel.com Mon Jul 15 15:27:28 2013 From: philip.w.mcadams at intel.com (Mcadams, Philip W) Date: Mon, 15 Jul 2013 19:27:28 +0000 Subject: Question regarding building Python Windows installer In-Reply-To: References: <7B6560631CF08A4B8189A9CB669768F4228E86B7@FMSMSX101.amr.corp.intel.com> <7B6560631CF08A4B8189A9CB669768F4228EB578@FMSMSX101.amr.corp.intel.com> <7B6560631CF08A4B8189A9CB669768F4228EC7B2@FMSMSX101.amr.corp.intel.com> Message-ID: <7B6560631CF08A4B8189A9CB669768F4228ECF7B@FMSMSX101.amr.corp.intel.com> Yes. My goal was to create the installer to put the modified python on my Mercurial server. So I could have effectively copied over the \Lib\site-packages on the server? What I was trying to resolve was the issue with large Mercurial pushes. I instead am using the IIS Crypto tool to resolve the issue. I'd found a link that stated that modification to the _ssl.c module in Python could also fix the issue but that the python source would to be recompiled. Since we are going with IISCrypto the Python change is no longer needed. But out curiosity, and in case I ran into a situation where did indeed need to make a fix to Python I've wondered what's the best way to do that. Hopefully this gives you a little insight on what I'm trying to do. Thanks for your replies. Thank you. Philip McAdams Systems Administrator - NVM Solutions Group Systems Engineering Apps & Infrastructure Desk: (916) 377-6156 Cell: (916) 534-0092 Pole: FM3-1-D7 -----Original Message----- From: zachary.ware at gmail.com [mailto:zachary.ware at gmail.com] On Behalf Of Zachary Ware Sent: Monday, July 15, 2013 12:07 PM To: Mcadams, Philip W Cc: python-list at python.org Subject: Re: Question regarding building Python Windows installer On Mon, Jul 15, 2013 at 1:02 PM, Mcadams, Philip W wrote: > Thanks for the reply Zachery. We have decided to just use another solution. Out of curiosity though I wanted to clarification on your statement: > > just stick the hg modules somewhere on PYTHONPATH. > > Are you saying that I would just map hg modules i.e.: C:\Users\pwmcadam\Downloads\Python-2.7.4\Python-2.7.4\Modules to my environment variables in Windows. > > Wasn't exactly following your comment. > Did I understand correctly that in your initial message you said you needed an installer to try to resolve a Mercurial issue? If not, ignore that whole sentence of mine :) Otherwise, I meant that you can just copy the mercurial and hgext directories from \Lib\site-packages to \Lib\site-packages, which would add that dir to the start of sys.path. I did somewhat misuse 'PYTHONPATH' in my earlier message. -- Zach From zachary.ware+pylist at gmail.com Mon Jul 15 16:51:05 2013 From: zachary.ware+pylist at gmail.com (Zachary Ware) Date: Mon, 15 Jul 2013 15:51:05 -0500 Subject: Question regarding building Python Windows installer In-Reply-To: <7B6560631CF08A4B8189A9CB669768F4228ECF7B@FMSMSX101.amr.corp.intel.com> References: <7B6560631CF08A4B8189A9CB669768F4228E86B7@FMSMSX101.amr.corp.intel.com> <7B6560631CF08A4B8189A9CB669768F4228EB578@FMSMSX101.amr.corp.intel.com> <7B6560631CF08A4B8189A9CB669768F4228EC7B2@FMSMSX101.amr.corp.intel.com> <7B6560631CF08A4B8189A9CB669768F4228ECF7B@FMSMSX101.amr.corp.intel.com> Message-ID: (Side note: Please avoid top-posting in future. Bottom-posting keeps context more clearly) On Mon, Jul 15, 2013 at 2:27 PM, Mcadams, Philip W wrote: > Yes. My goal was to create the installer to put the modified python on my Mercurial server. So I could have effectively copied over the \Lib\site-packages on the server? What I was trying to resolve was the issue with large Mercurial pushes. I instead am using the IIS Crypto tool to resolve the issue. I'd found a link that stated that modification to the _ssl.c module in Python could also fix the issue but that the python source would to be recompiled. Since we are going with IISCrypto the Python change is no longer needed. But out curiosity, and in case I ran into a situation where did indeed need to make a fix to Python I've wondered what's the best way to do that. Hopefully this gives you a little insight on what I'm trying to do. Thanks for your replies. > Hmmm, not quite. \Lib\site-packages would be empty. I meant the other way around, copying the installed site-packages dir into the source tree to use mercurial from the source tree. I think you'd also have to copy hg and hg.bat from \Scripts as well, though. You might have been able to get away with just copying the newly compiled _ssl.pyd from your source tree to \DLLs, but I can't guarantee that. I think the solution you've gone for is a much better solution in the long run, though. Building your own Python (on Windows) should probably be a last resort. -- Zach From guxiaobo1982 at qq.com Mon Jul 15 09:47:17 2013 From: guxiaobo1982 at qq.com (=?ISO-8859-1?B?Z3V4aWFvYm8xOTgy?=) Date: Mon, 15 Jul 2013 21:47:17 +0800 Subject: How to build python with shared libraries. Message-ID: Hi, I am not so familiar with Python, I just want to use the multicorn external data wrapper and plpythonu2 language with PostgreSQL, my question is which option to specify when build Python 3.3 with shared libraries from source. Regards, Xiaobo Gu -------------- next part -------------- An HTML attachment was scrubbed... URL: From sol2ray at gmail.com Mon Jul 15 11:41:59 2013 From: sol2ray at gmail.com (Sol Toure) Date: Mon, 15 Jul 2013 11:41:59 -0400 Subject: [ANN]:JSONStream Message-ID: I was trying to process a large file containing a number of distinct JSON object as a stream, but I couldn't find anything readily available to that. (maybe I didn't search hard enough) So I came up with this: https://github.com/qrtz/JSONStream I hope you find it useful too. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cce at clarkevans.com Wed Jul 17 11:51:40 2013 From: cce at clarkevans.com (Clark C. Evans) Date: Wed, 17 Jul 2013 11:51:40 -0400 Subject: [ANN]:JSONStream In-Reply-To: References: Message-ID: <1374076300.20300.140661256765729.2467B0A8@webmail.messagingengine.com> Looks interesting. In YAML we used three dashes as the "stream separator". So already a YAML processor could handle a JSON stream ... >>> for doc in yaml.load_all(""" ... --- {"one": "value"} ... --- {"two": "another"} ... --- ... {"three": "a third item in the stream", ... "with": "more data"} ... """): ... print doc ... {'one': 'value'} {'two': 'another'} {'with': 'more data', 'three': 'a third item in the stream'} -------------- next part -------------- An HTML attachment was scrubbed... URL: From sol2ray at gmail.com Wed Jul 17 15:35:58 2013 From: sol2ray at gmail.com (Sol Toure) Date: Wed, 17 Jul 2013 15:35:58 -0400 Subject: [ANN]:JSONStream In-Reply-To: <1374076300.20300.140661256765729.2467B0A8@webmail.messagingengine.com> References: <1374076300.20300.140661256765729.2467B0A8@webmail.messagingengine.com> Message-ID: I didn't look into using YAML processor. Also that would have required pre-processing the data to add the separators. With this method you don't need the separators. You can have 0 or more white space between objects: for obj in JSONStream(StringIO('''{"one":1}{"two":2} {"three":3} 4 {"five": 5}''')): print(obj) {"one":1} {"two":2} {"three":3} 4 {"five":5} It solved my problem, so I thought someone might find it useful. On Wed, Jul 17, 2013 at 11:51 AM, Clark C. Evans wrote: > ** > Looks interesting. In YAML we used three dashes as the "stream separator". > So already a YAML processor could handle a JSON stream ... > > >>> for doc in yaml.load_all(""" > ... --- {"one": "value"} > ... --- {"two": "another"} > ... --- > ... {"three": "a third item in the stream", > ... "with": "more data"} > ... """): > ... print doc > ... > {'one': 'value'} > {'two': 'another'} > {'with': 'more data', 'three': 'a third item in the stream'} > > > > -- > http://mail.python.org/mailman/listinfo/python-list > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fabiosantosart at gmail.com Wed Jul 17 15:45:07 2013 From: fabiosantosart at gmail.com (=?ISO-8859-1?Q?F=E1bio_Santos?=) Date: Wed, 17 Jul 2013 20:45:07 +0100 Subject: [ANN]:JSONStream In-Reply-To: References: <1374076300.20300.140661256765729.2467B0A8@webmail.messagingengine.com> Message-ID: On 17 Jul 2013 20:40, "Sol Toure" wrote: > > I didn't look into using YAML processor. > Also that would have required pre-processing the data to add the separators. > With this method you don't need the separators. You can have 0 or more white space between objects: > > for obj in JSONStream(StringIO('''{"one":1}{"two":2} {"three":3} 4 {"five": 5}''')): > print(obj) > > {"one":1} > {"two":2} > {"three":3} > 4 > {"five":5} > > It solved my problem, so I thought someone might find it useful. > > > > On Wed, Jul 17, 2013 at 11:51 AM, Clark C. Evans wrote: >> >> Looks interesting. In YAML we used three dashes as the "stream separator". >> So already a YAML processor could handle a JSON stream ... >> >> >>> for doc in yaml.load_all(""" >> ... --- {"one": "value"} >> ... --- {"two": "another"} >> ... --- >> ... {"three": "a third item in the stream", >> ... "with": "more data"} >> ... """): >> ... print doc >> ... >> {'one': 'value'} >> {'two': 'another'} >> {'with': 'more data', 'three': 'a third item in the stream'} I think this could be useful for the twitter stream api. It feeds \n-separated json objects over time. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tdhfwh at nottheoilrig.com Mon Jul 15 11:50:52 2013 From: tdhfwh at nottheoilrig.com (Jack Bates) Date: Mon, 15 Jul 2013 08:50:52 -0700 Subject: Is this a bug? Message-ID: <51E41A5C.7060903@nottheoilrig.com> Hello, Is the following code supposed to be an UnboundLocalError? Currently it assigns the value 'bar' to the attribute baz.foo foo = 'bar' class baz: foo = foo From rosuav at gmail.com Mon Jul 15 12:05:59 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 16 Jul 2013 02:05:59 +1000 Subject: Is this a bug? In-Reply-To: <51E41A5C.7060903@nottheoilrig.com> References: <51E41A5C.7060903@nottheoilrig.com> Message-ID: On Tue, Jul 16, 2013 at 1:50 AM, Jack Bates wrote: > Hello, > > Is the following code supposed to be an UnboundLocalError? > Currently it assigns the value 'bar' to the attribute baz.foo > > foo = 'bar' > class baz: > foo = foo > -- > http://mail.python.org/mailman/listinfo/python-list Unless you're creating that class inside a function, it would be NameError, not UnboundLocalError. Due to the way class scopes work, it's actually possible and sometimes useful to do this, as it "snapshots" the current referent of that name. It's like what happens with default arguments to a function: foo = 'bar' def func(foo=foo): return foo foo = 'quux' print(func()) The newly-defined name isn't "in scope" until its assignment is complete; until then, the old name is fully in scope. ChrisA From zachary.ware+pylist at gmail.com Mon Jul 15 12:06:09 2013 From: zachary.ware+pylist at gmail.com (Zachary Ware) Date: Mon, 15 Jul 2013 11:06:09 -0500 Subject: Is this a bug? In-Reply-To: <51E41A5C.7060903@nottheoilrig.com> References: <51E41A5C.7060903@nottheoilrig.com> Message-ID: On Mon, Jul 15, 2013 at 10:50 AM, Jack Bates wrote: > Hello, > > Is the following code supposed to be an UnboundLocalError? > Currently it assigns the value 'bar' to the attribute baz.foo > > foo = 'bar' > class baz: > foo = foo No bug. It's not an error because of differences in the way classes and functions compile. -- Zach From joshua at landau.ws Mon Jul 15 12:13:12 2013 From: joshua at landau.ws (Joshua Landau) Date: Mon, 15 Jul 2013 17:13:12 +0100 Subject: Is this a bug? In-Reply-To: <51E41A5C.7060903@nottheoilrig.com> References: <51E41A5C.7060903@nottheoilrig.com> Message-ID: On 15 July 2013 16:50, Jack Bates wrote: > Hello, > > Is the following code supposed to be an UnboundLocalError? > Currently it assigns the value 'bar' to the attribute baz.foo > > foo = 'bar' > class baz: > foo = foo I have two responses because I'm not sure what you're saying. Take your pick. ---------------------------------- If you're getting an error from that, then yes. The code works fine for me. I assume you're just jumping to conclusions and not testing code and were doing something like: foo = "Outside foo" def this_will_break(): foo foo = "Inside foo" and wondering why it breaks. Is this correct? If so, it's because Python has consistent ideas about whether something is local to its scope -- some other languages will think that foo is nonlocal *until* they see the assignment. Python knows that foo will be assigned later in the function, so is a local variable the whole time. Being local, the outside version refers to *a different name*. If you want them to refer to the same name, use "nonlocal" or "global", depending on circumstance. foo = "Outside foo" def this_will_break(): global foo foo foo = "Still outside foo (outside foo is changed too)" ---------------------------------------- ALTERNATIVELY Are you saying you think that instead of doing what it does it should raise an UnboundLocalError? If so, then no. Assignments inside class bodies are special-cased in Python. This is because all assignments refer to properties of "self" on the LHS but external things too on the RHS. This is why you can do "def x(): ..." instead of "def self.x(): ..." or some other weird thing. There's also some extra special stuff that goes on. In order to make this an UnboundLocalError, lots of dramatic and unhelpful changes would have to take place, hence the current behaviour. The current behaviour is useful, too. From tdhfwh at nottheoilrig.com Tue Jul 16 11:25:35 2013 From: tdhfwh at nottheoilrig.com (Jack Bates) Date: Tue, 16 Jul 2013 08:25:35 -0700 Subject: Is this a bug? In-Reply-To: References: <51E41A5C.7060903@nottheoilrig.com> Message-ID: <51E565EF.30602@nottheoilrig.com> On 15/07/13 09:13 AM, Joshua Landau wrote: > On 15 July 2013 16:50, Jack Bates wrote: >> Hello, >> >> Is the following code supposed to be an UnboundLocalError? >> Currently it assigns the value 'bar' to the attribute baz.foo >> >> foo = 'bar' >> class baz: >> foo = foo > > If so, then no. Assignments inside class bodies are special-cased in > Python. This is because all assignments refer to properties of "self" > on the LHS but external things too on the RHS. This is why you can do > "def x(): ..." instead of "def self.x(): ..." or some other weird > thing. There's also some extra special stuff that goes on. > > In order to make this an UnboundLocalError, lots of dramatic and > unhelpful changes would have to take place, hence the current > behaviour. The current behaviour is useful, too. Ah, thank you Chris Angelico for explaining how this is like what happens with default arguments to a function and Joshua Landau for pointing out how assignments inside class bodies refer to properties of "self" on the LHS. It makes sense now. Only I'm struggling to find where the behavior is defined in the language reference. Can someone please help point me to where in the language reference this is discussed? I've been hunting through the section on naming and binding: http://docs.python.org/3/reference/executionmodel.html#naming-and-binding From ian.g.kelly at gmail.com Tue Jul 16 14:04:30 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Tue, 16 Jul 2013 12:04:30 -0600 Subject: Is this a bug? In-Reply-To: <51E565EF.30602@nottheoilrig.com> References: <51E41A5C.7060903@nottheoilrig.com> <51E565EF.30602@nottheoilrig.com> Message-ID: On Tue, Jul 16, 2013 at 9:25 AM, Jack Bates wrote: > Ah, thank you Chris Angelico for explaining how this is like what happens > with default arguments to a function and Joshua Landau for pointing out how > assignments inside class bodies refer to properties of "self" on the LHS. It > makes sense now. Only I'm struggling to find where the behavior is defined > in the language reference. Can someone please help point me to where in the > language reference this is discussed? I've been hunting through the section > on naming and binding: > > http://docs.python.org/3/reference/executionmodel.html#naming-and-binding The documentation appears to be wrong. It says: """ If a name binding operation occurs anywhere within a code block, all uses of the name within the block are treated as references to the current block. This can lead to errors when a name is used within a block before it is bound. This rule is subtle. Python lacks declarations and allows name binding operations to occur anywhere within a code block. The local variables of a code block can be determined by scanning the entire text of the block for name binding operations. """ But this only applies to function blocks, not the general case. In general, I believe it is more accurate to say that a variable is local to the block if its name is found in the locals() dict. That normally won't be true until the variable has been bound. Any references prior to that will look for a global variable. From tjreedy at udel.edu Tue Jul 16 19:50:43 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 16 Jul 2013 19:50:43 -0400 Subject: Is this a bug? In-Reply-To: References: <51E41A5C.7060903@nottheoilrig.com> <51E565EF.30602@nottheoilrig.com> Message-ID: On 7/16/2013 2:04 PM, Ian Kelly wrote: > The documentation appears to be wrong. It says: > > """ > If a name binding operation occurs anywhere within a code block, all > uses of the name within the block are treated as references to the > current block. This can lead to errors when a name is used within a > block before it is bound. This rule is subtle. Python lacks > declarations and allows name binding operations to occur anywhere > within a code block. The local variables of a code block can be > determined by scanning the entire text of the block for name binding > operations. > """ I agree that there is a problem. http://bugs.python.org/issue18478 > But this only applies to function blocks, not the general case. In > general, I believe it is more accurate to say that a variable is local > to the block if its name is found in the locals() dict. That is not true for functions, where names are classified as local *before* being added to the locals dict. (Actually, names are usually not added to the locals dict until locals() is called to update it). It would be better to say that names are local if found in the local namespace, and consider that names are added to a function local namespace (which is *not* the local() dict) when classified (before being bound), but otherwise only when bound. That normally > won't be true until the variable has been bound. Any references prior > to that will look for a global variable. At module scope, globals() == locals(). But feel free to suggest a different fix for the issue than I did. -- Terry Jan Reedy From benlast at gmail.com Mon Jul 15 18:21:05 2013 From: benlast at gmail.com (Ben Last) Date: Tue, 16 Jul 2013 06:21:05 +0800 Subject: grimace: a fluent regular expression generator in Python Message-ID: Hi all I'd be interested in comments on a fluent regular expression generator I've been playing with (inspired by the frustrations of a friend of mine who's learning). The general use case is to be able to construct RE strings such as: r'^\(\d{3,3}\)-{1,1}\d{3,3}\-{1,1}\d{4,4}$' (intended to match North American phone number) as: from grimace import RE north_american_number_re = (RE().start .literal('(').followed_by.exactly(3).digits.then.literal(')') .then.one.literal("-").then.exactly(3).digits .then.one.dash.followed_by.exactly(4).digits.then.end .as_string()) The intent is to provide clarity: since the strings would normally be generated and compiled when a module is first imported, there's minimal overhead. It's on github at https://github.com/benlast/grimace and the first blog post that explains it is here: http://benlast.livejournal.com/30871.html (I've added to it since then). Tests are the best explanation, and they're in the __init__ at the end so that they're self-executing if the init is executed. I'm thinking about other features to implement, and more use cases would be welcome. Cheers ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshua at landau.ws Tue Jul 16 04:38:26 2013 From: joshua at landau.ws (Joshua Landau) Date: Tue, 16 Jul 2013 09:38:26 +0100 Subject: grimace: a fluent regular expression generator in Python In-Reply-To: References: Message-ID: On 15 July 2013 23:21, Ben Last wrote: > Hi all > > I'd be interested in comments on a fluent regular expression generator I've > been playing with (inspired by the frustrations of a friend of mine who's > learning). > > The general use case is to be able to construct RE strings such as: > > r'^\(\d{3,3}\)-{1,1}\d{3,3}\-{1,1}\d{4,4}$' (intended to match North > American phone number) > > as: > > from grimace import RE > north_american_number_re = ( > RE().start > .literal('(').followed_by.exactly(3).digits.then.literal(')') > .then.one.literal("-").then.exactly(3).digits > .then.one.dash.followed_by.exactly(4).digits.then.end > .as_string() > ) This looks really busy. How about something more like: from grimace import RE, start, digits, dash, end RE(start, "(", digits[3], ")-", digits[3], dash, digits[4], end).as_string() ? and then you can do cool stuff like (Regex completely untested, I hardly use the things): RE((start | tab), (digits[:], inverse(dash)[4])[:2]) ? r"(^|\t)(\d*[^\-]{4,4}){0,2}" > The intent is to provide clarity: since the strings would normally be > generated and compiled when a module is first imported, there's minimal > overhead. > > It's on github at https://github.com/benlast/grimace and the first blog post > that explains it is here: http://benlast.livejournal.com/30871.html (I've > added to it since then). > > Tests are the best explanation, and they're in the __init__ at the end so > that they're self-executing if the init is executed. > > I'm thinking about other features to implement, and more use cases would be > welcome. > > Cheers > ben > > > -- > http://mail.python.org/mailman/listinfo/python-list > From 2013 at jmunch.dk Tue Jul 16 07:38:35 2013 From: 2013 at jmunch.dk (Anders J. Munch) Date: Tue, 16 Jul 2013 13:38:35 +0200 Subject: grimace: a fluent regular expression generator in Python In-Reply-To: References: Message-ID: <51E530BB.5070403@jmunch.dk> Ben Last wrote: > north_american_number_re = (RE().start > > .literal('(').followed_by.exactly(3).digits.then.literal(')') > .then.one.literal("-").then.exactly(3).digits > > .then.one.dash.followed_by.exactly(4).digits.then.end > .as_string()) Very cool. It's a bit verbose for my taste, and I'm not sure how well it will cope with nested structure. Here's my take on what readable regexps could look like: north_american_number_re = RE.compile(r""" ^ "(" digit{3} ")" # And why shouldn't a regexp "-" digit{3} # include en embedded comment? "-" digit{4} $ """) The problem with Perl-style regexp notation isn't so much that it's terse - it's that the syntax is irregular (sic) and doesn't follow modern principles for lexical structure in computer languages. You can get a long way just by ignoring whitespace, putting literals in quotes and allowing embedded comments. Setting the re.VERBOSE flag achieves two out of three, so you can write: north_american_number_re = RE.compile(r""" ^ ( \d{3} ) # Definite improvement, though I really miss putting - \d{3} # literals in quotes. - \d{4} $ """) It's too bad re.VERBOSE isn't the default. regards, Anders From ben at benlast.com Tue Jul 16 22:33:17 2013 From: ben at benlast.com (Ben Last) Date: Wed, 17 Jul 2013 10:33:17 +0800 Subject: grimace: a fluent regular expression generator in Python Message-ID: On 16 July 2013 20:48, wrote: > From: "Anders J. Munch" <2013 at jmunch.dk> > Date: Tue, 16 Jul 2013 13:38:35 +0200 > Ben Last wrote: > >> north_american_number_re = (RE().start >> .literal('(').followed_by.**exactly(3).digits.then.**literal(')') >> .then.one.literal("-").then.** >> exactly(3).digits >> .then.one.dash.followed_by.**exactly(4).digits.then.end >> .as_string()) >> > > Very cool. It's a bit verbose for my taste, and I'm not sure how well it > will cope with nested structure. > I guess verbosity is the aim, in that *explicit is better than implicit* :) And I suppose that's one of the attributes of a fluent system; they tend to need more typing. It's not Perl... > The problem with Perl-style regexp notation isn't so much that it's terse > - it's that the syntax is irregular (sic) and doesn't follow modern > principles for lexical structure in computer languages. You can get a long > way just by ignoring whitespace, putting literals in quotes and allowing > embedded comments. > Good points. I wanted to find a syntax that allows comments as well as being fluent: RE() .any_number_of.digits # Recall that any_number_of includes zero .followed_by.an_optional.dot.then.at_least_one.digit # The dot is specifically optional # but we must have one digit as a minimum .as_string() ... and yes, I aso specifically wanted to have literals quoted. Nested groups work, but I haven't tackled lookahead and backreferences : essentially because if you're writing an RE that complex, you should probably be working directly in RE strings. Depending on what you mean by "nested", re-use of RE objects is easy (example from the unit tests): identifier_start_chars = RE().regex("[a-zA-Z_]") identifier_chars = RE().regex("[a-zA-Z0-9_]") self.assertEqual(RE().one_or_more.of(identifier_start_chars) .followed_by.zero_or_more(identifier_chars) .as_string(), r"[a-zA-Z_]+[a-zA-Z0-9_]*") Thanks for the comments! ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhibschman at gmail.com Wed Jul 17 08:55:37 2013 From: jhibschman at gmail.com (Johann Hibschman) Date: Wed, 17 Jul 2013 07:55:37 -0500 Subject: grimace: a fluent regular expression generator in Python References: Message-ID: Ben Last writes: > Good points. I wanted to find a syntax that allows comments as well as > being fluent: > RE() > .any_number_of.digits # Recall that any_number_of includes zero > .followed_by.an_optional.dot.then.at_least_one.digit # The dot is > specifically optional > # but we must have one digit as a minimum > .as_string() Speaking of syntax, have you looked at pyparsing? I like their pattern-matching syntax, and I can see it being applied to regexes. They use an operator-heavy syntax, like: '(' + digits * 3 + ')-' + digits * 3 + '-' + digits * 4 That seems easier for me to read than the foo.then.follow syntax. That then makes me think of ometa, which is a fun read, but probably not completely relevant. Regards, Johann From greg.ewing at canterbury.ac.nz Wed Jul 17 19:51:34 2013 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Thu, 18 Jul 2013 11:51:34 +1200 Subject: grimace: a fluent regular expression generator in Python In-Reply-To: References: Message-ID: Ben Last wrote: > north_american_number_re = (RE().start > .literal('(').followed_by.__exactly(3).digits.then.__literal(')') > .then.one.literal("-").then.__exactly(3).digits > .then.one.dash.followed_by.__exactly(4).digits.then.end > .as_string()) Is 'dash' the same as 'literal("-")'? Is there any difference between 'then' and 'followed_by'? Why do some things have __ in front of them? Is there a difference between 'literal' and '__literal'? -- Greg From roy at panix.com Wed Jul 17 10:22:00 2013 From: roy at panix.com (Roy Smith) Date: Wed, 17 Jul 2013 10:22:00 -0400 Subject: grimace: a fluent regular expression generator in Python References: Message-ID: In article , "Anders J. Munch" <2013 at jmunch.dk> wrote: > The problem with Perl-style regexp notation isn't so much that it's terse - > it's > that the syntax is irregular (sic) and doesn't follow modern principles for > lexical structure in computer languages. There seem to be three basic ways to denote what's literal and what's not. 1) The Python (and C, Java, PHP, Fortran, etc) way, where all text is assumed to be evaluated as a language construct, unless explicitly quoted to make it a literal. 2) The shell way, where all text is assumed to be literal strings, unless explicitly marked with a $ (or other sigil) as a variable. 3) The regex way, where some characters are magic, but only sometimes (depending on context), and you just have to know which ones they are, and when, and can escape them to make them non-magic if you have to. Where things get really messy is when you try to embed one language into another, such as regexes in Python. Perl (and awk, from which it evolved) solves the problem in its own way by making regexes a built-in part of the language syntax. Python goes in the other direction, and says regexes are just strings that you pass around. > You can get a long way just by ignoring whitespace, putting literals > in quotes and allowing embedded comments. Setting the re.VERBOSE > flag achieves two out of three [example elided]. Yup. Here's a more complex example. We use this to parse haproxy log files (probably going to munged a bit as lines get refolded by news software). That would be insane without verbose mode (some might argue it's insane now, but that's another thread). pattern = re.compile(r'haproxy\[(?P\d+)]: ' r'(?P(\d{1,3}\.){3}\d{1,3}):' r'(?P\d{1,5}) ' r'\[(?P\d{2}/\w{3}/\d{4}(:\d{2}){3}\.\d{3})] ' r'(?P\S+) ' r'(?P\S+)/' r'(?P\S+) ' r'(?P(-1|\d+))/' r'(?P(-1|\d+))/' r'(?P(-1|\d+))/' r'(?P(-1|\d+))/' r'(?P\+?\d+) ' r'(?P\d{3}) ' r'(?P\d+) ' r'(?P\S+) ' r'(?P\S+) ' r'(?P[\w-]{4}) ' r'(?P\d+)/' r'(?P\d+)/' r'(?P\d+)/' r'(?P\d+)/' r'(?P\d+) ' r'(?P\d+)/' r'(?P\d+) ' r'(\{(?P.*?)\} )?' # Comment this out for a stock haproxy (see above) r'(\{(?P.*?)\} )?' r'(\{(?P.*?)\} )?' r'"(?P.+)"' ) From vito.detullio at gmail.com Tue Jul 16 01:44:45 2013 From: vito.detullio at gmail.com (Vito De Tullio) Date: Tue, 16 Jul 2013 07:44:45 +0200 Subject: a little more explicative error message? Message-ID: Hi I was writing a decorator and lost half an hour for a stupid bug in my code, but honestly the error the python interpreter returned to me doesn't helped... $ python3 Python 3.3.0 (default, Feb 24 2013, 09:34:27) [GCC 4.7.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from functools import wraps >>> def dec(fun): ... @wraps ... def ret(*args, **kwargs): ... return fun(*args, **kwargs) ... return ret ... >>> @dec ... def fun(): pass ... >>> fun() Traceback (most recent call last): File "", line 1, in TypeError: update_wrapper() missing 1 required positional argument: 'wrapper' >>> $ Soo... at a first glance, no tricks... can you tell where is the error? :D As I said, the error is totally mine, I just forgot to pass the function as parameter to wraps. But... what is "update_wrapper()"? and "wrapper"? There is no useful traceback or something... just... this. Ok, the documentation clearly says: This is a convenience function to simplify applying partial() to update_wrapper(). So, again, shame on me... I just read carefully the doc *after* 20 minutes trying everything else... still... I think should be useful if wraps() intercept this error saying something more explicit about the missing fun parameter... -- ZeD From tjreedy at udel.edu Tue Jul 16 04:25:06 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 16 Jul 2013 04:25:06 -0400 Subject: a little more explicative error message? In-Reply-To: References: Message-ID: On 7/16/2013 1:44 AM, Vito De Tullio wrote: > Hi > > I was writing a decorator and lost half an hour for a stupid bug in my code, > but honestly the error the python interpreter returned to me doesn't > helped... > > $ python3 > Python 3.3.0 (default, Feb 24 2013, 09:34:27) > [GCC 4.7.2] on linux > Type "help", "copyright", "credits" or "license" for more information. >>>> from functools import wraps >>>> def dec(fun): > ... @wraps > ... def ret(*args, **kwargs): > ... return fun(*args, **kwargs) At this point, when dec(fun) is called, the interpreter *successfully* executes ret = wraps(ret), which works, but is wrong, because wraps should be called with the wrapped function fun as its argument, not the wrapper function ret. The interpreter should instead execute ret = partial(update_wrapper, wrapped = fun, ...)(ret) which will update ret to look like fun. > ... return ret > ... >>>> @dec > ... def fun(): pass At this point, the interpreter *successfully* executes fun = dec(fun) = (wraps(ret))(fun), which causes ret = wraps(ret) before returning ret. Notice that when dec returns, the wraps code is over and done with. Instead the interpreter should execute fun = (partial(update_wrapper, wrapped = fun, ...)(ret))(fun) > ... >>>> fun() > Traceback (most recent call last): > File "", line 1, in > TypeError: update_wrapper() missing 1 required positional argument: > 'wrapper' Because fun has not been properly wrapped because you left out a call. > Soo... at a first glance, no tricks... can you tell where is the error? :D Not exactly, but it must have something to do with wraps, so the first thing I would do is to look at the wraps doc if I had not before. It only takes a couple of minutes. >>> from functools import wraps >>> help(wraps) Help on function wraps in module functools: wraps(wrapped, assigned=('__module__', '__name__', '__qualname__', '__doc__', '__annotations__'), updated=('__dict__',)) Decorator factory to apply update_wrapper() to a wrapper function Returns a decorator that invokes update_wrapper() with the decorated function as the wrapper argument and the arguments to wraps() as the remaining arguments. ... This is pretty clear that wraps is not a decorator but a function that returns decorator, which means that is must be called with an argument in the @del statement. To really understand what is intended to happen so I could write the commentary above, I looked at the code to see def wraps(wrapped, ...): '' return partial(update_wrapper, wrapped=wrapped, assigned=assigned, updated=updated) > As I said, the error is totally mine, I just forgot to pass the function as > parameter to wraps. But... what is "update_wrapper()"? and "wrapper"? There > is no useful traceback or something... just... this. > > Ok, the documentation clearly says: > > This is a convenience function to simplify applying partial() to > update_wrapper(). > > So, again, shame on me... I just read carefully the doc *after* 20 minutes > trying everything else... still... I think should be useful if wraps() > intercept this error saying something more explicit about the missing fun > parameter... How is a function that has already been called and has returned without apparent error supposed to catch a later error caused by its return not being correct, due to its input not being correct? Your request is like asking a function f(x) to catch ZeroDivisionError if it return a 0 and that 0 is later used as a divisor after f returns, as in 1/f(x). When you call wraps with a callable, there is no way for it to know that the callable in intended to the be the wrapper instead of the wrappee, unless it were clairvoyant ;-). -- Terry Jan Reedy From l.mohanphy at gmail.com Tue Jul 16 02:55:58 2013 From: l.mohanphy at gmail.com (Mohan L) Date: Tue, 16 Jul 2013 12:25:58 +0530 Subject: help on python regular expression named group Message-ID: Dear All, Here is my script : #!/usr/bin/python import re # A string. logs = "date=2012-11-28 time=21:14:59" # Match with named groups. m = re.match("(?P(date=(?P[^\s]+))\s+(time=(?P



class="gmail_quote"> [snip 70-odd lines of HTML...] > I'm guessing you may be posting with html.? So all your code runs > together. >

--
My irony meter didn't merely explode, it actually vaporized. -- Steven From devyncjohnson at gmail.com Fri Jul 19 18:10:06 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Fri, 19 Jul 2013 18:10:06 -0400 Subject: Share Code Tips In-Reply-To: <51e966f4$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <51E9445B.30509@Gmail.com> <51e966f4$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51E9B93E.6090300@Gmail.com> On 07/19/2013 12:19 PM, Steven D'Aprano wrote: > On Fri, 19 Jul 2013 10:21:10 -0400, Joel Goldstick wrote: > >>



> class="gmail_quote"> > [snip 70-odd lines of HTML...] > >> I'm guessing you may be posting with html. So all your code runs >> together. >>

--
> dir="ltr">
Joel Goldstick
http://joelgoldstick.com >>
> > My irony meter didn't merely explode, it actually vaporized. > > > What is the issue with my email? Should I resend it? Mahalo, DCJ From rosuav at gmail.com Fri Jul 19 19:04:39 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 20 Jul 2013 09:04:39 +1000 Subject: Share Code Tips In-Reply-To: <51E9B93E.6090300@Gmail.com> References: <51E9445B.30509@Gmail.com> <51e966f4$0$29971$c3e8da3$5496439d@news.astraweb.com> <51E9B93E.6090300@Gmail.com> Message-ID: On Sat, Jul 20, 2013 at 8:10 AM, Devyn Collier Johnson wrote: > > On 07/19/2013 12:19 PM, Steven D'Aprano wrote: >> >> On Fri, 19 Jul 2013 10:21:10 -0400, Joel Goldstick wrote: >> >>>



>> class="gmail_quote"> >> >> [snip 70-odd lines of HTML...] >> >>> I'm guessing you may be posting with html. So all your code runs >>> together. >>>

--
>> dir="ltr">
Joel Goldstick
http://joelgoldstick.com >>>
>> >> >> My irony meter didn't merely explode, it actually vaporized. >> >> >> > What is the issue with my email? Should I resend it? Either your lines are insanely long (and got wrapped by the mailer), or your multi-line functions lost their internal line breaks. Joel suggested that you may have been posting HTML mail (you're not), implying that it's worse than plain text mail (it is), and himself posted in HTML (that's irony). ChrisA (This is a parenthesis.) From steve+comp.lang.python at pearwood.info Fri Jul 19 13:59:10 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 19 Jul 2013 17:59:10 GMT Subject: Share Code Tips References: Message-ID: <51e97e6e$0$29971$c3e8da3$5496439d@news.astraweb.com> On Fri, 19 Jul 2013 09:51:23 -0400, Devyn Collier Johnson wrote: > def KDE_VERSION(): > print(subprocess.getoutput('kded4 --version | awk -F: > \'NR == 2 {print $2}\'').strip()) ##Get KDE version## I run KDE 3, and the above does not work for me. *half a wink* By the way, a comment that doesn't tell you anything that you don't already know is worse than useless. The function is called "KDE_VERSION, what else would it do other than return the KDE version? x += 1 # add 1 to x Worse than just being useless, redundant comments are dangerous, because as a general rule comments that don't say anything useful eventually become out-of-date, they become *inaccurate* rather than *redundant*, and that's worse than being useless. > Need a case-insensitive if-statement? Check this out: > > if 'YOUR_STRING'.lower() in SOMEVAR.lower(): Case-insensitivity is very hard. Take German for example: STRASSE <-> stra?e Or Turkish: ? <-> i I <-> ? In Python 3.3, you should use casefold rather than lowercase or uppercase: if some_string.casefold() in another_string.casefold(): ... but even that can't always take into account localised rules, e.g. in German, you should not convert SS to ? for placenames or person names, so for example Herr Mei?ner and Herr Meissner are two different people. This is one of the motivating reasons for introducing the uppercase ?. http://opentype.info/blog/2011/01/24/capital-sharp-s/ -- Steven From devyncjohnson at gmail.com Fri Jul 19 18:08:43 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Fri, 19 Jul 2013 18:08:43 -0400 Subject: Share Code Tips In-Reply-To: <51e97e6e$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <51e97e6e$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51E9B8EB.5060007@Gmail.com> On 07/19/2013 01:59 PM, Steven D'Aprano wrote: > On Fri, 19 Jul 2013 09:51:23 -0400, Devyn Collier Johnson wrote: > >> def KDE_VERSION(): >> print(subprocess.getoutput('kded4 --version | awk -F: >> \'NR == 2 {print $2}\'').strip()) ##Get KDE version## > I run KDE 3, and the above does not work for me. > > *half a wink* > > By the way, a comment that doesn't tell you anything that you don't > already know is worse than useless. The function is called "KDE_VERSION, > what else would it do other than return the KDE version? > > > x += 1 # add 1 to x > > Worse than just being useless, redundant comments are dangerous, because > as a general rule comments that don't say anything useful eventually > become out-of-date, they become *inaccurate* rather than *redundant*, and > that's worse than being useless. > > >> Need a case-insensitive if-statement? Check this out: >> >> if 'YOUR_STRING'.lower() in SOMEVAR.lower(): > Case-insensitivity is very hard. Take German for example: > > STRASSE <-> stra?e > > Or Turkish: > > ? <-> i > I <-> ? > > > In Python 3.3, you should use casefold rather than lowercase or uppercase: > > if some_string.casefold() in another_string.casefold(): ... > > > but even that can't always take into account localised rules, e.g. in > German, you should not convert SS to ? for placenames or person names, so > for example Herr Mei?ner and Herr Meissner are two different people. This > is one of the motivating reasons for introducing the uppercase ?. > > http://opentype.info/blog/2011/01/24/capital-sharp-s/ > > > Steven, thanks for your interesting comments. Your emails are very insightful. As for the KDE function, I should fix that. Thank you for catching that. Notice that the shell command in the function is "kded4". That would only check the version for the KDE4 series. The function will only work for KDE4 users. As for the comment, you would be amazed with the people that ask me "what does this do?". These people are redundant (^u^). As for the case-insensitive if-statements, most code uses Latin letters. Making a case-insensitive-international if-statement would be interesting. I can tackle that later. For now, I only wanted to take care of Latin letters. I hope to figure something out for all characters. Thank you for your reply. I found it to be very helpful. Mahalo, DCJ From rosuav at gmail.com Fri Jul 19 19:08:35 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 20 Jul 2013 09:08:35 +1000 Subject: Share Code Tips In-Reply-To: <51E9B8EB.5060007@Gmail.com> References: <51e97e6e$0$29971$c3e8da3$5496439d@news.astraweb.com> <51E9B8EB.5060007@Gmail.com> Message-ID: On Sat, Jul 20, 2013 at 8:08 AM, Devyn Collier Johnson wrote: > As for the case-insensitive if-statements, most code uses Latin letters. > Making a case-insensitive-international if-statement would be interesting. I > can tackle that later. For now, I only wanted to take care of Latin letters. > I hope to figure something out for all characters. Case insensitivity is a *hard* problem. Don't fool yourself that you can do it with a simple line of code and have it 'just work'. All you'll have is something that works "most of the time", and then breaks on certain input. As Steven said, using casefold() rather than lower() will help, but that's still not perfect. The simplest and safest way to solve Unicode capitalization issues is to declare that your protocol is case sensitive. I have a brother who couldn't understand why Unix file systems have to be case sensitive (why would anyone ever want to have "readme" and "README" in the same directory, after all?), until I explained how majorly hard it is with i18n, and how it suddenly becomes extremely unsafe for your *file system* to get this wrong. ChrisA From davea at davea.name Fri Jul 19 19:09:32 2013 From: davea at davea.name (Dave Angel) Date: Fri, 19 Jul 2013 19:09:32 -0400 Subject: Share Code Tips In-Reply-To: <51E9B8EB.5060007@Gmail.com> References: <51e97e6e$0$29971$c3e8da3$5496439d@news.astraweb.com> <51E9B8EB.5060007@Gmail.com> Message-ID: On 07/19/2013 06:08 PM, Devyn Collier Johnson wrote: > > On 07/19/2013 01:59 PM, Steven D'Aprano wrote: > > As for the case-insensitive if-statements, most code uses Latin letters. > Making a case-insensitive-international if-statement would be > interesting. I can tackle that later. For now, I only wanted to take > care of Latin letters. I hope to figure something out for all characters. > Once Steven gave you the answer, what's to figure out? You simply use casefold() instead of lower(). The only constraint is it's 3.3 and later, so you can't use it for anything earlier. http://docs.python.org/3.3/library/stdtypes.html#str.casefold """ str.casefold() Return a casefolded copy of the string. Casefolded strings may be used for caseless matching. Casefolding is similar to lowercasing but more aggressive because it is intended to remove all case distinctions in a string. For example, the German lowercase letter '?' is equivalent to "ss". Since it is already lowercase, lower() would do nothing to '?'; casefold() converts it to "ss". The casefolding algorithm is described in section 3.13 of the Unicode Standard. New in version 3.3. """ -- DaveA From devyncjohnson at gmail.com Fri Jul 19 21:04:55 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Fri, 19 Jul 2013 21:04:55 -0400 Subject: Share Code Tips In-Reply-To: References: <51e97e6e$0$29971$c3e8da3$5496439d@news.astraweb.com> <51E9B8EB.5060007@Gmail.com> Message-ID: <51E9E237.2040903@Gmail.com> On 07/19/2013 07:09 PM, Dave Angel wrote: > On 07/19/2013 06:08 PM, Devyn Collier Johnson wrote: >> >> On 07/19/2013 01:59 PM, Steven D'Aprano wrote: > > >> >> As for the case-insensitive if-statements, most code uses Latin letters. >> Making a case-insensitive-international if-statement would be >> interesting. I can tackle that later. For now, I only wanted to take >> care of Latin letters. I hope to figure something out for all >> characters. >> > > Once Steven gave you the answer, what's to figure out? You simply use > casefold() instead of lower(). The only constraint is it's 3.3 and > later, so you can't use it for anything earlier. > > http://docs.python.org/3.3/library/stdtypes.html#str.casefold > > """ > str.casefold() > Return a casefolded copy of the string. Casefolded strings may be used > for caseless matching. > > Casefolding is similar to lowercasing but more aggressive because it > is intended to remove all case distinctions in a string. For example, > the German lowercase letter '?' is equivalent to "ss". Since it is > already lowercase, lower() would do nothing to '?'; casefold() > converts it to "ss". > > The casefolding algorithm is described in section 3.13 of the Unicode > Standard. > > New in version 3.3. > """ > Chris Angelico said that casefold is not perfect. In the future, I want to make the perfect international-case-insensitive if-statement. For now, my code only supports a limited range of characters. Even with casefold, I will have some issues as Chris Angelico mentioned. Also, "?" is not really the same as "ss". Mahalo, DCJ From rosuav at gmail.com Fri Jul 19 21:13:17 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 20 Jul 2013 11:13:17 +1000 Subject: Share Code Tips In-Reply-To: <51E9E237.2040903@Gmail.com> References: <51e97e6e$0$29971$c3e8da3$5496439d@news.astraweb.com> <51E9B8EB.5060007@Gmail.com> <51E9E237.2040903@Gmail.com> Message-ID: On Sat, Jul 20, 2013 at 11:04 AM, Devyn Collier Johnson wrote: > > On 07/19/2013 07:09 PM, Dave Angel wrote: >> >> On 07/19/2013 06:08 PM, Devyn Collier Johnson wrote: >>> >>> >>> On 07/19/2013 01:59 PM, Steven D'Aprano wrote: >> >> >> >>> >>> >>> As for the case-insensitive if-statements, most code uses Latin letters. >>> Making a case-insensitive-international if-statement would be >>> interesting. I can tackle that later. For now, I only wanted to take >>> care of Latin letters. I hope to figure something out for all characters. >>> >> >> Once Steven gave you the answer, what's to figure out? You simply use >> casefold() instead of lower(). The only constraint is it's 3.3 and later, >> so you can't use it for anything earlier. >> >> http://docs.python.org/3.3/library/stdtypes.html#str.casefold >> >> """ >> str.casefold() >> Return a casefolded copy of the string. Casefolded strings may be used for >> caseless matching. >> >> Casefolding is similar to lowercasing but more aggressive because it is >> intended to remove all case distinctions in a string. For example, the >> German lowercase letter '?' is equivalent to "ss". Since it is already >> lowercase, lower() would do nothing to '?'; casefold() converts it to "ss". >> >> The casefolding algorithm is described in section 3.13 of the Unicode >> Standard. >> >> New in version 3.3. >> """ >> > Chris Angelico said that casefold is not perfect. In the future, I want to > make the perfect international-case-insensitive if-statement. For now, my > code only supports a limited range of characters. Even with casefold, I will > have some issues as Chris Angelico mentioned. Also, "?" is not really the > same as "ss". Well, casefold is about as good as it's ever going to be, but that's because "the perfect international-case-insensitive comparison" is a fundamentally impossible goal. Your last sentence hints as to why; there is no simple way to compare strings containing those characters, because the correct treatment varies according to context. Your two best options are: Be case sensitive (and then you need only worry about composition and combining characters and all those nightmares - the ones you have to worry about either way), or use casefold(). Of those, I prefer the first, because it's safer; the second is also a good option. ChrisA From dwightdhutto at gmail.com Fri Jul 19 23:42:00 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Fri, 19 Jul 2013 23:42:00 -0400 Subject: Share Code Tips In-Reply-To: References: <51e97e6e$0$29971$c3e8da3$5496439d@news.astraweb.com> <51E9B8EB.5060007@Gmail.com> <51E9E237.2040903@Gmail.com> Message-ID: Just use an explanatory user tip that states it should be case sensitive, just like with most sites, or apps. On Fri, Jul 19, 2013 at 9:13 PM, Chris Angelico wrote: > On Sat, Jul 20, 2013 at 11:04 AM, Devyn Collier Johnson > wrote: > > > > On 07/19/2013 07:09 PM, Dave Angel wrote: > >> > >> On 07/19/2013 06:08 PM, Devyn Collier Johnson wrote: > >>> > >>> > >>> On 07/19/2013 01:59 PM, Steven D'Aprano wrote: > >> > >> > >> > >>> > >>> > >>> As for the case-insensitive if-statements, most code uses Latin > letters. > >>> Making a case-insensitive-international if-statement would be > >>> interesting. I can tackle that later. For now, I only wanted to take > >>> care of Latin letters. I hope to figure something out for all > characters. > >>> > >> > >> Once Steven gave you the answer, what's to figure out? You simply use > >> casefold() instead of lower(). The only constraint is it's 3.3 and > later, > >> so you can't use it for anything earlier. > >> > >> http://docs.python.org/3.3/library/stdtypes.html#str.casefold > >> > >> """ > >> str.casefold() > >> Return a casefolded copy of the string. Casefolded strings may be used > for > >> caseless matching. > >> > >> Casefolding is similar to lowercasing but more aggressive because it is > >> intended to remove all case distinctions in a string. For example, the > >> German lowercase letter '?' is equivalent to "ss". Since it is already > >> lowercase, lower() would do nothing to '?'; casefold() converts it to > "ss". > >> > >> The casefolding algorithm is described in section 3.13 of the Unicode > >> Standard. > >> > >> New in version 3.3. > >> """ > >> > > Chris Angelico said that casefold is not perfect. In the future, I want > to > > make the perfect international-case-insensitive if-statement. For now, my > > code only supports a limited range of characters. Even with casefold, I > will > > have some issues as Chris Angelico mentioned. Also, "?" is not really the > > same as "ss". > > Well, casefold is about as good as it's ever going to be, but that's > because "the perfect international-case-insensitive comparison" is a > fundamentally impossible goal. Your last sentence hints as to why; > there is no simple way to compare strings containing those characters, > because the correct treatment varies according to context. > > Your two best options are: Be case sensitive (and then you need only > worry about composition and combining characters and all those > nightmares - the ones you have to worry about either way), or use > casefold(). Of those, I prefer the first, because it's safer; the > second is also a good option. > > ChrisA > -- > http://mail.python.org/mailman/listinfo/python-list > -- Best Regards, David Hutto *CEO:* *http://www.hitwebdevelopment.com* -------------- next part -------------- An HTML attachment was scrubbed... URL: From devyncjohnson at gmail.com Sat Jul 20 08:20:30 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Sat, 20 Jul 2013 08:20:30 -0400 Subject: Share Code Tips In-Reply-To: References: <51e97e6e$0$29971$c3e8da3$5496439d@news.astraweb.com> <51E9B8EB.5060007@Gmail.com> <51E9E237.2040903@Gmail.com> Message-ID: <51EA808E.5070602@Gmail.com> On 07/19/2013 09:13 PM, Chris Angelico wrote: > On Sat, Jul 20, 2013 at 11:04 AM, Devyn Collier Johnson > wrote: >> On 07/19/2013 07:09 PM, Dave Angel wrote: >>> On 07/19/2013 06:08 PM, Devyn Collier Johnson wrote: >>>> >>>> On 07/19/2013 01:59 PM, Steven D'Aprano wrote: >>> >>> >>>> >>>> As for the case-insensitive if-statements, most code uses Latin letters. >>>> Making a case-insensitive-international if-statement would be >>>> interesting. I can tackle that later. For now, I only wanted to take >>>> care of Latin letters. I hope to figure something out for all characters. >>>> >>> Once Steven gave you the answer, what's to figure out? You simply use >>> casefold() instead of lower(). The only constraint is it's 3.3 and later, >>> so you can't use it for anything earlier. >>> >>> http://docs.python.org/3.3/library/stdtypes.html#str.casefold >>> >>> """ >>> str.casefold() >>> Return a casefolded copy of the string. Casefolded strings may be used for >>> caseless matching. >>> >>> Casefolding is similar to lowercasing but more aggressive because it is >>> intended to remove all case distinctions in a string. For example, the >>> German lowercase letter '?' is equivalent to "ss". Since it is already >>> lowercase, lower() would do nothing to '?'; casefold() converts it to "ss". >>> >>> The casefolding algorithm is described in section 3.13 of the Unicode >>> Standard. >>> >>> New in version 3.3. >>> """ >>> >> Chris Angelico said that casefold is not perfect. In the future, I want to >> make the perfect international-case-insensitive if-statement. For now, my >> code only supports a limited range of characters. Even with casefold, I will >> have some issues as Chris Angelico mentioned. Also, "?" is not really the >> same as "ss". > Well, casefold is about as good as it's ever going to be, but that's > because "the perfect international-case-insensitive comparison" is a > fundamentally impossible goal. Your last sentence hints as to why; > there is no simple way to compare strings containing those characters, > because the correct treatment varies according to context. > > Your two best options are: Be case sensitive (and then you need only > worry about composition and combining characters and all those > nightmares - the ones you have to worry about either way), or use > casefold(). Of those, I prefer the first, because it's safer; the > second is also a good option. > > ChrisA Thanks everyone (especially Chris Angelico and Steven D'Aprano) for all of your helpful suggests and ideas. I plan to implement casefold() in some of my programs. Mahalo, DCJ From davea at davea.name Fri Jul 19 21:51:32 2013 From: davea at davea.name (Dave Angel) Date: Fri, 19 Jul 2013 21:51:32 -0400 Subject: Share Code Tips In-Reply-To: <51E9E237.2040903@Gmail.com> References: <51e97e6e$0$29971$c3e8da3$5496439d@news.astraweb.com> <51E9B8EB.5060007@Gmail.com> <51E9E237.2040903@Gmail.com> Message-ID: On 07/19/2013 09:04 PM, Devyn Collier Johnson wrote: > >> > Chris Angelico said that casefold is not perfect. In the future, I want > to make the perfect international-case-insensitive if-statement. For > now, my code only supports a limited range of characters. Even with > casefold, I will have some issues as Chris Angelico mentioned. Also, "?" > is not really the same as "ss". > Sure, the casefold() method has its problems. But you're going to avoid using it till you can do a "perfect" one? Perfect in what context? For "case sensitively" comparing people's names in a single language in a single country? Perhaps that can be made perfect. For certain combinations of language and country. But if you want to compare words in an unspecified language with an unspecified country, it cannot be done. If you've got a particular goal in mind, great. But as a library function, you're better off using the best standard method available, and document what its limitations are. One way of documenting such is to quote the appropriate standards, with their caveats. By the way, you mentioned earlier that you're restricting yourself to Latin characters. The lower() method is inadequate for many of those as well. Perhaps you meant ASCII instead. -- DaveA From devyncjohnson at gmail.com Sat Jul 20 06:06:07 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Sat, 20 Jul 2013 06:06:07 -0400 Subject: Share Code Tips In-Reply-To: References: <51e97e6e$0$29971$c3e8da3$5496439d@news.astraweb.com> <51E9B8EB.5060007@Gmail.com> <51E9E237.2040903@Gmail.com> Message-ID: <51EA610F.7050808@Gmail.com> On 07/19/2013 09:51 PM, Dave Angel wrote: > On 07/19/2013 09:04 PM, Devyn Collier Johnson wrote: >> > > >>> >> Chris Angelico said that casefold is not perfect. In the future, I want >> to make the perfect international-case-insensitive if-statement. For >> now, my code only supports a limited range of characters. Even with >> casefold, I will have some issues as Chris Angelico mentioned. Also, "?" >> is not really the same as "ss". >> > > Sure, the casefold() method has its problems. But you're going to > avoid using it till you can do a "perfect" one? > > Perfect in what context? For "case sensitively" comparing people's > names in a single language in a single country? Perhaps that can be > made perfect. For certain combinations of language and country. > > But if you want to compare words in an unspecified language with an > unspecified country, it cannot be done. > > If you've got a particular goal in mind, great. But as a library > function, you're better off using the best standard method available, > and document what its limitations are. One way of documenting such is > to quote the appropriate standards, with their caveats. > > > By the way, you mentioned earlier that you're restricting yourself to > Latin characters. The lower() method is inadequate for many of those > as well. Perhaps you meant ASCII instead. > Of course not, Dave; I will implement casefold. I just plan to not stop there. My program should not come across unspecified languages. Yeah, I meant ASCII, but I was unaware that lower() had some limitation on Latin letters. Mahalo, DCJ From steve+comp.lang.python at pearwood.info Fri Jul 19 23:44:36 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 20 Jul 2013 03:44:36 GMT Subject: Share Code Tips References: <51e97e6e$0$29971$c3e8da3$5496439d@news.astraweb.com> <51E9B8EB.5060007@Gmail.com> Message-ID: <51ea07a4$0$29971$c3e8da3$5496439d@news.astraweb.com> On Fri, 19 Jul 2013 21:04:55 -0400, Devyn Collier Johnson wrote: > In the future, I want to > make the perfect international-case-insensitive if-statement. For now, > my code only supports a limited range of characters. Even with casefold, > I will have some issues as Chris Angelico mentioned. There are hundreds of written languages in the world, with thousands of characters, and most of them have rules about case-sensitivity and character normalization. For example, in Greek, lowercase ? is ? except at the end of a word, when it is ?. ??? '???'.upper() '???' ??? '???'.lower() '???' ??? '???'.casefold() '???' So in this case, casefold() correctly solves the problem, provided you are comparing modern Greek text. But if you're comparing text in some other language which merely happens to use Greek letters, but doesn't have the same rules about letter sigma, then it will be inappropriate. So you cannot write a single "perfect" case-insensitive comparison, the best you can hope for is to write dozens or hundreds of separate case- insensitive comparisons, one for each language or family of languages. For an introduction to the problem: http://www.w3.org/International/wiki/Case_folding http://www.unicode.org/faq/casemap_charprop.html > Also, "?" is not really the same as "ss". Sometimes it is. Sometimes it isn't. -- Steven From dwightdhutto at gmail.com Sat Jul 20 00:15:21 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Sat, 20 Jul 2013 00:15:21 -0400 Subject: Share Code Tips In-Reply-To: <51ea07a4$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <51e97e6e$0$29971$c3e8da3$5496439d@news.astraweb.com> <51E9B8EB.5060007@Gmail.com> <51ea07a4$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: It seems, without utilizing this, or googling, that a case sensitive library is either developed, or could be implemented by utilizing case sensitive translation through a google translation page using an urlopener, and placing in the data to be processed back to the boolean value. Never attempted, but the algorithm seems simpler than the dozens of solutions method. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwightdhutto at gmail.com Sat Jul 20 00:22:04 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Sat, 20 Jul 2013 00:22:04 -0400 Subject: Share Code Tips In-Reply-To: References: <51e97e6e$0$29971$c3e8da3$5496439d@news.astraweb.com> <51E9B8EB.5060007@Gmail.com> <51ea07a4$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: It seems that you could use import re, in my mind's pseudo code, to compile a translational usage of usernames/passwords that could remain case sensitive by using just the translational dictionaries, and refining with data input tests/unit tests. On Sat, Jul 20, 2013 at 12:15 AM, David Hutto wrote: > It seems, without utilizing this, or googling, that a case sensitive > library is either developed, or could be implemented by utilizing case > sensitive translation through a google translation page using an urlopener, > and placing in the data to be processed back to the boolean value. Never > attempted, but the algorithm seems simpler than the dozens of solutions > method. > -- Best Regards, David Hutto *CEO:* *http://www.hitwebdevelopment.com* -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwightdhutto at gmail.com Sat Jul 20 00:26:16 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Sat, 20 Jul 2013 00:26:16 -0400 Subject: Share Code Tips In-Reply-To: References: <51e97e6e$0$29971$c3e8da3$5496439d@news.astraweb.com> <51E9B8EB.5060007@Gmail.com> <51ea07a4$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: I didn't see that this was for a chess game. That seems more point and click. Everyone can recognize a bishop from a queen, or a rook from a pawn. So why would case sensitivity matter other than the 16 pieces on the board? Or am I misunderstanding the question? On Sat, Jul 20, 2013 at 12:22 AM, David Hutto wrote: > It seems that you could use import re, in my mind's pseudo code, to > compile a translational usage of usernames/passwords that could remain case > sensitive by using just the translational dictionaries, and refining with > data input tests/unit tests. > > > On Sat, Jul 20, 2013 at 12:15 AM, David Hutto wrote: > >> It seems, without utilizing this, or googling, that a case sensitive >> library is either developed, or could be implemented by utilizing case >> sensitive translation through a google translation page using an urlopener, >> and placing in the data to be processed back to the boolean value. Never >> attempted, but the algorithm seems simpler than the dozens of solutions >> method. >> > > > > -- > Best Regards, > David Hutto > *CEO:* *http://www.hitwebdevelopment.com* > -- Best Regards, David Hutto *CEO:* *http://www.hitwebdevelopment.com* -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwightdhutto at gmail.com Sat Jul 20 00:27:26 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Sat, 20 Jul 2013 00:27:26 -0400 Subject: Share Code Tips In-Reply-To: References: <51e97e6e$0$29971$c3e8da3$5496439d@news.astraweb.com> <51E9B8EB.5060007@Gmail.com> <51ea07a4$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: 32 if you count black, and white. On Sat, Jul 20, 2013 at 12:26 AM, David Hutto wrote: > I didn't see that this was for a chess game. That seems more point and > click. Everyone can recognize a bishop from a queen, or a rook from a pawn. > So why would case sensitivity matter other than the 16 pieces on the board? > Or am I misunderstanding the question? > > > > On Sat, Jul 20, 2013 at 12:22 AM, David Hutto wrote: > >> It seems that you could use import re, in my mind's pseudo code, to >> compile a translational usage of usernames/passwords that could remain case >> sensitive by using just the translational dictionaries, and refining with >> data input tests/unit tests. >> >> >> On Sat, Jul 20, 2013 at 12:15 AM, David Hutto wrote: >> >>> It seems, without utilizing this, or googling, that a case sensitive >>> library is either developed, or could be implemented by utilizing case >>> sensitive translation through a google translation page using an urlopener, >>> and placing in the data to be processed back to the boolean value. Never >>> attempted, but the algorithm seems simpler than the dozens of solutions >>> method. >>> >> >> >> >> -- >> Best Regards, >> David Hutto >> *CEO:* *http://www.hitwebdevelopment.com* >> > > > > -- > Best Regards, > David Hutto > *CEO:* *http://www.hitwebdevelopment.com* > -- Best Regards, David Hutto *CEO:* *http://www.hitwebdevelopment.com* -------------- next part -------------- An HTML attachment was scrubbed... URL: From devyncjohnson at gmail.com Sat Jul 20 08:36:46 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Sat, 20 Jul 2013 08:36:46 -0400 Subject: Share Code Tips In-Reply-To: References: <51e97e6e$0$29971$c3e8da3$5496439d@news.astraweb.com> <51E9B8EB.5060007@Gmail.com> <51ea07a4$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51EA845E.3090101@Gmail.com> On 07/20/2013 12:26 AM, David Hutto wrote: > I didn't see that this was for a chess game. That seems more point and > click. Everyone can recognize a bishop from a queen, or a rook from a > pawn. So why would case sensitivity matter other than the 16 pieces on > the board? Or am I misunderstanding the question? > > > > On Sat, Jul 20, 2013 at 12:22 AM, David Hutto > wrote: > > It seems that you could use import re, in my mind's pseudo code, > to compile a translational usage of usernames/passwords that could > remain case sensitive by using just the translational > dictionaries, and refining with data input tests/unit tests. > > > On Sat, Jul 20, 2013 at 12:15 AM, David Hutto > > wrote: > > It seems, without utilizing this, or googling, that a case > sensitive library is either developed, or could be implemented > by utilizing case sensitive translation through a google > translation page using an urlopener, and placing in the data > to be processed back to the boolean value. Never attempted, > but the algorithm seems simpler than the dozens of solutions > method. > > > > > -- > Best Regards, > David Hutto > /*CEO:*/ _http://www.hitwebdevelopment.com_ > > > > > -- > Best Regards, > David Hutto > /*CEO:*/ _http://www.hitwebdevelopment.com_ > > In the email, I am sharing various code snippets to give others ideas and inspiration for coding. I that particular snippet, I am giving Python3 programmers the idea of making chess tags on a HTML or XML interpreter. It would be neat to type a tag that would generate chess pieces instead of remembering the HTML ASCII code. From my understanding, that email is not being displayed correctly. Are all of the lines run together? Thank you for asking. I want everyone to understand the purpose of the email and that particular snippet. Remember, assumption is the lowest form of knowledge. Mahalo, DCJ -------------- next part -------------- An HTML attachment was scrubbed... URL: From devyncjohnson at gmail.com Sat Jul 20 06:10:50 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Sat, 20 Jul 2013 06:10:50 -0400 Subject: Share Code Tips In-Reply-To: <51ea07a4$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <51e97e6e$0$29971$c3e8da3$5496439d@news.astraweb.com> <51E9B8EB.5060007@Gmail.com> <51ea07a4$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51EA622A.30102@Gmail.com> On 07/19/2013 11:44 PM, Steven D'Aprano wrote: > On Fri, 19 Jul 2013 21:04:55 -0400, Devyn Collier Johnson wrote: > >> In the future, I want to >> make the perfect international-case-insensitive if-statement. For now, >> my code only supports a limited range of characters. Even with casefold, >> I will have some issues as Chris Angelico mentioned. > There are hundreds of written languages in the world, with thousands of > characters, and most of them have rules about case-sensitivity and > character normalization. For example, in Greek, lowercase ? is ? except > at the end of a word, when it is ?. > > ??? '???'.upper() > '???' > ??? '???'.lower() > '???' > ??? '???'.casefold() > '???' > > > So in this case, casefold() correctly solves the problem, provided you > are comparing modern Greek text. But if you're comparing text in some > other language which merely happens to use Greek letters, but doesn't > have the same rules about letter sigma, then it will be inappropriate. So > you cannot write a single "perfect" case-insensitive comparison, the best > you can hope for is to write dozens or hundreds of separate case- > insensitive comparisons, one for each language or family of languages. > > For an introduction to the problem: > > http://www.w3.org/International/wiki/Case_folding > > http://www.unicode.org/faq/casemap_charprop.html > > > > >> Also, "?" is not really the same as "ss". > Sometimes it is. Sometimes it isn't. > > > Wow, my if-statement is so imperfect! Thankfully, only English people will talk to an English chatbot (I hope), so for my use of the code, it will work. Do the main Python3 developers plan to do something about this? Mahalo, DCJ From steve+comp.lang.python at pearwood.info Fri Jul 19 23:18:07 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 20 Jul 2013 03:18:07 GMT Subject: Share Code Tips References: <51e97e6e$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51ea016e$0$29971$c3e8da3$5496439d@news.astraweb.com> On Fri, 19 Jul 2013 18:08:43 -0400, Devyn Collier Johnson wrote: > As for the case-insensitive if-statements, most code uses Latin letters. > Making a case-insensitive-international if-statement would be > interesting. I can tackle that later. For now, I only wanted to take > care of Latin letters. I hope to figure something out for all > characters. As I showed, even for Latin letters, the trick of "if astring.lower() == bstring.lower()" doesn't *quite* work, although it can be "close enough" for some purposes. For example, some languages treat accents as mere guides to pronunciation, so ? == o, while other languages treat them as completely different letters. Same with ligatures: in modern English, ? should be treated as equal to ae, but in Old English, Danish, Norwegian and Icelandic it is a distinct letter. Case-insensitive testing may be easier in many non-European languages, because they don't have cases. A full solution to the problem of localized string matching requires expert knowledge for each language, but a 90% solution is pretty simple: astring.casefold() == bstring.casefold() or before version 3.3, just use lowercase. It's not a perfect solution, but it works reasonably well if you don't care about full localization. -- Steven From devyncjohnson at gmail.com Sat Jul 20 06:07:38 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Sat, 20 Jul 2013 06:07:38 -0400 Subject: Share Code Tips In-Reply-To: <51ea016e$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <51e97e6e$0$29971$c3e8da3$5496439d@news.astraweb.com> <51ea016e$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51EA616A.20609@Gmail.com> On 07/19/2013 11:18 PM, Steven D'Aprano wrote: > On Fri, 19 Jul 2013 18:08:43 -0400, Devyn Collier Johnson wrote: > >> As for the case-insensitive if-statements, most code uses Latin letters. >> Making a case-insensitive-international if-statement would be >> interesting. I can tackle that later. For now, I only wanted to take >> care of Latin letters. I hope to figure something out for all >> characters. > As I showed, even for Latin letters, the trick of "if astring.lower() == > bstring.lower()" doesn't *quite* work, although it can be "close enough" > for some purposes. For example, some languages treat accents as mere > guides to pronunciation, so ? == o, while other languages treat them as > completely different letters. Same with ligatures: in modern English, ? > should be treated as equal to ae, but in Old English, Danish, Norwegian > and Icelandic it is a distinct letter. > > Case-insensitive testing may be easier in many non-European languages, > because they don't have cases. > > A full solution to the problem of localized string matching requires > expert knowledge for each language, but a 90% solution is pretty simple: > > astring.casefold() == bstring.casefold() > > or before version 3.3, just use lowercase. It's not a perfect solution, > but it works reasonably well if you don't care about full localization. > > > Thanks for the tips. I am learning a lot from this mailing list. I hope my code helped some people though. Mahalo, DCJ From devyncjohnson at gmail.com Fri Jul 19 10:03:13 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Fri, 19 Jul 2013 10:03:13 -0400 Subject: Play Ogg Files Message-ID: <51E94721.3020906@Gmail.com> Aloha Python programmers! I am making a chatbot that I host on Launchpad.net/neobot. I am currently converting the engine from BASH code to Python3. I need to convert this for cross-platform compatibility. I do not need to use Mplayer; I just show the below code to give others a better idea what I am doing. I would prefer to be Python3 independent; I do not want to use the system shell. I am fine with using Python3 modules like Pygame (if there is a py3 module). As long as the code is fast, efficient, and simple without depending on the system shell or external apps, that would be nice. I also need the code to execute while the rest of the script continues running. jobs = multiprocessing.Process(SEND = subprocess.getoutput('mplayer -nogui -nolirc -noar -quiet ./conf/boot.ogg')) #Boot sound# Mahalo, Devyn Collier Johnson DevynCJohnson at Gmail.com From subhabangalore at gmail.com Fri Jul 19 16:58:20 2013 From: subhabangalore at gmail.com (subhabangalore at gmail.com) Date: Fri, 19 Jul 2013 13:58:20 -0700 (PDT) Subject: Topic Modeling LDA Gensim Message-ID: <2a0b9ce0-9751-4d3a-9859-5e8ec7924bcc@googlegroups.com> Dear Group, I am trying to use Gensim for Topic Modeling with LDA. I have trained LDA but now I want to test it with new documents. Should I use doc_lda = lda[doc_bow] or is it something else? If any one of the esteemed members of the group can kindly suggest? Thanking in Advance, Regards, Subhabrata From ms2597 at cornell.edu Fri Jul 19 18:56:24 2013 From: ms2597 at cornell.edu (cutems93) Date: Fri, 19 Jul 2013 15:56:24 -0700 (PDT) Subject: Python testing tools Message-ID: <402312e9-3952-48ff-99da-bd76856042e5@googlegroups.com> I am currently doing some research on testing software for Python. I found that there are many different types of testing tools. These are what I've found. 1.Unit test 2.Mock test 3.Fuzz test 4.Web test 5.Acceptance/business logic test 6.GUI test 7.Source code checking 8.Code coverage 9.Continuous integration 10.Automatic test runners 11.Test fixtures I know web and GUI testing tools are for specific uses. For instance, if you are not working with GUI or web pages, you don't need those testing tools. Other than these two, do you use all of the other nine testing tools? I think many of you are using unit testing tools, such as unittest and doctest, and source code checking tools, like pylint or pychecker. Do you guys use #2,3,5,8,9,10 and 11 often? Thanks! -Min S. From ben+python at benfinney.id.au Sat Jul 20 04:11:12 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 20 Jul 2013 18:11:12 +1000 Subject: Python testing tools References: <402312e9-3952-48ff-99da-bd76856042e5@googlegroups.com> Message-ID: <7w4nbp1win.fsf@benfinney.id.au> cutems93 writes: > I am currently doing some research on testing software for Python. I > found that there are many different types of testing tools. These are > what I've found. You will find these discussed at the Python Testing Tools Taxonomy . Hope that helps. From ms2597 at cornell.edu Tue Jul 23 13:51:50 2013 From: ms2597 at cornell.edu (cutems93) Date: Tue, 23 Jul 2013 10:51:50 -0700 (PDT) Subject: Python testing tools In-Reply-To: References: <402312e9-3952-48ff-99da-bd76856042e5@googlegroups.com> Message-ID: <9a7d2025-380a-43a9-add5-06691d77fc07@googlegroups.com> On Saturday, July 20, 2013 1:11:12 AM UTC-7, Ben Finney wrote: > cutems93 writes: > > > > > I am currently doing some research on testing software for Python. I > > > found that there are many different types of testing tools. These are > > > what I've found. > > > > You will find these discussed at the Python Testing Tools Taxonomy > > . > > > > Hope that helps. Thank you, but I already read this page before I posted this question. What I want to know is whether you personally use these tools other than unit testing tools. From skip at pobox.com Tue Jul 23 14:04:23 2013 From: skip at pobox.com (Skip Montanaro) Date: Tue, 23 Jul 2013 13:04:23 -0500 Subject: Python testing tools In-Reply-To: <9a7d2025-380a-43a9-add5-06691d77fc07@googlegroups.com> References: <402312e9-3952-48ff-99da-bd76856042e5@googlegroups.com> <9a7d2025-380a-43a9-add5-06691d77fc07@googlegroups.com> Message-ID: > Thank you, but I already read this page before I posted this question. What I want to > know is whether you personally use these tools other than unit testing tools. I tried using one of the mock tools a few years ago. I found it didn't fit my brain very well. (Maybe it was just me.) I use pylint all the time, and coverage from time-to-time, have used nose in the past, but not for my current stuff. All are worth your time. Skip From ms2597 at cornell.edu Tue Jul 23 14:12:03 2013 From: ms2597 at cornell.edu (cutems93) Date: Tue, 23 Jul 2013 11:12:03 -0700 (PDT) Subject: Python testing tools In-Reply-To: References: <402312e9-3952-48ff-99da-bd76856042e5@googlegroups.com> <9a7d2025-380a-43a9-add5-06691d77fc07@googlegroups.com> Message-ID: <2ff91ad7-a528-4a75-a4ea-5b6a9194a776@googlegroups.com> On Tuesday, July 23, 2013 11:04:23 AM UTC-7, Skip Montanaro wrote: > > Thank you, but I already read this page before I posted this question. What I want to > > > know is whether you personally use these tools other than unit testing tools. > > > > I tried using one of the mock tools a few years ago. I found it > > didn't fit my brain very well. (Maybe it was just me.) > > > > I use pylint all the time, and coverage from time-to-time, have used > > nose in the past, but not for my current stuff. All are worth your > > time. > > > > Skip Thank you! What tool do you use for coverage? And have you used pychecker? I heard it is as good as pylint. What do you think? -Min S. From skip at pobox.com Tue Jul 23 14:33:10 2013 From: skip at pobox.com (Skip Montanaro) Date: Tue, 23 Jul 2013 13:33:10 -0500 Subject: Python testing tools In-Reply-To: <2ff91ad7-a528-4a75-a4ea-5b6a9194a776@googlegroups.com> References: <402312e9-3952-48ff-99da-bd76856042e5@googlegroups.com> <9a7d2025-380a-43a9-add5-06691d77fc07@googlegroups.com> <2ff91ad7-a528-4a75-a4ea-5b6a9194a776@googlegroups.com> Message-ID: > Thank you! What tool do you use for coverage? coverage. :-) > And have you used pychecker? Yes, in fact, I used to use a wrapper script I wrote that ran both pylint and pychecker, then massaged the output into suitable-for-emacs-next-error-command > I heard it is as good as pylint. What do you think? They overlap a fair bit, but do somewhat different things. S From ms2597 at cornell.edu Tue Jul 23 14:56:17 2013 From: ms2597 at cornell.edu (cutems93) Date: Tue, 23 Jul 2013 11:56:17 -0700 (PDT) Subject: Python testing tools In-Reply-To: References: <402312e9-3952-48ff-99da-bd76856042e5@googlegroups.com> <9a7d2025-380a-43a9-add5-06691d77fc07@googlegroups.com> <2ff91ad7-a528-4a75-a4ea-5b6a9194a776@googlegroups.com> Message-ID: On Tuesday, July 23, 2013 11:33:10 AM UTC-7, Skip Montanaro wrote: > > Thank you! What tool do you use for coverage? > > > > coverage. :-) > > > > > And have you used pychecker? > > > > Yes, in fact, I used to use a wrapper script I wrote that ran both > > pylint and pychecker, then massaged the output into > > suitable-for-emacs-next-error-command > > > > > I heard it is as good as pylint. What do you think? > > > > They overlap a fair bit, but do somewhat different things. > > > > S Could you please elaborate on the difference of the two? I heard pylint does not import your source code when it is analyzing, while pychecker does. Does that make some difference? Moreover, do you personally like pylint or pycheker and why? Thank you!! -Min S. From skip at pobox.com Tue Jul 23 15:11:39 2013 From: skip at pobox.com (Skip Montanaro) Date: Tue, 23 Jul 2013 14:11:39 -0500 Subject: Python testing tools In-Reply-To: References: <402312e9-3952-48ff-99da-bd76856042e5@googlegroups.com> <9a7d2025-380a-43a9-add5-06691d77fc07@googlegroups.com> <2ff91ad7-a528-4a75-a4ea-5b6a9194a776@googlegroups.com> Message-ID: > Could you please elaborate on the difference of the two? I heard pylint > does not import your source code when it is analyzing, while pychecker does. > Does that make some difference? Moreover, do you personally like pylint or > pycheker and why? I haven't followed pychecker development for awhile. Pylint seems more actively maintained, though I could be wrong. The import issue is one significant difference, important if you are trying to check scripts which have side effects when imported. It's not a matter of like or not. I use what works and can easily be fit into the way I work. Both pylint and pychecker satisfy that constraint. S From ben+python at benfinney.id.au Tue Jul 23 18:51:00 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 24 Jul 2013 08:51:00 +1000 Subject: Python testing tools References: <402312e9-3952-48ff-99da-bd76856042e5@googlegroups.com> <9a7d2025-380a-43a9-add5-06691d77fc07@googlegroups.com> Message-ID: <7w38r4ao17.fsf@benfinney.id.au> cutems93 writes: > On Saturday, July 20, 2013 1:11:12 AM UTC-7, Ben Finney wrote: > > You will find these discussed at the Python Testing Tools Taxonomy > > . > > > > Hope that helps. > > Thank you, but I already read this page before I posted this question. (You will benefit from also reading and applying before using Google Groups. My advice: choose a different interface to this forum, Google Groups is terrible.) > What I want to know is whether you personally use these tools other > than unit testing tools. Yes, I do :-) What are you actually wanting to learn, beyond a collection of ?this is what I use? stories? -- \ ?The way to build large Python applications is to componentize | `\ and loosely-couple the hell out of everything.? ?Aahz | _o__) | Ben Finney From ms2597 at cornell.edu Tue Jul 30 20:18:40 2013 From: ms2597 at cornell.edu (cutems93) Date: Tue, 30 Jul 2013 17:18:40 -0700 (PDT) Subject: Python testing tools In-Reply-To: References: <402312e9-3952-48ff-99da-bd76856042e5@googlegroups.com> <9a7d2025-380a-43a9-add5-06691d77fc07@googlegroups.com> Message-ID: On Tuesday, July 23, 2013 3:51:00 PM UTC-7, Ben Finney wrote: > cutems93 writes: > > > > > On Saturday, July 20, 2013 1:11:12 AM UTC-7, Ben Finney wrote: > > > > You will find these discussed at the Python Testing Tools Taxonomy > > > > . > > > > > > > > Hope that helps. > > > > > > Thank you, but I already read this page before I posted this question. > > > > (You will benefit from also reading and applying > > before using Google > > Groups. My advice: choose a different interface to this forum, Google > > Groups is terrible.) > > > > > What I want to know is whether you personally use these tools other > > > than unit testing tools. > > > > Yes, I do :-) > > > > What are you actually wanting to learn, beyond a collection of ?this is > > what I use? stories? > > > > -- > > \ ?The way to build large Python applications is to componentize | > > `\ and loosely-couple the hell out of everything.? ?Aahz | > > _o__) | > > Ben Finney Sorry, I didn't notice that there are new replies. I want to know why you are using other software than unittest. Are those software more like "options" or "necessities" for you? Thank you! Min S. From srcmap.y.yen at gmail.com Wed Jul 24 18:02:28 2013 From: srcmap.y.yen at gmail.com (Y Yen) Date: Wed, 24 Jul 2013 15:02:28 -0700 (PDT) Subject: Python testing tools In-Reply-To: <402312e9-3952-48ff-99da-bd76856042e5@googlegroups.com> References: <402312e9-3952-48ff-99da-bd76856042e5@googlegroups.com> Message-ID: <9d2c755d-7dd5-4076-923a-7f38e4d1e655@googlegroups.com> On Friday, July 19, 2013 3:56:24 PM UTC-7, cutems93 wrote: > I am currently doing some research on testing software for Python. I found that there are many different types of testing tools. These are what I've found. > > > > 1.Unit test > > 2.Mock test > > 3.Fuzz test > > 4.Web test > > 5.Acceptance/business logic test > > 6.GUI test > > 7.Source code checking > > 8.Code coverage > > 9.Continuous integration > > 10.Automatic test runners > > 11.Test fixtures > > > > I know web and GUI testing tools are for specific uses. For instance, if you are not working with GUI or web pages, you don't need those testing tools. Other than these two, do you use all of the other nine testing tools? I think many of you are using unit testing tools, such as unittest and doctest, and source code checking tools, like pylint or pychecker. Do you guys use #2,3,5,8,9,10 and 11 often? > > > > Thanks! > > > > -Min S. I found the python's unittest framework lack good reporting tools. For my project (www.srcmap.com), I want something that does high level features integration tests, scriptable, generate report in HTML table format that label pass ->Green, fail -> Red. Separate by scripts, commands runs, etc. In python, it is so trivial to write you own. It parse the json test scripts and execute the commands inside one at a time. It works very well. From me at davecotter.com Fri Jul 19 19:52:39 2013 From: me at davecotter.com (David M. Cotter) Date: Fri, 19 Jul 2013 16:52:39 -0700 (PDT) Subject: how: embed + extend to control my running app? Message-ID: <6c1a5595-b2f7-4712-8c9c-be664dd8ed18@googlegroups.com> i'd like my app to be "available" to python while it's running. for example, say my app is "FooBar.app". when my FooBar.app is running, now there is a python interface available to python, and the user can write python scripts to make use of it. with their scripts, they can control my running application when FooBar.app is NOT running, perhaps making use of any of the python functions of "FooBar.app" would either return an error, or possibly launch "FooBar.app"? or do nothing since it's not running? can boost::python help with this? i've never worked with extending or embedding python, so any help would be super great From rosuav at gmail.com Fri Jul 19 21:04:15 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 20 Jul 2013 11:04:15 +1000 Subject: how: embed + extend to control my running app? In-Reply-To: <6c1a5595-b2f7-4712-8c9c-be664dd8ed18@googlegroups.com> References: <6c1a5595-b2f7-4712-8c9c-be664dd8ed18@googlegroups.com> Message-ID: On Sat, Jul 20, 2013 at 9:52 AM, David M. Cotter wrote: > i'd like my app to be "available" to python while it's running. > > for example, say my app is "FooBar.app". > > when my FooBar.app is running, now there is a python interface available to python, and the user can write python scripts to make use of it. > > with their scripts, they can control my running application > > when FooBar.app is NOT running, perhaps making use of any of the python functions of "FooBar.app" would either return an error, or possibly launch "FooBar.app"? or do nothing since it's not running? Interfacing C and Python like you suggest can't be done with embedding, because that requires that your app be already running, and probably not by having your code run as a pure module. In fact, I would actually suggest that you devise a separate protocol between your app and the Python script, and run them as separate processes. That way, they run independently, and you may (platform-specific code required here though) be able to invoke your app; most importantly, you'll be able to cleanly handle multiple Python scripts trying to control you simultaneously. What platforms are you aiming at? If it's just for Unix-like ones, the easiest way is probably to create a Unix domain socket, which the Python program can write to and/or read from. With a protocol based around simple operations like that, your Python module need not even involve C code - it simply opens a socket file and uses standard I/O methods on it. Alternatively, you may want to consider a TCP socket, which would let you split the client and server across a network, or possibly direct shared memory access. The world's your oyster. What kind of sauce would you like it with? ChrisA From me at davecotter.com Wed Jul 24 01:15:30 2013 From: me at davecotter.com (David M. Cotter) Date: Tue, 23 Jul 2013 22:15:30 -0700 (PDT) Subject: how: embed + extend to control my running app? In-Reply-To: References: <6c1a5595-b2f7-4712-8c9c-be664dd8ed18@googlegroups.com> Message-ID: <823607d0-9365-4dbb-b80c-8221bb514b01@googlegroups.com> i'm targeting Mac and Windows. Let's skip the thing about "it should work when my app isn't running", just assume it's going to be embedded, no pipes or sockets necessary. For Mac, I understand i need to "create" (?) a python.dylib, but i find no directions for that at the expected location: http://docs.python.org/2/extending/embedding.html is there some wiki page explaining how to create this for use in MacOS / Xcode? Now for Windows: same thing, i think i must create a .dll, right? Is there a tutorial for that? After that, i can link to these items, then in my C++ app, just #include "Python.h" and i've covered step 1. From greg.ewing at canterbury.ac.nz Wed Jul 24 06:53:57 2013 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Wed, 24 Jul 2013 22:53:57 +1200 Subject: how: embed + extend to control my running app? In-Reply-To: <823607d0-9365-4dbb-b80c-8221bb514b01@googlegroups.com> References: <6c1a5595-b2f7-4712-8c9c-be664dd8ed18@googlegroups.com> <823607d0-9365-4dbb-b80c-8221bb514b01@googlegroups.com> Message-ID: David M. Cotter wrote: > For Mac, I understand i need to "create" (?) a python.dylib, If your Python was installed as a framework, you should already have one. Just link your application with "-framework Python". > Now for Windows: same thing, i think i must create a .dll, right? Again, you should already have a python.dll in your installation somewhere. -- Greg From me at davecotter.com Wed Jul 24 12:10:28 2013 From: me at davecotter.com (David M. Cotter) Date: Wed, 24 Jul 2013 09:10:28 -0700 (PDT) Subject: how: embed + extend to control my running app? In-Reply-To: References: <6c1a5595-b2f7-4712-8c9c-be664dd8ed18@googlegroups.com> <823607d0-9365-4dbb-b80c-8221bb514b01@googlegroups.com> Message-ID: <54e0ad8a-2f27-445f-9abb-49addf107430@googlegroups.com> well, umm, gosh, now i feel quite silly. that was easy. okay that's done. next: i'd like to redirect the output of any "print" statements to my C function: > void Log(const unsigned char *utf8_cstrP); on the mac, python output sys.stdout goes into the debug console if you're in the debugger, and to the "console app" if not. On windows, i don't think it goes anywhere at all? So: i really want it to go to my own log file (via my Log() function). now, can i specify "please output to this FILE*" ?, i looked at all the python c headers but found nothing about redirecting the output. I see "PySys_GetFile()" which will get what it points to, but what i want is a "PySys_SetFile()" so i can set it. the only alternative seems to be: > PyObject *logObjectP = create ???; > > ERR(PySys_SetObject("stdout", logObjectP)); if that's the only way, how to create the logObjectP such that it redirects the write() python function to my Log() C function? i tried this: -------------------------------- const char *s_printFunc = "import sys\n" "class CustomPrint():\n" " def __init__(self):\n" " self.old_stdout=sys.stdout\n" "\n" " def write(self, text):\n" " self.old_stdout.write('foobar')\n" " text = text.rstrip()\n" " if len(text) == 0:\n" " return\n" " self.old_stdout.write('custom Print--->' + text + '\n')\n"; OSStatus CPython_PreAlloc(const char *utf8Z) { OSStatus err = noErr; PyCompilerFlags flags; PyObject *logObjectP = NULL; Py_SetProgramName(const_cast(utf8Z)); Py_Initialize(); flags.cf_flags = PyCF_SOURCE_IS_UTF8; logObjectP = Py_CompileStringFlags(s_printFunc, "CustomPrint", Py_single_input, &flags); ERR_NULL(logObjectP, tsmUnsupScriptLanguageErr); if (!err) { ERR(PySys_SetObject("stdout", logObjectP)); ERR(PySys_SetObject("stderr", logObjectP)); Py_DECREF(logObjectP); } return err; } void CPython_PostDispose() { Py_Finalize(); } void CPython_Test() { PyRun_SimpleString( "from time import time, ctime\n" "print 'Today is', ctime(time())\n"); } ----------------------------------------- and when i run CPython_Test(), there is no output at all. If i comment out the entire Py_CompileStringFlags() line, then the output works fine (going to stdout as expected), so i'm not sure what i'm doing wrong From rosuav at gmail.com Wed Jul 24 12:25:47 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 02:25:47 +1000 Subject: how: embed + extend to control my running app? In-Reply-To: <54e0ad8a-2f27-445f-9abb-49addf107430@googlegroups.com> References: <6c1a5595-b2f7-4712-8c9c-be664dd8ed18@googlegroups.com> <823607d0-9365-4dbb-b80c-8221bb514b01@googlegroups.com> <54e0ad8a-2f27-445f-9abb-49addf107430@googlegroups.com> Message-ID: On Thu, Jul 25, 2013 at 2:10 AM, David M. Cotter wrote: > So: i really want it to go to my own log file (via my Log() function). now, can i specify "please output to this FILE*" ?, i looked at all the python c headers but found nothing about redirecting the output. Are you able to simply redirect the OS-level stdout handle, or would that disrupt your own code? That might be an easier way to log to a file. ChrisA From me at davecotter.com Wed Jul 24 13:40:03 2013 From: me at davecotter.com (David M. Cotter) Date: Wed, 24 Jul 2013 10:40:03 -0700 (PDT) Subject: how: embed + extend to control my running app? In-Reply-To: References: <6c1a5595-b2f7-4712-8c9c-be664dd8ed18@googlegroups.com> <823607d0-9365-4dbb-b80c-8221bb514b01@googlegroups.com> <54e0ad8a-2f27-445f-9abb-49addf107430@googlegroups.com> Message-ID: i don't use stdout in my own code, my code goes to my own log file. i want the output from any python code to go to my existing log file, so log statements from my app and any python code are intermingled in that one file. my updated code is here, which now bridges my python print function to my C function: http://karaoke.kjams.com/wiki/Python but it seems that my custom "s_printFunc" is never called ? From me at davecotter.com Wed Jul 24 15:35:12 2013 From: me at davecotter.com (David M. Cotter) Date: Wed, 24 Jul 2013 12:35:12 -0700 (PDT) Subject: how: embed + extend to control my running app? In-Reply-To: References: <6c1a5595-b2f7-4712-8c9c-be664dd8ed18@googlegroups.com> <823607d0-9365-4dbb-b80c-8221bb514b01@googlegroups.com> <54e0ad8a-2f27-445f-9abb-49addf107430@googlegroups.com> Message-ID: <90fb39b6-747e-4248-9c05-3c48fc41b631@googlegroups.com> > http://karaoke.kjams.com/wiki/Python nevermind, i got it, it's working now (see link for code) From me at davecotter.com Wed Jul 24 16:37:46 2013 From: me at davecotter.com (David M. Cotter) Date: Wed, 24 Jul 2013 13:37:46 -0700 (PDT) Subject: how: embed + extend to control my running app? In-Reply-To: References: <6c1a5595-b2f7-4712-8c9c-be664dd8ed18@googlegroups.com> <823607d0-9365-4dbb-b80c-8221bb514b01@googlegroups.com> Message-ID: <522e7d68-6c26-4036-9389-bc780d792f86@googlegroups.com> > > Now for Windows: same thing, i think i must create a .dll, right? > you should already have a python.dll in your installation i can find "python27.lib" in the "libs" folder, but there is no "python27_d.lib", and there is no "python27.dll" in the DLLs folder? are there instructions for creating (or finding) these for Windows? From me at davecotter.com Wed Jul 24 20:51:45 2013 From: me at davecotter.com (David M. Cotter) Date: Wed, 24 Jul 2013 17:51:45 -0700 (PDT) Subject: how: embed + extend to control my running app? In-Reply-To: <522e7d68-6c26-4036-9389-bc780d792f86@googlegroups.com> References: <6c1a5595-b2f7-4712-8c9c-be664dd8ed18@googlegroups.com> <823607d0-9365-4dbb-b80c-8221bb514b01@googlegroups.com> <522e7d68-6c26-4036-9389-bc780d792f86@googlegroups.com> Message-ID: update: okay so the python27.dll is in /windows/system32 so ignore that i've set my include directory correct, so i can compile i've set my "additional libraries" directory to the "libs" directory (where the ".lib" files are. (note: NOT including "Lib" directory, cuz that's full of .py files and folders) (note: NOT including "DLLs" directory, cuz, why would i?) No need to specify "additional dependencies" for the .lib file, cuz the pyconfig.h file does that. but there is no "python27_d.dll" anywhere to be found, so i hacked pyconfig.h to get rid of the "_d". so it all compiles. but it won't link: LNK2001: unresolved external symbol __imp___Py_RefTotal LNK2001: unresolved external symbol __imp___Py_NoneStruct LNK2019: unresolved external symbol __imp__PyArg_ParseTuple LNK2019: unresolved external symbol __imp__PyFloat_FromDouble LNK2019: unresolved external symbol __imp__PyString_FromString LNK2019: unresolved external symbol __imp__PyRun_SimpleStringFlags LNK2019: unresolved external symbol __imp__Py_InitModule4TraceRefs LNK2019: unresolved external symbol __imp__Py_Initialize LNK2019: unresolved external symbol __imp__Py_SetProgramName LNK2019: unresolved external symbol __imp__Py_Finalize LNK2019: unresolved external symbol __imp__PyRun_SimpleFileExFlags what, pray tell, am i doing wrong? *hopeful face* From davea at davea.name Wed Jul 24 21:14:34 2013 From: davea at davea.name (Dave Angel) Date: Wed, 24 Jul 2013 21:14:34 -0400 Subject: how: embed + extend to control my running app? In-Reply-To: References: <6c1a5595-b2f7-4712-8c9c-be664dd8ed18@googlegroups.com> <823607d0-9365-4dbb-b80c-8221bb514b01@googlegroups.com> <522e7d68-6c26-4036-9389-bc780d792f86@googlegroups.com> Message-ID: On 07/24/2013 08:51 PM, David M. Cotter wrote: > update: okay so the python27.dll is in /windows/system32 so ignore that > > i've set my include directory correct, so i can compile > > i've set my "additional libraries" directory to the "libs" directory (where the ".lib" files are. (note: NOT including "Lib" directory, cuz that's full of .py files and folders) (note: NOT including "DLLs" directory, cuz, why would i?) > > No need to specify "additional dependencies" for the .lib file, cuz the pyconfig.h file does that. > > but there is no "python27_d.dll" anywhere to be found, so i hacked pyconfig.h to get rid of the "_d". > > so it all compiles. > > but it won't link: > > LNK2001: unresolved external symbol __imp___Py_RefTotal > LNK2001: unresolved external symbol __imp___Py_NoneStruct > LNK2019: unresolved external symbol __imp__PyArg_ParseTuple > LNK2019: unresolved external symbol __imp__PyFloat_FromDouble > LNK2019: unresolved external symbol __imp__PyString_FromString > LNK2019: unresolved external symbol __imp__PyRun_SimpleStringFlags > LNK2019: unresolved external symbol __imp__Py_InitModule4TraceRefs > LNK2019: unresolved external symbol __imp__Py_Initialize > LNK2019: unresolved external symbol __imp__Py_SetProgramName > LNK2019: unresolved external symbol __imp__Py_Finalize > LNK2019: unresolved external symbol __imp__PyRun_SimpleFileExFlags > > what, pray tell, am i doing wrong? *hopeful face* > Digging *far* back in my Windows memory, those look like imports. You probably need the import lib for the Python.dll. Probably called somethng like python.lib. You could check that by doing a dumpbin of python.dll and searching for those entry points. An import lib in Windows simply tells the linker that those symbols will be resolved at runtime, and from a particular dll. They can also change the names to be used (removing the __imp__ prefix) and even specify a numeric entry point (to slow down people who reverse engineer these things). If I recall right, there's a way in Microsoft's toolset to create an import lib from a dll, assuming the dll doesn't restrict itself to those numeric thingies. As for the _d suffix, that's commonly used to specify debug versions of things. They would have extra symbol information, and less optimized code so that it's easier to use a debugger on them. -- DaveA From me at davecotter.com Thu Jul 25 14:24:43 2013 From: me at davecotter.com (David M. Cotter) Date: Thu, 25 Jul 2013 11:24:43 -0700 (PDT) Subject: how: embed + extend to control my running app? In-Reply-To: References: <6c1a5595-b2f7-4712-8c9c-be664dd8ed18@googlegroups.com> <823607d0-9365-4dbb-b80c-8221bb514b01@googlegroups.com> <522e7d68-6c26-4036-9389-bc780d792f86@googlegroups.com> Message-ID: Okay the link problem was solved: i had installed a 64bit python and my app is 32bit. i'm using ActivePython installer from here: http://www.activestate.com/activepython/downloads it seems that now the problem is that this does not install the _d versions of the .lib. :( does anyone know how to get or create the _d version of the .lib out of the ActivePtyon installation? From devyncjohnson at gmail.com Fri Jul 19 21:06:38 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Fri, 19 Jul 2013 21:06:38 -0400 Subject: Play Ogg Files Message-ID: <51E9E29E.7010203@Gmail.com> Aloha Python programmers! I am making a chatbot that I host on Launchpad.net/neobot. I am currently converting the engine from BASH code to Python3. I need to convert this for cross-platform compatibility. I do not need to use Mplayer; I just show the below code to give others a better idea what I am doing. I would prefer to be Python3 independent; I do not want to use the system shell. I am fine with using Python3 modules like Pygame (if there is a py3 module). As long as the code is fast, efficient, and simple without depending on the system shell or external apps, that would be nice. I also need the code to execute while the rest of the script continues running. jobs = multiprocessing.Process(SEND = subprocess.getoutput('mplayer -nogui -nolirc -noar -quiet ./conf/boot.ogg')) #Boot sound# Mahalo, Devyn Collier Johnson DevynCJohnson at Gmail.com From stefan_ml at behnel.de Sat Jul 20 00:21:22 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 20 Jul 2013 06:21:22 +0200 Subject: Play Ogg Files In-Reply-To: <51E9E29E.7010203@Gmail.com> References: <51E9E29E.7010203@Gmail.com> Message-ID: Devyn Collier Johnson, 20.07.2013 03:06: > I am making a chatbot that I host on Launchpad.net/neobot. I am currently > converting the engine from BASH code to Python3. I need to convert this for > cross-platform compatibility. I do not need to use Mplayer; I just show the > below code to give others a better idea what I am doing. I would prefer to > be Python3 independent; I do not want to use the system shell. I am fine > with using Python3 modules like Pygame (if there is a py3 module). As long > as the code is fast, efficient, and simple without depending on the system > shell or external apps, that would be nice. I also need the code to execute > while the rest of the script continues running. > > jobs = multiprocessing.Process(SEND = subprocess.getoutput('mplayer > -nogui -nolirc -noar -quiet ./conf/boot.ogg')) #Boot sound# Well, since you mentioned it already, have you actually looked at pygame? It should be able to do what you want. There's also pyaudio, which is more specialised to, well, audio. A web search for python and ogg might provide more. Stefan From dwightdhutto at gmail.com Sat Jul 20 00:39:03 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Sat, 20 Jul 2013 00:39:03 -0400 Subject: Play Ogg Files In-Reply-To: References: <51E9E29E.7010203@Gmail.com> Message-ID: you could use , and I think its david at david:~$ python Python 2.7.3 (default, Aug 1 2012, 05:16:07) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import subprocess >>> subprocess.call(['espeak', 'word_spoken'], stdin=None, stdout=None, stderr=None, shell=False) This is on ubuntu linux, using espeak. this is on ubun On Sat, Jul 20, 2013 at 12:21 AM, Stefan Behnel wrote: > Devyn Collier Johnson, 20.07.2013 03:06: > > I am making a chatbot that I host on Launchpad.net/neobot. I am currently > > converting the engine from BASH code to Python3. I need to convert this > for > > cross-platform compatibility. I do not need to use Mplayer; I just show > the > > below code to give others a better idea what I am doing. I would prefer > to > > be Python3 independent; I do not want to use the system shell. I am fine > > with using Python3 modules like Pygame (if there is a py3 module). As > long > > as the code is fast, efficient, and simple without depending on the > system > > shell or external apps, that would be nice. I also need the code to > execute > > while the rest of the script continues running. > > > > jobs = multiprocessing.Process(SEND = subprocess.getoutput('mplayer > > -nogui -nolirc -noar -quiet ./conf/boot.ogg')) #Boot sound# > > Well, since you mentioned it already, have you actually looked at pygame? > It should be able to do what you want. There's also pyaudio, which is more > specialised to, well, audio. A web search for python and ogg might provide > more. > > Stefan > > > -- > http://mail.python.org/mailman/listinfo/python-list > -- Best Regards, David Hutto *CEO:* *http://www.hitwebdevelopment.com* -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwightdhutto at gmail.com Sat Jul 20 00:52:41 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Sat, 20 Jul 2013 00:52:41 -0400 Subject: Play Ogg Files In-Reply-To: References: <51E9E29E.7010203@Gmail.com> Message-ID: This is using 3.2, which shouldn't be far off, the latest I could get on ubuntu. david at david:~$ python3.2 Python 3.2.3 (default, Apr 10 2013, 05:29:11) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import subprocess >>> subprocess.call(['espeak', 'word_spoken'], stdin=None, stdout=None, stderr=None, shell=False) On Sat, Jul 20, 2013 at 12:39 AM, David Hutto wrote: > you could use , and I think its > > david at david:~$ python > Python 2.7.3 (default, Aug 1 2012, 05:16:07) > [GCC 4.6.3] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import subprocess > >>> subprocess.call(['espeak', 'word_spoken'], stdin=None, stdout=None, > stderr=None, shell=False) > > > > This is on ubuntu linux, using espeak. > this is on ubun > > > On Sat, Jul 20, 2013 at 12:21 AM, Stefan Behnel wrote: > >> Devyn Collier Johnson, 20.07.2013 03:06: >> > I am making a chatbot that I host on Launchpad.net/neobot. I am >> currently >> > converting the engine from BASH code to Python3. I need to convert this >> for >> > cross-platform compatibility. I do not need to use Mplayer; I just show >> the >> > below code to give others a better idea what I am doing. I would prefer >> to >> > be Python3 independent; I do not want to use the system shell. I am fine >> > with using Python3 modules like Pygame (if there is a py3 module). As >> long >> > as the code is fast, efficient, and simple without depending on the >> system >> > shell or external apps, that would be nice. I also need the code to >> execute >> > while the rest of the script continues running. >> > >> > jobs = multiprocessing.Process(SEND = subprocess.getoutput('mplayer >> > -nogui -nolirc -noar -quiet ./conf/boot.ogg')) #Boot sound# >> >> Well, since you mentioned it already, have you actually looked at pygame? >> It should be able to do what you want. There's also pyaudio, which is more >> specialised to, well, audio. A web search for python and ogg might provide >> more. >> >> Stefan >> >> >> -- >> http://mail.python.org/mailman/listinfo/python-list >> > > > > -- > Best Regards, > David Hutto > *CEO:* *http://www.hitwebdevelopment.com* > -- Best Regards, David Hutto *CEO:* *http://www.hitwebdevelopment.com* -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwightdhutto at gmail.com Sat Jul 20 00:59:59 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Sat, 20 Jul 2013 00:59:59 -0400 Subject: Play Ogg Files In-Reply-To: References: <51E9E29E.7010203@Gmail.com> Message-ID: Just get a good dictionary, and distutils I believe, someone a little bit more experienced in these should be along soon, or use the manual, and docs. On Sat, Jul 20, 2013 at 12:52 AM, David Hutto wrote: > This is using 3.2, which shouldn't be far off, the latest I could get on > ubuntu. > > david at david:~$ python3.2 > Python 3.2.3 (default, Apr 10 2013, 05:29:11) > [GCC 4.6.3] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import subprocess > >>> subprocess.call(['espeak', 'word_spoken'], stdin=None, stdout=None, > stderr=None, shell=False) > > > On Sat, Jul 20, 2013 at 12:39 AM, David Hutto wrote: > >> you could use , and I think its >> >> david at david:~$ python >> Python 2.7.3 (default, Aug 1 2012, 05:16:07) >> [GCC 4.6.3] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import subprocess >> >>> subprocess.call(['espeak', 'word_spoken'], stdin=None, stdout=None, >> stderr=None, shell=False) >> >> >> >> This is on ubuntu linux, using espeak. >> this is on ubun >> >> >> On Sat, Jul 20, 2013 at 12:21 AM, Stefan Behnel wrote: >> >>> Devyn Collier Johnson, 20.07.2013 03:06: >>> > I am making a chatbot that I host on Launchpad.net/neobot. I am >>> currently >>> > converting the engine from BASH code to Python3. I need to convert >>> this for >>> > cross-platform compatibility. I do not need to use Mplayer; I just >>> show the >>> > below code to give others a better idea what I am doing. I would >>> prefer to >>> > be Python3 independent; I do not want to use the system shell. I am >>> fine >>> > with using Python3 modules like Pygame (if there is a py3 module). As >>> long >>> > as the code is fast, efficient, and simple without depending on the >>> system >>> > shell or external apps, that would be nice. I also need the code to >>> execute >>> > while the rest of the script continues running. >>> > >>> > jobs = multiprocessing.Process(SEND = subprocess.getoutput('mplayer >>> > -nogui -nolirc -noar -quiet ./conf/boot.ogg')) #Boot sound# >>> >>> Well, since you mentioned it already, have you actually looked at pygame? >>> It should be able to do what you want. There's also pyaudio, which is >>> more >>> specialised to, well, audio. A web search for python and ogg might >>> provide >>> more. >>> >>> Stefan >>> >>> >>> -- >>> http://mail.python.org/mailman/listinfo/python-list >>> >> >> >> >> -- >> Best Regards, >> David Hutto >> *CEO:* *http://www.hitwebdevelopment.com* >> > > > > -- > Best Regards, > David Hutto > *CEO:* *http://www.hitwebdevelopment.com* > -- Best Regards, David Hutto *CEO:* *http://www.hitwebdevelopment.com* -------------- next part -------------- An HTML attachment was scrubbed... URL: From devyncjohnson at gmail.com Sat Jul 20 08:39:14 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Sat, 20 Jul 2013 08:39:14 -0400 Subject: Play Ogg Files In-Reply-To: References: <51E9E29E.7010203@Gmail.com> Message-ID: <51EA84F2.6080702@Gmail.com> On 07/20/2013 12:39 AM, David Hutto wrote: > you could use , and I think its > > david at david:~$ python > Python 2.7.3 (default, Aug 1 2012, 05:16:07) > [GCC 4.6.3] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import subprocess > >>> subprocess.call(['espeak', 'word_spoken'], stdin=None, > stdout=None, stderr=None, shell=False) > > > > This is on ubuntu linux, using espeak. > this is on ubun > > > On Sat, Jul 20, 2013 at 12:21 AM, Stefan Behnel > wrote: > > Devyn Collier Johnson, 20.07.2013 03:06: > > I am making a chatbot that I host on Launchpad.net/neobot. I am > currently > > converting the engine from BASH code to Python3. I need to > convert this for > > cross-platform compatibility. I do not need to use Mplayer; I > just show the > > below code to give others a better idea what I am doing. I would > prefer to > > be Python3 independent; I do not want to use the system shell. I > am fine > > with using Python3 modules like Pygame (if there is a py3 > module). As long > > as the code is fast, efficient, and simple without depending on > the system > > shell or external apps, that would be nice. I also need the code > to execute > > while the rest of the script continues running. > > > > jobs = multiprocessing.Process(SEND = > subprocess.getoutput('mplayer > > -nogui -nolirc -noar -quiet ./conf/boot.ogg')) #Boot sound# > > Well, since you mentioned it already, have you actually looked at > pygame? > It should be able to do what you want. There's also pyaudio, which > is more > specialised to, well, audio. A web search for python and ogg might > provide > more. > > Stefan > > > -- > http://mail.python.org/mailman/listinfo/python-list > > > > > -- > Best Regards, > David Hutto > /*CEO:*/ _http://www.hitwebdevelopment.com_ > > Where did Espeak come from? This is about playing an ogg file using Python3 code. DCJ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwightdhutto at gmail.com Sun Jul 21 01:32:05 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Sun, 21 Jul 2013 01:32:05 -0400 Subject: Play Ogg Files In-Reply-To: <51EA84F2.6080702@Gmail.com> References: <51E9E29E.7010203@Gmail.com> <51EA84F2.6080702@Gmail.com> Message-ID: It was supposed to show you that you can use a command line function from windows or linux that will play an ogg/.wav file/ etc with an if windows: do this or if linux do this. espeak was just a suggestion, unless you want your own voice played for the chatbot, or a selection of a male or female voice, and why ogg, why not wav? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwightdhutto at gmail.com Sun Jul 21 01:39:37 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Sun, 21 Jul 2013 01:39:37 -0400 Subject: Play Ogg Files In-Reply-To: References: <51E9E29E.7010203@Gmail.com> <51EA84F2.6080702@Gmail.com> Message-ID: With linux you can have your package listed in synaptic, and can use with a sudo apt-get install whatever ogg player like ogg123, and windows I don't work with that much, but I'm pretty sure I've played .wav files from the command line before while working with cross platform just for practice, so with python 3 you can use what's available in the system with an if command. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Jul 21 01:50:23 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 21 Jul 2013 15:50:23 +1000 Subject: Play Ogg Files In-Reply-To: References: <51E9E29E.7010203@Gmail.com> <51EA84F2.6080702@Gmail.com> Message-ID: On Sun, Jul 21, 2013 at 3:39 PM, David Hutto wrote: > With linux you can have your package listed in synaptic, and can use with a > sudo apt-get install whatever ogg player like ogg123, and windows I don't > work with that much, but I'm pretty sure I've played .wav files from the > command line before while working with cross platform just for practice, so > with python 3 you can use what's available in the system with an if command. Correction: "With Debian-based Linux distributions, you can etc etc" - aptitude is Debian's package manager, it's not something you'll find on other Linuxes. And the exact packages available depend on your repositories; again, most Debian-derived Linux distros will most likely have ogg123, but it's not guaranteed. However, it's reasonably likely that other package managers and repositories will have what you're looking for. ChrisA From dwightdhutto at gmail.com Sun Jul 21 02:07:38 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Sun, 21 Jul 2013 02:07:38 -0400 Subject: Play Ogg Files In-Reply-To: References: <51E9E29E.7010203@Gmail.com> <51EA84F2.6080702@Gmail.com> Message-ID: Yeah, its like yum used in others(or the point and click gui package installers). The main point kind of is in cross platform it would seem that you would just use what's available with try/except, or if statements, and the question is what os's is he going for. Then a simple usage of what's available in the command line for those os's. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Jul 21 02:29:20 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 21 Jul 2013 16:29:20 +1000 Subject: Play Ogg Files In-Reply-To: References: <51E9E29E.7010203@Gmail.com> <51EA84F2.6080702@Gmail.com> Message-ID: On Sun, Jul 21, 2013 at 4:07 PM, David Hutto wrote: > Yeah, its like yum used in others(or the point and click gui package > installers). The main point kind of is in cross platform it would seem that > you would just use what's available with try/except, or if statements, and > the question is what os's is he going for. Right, it's just safer to avoid annoying the Red Hat people by pretending the whole world is Debian :) ChrisA From dwightdhutto at gmail.com Sun Jul 21 02:42:12 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Sun, 21 Jul 2013 02:42:12 -0400 Subject: Play Ogg Files In-Reply-To: References: <51E9E29E.7010203@Gmail.com> <51EA84F2.6080702@Gmail.com> Message-ID: Mainly I just use my apps for my own purposes. So it's usually on the debian/ubuntu distro I have, although I do have Windows XP SP3 in virtual box. I have been meaning to install some other linux distros in virtual box that are the main ones, percentage of utilization based, that are used, and practice more with the cross platform, but haven't had a chance to lately. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwightdhutto at gmail.com Sun Jul 21 02:47:48 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Sun, 21 Jul 2013 02:47:48 -0400 Subject: Play Ogg Files In-Reply-To: References: <51E9E29E.7010203@Gmail.com> <51EA84F2.6080702@Gmail.com> Message-ID: And just thinking about it,,, Devyn might want to install virtual box, and place in the os's he wants to target first, because there's usually always a glitch in 3rd party imports, This is especially true when using a newer version of python that the other developers of packages you can import from haven't had time to update them, as well as update for os's that might have changed, and everyone has a newer version. On Sun, Jul 21, 2013 at 2:42 AM, David Hutto wrote: > Mainly I just use my apps for my own purposes. So it's usually on the > debian/ubuntu distro I have, although I do have Windows XP SP3 in virtual > box. > > I have been meaning to install some other linux distros in virtual box > that are the main ones, percentage of utilization based, that are used, and > practice more with the cross platform, but haven't had a chance to lately. > > > -- Best Regards, David Hutto *CEO:* *http://www.hitwebdevelopment.com* -------------- next part -------------- An HTML attachment was scrubbed... URL: From devyncjohnson at gmail.com Mon Jul 22 09:04:29 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Mon, 22 Jul 2013 09:04:29 -0400 Subject: Play Ogg Files In-Reply-To: References: <51E9E29E.7010203@Gmail.com> <51EA84F2.6080702@Gmail.com> Message-ID: <51ED2DDD.8070503@Gmail.com> On 07/21/2013 01:50 AM, Chris Angelico wrote: > On Sun, Jul 21, 2013 at 3:39 PM, David Hutto wrote: >> With linux you can have your package listed in synaptic, and can use with a >> sudo apt-get install whatever ogg player like ogg123, and windows I don't >> work with that much, but I'm pretty sure I've played .wav files from the >> command line before while working with cross platform just for practice, so >> with python 3 you can use what's available in the system with an if command. > Correction: "With Debian-based Linux distributions, you can etc etc" - > aptitude is Debian's package manager, it's not something you'll find > on other Linuxes. And the exact packages available depend on your > repositories; again, most Debian-derived Linux distros will most > likely have ogg123, but it's not guaranteed. However, it's reasonably > likely that other package managers and repositories will have what > you're looking for. > > ChrisA Good point, Chris. I do not want to make coding specifically for each package manager for many of the different distros. I follow the KISS princlple (Keep It Simple Stupid) when programming. Making a cross-platform/universal way to play an ogg file is better than being specific for each individual system. Remember KISS. Mahalo, DCJ From devyncjohnson at gmail.com Sat Jul 20 08:25:12 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Sat, 20 Jul 2013 08:25:12 -0400 Subject: Play Ogg Files In-Reply-To: References: <51E9E29E.7010203@Gmail.com> Message-ID: <51EA81A8.9060608@Gmail.com> On 07/20/2013 12:21 AM, Stefan Behnel wrote: > Devyn Collier Johnson, 20.07.2013 03:06: >> I am making a chatbot that I host on Launchpad.net/neobot. I am currently >> converting the engine from BASH code to Python3. I need to convert this for >> cross-platform compatibility. I do not need to use Mplayer; I just show the >> below code to give others a better idea what I am doing. I would prefer to >> be Python3 independent; I do not want to use the system shell. I am fine >> with using Python3 modules like Pygame (if there is a py3 module). As long >> as the code is fast, efficient, and simple without depending on the system >> shell or external apps, that would be nice. I also need the code to execute >> while the rest of the script continues running. >> >> jobs = multiprocessing.Process(SEND = subprocess.getoutput('mplayer >> -nogui -nolirc -noar -quiet ./conf/boot.ogg')) #Boot sound# > Well, since you mentioned it already, have you actually looked at pygame? > It should be able to do what you want. There's also pyaudio, which is more > specialised to, well, audio. A web search for python and ogg might provide > more. > > Stefan > > Thanks Stefan! I have not heard of Pyaudio; I will look into that. As for Pygame, I have not been able to find any good documentation for playing audio files. Plus, I recently learned that Pygame is not Python3 compatible. Mahalo, DCJ From stefan_ml at behnel.de Sun Jul 21 10:10:16 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 21 Jul 2013 16:10:16 +0200 Subject: Play Ogg Files In-Reply-To: <51EA81A8.9060608@Gmail.com> References: <51E9E29E.7010203@Gmail.com> <51EA81A8.9060608@Gmail.com> Message-ID: Devyn Collier Johnson, 20.07.2013 14:25: > On 07/20/2013 12:21 AM, Stefan Behnel wrote: >> Devyn Collier Johnson, 20.07.2013 03:06: >>> I am making a chatbot that I host on Launchpad.net/neobot. I am currently >>> converting the engine from BASH code to Python3. I need to convert this for >>> cross-platform compatibility. I do not need to use Mplayer; I just show the >>> below code to give others a better idea what I am doing. I would prefer to >>> be Python3 independent; I do not want to use the system shell. I am fine >>> with using Python3 modules like Pygame (if there is a py3 module). As long >>> as the code is fast, efficient, and simple without depending on the system >>> shell or external apps, that would be nice. I also need the code to execute >>> while the rest of the script continues running. >>> >>> jobs = multiprocessing.Process(SEND = subprocess.getoutput('mplayer >>> -nogui -nolirc -noar -quiet ./conf/boot.ogg')) #Boot sound# >> Well, since you mentioned it already, have you actually looked at pygame? >> It should be able to do what you want. There's also pyaudio, which is more >> specialised to, well, audio. A web search for python and ogg might provide >> more. > > Thanks Stefan! I have not heard of Pyaudio; I will look into that. As for > Pygame, I have not been able to find any good documentation for playing > audio files. A quick duckduckgo search gave me this, at least: http://www.pygame.org/docs/ref/mixer.html > Plus, I recently learned that Pygame is not Python3 compatible. Looks like it's your lucky day: http://www.pygame.org/wiki/FrequentlyAskedQuestions#Does%20Pygame%20work%20with%20Python%203? Stefan From devyncjohnson at gmail.com Mon Jul 22 08:55:38 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Mon, 22 Jul 2013 08:55:38 -0400 Subject: Play Ogg Files In-Reply-To: References: <51E9E29E.7010203@Gmail.com> <51EA81A8.9060608@Gmail.com> Message-ID: <51ED2BCA.8020403@Gmail.com> On 07/21/2013 10:10 AM, Stefan Behnel wrote: > Devyn Collier Johnson, 20.07.2013 14:25: >> On 07/20/2013 12:21 AM, Stefan Behnel wrote: >>> Devyn Collier Johnson, 20.07.2013 03:06: >>>> I am making a chatbot that I host on Launchpad.net/neobot. I am currently >>>> converting the engine from BASH code to Python3. I need to convert this for >>>> cross-platform compatibility. I do not need to use Mplayer; I just show the >>>> below code to give others a better idea what I am doing. I would prefer to >>>> be Python3 independent; I do not want to use the system shell. I am fine >>>> with using Python3 modules like Pygame (if there is a py3 module). As long >>>> as the code is fast, efficient, and simple without depending on the system >>>> shell or external apps, that would be nice. I also need the code to execute >>>> while the rest of the script continues running. >>>> >>>> jobs = multiprocessing.Process(SEND = subprocess.getoutput('mplayer >>>> -nogui -nolirc -noar -quiet ./conf/boot.ogg')) #Boot sound# >>> Well, since you mentioned it already, have you actually looked at pygame? >>> It should be able to do what you want. There's also pyaudio, which is more >>> specialised to, well, audio. A web search for python and ogg might provide >>> more. >> Thanks Stefan! I have not heard of Pyaudio; I will look into that. As for >> Pygame, I have not been able to find any good documentation for playing >> audio files. > A quick duckduckgo search gave me this, at least: > > http://www.pygame.org/docs/ref/mixer.html > > >> Plus, I recently learned that Pygame is not Python3 compatible. > Looks like it's your lucky day: > > http://www.pygame.org/wiki/FrequentlyAskedQuestions#Does%20Pygame%20work%20with%20Python%203? > > Stefan > > Thanks! DCJ From wuwei23 at gmail.com Mon Jul 22 20:37:03 2013 From: wuwei23 at gmail.com (alex23) Date: Tue, 23 Jul 2013 10:37:03 +1000 Subject: Play Ogg Files In-Reply-To: References: <51E9E29E.7010203@Gmail.com> Message-ID: On 20/07/2013 10:25 PM, Devyn Collier Johnson wrote: > I have not heard of Pyaudio; I will look into that. As > for Pygame, I have not been able to find any good documentation for > playing audio files. Plus, I recently learned that Pygame is not Python3 > compatible. Another option would be Pyglet, which uses the cross-platform binary AVBin to provide sound support. It may not provide as much control as PyAudio, but given your example usage it might be a bit more straightforward: pyglet.media.load('boot.ogg', streaming=False).play() http://www.pyglet.org/doc/programming_guide/simple_audio_playback.html The latest development release provides support for Python 3: https://code.google.com/p/pyglet/downloads/list?q=1.2alpha1 From dwightdhutto at gmail.com Tue Jul 23 01:19:12 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Tue, 23 Jul 2013 01:19:12 -0400 Subject: Play Ogg Files In-Reply-To: References: <51E9E29E.7010203@Gmail.com> Message-ID: Devyn, are you just trying to use this in an application? Would a browser based web app work. I ask because there will still be some sort of DB interaction, so could it be an option to go with a browser command? On Mon, Jul 22, 2013 at 8:37 PM, alex23 wrote: > On 20/07/2013 10:25 PM, Devyn Collier Johnson wrote: > >> I have not heard of Pyaudio; I will look into that. As >> for Pygame, I have not been able to find any good documentation for >> playing audio files. Plus, I recently learned that Pygame is not Python3 >> compatible. >> > > Another option would be Pyglet, which uses the cross-platform binary AVBin > to provide sound support. It may not provide as much control as PyAudio, > but given your example usage it might be a bit more straightforward: > > pyglet.media.load('boot.ogg', streaming=False).play() > > http://www.pyglet.org/doc/**programming_guide/simple_**audio_playback.html > > The latest development release provides support for Python 3: > > https://code.google.com/p/**pyglet/downloads/list?q=1.**2alpha1 > -- > http://mail.python.org/**mailman/listinfo/python-list > -- Best Regards, David Hutto *CEO:* *http://www.hitwebdevelopment.com* -------------- next part -------------- An HTML attachment was scrubbed... URL: From devyncjohnson at gmail.com Tue Jul 23 06:40:10 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Tue, 23 Jul 2013 06:40:10 -0400 Subject: Play Ogg Files In-Reply-To: References: <51E9E29E.7010203@Gmail.com> Message-ID: <51EE5D8A.7000704@Gmail.com> On 07/23/2013 01:19 AM, David Hutto wrote: > Devyn, are you just trying to use this in an application? Would a > browser based web app work. I ask because there will still be some > sort of DB interaction, so could it be an option to go with a browser > command? > > > On Mon, Jul 22, 2013 at 8:37 PM, alex23 > wrote: > > On 20/07/2013 10:25 PM, Devyn Collier Johnson wrote: > > I have not heard of Pyaudio; I will look into that. As > for Pygame, I have not been able to find any good > documentation for > playing audio files. Plus, I recently learned that Pygame is > not Python3 > compatible. > > > Another option would be Pyglet, which uses the cross-platform > binary AVBin to provide sound support. It may not provide as much > control as PyAudio, but given your example usage it might be a bit > more straightforward: > > pyglet.media.load('boot.ogg', streaming=False).play() > > http://www.pyglet.org/doc/programming_guide/simple_audio_playback.html > > The latest development release provides support for Python 3: > > https://code.google.com/p/pyglet/downloads/list?q=1.2alpha1 > -- > http://mail.python.org/mailman/listinfo/python-list > > > > > -- > Best Regards, > David Hutto > /*CEO:*/ _http://www.hitwebdevelopment.com_ > > I will be playing an ogg file as a bootup sound for a chatbot that runs in a terminal. There are no web-applications. I will be looking into the different suggestions that I was offered. So far, Pyglet seems to be the best. Once I have officially decided and implemented an idea, I will share my choice with everyone. Mahalo, DCJ -------------- next part -------------- An HTML attachment was scrubbed... URL: From devyncjohnson at gmail.com Sat Jul 20 08:48:10 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Sat, 20 Jul 2013 08:48:10 -0400 Subject: List as Contributor Message-ID: <51EA870A.70308@Gmail.com> Many users on here have answered my questions and given me ideas and suggestions for code that I am using in my open-source GPLv3 chatbot. When I release the next update (that will be in a month or two), does anyone that has contributed helpful ideas want to be listed as a contributor under the heading "Gave Suggestions/Ideas"? The chatbot, named Neobot, is hosted on Launchpad (Neobot is buggy right now). https://launchpad.net/neobot For those of you that want to be listed, email me with your name as you want it displayed and your email or other contact information if you want. Once I release version 0.8, I will inform this mailing list. Moderators: Would you like me to list the email address of this mailing list as a contributor? Mahalo, Devyn Collier Johnson DevynCJohnson at Gmail.com From rosuav at gmail.com Sat Jul 20 09:41:22 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 20 Jul 2013 23:41:22 +1000 Subject: List as Contributor In-Reply-To: <51EA870A.70308@Gmail.com> References: <51EA870A.70308@Gmail.com> Message-ID: On Sat, Jul 20, 2013 at 10:48 PM, Devyn Collier Johnson wrote: > Many users on here have answered my questions and given me ideas and > suggestions for code that I am using in my open-source GPLv3 chatbot. When I > release the next update (that will be in a month or two), does anyone that > has contributed helpful ideas want to be listed as a contributor under the > heading "Gave Suggestions/Ideas"? The chatbot, named Neobot, is hosted on > Launchpad (Neobot is buggy right now). https://launchpad.net/neobot A simple enumeration of names would, I think, be appropriate and simple ("With thanks to the following: Fred Foobar, Joe Citizen, John Smith."). And thanks for the consideration! :) ChrisA From pablobarhamalzas at gmail.com Sat Jul 20 16:22:03 2013 From: pablobarhamalzas at gmail.com (pablobarhamalzas at gmail.com) Date: Sat, 20 Jul 2013 13:22:03 -0700 (PDT) Subject: How can I make this piece of code even faster? Message-ID: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> Ok, I'm working on a predator/prey simulation, which evolve using genetic algorithms. At the moment, they use a quite simple feed-forward neural network, which can change size over time. Each brain "tick" is performed by the following function (inside the Brain class): def tick(self): input_num = self.input_num hidden_num = self.hidden_num output_num = self.output_num hidden = [0]*hidden_num output = [0]*output_num inputs = self.input h_weight = self.h_weight o_weight = self.o_weight e = math.e count = -1 for x in range(hidden_num): temp = 0 for y in range(input_num): count += 1 temp += inputs[y] * h_weight[count] hidden[x] = 1/(1+e**(-temp)) count = -1 for x in range(output_num): temp = 0 for y in range(hidden_num): count += 1 temp += hidden[y] * o_weight[count] output[x] = 1/(1+e**(-temp)) self.output = output The function is actually quite fast (~0.040 seconds per 200 calls, using 10 input, 20 hidden and 3 output neurons), and used to be much slower untill I fiddled about with it a bit to make it faster. However, it is still somewhat slow for what I need it. My question to you is if you an see any obvious (or not so obvious) way of making this faster. I've heard about numpy and have been reading about it, but I really can't see how it could be implemented here. Cheers! From fabiofz at gmail.com Sat Jul 20 17:05:43 2013 From: fabiofz at gmail.com (Fabio Zadrozny) Date: Sat, 20 Jul 2013 18:05:43 -0300 Subject: How can I make this piece of code even faster? In-Reply-To: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> References: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> Message-ID: On Sat, Jul 20, 2013 at 5:22 PM, wrote: > Ok, I'm working on a predator/prey simulation, which evolve using genetic > algorithms. At the moment, they use a quite simple feed-forward neural > network, which can change size over time. Each brain "tick" is performed by > the following function (inside the Brain class): > > def tick(self): > input_num = self.input_num > hidden_num = self.hidden_num > output_num = self.output_num > > hidden = [0]*hidden_num > output = [0]*output_num > > inputs = self.input > h_weight = self.h_weight > o_weight = self.o_weight > > e = math.e > > count = -1 > for x in range(hidden_num): > temp = 0 > for y in range(input_num): > count += 1 > temp += inputs[y] * h_weight[count] > hidden[x] = 1/(1+e**(-temp)) > > count = -1 > for x in range(output_num): > temp = 0 > for y in range(hidden_num): > count += 1 > temp += hidden[y] * o_weight[count] > output[x] = 1/(1+e**(-temp)) > > self.output = output > > The function is actually quite fast (~0.040 seconds per 200 calls, using > 10 input, 20 hidden and 3 output neurons), and used to be much slower > untill I fiddled about with it a bit to make it faster. However, it is > still somewhat slow for what I need it. > > My question to you is if you an see any obvious (or not so obvious) way of > making this faster. I've heard about numpy and have been reading about it, > but I really can't see how it could be implemented here. > > Cheers! > -- > http://mail.python.org/mailman/listinfo/python-list > Low level optimizations: If you're in Python 2.x (and not 3), you should use xrange() instead of range(), or maybe even create a local variable and increment it and check its value within a while (that way you can save a few instructions on method invocations from xrange/range). Anyways, if that's not fast enough, just port it to c/c++ (or one of the alternatives to speed it up while still in python: numba, cython, shedskin). Or (if you can), try to use PyPy and see if you get more speed without doing anything. Cheers, Fabio -------------- next part -------------- An HTML attachment was scrubbed... URL: From roy at panix.com Sat Jul 20 17:25:52 2013 From: roy at panix.com (Roy Smith) Date: Sat, 20 Jul 2013 17:25:52 -0400 Subject: How can I make this piece of code even faster? References: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> Message-ID: In article <6bf4d298-b425-4357-9c1a-192e6e6cd9f0 at googlegroups.com>, pablobarhamalzas at gmail.com wrote: > Ok, I'm working on a predator/prey simulation, which evolve using genetic > algorithms. At the moment, they use a quite simple feed-forward neural > network, which can change size over time. Each brain "tick" is performed by > the following function (inside the Brain class): > > def tick(self): > input_num = self.input_num > hidden_num = self.hidden_num > output_num = self.output_num > > hidden = [0]*hidden_num > output = [0]*output_num > > inputs = self.input > h_weight = self.h_weight > o_weight = self.o_weight > > e = math.e > > count = -1 > for x in range(hidden_num): > temp = 0 > for y in range(input_num): > count += 1 > temp += inputs[y] * h_weight[count] > hidden[x] = 1/(1+e**(-temp)) > > count = -1 > for x in range(output_num): > temp = 0 > for y in range(hidden_num): > count += 1 > temp += hidden[y] * o_weight[count] > output[x] = 1/(1+e**(-temp)) > > self.output = output > > The function is actually quite fast (~0.040 seconds per 200 calls, using 10 > input, 20 hidden and 3 output neurons), and used to be much slower untill I > fiddled about with it a bit to make it faster. However, it is still somewhat > slow for what I need it. > > My question to you is if you an see any obvious (or not so obvious) way of > making this faster. I've heard about numpy and have been reading about it, > but I really can't see how it could be implemented here. First thing, I would add some instrumentation to see where the most time is being spent. My guess is in the first set of nested loops, where the inner loop gets executed hidden_num * input_num (i.e. 10 * 20 = 200) times. But timing data is better than my guess. Assuming I'm right, though, you do compute range(input_num) 20 times. You don't need to do that. You might try xrange(), or you might just factor out creating the list outside the outer loop. But, none of that seems like it should make much difference. What possible values can temp take? If it can only take certain discrete values and you can enumerate them beforehand, you might want to build a dict mapping temp -> 1/(1+e**(-temp)) and then all that math becomes just a table lookup. From pablobarhamalzas at gmail.com Sat Jul 20 18:45:45 2013 From: pablobarhamalzas at gmail.com (pablobarhamalzas at gmail.com) Date: Sat, 20 Jul 2013 15:45:45 -0700 (PDT) Subject: How can I make this piece of code even faster? In-Reply-To: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> References: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> Message-ID: <49c89529-2e93-474a-9d1f-1ee2288085c1@googlegroups.com> Hi there. I'm using python 3, where xrange doesn't exist any more (range is now equivalent). And "temp" doesn't have any fixed discrete values it always takes. I have tried cython but it doesn't seem to work well (maybe using it wrong?). Any other ideas? From rosuav at gmail.com Sat Jul 20 18:55:07 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 21 Jul 2013 08:55:07 +1000 Subject: How can I make this piece of code even faster? In-Reply-To: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> References: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> Message-ID: On Sun, Jul 21, 2013 at 6:22 AM, wrote: > temp = 0 > for y in range(input_num): > count += 1 > temp += inputs[y] * h_weight[count] > hidden[x] = 1/(1+e**(-temp)) It's a micro-optimization that'll probably have negligible effect, but it can't hurt: Instead of adding to temp and raising e to -temp, carry the value of temp as a negative number: temp -= inputs[y] * h_weight[count] hidden[x] = 1/(1+e**temp) Ditto in the second loop. Not sure which way performance would go, but would it be more readable to take an iterator for h_weight and o_weight? Something like this: # Slot this into your existing structure inputs = self.input h_weight = iter(self.h_weight) o_weight = iter(self.o_weight) e = math.e for x in range(hidden_num): temp = 0 for y in inputs: temp += y * next(h_weight) hidden[x] = 1/(1+e**(-temp)) for x in range(output_num): temp = 0 for y in hidden: temp += y * next(o_weight) output[x] = 1/(1+e**(-temp)) # End. If that looks better, the next change I'd look to make is replacing the 'for y' loops with sum() calls on generators: temp = sum(y * next(o_weight) for y in hidden) And finally replace the entire 'for x' loops with list comps... which makes for two sizeable one-liners, which I like and many people detest: def tick(self): inputs = self.inputs h_weight = iter(self.h_weight) o_weight = iter(self.o_weight) e = math.e hidden = [1/(1+e**sum(-y * next(h_weight) for y in inputs)) for _ in range(hidden_num)] self.output = [1/(1+e**sum(-y * next(o_weight) for y in hidden)) for _ in range(output_num)] Up to you to decide whether you find that version more readable, or at least sufficiently readable, and then to test performance :) But it's shorter by quite a margin, which I personally like. Oh, and I'm relying on you to make sure I've made the translation correctly, which I can't confirm without a pile of input data to test it on. All I can say is that it's syntactically correct. ChrisA From pablobarhamalzas at gmail.com Sat Jul 20 19:24:24 2013 From: pablobarhamalzas at gmail.com (pablobarhamalzas at gmail.com) Date: Sat, 20 Jul 2013 16:24:24 -0700 (PDT) Subject: How can I make this piece of code even faster? In-Reply-To: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> References: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> Message-ID: Hi there Chris. Unfortunately, using iterations was about twice as slow as the original implementation, so that's not the solution. Thank's anyway. From rosuav at gmail.com Sat Jul 20 19:29:42 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 21 Jul 2013 09:29:42 +1000 Subject: How can I make this piece of code even faster? In-Reply-To: References: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> Message-ID: On Sun, Jul 21, 2013 at 9:24 AM, wrote: > Hi there Chris. > Unfortunately, using iterations was about twice as slow as the original implementation, so that's not the solution. > Thank's anyway. Fascinating! Well, was worth a try anyhow. But that's a very surprising result. ChrisA From steve+comp.lang.python at pearwood.info Sun Jul 21 01:11:51 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 21 Jul 2013 05:11:51 GMT Subject: How can I make this piece of code even faster? References: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> Message-ID: <51eb6d96$0$29971$c3e8da3$5496439d@news.astraweb.com> On Sat, 20 Jul 2013 13:22:03 -0700, pablobarhamalzas asked: "How can I make this piece of code even faster?" - Use a faster computer. - Put in more memory. - If using Unix or Linux, decrease the "nice" priority of the process. I mention these because sometimes people forget that if you have a choice between "spend 10 hours at $70 per hour to optimize code", and "spend $200 to put more memory in", putting more memory in may be more cost effective. Other than that, what you describe sounds like it could be a good candidate for PyPy to speed the code up, although PyPy is still (mostly) Python 2. You could take this question to the pypy mailing list and ask there. http://mail.python.org/mailman/listinfo/pypy-dev You also might like to try Cython or Numba. As far as pure-Python optimizations, once you have a decent algorithm, there's probably not a lot of room for major speed ups. But a couple of thoughts and a possible optimized version come to mind... 1) In general, it is better/faster to iterate over lists directly, than indirectly by index number: for item in sequence: process(item) rather than: for i in range(len(sequence)): item = sequence[i] process(item) If you need both the index and the value: for i, item in enumerate(sequence): print(i, process(item)) In your specific case, if I have understood your code's logic, you can just iterate directly over the appropriate lists, once each. 2) You perform an exponentiation using math.e**(-temp). You will probably find that math.exp(-temp) is both faster and more accurate. 3) If you need to add numbers, it is better to call sum() or math.fsum() than add them by hand. sum() may be a tiny bit faster, or maybe not, but fsum() is more accurate for floats. See below for my suggestion on an optimized version. > Ok, I'm working on a predator/prey simulation, which evolve using > genetic algorithms. At the moment, they use a quite simple feed-forward > neural network, which can change size over time. Each brain "tick" is > performed by the following function (inside the Brain class): > > def tick(self): > input_num = self.input_num > hidden_num = self.hidden_num > output_num = self.output_num > > hidden = [0]*hidden_num > output = [0]*output_num > > inputs = self.input > h_weight = self.h_weight > o_weight = self.o_weight > > e = math.e > > count = -1 > for x in range(hidden_num): > temp = 0 > for y in range(input_num): > count += 1 > temp += inputs[y] * h_weight[count] > hidden[x] = 1/(1+e**(-temp)) > > count = -1 > for x in range(output_num): > temp = 0 > for y in range(hidden_num): > count += 1 > temp += hidden[y] * o_weight[count] > output[x] = 1/(1+e**(-temp)) > > self.output = output > > The function is actually quite fast (~0.040 seconds per 200 calls, using > 10 input, 20 hidden and 3 output neurons), and used to be much slower > untill I fiddled about with it a bit to make it faster. However, it is > still somewhat slow for what I need it. > > My question to you is if you an see any obvious (or not so obvious) way > of making this faster. I've heard about numpy and have been reading > about it, but I really can't see how it could be implemented here. Here's my suggestion: def tick(self): exp = math.exp sum = math.fsum # more accurate than builtin sum # This assumes that both inputs and h_weight have exactly # self.input_num values. temp = fsum(i*w for (i, w) in zip(self.inputs, self.h_weight)) hidden = [1/(1+exp(-temp))]*self.hidden_num # This assumes that both outputs and o_weight have exactly # self.output_num values. temp = fsum(o*w for (o, w) in zip(self.outputs, self.o_weight)) self.output = [1/(1+exp(-temp))]*self.output_num I have neither tested that this works the same as your code (or even works at all!) nor that it is faster, but I would expect that it will be faster. Good luck! -- Steven From paul.nospam at rudin.co.uk Sun Jul 21 03:11:59 2013 From: paul.nospam at rudin.co.uk (Paul Rudin) Date: Sun, 21 Jul 2013 08:11:59 +0100 Subject: How can I make this piece of code even faster? References: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> <51eb6d96$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <87fvv8v134.fsf@no-fixed-abode.cable.virginmedia.net> Steven D'Aprano writes: > On Sat, 20 Jul 2013 13:22:03 -0700, pablobarhamalzas asked: > > "How can I make this piece of code even faster?" > > - Use a faster computer. > - Put in more memory. > - If using Unix or Linux, decrease the "nice" priority of the process. > > I mention these because sometimes people forget that if you have a choice > between "spend 10 hours at $70 per hour to optimize code", and "spend > $200 to put more memory in", putting more memory in may be more cost > effective. > Sure - but it's helpful if programmers understand a little bit about the computational complexity of algorithms. If it's just a question of making each basic step of your algorithm a bit faster, then it may well be better to spend money on better hardware than on squeezing more out of your code. OTOH if you've got an n^2 implementation and there's actually an n.log n solution available then you should probably re-code. Of course if what you've got is actually adequate for your use-case then it maybe that you don't actually need to do anything at all... From rosuav at gmail.com Sun Jul 21 05:21:52 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 21 Jul 2013 19:21:52 +1000 Subject: How can I make this piece of code even faster? In-Reply-To: <87fvv8v134.fsf@no-fixed-abode.cable.virginmedia.net> References: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> <51eb6d96$0$29971$c3e8da3$5496439d@news.astraweb.com> <87fvv8v134.fsf@no-fixed-abode.cable.virginmedia.net> Message-ID: On Sun, Jul 21, 2013 at 5:11 PM, Paul Rudin wrote: > Steven D'Aprano writes: > >> On Sat, 20 Jul 2013 13:22:03 -0700, pablobarhamalzas asked: >> >> "How can I make this piece of code even faster?" >> >> - Use a faster computer. >> - Put in more memory. >> - If using Unix or Linux, decrease the "nice" priority of the process. >> >> I mention these because sometimes people forget that if you have a choice >> between "spend 10 hours at $70 per hour to optimize code", and "spend >> $200 to put more memory in", putting more memory in may be more cost >> effective. >> > > Sure - but it's helpful if programmers understand a little bit about the > computational complexity of algorithms. If it's just a question of > making each basic step of your algorithm a bit faster, then it may well > be better to spend money on better hardware than on squeezing more out > of your code. OTOH if you've got an n^2 implementation and there's > actually an n.log n solution available then you should probably re-code. I haven't analyzed every suggestion in this thread in detail, but I don't think any of them affects the algorithmic complexity of the code. They're all incremental changes. ChrisA From __peter__ at web.de Sun Jul 21 03:10:10 2013 From: __peter__ at web.de (Peter Otten) Date: Sun, 21 Jul 2013 09:10:10 +0200 Subject: How can I make this piece of code even faster? References: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> Message-ID: pablobarhamalzas at gmail.com wrote: > Ok, I'm working on a predator/prey simulation, which evolve using genetic > algorithms. At the moment, they use a quite simple feed-forward neural > network, which can change size over time. Each brain "tick" is performed > by the following function (inside the Brain class): > > def tick(self): > input_num = self.input_num > hidden_num = self.hidden_num > output_num = self.output_num > > hidden = [0]*hidden_num > output = [0]*output_num > > inputs = self.input > h_weight = self.h_weight > o_weight = self.o_weight > > e = math.e > > count = -1 > for x in range(hidden_num): > temp = 0 > for y in range(input_num): > count += 1 > temp += inputs[y] * h_weight[count] > hidden[x] = 1/(1+e**(-temp)) > > count = -1 > for x in range(output_num): > temp = 0 > for y in range(hidden_num): > count += 1 > temp += hidden[y] * o_weight[count] > output[x] = 1/(1+e**(-temp)) > > self.output = output > > The function is actually quite fast (~0.040 seconds per 200 calls, using > 10 input, 20 hidden and 3 output neurons), and used to be much slower > untill I fiddled about with it a bit to make it faster. However, it is > still somewhat slow for what I need it. > > My question to you is if you an see any obvious (or not so obvious) way of > making this faster. I've heard about numpy and have been reading about it, > but I really can't see how it could be implemented here. > > Cheers! Assuming every list is replaced with a numpy.array, h_weight.shape == (hidden_num, input_num) o_weight.shape == (output_num, hidden_num) and as untested as it gets: def tick(self): temp = numpy.dot(self.inputs, self.h_weight) hidden = 1/(1+numpy.exp(-temp)) temp = numpy.dot(hidden, self.o_weight) self.output = 1/(1+numpy.exp(-temp)) My prediction: this is probably wrong, but if you can fix the code it will be stinkin' fast ;) From storchaka at gmail.com Sun Jul 21 03:11:27 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 21 Jul 2013 10:11:27 +0300 Subject: How can I make this piece of code even faster? In-Reply-To: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> References: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> Message-ID: 20.07.13 23:22, pablobarhamalzas at gmail.com ???????(??): > e = math.e > > count = -1 > for x in range(hidden_num): > temp = 0 > for y in range(input_num): > count += 1 > temp += inputs[y] * h_weight[count] > hidden[x] = 1/(1+e**(-temp)) [...] > My question to you is if you an see any obvious (or not so obvious) way of making this faster. 1. Use math.exp() instead of math.e**. 2. I'm not sure that it will be faster, but try to use sum(). temp = sum(inputs[y] * h_weight[count + y] for y in range(input_num)) count += input_num or temp = sum(map(operator.mul, inputs, h_weight[count:count+input_num])) count += input_num From auriocus at gmx.de Sun Jul 21 03:24:25 2013 From: auriocus at gmx.de (Christian Gollwitzer) Date: Sun, 21 Jul 2013 09:24:25 +0200 Subject: How can I make this piece of code even faster? In-Reply-To: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> References: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> Message-ID: How about using numpy? Am 20.07.13 22:22, schrieb pablobarhamalzas at gmail.com: > Ok, I'm working on a predator/prey simulation, which evolve using genetic algorithms. At the moment, they use a quite simple feed-forward neural network, which can change size over time. Each brain "tick" is performed by the following function (inside the Brain class): > > count = -1 > for x in range(hidden_num): > temp = 0 > for y in range(input_num): > count += 1 > temp += inputs[y] * h_weight[count] > hidden[x] = 1/(1+e**(-temp)) I don't really understand this loop, but it looks to me like a matrix-vector multiplication of the matrix of weights (indexed with a single continous index) with the inputs. Given that you reshape the weights() array correctly into a hidden_num-by-input_num array, this would result in import numpy as np hidden = 1.0/(1.0 + np.exp(-np.dot(h_weights, inputs))) The matrix-vector product is then executed by a compiled loop. You can reshape your existing lists into that form by inputs = np.array(inputs) h_weight = np.reshape(h_weight, (hidden_num, input_num)) ... but of course you should use this only to check whether you get the correct result. You don't want to do that in the loop, instead, store the weights always in matrix form. Christian From pablobarhamalzas at gmail.com Sun Jul 21 06:19:24 2013 From: pablobarhamalzas at gmail.com (pablobarhamalzas at gmail.com) Date: Sun, 21 Jul 2013 03:19:24 -0700 (PDT) Subject: How can I make this piece of code even faster? In-Reply-To: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> References: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> Message-ID: <9a207133-1f52-414a-bbbd-581bcee8dc93@googlegroups.com> Thank's for all the replies! I've tried some of the imporovements you suggested (using math.exp() and sum() or math.fsum()). None of that made the code faster, because they are functions you are calling lots of times, and function calling is quite time expensive (same as x**(1/2) is faster than math.sqrt(x)). I'm going to try to convert tu numpy now (I have no idea how to do it at the moment), thank's again to everyone. From steve+comp.lang.python at pearwood.info Sun Jul 21 06:31:42 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 21 Jul 2013 10:31:42 GMT Subject: How can I make this piece of code even faster? References: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> <9a207133-1f52-414a-bbbd-581bcee8dc93@googlegroups.com> Message-ID: <51ebb88e$0$29971$c3e8da3$5496439d@news.astraweb.com> On Sun, 21 Jul 2013 03:19:24 -0700, pablobarhamalzas wrote: > Thank's for all the replies! I've tried some of the imporovements you > suggested (using math.exp() and sum() or math.fsum()). None of that made > the code faster, because they are functions you are calling lots of > times, and function calling is quite time expensive (same as x**(1/2) is > faster than math.sqrt(x)). You are *badly* mistaken. Not only is sqrt more accurate, but it is also much faster. [steve at ando ~]$ python3.3 -m timeit -s "x = 2.357e7" "x**0.5" 1000000 loops, best of 3: 0.319 usec per loop [steve at ando ~]$ python3.3 -m timeit -s "x = 2.357e7" -s "from math import sqrt" "sqrt(x)" 10000000 loops, best of 3: 0.172 usec per loop How exactly are you timing the code? -- Steven From rosuav at gmail.com Sun Jul 21 06:48:46 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 21 Jul 2013 20:48:46 +1000 Subject: How can I make this piece of code even faster? In-Reply-To: <51ebb88e$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> <9a207133-1f52-414a-bbbd-581bcee8dc93@googlegroups.com> <51ebb88e$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sun, Jul 21, 2013 at 8:31 PM, Steven D'Aprano wrote: > On Sun, 21 Jul 2013 03:19:24 -0700, pablobarhamalzas wrote: > >> Thank's for all the replies! I've tried some of the imporovements you >> suggested (using math.exp() and sum() or math.fsum()). None of that made >> the code faster, because they are functions you are calling lots of >> times, and function calling is quite time expensive (same as x**(1/2) is >> faster than math.sqrt(x)). > > You are *badly* mistaken. Not only is sqrt more accurate, but it is also > much faster. > > > [steve at ando ~]$ python3.3 -m timeit -s "x = 2.357e7" "x**0.5" > 1000000 loops, best of 3: 0.319 usec per loop > [steve at ando ~]$ python3.3 -m timeit -s "x = 2.357e7" -s "from math import > sqrt" "sqrt(x)" > 10000000 loops, best of 3: 0.172 usec per loop Don't forget the cost of attribute lookup, which adds 50% to the sqrt() figure. Still faster than exponentiation. (Figures from Python 3.4 alpha, but unlikely to be materially different.) rosuav at sikorsky:~$ python3 -m timeit -s "x = 2.357e7" "x**0.5" 1000000 loops, best of 3: 0.239 usec per loop rosuav at sikorsky:~$ python3 -m timeit -s "x = 2.357e7" -s "from math import sqrt" "sqrt(x)" 10000000 loops, best of 3: 0.102 usec per loop rosuav at sikorsky:~$ python3 -m timeit -s "x = 2.357e7" -s "import math" "math.sqrt(x)" 10000000 loops, best of 3: 0.155 usec per loop ChrisA From pablobarhamalzas at gmail.com Sun Jul 21 06:48:36 2013 From: pablobarhamalzas at gmail.com (pablobarhamalzas at gmail.com) Date: Sun, 21 Jul 2013 03:48:36 -0700 (PDT) Subject: How can I make this piece of code even faster? In-Reply-To: <51ebb88e$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> <9a207133-1f52-414a-bbbd-581bcee8dc93@googlegroups.com> <51ebb88e$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <2ff99afd-b734-417f-bbc2-65cf8cc0858e@googlegroups.com> El domingo, 21 de julio de 2013 12:31:42 UTC+2, Steven D'Aprano escribi?: > On Sun, 21 Jul 2013 03:19:24 -0700, pablobarhamalzas wrote: > > > > > Thank's for all the replies! I've tried some of the imporovements you > > > suggested (using math.exp() and sum() or math.fsum()). None of that made > > > the code faster, because they are functions you are calling lots of > > > times, and function calling is quite time expensive (same as x**(1/2) is > > > faster than math.sqrt(x)). > > > > You are *badly* mistaken. Not only is sqrt more accurate, but it is also > > much faster. > > > > > > [steve at ando ~]$ python3.3 -m timeit -s "x = 2.357e7" "x**0.5" > > 1000000 loops, best of 3: 0.319 usec per loop > > [steve at ando ~]$ python3.3 -m timeit -s "x = 2.357e7" -s "from math import > > sqrt" "sqrt(x)" > > 10000000 loops, best of 3: 0.172 usec per loop > > > > > > How exactly are you timing the code? I'm timing the whole program with cProfile. Removing math.sqrt() from a function and using **(1/2) instead cut the execution time for a significant amount (~0.035 to ~0.020). I can't see another explanation for the speed increase... From stefan_ml at behnel.de Sun Jul 21 08:49:44 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 21 Jul 2013 14:49:44 +0200 Subject: How can I make this piece of code even faster? In-Reply-To: <2ff99afd-b734-417f-bbc2-65cf8cc0858e@googlegroups.com> References: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> <9a207133-1f52-414a-bbbd-581bcee8dc93@googlegroups.com> <51ebb88e$0$29971$c3e8da3$5496439d@news.astraweb.com> <2ff99afd-b734-417f-bbc2-65cf8cc0858e@googlegroups.com> Message-ID: pablobarhamalzas at gmail.com, 21.07.2013 12:48: > El domingo, 21 de julio de 2013 12:31:42 UTC+2, Steven D'Aprano escribi?: >> [steve at ando ~]$ python3.3 -m timeit -s "x = 2.357e7" "x**0.5" >> 1000000 loops, best of 3: 0.319 usec per loop >> [steve at ando ~]$ python3.3 -m timeit -s "x = 2.357e7" -s "from math import >> sqrt" "sqrt(x)" >> 10000000 loops, best of 3: 0.172 usec per loop >> >> How exactly are you timing the code? > > I'm timing the whole program with cProfile. Removing math.sqrt() from a function and using **(1/2) instead cut the execution time for a significant amount (~0.035 to ~0.020). With or without the profiler running? Note that profiling will slow down your code (especially function calls), often significantly and sometimes even in such an unbalanced way that it visibly changes its execution profile. Always make sure you validate your code changes with benchmarks, outside of the profiler. Stefan From torriem at gmail.com Sun Jul 21 11:27:36 2013 From: torriem at gmail.com (Michael Torrie) Date: Sun, 21 Jul 2013 09:27:36 -0600 Subject: How can I make this piece of code even faster? In-Reply-To: <9a207133-1f52-414a-bbbd-581bcee8dc93@googlegroups.com> References: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> <9a207133-1f52-414a-bbbd-581bcee8dc93@googlegroups.com> Message-ID: <51EBFDE8.4030705@gmail.com> On 07/21/2013 04:19 AM, pablobarhamalzas at gmail.com wrote: > Thank's for all the replies! I've tried some of the imporovements you suggested (using math.exp() and sum() or math.fsum()). > None of that made the code faster, because they are functions you are calling lots of times, and function calling is quite time expensive (same as x**(1/2) is faster than math.sqrt(x)). > > I'm going to try to convert tu numpy now (I have no idea how to do it at the moment), thank's again to everyone. Perhaps you'd have better results if you'd post a runnable piece of code. Otherwise we're just guessing since no one has the ability to actually run your code. From joshua at landau.ws Sun Jul 21 07:39:46 2013 From: joshua at landau.ws (Joshua Landau) Date: Sun, 21 Jul 2013 12:39:46 +0100 Subject: How can I make this piece of code even faster? In-Reply-To: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> References: <6bf4d298-b425-4357-9c1a-192e6e6cd9f0@googlegroups.com> Message-ID: On 20 July 2013 21:22, wrote: > Ok, I'm working on a predator/prey simulation, which evolve using genetic algorithms. At the moment, they use a quite simple feed-forward neural network, which can change size over time. Each brain "tick" is performed by the following function (inside the Brain class): > > > The function is actually quite fast (~0.040 seconds per 200 calls, using 10 input, 20 hidden and 3 output neurons), and used to be much slower untill I fiddled about with it a bit to make it faster. However, it is still somewhat slow for what I need it. > > My question to you is if you an see any obvious (or not so obvious) way of making this faster. I've heard about numpy and have been reading about it, but I really can't see how it could be implemented here. Currently we're just guessing; if you gave us an appropriate stand-in for "self" (so that we can call the function) we could be helpful much more easily. From ptmcg at austin.rr.com Sat Jul 20 17:30:14 2013 From: ptmcg at austin.rr.com (Paul McGuire) Date: Sat, 20 Jul 2013 14:30:14 -0700 (PDT) Subject: [ANN] pyparsing 2.0.1 released - compatible with Python 2.6 and later Message-ID: In my releasing of Pyparsing 1.5.7/2.0.0 last November, I started to split supported Python versions: 2.x to the Pyparsing 1.5.x track, and 3.x to the Pyparsing 2.x track. Unfortunately, this caused a fair bit of pain for many current users of Python 2.6 and 2.7 (especially those using libs dependent on pyparsing), as the default installed pyparsing version using easy_install or pip would be the incompatible-to-them pyparsing 2.0.0. I hope I have rectified (or at least improved) this situation with the latest release of pyparsing 2.0.1. Version 2.0.1 takes advantage of the cross-major-version compatibility that was planned into Python, wherein many of the new features of Python 3.x were made available in Python 2.6 and 2.7. By avoiding the one usage of ?nonlocal? (a Python 3.x feature not available in any Python 2.x release), I?ve been able to release pyparsing 2.0.1 in a form that will work for all those using Python 2.6 and later. (If you are stuck on version 2.5 or earlier of Python, then you still have to explicitly download the 1.5.7 version of pyparsing.) This release also includes a bugfix to the new ?<<=? operator, so that ?< Message-ID: <51eb63b6$0$29971$c3e8da3$5496439d@news.astraweb.com> On Sat, 20 Jul 2013 14:30:14 -0700, Paul McGuire wrote: > Thanks for your continued support and interest in pyparsing! And thank you for pyparsing! Paul, I thought I would mention that over the last week or so on the Python-Dev mailing list, there has been some discussion about adding a parser generator to the standard library for Python 3.4. If this is of interest to you, have a look for the thread PLY in stdlib (was cffi in stdlib) at http://mail.python.org/mailman/listinfo/python-dev Regards, -- Steve From vasudevram at gmail.com Sat Jul 20 18:20:31 2013 From: vasudevram at gmail.com (vasudevram) Date: Sat, 20 Jul 2013 15:20:31 -0700 (PDT) Subject: PDFBuilder can now handle unlimited input files Message-ID: (Cross-posted to comp.lang.python.announce) PDFBuilder, which is a part of my xtopdf toolkit, can now handle an unlimited (*1) number of input files. PDFBuilder enables the creation of composite PDFs consisting of the content of multiple CSV and TDV/TSV files (*2). Support for other formats can be added easily. (*2) TDV = Tab Delimited Values, TSV = Tab Separated Values Blog post about the latest version of PDFBuilder: PDFBuilder can now handle unlimited input files: http://jugad2.blogspot.in/2013/07/pdfbuilder-can-now-handle-unlimited.html (*1) See the above post for the meaning of "unlimited". PDFBuilder is available here as part of the xtopdf toolkit: https://bitbucket.org/vasudevram/xtopdf xtopdf (and hence PDFBuilder) is released as open source software under the BSD License. Enjoy. --- Vasudev Ram Dancing Bison Enterprises Software consulting and training (Python, C, Linux, databases, open source, ...) http://www.dancingbison.com (site currently down for maintenance) From nospam at nospam.com Sun Jul 21 10:42:50 2013 From: nospam at nospam.com (Gilles) Date: Sun, 21 Jul 2013 16:42:50 +0200 Subject: Simple Python script as SMTP server for outgoing e-mails? Message-ID: Hello Every once in a while, my ISP's SMTP server refuses to send perfectly legit e-mails because it considers them as SPAM. So I'd like to install a dead-simple SMTP server on my XP computer just to act as SMTP backup server. All I'd need is to change the SMTP address in my e-mail client, and off they go. No need for anything else like user authentication or SPAM control. Is there a no-brainer, ready-to-use solution in Python that I could use for this? Thank you. From rosuav at gmail.com Sun Jul 21 10:48:29 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 22 Jul 2013 00:48:29 +1000 Subject: Simple Python script as SMTP server for outgoing e-mails? In-Reply-To: References: Message-ID: On Mon, Jul 22, 2013 at 12:42 AM, Gilles wrote: > Hello > > Every once in a while, my ISP's SMTP server refuses to send > perfectly legit e-mails because it considers them as SPAM. > > So I'd like to install a dead-simple SMTP server on my XP computer > just to act as SMTP backup server. > All I'd need is to change the SMTP address in my e-mail client, and > off they go. No need for anything else like user authentication or > SPAM control. > > Is there a no-brainer, ready-to-use solution in Python that I could > use for this? Rather than write something from scratch, I'd look at deploying something out-of-the-box - Postfix, for instance - which you will be able to configure much faster than writing your own. And then you could have it either send via your ISP or send directly to the receiving MTA, without much extra effort. ChrisA From nospam at nospam.com Sun Jul 21 12:19:03 2013 From: nospam at nospam.com (Gilles) Date: Sun, 21 Jul 2013 18:19:03 +0200 Subject: Simple Python script as SMTP server for outgoing e-mails? References: Message-ID: On Mon, 22 Jul 2013 00:48:29 +1000, Chris Angelico wrote: >Rather than write something from scratch, I'd look at deploying >something out-of-the-box - Postfix, for instance - which you will be >able to configure much faster than writing your own. And then you >could have it either send via your ISP or send directly to the >receiving MTA, without much extra effort. Thank you but precisely, I was looking for a "ready-to-use solution in Python" so that I wouldn't have to write it myself. Also, I don't need a full-fledged SMTP server, just a tiny script that will let me send the occasional e-mails from my e-mail client that my ISP wrongly considers as SPAM. So, does someone know of a good, SMTP server just to send e-mails? Thank you. From torriem at gmail.com Sun Jul 21 13:46:52 2013 From: torriem at gmail.com (Michael Torrie) Date: Sun, 21 Jul 2013 11:46:52 -0600 Subject: Simple Python script as SMTP server for outgoing e-mails? In-Reply-To: References: Message-ID: <51EC1E8C.2000705@gmail.com> On 07/21/2013 10:19 AM, Gilles wrote: > So, does someone know of a good, SMTP server just to send e-mails? What you're looking for is not an SMTP server but a Mail Transfer Agent, called an MTA. Pretty much all distros ship with an MTA by default, even if the SMTP server part of it isn't installed or running. And often the MTA is, for compatibility reasons, /usr/sbin/sendmail. http://stackoverflow.com/questions/73781/sending-mail-via-sendmail-from-python I'm sure there are MTA's implemented in python. Now that you know what they are called (not SMTP servers!) you can search for them. Dennis is correct, though, that most ISPs do block outbound port 25 connections for security and spam reasons, and require you to use their SMTP server, which precludes the use of the local MTA. From nospam at nospam.com Sun Jul 21 16:34:46 2013 From: nospam at nospam.com (Gilles) Date: Sun, 21 Jul 2013 22:34:46 +0200 Subject: Simple Python script as SMTP server for outgoing e-mails? References: Message-ID: On Sun, 21 Jul 2013 11:46:52 -0600, Michael Torrie wrote: >What you're looking for is not an SMTP server but a Mail Transfer Agent, >called an MTA. > >Pretty much all distros ship with an MTA by default, even if the SMTP >server part of it isn't installed or running. And often the MTA is, for >compatibility reasons, /usr/sbin/sendmail. > >http://stackoverflow.com/questions/73781/sending-mail-via-sendmail-from-python > >I'm sure there are MTA's implemented in python. Now that you know what >they are called (not SMTP servers!) you can search for them. > >Dennis is correct, though, that most ISPs do block outbound port 25 >connections for security and spam reasons, and require you to use their >SMTP server, which precludes the use of the local MTA. Thanks for the infos. Ideally, I was looking for a simple Windows app as MTA, but a Python script is OK. I'm not sure my ISP blocks outbound port 25 connections. I'll experiment with a small Linux box. I wist they would use a smarter SPAM filter that wouldn't flag perfectly legit-looking outgoing e-mails. From oneingray at gmail.com Sun Jul 21 16:53:07 2013 From: oneingray at gmail.com (Ivan Shmakov) Date: Sun, 21 Jul 2013 20:53:07 +0000 Subject: Simple Python script as SMTP server for outgoing e-mails? References: Message-ID: <87siz7tz2k.fsf@violet.siamics.net> >>>>> Gilles writes: >>>>> On Sun, 21 Jul 2013 11:46:52 -0600, Michael Torrie wrote: [Cross-posting to news:comp.mail.misc.] >> What you're looking for is not an SMTP server but a Mail Transfer >> Agent, called an MTA. [...] >> Dennis is correct, though, that most ISPs do block outbound port 25 >> connections for security and spam reasons, and require you to use >> their SMTP server, which precludes the use of the local MTA. > I'm not sure my ISP blocks outbound port 25 connections. I'll > experiment with a small Linux box. There's yet another issue: certain email "operators" may block /inbound/ port 25 connections from ISP "customer" networks. > I wist they would use a smarter SPAM filter that wouldn't flag > perfectly legit-looking outgoing e-mails. FWIW, it may also be possible to use an email service (such as Google Mail) provided by a third-party. -- FSF associate member #7257 From torriem at gmail.com Sun Jul 21 20:28:27 2013 From: torriem at gmail.com (Michael Torrie) Date: Sun, 21 Jul 2013 18:28:27 -0600 Subject: Simple Python script as SMTP server for outgoing e-mails? In-Reply-To: References: Message-ID: <51EC7CAB.7020201@gmail.com> On 07/21/2013 02:34 PM, Gilles wrote: > Thanks for the infos. Ideally, I was looking for a simple Windows app > as MTA, but a Python script is OK. The Sendmail MTA has been ported to many platforms including windows. But... > I'm not sure my ISP blocks outbound port 25 connections. I'll > experiment with a small Linux box. Having spent a long time managing e-mail servers, everything Ivan said in his reply is true as well. I had forgotten a lot of that since I haven't been running my own mail server (MTA or server part) in a while. I've sold my soul to Google for e-mail now with Google Apps for my domain. > I wist they would use a smarter SPAM filter that wouldn't flag > perfectly legit-looking outgoing e-mails. But then how would it know that legit-looking e-mails aren't in fact SPAM? E-mail is starting to be an almost intractable problem. No wonder the younger generations are just abandoning it entirely in favor of centralized, cathedral-style messaging systems such as facebook. From nospam at nospam.com Mon Jul 22 08:11:25 2013 From: nospam at nospam.com (Gilles) Date: Mon, 22 Jul 2013 14:11:25 +0200 Subject: Simple Python script as SMTP server for outgoing e-mails? References: Message-ID: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> On Sun, 21 Jul 2013 18:28:27 -0600, Michael Torrie wrote: >The Sendmail MTA has been ported to many platforms including windows. >But... Thanks for the tip. Since I couldn't find a good, basic, native Windows app, I was indeed about to look at eg. Exim + Cygwin, and resort to a Linux appliance if none footed the bill. >> I'm not sure my ISP blocks outbound port 25 connections. I'll >> experiment with a small Linux box. > >Having spent a long time managing e-mail servers, everything Ivan said >in his reply is true as well. I had forgotten a lot of that since I >haven't been running my own mail server (MTA or server part) in a while. Indeed, I had forgotten about some MTAs refusing incoming e-mails from other ISP's customer hosts. I'll experiment. >But then how would it know that legit-looking e-mails aren't in fact >SPAM? It generally does a good job, but every once in a while, some perfectly good e-mail I'm sending is flagged as SPAM. To keep all my e-mails in the same client, I'd rather use a local MTA than sending the e-mail from Gmail. Thank you. From rosuav at gmail.com Mon Jul 22 08:29:42 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 22 Jul 2013 22:29:42 +1000 Subject: Simple Python script as SMTP server for outgoing e-mails? In-Reply-To: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> Message-ID: On Mon, Jul 22, 2013 at 10:11 PM, Gilles wrote: > On Sun, 21 Jul 2013 18:28:27 -0600, Michael Torrie > wrote: >>Having spent a long time managing e-mail servers, everything Ivan said >>in his reply is true as well. I had forgotten a lot of that since I >>haven't been running my own mail server (MTA or server part) in a while. > > Indeed, I had forgotten about some MTAs refusing incoming e-mails from > other ISP's customer hosts. I'll experiment. One thing to check when you change how you send mail is your SPF record. I run the mail server for kepl.com.au and have set its SPF to: "v=spf1 ip4:122.107.147.136 ip4:203.214.67.43 ip4:192.168.0.0/16 -all" If your SPF is as strict as mine (and if it's not, please make it so, for the sake of the rest of the world!), you'll want to check it before you start sending mail directly from your own computer. Otherwise your mail _will_ be rejected as spam. ChrisA From nospam at nospam.com Mon Jul 22 08:38:16 2013 From: nospam at nospam.com (Gilles) Date: Mon, 22 Jul 2013 14:38:16 +0200 Subject: Simple Python script as SMTP server for outgoing e-mails? References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> Message-ID: On Mon, 22 Jul 2013 22:29:42 +1000, Chris Angelico wrote: >One thing to check when you change how you send mail is your SPF >record. I run the mail server for kepl.com.au and have set its SPF to: > >"v=spf1 ip4:122.107.147.136 ip4:203.214.67.43 ip4:192.168.0.0/16 -all" > >If your SPF is as strict as mine (and if it's not, please make it so, >for the sake of the rest of the world!), you'll want to check it >before you start sending mail directly from your own computer. >Otherwise your mail _will_ be rejected as spam. Thanks for the tip. I didn't know about SPF http://en.wikipedia.org/wiki/Sender_Policy_Framework From rosuav at gmail.com Mon Jul 22 08:51:12 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 22 Jul 2013 22:51:12 +1000 Subject: Simple Python script as SMTP server for outgoing e-mails? In-Reply-To: References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> Message-ID: On Mon, Jul 22, 2013 at 10:38 PM, Gilles wrote: > On Mon, 22 Jul 2013 22:29:42 +1000, Chris Angelico > wrote: >>One thing to check when you change how you send mail is your SPF >>record. I run the mail server for kepl.com.au and have set its SPF to: >> >>"v=spf1 ip4:122.107.147.136 ip4:203.214.67.43 ip4:192.168.0.0/16 -all" >> >>If your SPF is as strict as mine (and if it's not, please make it so, >>for the sake of the rest of the world!), you'll want to check it >>before you start sending mail directly from your own computer. >>Otherwise your mail _will_ be rejected as spam. > > Thanks for the tip. I didn't know about SPF > http://en.wikipedia.org/wiki/Sender_Policy_Framework It's a great way of detecting legit vs forged mail. If anyone tries to send mail purporting to be from anything at kepl.com.au and the receiving mail server is checking SPF records, it'll be rejected after one cheap DNS lookup. It's a simple and cacheable way to ask the owning server, "Is this guy allowed to send mail for you?". (The 192.168 block in my SPF record above is permitted to allow some intranet conveniences; omit it unless you need it.) ChrisA From torriem at gmail.com Mon Jul 22 10:08:43 2013 From: torriem at gmail.com (Michael Torrie) Date: Mon, 22 Jul 2013 08:08:43 -0600 Subject: Simple Python script as SMTP server for outgoing e-mails? In-Reply-To: References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> Message-ID: <51ED3CEB.1070706@gmail.com> On 07/22/2013 06:51 AM, Chris Angelico wrote: >> Thanks for the tip. I didn't know about SPF >> http://en.wikipedia.org/wiki/Sender_Policy_Framework > > It's a great way of detecting legit vs forged mail. If anyone tries to > send mail purporting to be from anything at kepl.com.au and the receiving > mail server is checking SPF records, it'll be rejected after one cheap > DNS lookup. It's a simple and cacheable way to ask the owning server, > "Is this guy allowed to send mail for you?". (The 192.168 block in my > SPF record above is permitted to allow some intranet conveniences; > omit it unless you need it.) Yes setting SPF records will help your mail be accepted by other servers, but I disagree with your appeal to make mail server SPF handling as strict as your server does. SPF has problems in a number of situations which could cause legitimate mail to be rejected. In my last job I could only use SPF as one spam factor, not as a basis for rejection. From rosuav at gmail.com Mon Jul 22 10:15:28 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 23 Jul 2013 00:15:28 +1000 Subject: Simple Python script as SMTP server for outgoing e-mails? In-Reply-To: <51ED3CEB.1070706@gmail.com> References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> <51ED3CEB.1070706@gmail.com> Message-ID: On Tue, Jul 23, 2013 at 12:08 AM, Michael Torrie wrote: > On 07/22/2013 06:51 AM, Chris Angelico wrote: >>> Thanks for the tip. I didn't know about SPF >>> http://en.wikipedia.org/wiki/Sender_Policy_Framework >> >> It's a great way of detecting legit vs forged mail. If anyone tries to >> send mail purporting to be from anything at kepl.com.au and the receiving >> mail server is checking SPF records, it'll be rejected after one cheap >> DNS lookup. It's a simple and cacheable way to ask the owning server, >> "Is this guy allowed to send mail for you?". (The 192.168 block in my >> SPF record above is permitted to allow some intranet conveniences; >> omit it unless you need it.) > > Yes setting SPF records will help your mail be accepted by other > servers, but I disagree with your appeal to make mail server SPF > handling as strict as your server does. SPF has problems in a number of > situations which could cause legitimate mail to be rejected. In my last > job I could only use SPF as one spam factor, not as a basis for rejection. If legit mail is rejected for failing an SPF check, it's the sending admin's problem, not yours. You should never have problems with it if it's set up correctly. And since rejected mail gets reported to the transmitting MTA, you don't need to drop it in a spambox or anything. It's not spam, it's simply invalid mail (equivalent to something sent to a dud address). ChrisA From torriem at gmail.com Mon Jul 22 12:25:23 2013 From: torriem at gmail.com (Michael Torrie) Date: Mon, 22 Jul 2013 10:25:23 -0600 Subject: Simple Python script as SMTP server for outgoing e-mails? In-Reply-To: References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> <51ED3CEB.1070706@gmail.com> Message-ID: <51ED5CF3.4030200@gmail.com> On 07/22/2013 08:15 AM, Chris Angelico wrote: > If legit mail is rejected for failing an SPF check, it's the sending > admin's problem, not yours. You should never have problems with it if > it's set up correctly. And since rejected mail gets reported to the > transmitting MTA, you don't need to drop it in a spambox or anything. > It's not spam, it's simply invalid mail (equivalent to something sent > to a dud address). Sure. Tell that to the people you work for who depend on e-mail. When I was a sysadmin (quite recently), I'd have gotten fired for enforcing such an arbitrary policy. Indeed when mail wasn't coming through that someone in the organization was expecting and wanting, regardless of SPF, it was indeed *my* problem and my job was on the line. BOFH attitudes simply aren't going to change that reality. SPF is just one more of the many things that are contributing overall to absolutely breaking and demise of SMTP. I'm afraid when it does finally cease to work, it's going to be replaced with less open, centrally-controlled messaging systems like facebook. Which is unfortunate. From rosuav at gmail.com Mon Jul 22 12:32:54 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 23 Jul 2013 02:32:54 +1000 Subject: Simple Python script as SMTP server for outgoing e-mails? In-Reply-To: <51ED5CF3.4030200@gmail.com> References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> <51ED3CEB.1070706@gmail.com> <51ED5CF3.4030200@gmail.com> Message-ID: On Tue, Jul 23, 2013 at 2:25 AM, Michael Torrie wrote: > On 07/22/2013 08:15 AM, Chris Angelico wrote: >> If legit mail is rejected for failing an SPF check, it's the sending >> admin's problem, not yours. You should never have problems with it if >> it's set up correctly. And since rejected mail gets reported to the >> transmitting MTA, you don't need to drop it in a spambox or anything. >> It's not spam, it's simply invalid mail (equivalent to something sent >> to a dud address). > > Sure. Tell that to the people you work for who depend on e-mail. When I > was a sysadmin (quite recently), I'd have gotten fired for enforcing > such an arbitrary policy. Indeed when mail wasn't coming through that > someone in the organization was expecting and wanting, regardless of > SPF, it was indeed *my* problem and my job was on the line. BOFH > attitudes simply aren't going to change that reality. Is your job on the line if the sender of that email got the recipient's address right? Is your job on the line if the sender mucked up his SMTP settings and the message didn't even get to your server? Is your job on the line if the email never even got sent? Then why should your job be on the line if the sender violates his own declared protocol? Remember, if you don't publish an SPF record, your emails will be accepted regardless. It's only if you explicitly create that DNS record that ends with "-all" that any of this will happen - which means you *asked* for that mail to be rejected. If you do that and then send mail from a different IP, then I *will* reject it. Accepting mail and just giving it a spam score is *worse*, because the sender won't even know why it didn't get through (what if most of his mail gets accepted, but that one email when he sent a blank body, subject "RE: your invoice", and a zip file attachment, managed to trip the spam cutoff and get dumped?), whereas rejecting will result in a quick and easy bounce, probably within seconds (minutes maybe). I stand by SPF checking. It has never been a problem. If you don't stand by protocols, you weaken those protocols. And speaking of protocols, I'm now going to have to follow the "I'm on an airliner and mobile phones have to be turned off" protocol, as the flight's due to depart shortly. Ah, protocols... some you love, some not so much. ChrisA From duncan.booth at invalid.invalid Tue Jul 23 04:06:00 2013 From: duncan.booth at invalid.invalid (Duncan Booth) Date: 23 Jul 2013 08:06:00 GMT Subject: Simple Python script as SMTP server for outgoing e-mails? References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> <51ED3CEB.1070706@gmail.com> Message-ID: Chris Angelico wrote: > On Tue, Jul 23, 2013 at 12:08 AM, Michael Torrie > wrote: >> On 07/22/2013 06:51 AM, Chris Angelico wrote: >>>> Thanks for the tip. I didn't know about SPF >>>> http://en.wikipedia.org/wiki/Sender_Policy_Framework >>> >>> It's a great way of detecting legit vs forged mail. If anyone tries >>> to send mail purporting to be from anything at kepl.com.au and the >>> receiving mail server is checking SPF records, it'll be rejected >>> after one cheap DNS lookup. It's a simple and cacheable way to ask >>> the owning server, "Is this guy allowed to send mail for you?". (The >>> 192.168 block in my SPF record above is permitted to allow some >>> intranet conveniences; omit it unless you need it.) >> >> Yes setting SPF records will help your mail be accepted by other >> servers, but I disagree with your appeal to make mail server SPF >> handling as strict as your server does. SPF has problems in a number >> of situations which could cause legitimate mail to be rejected. In >> my last job I could only use SPF as one spam factor, not as a basis >> for rejection. > > If legit mail is rejected for failing an SPF check, it's the sending > admin's problem, not yours. You should never have problems with it if > it's set up correctly. And since rejected mail gets reported to the > transmitting MTA, you don't need to drop it in a spambox or anything. > It's not spam, it's simply invalid mail (equivalent to something sent > to a dud address). > If you want your emails to have the best chance of arriving your SPF should list servers you use but not deny that there might be others. I have a very common situation where an overly strict SPF may cause problems: Like many people I have multiple email addresses which all end up in the same inbox. The one I most commonly give out to businesses bounces the email unchanged to the gmail inbox that I use. That means all emails I receive through that email address appear to Google to have originated from the forwarding servers. An SPF record from the original sender that claims to have a complete list of originating servers will therefore fail validation. It isn't Google's fault: they can't ignore the forwarding step otherwise spammers could bypass SPF simply by claiming to be forwarding the emails. It is simply a limitation of the SPF protocol. Fortunately they only use SPF as one indicator so real messages still get through. -- Duncan Booth From rosuav at gmail.com Tue Jul 23 05:19:25 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 23 Jul 2013 19:19:25 +1000 Subject: Simple Python script as SMTP server for outgoing e-mails? In-Reply-To: References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> <51ED3CEB.1070706@gmail.com> Message-ID: On Tue, Jul 23, 2013 at 6:06 PM, Duncan Booth wrote: > Chris Angelico wrote: > >> On Tue, Jul 23, 2013 at 12:08 AM, Michael Torrie >> wrote: >>> On 07/22/2013 06:51 AM, Chris Angelico wrote: >>>>> Thanks for the tip. I didn't know about SPF >>>>> http://en.wikipedia.org/wiki/Sender_Policy_Framework >>>> >>>> It's a great way of detecting legit vs forged mail. If anyone tries >>>> to send mail purporting to be from anything at kepl.com.au and the >>>> receiving mail server is checking SPF records, it'll be rejected >>>> after one cheap DNS lookup. It's a simple and cacheable way to ask >>>> the owning server, "Is this guy allowed to send mail for you?". (The >>>> 192.168 block in my SPF record above is permitted to allow some >>>> intranet conveniences; omit it unless you need it.) >>> >>> Yes setting SPF records will help your mail be accepted by other >>> servers, but I disagree with your appeal to make mail server SPF >>> handling as strict as your server does. SPF has problems in a number >>> of situations which could cause legitimate mail to be rejected. In >>> my last job I could only use SPF as one spam factor, not as a basis >>> for rejection. >> >> If legit mail is rejected for failing an SPF check, it's the sending >> admin's problem, not yours. You should never have problems with it if >> it's set up correctly. And since rejected mail gets reported to the >> transmitting MTA, you don't need to drop it in a spambox or anything. >> It's not spam, it's simply invalid mail (equivalent to something sent >> to a dud address). >> > If you want your emails to have the best chance of arriving your SPF should > list servers you use but not deny that there might be others. That usually makes the SPF record completely useless. The whole point is to say that random addresses on the internet _will not_ send mail from you. > I have a very common situation where an overly strict SPF may cause > problems: > > Like many people I have multiple email addresses which all end up in the > same inbox. The one I most commonly give out to businesses bounces the > email unchanged to the gmail inbox that I use. That means all emails I > receive through that email address appear to Google to have originated from > the forwarding servers. An SPF record from the original sender that claims > to have a complete list of originating servers will therefore fail > validation. Ah, there's a solution to this one. You simply use your own envelope-from address; SPF shouldn't be being checked for the From: header. Forwarding and using the original sender's address in the SMTP 'MAIL FROM' command is forging mail from them, so it is correct for that to be thrown out. The mail is coming from your own account, so you put your address in it, and you might even be able to put an uber-strict SPF record like "v=spf1 ip4:1.2.3.4 -all" which is quick to process and guarantees that nobody can pretend to forward mail on your behalf. The checks are for the *current connection*, not anything earlier. ChrisA From rosuav at gmail.com Tue Jul 23 05:30:36 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 23 Jul 2013 19:30:36 +1000 Subject: Simple Python script as SMTP server for outgoing e-mails? In-Reply-To: References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> <51ED3CEB.1070706@gmail.com> Message-ID: On Tue, Jul 23, 2013 at 7:19 PM, Chris Angelico wrote: > Ah, there's a solution to this one. You simply use your own > envelope-from address; SPF shouldn't be being checked for the From: > header. There's an example, by the way, of this exact technique right here - python-list at python.org sends mail to me with an envelope-from of "python-list-bounces+rosuav=gmail.com at python.org" - which passes SPF, since python.org has a TXT record designating the sending IP as one of theirs. It doesn't matter that invalid.invalid (your supposed domain) doesn't have an SPF record, nor would it be a problem if it had one that said "v=spf1 -all", because that domain wasn't checked. Mailing lists are doing the same sort of forwarding that you're doing. (Apologies to those who read this as a newsgroup, for whom this won't be as parallel an example. But it's still the case, just not for the posts you receive.) ChrisA From torriem at gmail.com Tue Jul 23 11:12:47 2013 From: torriem at gmail.com (Michael Torrie) Date: Tue, 23 Jul 2013 09:12:47 -0600 Subject: [OT] SPF - was Re: Simple Python script as SMTP server for outgoing e-mails? In-Reply-To: References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> <51ED3CEB.1070706@gmail.com> Message-ID: <51EE9D6F.80403@gmail.com> On 07/23/2013 03:30 AM, Chris Angelico wrote: > On Tue, Jul 23, 2013 at 7:19 PM, Chris Angelico wrote: >> Ah, there's a solution to this one. You simply use your own >> envelope-from address; SPF shouldn't be being checked for the From: >> header. > > There's an example, by the way, of this exact technique right here - > python-list at python.org sends mail to me with an envelope-from of > "python-list-bounces+rosuav=gmail.com at python.org" - which passes SPF, > since python.org has a TXT record designating the sending IP as one of > theirs. It doesn't matter that invalid.invalid (your supposed domain) > doesn't have an SPF record, nor would it be a problem if it had one > that said "v=spf1 -all", because that domain wasn't checked. Mailing > lists are doing the same sort of forwarding that you're doing. This is good and all, and I think I will modify my local postfix mail server I use for personal stuff, just for correctness' sake. I hadn't spent much time studying SPF in depth before, but after reading your comments (which were insightful) I'm now more convinced that SPF is worthless than ever, at least as a spam prevention mechanism. Spammers can use throwaway domains that publish very non-strict SPF records, and spam to their hearts content with random forged from addresses and SPF checks pass. The only way around that is to enforce SPF on the From: header in the e-mail itself, which we all agree is broken. I've been reading this: http://www.openspf.org/FAQ/SPF_is_not_about_spam Not very encouraging. When the other expensive options for going after spammers who have valid SPF records, they propose domain blacklists as a solution. From rosuav at gmail.com Tue Jul 23 17:47:28 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 24 Jul 2013 07:47:28 +1000 Subject: [OT] SPF - was Re: Simple Python script as SMTP server for outgoing e-mails? In-Reply-To: <51EE9D6F.80403@gmail.com> References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> <51ED3CEB.1070706@gmail.com> <51EE9D6F.80403@gmail.com> Message-ID: On Wed, Jul 24, 2013 at 1:12 AM, Michael Torrie wrote: > On 07/23/2013 03:30 AM, Chris Angelico wrote: >> On Tue, Jul 23, 2013 at 7:19 PM, Chris Angelico wrote: >>> Ah, there's a solution to this one. You simply use your own >>> envelope-from address; SPF shouldn't be being checked for the From: >>> header. >> >> There's an example, by the way, of this exact technique right here - >> python-list at python.org sends mail to me with an envelope-from of >> "python-list-bounces+rosuav=gmail.com at python.org" - which passes SPF, >> since python.org has a TXT record designating the sending IP as one of >> theirs. It doesn't matter that invalid.invalid (your supposed domain) >> doesn't have an SPF record, nor would it be a problem if it had one >> that said "v=spf1 -all", because that domain wasn't checked. Mailing >> lists are doing the same sort of forwarding that you're doing. > > This is good and all, and I think I will modify my local postfix mail > server I use for personal stuff, just for correctness' sake. Correctness is a worthwhile reason to do something :) > I hadn't spent much time studying SPF in depth before, but after reading > your comments (which were insightful) I'm now more convinced that SPF is > worthless than ever, at least as a spam prevention mechanism. Spammers > can use throwaway domains that publish very non-strict SPF records, and > spam to their hearts content with random forged from addresses and SPF > checks pass. The only way around that is to enforce SPF on the From: > header in the e-mail itself, which we all agree is broken. I've been > reading this: > > http://www.openspf.org/FAQ/SPF_is_not_about_spam There are several things that SPF achieves, but mainly it's a measure of trust. If you receive email from a domain I run, and the SPF record permits the IP that sent it to you, you can have a high degree of confidence that it really is from that domain. Suppose, for instance, that (pick a bank, any bank) has a strict SPF record. Someone tries to send a phishing email purporting to be from that bank. They then have to use a different envelope-from address, which instantly marks the mail as suspicious to anyone who's checking. But more likely, what they'll do is simply ignore SPF and send it anyway. That means that any MTA that checks SPF records is immediately freed of all that bad mail - which is more than just spam, it's a major vulnerability (thinking here of corporate networks where all the company's mail goes through a central server, and then a whole lot of non-technical people read it). In the same way that banks assure us that they will *never* ask for your password, they could also assure us that they will *never* send account information from any other domain. Spammers look for the easy pickings. If your X million addresses become (X-1),999,900 because a few servers are rejecting their mail, what do they care? But those hundred people now haven't seen that spam. Sure, spammers can easily get around SPF checks... but that won't get all that likely until the bulk of MTAs start checking. For now, we can take all the benefit. Later on, the world can look to other solutions. ChrisA From steve+comp.lang.python at pearwood.info Tue Jul 23 21:42:26 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 24 Jul 2013 01:42:26 GMT Subject: non sequitur: [OT] SPF - was Re: Simple Python script as SMTP server for outgoing e-mails? References: <51ED3CEB.1070706@gmail.com> <51EE9D6F.80403@gmail.com> Message-ID: <51ef3101$0$29971$c3e8da3$5496439d@news.astraweb.com> On Tue, 23 Jul 2013 19:59:01 -0400, Dennis Lee Bieber wrote: > {Liaden culture seems heavy on personal honor, and comments tend (to me) > be worded to avoid any chance of being interpreted as disparaging of the > person with whom one is speaking... Hmmm, pity such modes can't be > enforced on the newsgroups } Are you implying that failure to avoid disparaging others in newsgroups is harmful? That disparages me. -- Steven From duncan.booth at invalid.invalid Tue Jul 23 06:06:44 2013 From: duncan.booth at invalid.invalid (Duncan Booth) Date: 23 Jul 2013 10:06:44 GMT Subject: Simple Python script as SMTP server for outgoing e-mails? References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> <51ED3CEB.1070706@gmail.com> Message-ID: Chris Angelico wrote: > On Tue, Jul 23, 2013 at 6:06 PM, Duncan Booth wrote: >> I have a very common situation where an overly strict SPF may cause >> problems: >> >> Like many people I have multiple email addresses which all end up in >> the same inbox. The one I most commonly give out to businesses >> bounces the email unchanged to the gmail inbox that I use. That means >> all emails I receive through that email address appear to Google to >> have originated from the forwarding servers. An SPF record from the >> original sender that claims to have a complete list of originating >> servers will therefore fail validation. > > Ah, there's a solution to this one. You simply use your own > envelope-from address; SPF shouldn't be being checked for the From: > header. Forwarding and using the original sender's address in the SMTP > 'MAIL FROM' command is forging mail from them, so it is correct for > that to be thrown out. The mail is coming from your own account, so > you put your address in it, and you might even be able to put an > uber-strict SPF record like "v=spf1 ip4:1.2.3.4 -all" which is quick > to process and guarantees that nobody can pretend to forward mail on > your behalf. The checks are for the *current connection*, not anything > earlier. > Excellent idea, I'll tell the email forwarding service to rewrite their system immediately. Or I could just tell Google to rewrite their email system to know about and strip off the forwarding service's headers: that's probably about as easy. Or maybe I could just ask you to add the forwarder's SPF record into your own? I know that I could arrange things so that my emails don't trigger this situation, but that isn't the point. The point is that this situation happens quite commonly, therefore you as the sender of an email with a strict SPF are going to find systems rejecting emails you send that would get through if you have a less strict one. That is of course your choice, but many users of email would prefer to maximise the chance of the email they send arriving rather than reducing slightly the chance of people they may not even know receiving spam. You could also try combining SPF with DKIM although that has its own, different failure scenarios. -- Duncan Booth From vincent.vandevyvre at swing.be Tue Jul 23 07:42:13 2013 From: vincent.vandevyvre at swing.be (Vincent Vande Vyvre) Date: Tue, 23 Jul 2013 13:42:13 +0200 Subject: Strange behaviour with os.linesep In-Reply-To: References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> <51ED3CEB.1070706@gmail.com> Message-ID: <51EE6C15.5070701@swing.be> On Windows a script where de endline are the system line sep, the files are open with a double line in Eric4, Notepad++ or Gedit but they are correctly displayed in the MS Bloc-Notes. Example with this code: ---------------------------------------------- # -*- coding: utf-8 -*- import os L_SEP = os.linesep def write(): strings = ['# -*- coding: utf-8 -*-\n', 'import os\n', 'import sys\n'] with open('writetest.py', 'w') as outf: for s in strings: outf.write(s.replace('\n', L_SEP)) write() ---------------------------------------------- The syntax `s.replace('\n', L_SEP)`is required for portability. Regards - Vincent V.V From jason.swails at gmail.com Tue Jul 23 08:39:59 2013 From: jason.swails at gmail.com (Jason Swails) Date: Tue, 23 Jul 2013 08:39:59 -0400 Subject: Strange behaviour with os.linesep In-Reply-To: <51EE6C15.5070701@swing.be> References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> <51ED3CEB.1070706@gmail.com> <51EE6C15.5070701@swing.be> Message-ID: On Tue, Jul 23, 2013 at 7:42 AM, Vincent Vande Vyvre < vincent.vandevyvre at swing.be> wrote: > On Windows a script where de endline are the system line sep, the files > are open with a double line in Eric4, Notepad++ or Gedit but they are > correctly displayed in the MS Bloc-Notes. > > Example with this code: > ------------------------------**---------------- > # -*- coding: utf-8 -*- > > import os > L_SEP = os.linesep > > def write(): > strings = ['# -*- coding: utf-8 -*-\n', > 'import os\n', > 'import sys\n'] > with open('writetest.py', 'w') as outf: > for s in strings: > outf.write(s.replace('\n', L_SEP)) > I must ask why you are setting strings with a newline line ending only to replace them later with os.linesep. This seems convoluted compared to doing something like def write(): strings = ['#-*- coding: utf-8 -*-', 'import os', 'import sys'] with open('writetest.py', 'w') as outf: for s in strings: outf.write(s) outf.write(L_SEP) Or something equivalent. If, however, the source strings come from a file you've created somewhere (and are loaded by reading in that file line by line), then I can see a problem. DOS line endings are carriage returns ('\r\n'), whereas standard UNIX files use just newlines ('\n'). Therefore, if you are using the code: s.replace('\n', L_SEP) in Windows, using a Windows-generated file, then what you are likely doing is converting the string sequence '\r\n' into '\r\r\n', which is not what you want to do. I can imagine some text editors interpreting that as two endlines (since there are 2 \r's). Indeed, when I execute the code: >>> l = open('test.txt', 'w') >>> l.write('This is the first line\r\r\n') >>> l.write('This is the second\r\r\n') >>> l.close() on UNIX and open the resulting file in gedit, it is double-spaced, but if I just dump it to the screen using 'cat', it is single-spaced. If you want to make your code a bit more cross-platform, you should strip out all types of end line characters from the strings before you write them. So something like this: with open('writetest.py', 'w') as outf: for s in strings: outf.write(s.rstrip('\r\n')) outf.write(L_SEP) Hope this helps, Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: From vincent.vandevyvre at swing.be Tue Jul 23 09:10:35 2013 From: vincent.vandevyvre at swing.be (Vincent Vande Vyvre) Date: Tue, 23 Jul 2013 15:10:35 +0200 Subject: Strange behaviour with os.linesep In-Reply-To: References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> <51ED3CEB.1070706@gmail.com> <51EE6C15.5070701@swing.be> Message-ID: <51EE80CB.2010100@swing.be> Le 23/07/2013 14:39, Jason Swails a ?crit : > > > > On Tue, Jul 23, 2013 at 7:42 AM, Vincent Vande Vyvre > > wrote: > > On Windows a script where de endline are the system line sep, the > files are open with a double line in Eric4, Notepad++ or Gedit but > they are correctly displayed in the MS Bloc-Notes. > > Example with this code: > ---------------------------------------------- > # -*- coding: utf-8 -*- > > import os > L_SEP = os.linesep > > def write(): > strings = ['# -*- coding: utf-8 -*-\n', > 'import os\n', > 'import sys\n'] > with open('writetest.py', 'w') as outf: > for s in strings: > outf.write(s.replace('\n', L_SEP)) > > > I must ask why you are setting strings with a newline line ending only > to replace them later with os.linesep. This seems convoluted compared > to doing something like > > def write(): > strings = ['#-*- coding: utf-8 -*-', 'import os', 'import sys'] > with open('writetest.py', 'w') as outf: > for s in strings: > outf.write(s) > outf.write(L_SEP) > > Or something equivalent. > > If, however, the source strings come from a file you've created > somewhere (and are loaded by reading in that file line by line), then > I can see a problem. DOS line endings are carriage returns ('\r\n'), > whereas standard UNIX files use just newlines ('\n'). Therefore, if > you are using the code: > > s.replace('\n', L_SEP) > > in Windows, using a Windows-generated file, then what you are likely > doing is converting the string sequence '\r\n' into '\r\r\n', which is > not what you want to do. I can imagine some text editors interpreting > that as two endlines (since there are 2 \r's). Indeed, when I execute > the code: > > >>> l = open('test.txt', 'w') > >>> l.write('This is the first line\r\r\n') > >>> l.write('This is the second\r\r\n') > >>> l.close() > > on UNIX and open the resulting file in gedit, it is double-spaced, but > if I just dump it to the screen using 'cat', it is single-spaced. > > If you want to make your code a bit more cross-platform, you should > strip out all types of end line characters from the strings before you > write them. So something like this: > > with open('writetest.py', 'w') as outf: > for s in strings: > outf.write(s.rstrip('\r\n')) > outf.write(L_SEP) > > Hope this helps, > Jason The '\n' are in the original file. I've tested these other versions: ------------------------------- def write(): strings = ['# -*- coding: utf-8 -*-\n', 'import os\n', 'import sys\n'] with open('writetest.py', 'w') as outf: txt = L_SEP.join([s.rstip() for s in strings]): outf.write(txt) ------------------------------ ------------------------------- def write(): strings = ['# -*- coding: utf-8 -*-', 'import os', 'import sys'] with open('writetest.py', 'w') as outf: txt = L_SEP.join( strings): outf.write(txt) ------------------------------ Las, no changes, always correctly displayed in MS bloc-notes but with double line in other ?ditors. -- Vincent V.V. Oqapy . Qarte . PaQager From vincent.vandevyvre at swing.be Tue Jul 23 09:26:07 2013 From: vincent.vandevyvre at swing.be (Vincent Vande Vyvre) Date: Tue, 23 Jul 2013 15:26:07 +0200 Subject: Strange behaviour with os.linesep In-Reply-To: <51EE80CB.2010100@swing.be> References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> <51ED3CEB.1070706@gmail.com> <51EE6C15.5070701@swing.be> <51EE80CB.2010100@swing.be> Message-ID: <51EE846F.7010209@swing.be> Le 23/07/2013 15:10, Vincent Vande Vyvre a ?crit : > The '\n' are in the original file. > > I've tested these other versions: > > ------------------------------- > def write(): > strings = ['# -*- coding: utf-8 -*-\n', > 'import os\n', > 'import sys\n'] > with open('writetest.py', 'w') as outf: > txt = L_SEP.join([s.rstip() for s in strings]): > outf.write(txt) > ------------------------------ > > ------------------------------- > def write(): > strings = ['# -*- coding: utf-8 -*-', > 'import os', > 'import sys'] > with open('writetest.py', 'w') as outf: > txt = L_SEP.join( strings): > outf.write(txt) > ------------------------------ > > Las, no changes, always correctly displayed in MS bloc-notes but with > double line in other ?ditors. > Also with: ---------------------------------------- def count(): with open('c:\\Users\\Vincent\\writetest.py', 'r') as inf: lines = inf.readlines() for l in lines: print(l, len(l)) count() -------------------------------------- The output is: -------------------------------------- ('# -*- coding: utf-8 -*-\r\n', 25) ('import os\r\n', 11) ('import sys', 10) -- Vincent V.V. Oqapy . Qarte . PaQager From jason.swails at gmail.com Tue Jul 23 09:35:27 2013 From: jason.swails at gmail.com (Jason Swails) Date: Tue, 23 Jul 2013 09:35:27 -0400 Subject: Strange behaviour with os.linesep In-Reply-To: <51EE846F.7010209@swing.be> References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> <51ED3CEB.1070706@gmail.com> <51EE6C15.5070701@swing.be> <51EE80CB.2010100@swing.be> <51EE846F.7010209@swing.be> Message-ID: On Tue, Jul 23, 2013 at 9:26 AM, Vincent Vande Vyvre < vincent.vandevyvre at swing.be> wrote: > Le 23/07/2013 15:10, Vincent Vande Vyvre a ?crit : > > The '\n' are in the original file. >> >> I've tested these other versions: >> >> ------------------------------**- >> def write(): >> strings = ['# -*- coding: utf-8 -*-\n', >> 'import os\n', >> 'import sys\n'] >> with open('writetest.py', 'w') as outf: >> txt = L_SEP.join([s.rstip() for s in strings]): >> outf.write(txt) >> ------------------------------ >> >> ------------------------------**- >> def write(): >> strings = ['# -*- coding: utf-8 -*-', >> 'import os', >> 'import sys'] >> with open('writetest.py', 'w') as outf: >> txt = L_SEP.join( strings): >> outf.write(txt) >> ------------------------------ >> >> Las, no changes, always correctly displayed in MS bloc-notes but with >> double line in other ?ditors. >> >> > Also with: > > ------------------------------**---------- > def count(): > with open('c:\\Users\\Vincent\\**writetest.py', 'r') as inf: > lines = inf.readlines() > for l in lines: > print(l, len(l)) > Unrelated comment, but in general it's (much) more efficient to iterate through a file rather than iterate through a list of strings generated by readlines(): def count(): with open('c:\\Users\\Vincent\\writetest.py', 'r') as inf: for l in lines: print(l, len(l)) It's also fewer lines of code. ('# -*- coding: utf-8 -*-\r\n', 25) > ('import os\r\n', 11) > ('import sys', 10) Then it seems like there is an issue with your text editors that do not play nicely with DOS-style line endings. Gedit on my linux machine displays the line endings correctly (that is, '\r\n' is rendered as a single line). Good luck, Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve+comp.lang.python at pearwood.info Tue Jul 23 11:25:12 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 23 Jul 2013 15:25:12 GMT Subject: Strange behaviour with os.linesep References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> <51ED3CEB.1070706@gmail.com> Message-ID: <51eea057$0$29971$c3e8da3$5496439d@news.astraweb.com> On Tue, 23 Jul 2013 13:42:13 +0200, Vincent Vande Vyvre wrote: > On Windows a script where de endline are the system line sep, the files > are open with a double line in Eric4, Notepad++ or Gedit but they are > correctly displayed in the MS Bloc-Notes. I suspect the problem lies with Eric4, Notepad++ and Gedit. Do you perhaps have to manually tell them that the file uses Windows line separators? I recommend opening the file in a hex editor and seeing for yourself what line separators are used. > Example with this code: > ---------------------------------------------- > # -*- coding: utf-8 -*- > > import os > L_SEP = os.linesep > > def write(): > strings = ['# -*- coding: utf-8 -*-\n', > 'import os\n', > 'import sys\n'] > with open('writetest.py', 'w') as outf: > for s in strings: > outf.write(s.replace('\n', L_SEP)) > > write() > ---------------------------------------------- > > The syntax `s.replace('\n', L_SEP)`is required for portability. I don't think it is. Behaviour is a little different between Python 2 and 3, but by default, Python uses "Universal Newlines". When you open a file in text mode, arbitrary line separators should be automatically translated to \n when reading, and \n will be automatically translated to os.line_sep when writing. http://docs.python.org/3/library/functions.html#open http://docs.python.org/2/library/functions.html#open Some further discussion here: http://stackoverflow.com/questions/12193047/is-universal-newlines-mode- supposed-to-be-default-behaviour-for-open-in-python -- Steven From vincent.vandevyvre at swing.be Wed Jul 24 03:02:37 2013 From: vincent.vandevyvre at swing.be (Vincent Vande Vyvre) Date: Wed, 24 Jul 2013 09:02:37 +0200 Subject: Strange behaviour with os.linesep In-Reply-To: <51eea057$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> <51ED3CEB.1070706@gmail.com> <51eea057$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51EF7C0D.1040101@swing.be> Le 23/07/2013 17:25, Steven D'Aprano a ?crit : > On Tue, 23 Jul 2013 13:42:13 +0200, Vincent Vande Vyvre wrote: > >> On Windows a script where de endline are the system line sep, the files >> are open with a double line in Eric4, Notepad++ or Gedit but they are >> correctly displayed in the MS Bloc-Notes. > I suspect the problem lies with Eric4, Notepad++ and Gedit. Do you > perhaps have to manually tell them that the file uses Windows line > separators? > > I recommend opening the file in a hex editor and seeing for yourself what > line separators are used. > > >> Example with this code: >> ---------------------------------------------- >> # -*- coding: utf-8 -*- >> >> import os >> L_SEP = os.linesep >> >> def write(): >> strings = ['# -*- coding: utf-8 -*-\n', >> 'import os\n', >> 'import sys\n'] >> with open('writetest.py', 'w') as outf: >> for s in strings: >> outf.write(s.replace('\n', L_SEP)) >> >> write() >> ---------------------------------------------- >> >> The syntax `s.replace('\n', L_SEP)`is required for portability. > I don't think it is. Behaviour is a little different between Python 2 and > 3, but by default, Python uses "Universal Newlines". When you open a file > in text mode, arbitrary line separators should be automatically > translated to \n when reading, and \n will be automatically translated to > os.line_sep when writing. > > > http://docs.python.org/3/library/functions.html#open > http://docs.python.org/2/library/functions.html#open > > Some further discussion here: > > http://stackoverflow.com/questions/12193047/is-universal-newlines-mode- > supposed-to-be-default-behaviour-for-open-in-python > > > In fact, in my code, the original file is open in binary mode, the line separator is translate to \n and it is parsed by the module tokenise. I'm not a Windows user but my code must be run also on Win, this is the reason of the usage of os.linesep in writting. So, now I found the solution, just write the file in binary mode and now it is correctly open with all the editors. Thanks all -- Vincent V.V. Oqapy . Qarte . PaQager From rosuav at gmail.com Wed Jul 24 03:39:26 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 24 Jul 2013 17:39:26 +1000 Subject: Strange behaviour with os.linesep In-Reply-To: <51EF7C0D.1040101@swing.be> References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> <51ED3CEB.1070706@gmail.com> <51eea057$0$29971$c3e8da3$5496439d@news.astraweb.com> <51EF7C0D.1040101@swing.be> Message-ID: On Wed, Jul 24, 2013 at 5:02 PM, Vincent Vande Vyvre wrote: > In fact, in my code, the original file is open in binary mode, the line > separator is translate to \n and it is parsed by the module tokenise. > > I'm not a Windows user but my code must be run also on Win, this is the > reason of the usage of os.linesep in writting. > > So, now I found the solution, just write the file in binary mode and now it > is correctly open with all the editors. > Sounds to me like the problem was double-translation - you in your code turned \n into \r\n, but then the file was written in text mode, doing the same translation, so your lines were terminated with \r\r\n. That would result in what you're seeing (including the oddity that some editors will show the file differently). You may find it easier to write the file in text mode and let the underlying system do the translation for you. Chances are that'll be correct. ChrisA From tjreedy at udel.edu Wed Jul 24 12:01:33 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 24 Jul 2013 12:01:33 -0400 Subject: Strange behaviour with os.linesep In-Reply-To: References: <51ED3CEB.1070706@gmail.com> <51eea057$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 7/23/2013 7:41 PM, Dennis Lee Bieber wrote: > On 23 Jul 2013 15:25:12 GMT, Steven D'Aprano > declaimed the following: > >> On Tue, 23 Jul 2013 13:42:13 +0200, Vincent Vande Vyvre wrote: >> >>> On Windows a script where de endline are the system line sep, the files >>> are open with a double line in Eric4, Notepad++ or Gedit but they are >>> correctly displayed in the MS Bloc-Notes. >> >> I suspect the problem lies with Eric4, Notepad++ and Gedit. Do you >> perhaps have to manually tell them that the file uses Windows line >> separators? I suspect the problem likes in the file written. Notepad++ works fine with \r\n or \n on input and can produce either on output. > Don't know about those, but SciTE I know has both an menu option for > line ending (, , ), and one for "convert line endings" -- Terry Jan Reedy From rosuav at gmail.com Tue Jul 23 17:37:30 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 24 Jul 2013 07:37:30 +1000 Subject: Simple Python script as SMTP server for outgoing e-mails? In-Reply-To: References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> <51ED3CEB.1070706@gmail.com> Message-ID: On Tue, Jul 23, 2013 at 8:06 PM, Duncan Booth wrote: > Excellent idea, I'll tell the email forwarding service to rewrite their > system immediately. Yes. If they are using your domain in the MAIL FROM command and not using your mail servers, then yes, you should tell them, and use a different service until they fix that. It is not SPF's fault. It is a fundamental error of protocol, which SPF checks are highlighting. ChrisA From esj at harvee.org Mon Jul 22 08:54:11 2013 From: esj at harvee.org (Eric S. Johansson) Date: Mon, 22 Jul 2013 08:54:11 -0400 Subject: Simple Python script as SMTP server for outgoing e-mails? In-Reply-To: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> Message-ID: On Mon, 22 Jul 2013 08:11:25 -0400, Gilles wrote: > On Sun, 21 Jul 2013 18:28:27 -0600, Michael Torrie > wrote: >> The Sendmail MTA has been ported to many platforms including windows. >> But... > > Thanks for the tip. Since I couldn't find a good, basic, native > Windows app, I was indeed about to look at eg. Exim + Cygwin, and > resort to a Linux appliance if none footed the bill. > try http://emailrelay.sourceforge.net/ From torriem at gmail.com Mon Jul 22 10:10:10 2013 From: torriem at gmail.com (Michael Torrie) Date: Mon, 22 Jul 2013 08:10:10 -0600 Subject: Simple Python script as SMTP server for outgoing e-mails? In-Reply-To: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> Message-ID: <51ED3D42.9080102@gmail.com> On 07/22/2013 06:11 AM, Gilles wrote: > On Sun, 21 Jul 2013 18:28:27 -0600, Michael Torrie > wrote: >> The Sendmail MTA has been ported to many platforms including windows. >> But... > > Thanks for the tip. Since I couldn't find a good, basic, native > Windows app, I was indeed about to look at eg. Exim + Cygwin, and > resort to a Linux appliance if none footed the bill. Where did you look? Here's one I found. It's not the real sendmail program, but it implements the interface which is all you need: http://glob.com.au/sendmail/ I just googled for sendmail win32 From nospam at nospam.com Tue Jul 23 17:48:55 2013 From: nospam at nospam.com (Gilles) Date: Tue, 23 Jul 2013 23:48:55 +0200 Subject: Simple Python script as SMTP server for outgoing e-mails? References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> Message-ID: <5gutu89sifgpj4d1b363r00n7rqr33mf7p@4ax.com> On Mon, 22 Jul 2013 08:54:11 -0400, "Eric S. Johansson" wrote: >try http://emailrelay.sourceforge.net/ Thanks. I did find it, but it says it's not a full MTA: "E-MailRelay is not a routing MTA. It forwards e-mail to a pre-configured SMTP server, regardless of any message addressing or DNS redirects." http://emailrelay.sourceforge.net/userguide.html#SH_1_2 IOW, it'll just send outbound e-mails to my ISP's MTA, so I'm back at square one. From nospam at nospam.com Tue Jul 23 17:50:06 2013 From: nospam at nospam.com (Gilles) Date: Tue, 23 Jul 2013 23:50:06 +0200 Subject: Simple Python script as SMTP server for outgoing e-mails? References: <368qu85msgfhuk2j2s13qj0bqn4rkcint9@4ax.com> Message-ID: On Mon, 22 Jul 2013 08:10:10 -0600, Michael Torrie wrote: >Where did you look? Here's one I found. It's not the real sendmail >program, but it implements the interface which is all you need: > >http://glob.com.au/sendmail/ > >I just googled for sendmail win32 Thanks, but I need an MTA, not just a command-line app, so I can send e-mails from my e-mail client and just change the SMTP line in the configuration. From invalid at invalid.invalid Sun Jul 21 17:01:09 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Sun, 21 Jul 2013 21:01:09 +0000 (UTC) Subject: Simple Python script as SMTP server for outgoing e-mails? References: Message-ID: On 2013-07-21, Gilles wrote: > Every once in a while, my ISP's SMTP server refuses to send > perfectly legit e-mails because it considers them as SPAM. > > So I'd like to install a dead-simple SMTP server on my XP computer > just to act as SMTP backup server. All I'd need is to change the SMTP > address in my e-mail client, and off they go. No need for anything > else like user authentication or SPAM control. Unless you've got a static IP address, a domain name, and a valid MX record that will match up when they do a reverse DNS lookup, it's pretty unlikely that you're going to have much luck running an SMTP server. Most other SMTP servers are probably going to ignore or reject your attempts to transfer mail from your own SMTP server. > Is there a no-brainer, ready-to-use solution in Python that I could > use for this? I'd recommend postfix or exim if I was going to try to do it, but I think they're Unix-only. -- Grant From nospam at nospam.com Mon Jul 22 08:13:01 2013 From: nospam at nospam.com (Gilles) Date: Mon, 22 Jul 2013 14:13:01 +0200 Subject: Simple Python script as SMTP server for outgoing e-mails? References: Message-ID: <1c8qu81lnfu182dpl8u61qag0tvhpdmqsh@4ax.com> On Sun, 21 Jul 2013 21:01:09 +0000 (UTC), Grant Edwards wrote: >Unless you've got a static IP address, a domain name, and a valid MX >record that will match up when they do a reverse DNS lookup, it's >pretty unlikely that you're going to have much luck running an SMTP >server. Most other SMTP servers are probably going to ignore or >reject your attempts to transfer mail from your own SMTP server. I had forgotten about this. I'll give a try, and see how it goes. >I'd recommend postfix or exim if I was going to try to do it, but I >think they're Unix-only. Thanks for the tip. Looks like Exim is available on Windows through Cygwin http://blogostuffivelearnt.blogspot.fr/2012/07/smtp-mail-server-with-windows.html From nospam at nospam.com Mon Jul 22 08:19:57 2013 From: nospam at nospam.com (Gilles) Date: Mon, 22 Jul 2013 14:19:57 +0200 Subject: Simple Python script as SMTP server for outgoing e-mails? References: Message-ID: On Sun, 21 Jul 2013 21:01:09 +0000 (UTC), Grant Edwards wrote: >Unless you've got a static IP address, a domain name, and a valid MX >record that will match up when they do a reverse DNS lookup, it's >pretty unlikely that you're going to have much luck running an SMTP >server. Most other SMTP servers are probably going to ignore or >reject your attempts to transfer mail from your own SMTP server. Incidently, how do ISP MTAs find whether the remote MTA is legit or running on some regular user's computer? 1. Query Reverse DNS for IP 2. Find domain 3. Query DNS for MX 4. ? From invalid at invalid.invalid Mon Jul 22 10:10:00 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Mon, 22 Jul 2013 14:10:00 +0000 (UTC) Subject: Simple Python script as SMTP server for outgoing e-mails? References: Message-ID: On 2013-07-22, Gilles wrote: > On Sun, 21 Jul 2013 21:01:09 +0000 (UTC), Grant Edwards > wrote: >>Unless you've got a static IP address, a domain name, and a valid MX >>record that will match up when they do a reverse DNS lookup, it's >>pretty unlikely that you're going to have much luck running an SMTP >>server. Most other SMTP servers are probably going to ignore or >>reject your attempts to transfer mail from your own SMTP server. > > Incidently, how do ISP MTAs find whether the remote MTA is legit or > running on some regular user's computer? > > 1. Query Reverse DNS for IP > 2. Find domain > 3. Query DNS for MX > 4. ? There are a variety of things they check. They've got lists of IP address blocks that they know are residential DSL/cable customers, and sometimes they'll reject mail from those regardless of what you do. Some will compare the reverse-DNS lookup with the headers to make sure you're being honest about things like return-path, some will compare the IP address with the MX record for the domain they got when they did the reverse-lookup-DNS, and they've all probably got a variety of other secret heuristics they use to generate a "SPAM" score. For many years I ran my own SMTP server and had it configured to deliver mail directly to recipients. About 10 years, I had to give up on that because so many SMTP servers were rejecting/ignoring mail I sent. And I did have a static IP with a valid domain and MX record. But it was a residential DSL IP address, and I suspect that was enough to get mail rejected by some servers. -- Grant Edwards grant.b.edwards Yow! Well, O.K. at I'll compromise with my gmail.com principles because of EXISTENTIAL DESPAIR! From torriem at gmail.com Mon Jul 22 10:21:02 2013 From: torriem at gmail.com (Michael Torrie) Date: Mon, 22 Jul 2013 08:21:02 -0600 Subject: Simple Python script as SMTP server for outgoing e-mails? In-Reply-To: References: Message-ID: <51ED3FCE.3090506@gmail.com> On 07/22/2013 06:19 AM, Gilles wrote: > On Sun, 21 Jul 2013 21:01:09 +0000 (UTC), Grant Edwards > wrote: >> Unless you've got a static IP address, a domain name, and a valid MX >> record that will match up when they do a reverse DNS lookup, it's >> pretty unlikely that you're going to have much luck running an SMTP >> server. Most other SMTP servers are probably going to ignore or >> reject your attempts to transfer mail from your own SMTP server. > > Incidently, how do ISP MTAs find whether the remote MTA is legit or > running on some regular user's computer? > > 1. Query Reverse DNS for IP > 2. Find domain > 3. Query DNS for MX > 4. ? My mail server did a number of things: 1. ensure IP address of sending server has a reverse name (domain didn't particularly matter) 2. ensure the HELO address in SMTP matches IP address of sending server 3. check sender IP address against spam blacklists, which includes netblocks of home ISPs, some entire countries, flagged subnets 4. greylist sender IP if the recipient requested it. First connection always fails with a nonfatal server error, next connection must wait at least 5 minutes. If a reconnection happened too quickly, the IP was temporarily black listed. After success, IP address is whitelisted for a time. A commandline MTA will not be able to get through greylisting; only a mail server with queuing could. Spambots tend to give up on the first error, even now. Cheaper targets I guess. 5. spamassassin checked SPF (DNS) and domainkeys (message itself) and weighted the spam factor accordingly I think there were other basic rules that sendmail applied to the sender, but I can't remember all of what they are. This is well and truly off topic now for the python list, though. From rosuav at gmail.com Mon Jul 22 12:12:22 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 23 Jul 2013 02:12:22 +1000 Subject: Simple Python script as SMTP server for outgoing e-mails? In-Reply-To: <51ED3FCE.3090506@gmail.com> References: <51ED3FCE.3090506@gmail.com> Message-ID: On Tue, Jul 23, 2013 at 12:21 AM, Michael Torrie wrote: > My mail server did a number of things: > 1. ensure IP address of sending server has a reverse name (domain didn't > particularly matter) > 2. ensure the HELO address in SMTP matches IP address of sending server > 3. check sender IP address against spam blacklists, which includes > netblocks of home ISPs, some entire countries, flagged subnets > 4. greylist sender IP if the recipient requested it. First connection > always fails with a nonfatal server error, next connection must wait at > least 5 minutes. If a reconnection happened too quickly, the IP was > temporarily black listed. After success, IP address is whitelisted for > a time. A commandline MTA will not be able to get through greylisting; > only a mail server with queuing could. Spambots tend to give up on the > first error, even now. Cheaper targets I guess. > 5. spamassassin checked SPF (DNS) and domainkeys (message itself) and > weighted the spam factor accordingly > > I think there were other basic rules that sendmail applied to the > sender, but I can't remember all of what they are. This is well and > truly off topic now for the python list, though. And yet off-topic does happen... For what it's worth, here's how my server is set up: 1. A variety of protocol-level checks. If you don't say HELO, for instance, you get rejected. Surprisingly, these simple checks actually keep out a lot of spam - but I've yet to see any legiit mail blocked by them. (Not that I keep logs of these any more. I stopped watching after it looked clean for a while.) And if legit mail is rejected, it'll be resent or bounced by the sending MTA anyway. 2. SPF checks on the MAIL FROM:<> address. Again, if legit mail gets rejected (which would be the fault of the sending domain owner), the server at the previous hop will deal with it. Only hard failures get thrown out; anything else just gets marked (which we usually ignore) and delivered as normal, not even spam-scored. 3. Bayesian spam filter, set very conservatively so we get false negatives but (almost) no false positives. Any spam that gets through these three checks gets delivered, and then the users will drop it in their junk folder. Every week I do a train-and-wipe run across all junk folders, which logs spam counts from our primary mailboxes. Last week's run was 228 spam across the six logged accounts (some of those accounts collect from many addresses), or an average of five false negatives per account per day, and false positives are almost completely unheard-of. Considering how much spam assaults the outside of my fortress's walls, that's a fairly good ratio, I think. SPF for the win. ChrisA From nobody at nowhere.com Mon Jul 22 16:32:59 2013 From: nobody at nowhere.com (Nobody) Date: Mon, 22 Jul 2013 21:32:59 +0100 Subject: Simple Python script as SMTP server for outgoing e-mails? References: Message-ID: On Mon, 22 Jul 2013 14:19:57 +0200, Gilles wrote: > Incidently, how do ISP MTAs find whether the remote MTA is legit or > running on some regular user's computer? Look up the IP address in a database. If they don't have a database, perform a reverse DNS lookup and reject anything which looks like a typical auto-generated name for a consumer DSL/cable connection. FWIW, I've been running sendmail on my home system (ADSL with static IP) for years, and have had very few problems with mail being rejected. From kw at codebykevin.com Mon Jul 22 10:14:15 2013 From: kw at codebykevin.com (Kevin Walzer) Date: Mon, 22 Jul 2013 10:14:15 -0400 Subject: Simple Python script as SMTP server for outgoing e-mails? In-Reply-To: References: Message-ID: On 7/21/13 10:42 AM, Gilles wrote: > Hello > > Every once in a while, my ISP's SMTP server refuses to send > perfectly legit e-mails because it considers them as SPAM. > > So I'd like to install a dead-simple SMTP server on my XP computer > just to act as SMTP backup server. > All I'd need is to change the SMTP address in my e-mail client, and > off they go. No need for anything else like user authentication or > SPAM control. > > Is there a no-brainer, ready-to-use solution in Python that I could > use for this? > > Thank you. > http://www.hmailserver.com -- Kevin Walzer Code by Kevin/Mobile Code by Kevin http://www.codebykevin.com http://www.wtmobilesoftware.com From nospam at nospam.com Tue Jul 23 17:53:32 2013 From: nospam at nospam.com (Gilles) Date: Tue, 23 Jul 2013 23:53:32 +0200 Subject: Simple Python script as SMTP server for outgoing e-mails? References: Message-ID: <2nutu8la1eogr08vf2jdnrkbka7o7q4iei@4ax.com> On Mon, 22 Jul 2013 10:14:15 -0400, Kevin Walzer wrote: >http://www.hmailserver.com Thanks. hMailServer was one of the apps I checked, and I was just making sure there weren't something simpler, considering my needs, ideally something like Mongoose MTA. Regardless, because of the SPAM anti-measures mentioned above, it seems like I was over-optimistic about running an MTA and sending e-mails from my home computer :-/ From kw at codebykevin.com Wed Jul 24 10:38:52 2013 From: kw at codebykevin.com (Kevin Walzer) Date: Wed, 24 Jul 2013 10:38:52 -0400 Subject: Simple Python script as SMTP server for outgoing e-mails? In-Reply-To: <2nutu8la1eogr08vf2jdnrkbka7o7q4iei@4ax.com> References: <2nutu8la1eogr08vf2jdnrkbka7o7q4iei@4ax.com> Message-ID: On 7/23/13 5:53 PM, Gilles wrote: > On Mon, 22 Jul 2013 10:14:15 -0400, Kevin Walzer > wrote: >> http://www.hmailserver.com > > Thanks. hMailServer was one of the apps I checked, and I was just > making sure there weren't something simpler, considering my needs, > ideally something like Mongoose MTA. > > Regardless, because of the SPAM anti-measures mentioned above, it > seems like I was over-optimistic about running an MTA and sending > e-mails from my home computer :-/ > The reason I mentioned hMailServer is that I host my own mail server for my business--I have a static IP address, and correctly configured DNS--and so I'm able to send out large batches of e-mails to customers from my network without being blocked by my ISP. I'm running a Mac server so my mail system is the typical Unix setup (Postfix, etc.), but hMailServer was the closest thing I've found for Windows. Configuring your own server isn't cheap in terms of time even if you use FOSS components, and your ISP may charge more for a static IP, so I completely understand if you don't want to go that route. --Kevin -- Kevin Walzer Code by Kevin/Mobile Code by Kevin http://www.codebykevin.com http://www.wtmobilesoftware.com From dmitrey15 at gmail.com Mon Jul 22 11:44:50 2013 From: dmitrey15 at gmail.com (dmitrey15 at gmail.com) Date: Mon, 22 Jul 2013 08:44:50 -0700 (PDT) Subject: could you change PYPI downloads number for not-uploaded packages? Message-ID: <1510b4f9-5b17-4fde-a19f-cc4f0d277e40@googlegroups.com> Hi all, could you change PYPI downloads number for not-uploaded packages from zeros to real posivive numbers? For example, my projects download links are binded to my website , and thus people see misleading zeros, e.g. https://pypi.python.org/pypi/openopt Downloads (All Versions): 0 downloads in the last day 0 downloads in the last week Or, even better, taking into account that some people install packages from subversion/git/etc repository, invoke "+1" when someone runs "python setup.py install" (or "develop") (provided internet connection is present) ---------------- Regards, D. http://openopt.org/Dmitrey From robert.kern at gmail.com Mon Jul 22 11:52:01 2013 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 22 Jul 2013 16:52:01 +0100 Subject: could you change PYPI downloads number for not-uploaded packages? In-Reply-To: <1510b4f9-5b17-4fde-a19f-cc4f0d277e40@googlegroups.com> References: <1510b4f9-5b17-4fde-a19f-cc4f0d277e40@googlegroups.com> Message-ID: On 2013-07-22 16:44, dmitrey15 at gmail.com wrote: > Hi all, > could you change PYPI downloads number for not-uploaded packages from zeros to real posivive numbers? For example, my projects download links are binded to my website , and thus people see misleading zeros, e.g. > https://pypi.python.org/pypi/openopt > Downloads (All Versions): > 0 downloads in the last day > 0 downloads in the last week > > Or, even better, taking into account that some people install packages from subversion/git/etc repository, invoke "+1" when someone runs "python setup.py install" (or "develop") (provided internet connection is present) The maintenance and development of PyPI is discussed on the Distutils-SIG. Please bring your concerns there. http://www.python.org/community/sigs/current/distutils-sig/ In short, if you want to have download counts, you will need to host your package downloads from PyPI itself. There is no good way for PyPI to count downloads from any other source. What you might want to ask for instead is to have the download count not shown when the packages are not hosted on PyPI. That would be a reasonable change that I think the PyPI team would accept. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ben+python at benfinney.id.au Mon Jul 22 19:08:27 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 23 Jul 2013 09:08:27 +1000 Subject: could you change PYPI downloads number for not-uploaded packages? References: <1510b4f9-5b17-4fde-a19f-cc4f0d277e40@googlegroups.com> Message-ID: <7wa9ledwgk.fsf@benfinney.id.au> Robert Kern writes: > On 2013-07-22 16:44, dmitrey15 at gmail.com wrote: > > For example, my projects download links are binded to my website , > > and thus people see misleading zeros [?] > In short, if you want to have download counts, you will need to host > your package downloads from PyPI itself. Also of interest to this group is that PyPI is transitioning to strongly encourage hosting files at PyPI, and installer tools will default to not installing files hosted elsewhere. See PEP 438 ?Transitioning to release-file hosting on PyPI? for the details. As Robart said, it's best not discussed here, but at the ?distutils-sig? forum . -- \ ?I do not believe in immortality of the individual, and I | `\ consider ethics to be an exclusively human concern with no | _o__) superhuman authority behind it.? ?Albert Einstein, letter, 1953 | Ben Finney From chris.hinsley at gmail.com Mon Jul 22 14:36:41 2013 From: chris.hinsley at gmail.com (Chris Hinsley) Date: Mon, 22 Jul 2013 19:36:41 +0100 Subject: odd behavoiur seen Message-ID: <2013072219364160391-chrishinsley@gmailcom> Folks, I have this decorator: def memoize(maxsize): def _memoize(func): lru_cache = {} lru_list = [] def memoizer(*args, **kwargs): key = str(args) + str(kwargs) if key in lru_cache: lru_list.remove(key) lru_list.append(key) return lru_cache[key] print len(lru_list), if len(lru_list) >= maxsize: del(lru_cache[lru_list[0]]) del(lru_list[0]) ret = func(*args, **kwargs) lru_cache[key] = ret lru_list.append(key) return ret return memoizer return _memoize I didn't used to do the 'len(lru_list) >= maxsize' just '==' and noticed it sailing past the max number of entries, so put in the print statement, and now I see it ocationally printing a value 1 larger than maxsize !!! So if I use it as '@memoize(64)' I see some 65's in the output ! I'm at a loss to explain it, does anyone knows why ? Is it a bug or some threading issue ? I'm not useing threads BTW, and I've noticed this in both running it with Python or Pypy. Best Regards Chris From chris.hinsley at gmail.com Mon Jul 22 14:42:47 2013 From: chris.hinsley at gmail.com (Chris Hinsley) Date: Mon, 22 Jul 2013 19:42:47 +0100 Subject: odd behavoiur seen References: <2013072219364160391-chrishinsley@gmailcom> Message-ID: <2013072219424735368-chrishinsley@gmailcom> On 2013-07-22 18:36:41 +0000, Chris Hinsley said: > Folks, I have this decorator: > > def memoize(maxsize): > def _memoize(func): > lru_cache = {} > lru_list = [] Other clues, I use it on a recursive function: @memoize(64) def next_move(board, colour, alpha, beta, ply): if ply <= 0: return evaluate(board) * colour for new_board in all_moves(board[:], colour): score = -next_move(new_board, -colour, -beta, -alpha, ply - 1) if score >= beta: return score if score > alpha: alpha = score return alpha And I notice I don't get the strange problem on a non-recursive function ! Or at least I don't seam to. Chris From __peter__ at web.de Mon Jul 22 15:47:33 2013 From: __peter__ at web.de (Peter Otten) Date: Mon, 22 Jul 2013 21:47:33 +0200 Subject: odd behavoiur seen References: <2013072219364160391-chrishinsley@gmailcom> <2013072219424735368-chrishinsley@gmailcom> Message-ID: Chris Hinsley wrote: > On 2013-07-22 18:36:41 +0000, Chris Hinsley said: > >> Folks, I have this decorator: >> >> def memoize(maxsize): >> def _memoize(func): >> lru_cache = {} >> lru_list = [] > > Other clues, I use it on a recursive function: > > @memoize(64) > def next_move(board, colour, alpha, beta, ply): > if ply <= 0: > return evaluate(board) * colour > for new_board in all_moves(board[:], colour): > score = -next_move(new_board, -colour, -beta, -alpha, ply - 1) > if score >= beta: > return score > if score > alpha: > alpha = score > return alpha > > And I notice I don't get the strange problem on a non-recursive > function ! Or at least I don't seam to. That's indeed the problem: > if len(lru_list) >= maxsize: > del(lru_cache[lru_list[0]]) > del(lru_list[0]) > ret = func(*args, **kwargs) > lru_cache[key] = ret > lru_list.append(key) You delete a cached item, then call the original function which causes calls of the decorated function. This causes a length check which sees the already reduced length and decides that the cache is not yet full. If you remove the oldest item after calling the original function you should be OK. From chris.hinsley at gmail.com Mon Jul 22 16:12:52 2013 From: chris.hinsley at gmail.com (Chris Hinsley) Date: Mon, 22 Jul 2013 21:12:52 +0100 Subject: odd behavoiur seen References: <2013072219364160391-chrishinsley@gmailcom> <2013072219424735368-chrishinsley@gmailcom> Message-ID: <2013072221125291649-chrishinsley@gmailcom> On 2013-07-22 19:47:33 +0000, Peter Otten said: > Chris Hinsley wrote: > >> On 2013-07-22 18:36:41 +0000, Chris Hinsley said: >> >>> Folks, I have this decorator: >>> >>> def memoize(maxsize): >>> def _memoize(func): >>> lru_cache = {} >>> lru_list = [] >> >> Other clues, I use it on a recursive function: >> >> @memoize(64) >> def next_move(board, colour, alpha, beta, ply): >> if ply <= 0: >> return evaluate(board) * colour >> for new_board in all_moves(board[:], colour): >> score = -next_move(new_board, -colour, -beta, -alpha, ply - 1) >> if score >= beta: >> return score >> if score > alpha: >> alpha = score >> return alpha >> >> And I notice I don't get the strange problem on a non-recursive >> function ! Or at least I don't seam to. > > That's indeed the problem: > >> if len(lru_list) >= maxsize: >> del(lru_cache[lru_list[0]]) >> del(lru_list[0]) >> ret = func(*args, **kwargs) >> lru_cache[key] = ret >> lru_list.append(key) > > You delete a cached item, then call the original function which causes calls > of the decorated function. This causes a length check which sees the already > reduced length and decides that the cache is not yet full. > > If you remove the oldest item after calling the original function you should > be OK. Ah ! Thank you kindly sir ! Chris From santosh.ssit at gmail.com Mon Jul 22 15:09:05 2013 From: santosh.ssit at gmail.com (san) Date: Mon, 22 Jul 2013 12:09:05 -0700 (PDT) Subject: How to read a make file in python and access its elements Message-ID: How to read/load the cmake file in python and access its elements. I have a scenario, where i need to load the make file and access its elements. I have tried reading the make file as text file and parsing it,but its not the ideal solution Please let me know how to load the .mk file and access its elements in python. From ben+python at benfinney.id.au Mon Jul 22 19:14:08 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 23 Jul 2013 09:14:08 +1000 Subject: How to read a make file in python and access its elements References: Message-ID: <7w61w2dw73.fsf@benfinney.id.au> san writes: > I have a scenario, where i need to load the make file and access its > elements. What do you mean by ?elements? of a make file? Is that a term with a specific meaning, or do you mean some particular parts of the make file? > I have tried reading the make file as text file and parsing it,but its > not the ideal solution You might be interested in using a library purpose-built for creating a parser . I'm not aware of any makefile-syntax-aware tool for Python. Your best option could be to write your own parser using the above library. -- \ ?True greatness is measured by how much freedom you give to | `\ others, not by how much you can coerce others to do what you | _o__) want.? ?Larry Wall | Ben Finney From davea at davea.name Mon Jul 22 20:39:33 2013 From: davea at davea.name (Dave Angel) Date: Mon, 22 Jul 2013 20:39:33 -0400 Subject: How to read a make file in python and access its elements In-Reply-To: References: Message-ID: On 07/22/2013 03:09 PM, san wrote: > How to read/load the cmake file in python and access its elements. > I have a scenario, where i need to load the make file and access its elements. > I have tried reading the make file as text file and parsing it,but its not the ideal solution > Please let me know how to load the .mk file and access its elements in python. > First, just what do you mean by "make file"? You refer to cmake, make, and .mk in three places in your message. How about a link to the actual tool you're interested in? And a brief list of the other tools you're using it with. Is this it? http://www.cmake.org/ Once we're talking about the same tool, then the question is what "elements" are you interested in? If your cmake is anything like the traditional make program, the format is deceptively simple, and enormously complex in usage. But I suspect that your cmake has very little to do with Unix make tools. The cmake that I gave the link for uses a CMakeLists file, which looks nothing like a makefile. -- DaveA From wuwei23 at gmail.com Mon Jul 22 20:48:30 2013 From: wuwei23 at gmail.com (alex23) Date: Tue, 23 Jul 2013 10:48:30 +1000 Subject: How to read a make file in python and access its elements In-Reply-To: References: Message-ID: On 23/07/2013 5:09 AM, san wrote: > How to read/load the cmake file in python and access its elements. > I have a scenario, where i need to load the make file and access its elements. > I have tried reading the make file as text file and parsing it,but its not the ideal solution > Please let me know how to load the .mk file and access its elements in python. Take a look at pymake: "make.py (and the pymake modules that support it) are an implementation of the make tool which are mostly compatible with makefiles written for GNU make." http://hg.mozilla.org/users/bsmedberg_mozilla.com/pymake/ There is a parser.py file which might be useful to you. From malayrev at gmail.com Tue Jul 23 00:10:18 2013 From: malayrev at gmail.com (malayrev at gmail.com) Date: Mon, 22 Jul 2013 21:10:18 -0700 (PDT) Subject: How to tick checkboxes with the same name? Message-ID: <8fad5326-8518-4554-91bf-1c0b5c97d9bc@googlegroups.com> I faced a problem: to implement appropriate search program I need to tick few checkboxes which turned out to have the same name (name="a", id="a1","a2","a3","a4"). Set_input('a', True) does not work (I use Grab library), this command leads to the error "checkboxgroup must be set to a sequence". I don't understand what the sequence actually is, so I'm stuck with how to tick the checkboxes. It would be really great if someone would help me with how to handle this question. The code is available here: view-source:http://zakupki.gov.ru/pgz/public/action/contracts/search/ext/enter From joel.goldstick at gmail.com Tue Jul 23 01:05:14 2013 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Tue, 23 Jul 2013 01:05:14 -0400 Subject: How to tick checkboxes with the same name? In-Reply-To: <8fad5326-8518-4554-91bf-1c0b5c97d9bc@googlegroups.com> References: <8fad5326-8518-4554-91bf-1c0b5c97d9bc@googlegroups.com> Message-ID: On Tue, Jul 23, 2013 at 12:10 AM, wrote: > I faced a problem: to implement appropriate search program I need to tick few checkboxes which turned out to have the same name (name="a", id="a1","a2","a3","a4"). Set_input('a', True) does not work (I use Grab library), this command leads to the error "checkboxgroup must be set to a sequence". I don't understand what the sequence actually is, so I'm stuck with how to tick the checkboxes. It would be really great if someone would help me with how to handle this question. The code is available here: > view-source:http://zakupki.gov.ru/pgz/public/action/contracts/search/ext/enter > -- > http://mail.python.org/mailman/listinfo/python-list Have you tried a[0], a[1], etc. for the names? -- Joel Goldstick http://joelgoldstick.com From steve+comp.lang.python at pearwood.info Tue Jul 23 01:14:29 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 23 Jul 2013 05:14:29 GMT Subject: How to tick checkboxes with the same name? References: <8fad5326-8518-4554-91bf-1c0b5c97d9bc@googlegroups.com> Message-ID: <51ee1135$0$29971$c3e8da3$5496439d@news.astraweb.com> On Mon, 22 Jul 2013 21:10:18 -0700, malayrev wrote: > I faced a problem: to implement appropriate search program I need to > tick few checkboxes which turned out to have the same name (name="a", > id="a1","a2","a3","a4"). Set_input('a', True) does not work (I use Grab > library) Instructions for solving your problem can be found here: http://sscce.org/? Although it is written for Java, the same ideas will work for Python or any other programming language. Have a read of that page, and if it isn't enough to solve your problem with "Grab" (whatever that is), feel free to come back. -- Steven From __peter__ at web.de Tue Jul 23 03:25:00 2013 From: __peter__ at web.de (Peter Otten) Date: Tue, 23 Jul 2013 09:25 +0200 Subject: How to tick checkboxes with the same name? References: <8fad5326-8518-4554-91bf-1c0b5c97d9bc@googlegroups.com> Message-ID: malayrev at gmail.com wrote: > I faced a problem: to implement appropriate search program I need to tick > few checkboxes which turned out to have the same name (name="a", > id="a1","a2","a3","a4"). Set_input('a', True) does not work (I use Grab > library), For all but the most popular projects a url works wonders. I'm assuming http://grablib.org > this command leads to the error "checkboxgroup must be set to a > sequence". I don't understand what the sequence actually is, so I'm stuck > with how to tick the checkboxes. If I were to guess: set_input("a", [True, True, True, True]) but I don't see that form documented on page http://docs.grablib.org/api/ext_form.html If it doesn't work try set_input_by_id(_"a1", True) set_input_by_id(_"a2", True) and so on. From malayrev at gmail.com Thu Jul 25 11:42:19 2013 From: malayrev at gmail.com (malayrev at gmail.com) Date: Thu, 25 Jul 2013 08:42:19 -0700 (PDT) Subject: How to tick checkboxes with the same name? In-Reply-To: References: <8fad5326-8518-4554-91bf-1c0b5c97d9bc@googlegroups.com> Message-ID: ???????, 23 ???? 2013??., 9:05:14 UTC+4 ???????????? Joel Goldstick ???????: > On Tue, Jul 23, 2013 at 12:10 AM, wrote: > > > I faced a problem: to implement appropriate search program I need to tick few checkboxes which turned out to have the same name (name="a", id="a1","a2","a3","a4"). Set_input('a', True) does not work (I use Grab library), this command leads to the error "checkboxgroup must be set to a sequence". I don't understand what the sequence actually is, so I'm stuck with how to tick the checkboxes. It would be really great if someone would help me with how to handle this question. The code is available here: > > > view-source:http://zakupki.gov.ru/pgz/public/action/contracts/search/ext/enter > > > -- > > > http://mail.python.org/mailman/listinfo/python-list > > > > Have you tried a[0], a[1], etc. for the names? > > > > -- > > Joel Goldstick > > http://joelgoldstick.com sure, I tried, doesn't work From malayrev at gmail.com Thu Jul 25 11:46:05 2013 From: malayrev at gmail.com (malayrev at gmail.com) Date: Thu, 25 Jul 2013 08:46:05 -0700 (PDT) Subject: How to tick checkboxes with the same name? In-Reply-To: References: <8fad5326-8518-4554-91bf-1c0b5c97d9bc@googlegroups.com> Message-ID: <85c5f224-3ccc-421b-bb5d-d54c3defbd27@googlegroups.com> ???????, 23 ???? 2013??., 11:25:00 UTC+4 ???????????? Peter Otten ???????: > malayrev at gmail.com wrote: > > > > > I faced a problem: to implement appropriate search program I need to tick > > > few checkboxes which turned out to have the same name (name="a", > > > id="a1","a2","a3","a4"). Set_input('a', True) does not work (I use Grab > > > library), > > > > For all but the most popular projects a url works wonders. I'm assuming > > > > http://grablib.org > > > > > this command leads to the error "checkboxgroup must be set to a > > > sequence". I don't understand what the sequence actually is, so I'm stuck > > > with how to tick the checkboxes. > > > > If I were to guess: > > > > set_input("a", [True, True, True, True]) > > > > but I don't see that form documented on page > > > > http://docs.grablib.org/api/ext_form.html > > > > If it doesn't work try > > > > set_input_by_id(_"a1", True) > > set_input_by_id(_"a2", True) > > > > and so on. Well, I have read the documentation, I guess the problem lies in somewhat different field. As long as I understand it refers to Checkboxgroup classes. As checkboxes of interest belong to some common group with the common name "a", they do have values different from "True" or "False" (probably a sequence?) and I should guess somehow what are the real ones. How to do this? From __peter__ at web.de Fri Jul 26 03:05:15 2013 From: __peter__ at web.de (Peter Otten) Date: Fri, 26 Jul 2013 09:05:15 +0200 Subject: How to tick checkboxes with the same name? References: <8fad5326-8518-4554-91bf-1c0b5c97d9bc@googlegroups.com> <85c5f224-3ccc-421b-bb5d-d54c3defbd27@googlegroups.com> Message-ID: malayrev at gmail.com wrote: > ???????, 23 ???? 2013 ?., 11:25:00 UTC+4 ???????????? Peter Otten ???????: >> malayrev at gmail.com wrote: >> For all but the most popular projects a url works wonders. I'm assuming >> http://grablib.org > Well, I have read the documentation, I guess the problem lies in somewhat > different field. As long as I understand it refers to Checkboxgroup > classes. As checkboxes of interest belong to some common group with the > common name "a", they do have values different from "True" or "False" > (probably a sequence?) and I should guess somehow what are the real ones. > How to do this? I took a quick look into the source, and it seems that grab has no notion of multiple inputs with the same name. My suggested workaround >> set_input_by_id(_"a1", True) doesn't work because (among other things) the id is immediately translated into the name. I fear you have to construct the request manually or choose another library -- maybe mechanize (Python 2 only)? From hsiwrek at walla.com Tue Jul 23 02:52:42 2013 From: hsiwrek at walla.com (hsiwrek at walla.com) Date: Mon, 22 Jul 2013 23:52:42 -0700 (PDT) Subject: tkinter progress bar Message-ID: Hi, How can I add a tkinter progress bar in python 3.2 to start before a loop and end after it. I am looking for a very simple solution. def MyFunc(): Start progress bar for fileName in fileList: ? End progress bar Thanks a lot in advance. From auriocus at gmx.de Tue Jul 23 04:43:02 2013 From: auriocus at gmx.de (Christian Gollwitzer) Date: Tue, 23 Jul 2013 10:43:02 +0200 Subject: tkinter progress bar In-Reply-To: References: Message-ID: Am 23.07.13 08:52, schrieb hsiwrek at walla.com: > Hi, > > How can I add a tkinter progress bar in python 3.2 to start before a loop and end after it. I am looking for a very simple solution. > > def MyFunc(): > Start progress bar > > for fileName in fileList: > ? > > End progress bar > 1. There is a progress bar widget in ttk. At the beginning, you set maximum to the number of files in your list 2. In the loop, you set "value" of the progressbar to the current file number. You can also attach a variable 3. The bar is only redrawn when you process events. The simplest way to do this is by calling update() on the progress bar, which processes all pending events. Despite of it looking like a method, update() is really a global function within Tcl and updates all widgets in your interface. You must make sure, therefore, that the user does not trigger another event which interferes with your download, such as pressing the button for starting it again. The easiest way is to disable the button at the begin and reenable it at the end. 4. If processing of a single file takes a long time, the only way to have the GUI responsive is to put the work in a background thread. That seems to be more involved. Christian From hsiwrek at walla.com Tue Jul 23 05:38:34 2013 From: hsiwrek at walla.com (hsiwrek at walla.com) Date: Tue, 23 Jul 2013 02:38:34 -0700 (PDT) Subject: tkinter progress bar In-Reply-To: References: Message-ID: Dear Christian, Thanks for the help. Can you please add a source example as I am new with Tkinter. Cheers. From jason.swails at gmail.com Tue Jul 23 09:27:59 2013 From: jason.swails at gmail.com (Jason Swails) Date: Tue, 23 Jul 2013 09:27:59 -0400 Subject: tkinter progress bar In-Reply-To: References: Message-ID: On Tue, Jul 23, 2013 at 5:38 AM, wrote: > Dear Christian, > > Thanks for the help. Can you please add a source example as I am new with > Tkinter. > http://docs.python.org/2/library/ttk.html#progressbar You can do something like this: #!/usr/bin/env python import Tkinter as tk import ttk import time class MainApp(tk.Frame): def __init__(self, master): tk.Frame.__init__(self, master) self.progress = ttk.Progressbar(self, maximum=10) self.progress.pack(expand=1, fill=tk.BOTH) self.progress.bind("", self._loop_progress) def _loop_progress(self, *args): for i in range(10): self.progress.step(1) # Necessary to update the progress bar appearance self.update() # Busy-wait time.sleep(2) if __name__ == '__main__': root = tk.Tk() app = MainApp(root) app.pack(expand=1, fill=tk.BOTH) root.mainloop() This is a simple stand-alone app that (just) demonstrates how to use the ttk.Progressbar widget. HTH, Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: From enmce at yandex.ru Tue Jul 23 08:34:25 2013 From: enmce at yandex.ru (enmce at yandex.ru) Date: Tue, 23 Jul 2013 05:34:25 -0700 (PDT) Subject: Beginner. 2d rotation gives unexpected results. Message-ID: <0a905ff1-199c-4900-81e6-d9b7bb63bb44@googlegroups.com> Hello! This is my first post, nice to meet you all! I`m biology student from Russia, trying to learn python to perform some simple simulations. Here`s my first problem. I`m trying to perform some simple 2d vector rotations in pygame, in order to learn the basics of linear algebra and 2d transformations. So far i understand matrix multiplication pretty well, and probably all my math is right. Eventually i`m planning to write Poly class, and use it to rotate and translate some simple shapes. But when i try and write it in the program, i get very weird results, like all points of rectangle with coordinates [0,0],[0,100],[100,0],[100,100] start to go spiral and eventually shrink to the center. Although even Excel calculations with this formulas give me right result. I use Python 3.3 on Windows Xp. What is wrong with my code? [code]import pygame import math as m black = ( 0, 0, 0) white = ( 255, 255, 255) green = ( 0, 255, 0) red = ( 255, 0, 0) class Poly(): pos = [100,100] #x and y coordinates of a point rot = m.radians(1) #rotation in degrees def draw(self): #draw point pygame.draw.circle(screen,white,self.pos,10,0) def rotate(self): # rotation method sin = m.sin(self.rot) #calculationg sin and cos cos = m.cos(self.rot) x_rot = int(self.pos[0]*cos-self.pos[1]*sin) #mulpitplicating vector to rotation matrix y_rot = int(self.pos[0]*sin+self.pos[1]*cos) self.pos[0] = x_rot #set new coordinates to a point self.pos[1] = y_rot a = Poly() #Some simple sample points giving rectangle b = Poly() c = Poly() d = Poly() b.pos = [0,100] c.pos = [100,0] d.pos = [0,0] pygame.init() size = [700,500] screen = pygame.display.set_mode(size) done = False clock = pygame.time.Clock() while done == False: for event in pygame.event.get(): if event.type == pygame.QUIT: done = True a.rotate() #perform rotation b.rotate() c.rotate() d.rotate() screen.fill(black) a.draw() #draw point b.draw() c.draw() d.draw() pygame.display.flip() clock.tick(30) pygame.quit()[/code] P.S. Sorry for my english, bit rusty in that department. From dwightdhutto at gmail.com Tue Jul 23 09:04:27 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Tue, 23 Jul 2013 09:04:27 -0400 Subject: Beginner. 2d rotation gives unexpected results. In-Reply-To: <0a905ff1-199c-4900-81e6-d9b7bb63bb44@googlegroups.com> References: <0a905ff1-199c-4900-81e6-d9b7bb63bb44@googlegroups.com> Message-ID: haven't used pygame that much, but it sounds like you drew Z. You have [0,0],[0,100],[100,0],[100, 100] 0,0 is the top left, if I recall 0, 100 would be the lower left, then you move to100, 0 which would go diagonal to the top right, and then 100,100 to the lower right, this is assuming 0,0 is the upper left. for a square you would go,[0,0],[0,100],[100,100],[100,0]then back to [0,0] to complete the square. This is assuming that 0,0 is the upper left, the coords are x,y in the brackets, and the increase in x takes you the right, and the increase in y takes you down. If that doesn't work,I'll download it later, and try it out. On Tue, Jul 23, 2013 at 8:34 AM, wrote: > Hello! > This is my first post, nice to meet you all! > I`m biology student from Russia, trying to learn python to perform some > > simple simulations. > > Here`s my first problem. > I`m trying to perform some simple 2d vector rotations in pygame, in order > > to learn the basics of linear algebra and 2d transformations. So far i > > understand matrix multiplication pretty well, and probably all my math is > > right. Eventually i`m planning to write Poly class, and use it to rotate > > and translate some simple shapes. But when i try and write it in the > > program, i get very weird results, like all points of rectangle with > > coordinates [0,0],[0,100],[100,0],[100,100] start to go spiral and > > eventually shrink to the center. Although even Excel calculations with > > this formulas give me right result. > I use Python 3.3 on Windows Xp. > What is wrong with my code? > > [code]import pygame > import math as m > > black = ( 0, 0, 0) > white = ( 255, 255, 255) > green = ( 0, 255, 0) > red = ( 255, 0, 0) > > class Poly(): > pos = [100,100] #x and y coordinates of a point > rot = m.radians(1) #rotation in degrees > def draw(self): #draw point > pygame.draw.circle(screen,white,self.pos,10,0) > def rotate(self): # rotation method > sin = m.sin(self.rot) #calculationg sin and cos > cos = m.cos(self.rot) > x_rot = int(self.pos[0]*cos-self.pos[1]*sin) #mulpitplicating > > vector to rotation matrix > y_rot = int(self.pos[0]*sin+self.pos[1]*cos) > > self.pos[0] = x_rot #set new coordinates to a point > self.pos[1] = y_rot > > a = Poly() #Some simple sample points giving rectangle > b = Poly() > c = Poly() > d = Poly() > > b.pos = [0,100] > c.pos = [100,0] > d.pos = [0,0] > > pygame.init() > size = [700,500] > screen = pygame.display.set_mode(size) > done = False > clock = pygame.time.Clock() > while done == False: > for event in pygame.event.get(): > if event.type == pygame.QUIT: > done = True > > a.rotate() #perform rotation > b.rotate() > c.rotate() > d.rotate() > > screen.fill(black) > > a.draw() #draw point > b.draw() > c.draw() > d.draw() > pygame.display.flip() > clock.tick(30) > > pygame.quit()[/code] > > P.S. Sorry for my english, bit rusty in that department. > -- > http://mail.python.org/mailman/listinfo/python-list > -- Best Regards, David Hutto *CEO:* *http://www.hitwebdevelopment.com* -------------- next part -------------- An HTML attachment was scrubbed... URL: From __peter__ at web.de Tue Jul 23 09:11:43 2013 From: __peter__ at web.de (Peter Otten) Date: Tue, 23 Jul 2013 15:11:43 +0200 Subject: Beginner. 2d rotation gives unexpected results. References: <0a905ff1-199c-4900-81e6-d9b7bb63bb44@googlegroups.com> Message-ID: enmce at yandex.ru wrote: > This is my first post, nice to meet you all! Welcome! > I`m biology student from Russia, trying to learn python to perform some > > simple simulations. > > Here`s my first problem. > I`m trying to perform some simple 2d vector rotations in pygame, in order > > to learn the basics of linear algebra and 2d transformations. So far i > > understand matrix multiplication pretty well, and probably all my math is > > right. Eventually i`m planning to write Poly class, and use it to rotate > > and translate some simple shapes. But when i try and write it in the > > program, i get very weird results, like all points of rectangle with > > coordinates [0,0],[0,100],[100,0],[100,100] start to go spiral and > > eventually shrink to the center. Although even Excel calculations with > > this formulas give me right result. > I use Python 3.3 on Windows Xp. > What is wrong with my code? > def rotate(self): # rotation method > sin = m.sin(self.rot) #calculationg sin and cos > cos = m.cos(self.rot) > x_rot = int(self.pos[0]*cos-self.pos[1]*sin) #mulpitplicating The conversion to int introduces a rounding error that accumulates over time. > vector to rotation matrix > y_rot = int(self.pos[0]*sin+self.pos[1]*cos) > > self.pos[0] = x_rot #set new coordinates to a point > self.pos[1] = y_rot One way to keep the error low is to keep the float values in self.pos and do the rounding on the fly when you display the point: class Poly(): def __init__(self, color, pos, rot=m.radians(1)): self.color = color self.pos = pos self.rot = rot def draw(self): x, y = self.pos pygame.draw.circle(screen, self.color, [350+int(x), 250+int(y)], 10, 0) def rotate(self): sin = m.sin(self.rot) cos = m.cos(self.rot) x_rot = self.pos[0]*cos-self.pos[1]*sin y_rot = self.pos[0]*sin+self.pos[1]*cos self.pos = [x_rot, y_rot] a = Poly(white, [100, 100]) b = Poly(green, [0, 100]) c = Poly(blue, [100, 0]) d = Poly(red, [0, 0]) From nobody at nowhere.com Tue Jul 23 11:20:35 2013 From: nobody at nowhere.com (Nobody) Date: Tue, 23 Jul 2013 16:20:35 +0100 Subject: Beginner. 2d rotation gives unexpected results. References: <0a905ff1-199c-4900-81e6-d9b7bb63bb44@googlegroups.com> Message-ID: On Tue, 23 Jul 2013 15:11:43 +0200, Peter Otten wrote: > The conversion to int introduces a rounding error that accumulates over > time. Most floating point calculations introduce a rounding error. If the calculations are iterated, the error will accumulate. In general, you want to avoid accumulating entire transformations. E.g. if you want a spinning object, maintain the cumulative rotation angle and rotate the original points each frame. If you must accumulate transformations, you need to actively work to maintain any desired invariants. E.g. if a transformation is supposed to be orthonormal (all axes perpendicular and of unit length), you should renormalise it periodically, otherwise the lengths and angles will change over time. From joshua at landau.ws Wed Jul 24 17:17:22 2013 From: joshua at landau.ws (Joshua Landau) Date: Wed, 24 Jul 2013 22:17:22 +0100 Subject: Beginner. 2d rotation gives unexpected results. In-Reply-To: <0a905ff1-199c-4900-81e6-d9b7bb63bb44@googlegroups.com> References: <0a905ff1-199c-4900-81e6-d9b7bb63bb44@googlegroups.com> Message-ID: On 23 July 2013 13:34, wrote: > Hello! > This is my first post, nice to meet you all! > I`m biology student from Russia, trying to learn python to perform some > > simple simulations. > > Here`s my first problem. > I`m trying to perform some simple 2d vector rotations in pygame, in order > > to learn the basics of linear algebra and 2d transformations. So far i > > understand matrix multiplication pretty well, and probably all my math is > > right. Eventually i`m planning to write Poly class, and use it to rotate > > and translate some simple shapes. But when i try and write it in the > > program, i get very weird results, like all points of rectangle with > > coordinates [0,0],[0,100],[100,0],[100,100] start to go spiral and > > eventually shrink to the center. Although even Excel calculations with > > this formulas give me right result. > I use Python 3.3 on Windows Xp. > What is wrong with my code? > > [code]import pygame > import math as m > GAH! Why on earth would you do such a thing? Just "import math", there's no need to obfuscate your code. > black = ( 0, 0, 0) > white = ( 255, 255, 255) > green = ( 0, 255, 0) > red = ( 255, 0, 0) > It's probably better to do: black = pygame.Color("Black") white = pygame.Color("white") green = pygame.Color("green") red = pygame.Color("red") > class Poly(): > pos = [100,100] #x and y coordinates of a point rot = m.radians(1) #rotation in degrees > *Do not do this* This is because classes have shared values -- these "pos" and "rot" values are shared within the class. >>> class P: ... n = [] ... def more_n(self): ... self.n.append(len(self.n)) ... ... ... >>> one_P = P() >>> two_P = P() >>> >>> one_P.more_n() >>> one_P.more_n() >>> one_P.more_n() >>> >>> two_P.n [0, 1, 2] Normally you want to set these at initialisation: class Poly(): def __init__(self, pos=None, rot=math.radians(1)): self.pos = [100, 100] if pos is None else pos self.rot = rot super().__init__(self) > def draw(self): #draw point > pygame.draw.circle(screen,white,self.pos,10,0) > Add some spaces, dude. I was going to say: > Also, use keyword arguments instead of throwing around "10" and "0" with no context: > def draw(self): > pygame.draw.circle(screen, white, self.pos, radius=10, width=0) > Pygame-ists will know that "width" means border_width, by now. Pygame isn't known for it's clean design ;). But pygame, being brilliant (not) decided that it won't let you. def rotate(self): # rotation method > sin = m.sin(self.rot) #calculationg sin and cos > cos = m.cos(self.rot) > x_rot = int(self.pos[0]*cos-self.pos[1]*sin) #mulpitplicating vector to rotation matrix > y_rot = int(self.pos[0]*sin+self.pos[1]*cos) > > self.pos[0] = x_rot #set new coordinates to a point > self.pos[1] = y_rot > A lot of your comments are ridiculous. This one is particularly so: #mulpitplicating vector to rotation matrix. Don't add comments that talk about lines. Here is a quick guide for when to use comments: 1) API usage, when docstrings aren't usable 2) When something funny or unexpected occurs in the code, such as: # Goes down to 0 (does not include end-point) for i in range(foo, -1, -1): ... 3) To explain large-scale methodologies and algorithms Other than this, are you trying to obfuscate this line? HINT: ADD SPACES ;). A big problem here (this solves your problem) is your int(...) conversions. Do *not* store important information less accurately than it deserves. This is one very important instance. Only convert to int when you need to. Another thing: "sin = m.sin(self.rot)"??? Really? Write "sin_1deg = ..." instead, at least. > a = Poly() #Some simple sample points giving rectangle > b = Poly() > c = Poly() > d = Poly() > > b.pos = [0,100] > c.pos = [100,0] > d.pos = [0,0] > Use: a = Poly() b = Poly([0, 100]) c = Poly([100, 0]) d = Poly([0, 0]) pygame.init() > size = [700,500] > screen = pygame.display.set_mode(size) > done = False > clock = pygame.time.Clock() > while done == False: > "while not done" Also, just use "while True" and a "break". I know some C-people or what not think this is "evil" or something, but they're wrong. Personally, I actually like using: while "rotating the squares": ... instead of "while True". It's no slower and it's free documentation. > for event in pygame.event.get(): > if event.type == pygame.QUIT: > done = True > > a.rotate() #perform rotation > b.rotate() > c.rotate() > d.rotate() > > screen.fill(black) > > a.draw() #draw point > b.draw() > c.draw() > d.draw() > pygame.display.flip() > clock.tick(30) > > pygame.quit()[/code] > You don't need pygame.quit(). You *only* want pygame.quit() if you're quitting Pygame *before* the program is finished. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Jul 24 19:55:39 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 24 Jul 2013 19:55:39 -0400 Subject: Beginner. 2d rotation gives unexpected results. In-Reply-To: References: <0a905ff1-199c-4900-81e6-d9b7bb63bb44@googlegroups.com> Message-ID: On 7/24/2013 5:17 PM, Joshua Landau wrote: > import math as m > > > GAH! > > Why on earth would you do such a thing? for the same reason people do 'import tkinter as tk': to minimize typing and maximize clarity. In this case, from math import sin, cos, radians also works well -- Terry Jan Reedy From steve at divillo.com Tue Jul 23 17:52:21 2013 From: steve at divillo.com (steve at divillo.com) Date: Tue, 23 Jul 2013 14:52:21 -0700 (PDT) Subject: Converting a list of lists to a single list Message-ID: I think that itertools may be able to do what I want but I have not been able to figure out how. I want to convert an arbitrary number of lists with an arbitrary number of elements in each list into a single list as follows. Say I have three lists: [[A0,A1,A2], [B0,B1,B2] [C0,C1,C2]] I would like to convert those to a single list that looks like this: [A0,B0,C0,C1,C2,B1,C0,C1,C2,B2,C0,C1,C2,A1,B0,C0,C1,C2,B1,C0,C1,C2,B2,C0,C1,C2,A2,B0,C0,C1,C2,B1,C0,C1,C2,B2,C0,C1,C2] An easier way to visualize the pattern I want is as a tree. A0 B0 C0 C1 C2 B1 C0 C1 C2 B2 C0 C1 C2 A1 B0 C0 C1 C2 B1 C0 C1 C2 B2 C0 C1 C2 A2 B0 C0 C1 C2 B1 C0 C1 C2 B2 C0 C1 C2 From rafadurancastaneda at gmail.com Tue Jul 23 18:34:31 2013 From: rafadurancastaneda at gmail.com (=?ISO-8859-1?Q?Rafael_Dur=E1n_Casta=F1eda?=) Date: Wed, 24 Jul 2013 00:34:31 +0200 Subject: Converting a list of lists to a single list In-Reply-To: References: Message-ID: <51EF04F7.9040401@gmail.com> El 23/07/13 23:52, steve at divillo.com escribi?: > [[A0,A1,A2], [B0,B1,B2] [C0,C1,C2]] Hi, I think you are looking for itertools.chain, or in this case, itertools.chain.from_iterable: In [1]: x = [['A0','A1','A2'], ['B0','B1','B2'], ['C0','C1','C2']] In [2]: import itertools In [3]: [ y for y in itertools.chain.from_iterable(x)] Out[3]: ['A0', 'A1', 'A2', 'B0', 'B1', 'B2', 'C0', 'C1', 'C2'] HTH From rosuav at gmail.com Wed Jul 24 01:40:36 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 24 Jul 2013 15:40:36 +1000 Subject: Converting a list of lists to a single list In-Reply-To: <51EF04F7.9040401@gmail.com> References: <51EF04F7.9040401@gmail.com> Message-ID: On Wed, Jul 24, 2013 at 8:34 AM, Rafael Dur?n Casta?eda wrote: > In [3]: [ y for y in itertools.chain.from_iterable(x)] > Out[3]: ['A0', 'A1', 'A2', 'B0', 'B1', 'B2', 'C0', 'C1', 'C2'] Complete aside, given that this has already been pointed out as solving a different problem: Any time you see a list comp that just does "[ x for x in foo ]", check to see if it can be replaced by a simple list constructor: >>> [ y for y in itertools.chain.from_iterable(x)] ['A0', 'A1', 'A2', 'B0', 'B1', 'B2', 'C0', 'C1', 'C2'] >>> list(itertools.chain.from_iterable(x)) ['A0', 'A1', 'A2', 'B0', 'B1', 'B2', 'C0', 'C1', 'C2'] A bit simpler and achieves the same. ChrisA From python at mrabarnett.plus.com Tue Jul 23 18:52:21 2013 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 23 Jul 2013 23:52:21 +0100 Subject: Converting a list of lists to a single list In-Reply-To: References: Message-ID: <51EF0925.9020406@mrabarnett.plus.com> On 23/07/2013 22:52, steve at divillo.com wrote: > I think that itertools may be able to do what I want but I have not been able to figure out how. > > I want to convert an arbitrary number of lists with an arbitrary number of elements in each list into a single list as follows. > > Say I have three lists: > > [[A0,A1,A2], [B0,B1,B2] [C0,C1,C2]] > > I would like to convert those to a single list that looks like this: > > [A0,B0,C0,C1,C2,B1,C0,C1,C2,B2,C0,C1,C2,A1,B0,C0,C1,C2,B1,C0,C1,C2,B2,C0,C1,C2,A2,B0,C0,C1,C2,B1,C0,C1,C2,B2,C0,C1,C2] > > An easier way to visualize the pattern I want is as a tree. > > A0 > B0 > C0 > C1 > C2 > B1 > C0 > C1 > C2 > B2 > C0 > C1 > C2 > A1 > B0 > C0 > C1 > C2 > B1 > C0 > C1 > C2 > B2 > C0 > C1 > C2 > A2 > B0 > C0 > C1 > C2 > B1 > C0 > C1 > C2 > B2 > C0 > C1 > C2 > Using recursion: def tree_list(items): if len(items) == 1: return items[0] sublist = tree_list(items[1 : ]) result = [] for item in items[0]: result.append(item) result.extend(sublist) return result items = [["A0","A1","A2"], ["B0","B1","B2"], ["C0","C1","C2"]] print(tree_list(items)) From schesis at gmail.com Tue Jul 23 18:49:19 2013 From: schesis at gmail.com (Zero Piraeus) Date: Tue, 23 Jul 2013 18:49:19 -0400 Subject: Converting a list of lists to a single list In-Reply-To: References: Message-ID: : On 23 July 2013 17:52, wrote: > > Say I have three lists: > > [[A0,A1,A2], [B0,B1,B2] [C0,C1,C2]] > > I would like to convert those to a single list that looks like this: > [A0,B0,C0,C1,C2,B1,C0,C1,C2,B2,C0,C1,C2,A1,B0,C0,C1,C2,B1,C0,C1,C2,B2,C0,C1,C2,A2,B0,C0,C1,C2,B1,C0,C1,C2,B2,C0,C1,C2] How's this: from itertools import chain def treeify(seq): if seq: return list(chain(*([x] + treeify(seq[1:]) for x in seq[0]))) else: return [] >>> treeify([['A0', 'A1', 'A2'], ['B0', 'B1', 'B2'], ['C0', 'C1', 'C2']]) ['A0', 'B0', 'C0', 'C1', 'C2', 'B1', 'C0', 'C1', 'C2', 'B2', 'C0', 'C1', 'C2', 'A1', 'B0', 'C0', 'C1', 'C2', 'B1', 'C0', 'C1', 'C2', 'B2', 'C0', 'C1', 'C2', 'A2', 'B0', 'C0', 'C1', 'C2', 'B1', 'C0', 'C1', 'C2', 'B2', 'C0', 'C1', 'C2'] -[]z. From tjreedy at udel.edu Tue Jul 23 19:02:44 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 23 Jul 2013 19:02:44 -0400 Subject: Converting a list of lists to a single list In-Reply-To: References: Message-ID: On 7/23/2013 5:52 PM, steve at divillo.com wrote: > I think that itertools may be able to do what I want but I have not > been able to figure out how. A recursive generator suffices. > I want to convert an arbitrary number of lists with an arbitrary > number of elements in each list into a single list as follows. > > Say I have three lists: > > [[A0,A1,A2], [B0,B1,B2] [C0,C1,C2]] > > I would like to convert those to a single list that looks like this: > > [A0,B0,C0,C1,C2,B1,C0,C1,C2,B2,C0,C1,C2, > A1,B0,C0,C1,C2,B1,C0,C1,C2,B2,C0,C1,C2, > A2,B0,C0,C1,C2,B1,C0,C1,C2,B2,C0,C1,C2] def crossflat(lofl): if lofl: first = lofl.pop(0) for o in first: yield o yield from crossflat(lofl.copy()) A0, A1, A2 = 100, 101, 102 B0, B1, B2 = 10, 11, 12 C0, C1, C2 = 0, 1, 2 LL = [[A0, A1, A2], [B0, B1, B2], [C0, C1, C2]] cfLL = list(crossflat(LL)) print(cfLL) assert cfLL == [ A0, B0, C0, C1, C2, B1, C0, C1, C2, B2, C0, C1, C2, A1, B0, C0, C1, C2, B1, C0, C1, C2, B2, C0, C1, C2, A2, B0, C0, C1, C2, B1, C0, C1, C2, B2, C0, C1, C2] passes -- Terry Jan Reedy From tjreedy at udel.edu Wed Jul 24 11:56:30 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 24 Jul 2013 11:56:30 -0400 Subject: Converting a list of lists to a single list In-Reply-To: References: Message-ID: On 7/23/2013 7:02 PM, Terry Reedy wrote: > On 7/23/2013 5:52 PM, steve at divillo.com wrote: >> I think that itertools may be able to do what I want but I have not >> been able to figure out how. What you want is a flattened product with unchanged components of the successive products omitted in the flattening. The omission is the difficulty. > A recursive generator suffices. But see below for how to use itertools.product. >> I want to convert an arbitrary number of lists with an arbitrary >> number of elements in each list into a single list as follows. While others answered the Python2-oriented question ("How do I produce a list from a list of lists"), I answered the Python-3 oriented question of how to produce an iterator from an iterable of iterables. This scales better to an input with lots of long sequences. There is usually no need to manifest the output as a list, as the typical use of the list will be to iterate it. > def crossflat(lofl): > if lofl: > first = lofl.pop(0) > for o in first: > yield o > yield from crossflat(lofl.copy()) > > A0, A1, A2 = 100, 101, 102 > B0, B1, B2 = 10, 11, 12 > C0, C1, C2 = 0, 1, 2 > LL = [[A0, A1, A2], [B0, B1, B2], [C0, C1, C2]] > cfLL = list(crossflat(LL)) > print(cfLL) > assert cfLL == [ > A0, B0, C0, C1, C2, B1, C0, C1, C2, B2, C0, C1, C2, > A1, B0, C0, C1, C2, B1, C0, C1, C2, B2, C0, C1, C2, > A2, B0, C0, C1, C2, B1, C0, C1, C2, B2, C0, C1, C2] > > passes Here is filtered flattened product version. I think it clumsier than directly producing the items wanted, but it is good to know of this approach as a backup. from itertools import product def flatprod(iofi): # iterable of iterables lofi = list(iofi) now = [object()] * len(lofi) for new in product(*lofi): i = 0 while now[i] == new[i]: i += 1 yield from new[i:] now = new cfLL = list(flatprod(LL)) Same assert as before passes. -- Terry Jan Reedy From steve at divillo.com Wed Jul 24 14:11:54 2013 From: steve at divillo.com (steve at divillo.com) Date: Wed, 24 Jul 2013 11:11:54 -0700 (PDT) Subject: Converting a list of lists to a single list In-Reply-To: References: Message-ID: <2f857fe1-256f-48e7-9ad6-b686c84d32d6@googlegroups.com> Wow, thanks everyone. Very helpful indeed! On Tuesday, July 23, 2013 2:52:21 PM UTC-7, st... at divillo.com wrote: > I think that itertools may be able to do what I want but I have not been able to figure out how. > > > > I want to convert an arbitrary number of lists with an arbitrary number of elements in each list into a single list as follows. > > > > Say I have three lists: > > > > [[A0,A1,A2], [B0,B1,B2] [C0,C1,C2]] > > > > I would like to convert those to a single list that looks like this: > > > > [A0,B0,C0,C1,C2,B1,C0,C1,C2,B2,C0,C1,C2,A1,B0,C0,C1,C2,B1,C0,C1,C2,B2,C0,C1,C2,A2,B0,C0,C1,C2,B1,C0,C1,C2,B2,C0,C1,C2] > > > > An easier way to visualize the pattern I want is as a tree. > > > > A0 > > B0 > > C0 > > C1 > > C2 > > B1 > > C0 > > C1 > > C2 > > B2 > > C0 > > C1 > > C2 > > A1 > > B0 > > C0 > > C1 > > C2 > > B1 > > C0 > > C1 > > C2 > > B2 > > C0 > > C1 > > C2 > > A2 > > B0 > > C0 > > C1 > > C2 > > B1 > > C0 > > C1 > > C2 > > B2 > > C0 > > C1 > > C2 From ethan at stoneleaf.us Tue Jul 23 21:16:08 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 23 Jul 2013 18:16:08 -0700 Subject: Python 3: dict & dict.keys() Message-ID: <51EF2AD8.3080105@stoneleaf.us> Back in Python 2.x days I had a good grip on dict and dict.keys(), and when to use one or the other. Then Python 3 came on the scene with these things called 'views', and while range couldn't be bothered, dict jumped up and down shouting, "I want some!" So now, in Python 3, .keys(), .values(), even .items() all return these 'view' thingies. And everything I thought I knew about when to use one or the other went out the window. For example, if you need to modify a dict while iterating over it, use .keys(), right? Wrong: --> d = {1: 'one', 2:'two', 3:'three'} --> for k in d.keys(): ... if k == 1: ... del d[k] ... Traceback (most recent call last): File "", line 1, in RuntimeError: dictionary changed size during iteration If you need to manipulate the keys (maybe adding some, maybe deleting some) before doing something else with final key collection, use .keys(), right? Wrong: --> dk = d.keys() --> dk.remove(2) Traceback (most recent call last): File "", line 1, in AttributeError: 'dict_keys' object has no attribute 'remove' I understand that the appropriate incantation in Python 3 is: --> for k in list(d) ... ... or --> dk = list(d) --> dk.remove(2) which also has the added benefit of working the same way in Python 2. So, my question boils down to: in Python 3 how is dict.keys() different from dict? What are the use cases? -- ~Ethan~ From steve+comp.lang.python at pearwood.info Tue Jul 23 22:11:47 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 24 Jul 2013 02:11:47 GMT Subject: Python 3: dict & dict.keys() References: Message-ID: <51ef37e3$0$29971$c3e8da3$5496439d@news.astraweb.com> On Tue, 23 Jul 2013 18:16:08 -0700, Ethan Furman wrote: > Back in Python 2.x days I had a good grip on dict and dict.keys(), and > when to use one or the other. > > Then Python 3 came on the scene with these things called 'views', and > while range couldn't be bothered, dict jumped up and down shouting, "I > want some!" > > So now, in Python 3, .keys(), .values(), even .items() all return these > 'view' thingies. > > And everything I thought I knew about when to use one or the other went > out the window. Surely not. The fundamental behaviour of Python's data model hasn't changed. Lists are lists, views are views, and iterators are iterators. Only the way you get each has changed. - If in Python 2, you used the viewkeys() method, that's been renamed keys() in Python 3. So d.viewkeys() => d.keys(). - If in Python 2, you used the keys() method, it returns a list, and like any function that has been made lazy instead of eager in Python 3 (e.g. map, zip, filter) if you want the same behaviour, simply call list manually. So d.keys() => list(d.keys()). - If in Python 2, you used the iterkeys() methods, it returns a simple iterator, not a view. So d.iterkeys() => iter(d.keys()). None of these distinctions really matter if all you are doing is iterating over the keys, without modifying the dict. Not in Python 2, nor in Python 3. And naturally the same applies to the various flavours of *items and *values. > For example, if you need to modify a dict while iterating over it, use > .keys(), right? Wrong: > > --> d = {1: 'one', 2:'two', 3:'three'} --> for k in d.keys(): > ... if k == 1: > ... del d[k] > ... > Traceback (most recent call last): > File "", line 1, in > RuntimeError: dictionary changed size during iteration Fundamentally, this behaviour has not changed from Python 2: you should not iterate over a data structure while changing it, instead you should make a copy of the data you iterate over. In Python 2, d.keys() makes a copy and returns a list, so in Python 3 you would call list(d.keys()). > If you need to manipulate the keys (maybe adding some, maybe deleting > some) before doing something else with final key collection, use > .keys(), right? Wrong: > > --> dk = d.keys() > --> dk.remove(2) > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'dict_keys' object has no attribute 'remove' Repeat after me: "In Python 2, d.keys() returns a list of keys, so if I want a list of keys in Python 3, call list explicitly list(d.keys())." > I understand that the appropriate incantation in Python 3 is: > > --> for k in list(d) > ... ... > > or > > --> dk = list(d) > --> dk.remove(2) > > which also has the added benefit of working the same way in Python 2. > > So, my question boils down to: in Python 3 how is dict.keys() different > from dict? What are the use cases? *shrug* For most purposes, there is no difference, especially when merely iterating over the dict. Such differences as exist are trivial: - if you need an actual callable function or method, say to pass to some other function, you can do this: for method in (d.items, d.keys, d.values): process(method) instead of this: # untested for method in (d.items, d.keys, lambda d=d: iter(d)): process(method) - d.keys() is a view, not the dict itself. That's a pretty fundamental difference: compare dir(d.keys()) with dir(d). Basically, views are set-like, not list-like. -- Steven From ian.g.kelly at gmail.com Wed Jul 24 11:02:29 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 24 Jul 2013 09:02:29 -0600 Subject: Python 3: dict & dict.keys() In-Reply-To: <51ef37e3$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <51ef37e3$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, Jul 23, 2013 at 8:11 PM, Steven D'Aprano wrote: > Basically, views are set-like, not list-like. The keys and items views are set-like. The values view is not. From ethan at stoneleaf.us Wed Jul 24 20:59:43 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 24 Jul 2013 17:59:43 -0700 Subject: Python 3: dict & dict.keys() In-Reply-To: <51ef37e3$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <51ef37e3$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51F0787F.9010406@stoneleaf.us> On 07/23/2013 07:11 PM, Steven D'Aprano wrote: > On Tue, 23 Jul 2013 18:16:08 -0700, Ethan Furman wrote: >> >> So now, in Python 3, .keys(), .values(), even .items() all return these >> 'view' thingies. >> >> And everything I thought I knew about when to use one or the other went >> out the window. > > Surely not. The fundamental behaviour of Python's data model hasn't > changed. Poetic effect. Dramatic license. Blah blah. ;) > Repeat after me: "In Python 2, d.keys() returns a list of keys, so if I > want a list of keys in Python 3, call list explicitly list(d.keys())." Actually, I would recommend `list(d)`, which also works the same in both 2 and 3. -- ~Ethan~ From ben+python at benfinney.id.au Wed Jul 24 22:20:18 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 25 Jul 2013 12:20:18 +1000 Subject: Python 3: dict & dict.keys() References: <51ef37e3$0$29971$c3e8da3$5496439d@news.astraweb.com> <51F0787F.9010406@stoneleaf.us> Message-ID: <7wwqofidnh.fsf@benfinney.id.au> Ethan Furman writes: > On 07/23/2013 07:11 PM, Steven D'Aprano wrote: > > On Tue, 23 Jul 2013 18:16:08 -0700, Ethan Furman wrote: > >> And everything I thought I knew about when to use one or the other went > >> out the window. > > > > Surely not. The fundamental behaviour of Python's data model hasn't > > changed. > > Poetic effect. Dramatic license. Blah blah. ;) Text-only medium. Clarity of communication. Et cetera. :-) -- \ ?Two hands working can do more than a thousand clasped in | `\ prayer.? ?Anonymous | _o__) | Ben Finney From steve+comp.lang.python at pearwood.info Thu Jul 25 01:57:37 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 25 Jul 2013 05:57:37 GMT Subject: Python 3: dict & dict.keys() References: <51ef37e3$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51f0be51$0$29971$c3e8da3$5496439d@news.astraweb.com> On Wed, 24 Jul 2013 17:59:43 -0700, Ethan Furman wrote: >> Repeat after me: "In Python 2, d.keys() returns a list of keys, so if I >> want a list of keys in Python 3, call list explicitly list(d.keys())." > > Actually, I would recommend `list(d)`, which also works the same in both > 2 and 3. Fair point. -- Steven From __peter__ at web.de Wed Jul 24 02:23:08 2013 From: __peter__ at web.de (Peter Otten) Date: Wed, 24 Jul 2013 08:23:08 +0200 Subject: Python 3: dict & dict.keys() References: <51EF2AD8.3080105@stoneleaf.us> Message-ID: Ethan Furman wrote: > So, my question boils down to: in Python 3 how is dict.keys() different > from dict? What are the use cases? I just grepped through /usr/lib/python3, and could not identify a single line where some_object.keys() wasn't either wrapped in a list (or set, sorted, max) call, or iterated over. To me it looks like views are a solution waiting for a problem. From oscar.j.benjamin at gmail.com Wed Jul 24 08:51:34 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 24 Jul 2013 13:51:34 +0100 Subject: Python 3: dict & dict.keys() In-Reply-To: References: <51EF2AD8.3080105@stoneleaf.us> Message-ID: On Jul 24, 2013 7:25 AM, "Peter Otten" <__peter__ at web.de> wrote: > > Ethan Furman wrote: > > > So, my question boils down to: in Python 3 how is dict.keys() different > > from dict? What are the use cases? > > I just grepped through /usr/lib/python3, and could not identify a single > line where some_object.keys() wasn't either wrapped in a list (or set, > sorted, max) call, or iterated over. > > To me it looks like views are a solution waiting for a problem. What do you mean? Why would you want to create a temporary list just to iterate over it explicitly or implicitly (set, sorted, max,...)? Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From __peter__ at web.de Wed Jul 24 09:25:14 2013 From: __peter__ at web.de (Peter Otten) Date: Wed, 24 Jul 2013 15:25:14 +0200 Subject: Python 3: dict & dict.keys() References: <51EF2AD8.3080105@stoneleaf.us> Message-ID: Oscar Benjamin wrote: > On Jul 24, 2013 7:25 AM, "Peter Otten" <__peter__ at web.de> wrote: >> >> Ethan Furman wrote: >> >> > So, my question boils down to: in Python 3 how is dict.keys() >> > different >> > from dict? What are the use cases? >> >> I just grepped through /usr/lib/python3, and could not identify a single >> line where some_object.keys() wasn't either wrapped in a list (or set, >> sorted, max) call, or iterated over. >> >> To me it looks like views are a solution waiting for a problem. > > What do you mean? Why would you want to create a temporary list just to > iterate over it explicitly or implicitly (set, sorted, max,...)? I mean I don't understand the necessity of views when all actual usecases need iterators. The 2.x iterkeys()/iteritems()/itervalues() methods didn't create lists either. Do you have 2.x code lying around where you get a significant advantage by picking some_dict.viewkeys() over some_dict.iterkeys()? I could construct one >>> d = dict(a=1, b=2, c=3) >>> e = dict(b=4, c=5, d=6) >>> d.viewkeys() & e.viewkeys() set(['c', 'b']) but have not seen it in the wild. My guess is that most non-hardcore users don't even know about viewkeys(). By the way, my favourite idiom to iterate over the keys in both Python 2 and 3 is -- for example -- max(some_dict) rather than max(some_dict.whateverkeys()). From skip at pobox.com Wed Jul 24 10:58:21 2013 From: skip at pobox.com (Skip Montanaro) Date: Wed, 24 Jul 2013 09:58:21 -0500 Subject: Python 3: dict & dict.keys() In-Reply-To: References: <51EF2AD8.3080105@stoneleaf.us> Message-ID: > What do you mean? Why would you want to create a temporary list just to > iterate over it explicitly or implicitly (set, sorted, max,...)? Because while iterating over the keys, he might also want to add or delete keys to/from the dict. You can't do that while iterating over them in-place. This example demonstrates the issue and also shows that the modification actually takes place: >>> d = dict(zip(range(10), range(10, 0, -1))) >>> d {0: 10, 1: 9, 2: 8, 3: 7, 4: 6, 5: 5, 6: 4, 7: 3, 8: 2, 9: 1} >>> for k in d: ... if k == 3: ... del d[k+1] ... Traceback (most recent call last): File "", line 1, in RuntimeError: dictionary changed size during iteration >>> for k in list(d): ... if k == 3: ... del d[k+1] ... Traceback (most recent call last): File "", line 3, in KeyError: 4 >>> d.keys() dict_keys([0, 1, 2, 3, 5, 6, 7, 8, 9]) Skip From ian.g.kelly at gmail.com Wed Jul 24 11:15:06 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 24 Jul 2013 09:15:06 -0600 Subject: Python 3: dict & dict.keys() In-Reply-To: References: <51EF2AD8.3080105@stoneleaf.us> Message-ID: On Wed, Jul 24, 2013 at 8:58 AM, Skip Montanaro wrote: >> What do you mean? Why would you want to create a temporary list just to >> iterate over it explicitly or implicitly (set, sorted, max,...)? > > Because while iterating over the keys, he might also want to add or > delete keys to/from the dict. You can't do that while iterating over > them in-place. None of the (set, sorted, max, ...) cases will add or delete keys while iterating. For the occasional for loop where the programmer does want to do that, you can still explicitly create a temporary list with list(). From ethan at stoneleaf.us Wed Jul 24 11:57:11 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 24 Jul 2013 08:57:11 -0700 Subject: Python 3: dict & dict.keys() In-Reply-To: References: <51EF2AD8.3080105@stoneleaf.us> Message-ID: <51EFF957.1040707@stoneleaf.us> On 07/24/2013 05:51 AM, Oscar Benjamin wrote: > > On Jul 24, 2013 7:25 AM, "Peter Otten" <__peter__ at web.de > wrote: >> >> Ethan Furman wrote: >> >> > So, my question boils down to: in Python 3 how is dict.keys() different >> > from dict? What are the use cases? >> >> I just grepped through /usr/lib/python3, and could not identify a single >> line where some_object.keys() wasn't either wrapped in a list (or set, >> sorted, max) call, or iterated over. >> >> To me it looks like views are a solution waiting for a problem. > > What do you mean? Why would you want to create a temporary list just to iterate over it explicitly or implicitly (set, > sorted, max,...)? You wouldn't. But you don't need .keys() for that either as you can just use the dict itself. My point is that in 2.x .keys() did something different from the dict, while in 3.x it appears to me that they are the same. Peter's point is that in the stdlib the new functionality of .keys() is never used, not even once. -- ~Ethan~ From rosuav at gmail.com Wed Jul 24 12:34:50 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 02:34:50 +1000 Subject: Python 3: dict & dict.keys() In-Reply-To: <51EFF957.1040707@stoneleaf.us> References: <51EF2AD8.3080105@stoneleaf.us> <51EFF957.1040707@stoneleaf.us> Message-ID: On Thu, Jul 25, 2013 at 1:57 AM, Ethan Furman wrote: > On 07/24/2013 05:51 AM, Oscar Benjamin wrote: >> What do you mean? Why would you want to create a temporary list just to >> iterate over it explicitly or implicitly (set, >> sorted, max,...)? > > You wouldn't. But you don't need .keys() for that either as you can just > use the dict itself. Side point: Why is iterating over a dict equivalent to .keys() rather than .items()? It feels odd that, with both options viable, the implicit version iterates over half the dict instead of all of it. Obviously it can't be changed now, even if .items() were the better choice, but I'm curious as to the reason for the decision. ChrisA From stefan_ml at behnel.de Wed Jul 24 13:16:40 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 24 Jul 2013 19:16:40 +0200 Subject: Python 3: dict & dict.keys() In-Reply-To: References: <51EF2AD8.3080105@stoneleaf.us> <51EFF957.1040707@stoneleaf.us> Message-ID: Chris Angelico, 24.07.2013 18:34: > On Thu, Jul 25, 2013 at 1:57 AM, Ethan Furman wrote: >> On 07/24/2013 05:51 AM, Oscar Benjamin wrote: >>> What do you mean? Why would you want to create a temporary list just to >>> iterate over it explicitly or implicitly (set, >>> sorted, max,...)? >> >> You wouldn't. But you don't need .keys() for that either as you can just >> use the dict itself. > > Side point: Why is iterating over a dict equivalent to .keys() rather > than .items()? It feels odd that, with both options viable, the > implicit version iterates over half the dict instead of all of it. > Obviously it can't be changed now, even if .items() were the better > choice, but I'm curious as to the reason for the decision. The reason is that you can easily get at the values when iterating over the keys, or simply decide not to care about them and be happy with the keys only. Note that there are also many use cases that need all keys but not all values. If iteration always returned an item tuple by default, many use cases would have to resort to using .keys() in order to be efficient. And for the simple case, you'd have to type more, either the additional .keys() or the useless tuple unpacking. So, the reasoning is that iteration should do the basic thing that still allows you to do everything, instead of doing everything and pushing unnecessary work on the users by default. Stefan From tjreedy at udel.edu Wed Jul 24 13:17:12 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 24 Jul 2013 13:17:12 -0400 Subject: Python 3: dict & dict.keys() In-Reply-To: References: <51EF2AD8.3080105@stoneleaf.us> <51EFF957.1040707@stoneleaf.us> Message-ID: On 7/24/2013 12:34 PM, Chris Angelico wrote: > Side point: Why is iterating over a dict equivalent to .keys() rather > than .items()? It feels odd that, with both options viable, the > implicit version iterates over half the dict instead of all of it. > Obviously it can't be changed now, even if .items() were the better > choice, but I'm curious as to the reason for the decision. Both were considered and I think there were and are two somewhat-linked practical reasons. First, iterating over keys in more common than iterating over items. The more common one should be the default. Second, people ask much more often if 'key' is in dict than if 'key, value' is in dict. This is true as well for keyed reference books such as phone books, dictionaries, encyclopedias, and for the same reason. This is coupled with the fact that the default meaning of 'item in collection' is that iterating over 'collection' eventually produces 'item' or a value equal to 'item'. -- Terry Jan Reedy From rosuav at gmail.com Wed Jul 24 13:58:55 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 03:58:55 +1000 Subject: Python 3: dict & dict.keys() In-Reply-To: References: <51EF2AD8.3080105@stoneleaf.us> <51EFF957.1040707@stoneleaf.us> Message-ID: On Thu, Jul 25, 2013 at 3:17 AM, Terry Reedy wrote: > On 7/24/2013 12:34 PM, Chris Angelico wrote: > >> Side point: Why is iterating over a dict equivalent to .keys() rather >> than .items()? It feels odd that, with both options viable, the >> implicit version iterates over half the dict instead of all of it. >> Obviously it can't be changed now, even if .items() were the better >> choice, but I'm curious as to the reason for the decision. > > This is > coupled with the fact that the default meaning of 'item in collection' is > that iterating over 'collection' eventually produces 'item' or a value equal > to 'item'. Ahh, that makes sense. I never thought of iteration and 'in' being connected like that, but yes, that's a solid reason for doing it that way. ChrisA From christian at python.org Wed Jul 24 17:06:56 2013 From: christian at python.org (Christian Heimes) Date: Wed, 24 Jul 2013 23:06:56 +0200 Subject: Python 3: dict & dict.keys() In-Reply-To: References: <51EF2AD8.3080105@stoneleaf.us> <51EFF957.1040707@stoneleaf.us> Message-ID: Am 24.07.2013 18:34, schrieb Chris Angelico: > Side point: Why is iterating over a dict equivalent to .keys() rather > than .items()? It feels odd that, with both options viable, the > implicit version iterates over half the dict instead of all of it. > Obviously it can't be changed now, even if .items() were the better > choice, but I'm curious as to the reason for the decision. Consider this: if key in dict: ... for key in dict: ... It would be rather surprising if "in" as containment checks operates on keys and "in" as iterator returns (key, value) tuples. Christian From steve+comp.lang.python at pearwood.info Thu Jul 25 01:52:50 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 25 Jul 2013 05:52:50 GMT Subject: Python 3: dict & dict.keys() References: <51EF2AD8.3080105@stoneleaf.us> <51EFF957.1040707@stoneleaf.us> Message-ID: <51f0bd32$0$29971$c3e8da3$5496439d@news.astraweb.com> On Wed, 24 Jul 2013 13:17:12 -0400, Terry Reedy wrote: > On 7/24/2013 12:34 PM, Chris Angelico wrote: > >> Side point: Why is iterating over a dict equivalent to .keys() rather >> than .items()? It feels odd that, with both options viable, the >> implicit version iterates over half the dict instead of all of it. >> Obviously it can't be changed now, even if .items() were the better >> choice, but I'm curious as to the reason for the decision. > > Both were considered and I think there were and are two somewhat-linked > practical reasons. First, iterating over keys in more common than > iterating over items. The more common one should be the default. > > Second, people ask much more often if 'key' is in dict than if 'key, > value' is in dict. This is true as well for keyed reference books such > as phone books, dictionaries, encyclopedias, and for the same reason. > This is coupled with the fact that the default meaning of 'item in > collection' is that iterating over 'collection' eventually produces > 'item' or a value equal to 'item'. That second point was the deciding factor when direct iteration over dicts was added. has_key() was deprecated in favour of "key in dict", and that pretty much forced iteration to go over keys by default. The reasoning is, "x in y" ought to be equivalent to: for tmp in y: if x == tmp: return True return False There's probably even a PEP about this, if anyone is less lazy/busy and can be bothered looking for it. -- Steven From steve+comp.lang.python at pearwood.info Thu Jul 25 01:48:36 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 25 Jul 2013 05:48:36 GMT Subject: Python 3: dict & dict.keys() References: <51EF2AD8.3080105@stoneleaf.us> Message-ID: <51f0bc34$0$29971$c3e8da3$5496439d@news.astraweb.com> On Wed, 24 Jul 2013 08:57:11 -0700, Ethan Furman wrote: > My point is that in 2.x .keys() did something different from the dict, > while in 3.x it appears to me that they are the same. Then you aren't looking very closely. d.keys() returns a set-like view into the dict, which is great for comparing elements: py> d1 = dict.fromkeys([1, 2, 3, 4]) py> d2 = dict.fromkeys([3, 4, 5, 6]) py> d1.keys() & d2.keys() # keys that are in both {3, 4} py> d1.keys() ^ d2.keys() # keys not in both {1, 2, 5, 6} py> d1.keys() - d2.keys() # keys only in d1 {1, 2} py> d2.keys() - d1.keys() # keys only in d2 {5, 6} Dicts aren't sets, and don't support set methods: py> d1 - d2 Traceback (most recent call last): File "", line 1, in TypeError: unsupported operand type(s) for -: 'dict' and 'dict' > Peter's point is that in the stdlib the new functionality of .keys() is > never used, not even once. The standard library is not the universe of Python code, and most of the standard library predates Python 3. Some of it goes back to Python 1 idioms. In general, working code doesn't get updated until it stops working. I have code that manually walks over each dict and extracts keys that are in both, or one but not the other. Once I drop support for Python 2.6, I throw that code away and just use views. But until then, I'm stuck doing it the horrible way. Judging by a naive grep of my code, you might think I never used views. I do, I just have to reinvent the wheel. -- Steven From rosuav at gmail.com Thu Jul 25 02:02:42 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 16:02:42 +1000 Subject: Python 3: dict & dict.keys() In-Reply-To: <51f0bc34$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <51EF2AD8.3080105@stoneleaf.us> <51f0bc34$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, Jul 25, 2013 at 3:48 PM, Steven D'Aprano wrote: > Dicts aren't sets, and don't support set methods: > > py> d1 - d2 > Traceback (most recent call last): > File "", line 1, in > TypeError: unsupported operand type(s) for -: 'dict' and 'dict' I wouldn't take this as particularly significant, though. A future version of Python could add that support (and it might well be very useful), without breaking any of the effects of views. ChrisA From steve+comp.lang.python at pearwood.info Thu Jul 25 03:27:40 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 25 Jul 2013 07:27:40 GMT Subject: Python 3: dict & dict.keys() References: <51EF2AD8.3080105@stoneleaf.us> <51f0bc34$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51f0d36c$0$29971$c3e8da3$5496439d@news.astraweb.com> On Thu, 25 Jul 2013 16:02:42 +1000, Chris Angelico wrote: > On Thu, Jul 25, 2013 at 3:48 PM, Steven D'Aprano > wrote: >> Dicts aren't sets, and don't support set methods: >> >> py> d1 - d2 >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: unsupported operand type(s) for -: 'dict' and 'dict' > > I wouldn't take this as particularly significant, though. A future > version of Python could add that support (and it might well be very > useful), without breaking any of the effects of views. I don't think dicts can ever support set methods, since *they aren't sets*. Every element consists of both a key and a value, so you have to consider both. Set methods are defined in terms of singleton elements, not binary elements, so before you even begin, you have to decide what does it mean when two elements differ in only one of the two parts? Given dicts {1: 'a'}, {1: 'b'}, what is the union of them? I can see five possibilities: {1: 'a'} {1: 'b'} {1: ['a', 'b']} {1: set(['a', 'b'])} Error Each of the five results may be what you want in some circumstances. It would be a stupid thing for dict.union to pick one behaviour and make it the One True Way to perform union on two dicts. -- Steven From rosuav at gmail.com Thu Jul 25 04:15:22 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 18:15:22 +1000 Subject: Python 3: dict & dict.keys() In-Reply-To: <51f0d36c$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <51EF2AD8.3080105@stoneleaf.us> <51f0bc34$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f0d36c$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, Jul 25, 2013 at 5:27 PM, Steven D'Aprano wrote: > On Thu, 25 Jul 2013 16:02:42 +1000, Chris Angelico wrote: > >> On Thu, Jul 25, 2013 at 3:48 PM, Steven D'Aprano >> wrote: >>> Dicts aren't sets, and don't support set methods: >>> >>> py> d1 - d2 >>> Traceback (most recent call last): >>> File "", line 1, in >>> TypeError: unsupported operand type(s) for -: 'dict' and 'dict' >> >> I wouldn't take this as particularly significant, though. A future >> version of Python could add that support (and it might well be very >> useful), without breaking any of the effects of views. > > I don't think dicts can ever support set methods, since *they aren't > sets*. Every element consists of both a key and a value, so you have to > consider both. Set methods are defined in terms of singleton elements, > not binary elements, so before you even begin, you have to decide what > does it mean when two elements differ in only one of the two parts? > > Given dicts {1: 'a'}, {1: 'b'}, what is the union of them? I can see five > possibilities: > > {1: 'a'} > {1: 'b'} > {1: ['a', 'b']} > {1: set(['a', 'b'])} > Error > > Each of the five results may be what you want in some circumstances. It > would be a stupid thing for dict.union to pick one behaviour and make it > the One True Way to perform union on two dicts. That's true, but we already have that issue with sets. What's the union of {0} and {0.0}? Python's answer: It depends on the order of the operands. >>> i={0} >>> f={0.0} >>> i | f {0} >>> f | i {0.0} I would say that Python can freely pick from the first two options you offered (either keep-first or keep-last), most likely the first one, and it'd make good sense. Your third option would be good for a few specific circumstances, but then you probably would also want the combination of {1:'a'} and {1:'a'} to be {1:['a','a']} for consistency. This would make a good utility function, but isn't what I'd normally see set union doing. Similarly with the fourth option, though there it's a little more arguable. Raising an error would work, but is IMO unnecessary. (Pike has dictionary operations, but has made different choices. For instance, 0 and 0.0 are considered distinct, so a set can contain both. Mappings (dicts) merge by keeping the last, not the first. But the specifics don't much matter.) A Python set already has to distinguish between object value and object identity; a dict simply adds a bit more distinction between otherwise-considered-identical keys, namely their values. >>> a="This is a test." >>> b="This is a test." >>> a is b False >>> id(a) 16241512 >>> id(b) 16241392 >>> id(({a}|{b}).pop()) 16241512 Assuming a and b have different ids, which is true in the above example of Py3.3 on Windows, the set union *must* be different from one of them. Suppose you do a dictionary of id(key) -> value, and a set of the keys themselves. You could then do set operations on the keys, and then go and retrieve the values. Sure, maybe the way of doing things won't be exactly what everyone expects... but it works, and it makes plausible sense. And as a theoretical "might be implemented in Python 3.5", it still has no impact on views, beyond that there are some operations which must be done with views in <=3.3 that could be done on the dicts themselves in future. ChrisA From steve+comp.lang.python at pearwood.info Thu Jul 25 05:44:43 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 25 Jul 2013 09:44:43 GMT Subject: Python 3: dict & dict.keys() References: <51EF2AD8.3080105@stoneleaf.us> <51f0bc34$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f0d36c$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51f0f38b$0$29971$c3e8da3$5496439d@news.astraweb.com> On Thu, 25 Jul 2013 18:15:22 +1000, Chris Angelico wrote: > On Thu, Jul 25, 2013 at 5:27 PM, Steven D'Aprano > wrote: >> On Thu, 25 Jul 2013 16:02:42 +1000, Chris Angelico wrote: >> >>> On Thu, Jul 25, 2013 at 3:48 PM, Steven D'Aprano >>> wrote: >>>> Dicts aren't sets, and don't support set methods: >>>> >>>> py> d1 - d2 >>>> Traceback (most recent call last): >>>> File "", line 1, in >>>> TypeError: unsupported operand type(s) for -: 'dict' and 'dict' >>> >>> I wouldn't take this as particularly significant, though. A future >>> version of Python could add that support (and it might well be very >>> useful), without breaking any of the effects of views. >> >> I don't think dicts can ever support set methods, since *they aren't >> sets*. Every element consists of both a key and a value, so you have to >> consider both. Set methods are defined in terms of singleton elements, >> not binary elements, so before you even begin, you have to decide what >> does it mean when two elements differ in only one of the two parts? >> >> Given dicts {1: 'a'}, {1: 'b'}, what is the union of them? I can see >> five possibilities: >> >> {1: 'a'} >> {1: 'b'} >> {1: ['a', 'b']} >> {1: set(['a', 'b'])} >> Error >> >> Each of the five results may be what you want in some circumstances. It >> would be a stupid thing for dict.union to pick one behaviour and make >> it the One True Way to perform union on two dicts. > > That's true, but we already have that issue with sets. What's the union > of {0} and {0.0}? Python's answer: It depends on the order of the > operands. That's a side-effect of how numeric equality works in Python. Since 0 == 0.0, you can't have both as keys in the same dict, or set. Indeed, the same numeric equality issue occurs here: py> from fractions import Fraction py> [0, 2.5] == [0.0, Fraction(5, 2)] True So nothing really to do with sets or dicts specifically. Aside: I think the contrary behaviour is, well, contrary. It would be strange and disturbing to do this: for key in some_dict: if key == 0: print("found") print(some_dict[key]) and have the loop print "found" and then have the key lookup fail, but apparently that's how things work in Pike :-( > I would say that Python can freely pick from the first two options you > offered (either keep-first or keep-last), most likely the first one, and > it'd make good sense. Your third option would be good for a few specific > circumstances, but then you probably would also want the combination of > {1:'a'} and {1:'a'} to be {1:['a','a']} for consistency. Okay, that's six variations. And no, I don't think the "consistency" argument is right -- the idea is that you can have multiple values per key. Since 'a' == 'a', that's only one value, not two. The variation using a list, versus the set, depends on whether you care about order or hashability. [...] > Raising an error would work, but is IMO unnecessary. I believe that's the only reasonable way for a dict union method to work. As the Zen says: In the face of ambiguity, refuse the temptation to guess. Since there is ambiguity which value should be associated with the key, don't guess. [...] > A Python set already has to distinguish between object value and object > identity; a dict simply adds a bit more distinction between > otherwise-considered-identical keys, namely their values. Object identity is a red herring. It would be perfectly valid for a Python implementation to create new instances of each element in the set union, assuming such creation was free of side-effects (apart from memory usage and time, naturally). set.union() makes no promise about the identity of elements, and it is defined the same way for languages where object identity does not exist (say, old-school Pascal). -- Steven From rosuav at gmail.com Thu Jul 25 06:34:23 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 20:34:23 +1000 Subject: Python 3: dict & dict.keys() In-Reply-To: <51f0f38b$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <51EF2AD8.3080105@stoneleaf.us> <51f0bc34$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f0d36c$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f0f38b$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, Jul 25, 2013 at 7:44 PM, Steven D'Aprano wrote: > On Thu, 25 Jul 2013 18:15:22 +1000, Chris Angelico wrote: >> That's true, but we already have that issue with sets. What's the union >> of {0} and {0.0}? Python's answer: It depends on the order of the >> operands. > > That's a side-effect of how numeric equality works in Python. Since 0 == > 0.0, you can't have both as keys in the same dict, or set. Indeed, the > same numeric equality issue occurs here: > > py> from fractions import Fraction > py> [0, 2.5] == [0.0, Fraction(5, 2)] > True > > So nothing really to do with sets or dicts specifically. Here's how I imagine set/dict union: 1) Take a copy of the first object 2) Iterate through the second. If the key doesn't exist in the result, add it. This works just fine even when "add it" means "store this value against this key". The dict's value and the object's identity are both ignored, and you simply take the first one you find. > Aside: I think the contrary behaviour is, well, contrary. It would be > strange and disturbing to do this: > > for key in some_dict: > if key == 0: > print("found") > print(some_dict[key]) > > and have the loop print "found" and then have the key lookup fail, but > apparently that's how things work in Pike :-( I agree, that would be very strange and disturbing. I mentioned that aspect merely in passing, but the reason for the difference is not an oddity of key lookup, but a different decision about float and int: in Pike, 0 and 0.0 are not equal. (Nor are 1 and 1.0, in case you thought this was a weirdness of zero.) It's a debatable point; are we trying to say that all numeric types represent real numbers, and are equal if they represent the same real number? Or are different representations distinct, just as much as the string "0" is different from the integer 0? Pike took the latter approach. PHP took the former approach to its illogical extreme, that the string "0001E1" is equal to "000010" (both strings). No, the dictionary definitely needs to use object equality to do its lookup, although I could well imagine an implementation that runs orders of magnitude faster when object identity can be used. >> I would say that Python can freely pick from the first two options you >> offered (either keep-first or keep-last), most likely the first one, and >> it'd make good sense. Your third option would be good for a few specific >> circumstances, but then you probably would also want the combination of >> {1:'a'} and {1:'a'} to be {1:['a','a']} for consistency. > > Okay, that's six variations. And no, I don't think the "consistency" > argument is right -- the idea is that you can have multiple values per > key. Since 'a' == 'a', that's only one value, not two. Well, it depends what you're doing with the merging of the dicts. But all of these extra ways to do things would be explicitly-named functions with much rarer usage (and quite possibly not part of the standard library, they'd be snippets shared around and put directly in application code). >> Raising an error would work, but is IMO unnecessary. > > I believe that's the only reasonable way for a dict union method to work. > As the Zen says: > > In the face of ambiguity, refuse the temptation to guess. > > Since there is ambiguity which value should be associated with the key, > don't guess. There's already ambiguity as to which of two equal values should be retained by the set. Python takes the first. Is that guessing? Is that violating the zen? I don't see a problem with the current set implementation, and I also don't see a problem with using that for dict merging. > Object identity is a red herring. It would be perfectly valid for a > Python implementation to create new instances of each element in the set > union, assuming such creation was free of side-effects (apart from memory > usage and time, naturally). set.union() makes no promise about the > identity of elements, and it is defined the same way for languages where > object identity does not exist (say, old-school Pascal). That still doesn't deal with the "which type should the new object be". We're back to this question: What is the union of 0 and 0.0? >>> {0} | {0.0} {0} >>> {0.0} | {0} {0.0} Maybe Python could create a brand new object, but would it be an int or a float? The only way I could imagine this working is with a modified-set class that takes an object constructor, and passes every object through it. That way, you could have set(float) that coerces everything to float on entry, which would enforce what you're saying (even down to potentially creating a new object with a new id, though float() seems to return a float argument unchanged in CPython 3.3). Would that really help anything, though? Do we gain anything by not simply accepting, in the manner of Colonel Fairfax, the first that comes? ChrisA From steve+comp.lang.python at pearwood.info Thu Jul 25 10:57:10 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 25 Jul 2013 14:57:10 GMT Subject: Python 3: dict & dict.keys() References: <51EF2AD8.3080105@stoneleaf.us> <51f0bc34$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f0d36c$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f0f38b$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51f13cc5$0$29971$c3e8da3$5496439d@news.astraweb.com> On Thu, 25 Jul 2013 20:34:23 +1000, Chris Angelico wrote: > On Thu, Jul 25, 2013 at 7:44 PM, Steven D'Aprano > wrote: >> On Thu, 25 Jul 2013 18:15:22 +1000, Chris Angelico wrote: >>> That's true, but we already have that issue with sets. What's the >>> union of {0} and {0.0}? Python's answer: It depends on the order of >>> the operands. >> >> That's a side-effect of how numeric equality works in Python. Since 0 >> == 0.0, you can't have both as keys in the same dict, or set. Indeed, >> the same numeric equality issue occurs here: >> >> py> from fractions import Fraction >> py> [0, 2.5] == [0.0, Fraction(5, 2)] True >> >> So nothing really to do with sets or dicts specifically. > > Here's how I imagine set/dict union: > 1) Take a copy of the first object > 2) Iterate through the second. If the key doesn't exist in the result, > add it. That's because you're too much of a programmer to step away from the implementation. Fundamentally, set union has nothing to do with objects, or bit strings, or any concrete implementation. Sets might be infinite, and "take a copy" impossible or meaningless. Logically, the union of set A and set B is the set containing every element which is in A, every element in B, and no element which is not. How you assemble those elements in a concrete implementation is, in a sense, irrelevant. In old-school Pascal, the universe of possible elements is taken from the 16-bit, or 32-bit if you're lucky, integers; in Python, it's taken from hashable objects. Even using your suggested algorithm above, since union is symmetric, it should make no difference whether you start with the first, or with the second. > This works just fine even when "add it" means "store this value against > this key". The dict's value and the object's identity are both ignored, > and you simply take the first one you find. I don't believe that "works", since the whole point of dicts is to store the values. In practice, the values are more important than the keys. The key only exists so you can get to the value -- the key is equivalent to the index in a list, the value to the value at that index. We normally end up doing something like "print adict[key]", not "print key". So throwing away the values just because they happen to have the same key is a fairly dubious thing to do, at least for union or intersection. (In contrast, that's exactly what you want an update method to do. Different behaviour for different methods.) [...] >>> Raising an error would work, but is IMO unnecessary. >> >> I believe that's the only reasonable way for a dict union method to >> work. As the Zen says: >> >> In the face of ambiguity, refuse the temptation to guess. >> >> Since there is ambiguity which value should be associated with the key, >> don't guess. > > There's already ambiguity as to which of two equal values should be > retained by the set. In an ideal world of Platonic Ideals, it wouldn't matter, since everything is equal to itself, and to nothing else. There's only one number "two", whether you write it as 2 or 2.0 or 800/400 or ? or 0b10, and it is *impossible even in principle* to distinguish them since there is no "them" to distinguish between. Things that are equal shouldn't be distinguishable, not by value, not by type, not by identity. But that would be *too abstract* to be useful, and so we allow some of the abstractness leak away, to the benefit of all. But the consequence of this is that we sometimes have to make hard decisions, like, which one of these various "twos" do we want to keep? Or more often, we stumble into a decision by allowing the implementation specify the behaviour, rather than choosing the behaviour and the finding an implementation to match it. Given the two behaviours: {2} | {2.0} => {2} or {2.0}, which should it be? Why not Fraction(2) or Decimal(2) or 2+0j? there's no reason to prefer Python's answer, "the value on the left", except that it simplifies the implementation. The union operator ought to be symmetrical, a ? b should be identical to b ? a, but isn't. Another leaky abstraction. -- Steven From rosuav at gmail.com Thu Jul 25 11:07:55 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 26 Jul 2013 01:07:55 +1000 Subject: Python 3: dict & dict.keys() In-Reply-To: <51f13cc5$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <51EF2AD8.3080105@stoneleaf.us> <51f0bc34$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f0d36c$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f0f38b$0$29971$c3e8da3$5496439d@news.astraweb.com> <51f13cc5$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Fri, Jul 26, 2013 at 12:57 AM, Steven D'Aprano wrote: > [ snip lengthy explanation of sets ] > The union operator ought to > be symmetrical, a ? b should be identical to b ? a, but isn't. Another > leaky abstraction. Right. I agree with all your theory, which is fine and good. If we had a "set of real numbers", then each one would be both equal to and indistinguishable from any other representation of itself. But Python doesn't work with real numbers. It works with ints and floats and Fractions and Decimals and Guido-knows-what. (Sorry, I don't normally talk like that, but the expression begged to be said. :) ) So since Python already lets its abstraction leak a bit for usefulness, why not retain the exact same leak when working with a dict? A set is a dict with no values... a dict is a set with extra payload. They're given very similar literal notation; if they were meant to be more distinct, why was no alternative symbol used? (I love how a random side comment can become a topic of its own.) ChrisA From dfnsonfsduifb at gmx.de Thu Jul 25 09:22:59 2013 From: dfnsonfsduifb at gmx.de (Johannes Bauer) Date: Thu, 25 Jul 2013 15:22:59 +0200 Subject: Python 3: dict & dict.keys() In-Reply-To: <51f0bc34$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <51EF2AD8.3080105@stoneleaf.us> <51f0bc34$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 25.07.2013 07:48, Steven D'Aprano wrote: > Then you aren't looking very closely. d.keys() returns a set-like view > into the dict, which is great for comparing elements: > > py> d1 = dict.fromkeys([1, 2, 3, 4]) > py> d2 = dict.fromkeys([3, 4, 5, 6]) > py> d1.keys() & d2.keys() # keys that are in both > {3, 4} > py> d1.keys() ^ d2.keys() # keys not in both > {1, 2, 5, 6} > py> d1.keys() - d2.keys() # keys only in d1 > {1, 2} > py> d2.keys() - d1.keys() # keys only in d2 > {5, 6} I know this is completely off-topic, but I really must thank you for showing that neat trick. I didn't know set()'s operators &, ^, - were overloaded (and always used difference/intersection, etc). That's really, really neat. Thanks again, Joe -- >> Wo hattest Du das Beben nochmal GENAU vorhergesagt? > Zumindest nicht ?ffentlich! Ah, der neueste und bis heute genialste Streich unsere gro?en Kosmologen: Die Geheim-Vorhersage. - Karl Kaos ?ber R?diger Thomas in dsa From ethan at stoneleaf.us Thu Jul 25 09:57:15 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 25 Jul 2013 06:57:15 -0700 Subject: Python 3: dict & dict.keys() In-Reply-To: <51f0bc34$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <51EF2AD8.3080105@stoneleaf.us> <51f0bc34$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51F12EBB.8070008@stoneleaf.us> On 07/24/2013 10:48 PM, Steven D'Aprano wrote: > On Wed, 24 Jul 2013 08:57:11 -0700, Ethan Furman wrote: > >> My point is that in 2.x .keys() did something different from the dict, >> while in 3.x it appears to me that they are the same. > > Then you aren't looking very closely. Actually, I am. That's why I started this thread. Thank you for the insights. -- ~Ethan~ From stefan_ml at behnel.de Wed Jul 24 13:23:47 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 24 Jul 2013 19:23:47 +0200 Subject: Python 3: dict & dict.keys() In-Reply-To: References: <51EF2AD8.3080105@stoneleaf.us> Message-ID: Peter Otten, 24.07.2013 08:23: > Ethan Furman wrote: >> So, my question boils down to: in Python 3 how is dict.keys() different >> from dict? What are the use cases? > > I just grepped through /usr/lib/python3, and could not identify a single > line where some_object.keys() wasn't either wrapped in a list (or set, > sorted, max) call, or iterated over. In case that directory mainly consists of the standard library, then you shouldn't forget that most of the code in there predates Python 3 by ages and was only adapted to work with the new syntax/semantics, not rewritten in a "better" way. Even if it's not just the stdlib, that argument still holds. There is still fairly little code out there that was specifically written for Py2.6+, as opposed to just being adapted. > To me it looks like views are a solution waiting for a problem. They reduce the API overhead. Previously, you needed values() and itervalues(), with values() being not more than a special case of what itervalues() provides anyway. Now it's just one method that gives you everything. It simply has corrected the tradeoff from two special purpose APIs to one general purpose API, that's all. Stefan From ethan at stoneleaf.us Wed Jul 24 14:31:58 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 24 Jul 2013 11:31:58 -0700 Subject: Python 3: dict & dict.keys() In-Reply-To: References: <51EF2AD8.3080105@stoneleaf.us> Message-ID: <51F01D9E.2030903@stoneleaf.us> On 07/24/2013 10:23 AM, Stefan Behnel wrote: > Peter Otten, 24.07.2013 08:23: >> Ethan Furman wrote: >>> >>> So, my question boils down to: in Python 3 how is dict.keys() different >>> from dict? What are the use cases? >> >> To me it looks like views are a solution waiting for a problem. > > They reduce the API overhead. Previously, you needed values() and > itervalues(), with values() being not more than a special case of what > itervalues() provides anyway. Now it's just one method that gives you > everything. It simply has corrected the tradeoff from two special purpose > APIs to one general purpose API, that's all. I started this thread for two reasons: 1) Increase awareness that using `list(dict)` is a cross-version replacement for `dict.keys()` 2) Hopefully learn something about when a view is useful. So far #2 is pretty much a failure. Only one use-case so far (and it feels pretty rare). But hey, I have learned that while some set operations are allowed (&, ^, |, .isdisjoint()), others are not (.remove(), .discard(), .union(), etc.). The old .keys(), .values(), and .items() (and their .iter...() variations) did something commonly useful. Of what common use are these views? -- ~Ethan~ From stefan_ml at behnel.de Wed Jul 24 15:59:31 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 24 Jul 2013 21:59:31 +0200 Subject: Python 3: dict & dict.keys() In-Reply-To: <51F01D9E.2030903@stoneleaf.us> References: <51EF2AD8.3080105@stoneleaf.us> <51F01D9E.2030903@stoneleaf.us> Message-ID: Ethan Furman, 24.07.2013 20:31: > On 07/24/2013 10:23 AM, Stefan Behnel wrote: >> Peter Otten, 24.07.2013 08:23: >>> Ethan Furman wrote: >>>> >>>> So, my question boils down to: in Python 3 how is dict.keys() different >>>> from dict? What are the use cases? >>> >>> To me it looks like views are a solution waiting for a problem. >> >> They reduce the API overhead. Previously, you needed values() and >> itervalues(), with values() being not more than a special case of what >> itervalues() provides anyway. Now it's just one method that gives you >> everything. It simply has corrected the tradeoff from two special purpose >> APIs to one general purpose API, that's all. > > I started this thread for two reasons: > > 1) Increase awareness that using `list(dict)` is a cross-version > replacement for `dict.keys()` > > 2) Hopefully learn something about when a view is useful. > > So far #2 is pretty much a failure. I think the question is: how else would you implement an interface that doesn't restrict itself to returning a list? I mean, previously, the following was totally inefficient in terms of memory: value in d.values() It now avoids creating an intermediate list copy of the values, thus running with no additional memory overhead (well, a constant, ok, but definitely not linear) and keeps users from resorting to the much more unfriendly for v in d.itervalues(): if v == value: return True else: return False in order to achieve the same thing. You can now even efficiently do this for items, i.e. (key, value) in d.items() That's equivalent to "d[key] == value", but uses a different protocol, meaning that you don't have to make a copy of the dict items in order to pass it into something that works on a set or iterable of 2-tuples (which is a way more generic interface than requiring a dict as input). These things chain much more cleanly now, without first having to explain the difference between items() and iteritems() and when to use which. It's all about replacing the old copy-to-list interface by something that is efficiently processable step by step. All of this started back when iterators became a part of the language, then generators, and now dict views. They may not be the hugest feature ever, but they definitely fit into the language much better and much more cleanly than the old copy-to-list way. Ask yourself, if they had been there in Python 1.x, would you even have thought about making the iter*() methods a part of the language? Would you really have wanted a shorter way to create a list of dict values than list(d.values())? Stefan From ethan at stoneleaf.us Wed Jul 24 16:16:54 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 24 Jul 2013 13:16:54 -0700 Subject: Python 3: dict & dict.keys() In-Reply-To: References: <51EF2AD8.3080105@stoneleaf.us> <51F01D9E.2030903@stoneleaf.us> Message-ID: <51F03636.9050501@stoneleaf.us> On 07/24/2013 12:59 PM, Stefan Behnel wrote: > > I think the question is: how else would you implement an interface that > doesn't restrict itself to returning a list? I mean, previously, the > following was totally inefficient in terms of memory: > > value in d.values() > > It now avoids creating an intermediate list copy of the values, thus > running with no additional memory overhead (well, a constant, ok, but > definitely not linear) and keeps users from resorting to the much more > unfriendly > > for v in d.itervalues(): > if v == value: > return True > else: > return False > > in order to achieve the same thing. You can now even efficiently do this > for items, i.e. > > (key, value) in d.items() > > That's equivalent to "d[key] == value", but uses a different protocol, > meaning that you don't have to make a copy of the dict items in order to > pass it into something that works on a set or iterable of 2-tuples (which > is a way more generic interface than requiring a dict as input). These > things chain much more cleanly now, without first having to explain the > difference between items() and iteritems() and when to use which. > > It's all about replacing the old copy-to-list interface by something that > is efficiently processable step by step. All of this started back when > iterators became a part of the language, then generators, and now dict > views. They may not be the hugest feature ever, but they definitely fit > into the language much better and much more cleanly than the old > copy-to-list way. Thank you. :) -- ~Ethan~ From ramit.prasad at jpmorgan.com Wed Jul 24 16:34:28 2013 From: ramit.prasad at jpmorgan.com (Prasad, Ramit) Date: Wed, 24 Jul 2013 20:34:28 +0000 Subject: Python 3: dict & dict.keys() In-Reply-To: References: <51EF2AD8.3080105@stoneleaf.us> <51F01D9E.2030903@stoneleaf.us> Message-ID: <5B80DD153D7D744689F57F4FB69AF474185C131B@SCACMX008.exchad.jpmchase.net> Stefan Behnel wrote: > Ethan Furman, 24.07.2013 20:31: > > On 07/24/2013 10:23 AM, Stefan Behnel wrote: > >> Peter Otten, 24.07.2013 08:23: > >>> Ethan Furman wrote: > >>>> > >>>> So, my question boils down to: in Python 3 how is dict.keys() different > >>>> from dict? What are the use cases? > >>> > >>> To me it looks like views are a solution waiting for a problem. > >> > >> They reduce the API overhead. Previously, you needed values() and > >> itervalues(), with values() being not more than a special case of what > >> itervalues() provides anyway. Now it's just one method that gives you > >> everything. It simply has corrected the tradeoff from two special purpose > >> APIs to one general purpose API, that's all. > > > > I started this thread for two reasons: > > > > 1) Increase awareness that using `list(dict)` is a cross-version > > replacement for `dict.keys()` > > > > 2) Hopefully learn something about when a view is useful. > > > > So far #2 is pretty much a failure. > > I think the question is: how else would you implement an interface that > doesn't restrict itself to returning a list? I mean, previously, the > following was totally inefficient in terms of memory: > > value in d.values() > > It now avoids creating an intermediate list copy of the values, thus > running with no additional memory overhead (well, a constant, ok, but > definitely not linear) and keeps users from resorting to the much more > unfriendly > > for v in d.itervalues(): > if v == value: > return True > else: > return False > > in order to achieve the same thing. You can now even efficiently do this > for items, i.e. > > (key, value) in d.items() > > That's equivalent to "d[key] == value", but uses a different protocol, > meaning that you don't have to make a copy of the dict items in order to > pass it into something that works on a set or iterable of 2-tuples (which > is a way more generic interface than requiring a dict as input). These > things chain much more cleanly now, without first having to explain the > difference between items() and iteritems() and when to use which. > > It's all about replacing the old copy-to-list interface by something that > is efficiently processable step by step. All of this started back when > iterators became a part of the language, then generators, and now dict > views. They may not be the hugest feature ever, but they definitely fit > into the language much better and much more cleanly than the old > copy-to-list way. > > Ask yourself, if they had been there in Python 1.x, would you even have > thought about making the iter*() methods a part of the language? Would you > really have wanted a shorter way to create a list of dict values than > list(d.values())? > > Stefan > I am still not clear on the advantage of views vs. iterators. What makes d.viewkeys() better than d.iterkeys()? Why did they decide not to rename d.iterkeys() to d.keys() and instead use d.viewkeys()? Is the iteration over a set operation on keys really that common a use case? Ramit This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email. From tjreedy at udel.edu Wed Jul 24 19:45:37 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 24 Jul 2013 19:45:37 -0400 Subject: Python 3: dict & dict.keys() In-Reply-To: <5B80DD153D7D744689F57F4FB69AF474185C131B@SCACMX008.exchad.jpmchase.net> References: <51EF2AD8.3080105@stoneleaf.us> <51F01D9E.2030903@stoneleaf.us> <5B80DD153D7D744689F57F4FB69AF474185C131B@SCACMX008.exchad.jpmchase.net> Message-ID: On 7/24/2013 4:34 PM, Prasad, Ramit wrote: > I am still not clear on the advantage of views vs. iterators. A1: Views are iterables that can be iterated more than once. Therefore, they can be passed to a function that re-iterates its inputs, or to multiple functions. They support 'x in view' as efficiently as possible. Think about how you would write the non-view equivalent of '(0,None) in somedict.views())'. When set-like, views support some set operations. For .keys, which are always set-like, these operations are easy to implement as dicts are based on a hashed array of keys. Q2: What is the advantage of views vs. lists? A2: They do not take up space that is not needed. They can be converted to lists, to get all the features of lists, but not vice versa. > What makes d.viewkeys() better than d.iterkeys()? Why did they decide > not to rename d.iterkeys() to d.keys() and instead use d.viewkeys()? This is historically the wrong way to phrase the question. The 2.7 .viewxyz methods were *not* used to make the 3.x .xyz methods. It was the other way around. 3.0 came out with view methods replacing both list and iter methods just after 2.6, after a couple of years of design, and a year and a half before 2.7. The view methods were backported from 3.1 to 2.7, with 'view' added to the name to avoid name conflicts, to make it easier to write code that would either run on both 2.7 and 3.x or be converted with 2to3. A better question is: 'When 3.0 was designed, why were views invented for the .xyz methods rather than just renaming the .iterxyz methods. The advantages given above are the answer. View methods replace both list and iterator methods and are more flexible than either and directly or indirectly have all the advantages of both. My question is why some people are fussing so much because Python developers gave them one thing that is better than either of the two things it replaces? The mis-phrased question above illustrates why people new to Python should use the latest 3.x and ignore 2.x unless they must use 2.x libraries. 2.7 has all the old stuff, for back compatibility, and as much of the new stuff in 3.1 as seemed sensible, for forward compatibility. Thus it has lots of confusing duplication, and in this case, triplication -- Terry Jan Reedy From ramit.prasad at jpmorgan.com Thu Jul 25 12:11:47 2013 From: ramit.prasad at jpmorgan.com (Prasad, Ramit) Date: Thu, 25 Jul 2013 16:11:47 +0000 Subject: Python 3: dict & dict.keys() In-Reply-To: References: <51EF2AD8.3080105@stoneleaf.us> <51F01D9E.2030903@stoneleaf.us> <5B80DD153D7D744689F57F4FB69AF474185C131B@SCACMX008.exchad.jpmchase.net> Message-ID: <5B80DD153D7D744689F57F4FB69AF474185CA82D@SCACMX008.exchad.jpmchase.net> Terry Reedy wrote: > > On 7/24/2013 4:34 PM, Prasad, Ramit wrote: > > > I am still not clear on the advantage of views vs. iterators. > > A1: Views are iterables that can be iterated more than once. Therefore, > they can be passed to a function that re-iterates its inputs, or to > multiple functions. They support 'x in view' as efficiently as possible. > Think about how you would write the non-view equivalent of '(0,None) in > somedict.views())'. When set-like, views support some set operations. > For .keys, which are always set-like, these operations are easy to > implement as dicts are based on a hashed array of keys. Hmm, that is a change that makes some sense to me. Does the view get updated when dictionary changes or is a new view needed? I assume the latter. > > Q2: What is the advantage of views vs. lists? > > A2: They do not take up space that is not needed. They can be converted > to lists, to get all the features of lists, but not vice versa. > > > What makes d.viewkeys() better than d.iterkeys()? Why did they decide > > not to rename d.iterkeys() to d.keys() and instead use d.viewkeys()? > > This is historically the wrong way to phrase the question. The 2.7 > .viewxyz methods were *not* used to make the 3.x .xyz methods. It was > the other way around. 3.0 came out with view methods replacing both list > and iter methods just after 2.6, after a couple of years of design, and > a year and a half before 2.7. The view methods were backported from 3.1 > to 2.7, with 'view' added to the name to avoid name conflicts, to make > it easier to write code that would either run on both 2.7 and 3.x or be > converted with 2to3. > > A better question is: 'When 3.0 was designed, why were views invented > for the .xyz methods rather than just renaming the .iterxyz methods. The > advantages given above are the answer. View methods replace both list > and iterator methods and are more flexible than either and directly or > indirectly have all the advantages of both. > > My question is why some people are fussing so much because Python > developers gave them one thing that is better than either of the two > things it replaces? I personally am not "fussing" as existing functionality was preserved (and improved). I just was not clear on the difference. Thanks for all the detail and context. > > The mis-phrased question above illustrates why people new to Python > should use the latest 3.x and ignore 2.x unless they must use 2.x > libraries. 2.7 has all the old stuff, for back compatibility, and as > much of the new stuff in 3.1 as seemed sensible, for forward > compatibility. Thus it has lots of confusing duplication, and in this > case, triplication I work with 2.6 so no choice there... :) > > -- > Terry Jan Reedy > > -- > http://mail.python.org/mailman/listinfo/python-list This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email. From ethan at stoneleaf.us Thu Jul 25 12:21:02 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 25 Jul 2013 09:21:02 -0700 Subject: Python 3: dict & dict.keys() In-Reply-To: <5B80DD153D7D744689F57F4FB69AF474185CA82D@SCACMX008.exchad.jpmchase.net> References: <51EF2AD8.3080105@stoneleaf.us> <51F01D9E.2030903@stoneleaf.us> <5B80DD153D7D744689F57F4FB69AF474185C131B@SCACMX008.exchad.jpmchase.net> <5B80DD153D7D744689F57F4FB69AF474185CA82D@SCACMX008.exchad.jpmchase.net> Message-ID: <51F1506E.2020800@stoneleaf.us> On 07/25/2013 09:11 AM, Prasad, Ramit wrote: > Terry Reedy wrote: >> >> On 7/24/2013 4:34 PM, Prasad, Ramit wrote: >> >>> I am still not clear on the advantage of views vs. iterators. >> >> A1: Views are iterables that can be iterated more than once. Therefore, >> they can be passed to a function that re-iterates its inputs, or to >> multiple functions. They support 'x in view' as efficiently as possible. >> Think about how you would write the non-view equivalent of '(0,None) in >> somedict.views())'. When set-like, views support some set operations. >> For .keys, which are always set-like, these operations are easy to >> implement as dicts are based on a hashed array of keys. > > Hmm, that is a change that makes some sense to me. Does the view > get updated when dictionary changes or is a new view needed? I > assume the latter. Nope, the former. That is a big advantage that the views have over concrete lists: they show the /current/ state, and so are always up-do-date. -- ~Ethan~ From tjreedy at udel.edu Thu Jul 25 14:37:14 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 25 Jul 2013 14:37:14 -0400 Subject: Python 3: dict & dict.keys() In-Reply-To: <51F1506E.2020800@stoneleaf.us> References: <51EF2AD8.3080105@stoneleaf.us> <51F01D9E.2030903@stoneleaf.us> <5B80DD153D7D744689F57F4FB69AF474185C131B@SCACMX008.exchad.jpmchase.net> <5B80DD153D7D744689F57F4FB69AF474185CA82D@SCACMX008.exchad.jpmchase.net> <51F1506E.2020800@stoneleaf.us> Message-ID: On 7/25/2013 12:21 PM, Ethan Furman wrote: > On 07/25/2013 09:11 AM, Prasad, Ramit wrote: >> Hmm, that is a change that makes some sense to me. Does the view >> get updated when dictionary changes or is a new view needed? I >> assume the latter. > > Nope, the former. That is a big advantage that the views have over > concrete lists: they show the /current/ state, and so are always > up-do-date. I think 'view' is generally used in CS to mean a live view, as opposed to a snapshot. Memoryviews in 3.x are also live views. Dictionary views are read-only. I believe memoryviews can be read-write if allowed by the object being viewed. Python slices are snapshots. It has been proposed that they should be views to avoid copying memory, but that has been rejected since views necessarily keep the underlying object alive. Instead, applications can define the views they need. (They might, for instance, allow multiple slices in a view, as tk Text widgets do.) -- Terry Jan Reedy From ethan at stoneleaf.us Wed Jul 24 19:22:11 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 24 Jul 2013 16:22:11 -0700 Subject: Python 3: dict & dict.keys() In-Reply-To: <5B80DD153D7D744689F57F4FB69AF474185C131B@SCACMX008.exchad.jpmchase.net> References: <51EF2AD8.3080105@stoneleaf.us> <51F01D9E.2030903@stoneleaf.us> <5B80DD153D7D744689F57F4FB69AF474185C131B@SCACMX008.exchad.jpmchase.net> Message-ID: <51F061A3.6090505@stoneleaf.us> On 07/24/2013 01:34 PM, Prasad, Ramit wrote: > > I am still not clear on the advantage of views vs. iterators. What > makes d.viewkeys() better than d.iterkeys()? Why did they decide > not to rename d.iterkeys() to d.keys() and instead use d.viewkeys()? > Is the iteration over a set operation on keys really that common a > use case? From a practical standpoint, iterkeys() is a one-shot deal, while viewkeys() can be iterated over multiple times: --> d = {1: 'one', 2: 'two', 3: 'three'} --> di = d.iterkeys() --> list(di) [1, 2, 3] --> list(di) [] --> dv = d.viewkeys() --> list(dv) [1, 2, 3] --> list(dv) [1, 2, 3] And views are not sets -- they just support a couple set-like operations. -- ~Ethan~ From wuwei23 at gmail.com Thu Jul 25 02:01:25 2013 From: wuwei23 at gmail.com (alex23) Date: Thu, 25 Jul 2013 16:01:25 +1000 Subject: Python 3: dict & dict.keys() In-Reply-To: References: <51EF2AD8.3080105@stoneleaf.us> Message-ID: On 25/07/2013 4:31 AM, Ethan Furman wrote: > 2) Hopefully learn something about when a view is useful. I haven't seeen this mentioned - forgive me if it's a repeat - but views are constant references to whichever set they represent. Python 2.7: >>> dd = dict(a=1,b=2,c=3) >>> keys = dd.keys() >>> 'a' in keys True >>> dd['d'] = 4 >>> 'd' in keys False Python 3.3: >>> dd = dict(a=1,b=2,c=3) >>> keys = dd.keys() >>> 'a' in keys True >>> dd['d'] = 4 >>> 'd' in keys True If part of my code is only interested in what keys or values are present, it doesn't need to be given a reference to the full dictionary, just to whichever view it cares about. From ethan at stoneleaf.us Thu Jul 25 09:47:08 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 25 Jul 2013 06:47:08 -0700 Subject: Python 3: dict & dict.keys() In-Reply-To: References: <51EF2AD8.3080105@stoneleaf.us> Message-ID: <51F12C5C.6030305@stoneleaf.us> On 07/24/2013 11:01 PM, alex23 wrote: > On 25/07/2013 4:31 AM, Ethan Furman wrote: >> 2) Hopefully learn something about when a view is useful. > > I haven't seeen this mentioned - forgive me if it's a repeat - but views are constant references to whichever set they > represent. > > Python 2.7: > >>>> dd = dict(a=1,b=2,c=3) >>>> keys = dd.keys() >>>> 'a' in keys > True >>>> dd['d'] = 4 >>>> 'd' in keys > False > > Python 3.3: >>>> dd = dict(a=1,b=2,c=3) >>>> keys = dd.keys() >>>> 'a' in keys > True >>>> dd['d'] = 4 >>>> 'd' in keys > True > > If part of my code is only interested in what keys or values are present, it doesn't need to be given a reference to the > full dictionary, just to whichever view it cares about. In these cases why is a view better than just using the dict? Is it a safety so the you don't accidentally modify the dict? -- ~Ethan~ From steve+comp.lang.python at pearwood.info Thu Jul 25 03:04:38 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 25 Jul 2013 07:04:38 GMT Subject: Python 3: dict & dict.keys() References: <51EF2AD8.3080105@stoneleaf.us> Message-ID: <51f0ce06$0$29971$c3e8da3$5496439d@news.astraweb.com> On Wed, 24 Jul 2013 11:31:58 -0700, Ethan Furman wrote: > On 07/24/2013 10:23 AM, Stefan Behnel wrote: >> Peter Otten, 24.07.2013 08:23: >>> Ethan Furman wrote: >>>> >>>> So, my question boils down to: in Python 3 how is dict.keys() >>>> different from dict? What are the use cases? >>> >>> To me it looks like views are a solution waiting for a problem. >> >> They reduce the API overhead. Previously, you needed values() and >> itervalues(), with values() being not more than a special case of what >> itervalues() provides anyway. Now it's just one method that gives you >> everything. It simply has corrected the tradeoff from two special >> purpose APIs to one general purpose API, that's all. > > I started this thread for two reasons: > > 1) Increase awareness that using `list(dict)` is a cross-version > replacement for `dict.keys()` > > 2) Hopefully learn something about when a view is useful. > > So far #2 is pretty much a failure. I don't think so. - viewkeys() can be used as a drop-in replacement for iterkeys(), provided you remember not to iterate over it a second time. Doing so actually iterates over the view, instead of failing as with the iterator. If you actually want a one-shot iterator, call iter() on the view. - viewkeys() can be used as a drop-in replacement for Python2 keys(), provided you only iterate over it once. If you want an actual list, call list() on the view. - Views support set methods which don't modify the set. If there is a non- modifying set method which is not supported, it probably should be, and somebody should raise an enhancement request in the bug tracker. If you want an actual independent set you can modify without changing the dict, call set() (or frozenset) on the view. - Views support efficient (O(1) in the case of keys) membership testing, which neither iterkeys() nor Python2 keys() does. - Views support live, read-only access to dict keys and values. > Only one use-case so far (and it > feels pretty rare). Iterating over a dict's values or items is not rare. Using a view is better than making a list-copy of the dict and iterating over the list, because it avoids copying. For one-shot iteration, there's no benefit of a view over an iterator, or vice versa, but views are useful for more than just one-shot iteration. > But hey, I have learned that while some set > operations are allowed (&, ^, |, .isdisjoint()), others are not > (.remove(), .discard(), .union(), etc.). > > The old .keys(), .values(), and .items() (and their .iter...() > variations) did something commonly useful. Of what common use are these > views? Everything that dict iteration does, dict views do, and more. So if iterkeys() is useful, then so is viewkeys(). Had viewkeys come first, iterkeys would never have been invented. Making an actual list copy of the keys (values, items) is useful, but it's not useful enough to dedicate a method (three methods) for it. Just call list() on the view (or, in the case of keys, directly on the dict). -- Steven From rosuav at gmail.com Thu Jul 25 04:02:11 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 18:02:11 +1000 Subject: Python 3: dict & dict.keys() In-Reply-To: <51f0ce06$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <51EF2AD8.3080105@stoneleaf.us> <51f0ce06$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, Jul 25, 2013 at 5:04 PM, Steven D'Aprano wrote: > - Views support efficient (O(1) in the case of keys) membership testing, > which neither iterkeys() nor Python2 keys() does. To save me the trouble and potential error of digging through the source code: What's the complexity of membership testing on values/items? Since you're calling it "efficient" it must be better than O(n) which the list form would be, yet it isn't O(1) or you wouldn't have qualified "in the case of keys". Does this mean membership testing of the values and items views is O(log n) in some way, eg a binary search? ChrisA From __peter__ at web.de Thu Jul 25 04:13:47 2013 From: __peter__ at web.de (Peter Otten) Date: Thu, 25 Jul 2013 10:13:47 +0200 Subject: Python 3: dict & dict.keys() References: <51EF2AD8.3080105@stoneleaf.us> <51f0ce06$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: Chris Angelico wrote: > On Thu, Jul 25, 2013 at 5:04 PM, Steven D'Aprano > wrote: >> - Views support efficient (O(1) in the case of keys) membership testing, >> which neither iterkeys() nor Python2 keys() does. > > To save me the trouble and potential error of digging through the > source code: What's the complexity of membership testing on > values/items? Since you're calling it "efficient" it must be better > than O(n) which the list form would be, yet it isn't O(1) or you > wouldn't have qualified "in the case of keys". Does this mean > membership testing of the values and items views is O(log n) in some > way, eg a binary search? keys() and items() is O(1); both look up the key in the dictionary and items() then proceeds to compare the value. values() is O(n). From ian.g.kelly at gmail.com Thu Jul 25 11:53:33 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Thu, 25 Jul 2013 09:53:33 -0600 Subject: Python 3: dict & dict.keys() In-Reply-To: References: <51EF2AD8.3080105@stoneleaf.us> <51f0ce06$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Thu, Jul 25, 2013 at 2:13 AM, Peter Otten <__peter__ at web.de> wrote: > Chris Angelico wrote: > >> On Thu, Jul 25, 2013 at 5:04 PM, Steven D'Aprano >> wrote: >>> - Views support efficient (O(1) in the case of keys) membership testing, >>> which neither iterkeys() nor Python2 keys() does. >> >> To save me the trouble and potential error of digging through the >> source code: What's the complexity of membership testing on >> values/items? Since you're calling it "efficient" it must be better >> than O(n) which the list form would be, yet it isn't O(1) or you >> wouldn't have qualified "in the case of keys". Does this mean >> membership testing of the values and items views is O(log n) in some >> way, eg a binary search? > > keys() and items() is O(1); both look up the key in the dictionary and > items() then proceeds to compare the value. values() is O(n). 3.x values() is O(n) but avoids the unnecessary step of copying all the values in the dict that you get when performing the same operation using 2.x values(). Hence, although the asymptotic complexity is the same, it's still more efficient. From __peter__ at web.de Thu Jul 25 12:25:19 2013 From: __peter__ at web.de (Peter Otten) Date: Thu, 25 Jul 2013 18:25:19 +0200 Subject: Python 3: dict & dict.keys() References: <51EF2AD8.3080105@stoneleaf.us> <51f0ce06$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: Ian Kelly wrote: > On Thu, Jul 25, 2013 at 2:13 AM, Peter Otten <__peter__ at web.de> wrote: >> Chris Angelico wrote: >> >>> On Thu, Jul 25, 2013 at 5:04 PM, Steven D'Aprano >>> wrote: >>>> - Views support efficient (O(1) in the case of keys) membership >>>> testing, which neither iterkeys() nor Python2 keys() does. >>> >>> To save me the trouble and potential error of digging through the >>> source code: What's the complexity of membership testing on >>> values/items? Since you're calling it "efficient" it must be better >>> than O(n) which the list form would be, yet it isn't O(1) or you >>> wouldn't have qualified "in the case of keys". Does this mean >>> membership testing of the values and items views is O(log n) in some >>> way, eg a binary search? >> >> keys() and items() is O(1); both look up the key in the dictionary and >> items() then proceeds to compare the value. values() is O(n). > > 3.x values() is O(n) but avoids the unnecessary step of copying all the > values in the dict that you get when performing the same operation > using 2.x values(). Hence, although the asymptotic complexity is the > same, it's still more efficient. In Python 2 the prudent pythonista used itervalues() to avoid unnecessary intermediate list... From neilc at norwich.edu Wed Jul 24 08:54:36 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 24 Jul 2013 12:54:36 GMT Subject: Python 3: dict & dict.keys() References: <51EF2AD8.3080105@stoneleaf.us> Message-ID: On 2013-07-24, Peter Otten <__peter__ at web.de> wrote: >> So, my question boils down to: in Python 3 how is dict.keys() >> different from dict? What are the use cases? > > I just grepped through /usr/lib/python3, and could not identify > a single line where some_object.keys() wasn't either wrapped in > a list (or set, sorted, max) call, or iterated over. > > To me it looks like views are a solution waiting for a problem. Here's a case of using keys as a set-like view from my own "production" code (i.e., I used it once for one important job): seen = set() students = {} dates = [] for fname in sorted(glob.glob("currentterm201320?.csv")): print(fname, end="\n\t") date = get_date(fname) dates.append(date) term = fname[-11:-4] r = reg.registration(term, path=".") regs = r.keys() for alt_id in regs & seen: students[alt_id].append(r[alt_id]) for alt_id in seen - regs: students[alt_id].append(None) for alt_id in regs - seen: students[alt_id] = [None]*(len(dates)-1) + [r[alt_id]] seen.add(alt_id) It was a very nice way to to do three different things depending on the student sin the set I was working with, compared to a registration list: Granted the line was originally "regs = set(regs.keys())" before it occurred to me that it sucked to take what must be equivalent to a set, convert to a list, and then back to set again. Thanks to the set-like view of dict.keys it worked just like one might hope. Looking at it again "seen" might be a redundant parallel version of students.keys(). -- Neil Cerutti From oscar.j.benjamin at gmail.com Wed Jul 24 12:32:01 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 24 Jul 2013 17:32:01 +0100 Subject: Python 3: dict & dict.keys() Message-ID: On Jul 24, 2013 2:27 PM, "Peter Otten" <__peter__ at web.de> wrote: > > Oscar Benjamin wrote: > > > On Jul 24, 2013 7:25 AM, "Peter Otten" <__peter__ at web.de> wrote: > >> > >> Ethan Furman wrote: > >> > >> > So, my question boils down to: in Python 3 how is dict.keys() > >> > different > >> > from dict? What are the use cases? > >> > >> I just grepped through /usr/lib/python3, and could not identify a single > >> line where some_object.keys() wasn't either wrapped in a list (or set, > >> sorted, max) call, or iterated over. > >> > >> To me it looks like views are a solution waiting for a problem. > > > > What do you mean? Why would you want to create a temporary list just to > > iterate over it explicitly or implicitly (set, sorted, max,...)? > > I mean I don't understand the necessity of views when all actual usecases > need iterators. The 2.x iterkeys()/iteritems()/itervalues() methods didn't > create lists either. Oh, okay. I see what you mean. > > Do you have 2.x code lying around where you get a significant advantage by > picking some_dict.viewkeys() over some_dict.iterkeys()? No. I don't think I've ever used viewkeys. I noticed it once, didn't see an immediate use and forgot about it but... > I could construct > one > > >>> d = dict(a=1, b=2, c=3) > >>> e = dict(b=4, c=5, d=6) > >>> d.viewkeys() & e.viewkeys() > set(['c', 'b']) that might be useful. > > but have not seen it in the wild. > My guess is that most non-hardcore users don't even know about viewkeys(). > By the way, my favourite idiom to iterate over the keys in both Python 2 and > 3 is -- for example -- max(some_dict) rather than > max(some_dict.whateverkeys()). Agreed. Earlier I saw that I had list(some_dict) in some code. Not sure why but maybe because it's the same in Python 2 and 3. Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Mon Jul 22 17:33:47 2013 From: larry at hastings.org (Larry Hastings) Date: Mon, 22 Jul 2013 14:33:47 -0700 Subject: Contact information for Jim Hugunin? Message-ID: <51EDA53B.9050109@hastings.org> Does anybody have an email address (or anything, really) for Jim Hugunin? He left Google in May and appears to have dropped off the face of the internet. Please email me privately. I swear I will use the information only for good and never for evil, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From marduk at letterboxes.org Wed Jul 24 06:46:52 2013 From: marduk at letterboxes.org (Albert Hopkins) Date: Wed, 24 Jul 2013 06:46:52 -0400 Subject: Contact information for Jim Hugunin? In-Reply-To: <51EDA53B.9050109@hastings.org> References: <51EDA53B.9050109@hastings.org> Message-ID: <1374662812.25545.9223372036856487529.4F1201B5@webmail.messagingengine.com> On Mon, Jul 22, 2013, at 05:33 PM, Larry Hastings wrote: > > > Does anybody have an email address (or anything, really) for Jim > Hugunin? He left Google in May and appears to have dropped off the face > of the internet. Please email me privately. > > I swear I will use the information only for good and never for evil, Is that your definition of "good" and "evil" or mine? From maus at mail.com Sat Jul 27 05:33:11 2013 From: maus at mail.com (greymausg) Date: 27 Jul 2013 09:33:11 GMT Subject: Contact information for Jim Hugunin? References: <51EDA53B.9050109@hastings.org> Message-ID: On 2013-07-24, Albert Hopkins wrote: > On Mon, Jul 22, 2013, at 05:33 PM, Larry Hastings wrote: >> >> >> Does anybody have an email address (or anything, really) for Jim >> Hugunin? He left Google in May and appears to have dropped off the face >> of the internet. Please email me privately. >> >> I swear I will use the information only for good and never for evil, > > Is that your definition of "good" and "evil" or mine? After all, Google has its own definition! -- maus . . ... From navnathgadakh at gmail.com Wed Jul 24 03:47:08 2013 From: navnathgadakh at gmail.com (navnathgadakh at gmail.com) Date: Wed, 24 Jul 2013 00:47:08 -0700 (PDT) Subject: i want to run user specific cronjobs so where to specify user in ubuntu? Message-ID: <3f986e15-1ab6-4397-970b-b7cdb85d7291@googlegroups.com> From rosuav at gmail.com Wed Jul 24 03:54:16 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 24 Jul 2013 17:54:16 +1000 Subject: i want to run user specific cronjobs so where to specify user in ubuntu? In-Reply-To: <3f986e15-1ab6-4397-970b-b7cdb85d7291@googlegroups.com> References: <3f986e15-1ab6-4397-970b-b7cdb85d7291@googlegroups.com> Message-ID: On Wed, Jul 24, 2013 at 5:47 PM, wrote: > > -- > http://mail.python.org/mailman/listinfo/python-list Not a Python question. I'm sure Google can help you with this one - just take three words from your question, 'user specific cronjobs', and you'll get plenty of advice. ChrisA From mmanns at gmx.net Tue Jul 23 13:07:42 2013 From: mmanns at gmx.net (Martin Manns) Date: Tue, 23 Jul 2013 19:07:42 +0200 Subject: [ANN] pyspread 0.2.4 Message-ID: ============== pyspread 0.2.4 ============== Pyspread 0.2.4 is released. Besides Linux, the new version also runs on Windows (Windows 7 64bit and Windows XP 32bit tested). About pyspread ============== Pyspread is a non-traditional spreadsheet application that is based on and written in the programming language Python. The goal of pyspread is to be the most pythonic spreadsheet application. Pyspread is free software. It is released under the GPL v3. Project website: http://manns.github.com/pyspread/ What is new in 0.2.4 ==================== + Windows compatibility + More charts and chart options (box plots, histograms, pie charts) + Export of matplotlib charts to svg, png, pdf, eps and ps + Import from UTF-8 encoded csv files (no other encoding, yet) + More than 64k characters supported in one cell + Insertion and deletion of rows and columns now affect only the current table + High quality in-cell bitmaps images (with transparency) supported + Iterable data can be pasted into multiple cells via Paste As... + Pyspread can now be started from other Python applications + Partly localized in Nowegian Nynorsk and Bokmaal Known issues ============ + Selection mode is disabled in Windows. + "Find & Replace All" is slow in grids with much data unless pyspread is minimized. + Paste As... does not work if dates are present in the data structure. + Sometimes, pressing redo when there is nothing left to redo has undesired effects such as redoing an operation again. + When updating from a non-release version (from git), the file ~/.pyspreadrc (in Windows pyspread's registry entry) may have to be deleted. Enjoy Martin From peins0242 at gmail.com Wed Jul 24 22:03:07 2013 From: peins0242 at gmail.com (peins0242 at gmail.com) Date: Wed, 24 Jul 2013 19:03:07 -0700 (PDT) Subject: I have a little problem with Django when I am trying to create a new app Message-ID: <460c20e0-b011-47bb-ad95-c24da160315e@googlegroups.com> Hello everyone I'm watching a tutorial on how to create a project on Django... django-admin.py startproject carabali when I run this code on terminal.. happens : http://nsae01.casimages.net/img/2013/07/25/130725021220676239.png There's a mistake or something but I can't figure out, help me please.... From rosuav at gmail.com Thu Jul 25 00:12:29 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 14:12:29 +1000 Subject: I have a little problem with Django when I am trying to create a new app In-Reply-To: <460c20e0-b011-47bb-ad95-c24da160315e@googlegroups.com> References: <460c20e0-b011-47bb-ad95-c24da160315e@googlegroups.com> Message-ID: On Thu, Jul 25, 2013 at 12:03 PM, wrote: > Hello everyone I'm watching a tutorial on how to create a project on Django... > > django-admin.py startproject carabali > > when I run this code on terminal.. happens : > > > http://nsae01.casimages.net/img/2013/07/25/130725021220676239.png > > > There's a mistake or something but I can't figure out, help me please.... It's saying that it can't find the zlib module. What Linux distribution is this? You may need to 'apt-get install' / 'yum install' / 'pacman -S' another package to make this work - possibly it'll be called 'zlib' or 'python-zlib' or something. I would recommend a Google search for your Linux distribution and the words 'python importerror zlib'; there's a high chance that'll tell you what you need to install. ChrisA From jotavrj at gmail.com Wed Jul 24 22:08:55 2013 From: jotavrj at gmail.com (jotavrj at gmail.com) Date: Wed, 24 Jul 2013 19:08:55 -0700 (PDT) Subject: High Availability with neo4j-embedded in Python -- Is that possible? Message-ID: <9fc98759-d78a-48a1-8ae8-5c30131e62b8@googlegroups.com> I'm almost switching from Python to Groovy because I can't find a way to use the neo4j-embedded's Highly Available mode. But no, I certainly don't want Java... I'm using JPype. Is there a way to do it? Kind regards, Jota Junior From r1chardj0n3s at gmail.com Thu Jul 25 00:59:42 2013 From: r1chardj0n3s at gmail.com (Richard Jones) Date: Thu, 25 Jul 2013 14:59:42 +1000 Subject: [ANN] PyWeek 17 will run in the first week of September (1st to 8th) - write a game in Python in a week Message-ID: Hi all, The Python Game Programming Challenge will run its 17th challenge during the first week of September, from the 1st to the 8th. The PyWeek challenge: 1. Invites entrants to write a game in one week from scratch either as an individual or in a team, 2. Is intended to be challenging and fun, 3. Will increase the public body of game tools, code and expertise, 4. Will let a lot of people actually finish a game, and 5. May inspire new projects (with ready made teams!) Check out the help page for how to compete (and prepare) and the growing resources message board post: http://pyweek.org/s/help/ http://pyweek.org/d/4008/ From john_ladasky at sbcglobal.net Thu Jul 25 01:26:45 2013 From: john_ladasky at sbcglobal.net (John Ladasky) Date: Wed, 24 Jul 2013 22:26:45 -0700 (PDT) Subject: Python3, GUIs, game APIs, PyGLet, 2to3...? Message-ID: <4ddada8b-81fd-46e7-bd5d-eeab550a9fa7@googlegroups.com> I am teaching Python 3 to a few beginning computer programming students. Being high-school age boys, they are, unsurprisingly, interested in games. I want to introduce them to real-time programming and GUI in the most painless way possible. I know that Python comes bundled with Tkinter. Aside from the fact that I dislike its look and feel, Tkinter is not a beginners' GUI tool. You have to write a fair amount of boiler-plate code, and you have to understand classes pretty well. My students are not at the OOP level yet. Yet Python's turtle package, which is built on Tkinter, manages to avoid a lot of that complexity, at least at first. I am looking for a tool which gives my students the simplicity of turtle, but which works with 2D raster graphics. And, it must work on Py3. Deep in the middle of another thread, I had PyGLet recommended to me. So I tried to get PyGLet working on Python 3, which it supposedly will do. It installs on Py3, but importing pyglet fails. The test program also fails. The tracebacks were showing me that the package code is full of Python 2.x print statements. I started fixing them manually and, after a while, gave up. https://groups.google.com/d/msg/comp.lang.python/J64gfFg3ZKw/hH-lXurR70EJ There may be other Py3-incompatible code in the PyGLet package that I haven't encountered yet. Thus I have looked at the Python docs for the "2to3" utility. 2to3 probably does what I want, except for one thing: recursive operations on subfolders. Do I really have to manually traverse the directory tree of the package, modify one folder's worth of .py files, and then repeat ad nauseam? Yes, I could write a script to do that -- but shouldn't that functionality be built into 2to3? Any advice that you folks might offer -- either about getting 2to3 to execute recursively, or about installing any GUI with a shallow learning curve installed on Py3, would be appreciated. Thanks. From john_ladasky at sbcglobal.net Thu Jul 25 02:56:19 2013 From: john_ladasky at sbcglobal.net (John Ladasky) Date: Wed, 24 Jul 2013 23:56:19 -0700 (PDT) Subject: Python3, GUIs, game APIs, PyGLet, 2to3...? In-Reply-To: <4ddada8b-81fd-46e7-bd5d-eeab550a9fa7@googlegroups.com> References: <4ddada8b-81fd-46e7-bd5d-eeab550a9fa7@googlegroups.com> Message-ID: Followup to my own post: I've made progress with PyGLet. I should mention that I'm using Ubuntu Linux 13.04 64-bit, in case it matters. I tried executing "2to3 -w *.py" on just the files in the directory pyglet-1.2alpha1/pyglet. I then changed back to the pyglet-1.2alpha1 directory, and executed "sudo python setup.py install". Finally, I started my Python3 interpreter. This time, "import pyglet" generated no errors. The PyGLet "Hello, World" code found on this web page runs: http://www.pyglet.org/doc/programming_guide/hello_world.html While unexplored parts of the PyGLet package may yet contain Py2-specific code, I'm definitely on my way. From kushal.kumaran+python at gmail.com Thu Jul 25 04:35:43 2013 From: kushal.kumaran+python at gmail.com (Kushal Kumaran) Date: Thu, 25 Jul 2013 14:05:43 +0530 Subject: Python3, GUIs, game APIs, PyGLet, 2to3...? In-Reply-To: References: <4ddada8b-81fd-46e7-bd5d-eeab550a9fa7@googlegroups.com> Message-ID: <51f13889.4734420a.36bb.ffffa22f@mx.google.com> John Ladasky writes: > Followup to my own post: I've made progress with PyGLet. I should mention that I'm using Ubuntu Linux 13.04 64-bit, in case it matters. > > I tried executing "2to3 -w *.py" on just the files in the directory pyglet-1.2alpha1/pyglet. I then changed back to the pyglet-1.2alpha1 directory, and executed "sudo python setup.py install". Finally, I started my Python3 interpreter. This time, "import pyglet" generated no errors. > Does your python command mean python2 or python3? The setup.py at https://code.google.com/p/pyglet/source/browse/setup.py seems to run 2to3 automatically, but that will only happen if you actually use python3 to run setup.py. > The PyGLet "Hello, World" code found on this web page runs: > > http://www.pyglet.org/doc/programming_guide/hello_world.html > > While unexplored parts of the PyGLet package may yet contain Py2-specific code, I'm definitely on my way. -- regards, kushal From john_ladasky at sbcglobal.net Thu Jul 25 18:26:01 2013 From: john_ladasky at sbcglobal.net (John Ladasky) Date: Thu, 25 Jul 2013 15:26:01 -0700 (PDT) Subject: Python3, GUIs, game APIs, PyGLet, 2to3...? In-Reply-To: References: <4ddada8b-81fd-46e7-bd5d-eeab550a9fa7@googlegroups.com> Message-ID: <5c35c252-67ed-4810-88b2-6708ed321bee@googlegroups.com> On Thursday, July 25, 2013 1:35:43 AM UTC-7, Kushal Kumaran wrote: > Does your python command mean python2 or python3? The setup.py at > https://code.google.com/p/pyglet/source/browse/setup.py seems to run > 2to3 automatically, but that will only happen if you actually use > python3 to run setup.py. Thank you for that sharp observation, Kushal! (And I hope that more packages besides pyglet are configured this way -- to automatically run 2to3, when you install on Python 3.) How strange that I remember typing "sudo python setup.py install". Because on my Ubuntu 13.04 system, that would invoke Python 2.7, which is an essential part of the OS. Nevertheless, I was able to import pyglet after invoking "python3" from the shell. I didn't think that installed modules could be shared between multiple Python versions. I'll try again from scratch, and see whether that clears up my problems. From john_ladasky at sbcglobal.net Thu Jul 25 19:49:16 2013 From: john_ladasky at sbcglobal.net (John Ladasky) Date: Thu, 25 Jul 2013 16:49:16 -0700 (PDT) Subject: PyGLet, 2to3...? In-Reply-To: <5c35c252-67ed-4810-88b2-6708ed321bee@googlegroups.com> References: <4ddada8b-81fd-46e7-bd5d-eeab550a9fa7@googlegroups.com> <5c35c252-67ed-4810-88b2-6708ed321bee@googlegroups.com> Message-ID: On Thursday, July 25, 2013 3:26:01 PM UTC-7, John Ladasky wrote: > I'll try again from scratch, and see whether that clears up my problems. Nope, that didn't work. ======================================= john at john:~/Desktop/pyglet-1.2alpha1$ sudo python3 setup.py install [sudo] password for john: running install running build running build_py running install_lib running install_egg_info Removing /usr/local/lib/python3.3/dist-packages/pyglet-1.2alpha1.egg-info Writing /usr/local/lib/python3.3/dist-packages/pyglet-1.2alpha1.egg-info john at john:~/Desktop/pyglet-1.2alpha1$ python3 Python 3.3.1 (default, Apr 17 2013, 22:30:32) [GCC 4.7.3] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pyglet Traceback (most recent call last): File "", line 1, in File "./pyglet/__init__.py", line 276 print '[%d] %s%s %s' % (thread, indent, name, location) ^ SyntaxError: invalid syntax ======================================= The source code link that Kushal posted is dated December 19, 2012. Since that was several months ago, I double-checked the source code of setup.py in the 1.2alpha1 package that I downloaded. It would appear to perform the same check of sys.version_info that was shown on the Google Code page. To see how that check actually runs, I saved a copy of setup.py as setup2.py, adding diagnostic calls to print() as shown in the code block below: ======================================= if sys.version_info >= (3,): # Automatically run 2to3 when using Python 3 print("Python version is 3.0 or later.") # I added this if _have_setuptools: print("Have setuptools.") # I added this setup_info["use_2to3"] = True else: print("Do not have setuptools.") # I added this from distutils.command.build_py import build_py_2to3 setup_info["cmdclass"] = {"build_py" : build_py_2to3} ======================================= Here's the output: ======================================= john at john:~/Desktop/pyglet-1.2alpha1$ sudo python3 setup2.py install Python version is 3.0 or later. Do not have setuptools. running install running build running build_py running install_lib running install_egg_info Removing /usr/local/lib/python3.3/dist-packages/pyglet-1.2alpha1.egg-info Writing /usr/local/lib/python3.3/dist-packages/pyglet-1.2alpha1.egg-info ======================================= So, I don't know much yet about this _have_setuptools flag. I don't know whether it has to be True, instead of False, in order for 2to3 to work properly. I get the impression from the code that 2to3 should run regardless of the _have_setuptools flag, it is just that the task is accomplished in two different ways? From malaclypse2 at gmail.com Fri Jul 26 09:39:54 2013 From: malaclypse2 at gmail.com (Jerry Hill) Date: Fri, 26 Jul 2013 09:39:54 -0400 Subject: PyGLet, 2to3...? In-Reply-To: References: <4ddada8b-81fd-46e7-bd5d-eeab550a9fa7@googlegroups.com> <5c35c252-67ed-4810-88b2-6708ed321bee@googlegroups.com> Message-ID: On Thu, Jul 25, 2013 at 7:49 PM, John Ladasky wrote: > ======================================= > > john at john:~/Desktop/pyglet-1.2alpha1$ sudo python3 setup.py install > > [sudo] password for john: > > running install > running build > running build_py > running install_lib > running install_egg_info > Removing /usr/local/lib/python3.3/dist-packages/pyglet-1.2alpha1.egg-info > Writing /usr/local/lib/python3.3/dist-packages/pyglet-1.2alpha1.egg-info Pyglet was installed to /usr/local/lib/python3.3/dist-packages ... > john at john:~/Desktop/pyglet-1.2alpha1$ python3 > > Python 3.3.1 (default, Apr 17 2013, 22:30:32) > [GCC 4.7.3] on linux > Type "help", "copyright", "credits" or "license" for more information. >>>> import pyglet > Traceback (most recent call last): > File "", line 1, in > File "./pyglet/__init__.py", line 276 > print '[%d] %s%s %s' % (thread, indent, name, location) > ^ > SyntaxError: invalid syntax ... But here, the error message is talking about ./pyglet/__init__.py. I think you're accidentally importing the pyglet package from the local directory instead of from the proper location in dist-packages. Try changing back to your home directory and trying this again. I think you're picking up the code from ~/Desktop/pyglet-1.2alpha1/pyglet instead of from /usr/local/lib/python3.3/dist-packages. -- Jerry From kushal.kumaran+python at gmail.com Fri Jul 26 00:52:16 2013 From: kushal.kumaran+python at gmail.com (Kushal Kumaran) Date: Fri, 26 Jul 2013 10:22:16 +0530 Subject: PyGLet, 2to3...? In-Reply-To: References: <4ddada8b-81fd-46e7-bd5d-eeab550a9fa7@googlegroups.com> <5c35c252-67ed-4810-88b2-6708ed321bee@googlegroups.com> Message-ID: <51f28eb6.8358440a.62dc.ffffad74@mx.google.com> John Ladasky writes: > On Thursday, July 25, 2013 3:26:01 PM UTC-7, John Ladasky wrote: >> I'll try again from scratch, and see whether that clears up my problems. > > Nope, that didn't work. > > ======================================= > > john at john:~/Desktop/pyglet-1.2alpha1$ sudo python3 setup.py install > > [sudo] password for john: > > running install > running build > running build_py > running install_lib > running install_egg_info > Removing /usr/local/lib/python3.3/dist-packages/pyglet-1.2alpha1.egg-info > Writing /usr/local/lib/python3.3/dist-packages/pyglet-1.2alpha1.egg-info > > john at john:~/Desktop/pyglet-1.2alpha1$ python3 > > Python 3.3.1 (default, Apr 17 2013, 22:30:32) > [GCC 4.7.3] on linux > Type "help", "copyright", "credits" or "license" for more information. >>>> import pyglet > Traceback (most recent call last): > File "", line 1, in > File "./pyglet/__init__.py", line 276 ----------^ Your import is attempting to import from your current directory, which has the pre-2to3 version of the files packaged in the original distribution. Switch away and try again. > print '[%d] %s%s %s' % (thread, indent, name, location) > ^ > SyntaxError: invalid syntax > > ======================================= > > The source code link that Kushal posted is dated December 19, 2012. Since that was several months ago, I double-checked the source code of setup.py in the 1.2alpha1 package that I downloaded. It would appear to perform the same check of sys.version_info that was shown on the Google Code page. > > To see how that check actually runs, I saved a copy of setup.py as setup2.py, adding diagnostic calls to print() as shown in the code block below: > > ======================================= > > if sys.version_info >= (3,): > # Automatically run 2to3 when using Python 3 > print("Python version is 3.0 or later.") # I added this > if _have_setuptools: > print("Have setuptools.") # I added this > setup_info["use_2to3"] = True > else: > print("Do not have setuptools.") # I added this > from distutils.command.build_py import build_py_2to3 > setup_info["cmdclass"] = {"build_py" : build_py_2to3} > > ======================================= > > Here's the output: > > ======================================= > > john at john:~/Desktop/pyglet-1.2alpha1$ sudo python3 setup2.py install > > Python version is 3.0 or later. > Do not have setuptools. > running install > running build > running build_py > running install_lib > running install_egg_info > Removing /usr/local/lib/python3.3/dist-packages/pyglet-1.2alpha1.egg-info > Writing /usr/local/lib/python3.3/dist-packages/pyglet-1.2alpha1.egg-info > > ======================================= > > So, I don't know much yet about this _have_setuptools flag. I don't know whether it has to be True, instead of False, in order for 2to3 to work properly. I get the impression from the code that 2to3 should run regardless of the _have_setuptools flag, it is just that the task is accomplished in two different ways? That seems correct. My familiarity with the python packaging tools is limited, though. -- regards, kushal From john_ladasky at sbcglobal.net Fri Jul 26 19:35:18 2013 From: john_ladasky at sbcglobal.net (John Ladasky) Date: Fri, 26 Jul 2013 16:35:18 -0700 (PDT) Subject: PyGLet, 2to3...? In-Reply-To: References: <4ddada8b-81fd-46e7-bd5d-eeab550a9fa7@googlegroups.com> <5c35c252-67ed-4810-88b2-6708ed321bee@googlegroups.com> Message-ID: <31cf07ec-3f37-45da-84b1-06b12609319d@googlegroups.com> On Thursday, July 25, 2013 4:49:16 PM UTC-7, John Ladasky wrote: > On Thursday, July 25, 2013 3:26:01 PM UTC-7, John Ladasky wrote: > > > I'll try again from scratch, and see whether that clears up my problems. > > Nope, that didn't work. Thanks to both Jerry and Kushal. You were right, I was doing a local import of Py2.x code, instead of importing the 2to3-converted code from site-packages. From john_ladasky at sbcglobal.net Fri Jul 26 21:19:48 2013 From: john_ladasky at sbcglobal.net (John Ladasky) Date: Fri, 26 Jul 2013 18:19:48 -0700 (PDT) Subject: PyGLet, 2to3...? In-Reply-To: <31cf07ec-3f37-45da-84b1-06b12609319d@googlegroups.com> References: <4ddada8b-81fd-46e7-bd5d-eeab550a9fa7@googlegroups.com> <5c35c252-67ed-4810-88b2-6708ed321bee@googlegroups.com> <31cf07ec-3f37-45da-84b1-06b12609319d@googlegroups.com> Message-ID: <0e68990c-1928-41f5-bfa5-cf6b4437050c@googlegroups.com> I'm making progress, but I'm not out of the woods yet. I'm trying to run some of the programs from the tutorial web pages, and from the pyglet1.2alpha1/examples directory. I've realized that I will probably need to run 2to3 on the many of the latter. The Hello, World example runs. http://www.pyglet.org/doc/programming_guide/hello_world.html The Image Viewer example... http://www.pyglet.org/doc/programming_guide/image_viewer.html ...runs if I provide a local image file, and load it using pyglet.image.load(). As written, the example fails on the line: image = pyglet.resource.image('kitten.jpg') It's possible that the alpha1.2 version of PyGLet is missing some resources. It looks like the code for the bouncing ball example ought to work, but it doesn't. That code can be found on the web at: http://www.pyglet.org/doc/programming_guide/noisy.py and also in the PyGLet package, in the folder pyglet1.2alpha1/examples/noisy. Here's my traceback which, I am sorry to report, I find less than fully informative: =============================== Traceback (most recent call last): File "/usr/local/lib/python3.3/dist-packages/pyglet/__init__.py", line 332, in __getattr__ return getattr(self._module, name) AttributeError: 'NoneType' object has no attribute 'load' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.3/dist-packages/pyglet/lib.py", line 111, in load_library lib = ctypes.cdll.LoadLibrary(name) File "/usr/lib/python3.3/ctypes/__init__.py", line 431, in LoadLibrary return self._dlltype(name) File "/usr/lib/python3.3/ctypes/__init__.py", line 353, in __init__ self._handle = _dlopen(self._name, mode) OSError: libavbin.so: cannot open shared object file: No such file or directory During handling of the above exception, another exception occurred: Traceback (most recent call last): File "noisy.py", line 56, in sound = pyglet.resource.media(BALL_SOUND, streaming=False) File "/usr/local/lib/python3.3/dist-packages/pyglet/resource.py", line 610, in media return media.load(path, streaming=streaming) File "/usr/local/lib/python3.3/dist-packages/pyglet/__init__.py", line 338, in __getattr__ __import__(import_name) File "/usr/local/lib/python3.3/dist-packages/pyglet/media/__init__.py", line 1469, in from . import avbin File "/usr/local/lib/python3.3/dist-packages/pyglet/media/avbin.py", line 64, in darwin='/usr/local/lib/libavbin.dylib') File "/usr/local/lib/python3.3/dist-packages/pyglet/lib.py", line 118, in load_library if ((self.linux_not_found_error not in o.message) and AttributeError: 'OSError' object has no attribute 'message' =============================== I think that only the top few lines of this traceback are relevant. Somehow a None is being passed into some function in pyglet/__init__.py, when that function expects an object with an attribute named "load". Looking at the source, the function being called is _ModuleProxy.__getattr__(). I often find that Python's tracebacks stop one level short of what I really want to know. The top level calling function is frequently not named. What part of noisy.py called the code that crashed? Is there any way to get traceback to tell you more? Anyway, I wondered whether this might be a Python compatibility wart in noisy.py. Thus I tried running 2to3 on noisy.py, even though my eyes told me that the code was OK. 2to3 agreed with me, reporting: "RefactoringTool: No changes to noisy.py". I'm not sure where to go next with this. One thing is for certain, all of this is way over the heads of my students... From john_ladasky at sbcglobal.net Sat Jul 27 00:00:55 2013 From: john_ladasky at sbcglobal.net (John Ladasky) Date: Fri, 26 Jul 2013 21:00:55 -0700 (PDT) Subject: PyGLet, 2to3...? In-Reply-To: <0e68990c-1928-41f5-bfa5-cf6b4437050c@googlegroups.com> References: <4ddada8b-81fd-46e7-bd5d-eeab550a9fa7@googlegroups.com> <5c35c252-67ed-4810-88b2-6708ed321bee@googlegroups.com> <31cf07ec-3f37-45da-84b1-06b12609319d@googlegroups.com> <0e68990c-1928-41f5-bfa5-cf6b4437050c@googlegroups.com> Message-ID: <5cb707df-583e-4364-935e-28ca05419c8c@googlegroups.com> On Friday, July 26, 2013 6:19:48 PM UTC-7, John Ladasky wrote: > I'm making progress, but I'm not out of the woods yet. And while I appreciate any comments that may appear here, I've just found the pyglet-users group... https://groups.google.com/forum/#!forum/pyglet-users ...so that's probably the best place for me to ask further questions. From cool.tanaya16 at gmail.com Thu Jul 25 02:58:16 2013 From: cool.tanaya16 at gmail.com (Tanaya D) Date: Wed, 24 Jul 2013 23:58:16 -0700 (PDT) Subject: must be dicts in tuple Message-ID: Hi, I am using Python with Bots(EDI translator) But i am getting the following error: MappingFormatError: must be dicts in tuple: get((({'BOTSID': 'HEADER'},),)) Can anyone pls help me with it. Thanks From rosuav at gmail.com Thu Jul 25 04:19:02 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 18:19:02 +1000 Subject: must be dicts in tuple In-Reply-To: References: Message-ID: On Thu, Jul 25, 2013 at 4:58 PM, Tanaya D wrote: > Hi, > > I am using Python with Bots(EDI translator) > > But i am getting the following error: > MappingFormatError: must be dicts in tuple: get((({'BOTSID': 'HEADER'},),)) The first thing to do is to construct a short piece of code that demonstrates the problem. See http://sscce.org/ for some tips. Once you've done that, run that script and copy and paste its entire output (which will be your exception traceback) into an email. That said, though, I believe the problem you're seeing is going to be Bots-specific. You'll possibly do better on a Bots mailing list rather than a general Python one. ChrisA From timr at probo.com Tue Jul 30 00:15:11 2013 From: timr at probo.com (Tim Roberts) Date: Mon, 29 Jul 2013 21:15:11 -0700 Subject: must be dicts in tuple References: Message-ID: <7dfev8lj22630fpsoqvc4s7e82av7p3nh2@4ax.com> Tanaya D wrote: > >I am using Python with Bots(EDI translator) > >But i am getting the following error: >MappingFormatError: must be dicts in tuple: get((({'BOTSID': 'HEADER'},),)) > >Can anyone pls help me with it. Possible hint: the error says you're supposed to have dicts in a tuple. What you have is a dict in a tuple in a tuple. -- Tim Roberts, timr at probo.com Providenza & Boekelheide, Inc. From mauriceling at gmail.com Thu Jul 25 03:08:13 2013 From: mauriceling at gmail.com (mauriceling@acm.org) Date: Thu, 25 Jul 2013 00:08:13 -0700 (PDT) Subject: [ANN] New article in The Python Papers Source Codes - COPADS III (Compendium of Distributions II): Cauchy, Cosine, Exponential, Hypergeometric, Logarithmic, Semicircular, Triangular, and Weibull Message-ID: Hi everyone We have a new article in The Python Papers Source Codes. Title: COPADS III (Compendium of Distributions II): Cauchy, Cosine, Exponential, Hypergeometric, Logarithmic, Semicircular, Triangular, and Weibull Abstract: This manuscript illustrates the implementation and testing of eight statistical distributions, namely Cauchy, Cosine, Exponential, Hypergeometric, Logarithmic, Semicircular, Triangular, and Weibull distribution, where each distribution consists of three common functions ? Probability Density Function (PDF), Cumulative Density Function (CDF) and the inverse of CDF (inverseCDF). These codes had been incorporated into COPADS codebase (https://github.com/copads/ copads) are licensed under Lesser General Public Licence version 3. URL: http://ojs.pythonpapers.org/index.php/tppsc/article/view/253 Maurice Ling Co-EIC, The Python Papers Source Codes From devyncjohnson at gmail.com Thu Jul 25 09:24:30 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Thu, 25 Jul 2013 09:24:30 -0400 Subject: Critic my module Message-ID: <51F1270E.6000704@Gmail.com> Aloha Python Users! I made a Python3 module that allows users to use certain Linux shell commands from Python3 more easily than using os.system(), subprocess.Popen(), or subprocess.getoutput(). This module (once placed with the other modules) can be used like this import boash; boash.ls() I attached the module. I plan to release it on the Internet soon, but feel free to use it now. It is licensed under LGPLv3. The name comes from combining "Boa" with "SHell". Notice that the module's name almost looks like "BASH", a common Linux shell. The Boa is a constrictor snake. This module makes Unix shells easier to use via Python3. This brings the system shell closer to the Python shell. Mahalo, Devyn Collier Johnson DevynCJohnson at Gmail.com -------------- next part -------------- #!/usr/bin/python3 #Made by Devyn Collier Johnson, NCLA, Linux+, LPIC-1, DCTS #Made using the Geany IDE #LGPLv3 import re, sys, subprocess, platform def grep(regex,textf): #Sample Command: grep.grep("^x",dir()) #Syntax: boash.grep(regexp_string,list_of_strings_to_search) version = '0.2a' expr = re.compile(regex) match = re.findall(expr, textf) if match != None: print(match) def ls(): version = '0.3' print(subprocess.getoutput('ls')) def dir(): version = '0.3' print(subprocess.getoutput('dir')) def ll(): version = '0.3' print(subprocess.getoutput('ls -l')) def vdir(): version = '0.3' print(subprocess.getoutput('ls -l')) def uname(): version = '0.3' print(platform.uname()) def cmd(*arg): version = '0.3' print(subprocess.getoutput(arg)) def command(*arg): version = '0.3' print(subprocess.getoutput(arg)) def man(x): version = '0.3' print(subprocess.getoutput('man' + x)) def apropos(*arg): version = '0.3' print(subprocess.getoutput('apropos' + arg)) def bash(*arg): version = '0.3' print(subprocess.getoutput(arg)) def shell(*arg): version = '0.3' print(subprocess.getoutput(arg)) def clear_bash_history(): version = '0.3' print(subprocess.getoutput('history -c')) def clear_bash_hist(): version = '0.3' print(subprocess.getoutput('history -c')) def clear_hist(): version = '0.3' print(subprocess.getoutput('history -c')) def clear_history(): version = '0.3' print(subprocess.getoutput('history -c')) def del_bash_hist(): version = '0.3' print(subprocess.getoutput('history -c')) def delete_bash_hist(): version = '0.3' print(subprocess.getoutput('history -c')) def del_hist(): version = '0.3' print(subprocess.getoutput('history -c')) def delete_history(): version = '0.3' print(subprocess.getoutput('history -c')) def delete_bash_history(): version = '0.3' print(subprocess.getoutput('history -c')) def delete_hist(): version = '0.3' print(subprocess.getoutput('history -c')) def firefox(): version = '0.3' print(subprocess.Popen('(firefox &)')) def opera(): version = '0.3' print(subprocess.Popen('(opera &)')) def arora(): version = '0.3' print(subprocess.Popen('(arora &)')) def dolphin(): version = '0.3' print(subprocess.Popen('(dolphin &)')) def nautilus(): version = '0.3' print(subprocess.Popen('(nautilus &)')) def konqueror(): version = '0.3' print(subprocess.Popen('(konqueror &)')) def smplayer(): version = '0.3' print(subprocess.Popen('(smplayer &)')) def mplayer(): version = '0.3' print(subprocess.Popen('(mplayer &)')) def vlc(): version = '0.3' print(subprocess.Popen('(vlc &)')) def qvlc(): version = '0.3' print(subprocess.Popen('(qvlc &)')) def nvlc(): version = '0.3' jprint(subprocess.Popen('(nvlc &)')) def svlc(): version = '0.3' print(subprocess.Popen('(svlc &)')) def rvlc(): version = '0.3' print(subprocess.Popen('(rvlc &)')) def xterm(): version = '0.3' print(subprocess.Popen('(xterm &)')) def geany(): version = '0.3' print(subprocess.Popen('(geany &)')) def lsof(): version = '0.3' print(subprocess.getoutput(lsof)) def free(*arg): version = '0.3' print(subprocess.getoutput('free' + arg)) def lsof(*arg): version = '0.3' print(subprocess.getoutput('lsof' + arg)) def pwd(): version = '0.3' print(subprocess.getoutput('pwd')) def getcwd(): version = '0.3' print(subprocess.getoutput('pwd')) def whoami(): version = '0.3' print(subprocess.getoutput('whoami')) def finger(): version = '0.3' print(subprocess.getoutput('finger')) def hostname(): version = '0.3' print(subprocess.getoutput('hostname')) def arch(): version = '0.3' print(subprocess.getoutput('arch')) def architecture(): version = '0.3' print(subprocess.getoutput('arch')) def go_back(): version = '0.3' print(subprocess.Popen('cd !!:1')) def repeat(): version = '0.3' print(subprocess.Popen('!!') def repeat_cmd(): version = '0.3' print(subprocess.Popen('!!')) def last_cmd(): version = '0.3' print(subprocess.Popen('!!')) def last_command(): version = '0.3' print(subprocess.Popen('!!')) def ejcd(): version = '0.3' print(subprocess.Popen('eject cdrom1')) def eject_cd(): version = '0.3' print(subprocess.Popen('eject cdrom1')) def eject_cdrom(): version = '0.3' print(subprocess.Popen('eject cdrom1')) def ejectcd(): version = '0.3' print(subprocess.Popen('eject cdrom1')) def ejectcdrom(): version = '0.3' print(subprocess.Popen('eject cdrom1')) def eject_disc(): version = '0.3' print(subprocess.Popen('eject cdrom1')) def eject_disc_tray(): version = '0.3' print(subprocess.Popen('eject cdrom1')) def eject_tray(): version = '0.3' print(subprocess.Popen('eject cdrom1')) def ejectdisc(): version = '0.3' print(subprocess.Popen('eject cdrom1')) def ejectdisctray(): version = '0.3' print(subprocess.Popen('eject cdrom1')) def ejecttray(): version = '0.3' print(subprocess.Popen('eject cdrom1')) def ejtray(): version = '0.3' print(subprocess.Popen('eject cdrom1')) def ejdvd(): version = '0.3' print(subprocess.Popen('eject cdrom1')) def eject_dvd(): version = '0.3' print(subprocess.Popen('eject cdrom1')) def ejectdvd(): version = '0.3' print(subprocess.Popen('eject cdrom1')) def env(): version = '0.3' print(subprocess.getoutput('env')) def runlevel(): version = '0.3' print(subprocess.getoutput('runlevel')) def restart(): version = '0.3' print(subprocess.Popen('shutdown -r now')) def reboot(): version = '0.3' print(subprocess.Popen('shutdown -r now')) def shutdown(): version = '0.3' print(subprocess.Popen('shutdown now')) def shut_down(): version = '0.3' print(subprocess.Popen('shutdown now')) def halt(): version = '0.3' print(subprocess.Popen('shutdown -h now')) def poweroff(): version = '0.3' print(subprocess.Popen('shutdown -P now')) def power_off(): version = '0.3' print(subprocess.Popen('shutdown -P now')) def powerdown(): version = '0.3' print(subprocess.Popen('shutdown -P now')) def power_down(): version = '0.3' print(subprocess.Popen('shutdown -P now')) def nologin(): version = '0.3' print(subprocess.Popen('shutdown -k now')) def no_login(): version = '0.3' print(subprocess.Popen('shutdown -k now')) def nologins(): version = '0.3' print(subprocess.Popen('shutdown -k now')) def no_logins(): version = '0.3' print(subprocess.Popen('shutdown -k now')) version = '0.6b' From joshua at landau.ws Thu Jul 25 10:03:55 2013 From: joshua at landau.ws (Joshua Landau) Date: Thu, 25 Jul 2013 15:03:55 +0100 Subject: Critic my module In-Reply-To: <51F1270E.6000704@Gmail.com> References: <51F1270E.6000704@Gmail.com> Message-ID: On 25 July 2013 14:24, Devyn Collier Johnson wrote: > Aloha Python Users! > > I made a Python3 module that allows users to use certain Linux shell > commands from Python3 more easily than using os.system(), > subprocess.Popen(), or subprocess.getoutput(). This module (once placed > with the other modules) can be used like this > > import boash; boash.ls() > > I attached the module. I plan to release it on the Internet soon, but > feel free to use it now. It is licensed under LGPLv3. > > The name comes from combining "Boa" with "SHell". Notice that the > module's name almost looks like "BASH", a common Linux shell. The Boa is a > constrictor snake. This module makes Unix shells easier to use via Python3. > This brings the system shell closer to the Python shell. > 1) Have you tested everything? At first glance some of those look like they won't work. 2) Whenever you repeat yourself, *especially* at this magnitude, you're doing something seriously wrong - *especially* in Python. *Completely-untestedly-and-super-quick-hackedly:* import re, sys, subprocess, platform class Command: def __init__(self, command=None): self.command = command def __call__(self, *args): command_list = [self.command] if self.command else [] command_list.extend(args) print(subprocess.getoutput([command_list])) def uname(): print(platform.uname()) def lsof(): print(subprocess.getoutput(lsof)) apropos = Command("apropos") arora = Command("(arora &)") dir = Command("dir") dolphin = Command("(dolphin &)") env = Command("env") finger = Command("finger") firefox = Command("(firefox &)") free = Command("free") geany = Command("(geany &)") getcwd = Command("pwd") go_back = Command("cd !!:1") halt = Command("shutdown -h now") hostname = Command("hostname") konqueror = Command("(konqueror &)") ls = Command("ls") lsof = Command("lsof") man = Command("man") mplayer = Command("(mplayer &)") nautilus = Command("(nautilus &)") nvlc = Command("(nvlc &)") opera = Command("(opera &)") pwd = Command("pwd") qvlc = Command("(qvlc &)") repeat = Command("!") runlevel = Command("runlevel") rvlc = Command("(rvlc &)") smplayer = Command("(smplayer &)") svlc = Command("(svlc &)") vlc = Command("(vlc &)") whoami = Command("whoami") xterm = Command("(xterm &)") arch = architecture = Command("arch") bash = cmd = command = shell = Command() last_cmd = last_command = repeat_cmd = Command("!!") ll = vdir = Command("ls - l") no_login = no_logins = nologin = nologins = Command("shutdown -k now") power_down = power_off = powerdown = poweroff = Command("shutdown -P now") reboot = restart = Command("shutdown -r now") shut_down = shutdown = Command("shutdown now") clear_bash_hist = clear_bash_history = clear_hist = clear_history = \ del_bash_hist = del_hist = delete_bash_hist = delete_bash_history = \ delete_hist = delete_history = Command("history -c") ejcd = ejdvd = eject_cd = eject_cdrom = eject_disc = eject_disc_tray = \ eject_dvd = eject_tray = ejectcd = ejectcdrom = ejectdisc = ejectdisctray = ejectdvd = ejecttray = ejtray = Command("eject cdrom1") I wouldn't actually do it like this (I'd probably start with a dict and add to a class programmatically), but this is a simple display of how one can use classes and other meta-constructs to make things look nicer. It also doesn't deal with forcing the number of arguments to be correct, but this is demo material and so I leave that to you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alain at dpt-info.u-strasbg.fr Thu Jul 25 10:09:15 2013 From: alain at dpt-info.u-strasbg.fr (Alain Ketterlin) Date: Thu, 25 Jul 2013 16:09:15 +0200 Subject: Critic my module References: Message-ID: <87y58u20l0.fsf@dpt-info.u-strasbg.fr> Devyn Collier Johnson writes: > I made a Python3 module that allows users to use certain Linux > shell commands from Python3 more easily than using os.system(), > subprocess.Popen(), or subprocess.getoutput(). This module (once > placed with the other modules) can be used like this Good, but I doubt it's really useful: I think nobody is going to add a dependency on your module for, basically, one-line wrappers... Here are a few comments: > def ls(): > version = '0.3' > print(subprocess.getoutput('ls')) version is local here, so basically your first statement is useless (search for "global" in python's language ref). > def uname(): > version = '0.3' > print(platform.uname()) I once learned: "never print anything in a library function". This is a bad thing to do, for a variety of reasons. For instance, stdout may be redirected during this call... > def man(x): > version = '0.3' > print(subprocess.getoutput('man' + x)) getoutput is (essentially) Popen(...,shell=True), and the doc says: "the use of shell=True is strongly discouraged in cases where the command string is constructed from external input" (for very good reasons) > def clear_bash_history(): > version = '0.3' > print(subprocess.getoutput('history -c')) Who told you subprocess will use bash? Again, the doc: "On Unix with shell=True, the shell defaults to /bin/sh." All your uses of bash-isms may break (esp. "!!") > def firefox(): > version = '0.3' > print(subprocess.Popen('(firefox &)')) See section "Replacing the os.spawn family" in... the doc. > def go_back(): > version = '0.3' > print(subprocess.Popen('cd !!:1')) Hopeless. Have you tried this? > def reboot(): > version = '0.3' > print(subprocess.Popen('shutdown -r now')) What do you expect this to print? I mean, after shutdown/reboot. > version = '0.6b' So, what's the version? 0.3 or 0.6b (btw, are you sure this "version" is the same as the one you use in all functions?). -- Alain. From devyncjohnson at gmail.com Fri Jul 26 06:59:15 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Fri, 26 Jul 2013 06:59:15 -0400 Subject: Critic my module In-Reply-To: <87y58u20l0.fsf@dpt-info.u-strasbg.fr> References: <87y58u20l0.fsf@dpt-info.u-strasbg.fr> Message-ID: <51F25683.9010809@Gmail.com> On 07/25/2013 10:09 AM, Alain Ketterlin wrote: > Devyn Collier Johnson writes: > >> I made a Python3 module that allows users to use certain Linux >> shell commands from Python3 more easily than using os.system(), >> subprocess.Popen(), or subprocess.getoutput(). This module (once >> placed with the other modules) can be used like this > Good, but I doubt it's really useful: I think nobody is going to add a > dependency on your module for, basically, one-line wrappers... > > Here are a few comments: > >> def ls(): >> version = '0.3' >> print(subprocess.getoutput('ls')) > version is local here, so basically your first statement is useless > (search for "global" in python's language ref). > >> def uname(): >> version = '0.3' >> print(platform.uname()) > I once learned: "never print anything in a library function". This is a > bad thing to do, for a variety of reasons. For instance, stdout may be > redirected during this call... > >> def man(x): >> version = '0.3' >> print(subprocess.getoutput('man' + x)) > getoutput is (essentially) Popen(...,shell=True), and the doc says: > > "the use of shell=True is strongly discouraged in cases where the > command string is constructed from external input" > > (for very good reasons) > >> def clear_bash_history(): >> version = '0.3' >> print(subprocess.getoutput('history -c')) > Who told you subprocess will use bash? Again, the doc: > > "On Unix with shell=True, the shell defaults to /bin/sh." > > All your uses of bash-isms may break (esp. "!!") > >> def firefox(): >> version = '0.3' >> print(subprocess.Popen('(firefox &)')) > See section "Replacing the os.spawn family" in... the doc. > >> def go_back(): >> version = '0.3' >> print(subprocess.Popen('cd !!:1')) > Hopeless. Have you tried this? > >> def reboot(): >> version = '0.3' >> print(subprocess.Popen('shutdown -r now')) > What do you expect this to print? I mean, after shutdown/reboot. > >> version = '0.6b' > So, what's the version? 0.3 or 0.6b > > (btw, are you sure this "version" is the same as the one you use in all > functions?). > > -- Alain. The version in each function is the version of that function if users want to know what version they are using. The last version is for the whole module. The module overall is version 0.6b. The module started with a few functions and as I increased the number of functions, I increased the module version number. It is a coincidence that all of the modules happen to have the same version number. I increase the version number after I work on a function. I cannot remember the command to print a module's/function's version number, but with that command, you could see the version of a particular function or module. No, I have not tried go_back(), thank you for catching that. The main point of this is for shell users that are using Python and do not know some of the Python commands. This module would make Python more like a Linux shell. For instance, a shell user would type boash.uname() because they may not know they can type "import platform; platform.uname()". I know that printing is not really the best of ideas, but how else can I make the output be displayed without quotes or newline marks? Thank you very much Alain Ketterlin for your feedback! Mahalo, DCJ From alister.ware at ntlworld.com Fri Jul 26 11:08:35 2013 From: alister.ware at ntlworld.com (Alister) Date: Fri, 26 Jul 2013 15:08:35 GMT Subject: Critic my module References: <87y58u20l0.fsf@dpt-info.u-strasbg.fr> Message-ID: > > The main point of this is for shell users that are using Python and do > not know some of the Python commands. This module would make Python more > like a Linux shell. For instance, a shell user would type boash.uname() > because they may not know they can type "import platform; > platform.uname()". But they will know how to import your module? to be honest I think this module is a solution looking for a problem & you have re-invented the wheel (Square). don't let that put you off, there are many command line tools that could do with a good wrapper to make them easier to use,perhaps you are tryingto be too general. > > I know that printing is not really the best of ideas, but how else can I > make the output be displayed without quotes or newline marks? as this is a module intended for others to use in their programs (rather than a tool in its own right) the correct approach is to return the result & lave it for the user to print the result, or process it however they require. -- I also never expected Intel to dispose of themselves in such a cute way. - Rik van Riel on linux-kernel From joshua at landau.ws Fri Jul 26 13:24:29 2013 From: joshua at landau.ws (Joshua Landau) Date: Fri, 26 Jul 2013 18:24:29 +0100 Subject: Critic my module In-Reply-To: References: <87y58u20l0.fsf@dpt-info.u-strasbg.fr> Message-ID: On 26 July 2013 16:08, Alister wrote: > > > > The main point of this is for shell users that are using Python and do > > not know some of the Python commands. This module would make Python more > > like a Linux shell. For instance, a shell user would type boash.uname() > > because they may not know they can type "import platform; > > platform.uname()". > > But they will know how to import your module? > > to be honest I think this module is a solution looking for a problem & > you have re-invented the wheel (Square). > don't let that put you off, there are many command line tools that could > do with a good wrapper to make them easier to use,perhaps you are tryingto > be too general. I actually think http://plumbum.readthedocs.org , http://sarge.readthedocs.org and http://amoffat.github.io/sh/ do a really good job at this. I might one day even try them ;). -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve+comp.lang.python at pearwood.info Fri Jul 26 22:48:12 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 27 Jul 2013 02:48:12 GMT Subject: Critic my module References: Message-ID: <51f334eb$0$29971$c3e8da3$5496439d@news.astraweb.com> As requested, some constructive criticism of your module. On Thu, 25 Jul 2013 09:24:30 -0400, Devyn Collier Johnson wrote: > #!/usr/bin/python3 > #Made by Devyn Collier Johnson, NCLA, Linux+, LPIC-1, DCTS What's NCLA, Linux+, LPIC-1, DCTS? Do these mean anything? Are we supposed to know what they mean? "Made by" has no legal significance. You probably want: Copyright ? 2013 Devyn Collier Johnson. > #Made using the Geany IDE Nobody gives a monkey's toss what editor you used to type up the module. You might as well mention the brand of monitor you used, or whether the keyboard is Dvorak or Qwerty. > #LGPLv3 You can't just drop in a mention of "LGPLv3" and expect it to mean anything. You actually have to obey the licence yourself, and that includes *actually including the licence in your work*. (You're technically in violation of the licence at the moment, however since the only person whose copyright you are infringing is yourself, it doesn't matter. However anyone else using your code is at risk.) http://www.gnu.org/licenses/gpl-howto.html In the case of the LGPL, you have to include the text of *both* the GPL and the LGPL, not just one. > import re, sys, subprocess, platform > def grep(regex,textf): > #Sample Command: grep.grep("^x",dir()) #Syntax: > boash.grep(regexp_string,list_of_strings_to_search) Comments using # are only of use to people reading the source code. If you want comments to be available at the interactive prompt, you should write them as doc strings: def grep(regex, textf): """This string is a docstring. Sample command: ... Blah blah blah """ Then, at the interactive prompt, the user can say: help(boash.grep) to read the docstring. > version = '0.2a' That's quite useless, since it is a local variable invisible outside of the function. Also, why would you bother giving every individual function a version number? That's rather pointless. The user cannot pick and choose function A with version number 0.6 and function B with version number 0.7 if the module provides versions 0.7 of both. > expr = re.compile(regex) > match = re.findall(expr, textf) > if match != None: > print(match) When comparing with None, it is preferred to use "is" and "is not" rather than equality tests. > def ls(): > version = '0.3' > print(subprocess.getoutput('ls')) > def dir(): > version = '0.3' > print(subprocess.getoutput('dir')) A blank line or two between functions does wonders for readability. There is no prize for conserving newlines. You might like to read PEP 8, the Python style guide. It is optional, but still makes a very good guide. http://www.python.org/dev/peps/pep-0008/ > def bash(*arg): > version = '0.3' > print(subprocess.getoutput(arg)) > def shell(*arg): > version = '0.3' > print(subprocess.getoutput(arg)) bash is not a synonym for "shell". "The shell" might be sh, csh, bash, or any one of many other shells, all of which are slightly (or not so slightly) different. > def clear_bash_history(): > version = '0.3' > print(subprocess.getoutput('history -c')) [...] Do you really need ten aliases for 'history -c'? If you want to define aliases for a function, don't recreate the entire function ten times. Start with defining the function once, then: clear_bash_hist = clear_hist = clear_history = clear_bash_history etc. But really, having ten names for the one function just confuses people, who then wonder what subtle difference there is between delete_history and clear_history. > def firefox(): > version = '0.3' > print(subprocess.Popen('(firefox &)')) Is Firefox really so important that it needs a dedicated command? What about Debian users? Doesn't Iceweasel get a command? > def xterm(): > version = '0.3' > print(subprocess.Popen('(xterm &)')) Surely the user already has an xterm open, if they are running this interactively? Why not just use your xterm's "new window" or "new tab" command? [...delete more trivial calls to subprocess...] > def repeat_cmd(): > version = '0.3' > print(subprocess.Popen('!!')) [... delete two exact copies of this function...] > def ejcd(): > version = '0.3' > print(subprocess.Popen('eject cdrom1')) [... delete FOURTEEN exact copies of this function...] Really? Is anyone going to type "eject_disc_tray" instead of "eject"? I think that will do. This doesn't really do anything except define a large number of trivial wrappers to commands already available in the shell. Emphasis on the *trivial* -- with the exception of the grep wrapper, which is all of four lines (ignoring the useless internal version number), every single one of these wrapper functions is a one-liner.[1] In other words, you're not adding any value to the shell commands by wrapping them in Python. There are plenty of big, complex shell commands that take a plethora of options and could do with some useful Python wrappers, like wget. But you haven't done them. Nor have you added extra security, or even extra convenience. You've done nothing that couldn't be done using the shell "alias" command, except in Python where the syntax is less convenient (e.g. "ls" in the shell, versus "ls()" in Python). [1] I think every newbie programmer goes through a stage of pointlessly writing one-liner wrappers to every second function they see. I know I did. The difference is, before the Internet, nobody did it publicly. -- Steven From devyncjohnson at gmail.com Sat Jul 27 08:56:10 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Sat, 27 Jul 2013 08:56:10 -0400 Subject: Critic my module In-Reply-To: <51f334eb$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <51f334eb$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51F3C36A.4020400@Gmail.com> On 07/26/2013 10:48 PM, Steven D'Aprano wrote: > As requested, some constructive criticism of your module. > > On Thu, 25 Jul 2013 09:24:30 -0400, Devyn Collier Johnson wrote: > >> #!/usr/bin/python3 >> #Made by Devyn Collier Johnson, NCLA, Linux+, LPIC-1, DCTS > What's NCLA, Linux+, LPIC-1, DCTS? Do these mean anything? Are we > supposed to know what they mean? > > "Made by" has no legal significance. You probably want: > > Copyright ? 2013 Devyn Collier Johnson. > > >> #Made using the Geany IDE > Nobody gives a monkey's toss what editor you used to type up the module. > You might as well mention the brand of monitor you used, or whether the > keyboard is Dvorak or Qwerty. > > >> #LGPLv3 > You can't just drop in a mention of "LGPLv3" and expect it to mean > anything. You actually have to obey the licence yourself, and that > includes *actually including the licence in your work*. (You're > technically in violation of the licence at the moment, however since the > only person whose copyright you are infringing is yourself, it doesn't > matter. However anyone else using your code is at risk.) > > http://www.gnu.org/licenses/gpl-howto.html > > In the case of the LGPL, you have to include the text of *both* the GPL > and the LGPL, not just one. > > > >> import re, sys, subprocess, platform >> def grep(regex,textf): >> #Sample Command: grep.grep("^x",dir()) #Syntax: >> boash.grep(regexp_string,list_of_strings_to_search) > Comments using # are only of use to people reading the source code. If > you want comments to be available at the interactive prompt, you should > write them as doc strings: > > def grep(regex, textf): > """This string is a docstring. > > Sample command: ... > Blah blah blah > """ > > Then, at the interactive prompt, the user can say: > > help(boash.grep) > > to read the docstring. > > >> version = '0.2a' > That's quite useless, since it is a local variable invisible outside of > the function. > > Also, why would you bother giving every individual function a version > number? That's rather pointless. The user cannot pick and choose function > A with version number 0.6 and function B with version number 0.7 if the > module provides versions 0.7 of both. > > >> expr = re.compile(regex) >> match = re.findall(expr, textf) >> if match != None: >> print(match) > When comparing with None, it is preferred to use "is" and "is not" rather > than equality tests. > > >> def ls(): >> version = '0.3' >> print(subprocess.getoutput('ls')) >> def dir(): >> version = '0.3' >> print(subprocess.getoutput('dir')) > A blank line or two between functions does wonders for readability. There > is no prize for conserving newlines. > > You might like to read PEP 8, the Python style guide. It is optional, but > still makes a very good guide. > > http://www.python.org/dev/peps/pep-0008/ > > >> def bash(*arg): >> version = '0.3' >> print(subprocess.getoutput(arg)) >> def shell(*arg): >> version = '0.3' >> print(subprocess.getoutput(arg)) > bash is not a synonym for "shell". "The shell" might be sh, csh, bash, or > any one of many other shells, all of which are slightly (or not so > slightly) different. > > >> def clear_bash_history(): >> version = '0.3' >> print(subprocess.getoutput('history -c')) > [...] > > Do you really need ten aliases for 'history -c'? > > If you want to define aliases for a function, don't recreate the entire > function ten times. Start with defining the function once, then: > > clear_bash_hist = clear_hist = clear_history = clear_bash_history > > etc. But really, having ten names for the one function just confuses > people, who then wonder what subtle difference there is between > delete_history and clear_history. > >> def firefox(): >> version = '0.3' >> print(subprocess.Popen('(firefox &)')) > Is Firefox really so important that it needs a dedicated command? > > What about Debian users? Doesn't Iceweasel get a command? > > >> def xterm(): >> version = '0.3' >> print(subprocess.Popen('(xterm &)')) > Surely the user already has an xterm open, if they are running this > interactively? Why not just use your xterm's "new window" or "new tab" > command? > > > [...delete more trivial calls to subprocess...] > >> def repeat_cmd(): >> version = '0.3' >> print(subprocess.Popen('!!')) > [... delete two exact copies of this function...] > >> def ejcd(): >> version = '0.3' >> print(subprocess.Popen('eject cdrom1')) > [... delete FOURTEEN exact copies of this function...] > > Really? Is anyone going to type "eject_disc_tray" instead of "eject"? > > > I think that will do. > > This doesn't really do anything except define a large number of trivial > wrappers to commands already available in the shell. Emphasis on the > *trivial* -- with the exception of the grep wrapper, which is all of four > lines (ignoring the useless internal version number), every single one of > these wrapper functions is a one-liner.[1] In other words, you're not > adding any value to the shell commands by wrapping them in Python. There > are plenty of big, complex shell commands that take a plethora of options > and could do with some useful Python wrappers, like wget. But you haven't > done them. > > Nor have you added extra security, or even extra convenience. You've done > nothing that couldn't be done using the shell "alias" command, except in > Python where the syntax is less convenient (e.g. "ls" in the shell, > versus "ls()" in Python). > > > > > [1] I think every newbie programmer goes through a stage of pointlessly > writing one-liner wrappers to every second function they see. I know I > did. The difference is, before the Internet, nobody did it publicly. > > Wow! Thanks for the thorough critic. I appreciate your feed back and thank you so much for the PEP link. I learned a lot. I never saw that page before. The "NCLA, Linux+, LPIC-1, DCTS" are my computer certifications. As for mentioning Geany, I am trying to promote and give credit to Geany. Good point about the Made by/Copyright suggestion. Although, I have not copyrighted the file, can I still say "Copyrighted by ...". Thank you for the LGPLv3 suggestion. I know that I must include the GPL license for GPL programs, but I thought for LGPL code I could just have "#LGPLv3". Thank you so much for that feedback. I definitely need to read about all of the types of licenses. I thought it would be helpful to include the version numbers for each function, but you and another Python developer said it is pointless. I see what you mean. The grep emulating function does not work yet. I am still working on that. Yeah, I have a VERY BAD habit of treating bash and the Linux shell (or any/all shells) as the same thing. I know they are all very different, but for some reason I still keep calling all shells in general BASH. I will be sure to create aliases. I make alias commands so that it is easier to guess or remember a command. For instance, a Python user my want to clear the shell's history, but can only remember one form of the command or must guess. On my Ubuntu system, I have set up numerous shell aliases. I am addicted to aliases. I still need to add the other browsers. Do very many people use Iceweasel? I did not notice that I have "print(subprocess.Popen('(xterm &)'))" instead of "subprocess.Popen('(xterm &)')". The worst computer errors are ID-10-T errors. True, the user my have Xterm open, but what if they use Guake (like me) or Pterm, EvilVTE, Valaterm, Gnome-Terminal, Konsole, etc.? How could I add security and convenience? Okay, I will try to add wget. Are there any other shell commands that anyone feels I should add? The point of this module is to allow Linux shell users to use Python3 as a regular shell. Instead of using CSH, Bash, Tcsh, FISH, etc., users could use Python3 and import this module. Python is more powerful than any shell, so I want to make it easier for anyone to use Python as the default shell. For instance, instead of typing "print(os.getcwd())" to get the current working directory, users could type "boash.ls()". I hope that is easier to remember than "print(os.getcwd())". As for the print() command, I do not like how os.getcwd() has single quotes around the output. Plus, Linux shell do not print output with quotes. I want to make this a very useful and popular module, so I will use the suggestions and add more useful wrappers. Would it help if I made a Youtube video showing how this module can be used? I will post the next version on this mailing list for another review. Thanks everyone, and thanks a lot Steven D'Aprano! Mahalo, Devyn Collier Johnson DevynCJohnson at Gmail.com From davea at davea.name Sat Jul 27 10:33:33 2013 From: davea at davea.name (Dave Angel) Date: Sat, 27 Jul 2013 10:33:33 -0400 Subject: Critic my module In-Reply-To: <51F3C36A.4020400@Gmail.com> References: <51f334eb$0$29971$c3e8da3$5496439d@news.astraweb.com> <51F3C36A.4020400@Gmail.com> Message-ID: On 07/27/2013 08:56 AM, Devyn Collier Johnson wrote: > Somehow during this thread, you have changed your purpose for this library. It used to be a library that Python programmers could import and use. And now, it's a shell replacement? The user runs the Python interpreter, and types the line import boash to get started. > > Yeah, I have a VERY BAD habit of treating bash and the Linux shell (or > any/all shells) as the same thing. I know they are all very different, > but for some reason I still keep calling all shells in general BASH. I seem to recall that BASH is an acronym, for Bourne Again SHell. > > I will be sure to create aliases. I make alias commands so that it is > easier to guess or remember a command. For instance, a Python user my > want to clear the shell's history, but can only remember one form of the > command or must guess. On my Ubuntu system, I have set up numerous shell > aliases. I am addicted to aliases. Nothing wrong with aliases. But how does your user create his own aliases? That's much more important than having yours available. Remember that any new globals he defines are lost when he exits the interpreter (or crashes). > > I still need to add the other browsers. Do very many people use Iceweasel? > > I did not notice that I have "print(subprocess.Popen('(xterm &)'))" > instead of "subprocess.Popen('(xterm &)')". The worst computer errors > are ID-10-T errors. > > True, the user my have Xterm open, but what if they use Guake (like me) > or Pterm, EvilVTE, Valaterm, Gnome-Terminal, Konsole, etc.? Exactly. If they're using a terminal with tabs, they might want to create a new tab, not a new instance of the terminal. Or if they're running screen or the equivalent, they want the new terminal to show up on their (remote) console. > > How could I add security and convenience? Okay, I will try to add wget. > Are there any other shell commands that anyone feels I should add? How about tab? The command-completion and filename-completion and parameter-completion logic of a good shell is extremely complex, and very useful. > > The point of this module is to allow Linux shell users to use Python3 as > a regular shell. Instead of using CSH, Bash, Tcsh, FISH, etc., users > could use Python3 and import this module. Python is more powerful than > any shell, so I want to make it easier for anyone to use Python as the > default shell. For instance, instead of typing "print(os.getcwd())" to > get the current working directory, users could type "boash.ls()". I hope > that is easier to remember than "print(os.getcwd())". It's easier, but doesn't begin to do the same thing. As for using this INSTEAD of csh, bash, etc., that might be likely once it gets beyond 10% of the usefulness. Right now, it's at about 0.01% And any habits a user gets within this environment have to be unlearned when he returns to a standard shell. Back in the early days of MSDOS, the FORMAT command didn't require you to specify a drive letter. So people could intend to format their floppy, and actually trash their hard disk. So I had colleagues who put a FORMAT.BAt command on their path which hard-wired the A: parameter. Now what happens to one of those folks when he goes to another machine which doesn't have that batch file? Ouch! Instead I wrote a batch file that checked to make sure you had the A: parameter. Rapidly, my fingers learned that FORMAT A: was the valid syntax, and pretty soon the batch file was unnecessary (for me). If I were going to define a dozen aliases for other people to use, I'd make them something like: def clear_hist(): print "The function you want is probably clear_history()" > As for the print() > command, I do not like how os.getcwd() has single quotes around the > output. Those quotes come from the Python interpreter, not from getcwd(). > Plus, Linux shell do not print output with quotes. > > I want to make this a very useful and popular module, so I will use the > suggestions and add more useful wrappers. Would it help if I made a > Youtube video showing how this module can be used? > > I will post the next version on this mailing list for another review. > Thanks everyone, and thanks a lot Steven D'Aprano! > Have you figured out how you're going to do things like cd (os.chdir), which have to remember state? And how to pass the new current directory to the shell that launched Python? Have you looked at ipython (ipython.org) ? At least from there, you can get command completion with tab, one-third of the bash functionality. So if you type boash.cle then it'll fill in the rest. -- DaveA From devyncjohnson at gmail.com Sat Jul 27 10:53:16 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Sat, 27 Jul 2013 10:53:16 -0400 Subject: Critic my module In-Reply-To: References: <51f334eb$0$29971$c3e8da3$5496439d@news.astraweb.com> <51F3C36A.4020400@Gmail.com> Message-ID: <51F3DEDC.2020303@Gmail.com> On 07/27/2013 10:33 AM, Dave Angel wrote: > On 07/27/2013 08:56 AM, Devyn Collier Johnson wrote: >> > > > > Somehow during this thread, you have changed your purpose for this > library. It used to be a library that Python programmers could import > and use. And now, it's a shell replacement? The user runs the Python > interpreter, and types the line import boash to get started. > >> >> Yeah, I have a VERY BAD habit of treating bash and the Linux shell (or >> any/all shells) as the same thing. I know they are all very different, >> but for some reason I still keep calling all shells in general BASH. > > I seem to recall that BASH is an acronym, for Bourne Again SHell. > >> >> I will be sure to create aliases. I make alias commands so that it is >> easier to guess or remember a command. For instance, a Python user my >> want to clear the shell's history, but can only remember one form of the >> command or must guess. On my Ubuntu system, I have set up numerous shell >> aliases. I am addicted to aliases. > > Nothing wrong with aliases. But how does your user create his own > aliases? That's much more important than having yours available. > Remember that any new globals he defines are lost when he exits the > interpreter (or crashes). > >> >> I still need to add the other browsers. Do very many people use >> Iceweasel? >> >> I did not notice that I have "print(subprocess.Popen('(xterm &)'))" >> instead of "subprocess.Popen('(xterm &)')". The worst computer errors >> are ID-10-T errors. >> >> True, the user my have Xterm open, but what if they use Guake (like me) >> or Pterm, EvilVTE, Valaterm, Gnome-Terminal, Konsole, etc.? > > Exactly. If they're using a terminal with tabs, they might want to > create a new tab, not a new instance of the terminal. Or if they're > running screen or the equivalent, they want the new terminal to show > up on their (remote) console. > >> >> How could I add security and convenience? Okay, I will try to add wget. >> Are there any other shell commands that anyone feels I should add? > > How about tab? The command-completion and filename-completion and > parameter-completion logic of a good shell is extremely complex, and > very useful. > >> >> The point of this module is to allow Linux shell users to use Python3 as >> a regular shell. Instead of using CSH, Bash, Tcsh, FISH, etc., users >> could use Python3 and import this module. Python is more powerful than >> any shell, so I want to make it easier for anyone to use Python as the >> default shell. For instance, instead of typing "print(os.getcwd())" to >> get the current working directory, users could type "boash.ls()". I hope >> that is easier to remember than "print(os.getcwd())". > > It's easier, but doesn't begin to do the same thing. > > As for using this INSTEAD of csh, bash, etc., that might be likely > once it gets beyond 10% of the usefulness. Right now, it's at about > 0.01% And any habits a user gets within this environment have to be > unlearned when he returns to a standard shell. > > Back in the early days of MSDOS, the FORMAT command didn't require you > to specify a drive letter. So people could intend to format their > floppy, and actually trash their hard disk. So I had colleagues who > put a FORMAT.BAt command on their path which hard-wired the A: > parameter. Now what happens to one of those folks when he goes to > another machine which doesn't have that batch file? Ouch! Instead I > wrote a batch file that checked to make sure you had the A: > parameter. Rapidly, my fingers learned that FORMAT A: was the valid > syntax, and pretty soon the batch file was unnecessary (for me). > > If I were going to define a dozen aliases for other people to use, I'd > make them something like: > > def clear_hist(): > print "The function you want is probably clear_history()" > > >> As for the print() >> command, I do not like how os.getcwd() has single quotes around the >> output. > > Those quotes come from the Python interpreter, not from getcwd(). > >> Plus, Linux shell do not print output with quotes. >> >> I want to make this a very useful and popular module, so I will use the >> suggestions and add more useful wrappers. Would it help if I made a >> Youtube video showing how this module can be used? >> >> I will post the next version on this mailing list for another review. >> Thanks everyone, and thanks a lot Steven D'Aprano! >> > > Have you figured out how you're going to do things like cd (os.chdir), > which have to remember state? And how to pass the new current > directory to the shell that launched Python? > > Have you looked at ipython (ipython.org) ? At least from there, you > can get command completion with tab, one-third of the bash > functionality. So if you type boash.cle then it'll fill in > the rest. > > Thanks! I will look into IPython. I am familiar with it already. Yes, I have two purposes for the module, but after reading these suggestions I have modified my goal and purpose to achieve the goal of making a useful and popular Python3 module. The whole point of my boash project is to make a useful module. How can I make this module useful? I am fixing the problems and implementing suggestions. Would a Python3 game module be more useful? I plan to make a function that rolls a die and prints the output (You got a 5) and other similar random games. Mahalo, DCJ From rosuav at gmail.com Sat Jul 27 11:15:50 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 27 Jul 2013 16:15:50 +0100 Subject: Critic my module In-Reply-To: <51F3DEDC.2020303@Gmail.com> References: <51f334eb$0$29971$c3e8da3$5496439d@news.astraweb.com> <51F3C36A.4020400@Gmail.com> <51F3DEDC.2020303@Gmail.com> Message-ID: On Sat, Jul 27, 2013 at 3:53 PM, Devyn Collier Johnson wrote: > Would a Python3 game module be more useful? I plan to make a function that > rolls a die and prints the output (You got a 5) and other similar random > games. Taking someone else's module and learning to use it has a cost. Plus there's licensing and other issues (if you release your library GPL3, you force anyone who uses it to do the same - though I'm not 100% sure how that goes with Python modules, since they're not 'linked' the way others are), not to mention the time spent finding out that your module even exists. For a module to be useful, all those costs combined have to be lower than the cost of just writing the code yourself when you need it. On the other hand, it's VERY common for a programmer to have his own *personal* utilities module. Stuff stuff in there whenever you think it'll be useful, import it into your applications, et voila. The bar is way lower for that. Your dice-roller is perhaps useful to yourself, without being worth the effort for someone else to learn. Plus, you get to decide exactly how much flexibility you need. Do you only ever need to roll a single six-sider at a time? Then don't bother implementing stuff like I did for Minstrel Hall, where we play Dungeons and Dragons: [ROLL] Rosuav (Gaston crit dmg) rolls 4d8: 7, 3, 6, 1, totalling 17. [ROLL] Rosuav (Gaston crit dmg) rolls d6: 1 (elec) [ROLL] Rosuav (Gaston crit dmg) rolls d10: 6 (burst) [ROLL] Rosuav (Gaston crit dmg) rolls d8: 8 (thunder) [ROLL] For 4d8+12 STR+10 ench+4 specialization+d6 elec+d10 burst+d8 thunder+20 PA, Rosuav (Gaston crit dmg) totals: 78 Okay, that's a somewhat extreme example, but it's common to roll damage as, say, 2d6+13, which means two six-sided dice plus a constant 13. (This will result in damage between 15 and 25, with 20 being significantly more likely than either of the extremes.) And even that is probably a lot more complicated than you'll need for your purposes... yet for a D&D system, a dice roller that can only do a single d6 at a time is utterly useless. There's actually a dice roller module on PyPI already [1]; and it's probably of no use to you, because it's as complicated as I described above. I personally wouldn't use it, though, because I can't see a license - which comes back to the issues I listed above. Again, not an issue for your own code; if it's your copyright, you can do with it as you wish. [1] https://pypi.python.org/pypi/diceroll ChrisA From devyncjohnson at gmail.com Sat Jul 27 09:19:54 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Sat, 27 Jul 2013 09:19:54 -0400 Subject: Critic my module In-Reply-To: <51f334eb$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <51f334eb$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51F3C8FA.9040801@Gmail.com> On 07/26/2013 10:48 PM, Steven D'Aprano wrote: > As requested, some constructive criticism of your module. > > On Thu, 25 Jul 2013 09:24:30 -0400, Devyn Collier Johnson wrote: > >> #!/usr/bin/python3 >> #Made by Devyn Collier Johnson, NCLA, Linux+, LPIC-1, DCTS > What's NCLA, Linux+, LPIC-1, DCTS? Do these mean anything? Are we > supposed to know what they mean? > > "Made by" has no legal significance. You probably want: > > Copyright ? 2013 Devyn Collier Johnson. > > >> #Made using the Geany IDE > Nobody gives a monkey's toss what editor you used to type up the module. > You might as well mention the brand of monitor you used, or whether the > keyboard is Dvorak or Qwerty. > > >> #LGPLv3 > You can't just drop in a mention of "LGPLv3" and expect it to mean > anything. You actually have to obey the licence yourself, and that > includes *actually including the licence in your work*. (You're > technically in violation of the licence at the moment, however since the > only person whose copyright you are infringing is yourself, it doesn't > matter. However anyone else using your code is at risk.) > > http://www.gnu.org/licenses/gpl-howto.html > > In the case of the LGPL, you have to include the text of *both* the GPL > and the LGPL, not just one. > > > >> import re, sys, subprocess, platform >> def grep(regex,textf): >> #Sample Command: grep.grep("^x",dir()) #Syntax: >> boash.grep(regexp_string,list_of_strings_to_search) > Comments using # are only of use to people reading the source code. If > you want comments to be available at the interactive prompt, you should > write them as doc strings: > > def grep(regex, textf): > """This string is a docstring. > > Sample command: ... > Blah blah blah > """ > > Then, at the interactive prompt, the user can say: > > help(boash.grep) > > to read the docstring. > > >> version = '0.2a' > That's quite useless, since it is a local variable invisible outside of > the function. > > Also, why would you bother giving every individual function a version > number? That's rather pointless. The user cannot pick and choose function > A with version number 0.6 and function B with version number 0.7 if the > module provides versions 0.7 of both. > > >> expr = re.compile(regex) >> match = re.findall(expr, textf) >> if match != None: >> print(match) > When comparing with None, it is preferred to use "is" and "is not" rather > than equality tests. > > >> def ls(): >> version = '0.3' >> print(subprocess.getoutput('ls')) >> def dir(): >> version = '0.3' >> print(subprocess.getoutput('dir')) > A blank line or two between functions does wonders for readability. There > is no prize for conserving newlines. > > You might like to read PEP 8, the Python style guide. It is optional, but > still makes a very good guide. > > http://www.python.org/dev/peps/pep-0008/ > > >> def bash(*arg): >> version = '0.3' >> print(subprocess.getoutput(arg)) >> def shell(*arg): >> version = '0.3' >> print(subprocess.getoutput(arg)) > bash is not a synonym for "shell". "The shell" might be sh, csh, bash, or > any one of many other shells, all of which are slightly (or not so > slightly) different. > > >> def clear_bash_history(): >> version = '0.3' >> print(subprocess.getoutput('history -c')) > [...] > > Do you really need ten aliases for 'history -c'? > > If you want to define aliases for a function, don't recreate the entire > function ten times. Start with defining the function once, then: > > clear_bash_hist = clear_hist = clear_history = clear_bash_history > > etc. But really, having ten names for the one function just confuses > people, who then wonder what subtle difference there is between > delete_history and clear_history. > >> def firefox(): >> version = '0.3' >> print(subprocess.Popen('(firefox &)')) > Is Firefox really so important that it needs a dedicated command? > > What about Debian users? Doesn't Iceweasel get a command? > > >> def xterm(): >> version = '0.3' >> print(subprocess.Popen('(xterm &)')) > Surely the user already has an xterm open, if they are running this > interactively? Why not just use your xterm's "new window" or "new tab" > command? > > > [...delete more trivial calls to subprocess...] > >> def repeat_cmd(): >> version = '0.3' >> print(subprocess.Popen('!!')) > [... delete two exact copies of this function...] > >> def ejcd(): >> version = '0.3' >> print(subprocess.Popen('eject cdrom1')) > [... delete FOURTEEN exact copies of this function...] > > Really? Is anyone going to type "eject_disc_tray" instead of "eject"? > > > I think that will do. > > This doesn't really do anything except define a large number of trivial > wrappers to commands already available in the shell. Emphasis on the > *trivial* -- with the exception of the grep wrapper, which is all of four > lines (ignoring the useless internal version number), every single one of > these wrapper functions is a one-liner.[1] In other words, you're not > adding any value to the shell commands by wrapping them in Python. There > are plenty of big, complex shell commands that take a plethora of options > and could do with some useful Python wrappers, like wget. But you haven't > done them. > > Nor have you added extra security, or even extra convenience. You've done > nothing that couldn't be done using the shell "alias" command, except in > Python where the syntax is less convenient (e.g. "ls" in the shell, > versus "ls()" in Python). > > > > > [1] I think every newbie programmer goes through a stage of pointlessly > writing one-liner wrappers to every second function they see. I know I > did. The difference is, before the Internet, nobody did it publicly. > > About the aliases, I have tried setting pwd() as an alias for "os.getcwd()", but I cannot type "pwd()" and get the desired output. Instead, I must type "pwd". I tested this in Guake running Python3.3. >>> os.getcwd() '/home/collier' >>> pwd = os.getcwd() >>> pwd() Traceback (most recent call last): File "", line 1, in TypeError: 'str' object is not callable >>> pwd '/home/collier' >>> pwd() = os.getcwd() File "", line 1 SyntaxError: can't assign to function call How could I make pwd() work? Mahalo, DCJ From davea at davea.name Sat Jul 27 09:35:50 2013 From: davea at davea.name (Dave Angel) Date: Sat, 27 Jul 2013 09:35:50 -0400 Subject: Critic my module In-Reply-To: <51F3C8FA.9040801@Gmail.com> References: <51f334eb$0$29971$c3e8da3$5496439d@news.astraweb.com> <51F3C8FA.9040801@Gmail.com> Message-ID: On 07/27/2013 09:19 AM, Devyn Collier Johnson wrote: > > About the aliases, I have tried setting pwd() as an alias for > "os.getcwd()", but I cannot type "pwd()" and get the desired output. > Instead, I must type "pwd". I tested this in Guake running Python3.3. > > >>> os.getcwd() > '/home/collier' > >>> pwd = os.getcwd() > >>> pwd() > Traceback (most recent call last): > File "", line 1, in > TypeError: 'str' object is not callable > >>> pwd > '/home/collier' > >>> pwd() = os.getcwd() > File "", line 1 > SyntaxError: can't assign to function call > > > How could I make pwd() work? Don't call getcwd() when making the alias. You want it to be called when USING the alias. pwd = os.getcwd #creates the alias pwd() #runs the alias -- DaveA From devyncjohnson at gmail.com Sat Jul 27 09:44:24 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Sat, 27 Jul 2013 09:44:24 -0400 Subject: Critic my module In-Reply-To: References: <51f334eb$0$29971$c3e8da3$5496439d@news.astraweb.com> <51F3C8FA.9040801@Gmail.com> Message-ID: <51F3CEB8.2040107@Gmail.com> On 07/27/2013 09:35 AM, Dave Angel wrote: > On 07/27/2013 09:19 AM, Devyn Collier Johnson wrote: >> > > > >> About the aliases, I have tried setting pwd() as an alias for >> "os.getcwd()", but I cannot type "pwd()" and get the desired output. >> Instead, I must type "pwd". I tested this in Guake running Python3.3. >> >> >>> os.getcwd() >> '/home/collier' >> >>> pwd = os.getcwd() >> >>> pwd() >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: 'str' object is not callable >> >>> pwd >> '/home/collier' >> >>> pwd() = os.getcwd() >> File "", line 1 >> SyntaxError: can't assign to function call >> >> >> How could I make pwd() work? > > Don't call getcwd() when making the alias. You want it to be called > when USING the alias. > > pwd = os.getcwd #creates the alias > > pwd() #runs the alias > > > Thanks! It works! >>> pwd = os.getcwd >>> pwd() '/home/collier' Mahalo, Dave! DCJ From kwpolska at gmail.com Sat Jul 27 09:36:44 2013 From: kwpolska at gmail.com (=?UTF-8?B?Q2hyaXMg4oCcS3dwb2xza2HigJ0gV2Fycmljaw==?=) Date: Sat, 27 Jul 2013 15:36:44 +0200 Subject: Critic my module In-Reply-To: <51F3C8FA.9040801@Gmail.com> References: <51f334eb$0$29971$c3e8da3$5496439d@news.astraweb.com> <51F3C8FA.9040801@Gmail.com> Message-ID: On Sat, Jul 27, 2013 at 3:19 PM, Devyn Collier Johnson wrote: > About the aliases, I have tried setting pwd() as an alias for "os.getcwd()", > but I cannot type "pwd()" and get the desired output. Instead, I must type > "pwd". I tested this in Guake running Python3.3. > >>>> os.getcwd() > '/home/collier' >>>> pwd = os.getcwd() >>>> pwd() > Traceback (most recent call last): > File "", line 1, in > TypeError: 'str' object is not callable >>>> pwd > '/home/collier' >>>> pwd() = os.getcwd() > File "", line 1 > SyntaxError: can't assign to function call > > > How could I make pwd() work? > > > Mahalo, > > DCJ > -- > http://mail.python.org/mailman/listinfo/python-list >>> import os >>> pwd = os.getcwd >>> pwd() '/home/kwpolska' >>> os.chdir('/') >>> pwd() '/' >>> -- Chris ?Kwpolska? Warrick PGP: 5EAAEA16 stop html mail | always bottom-post | only UTF-8 makes sense From rosuav at gmail.com Sat Jul 27 09:37:25 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 27 Jul 2013 14:37:25 +0100 Subject: Critic my module In-Reply-To: <51F3C8FA.9040801@Gmail.com> References: <51f334eb$0$29971$c3e8da3$5496439d@news.astraweb.com> <51F3C8FA.9040801@Gmail.com> Message-ID: On Sat, Jul 27, 2013 at 2:19 PM, Devyn Collier Johnson wrote: > About the aliases, I have tried setting pwd() as an alias for "os.getcwd()", > but I cannot type "pwd()" and get the desired output. Instead, I must type > "pwd". I tested this in Guake running Python3.3. > >>>> os.getcwd() > '/home/collier' >>>> pwd = os.getcwd() >>>> pwd() Try: >>> pwd = os.getcwd Otherwise you're calling it immediately. ChrisA From alister.ware at ntlworld.com Sat Jul 27 12:32:38 2013 From: alister.ware at ntlworld.com (Alister) Date: Sat, 27 Jul 2013 16:32:38 GMT Subject: Critic my module References: <51f334eb$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sat, 27 Jul 2013 08:56:10 -0400, Devyn Collier Johnson wrote: > > Good point about the Made by/Copyright suggestion. Although, I have not > copyrighted the file, can I still say "Copyrighted by ...".-- There is no special process to Copyright anything. the simple act of writing it automatically gives you the copyright on your own work. Proving that it was you that created the work & when may be a little trickier if you do not find a reliable means of recording the event though. (posting to this news group actually gives a reasonable time stamp provided the article has not expired by the time it is needed) Stenderup's Law: The sooner you fall behind, the more time you will have to catch up. From davea at davea.name Sat Jul 27 12:58:11 2013 From: davea at davea.name (Dave Angel) Date: Sat, 27 Jul 2013 12:58:11 -0400 Subject: Critic my module In-Reply-To: References: <51f334eb$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 07/27/2013 12:32 PM, Alister wrote: > On Sat, 27 Jul 2013 08:56:10 -0400, Devyn Collier Johnson wrote: >> >> Good point about the Made by/Copyright suggestion. Although, I have not >> copyrighted the file, can I still say "Copyrighted by ...".-- > > There is no special process to Copyright anything. > the simple act of writing it automatically gives you the copyright on > your own work. > > Proving that it was you that created the work & when may be a little > trickier if you do not find a reliable means of recording the event > though. > (posting to this news group actually gives a reasonable time stamp > provided the article has not expired by the time it is needed) > The copyright law varies by country, and it's wise to look up your own country's rules, as well as investigate international law. The following is based only on my limited recollection of US law. I include references below, but have not re-studied them. Nor am I a lawyer. Generally, a copyright does belong to the author, as soon as he provably commits the text to medium. However, if you ever expect to defend a copyright, it can be useful to register it. Registration is limited by law to certain time limits. I don't recall exactly, but I believe that once something is published, it has to be registered within 3 months or so. Registration is easy, but not free. You can copyright a number of works simultaneously, but I think they have to be of the same type. And if it is registered, you can sue for larger amounts, and you'll have a much easier time getting a lawyer to take the case for you. See: http://www.copyright.gov/ and especially: http://www.copyright.gov/circs/circ01.pdf -- DaveA From js at globe.de Thu Jul 25 10:06:48 2013 From: js at globe.de (Schneider) Date: Thu, 25 Jul 2013 16:06:48 +0200 Subject: Critic my module In-Reply-To: <51F1270E.6000704@Gmail.com> References: <51F1270E.6000704@Gmail.com> Message-ID: <51F130F8.60801@globe.de> Hi, nice idea. mybe - for security reasons - you should ensure, that the right tool is called and not some tool put the path with the same name. bg, Johannes On Thu 25 Jul 2013 03:24:30 PM CEST, Devyn Collier Johnson wrote: > Aloha Python Users! > > I made a Python3 module that allows users to use certain Linux > shell commands from Python3 more easily than using os.system(), > subprocess.Popen(), or subprocess.getoutput(). This module (once > placed with the other modules) can be used like this > > import boash; boash.ls() > > I attached the module. I plan to release it on the Internet soon, > but feel free to use it now. It is licensed under LGPLv3. > > The name comes from combining "Boa" with "SHell". Notice that the > module's name almost looks like "BASH", a common Linux shell. The Boa > is a constrictor snake. This module makes Unix shells easier to use > via Python3. This brings the system shell closer to the Python shell. > > > Mahalo, > > Devyn Collier Johnson > DevynCJohnson at Gmail.com > > -- GLOBE Development GmbH K?nigsberger Strasse 260 48157 M?nsterGLOBE Development GmbH K?nigsberger Strasse 260 48157 M?nster 0251/5205 390 From devyncjohnson at gmail.com Fri Jul 26 06:39:47 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Fri, 26 Jul 2013 06:39:47 -0400 Subject: Critic my module In-Reply-To: <51F12EFB.3070906@globe.de> References: <51F1270E.6000704@Gmail.com> <51F12EFB.3070906@globe.de> Message-ID: <51F251F3.5030800@Gmail.com> On 07/25/2013 09:58 AM, Schneider wrote: > Hi, > > nice idea. > > mybe - for security reasons - you should ensure, that the right tool > is called and not some tool put the path with the same name. > > bg, > Johannes > > On Thu 25 Jul 2013 03:24:30 PM CEST, Devyn Collier Johnson wrote: >> Aloha Python Users! >> >> I made a Python3 module that allows users to use certain Linux >> shell commands from Python3 more easily than using os.system(), >> subprocess.Popen(), or subprocess.getoutput(). This module (once >> placed with the other modules) can be used like this >> >> import boash; boash.ls() >> >> I attached the module. I plan to release it on the Internet soon, >> but feel free to use it now. It is licensed under LGPLv3. >> >> The name comes from combining "Boa" with "SHell". Notice that the >> module's name almost looks like "BASH", a common Linux shell. The Boa >> is a constrictor snake. This module makes Unix shells easier to use >> via Python3. This brings the system shell closer to the Python shell. >> >> >> Mahalo, >> >> Devyn Collier Johnson >> DevynCJohnson at Gmail.com >> >> > > > > -- > GLOBE Development GmbH > K?nigsberger Strasse 260 > 48157 M?nsterGLOBE Development GmbH > K?nigsberger Strasse 260 > 48157 M?nster > 0251/5205 390 What do you mean by that Schneider? Mahalo, Devyn Collier Johnson DevynCJohnson at Gmail.com From js at globe.de Mon Jul 29 06:58:16 2013 From: js at globe.de (Schneider) Date: Mon, 29 Jul 2013 12:58:16 +0200 Subject: Critic my module In-Reply-To: <51F251F3.5030800@Gmail.com> References: <51F1270E.6000704@Gmail.com> <51F12EFB.3070906@globe.de> <51F251F3.5030800@Gmail.com> Message-ID: <51F64AC8.2030003@globe.de> Hi, lets uses the ls example: the way you do it now implies, that you search your PATH variable until it finds a program called 'ls'. So if we are able to change the PATH variable, and put out own 'ls' somewhere in the (new) paths, calling you ls() will execute whatever we want our own ls' to do. Second remark: if the behavior of some tools is changed (for examples with using aliases) we cannot expect the called tool (in the example: 'ls') to give the same output on every system. this can be avoided (mostly) by ensuring that the right program (in the example /bin/ls) is called, and not only ls. bg, Johannes On 07/26/2013 12:39 PM, Devyn Collier Johnson wrote: > > On 07/25/2013 09:58 AM, Schneider wrote: >> Hi, >> >> nice idea. >> >> mybe - for security reasons - you should ensure, that the right tool >> is called and not some tool put the path with the same name. >> >> bg, >> Johannes >> >> Devyn Collier Johnson On Thu 25 Jul 2013 >> 03:24:30 PM CEST, Devyn Collier Johnson wrote: >>> Aloha Python Users! >>> >>> I made a Python3 module that allows users to use certain Linux >>> shell commands from Python3 more easily than using os.system(), >>> subprocess.Popen(), or subprocess.getoutput(). This module (once >>> placed with the other modules) can be used like this >>> >>> import boash; boash.ls() >>> >>> I attached the module. I plan to release it on the Internet soon, >>> but feel free to use it now. It is licensed under LGPLv3. >>> >>> The name comes from combining "Boa" with "SHell". Notice that the >>> module's name almost looks like "BASH", a common Linux shell. The Boa >>> is a constrictor snake. This module makes Unix shells easier to use >>> via Python3. This brings the system shell closer to the Python shell. >>> >>> >>> Mahalo, >>> >>> Devyn Collier Johnson >>> DevynCJohnson at Gmail.com >>> >>> >> >> >> >> -- >> GLOBE Development GmbH >> K?nigsberger Strasse 260 >> 48157 M?nsterGLOBE Development GmbH >> K?nigsberger Strasse 260 >> 48157 M?nster >> 0251/5205 390 > > What do you mean by that Schneider? > > Mahalo, > > Devyn Collier Johnson > DevynCJohnson at Gmail.com -- GLOBE Development GmbH K?nigsberger Strasse 260 48157 M?nsterGLOBE Development GmbH K?nigsberger Strasse 260 48157 M?nster 0251/5205 390 From devyncjohnson at gmail.com Mon Jul 29 07:21:14 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Mon, 29 Jul 2013 07:21:14 -0400 Subject: Critic my module In-Reply-To: <51F64AC8.2030003@globe.de> References: <51F1270E.6000704@Gmail.com> <51F12EFB.3070906@globe.de> <51F251F3.5030800@Gmail.com> <51F64AC8.2030003@globe.de> Message-ID: <51F6502A.8080605@Gmail.com> On 07/29/2013 06:58 AM, Schneider wrote: > Hi, > > lets uses the ls example: > the way you do it now implies, that you search your PATH variable > until it finds a > program called 'ls'. So if we are able to change the PATH variable, > and put out own > 'ls' somewhere in the (new) paths, calling you ls() will execute > whatever we want our > own ls' to do. > > Second remark: if the behavior of some tools is changed (for examples > with using aliases) > we cannot expect the called tool (in the example: 'ls') to give the > same output on > every system. > > this can be avoided (mostly) by ensuring that the right program (in > the example /bin/ls) is called, and not only ls. > > bg, > Johannes > > On 07/26/2013 12:39 PM, Devyn Collier Johnson wrote: >> >> On 07/25/2013 09:58 AM, Schneider wrote: >>> Hi, >>> >>> nice idea. >>> >>> mybe - for security reasons - you should ensure, that the right >>> tool is called and not some tool put the path with the same name. >>> >>> bg, >>> Johannes >>> >>> Devyn Collier Johnson On Thu 25 Jul 2013 >>> 03:24:30 PM CEST, Devyn Collier Johnson wrote: >>>> Aloha Python Users! >>>> >>>> I made a Python3 module that allows users to use certain Linux >>>> shell commands from Python3 more easily than using os.system(), >>>> subprocess.Popen(), or subprocess.getoutput(). This module (once >>>> placed with the other modules) can be used like this >>>> >>>> import boash; boash.ls() >>>> >>>> I attached the module. I plan to release it on the Internet soon, >>>> but feel free to use it now. It is licensed under LGPLv3. >>>> >>>> The name comes from combining "Boa" with "SHell". Notice that the >>>> module's name almost looks like "BASH", a common Linux shell. The Boa >>>> is a constrictor snake. This module makes Unix shells easier to use >>>> via Python3. This brings the system shell closer to the Python shell. >>>> >>>> >>>> Mahalo, >>>> >>>> Devyn Collier Johnson >>>> DevynCJohnson at Gmail.com >>>> >>>> >>> >>> >>> >>> -- >>> GLOBE Development GmbH >>> K?nigsberger Strasse 260 >>> 48157 M?nsterGLOBE Development GmbH >>> K?nigsberger Strasse 260 >>> 48157 M?nster >>> 0251/5205 390 >> >> What do you mean by that Schneider? >> >> Mahalo, >> >> Devyn Collier Johnson >> DevynCJohnson at Gmail.com > > Thanks, good point. I will fix that. BTW, we bottom post on this mailing list. Mahalo, DCJ From lele at metapensiero.it Mon Jul 29 14:36:10 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Mon, 29 Jul 2013 20:36:10 +0200 Subject: Critic my module References: <51F1270E.6000704@Gmail.com> <51F12EFB.3070906@globe.de> <51F251F3.5030800@Gmail.com> <51F64AC8.2030003@globe.de> <51F6502A.8080605@Gmail.com> Message-ID: <87txjdi57p.fsf@nautilus.nautilus> This thread did not mention alternative and existing modules with (almost) the same goal, two come to mind: * https://pypi.python.org/pypi/sh * https://pypi.python.org/pypi/sarge bye, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From devyncjohnson at gmail.com Mon Jul 29 15:19:03 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Mon, 29 Jul 2013 15:19:03 -0400 Subject: Critic my module In-Reply-To: <87txjdi57p.fsf@nautilus.nautilus> References: <51F1270E.6000704@Gmail.com> <51F12EFB.3070906@globe.de> <51F251F3.5030800@Gmail.com> <51F64AC8.2030003@globe.de> <51F6502A.8080605@Gmail.com> <87txjdi57p.fsf@nautilus.nautilus> Message-ID: <51F6C027.1050001@Gmail.com> On 07/29/2013 02:36 PM, Lele Gaifax wrote: > This thread did not mention alternative and existing modules with > (almost) the same goal, two come to mind: > > * https://pypi.python.org/pypi/sh > * https://pypi.python.org/pypi/sarge > > bye, lele. Thanks everyone for the feedback. Clearly, I should stop my project. I will work on something else now (^u^)! Mahalo, DCJ From joshua at landau.ws Mon Jul 29 17:17:24 2013 From: joshua at landau.ws (Joshua Landau) Date: Mon, 29 Jul 2013 22:17:24 +0100 Subject: Critic my module In-Reply-To: <87txjdi57p.fsf@nautilus.nautilus> References: <51F1270E.6000704@Gmail.com> <51F12EFB.3070906@globe.de> <51F251F3.5030800@Gmail.com> <51F64AC8.2030003@globe.de> <51F6502A.8080605@Gmail.com> <87txjdi57p.fsf@nautilus.nautilus> Message-ID: On 29 July 2013 19:36, Lele Gaifax wrote: > This thread did not mention alternative and existing modules with > (almost) the same goal, two come to mind: > > * https://pypi.python.org/pypi/sh > * https://pypi.python.org/pypi/sarge Actually, I noted both those and plumbum. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jason.swails at gmail.com Sat Jul 27 11:14:57 2013 From: jason.swails at gmail.com (Jason Swails) Date: Sat, 27 Jul 2013 11:14:57 -0400 Subject: Critic my module In-Reply-To: <51F1270E.6000704@Gmail.com> References: <51F1270E.6000704@Gmail.com> Message-ID: You've gotten plenty of good advice from people discussing the coding and coding style itself, I'll provide some feedback from the vantage point of a perspective user. On Thu, Jul 25, 2013 at 9:24 AM, Devyn Collier Johnson < devyncjohnson at gmail.com> wrote: > Aloha Python Users! > > I made a Python3 module that allows users to use certain Linux shell > commands from Python3 more easily than using os.system(), > subprocess.Popen(), or subprocess.getoutput(). This module (once placed > with the other modules) can be used like this > > import boash; boash.ls() > I actually wrote a program recently in which I wanted access to unix "ls" command, and I wanted it to behave as close to the real, UNIX "ls" as possible. This would seem like a perfect use-case for your module, but the problem is that the 'ls' command in your module does not behave much like the real 'ls' command. You never let any of the 'system' commands in your module access any arguments. More often than not, I use "ls" with several command-line arguments, like: ls --color=auto -lthr dir_basename*/ Even if you're just spawning 'ls' directly, this is actually non-trivial to implement. You need globbing on all non-option arguments, you may want to pass up the return code somehow, depending on what the user wants to do: [bash ]$ ls nodir ls: nodir: No such file or directory [bash ]$ echo $? 1 Also, 'ls' in the terminal behaves like "ls -C" when called from your module. In the framework of my program, my 'ls' command looks like this: class ls(Action): """ Lists directory contents. Like UNIX 'ls' """ needs_parm = False def init(self, arg_list): from glob import glob self.args = [] # Process the argument list to mimic the real ls as much as possible while True: try: arg = arg_list.get_next_string() if not arg.startswith('-'): # Glob this argument globarg = glob(arg) if len(globarg) > 0: self.args.extend(globarg) else: self.args.append(arg) else: self.args.append(arg) except NoArgument: break def __str__(self): from subprocess import Popen, PIPE process = Popen(['/bin/ls', '-C'] + self.args, stdout=PIPE, stderr=PIPE) out, err = process.communicate('') process.wait() return out + err [I have omitted the Action base class, which processes the user command-line arguments and passes it to the init() method in arg_list -- this listing was just to give you a basic idea of the complexity of getting a true-er 'ls' command]. Your 'uname' command is likewise limited (and the printout looks strange: >>> print(platform.uname()) ('Linux', 'Batman', '3.3.8-gentoo', '#1 SMP Fri Oct 5 14:14:57 EDT 2012', 'x86_64', 'AMD FX(tm)-6100 Six-Core Processor') Whereas: [bash $] uname -a Linux Batman 3.3.8-gentoo #1 SMP Fri Oct 5 14:14:57 EDT 2012 x86_64 AMD FX(tm)-6100 Six-Core Processor AuthenticAMD GNU/Linux You may want to change that to: def uname(): print(' '.join(platform.uname())) Although again, oftentimes people want only something specific from uname (like -m or -n). HTH, Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: From devyncjohnson at gmail.com Sat Jul 27 14:06:34 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Sat, 27 Jul 2013 14:06:34 -0400 Subject: Critic my module In-Reply-To: References: <51F1270E.6000704@Gmail.com> Message-ID: <51F40C2A.3090306@Gmail.com> On 07/27/2013 11:14 AM, Jason Swails wrote: > You've gotten plenty of good advice from people discussing the coding > and coding style itself, I'll provide some feedback from the vantage > point of a perspective user. > > > On Thu, Jul 25, 2013 at 9:24 AM, Devyn Collier Johnson > > wrote: > > Aloha Python Users! > > I made a Python3 module that allows users to use certain Linux > shell commands from Python3 more easily than using os.system(), > subprocess.Popen(), or subprocess.getoutput(). This module (once > placed with the other modules) can be used like this > > import boash; boash.ls () > > > I actually wrote a program recently in which I wanted access to unix > "ls" command, and I wanted it to behave as close to the real, UNIX > "ls" as possible. > > This would seem like a perfect use-case for your module, but the > problem is that the 'ls' command in your module does not behave much > like the real 'ls' command. You never let any of the 'system' > commands in your module access any arguments. More often than not, I > use "ls" with several command-line arguments, like: > > ls --color=auto -lthr dir_basename*/ > > Even if you're just spawning 'ls' directly, this is actually > non-trivial to implement. You need globbing on all non-option > arguments, you may want to pass up the return code somehow, depending > on what the user wants to do: > > [bash ]$ ls nodir > ls: nodir: No such file or directory > [bash ]$ echo $? > 1 > > Also, 'ls' in the terminal behaves like "ls -C" when called from your > module. In the framework of my program, my 'ls' command looks like this: > > class ls(Action): > """ > Lists directory contents. Like UNIX 'ls' > """ > needs_parm = False > def init(self, arg_list): > from glob import glob > self.args = [] > # Process the argument list to mimic the real ls as much as possible > while True: > try: > arg = arg_list.get_next_string() > if not arg.startswith('-'): > # Glob this argument > globarg = glob(arg) > if len(globarg) > 0: > self.args.extend(globarg) > else: > self.args.append(arg) > else: > self.args.append(arg) > except NoArgument: > break > > def __str__(self): > from subprocess import Popen, PIPE > process = Popen(['/bin/ls', '-C'] + self.args, stdout=PIPE, > stderr=PIPE) > out, err = process.communicate('') > process.wait() > return out + err > > [I have omitted the Action base class, which processes the user > command-line arguments and passes it to the init() method in arg_list > -- this listing was just to give you a basic idea of the complexity of > getting a true-er 'ls' command]. > > Your 'uname' command is likewise limited (and the printout looks strange: > > >>> print(platform.uname()) > ('Linux', 'Batman', '3.3.8-gentoo', '#1 SMP Fri Oct 5 14:14:57 EDT > 2012', 'x86_64', 'AMD FX(tm)-6100 Six-Core Processor') > > Whereas: > > [bash $] uname -a > Linux Batman 3.3.8-gentoo #1 SMP Fri Oct 5 14:14:57 EDT 2012 x86_64 > AMD FX(tm)-6100 Six-Core Processor AuthenticAMD GNU/Linux > > You may want to change that to: > > def uname(): > print(' '.join(platform.uname())) > > Although again, oftentimes people want only something specific from > uname (like -m or -n). > > HTH, > Jason > For now, I will decide if it would be worth my time to make such a module seeing that many feel that it may not be useful. Thank you all for your feedback. Mahalo, DCJ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsf80238 at gmail.com Sat Jul 27 23:46:37 2013 From: jsf80238 at gmail.com (Jason Friedman) Date: Sat, 27 Jul 2013 21:46:37 -0600 Subject: Critic my module In-Reply-To: <51F1270E.6000704@Gmail.com> References: <51F1270E.6000704@Gmail.com> Message-ID: > > > I made a Python3 module that allows users to use certain Linux shell > commands from Python3 more easily than using os.system(), > subprocess.Popen(), or subprocess.getoutput(). This module (once placed > with the other modules) can be used like this > > Looks similar to https://pypi.python.org/pypi/sh. -------------- next part -------------- An HTML attachment was scrubbed... URL: From devyncjohnson at gmail.com Thu Jul 25 09:39:22 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Thu, 25 Jul 2013 09:39:22 -0400 Subject: Cross-Platform Python3 Equivalent to notify-send Message-ID: <51F12A8A.4050103@Gmail.com> Linux systems with the proper software can use the "notify-send" command. Is there a cross-platform Python3 equivalent? Mahalo, Devyn Collier Johnson DevynCJohnson at Gmail.com From devyncjohnson at gmail.com Thu Jul 25 09:42:25 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Thu, 25 Jul 2013 09:42:25 -0400 Subject: Python Script Hashplings Message-ID: <51F12B41.3050004@Gmail.com> If I execute a Python3 script with this haspling (#!/usr/bin/python3.3) and Python3.3 is not installed, but Python3.2 is installed, would the script still work? Would it fall back to Python3.2? I hope Dihedral is listening. I would like to see another response from HIM. Mahalo, DCJ From python at mrabarnett.plus.com Thu Jul 25 09:54:57 2013 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 25 Jul 2013 14:54:57 +0100 Subject: Python Script Hashplings In-Reply-To: <51F12B41.3050004@Gmail.com> References: <51F12B41.3050004@Gmail.com> Message-ID: <51F12E31.1070502@mrabarnett.plus.com> On 25/07/2013 14:42, Devyn Collier Johnson wrote: > If I execute a Python3 script with this haspling (#!/usr/bin/python3.3) > and Python3.3 is not installed, but Python3.2 is installed, would the > script still work? Would it fall back to Python3.2? > Why don't you try it? > I hope Dihedral is listening. I would like to see another response from HIM. > From mclefavor at gmail.com Thu Jul 25 10:01:33 2013 From: mclefavor at gmail.com (Matthew Lefavor) Date: Thu, 25 Jul 2013 10:01:33 -0400 Subject: Python Script Hashplings In-Reply-To: <51F12E31.1070502@mrabarnett.plus.com> References: <51F12B41.3050004@Gmail.com> <51F12E31.1070502@mrabarnett.plus.com> Message-ID: The answer is "probably not." If you just want to use the latest version of Python 3 you have installed on your system, use: "#!/usr/bin/python3". When you use the specific minor version numbers, they point to that specific minor version. Actually, the preferred shebang line is of the form: "#!/usr/bin/env python3". This way the end users can override the interpreter with, say, a virtualenv, rather than being stuck with the system default. On Thu, Jul 25, 2013 at 9:54 AM, MRAB wrote: > On 25/07/2013 14:42, Devyn Collier Johnson wrote: > >> If I execute a Python3 script with this haspling (#!/usr/bin/python3.3) >> and Python3.3 is not installed, but Python3.2 is installed, would the >> script still work? Would it fall back to Python3.2? >> >> Why don't you try it? > > > I hope Dihedral is listening. I would like to see another response from >> HIM. >> >> > -- > http://mail.python.org/**mailman/listinfo/python-list > -------------- next part -------------- An HTML attachment was scrubbed... URL: From devyncjohnson at gmail.com Fri Jul 26 06:41:29 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Fri, 26 Jul 2013 06:41:29 -0400 Subject: Python Script Hashplings In-Reply-To: References: <51F12B41.3050004@Gmail.com> <51F12E31.1070502@mrabarnett.plus.com> Message-ID: <51F25259.7010909@Gmail.com> On 07/25/2013 10:01 AM, Matthew Lefavor wrote: > The answer is "probably not." If you just want to use the latest > version of Python 3 you have installed on your system, use: > "#!/usr/bin/python3". When you use the specific minor version numbers, > they point to that specific minor version. > > Actually, the preferred shebang line is of the form: "#!/usr/bin/env > python3". This way the end users can override the interpreter with, > say, a virtualenv, rather than being stuck with the system default. > > > On Thu, Jul 25, 2013 at 9:54 AM, MRAB > wrote: > > On 25/07/2013 14:42, Devyn Collier Johnson wrote: > > If I execute a Python3 script with this haspling > (#!/usr/bin/python3.3) > and Python3.3 is not installed, but Python3.2 is installed, > would the > script still work? Would it fall back to Python3.2? > > Why don't you try it? > > > I hope Dihedral is listening. I would like to see another > response from HIM. > > > -- > http://mail.python.org/mailman/listinfo/python-list > > > > Thanks Matthew Lefavor! But specifically, why use "#!/usr/bin/env python3" instead of "#!/usr/bin/python3"? Mahalo, DCJ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mclefavor at gmail.com Fri Jul 26 10:58:24 2013 From: mclefavor at gmail.com (Matthew Lefavor) Date: Fri, 26 Jul 2013 10:58:24 -0400 Subject: Python Script Hashplings In-Reply-To: <51F25259.7010909@Gmail.com> References: <51F12B41.3050004@Gmail.com> <51F12E31.1070502@mrabarnett.plus.com> <51F25259.7010909@Gmail.com> Message-ID: > > > Thanks Matthew Lefavor! But specifically, why use "#!/usr/bin/env python3" > instead of "#!/usr/bin/python3"? > The "env" program looks up its argument in the current $PATH environment variable, and then executes that. This means you aren't necessarily tied to /usr/bin/python3. It makes things more portable. For example, old Linux distributions don't have Python 3 installed with them, and the user might not have permissions to install Python 3 system-wide. Instead they have it in some sort of ~/HOME/bin directory, and then that is placed on the path by their .bashrc file. If your shebang line was "#!/usr/bin/python3", the program wouldn't work without them changing that line. If the shebang ling was "#!/usr/bin/env python3", it would find the Python3 binary no problem. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ramit.prasad at jpmorgan.com Fri Jul 26 10:51:50 2013 From: ramit.prasad at jpmorgan.com (Prasad, Ramit) Date: Fri, 26 Jul 2013 14:51:50 +0000 Subject: Python Script Hashplings In-Reply-To: <51F25259.7010909@Gmail.com> References: <51F12B41.3050004@Gmail.com> <51F12E31.1070502@mrabarnett.plus.com> <51F25259.7010909@Gmail.com> Message-ID: <5B80DD153D7D744689F57F4FB69AF474185E65EF@SCACMX008.exchad.jpmchase.net> Devyn Collier Johnson wrote: > Thanks Matthew Lefavor! But specifically, why use "#!/usr/bin/env python3" instead of > "#!/usr/bin/python3"? > > Mahalo, > > DCJ I believe this will work on Windows for Python 3.3+ and also with virtualenv. https://pypi.python.org/pypi/virtualenv Virtualenv is highly recommended as it lets you create isolated Python environments on a per project basis. ~Ramit This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email. From devyncjohnson at gmail.com Fri Jul 26 06:37:49 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Fri, 26 Jul 2013 06:37:49 -0400 Subject: Python Script Hashplings In-Reply-To: <51F12E31.1070502@mrabarnett.plus.com> References: <51F12B41.3050004@Gmail.com> <51F12E31.1070502@mrabarnett.plus.com> Message-ID: <51F2517D.2070009@Gmail.com> On 07/25/2013 09:54 AM, MRAB wrote: > On 25/07/2013 14:42, Devyn Collier Johnson wrote: >> If I execute a Python3 script with this haspling (#!/usr/bin/python3.3) >> and Python3.3 is not installed, but Python3.2 is installed, would the >> script still work? Would it fall back to Python3.2? >> > Why don't you try it? > >> I hope Dihedral is listening. I would like to see another response >> from HIM. >> > Good point, but if it falls back to Python3.2, how would I know? Plus, I have Python3.3, 3.2, and 2.7 installed. I cannot uninstall them due to dependencies. DCJ From rosuav at gmail.com Fri Jul 26 06:43:07 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 26 Jul 2013 11:43:07 +0100 Subject: Python Script Hashplings In-Reply-To: <51F2517D.2070009@Gmail.com> References: <51F12B41.3050004@Gmail.com> <51F12E31.1070502@mrabarnett.plus.com> <51F2517D.2070009@Gmail.com> Message-ID: On Fri, Jul 26, 2013 at 11:37 AM, Devyn Collier Johnson wrote: > > On 07/25/2013 09:54 AM, MRAB wrote: >> >> On 25/07/2013 14:42, Devyn Collier Johnson wrote: >>> >>> If I execute a Python3 script with this haspling (#!/usr/bin/python3.3) >>> and Python3.3 is not installed, but Python3.2 is installed, would the >>> script still work? Would it fall back to Python3.2? >>> >> Why don't you try it? >> >>> I hope Dihedral is listening. I would like to see another response from >>> HIM. >>> >> > Good point, but if it falls back to Python3.2, how would I know? Plus, I > have Python3.3, 3.2, and 2.7 installed. I cannot uninstall them due to > dependencies. Easy: #!/usr/bin/python3.3 import sys print(sys.version) Now run that on lots of different computers (virtual computers work well for this). ChrisA From python at mrabarnett.plus.com Fri Jul 26 09:53:54 2013 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 26 Jul 2013 14:53:54 +0100 Subject: Python Script Hashplings In-Reply-To: References: <51F12B41.3050004@Gmail.com> <51F12E31.1070502@mrabarnett.plus.com> <51F2517D.2070009@Gmail.com> Message-ID: <51F27F72.3020807@mrabarnett.plus.com> On 26/07/2013 11:43, Chris Angelico wrote: > On Fri, Jul 26, 2013 at 11:37 AM, Devyn Collier Johnson > wrote: >> >> On 07/25/2013 09:54 AM, MRAB wrote: >>> >>> On 25/07/2013 14:42, Devyn Collier Johnson wrote: >>>> >>>> If I execute a Python3 script with this haspling (#!/usr/bin/python3.3) >>>> and Python3.3 is not installed, but Python3.2 is installed, would the >>>> script still work? Would it fall back to Python3.2? >>>> >>> Why don't you try it? >>> >>>> I hope Dihedral is listening. I would like to see another response from >>>> HIM. >>>> >>> >> Good point, but if it falls back to Python3.2, how would I know? Plus, I >> have Python3.3, 3.2, and 2.7 installed. I cannot uninstall them due to >> dependencies. > > Easy: > > #!/usr/bin/python3.3 > import sys > print(sys.version) > > Now run that on lots of different computers (virtual computers work > well for this). > There's also sys.version_info: >>> import sys >>> sys.version_info sys.version_info(major=3, minor=3, micro=2, releaselevel='final', serial=0) If you want to test what would happen if that version wasn't installed, set the shebang line to a future version, such as Python 3.4. I doubt you have that installed! :-) From rosuav at gmail.com Fri Jul 26 10:14:59 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 26 Jul 2013 15:14:59 +0100 Subject: Python Script Hashplings In-Reply-To: <51F27F72.3020807@mrabarnett.plus.com> References: <51F12B41.3050004@Gmail.com> <51F12E31.1070502@mrabarnett.plus.com> <51F2517D.2070009@Gmail.com> <51F27F72.3020807@mrabarnett.plus.com> Message-ID: On Fri, Jul 26, 2013 at 2:53 PM, MRAB wrote: > If you want to test what would happen if that version wasn't installed, > set the shebang line to a future version, such as Python 3.4. I doubt > you have that installed! :-) Be careful, some people DO have a python3.4 binary :) Go for 3.5 for a bit more reliability. rosuav at sikorsky:~$ python3.4 3.4.0a0 (default:da7d97ca1ef6, Jul 13 2013, 14:05:08) [GCC 4.7.2] ChrisA From devyncjohnson at gmail.com Fri Jul 26 19:33:10 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Fri, 26 Jul 2013 19:33:10 -0400 Subject: Python Script Hashplings In-Reply-To: References: <51F12B41.3050004@Gmail.com> <51F12E31.1070502@mrabarnett.plus.com> <51F2517D.2070009@Gmail.com> <51F27F72.3020807@mrabarnett.plus.com> Message-ID: <51F30736.6070706@Gmail.com> On 07/26/2013 10:14 AM, Chris Angelico wrote: > On Fri, Jul 26, 2013 at 2:53 PM, MRAB wrote: >> If you want to test what would happen if that version wasn't installed, >> set the shebang line to a future version, such as Python 3.4. I doubt >> you have that installed! :-) > Be careful, some people DO have a python3.4 binary :) Go for 3.5 for a > bit more reliability. > > rosuav at sikorsky:~$ python3.4 > 3.4.0a0 (default:da7d97ca1ef6, Jul 13 2013, 14:05:08) > [GCC 4.7.2] > > ChrisA Thank you everyone for your comments and suggestions. I made a script to test the question. see for yourself: SCRIPT: #!/usr/bin/env python3.5 import sys; print(sys.version) OUTPUT IN TERMINAL: collier at Nacho-Laptop:~$ ./test.py /usr/bin/env: python3.5: No such file or directory Mahalo, DCJ From mail at timgolden.me.uk Fri Jul 26 06:47:24 2013 From: mail at timgolden.me.uk (Tim Golden) Date: Fri, 26 Jul 2013 11:47:24 +0100 Subject: Python Script Hashplings In-Reply-To: <51F2517D.2070009@Gmail.com> References: <51F12B41.3050004@Gmail.com> <51F12E31.1070502@mrabarnett.plus.com> <51F2517D.2070009@Gmail.com> Message-ID: <51F253BC.1010709@timgolden.me.uk> On 26/07/2013 11:37, Devyn Collier Johnson wrote: > > On 07/25/2013 09:54 AM, MRAB wrote: >> On 25/07/2013 14:42, Devyn Collier Johnson wrote: >>> If I execute a Python3 script with this haspling (#!/usr/bin/python3.3) >>> and Python3.3 is not installed, but Python3.2 is installed, would the >>> script still work? Would it fall back to Python3.2? >>> >> Why don't you try it? >> >>> I hope Dihedral is listening. I would like to see another response >>> from HIM. >>> >> > Good point, but if it falls back to Python3.2, how would I know? Plus, I > have Python3.3, 3.2, and 2.7 installed. I cannot uninstall them due to > dependencies. Devyn, I'm not a *nix person so someone can point out if I'm wrong, but my understanding is that the shebang line (or whatever you want to call it) just tells the shell: run this command to run this file. So you can put "#!/usr/bin/fish-and-chips" as the first line and it will try to run the file using /usr/bin/fish-and-chips. If you put #!/usr/bin/python3.3 the shell will use the executable /usr/bin/python3.3. It doesn't know or care about Python or its versions: it won't go looking for some alternative binary. If /usr/bin/python3.3 isn't there, the shell will fail to run the code with some kind of error message. If you or your package manager symlink /usr/bin/python3 to whatever the latest Python 3.x is on your system then you can safely use /usr/bin/python3 throughout and let the symlink do the work! TJG From ben+python at benfinney.id.au Fri Jul 26 18:08:26 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 27 Jul 2013 08:08:26 +1000 Subject: Python Script Hashplings References: <51F12B41.3050004@Gmail.com> <51F12E31.1070502@mrabarnett.plus.com> <51F2517D.2070009@Gmail.com> <51F253BC.1010709@timgolden.me.uk> Message-ID: <7w38r1t1np.fsf@benfinney.id.au> Tim Golden writes: > Devyn, I'm not a *nix person so someone can point out if I'm wrong, > but my understanding is that the shebang line (or whatever you want to > call it) just tells the shell: run this command to run this file. So > you can put "#!/usr/bin/fish-and-chips" as the first line and it will > try to run the file using /usr/bin/fish-and-chips. Close: it's an instruction not to the shell, but to the kernel. The shell defers any ?run the program in this file? to the kernel, and it's the kernel that pays attention to the file's shebang line. -- \ ?When cryptography is outlawed, bayl bhgynjf jvyy unir | `\ cevinpl.? ?Anonymous | _o__) | Ben Finney From gouzounakis at hotmail.com Thu Jul 25 09:48:09 2013 From: gouzounakis at hotmail.com (D. Xenakis) Date: Thu, 25 Jul 2013 06:48:09 -0700 (PDT) Subject: virtualenv problem Message-ID: <626bd852-a79e-42d7-aaa9-c5cba477d155@googlegroups.com> Hi there. Im using windows 7 64bit I have installed python 3.3.2 in C:\Python33 and then easy_install , pip, and virtualenv. But i do not know if the virtualenv installation is correct as i cant seem to be able to create any virtual enviroment yet. How can i check if everything is correct? What exactly should i do to create a virtual enviroment into my new_project folder located here: in C:\new_project ? I think there is something wrong with the installation because when i run through idle the virtual-env scripts located in "C:\Python33\Scripts" then i get the following.. Python 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:06:53) [MSC v.1600 64 bit (AMD64)] on win32 Type "copyright", "credits" or "license()" for more information. >>> ================================ RESTART ================================ >>> You must provide a DEST_DIR Usage: virtualenv-3.3-script.py [OPTIONS] DEST_DIR Options: --version show program's version number and exit -h, --help show this help message and exit -v, --verbose Increase verbosity -q, --quiet Decrease verbosity -p PYTHON_EXE, --python=PYTHON_EXE The Python interpreter to use, e.g., --python=python2.5 will use the python2.5 interpreter to create the new environment. The default is the interpreter that virtualenv was installed with (C:\Python33\pythonw.exe) --clear Clear out the non-root install and start from scratch --no-site-packages Don't give access to the global site-packages dir to the virtual environment (default) --system-site-packages Give access to the global site-packages dir to the virtual environment --always-copy Always copy files rather than symlinking --unzip-setuptools Unzip Setuptools when installing it --relocatable Make an EXISTING virtualenv environment relocatable. This fixes up scripts and makes all .pth files relative --no-setuptools Do not install setuptools (or pip) in the new virtualenv. --no-pip Do not install pip in the new virtualenv. --extra-search-dir=SEARCH_DIRS Directory to look for setuptools/pip distributions in. You can add any number of additional --extra-search- dir paths. --never-download Never download anything from the network. This is now always the case. The option is only retained for backward compatibility, and does nothing. Virtualenv will fail if local distributions of setuptools/pip are not present. --prompt=PROMPT Provides an alternative prompt prefix for this environment --setuptools Backward compatibility. Does nothing. --distribute Backward compatibility. Does nothing. Traceback (most recent call last): File "C:\Python33\Scripts\virtualenv-3.3-script.py", line 9, in load_entry_point('virtualenv==1.10', 'console_scripts', 'virtualenv-3.3')() File "C:\Python33\lib\site-packages\virtualenv.py", line 786, in main sys.exit(2) SystemExit: 2 plz any help appreciated From wuwei23 at gmail.com Thu Jul 25 20:47:07 2013 From: wuwei23 at gmail.com (alex23) Date: Fri, 26 Jul 2013 10:47:07 +1000 Subject: virtualenv problem In-Reply-To: <626bd852-a79e-42d7-aaa9-c5cba477d155@googlegroups.com> References: <626bd852-a79e-42d7-aaa9-c5cba477d155@googlegroups.com> Message-ID: On 25/07/2013 11:48 PM, D. Xenakis wrote: > I think there is something wrong with the installation because when i run through idle the virtual-env scripts located in "C:\Python33\Scripts" then i get the following.. virtualenv is intended to be a command line tool, so running it through idle is your first problem :) > You must provide a DEST_DIR > Usage: virtualenv-3.3-script.py [OPTIONS] DEST_DIR The error you're receiving seems pretty explicit. Generally, you would go to, say, a projects folder and type at the command line: C:\Projects> virtualenv my-new-project C:\Projects> cd my-new-project C:\Projects\my-new-project> Scripts\activate.bat This will create & enable your virtualenv sandbox. For more info see: http://www.virtualenv.org/en/latest/#usage From gouzounakis at hotmail.com Fri Jul 26 08:25:13 2013 From: gouzounakis at hotmail.com (D. Xenakis) Date: Fri, 26 Jul 2013 05:25:13 -0700 (PDT) Subject: virtualenv problem In-Reply-To: References: <626bd852-a79e-42d7-aaa9-c5cba477d155@googlegroups.com> Message-ID: <4fdf3e7c-0c1f-4048-b4bb-006e8cd42f30@googlegroups.com> Yeah trying to run virtualenv under IDLE was a desperate move as i couldnt make anything work under cmd. Apparently my problem was that i did not have correctly setup the new path.. Solution for me was the following from "http://forums.udacity.com/questions/100064678/pip-installation-instructions" ---------------------------------------------- ..We want to add that directory to your Path environment variable. Path is a list of directories where your OS looks for executable files. You will need to change the directory if you installed Python in a non-default location. a. go to Control Panel ? System ? Advanced ? Environment Variables, make sure Path is selected under "user variables for user", and click edit. b. Add ;C:\Python33\Scripts\ and ;C:\Python33\ (no spaces after the previous entry, ';' is the delimiter) to the end of variable value, then click ok. You should not be erasing anything, just adding to what's already there. This just makes it so we don't have to type the full path name whenever we want to run pip or other programs in those directories. "C:\Python33\Scripts\" is the one we want now, but "C:\Python33\" might be useful for you in the future. Restart your computer. ---------------------------------------------- I realised that even if i didn't do the above, i could still have got things rolling just by cd-ing from the cmd, to the script folder of my Python installation. But hey - learning is a good thing From wuwei23 at gmail.com Mon Jul 29 20:25:54 2013 From: wuwei23 at gmail.com (alex23) Date: Tue, 30 Jul 2013 10:25:54 +1000 Subject: virtualenv problem In-Reply-To: <4fdf3e7c-0c1f-4048-b4bb-006e8cd42f30@googlegroups.com> References: <626bd852-a79e-42d7-aaa9-c5cba477d155@googlegroups.com> <4fdf3e7c-0c1f-4048-b4bb-006e8cd42f30@googlegroups.com> Message-ID: On 26/07/2013 10:25 PM, D. Xenakis wrote: > Apparently my problem was that i did not have correctly setup the new path.. > But hey - learning is a good thing +1! Also, good job on posting the solution you found as well, that's always helpful if anyone else hits the same problem. Personally, I tend to use the ActiveState Python installer; it not only takes care of a lot of the general Windows integration, it also includes a handful of Windows-oriented libraries. It's not a big enough difference to worry about if you've managed to get things working now, but for subsequent Python versions it might be convenient. From santiago.diez at caoba.fr Thu Jul 25 11:11:15 2013 From: santiago.diez at caoba.fr (santiago.diez at caoba.fr) Date: Thu, 25 Jul 2013 08:11:15 -0700 (PDT) Subject: Is it that easy to install Python ? Message-ID: Hi there, I never write any Python program but as a system administrator, I'm often asked to install python on Debian servers. I just finished downloading, configuring, making and installing. The binary is now installed in : /usr/local/Python-2.7.5/bin/python2.7 (the path is a deliberate administrator choice). Is that it? What else will my users need? Regards Santiago From maarten.sneep at knmi.nl Thu Jul 25 12:13:43 2013 From: maarten.sneep at knmi.nl (Maarten) Date: Thu, 25 Jul 2013 09:13:43 -0700 (PDT) Subject: Is it that easy to install Python ? In-Reply-To: References: Message-ID: On Thursday, July 25, 2013 5:11:15 PM UTC+2, santia... at caoba.fr wrote: > Hi there, > > I never write any Python program but as a system administrator, I'm often asked to install python on Debian servers. > > I just finished downloading, configuring, making and installing. > > The binary is now installed in : > /usr/local/Python-2.7.5/bin/python2.7 > (the path is a deliberate administrator choice). > > Is that it? Probably. > What else will my users need? The path must be search $PATH in the environment where Python is used, so that standard scripts starting with "#!/usr/bin/env python" will find your python. Next your users will probably start requesting additional packages (regex, lxml, numpy, scipy, matplotlib, ... depending on what they actually do). Maarten From mclefavor at gmail.com Thu Jul 25 12:15:47 2013 From: mclefavor at gmail.com (Matthew Lefavor) Date: Thu, 25 Jul 2013 12:15:47 -0400 Subject: Is it that easy to install Python ? In-Reply-To: References: Message-ID: On Thu, Jul 25, 2013 at 11:11 AM, wrote: > Hi there, > > I never write any Python program but as a system administrator, I'm often > asked to install python on Debian servers. > > I just finished downloading, configuring, making and installing. > > The binary is now installed in : > /usr/local/Python-2.7.5/bin/python2.7 > (the path is a deliberate administrator choice). > > Is that it? > > What else will my users need? > They may need permission to install third-party modules themselves. That would probably require write permissions to /usr/local/Python-2.7.5/lib. If you are unwilling to grant those permissions, you can suggest that they learn how to work with "virtualenvs" (http://www.virtualenv.org/en/latest/), which would allow them to install third-party modules locally. And, of course, they need to be saavy enough to put the new python installation on the PATH. -------------- next part -------------- An HTML attachment was scrubbed... URL: From irmen.NOSPAM at xs4all.nl Thu Jul 25 12:36:48 2013 From: irmen.NOSPAM at xs4all.nl (Irmen de Jong) Date: Thu, 25 Jul 2013 18:36:48 +0200 Subject: Is it that easy to install Python ? In-Reply-To: References: Message-ID: <51f1541f$0$15947$e4fe514c@news.xs4all.nl> On 25-7-2013 17:11, santiago.diez at caoba.fr wrote: > Hi there, > > I never write any Python program but as a system administrator, I'm often asked to install python on Debian servers. > > I just finished downloading, configuring, making and installing. > > The binary is now installed in : > /usr/local/Python-2.7.5/bin/python2.7 > (the path is a deliberate administrator choice). > > Is that it? > > What else will my users need? Why didn't you use the Debian package instead? You now have installed an unsupported, untested custom built Python version on your server. Why not simply $ apt-get install python and let the Debian package maintainers take care of properly testing and supporting it... Also, installing additional python packages will be much less of a hassle because there's hundreds of them readily available in Debian's package repositories and they can be installed (including correct dependencies) in the same way. Irmen From rosuav at gmail.com Thu Jul 25 12:44:55 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 26 Jul 2013 02:44:55 +1000 Subject: Is it that easy to install Python ? In-Reply-To: <51f1541f$0$15947$e4fe514c@news.xs4all.nl> References: <51f1541f$0$15947$e4fe514c@news.xs4all.nl> Message-ID: On Fri, Jul 26, 2013 at 2:36 AM, Irmen de Jong wrote: > On 25-7-2013 17:11, santiago.diez at caoba.fr wrote: >> Hi there, >> >> I never write any Python program but as a system administrator, I'm often asked to install python on Debian servers. >> >> I just finished downloading, configuring, making and installing. >> >> The binary is now installed in : >> /usr/local/Python-2.7.5/bin/python2.7 >> (the path is a deliberate administrator choice). >> > Why didn't you use the Debian package instead? You now have installed an unsupported, > untested custom built Python version on your server. Why not simply > > $ apt-get install python That'll do fine on Debian 7 (Wheezy, current stable). On Debian 6 (Squeeze, oldstable), that'll get you a 2.6 release IIRC. ChrisA From invalid at invalid.invalid Thu Jul 25 14:32:50 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Thu, 25 Jul 2013 18:32:50 +0000 (UTC) Subject: Is it that easy to install Python ? References: Message-ID: On 2013-07-25, santiago.diez at caoba.fr wrote: > Hi there, > > I never write any Python program but as a system administrator, I'm > often asked to install python on Debian servers. Are you sure it's not already installed? I haven't seen a Linux distro for a _long_ time that didn't include Python as part of the base installation. > I just finished downloading, configuring, making and installing. > > The binary is now installed in : > /usr/local/Python-2.7.5/bin/python2.7 Why not just apt-get install python? -- Grant Edwards grant.b.edwards Yow! Uh-oh!! I'm having at TOO MUCH FUN!! gmail.com From oscxxxxxxxaas at gmail.com Sat Jul 27 18:05:19 2013 From: oscxxxxxxxaas at gmail.com (wizzofozz) Date: Sun, 28 Jul 2013 00:05:19 +0200 Subject: Is it that easy to install Python ? In-Reply-To: References: Message-ID: On 25-7-2013 17:11, santiago.diez at caoba.fr wrote: > Hi there, > > I never write any Python program but as a system administrator, I'm often asked to install python on Debian servers. > > I just finished downloading, configuring, making and installing. > As a side note to all who have replied already; as a 'regular' linux user you can also just download, untar, configure and make. You can't do 'make install', but that's not required to run the executables (in this case 'python') built in the previous steps. just nitpicking .. :-) cheers, Ozz From santiago.diez at caoba.fr Wed Jul 31 06:08:21 2013 From: santiago.diez at caoba.fr (santiago.diez at caoba.fr) Date: Wed, 31 Jul 2013 03:08:21 -0700 (PDT) Subject: Is it that easy to install Python ? In-Reply-To: References: Message-ID: OK thanks for your answers. So Python is not a daemon. Is it? Does it have a config file? Actually, each site on the web server is jailed in a chrooted environment. In the chrooted environment, Python appears to be in /usr/bin/python. If I give developers write access to a folder like /usr/lib/python/dist-packages/ (in the chrooted environment), will this be sufficient? Regards Santiago PS: Thanks for all your opinions about how one should install Python but that wasn't the purpose of this thread. Most programs on my servers are compiled. Period. From ian.g.kelly at gmail.com Wed Jul 31 12:44:15 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 31 Jul 2013 10:44:15 -0600 Subject: Is it that easy to install Python ? In-Reply-To: References: Message-ID: On Wed, Jul 31, 2013 at 4:08 AM, wrote: > OK thanks for your answers. > So Python is not a daemon. Is it? No. > Does it have a config file? Not as such. Arbitrary site-specific customization can be done by creating a sitecustomize module, but this is not standard. There are also some environment variables that you may want to consider setting in the system profile, but none are required or generally recommended: http://docs.python.org/2/using/cmdline.html#environment-variables > Actually, each site on the web server is jailed in a chrooted environment. > In the chrooted environment, Python appears to be in /usr/bin/python. > > If I give developers write access to a folder like /usr/lib/python/dist-packages/ (in the chrooted environment), will this be sufficient? I don't have experience with this, but it seems to me that you should place the standard library and site-packages in the chrooted /usr/lib/python2.7 and then set the PYTHONHOME environment variable to /usr/lib. From ian.g.kelly at gmail.com Wed Jul 31 12:49:53 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 31 Jul 2013 10:49:53 -0600 Subject: Is it that easy to install Python ? In-Reply-To: References: Message-ID: On Wed, Jul 31, 2013 at 10:44 AM, Ian Kelly wrote: > and then set the PYTHONHOME environment variable to /usr/lib. Rather, just /usr. From scott.moore270 at gmail.com Thu Jul 25 12:03:16 2013 From: scott.moore270 at gmail.com (CTSB01) Date: Thu, 25 Jul 2013 09:03:16 -0700 (PDT) Subject: Creating a Simple User Interface for a Function Message-ID: I have the following code that runs perfectly: def psi_j(x, j): rtn = [] for n2 in range(0, len(x) * j - 2): n = n2 / j r = n2 - n * j rtn.append(j * x[n] + r * (x[n + 1] - x[n])) print 'n2 =', n2, ': n =', n, ' r =' , r, ' rtn =', rtn return rtn This code takes a string x = [0,1,1,1,2] for example (it must always begin with 0) and a parameter j, say 2, and outputs a string (x = [0, 1, 2, 2, 2, 2, 2, 3] in this example). It does this in two steps: First it decomposes some number m into a multiple of j and a remainder. Then it runs this decomposition through a function on the rtn.append line. Notice that this has cj - 1 terms where c is the number of terms in the input string and j is the parameter. Normally, we would like it to be able to calculate cj terms. This is an issue with the function that I am more than happy to put aside for the moment. My key interest is to be able to make this program usable for someone who has no knowledge of programming. In particular, I need some kind of user interface that prompts the user to input a string (ideally just by putting in numbers in the form 011123334 for example) and a parameter, and then displays the output sequence. This is essentially what the program already does but the idea is to make it usable for even the most technologically disinclined. Ideally it would do this without needing to run Python at all. If anyone is able to make this happen in Python I would be eternally grateful. From davea at davea.name Thu Jul 25 15:19:27 2013 From: davea at davea.name (Dave Angel) Date: Thu, 25 Jul 2013 15:19:27 -0400 Subject: Creating a Simple User Interface for a Function In-Reply-To: References: Message-ID: On 07/25/2013 12:03 PM, CTSB01 wrote: > I have the following code that runs perfectly: > > def psi_j(x, j): > rtn = [] > for n2 in range(0, len(x) * j - 2): > n = n2 / j > r = n2 - n * j > rtn.append(j * x[n] + r * (x[n + 1] - x[n])) > print 'n2 =', n2, ': n =', n, ' r =' , r, ' rtn =', rtn > return rtn > No it doesn't run perfectly. It'll get a syntax error on the print function call. That's assuming you're still using Python 3.3. You really need to start by specifying your environment, without making us look back through previous threads from you. > This code takes a string x = [0,1,1,1,2] for example That's not a string. A string would be like xx = psi_j("0abcd1234") Perhaps you mean list? And is it a list of integers, or of arbitrary numbers? Are there any constraints on the sizes or signs of those numbers? > (it must always begin with 0) and a parameter j, say 2, and outputs a string (x = [0, 1, 2, 2, 2, 2, 2, 3] in this example). > > It does this in two steps: First it decomposes some number m into a multiple of j and a remainder. Only if you replace the / with //. Or just use the function divmod(): n, r = divmod(n2, m) > Then it runs this decomposition through a function on the rtn.append line. > > Notice that this has cj - 1 terms where c is the number of terms in the input string and j is the parameter. Normally, we would like it to be able to calculate cj terms. > This is an issue with the function that I am more than happy to put aside for the moment. > > My key interest is to be able to make this program So far you have a function, not a program. If you put it in a text file and run it from python, it'll do nothing but display a syntax error message. And when you fix that, it'll just run without doing anything. usable for someone who has no knowledge of programming. In particular, I need some kind of user interface that prompts > the user to input a string (ideally just by putting in numbers in the form 011123334 for example) and a parameter, > and then displays the output sequence. This is essentially what the program already does but the idea is to make it usable > for even the most technologically disinclined. Ideally it would do this without needing to run Python at all. Then why are you asking on the Python forum? Or perhaps you mean without him knowing he's running Python? In that case, use a shebang line at the beginning, which will tell Linux to automatically invoke the specified program (or programming language in this case). > If anyone is able to make this happen in Python I would be eternally grateful. > If we assume you're running Python 3.3 on Linux, and the user is willing to us the terminal, then how about parsing the string from the command line he types? You can access it as011123334 a string from sys.argv, and convert it to separate numbers. Of course as it stands now, you cannot tell whether the user wanted 0,1,1,1,2,3,3,3,4 or 0, 111, 23, 3, 3, 4 or something else. -- DaveA From scott.moore270 at gmail.com Thu Jul 25 16:58:33 2013 From: scott.moore270 at gmail.com (CTSB01) Date: Thu, 25 Jul 2013 13:58:33 -0700 (PDT) Subject: Creating a Simple User Interface for a Function In-Reply-To: References: Message-ID: <97f8224f-e73b-4a4b-bf05-7cc3dba4e9d9@googlegroups.com> On Thursday, July 25, 2013 3:19:27 PM UTC-4, Dave Angel wrote: > On 07/25/2013 12:03 PM, CTSB01 wrote: > > > I have the following code that runs perfectly: > > > > def psi_j(x, j): > > > rtn = [] > > > for n2 in range(0, len(x) * j - 2): > > > n = n2 / j > > > r = n2 - n * j > > > rtn.append(j * x[n] + r * (x[n + 1] - x[n])) > > > print 'n2 =', n2, ': n =', n, ' r =' , r, ' rtn =', rtn > > > return rtn > > No it doesn't run perfectly. It'll get a syntax error on the print > > function call. That's assuming you're still using Python 3.3. You > > really need to start by specifying your environment, without making us > > look back through previous threads from you. > > > This code takes a string x = [0,1,1,1,2] for example > > That's not a string. A string would be like > > xx = psi_j("0abcd1234") > > Perhaps you mean list? And is it a list of integers, or of arbitrary > > numbers? Are there any constraints on the sizes or signs of those numbers? > > > (it must always begin with 0) and a parameter j, say 2, and outputs a string (x = [0, 1, 2, 2, 2, 2, 2, 3] in this example). > > > It does this in two steps: First it decomposes some number m into a multiple of j and a remainder. > > Only if you replace the / with //. Or just use the function divmod(): > > n, r = divmod(n2, m) > > > Then it runs this decomposition through a function on the rtn.append line. > > > Notice that this has cj - 1 terms where c is the number of terms in the input string and j is the parameter. Normally, we would like it to be able to calculate cj terms. > > > This is an issue with the function that I am more than happy to put aside for the moment. > > > My key interest is to be able to make this program > > So far you have a function, not a program. If you put it in a text file > > and run it from python, it'll do nothing but display a syntax error > > message. And when you fix that, it'll just run without doing anything. > > usable for someone who has no knowledge of programming. In > > particular, I need some kind of user interface that prompts > > > the user to input a string (ideally just by putting in numbers in the form 011123334 for example) and a parameter, > > > and then displays the output sequence. This is essentially what the program already does but the idea is to make it usable > > > for even the most technologically disinclined. Ideally it would do this without needing to run Python at all. > > Then why are you asking on the Python forum? Or perhaps you mean > > without him knowing he's running Python? In that case, use a shebang > > line at the beginning, which will tell Linux to automatically invoke the > > specified program (or programming language in this case). > > > If anyone is able to make this happen in Python I would be eternally grateful. > > If we assume you're running Python 3.3 on Linux, and the user is willing > > to us the terminal, then how about parsing the string from the command > > line he types? You can access it as011123334 a string from sys.argv, > > and convert it to separate numbers. Of course as it stands now, you > > cannot tell whether the user wanted > > 0,1,1,1,2,3,3,3,4 > > or > > 0, 111, 23, 3, 3, 4 > > or something else. > > DaveA Sorry Dave, to answer each part of your response: 1) I decided to use Python 2.7, and I will be sure to specify this in all future threads. 2) It is a list of positive integers. In fact, it is always going to be a list of positive increasing integers. 3) You're right. What I meant was that if after running that bit of code I enter >>> x = [0,1,2,3,4,5] >>> psi_j(x,2) I will get output that matches my requirements. 4) Yes, sorry that's what I meant (if I understood correctly). I was told elsewhere that I might want to try using tkinter. Essentially I'm trying to create a user interface that allows the user to just type in a string 01112345 for example, and choose a parameter (say j=2) and then click a button to run the function. I'd like to be able to run send a .exe file that the user can just open up and use with no further setup. So on top of the user interface I would also it looks like need to determine how to make Python change a string 01112345 into a list so that it does that automatically when the user clicks 'run'. Would a shebang still be the right way to go? Thanks again Dave, apologies for the ambiguity. From tjreedy at udel.edu Thu Jul 25 19:00:50 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 25 Jul 2013 19:00:50 -0400 Subject: Creating a Simple User Interface for a Function In-Reply-To: <97f8224f-e73b-4a4b-bf05-7cc3dba4e9d9@googlegroups.com> References: <97f8224f-e73b-4a4b-bf05-7cc3dba4e9d9@googlegroups.com> Message-ID: On 7/25/2013 4:58 PM, CTSB01 wrote: > 1) I decided to use Python 2.7, and I will be sure to specify this in > all future threads. Given that you are not using any libraries, let alone one that does not run on Python 3, I strongly recommend using the latest version (3.3). > 2) It is a list of positive integers. In fact, it is always going to > be a list of positive increasing integers. Your example below starts with 0, which is not positive. Perhaps you mean that all integers after a single leading 0 have to be positive and increasing. If you run digits together, then the max int is 9. Do you intend this? > 4) Yes, sorry that's what I meant (if I understood correctly). I was > told elsewhere that I might want to try using tkinter. If users start the program at a command line, the core of an input function would be input = (raw)input('Enter digits: ') # Include "raw" on 2.x You would need a more elaborate prompt printed first, and input checking with the request repeated if the input does not pass the check. It would be pretty simple to do the equivalent with a tkinter dialog box. > I'd like to be > able to run send a .exe file that the user can just open up and use > with no further setup. There are programs that will package your code with an interpreter. But do give people the option to get just the program without installing a duplicate interpreter. > So on top of the user interface I would also it looks like need to > determine how to make Python change a string 01112345 into a list so > that it does that automatically when the user clicks 'run'. >>> list('01112345') ['0', '1', '1', '1', '2', '3', '4', '5'] >>> '0,1,1,1,2,3,4,5'.split(',') ['0', '1', '1', '1', '2', '3', '4', '5'] > Would a shebang still be the right way to go? On Linux, definitely, whether you have user enter on the command line or in response to a prompt. On windows, it only helps with 3.3+. -- Terry Jan Reedy From tjreedy at udel.edu Thu Jul 25 19:46:44 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 25 Jul 2013 19:46:44 -0400 Subject: Creating a Simple User Interface for a Function In-Reply-To: References: <97f8224f-e73b-4a4b-bf05-7cc3dba4e9d9@googlegroups.com> Message-ID: Some additional comments. On 7/25/2013 7:00 PM, Terry Reedy wrote: > On 7/25/2013 4:58 PM, CTSB01 wrote: > >> 1) I decided to use Python 2.7, and I will be sure to specify this in >> all future threads. > > Given that you are not using any libraries, let alone one that does not > run on Python 3, I strongly recommend using the latest version (3.3). It would be pretty easy to make your simple code run on both 3.x and 2.6/7. Start your file (after any docstring or initial comment) with from __future__ import division, print_function Use "except XyxError as e:" instead of "except XyzError, e:". > If users start the program at a command line, the core of an input > function would be > numbers = input('Enter digits: ') # see below > You would need a more elaborate prompt printed first, and input checking > with the request repeated if the input does not pass the check. # To run on both 2.x and 3.x, put this after the __future__ import: try: input = raw_input except NameError: pass >> I'd like to be >> able to run send a .exe file that the user can just open up and use >> with no further setup. > > There are programs that will package your code with an interpreter. A Python pre-built binary is overkill for such a small function. The reason for doing so, packaging all dependencies together, does not apply. Any binary is limited to what machines it will run on. > do give people the option to get just the program without installing a > duplicate interpreter. A Python file, especially if designed to run on 2.6, will run on most any recent installation. -- Terry Jan Reedy From davea at davea.name Thu Jul 25 19:01:46 2013 From: davea at davea.name (Dave Angel) Date: Thu, 25 Jul 2013 19:01:46 -0400 Subject: Creating a Simple User Interface for a Function In-Reply-To: <97f8224f-e73b-4a4b-bf05-7cc3dba4e9d9@googlegroups.com> References: <97f8224f-e73b-4a4b-bf05-7cc3dba4e9d9@googlegroups.com> Message-ID: On 07/25/2013 04:58 PM, CTSB01 wrote: > > Sorry Dave, to answer each part of your response: > > 1) I decided to use Python 2.7, and I will be sure to specify this in all future threads. > 2) It is a list of positive integers. In fact, it is always going to be a list of positive increasing integers. > 3) You're right. What I meant was that if after running that bit of code I enter >>>> x = [0,1,2,3,4,5] >>>> psi_j(x,2) > I will get output that matches my requirements. > 4) Yes, sorry that's what I meant (if I understood correctly). I was told elsewhere that I might want to try using tkinter. Essentially I'm trying to create a user interface that allows the user to just type in a string 01112345 for example, and choose a parameter (say j=2) and then click a button to run the function. I'd like to be able to run send a .exe file But Linux doesn't understand exe files. Once again, you need to tell us your environment. I'm now supposing you mean your user will be using Windows. I have no experience with making a single runnable .exe file for Windows. But many others here do, and can presumably help you. > that the user can just open up and use with no further setup. > > So on top of the user interface I would also it looks like need to determine how to make Python change a string 01112345 into a list so that it does that automatically when the user clicks 'run'. Since you give no upper limits to the numbers the user might be specifying (other than saying their strictly positively monatonic), the string "01112345" could be interpreted only as: 0, 1, 11, 23, 45 0, 11, 12, 345 0, 11, 12345 0, 111, 2345 (If the individual numbers were limited to the range 0-9, then we'd come up with the sequence 0,1,1,1,2,3,4,5, but that's not increasing, so it wouldn't be legal.) Is there a reason you don't want the user to have to separate the numbers, either with spaces or commas? If they typed "0 1 11 23 45" you could trivially parse that with something like: [int(x) for x in data.split()] > > Would a shebang still be the right way to go? A shebang is a Linux/Unix concept. Although with Python 3.3 there's a similar Windows concept that lets a user switch between Python versions using the shebang, it's not really the same thing, and we shouldn't discuss it till we know whether you're talking Windows or Linux. > > Thanks again Dave, apologies for the ambiguity. > -- DaveA From ramit.prasad at jpmorgan.com Fri Jul 26 10:08:14 2013 From: ramit.prasad at jpmorgan.com (Prasad, Ramit) Date: Fri, 26 Jul 2013 14:08:14 +0000 Subject: Creating a Simple User Interface for a Function In-Reply-To: <97f8224f-e73b-4a4b-bf05-7cc3dba4e9d9@googlegroups.com> References: <97f8224f-e73b-4a4b-bf05-7cc3dba4e9d9@googlegroups.com> Message-ID: <5B80DD153D7D744689F57F4FB69AF474185E4168@SCACMX008.exchad.jpmchase.net> CTSB01 wrote: > On Thursday, July 25, 2013 3:19:27 PM UTC-4, Dave Angel wrote: > > On 07/25/2013 12:03 PM, CTSB01 wrote: > > > > > I have the following code that runs perfectly: > > > > > > > def psi_j(x, j): > > > > > rtn = [] > > > > > for n2 in range(0, len(x) * j - 2): > > > > > n = n2 / j > > > > > r = n2 - n * j > > > > > rtn.append(j * x[n] + r * (x[n + 1] - x[n])) > > > > > print 'n2 =', n2, ': n =', n, ' r =' , r, ' rtn =', rtn > > > > > return rtn > > > > No it doesn't run perfectly. It'll get a syntax error on the print > > > > function call. That's assuming you're still using Python 3.3. You > > > > really need to start by specifying your environment, without making us > > > > look back through previous threads from you. > > > > > This code takes a string x = [0,1,1,1,2] for example > > > > That's not a string. A string would be like > > > > xx = psi_j("0abcd1234") > > > > Perhaps you mean list? And is it a list of integers, or of arbitrary > > > > numbers? Are there any constraints on the sizes or signs of those numbers? > > > > > (it must always begin with 0) and a parameter j, say 2, and outputs a string (x = [0, 1, 2, 2, 2, > 2, 2, 3] in this example). > > > > > It does this in two steps: First it decomposes some number m into a multiple of j and a remainder. > > > > Only if you replace the / with //. Or just use the function divmod(): > > > > n, r = divmod(n2, m) > > > > > Then it runs this decomposition through a function on the rtn.append line. > > > > > Notice that this has cj - 1 terms where c is the number of terms in the input string and j is the > parameter. Normally, we would like it to be able to calculate cj terms. > > > > > This is an issue with the function that I am more than happy to put aside for the moment. > > > > > My key interest is to be able to make this program > > > > So far you have a function, not a program. If you put it in a text file > > > > and run it from python, it'll do nothing but display a syntax error > > > > message. And when you fix that, it'll just run without doing anything. > > > > usable for someone who has no knowledge of programming. In > > > > particular, I need some kind of user interface that prompts > > > > > the user to input a string (ideally just by putting in numbers in the form 011123334 for example) > and a parameter, > > > > > and then displays the output sequence. This is essentially what the program already does but the > idea is to make it usable > > > > > for even the most technologically disinclined. Ideally it would do this without needing to run > Python at all. > > > > Then why are you asking on the Python forum? Or perhaps you mean > > > > without him knowing he's running Python? In that case, use a shebang > > > > line at the beginning, which will tell Linux to automatically invoke the > > > > specified program (or programming language in this case). > > > > > If anyone is able to make this happen in Python I would be eternally grateful. > > > > If we assume you're running Python 3.3 on Linux, and the user is willing > > > > to us the terminal, then how about parsing the string from the command > > > > line he types? You can access it as011123334 a string from sys.argv, > > > > and convert it to separate numbers. Of course as it stands now, you > > > > cannot tell whether the user wanted > > > > 0,1,1,1,2,3,3,3,4 > > > > or > > > > 0, 111, 23, 3, 3, 4 > > > > or something else. > > > > DaveA > > Sorry Dave, to answer each part of your response: > > 1) I decided to use Python 2.7, and I will be sure to specify this in all future threads. > 2) It is a list of positive integers. In fact, it is always going to be a list of positive increasing > integers. > 3) You're right. What I meant was that if after running that bit of code I enter > >>> x = [0,1,2,3,4,5] > >>> psi_j(x,2) > I will get output that matches my requirements. > 4) Yes, sorry that's what I meant (if I understood correctly). I was told elsewhere that I might want > to try using tkinter. Essentially I'm trying to create a user interface that allows the user to just > type in a string 01112345 for example, and choose a parameter (say j=2) and then click a button to run > the function. I'd like to be able to run send a .exe file that the user can just open up and use with > no further setup. Any UI will work whether graphical or command line. TK is a good choice if you assume that the user has Python installed. If you are planning to create an exe (Windows) then you can probably bundle any GUI library (wx/gtk/qt) but you might be limited by the support of the exe creating tool. I have never created an executable like this, so I am not sure. > > So on top of the user interface I would also it looks like need to determine how to make Python change > a string 01112345 into a list so that it does that automatically when the user clicks 'run'. This really does not make sense to me. Does that mean 0, 1, 11, 23, 45 or 0, 111, 2345 or 0,11, 2345 or something else entirely? If you are doing this on the shell I would have the user pass in a string of delimited values. "0,1,11,23,45" and then do a .split(',') on the string. If you are doing this via GUI then you can create separate fields for each number or still ask for a delimited string. However, if the parameter (j=2) determines parsing of numbers then you can ignore my concerns. > > Would a shebang still be the right way to go? The shebang is useful when executing a script from cmd/shell directly. $ ./script.py # rather than $ python script.py If you are making an executable (.exe in Windows) then I do not think it will matter either way for 2.7. > > Thanks again Dave, apologies for the ambiguity. ~Ramit This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email. From klemily097 at gmail.com Thu Jul 25 14:30:35 2013 From: klemily097 at gmail.com (kle) Date: Thu, 25 Jul 2013 11:30:35 -0700 (PDT) Subject: Problem with PyObjC and NSEvents Message-ID: <26f3a525-19d5-49f0-9535-d13038bb3a6c@googlegroups.com> Hello all, I am having a rather specialized problem involving PyObjC and receiving NSEvents, and I'm wondering whether anybody here might have any insight. I'm writing a Python program, using PyObjC, that runs in the background and registers whenever the user clicks the mouse, in any application. I do this using an NSEvents global events monitor: NSEvent.addGlobalMonitorForEventsMatchingMask_handler_(NSLeftMouseDownMask, handler) where "handler" is the name of my handler method. The code works fine when I run the .py file from the command line -- my handler method is called whenever I click the mouse, and everything happens as expected. However, when I bundle it into a standalone app (using py2app) and run it, the app fails to register clicks. I have gotten it to work intermittently -- sometimes it will start working after the app has been brought to the front for the first time. But for the most part, it doesn't register the clicks at all. Any ideas on what I might try to get it working? I am not sure whether this is an NSEvents problem or a PyObjC problem, or just a me being stupid problem. But any help or guidance would be greatly appreciated. Thanks, Emily From fabiofz at gmail.com Thu Jul 25 15:34:58 2013 From: fabiofz at gmail.com (Fabio Zadrozny) Date: Thu, 25 Jul 2013 16:34:58 -0300 Subject: PyDev 2.8.0 Released Message-ID: Hi All, PyDev 2.8.0 has been released Details on PyDev: http://pydev.org Details on its development: http://pydev.blogspot.com Release Highlights: ------------------------------- * Type Inference now works with docstrings (Sphinx or Epydoc). See: http://pydev.org/manual_adv_type_hints.html * Fixed debugger to work on Google App Engine * Patch by Edward Catmur * Interactive console supports running with the Qt and Gtk event loops * Patches by Andrew Ferrazzutti * Multiple main modules/packages may be selected in the unittest run configuration * Properly handling unittest errors caused by setUpClass/setUpModule exceptions * It's possible to select the Working Set configuration in the New PyDev Project wizard * Patches by Christoph Zwerschke * It's possible to specify PyLint settings per project by passing --rcfile=.pylintrc (it's now run relative to the project directory) * PyLint now accepts an executable so that it does not have to rely on the configured interpreter. * Fixed OutOfMemoryError when large file was found in the workspace. * Editor startup is now faster due to improvements in Jython scripts. * Improved the way that the interpreter location is shown on the pydev package explorer. * PyDev Package Explorer icon no longer missing when top level elements is set to Working Sets * Other minor bugfixes Note: PyDev is now signed with a new (self-signed) certificate (see http://pydev.org/manual_101_install.html for the new certificate) . What is PyDev? --------------------------- PyDev is a plugin that enables users to use Eclipse for Python, Jython and IronPython development -- making Eclipse a first class Python IDE -- It comes with many goodies such as code completion, syntax highlighting, syntax analysis, refactor, debug and many others. Cheers, -- Fabio Zadrozny ------------------------------------------------------ Software Developer PyDev - Python Development Environment for Eclipse http://pydev.org http://pydev.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at davecotter.com Thu Jul 25 17:05:59 2013 From: me at davecotter.com (David M. Cotter) Date: Thu, 25 Jul 2013 14:05:59 -0700 (PDT) Subject: how to package embedded python? Message-ID: <99b6fbcb-5456-4421-89d6-d0453b722e8c@googlegroups.com> what must i include in my app package if i'm embedding python? i tried including *everything* in the "DLLs" directory, but my app still crashes as soon as i attempt to initialize python. this is on a system that does not have python installed, as most of my users won't have it. is it actually a requirement that they first install python? (cuz it does work then) From me at davecotter.com Fri Jul 26 11:54:54 2013 From: me at davecotter.com (David M. Cotter) Date: Fri, 26 Jul 2013 08:54:54 -0700 (PDT) Subject: how to package embedded python? In-Reply-To: <99b6fbcb-5456-4421-89d6-d0453b722e8c@googlegroups.com> References: <99b6fbcb-5456-4421-89d6-d0453b722e8c@googlegroups.com> Message-ID: does nobody know how to do this? does nobody know where proper documentation on this is? From me at davecotter.com Mon Jul 29 12:22:42 2013 From: me at davecotter.com (David M. Cotter) Date: Mon, 29 Jul 2013 09:22:42 -0700 (PDT) Subject: how to package embedded python? In-Reply-To: References: <99b6fbcb-5456-4421-89d6-d0453b722e8c@googlegroups.com> Message-ID: <7ea319ad-87e2-41a1-8251-023d784ac013@googlegroups.com> nooooooooooobody knoooooooooooowssss.... the trouble aaaaaaaaaaaaaaaaahhhh seeeeeeeeeeee....... From kw at codebykevin.com Mon Jul 29 17:38:44 2013 From: kw at codebykevin.com (Kevin Walzer) Date: Mon, 29 Jul 2013 17:38:44 -0400 Subject: how to package embedded python? In-Reply-To: <99b6fbcb-5456-4421-89d6-d0453b722e8c@googlegroups.com> References: <99b6fbcb-5456-4421-89d6-d0453b722e8c@googlegroups.com> Message-ID: On 7/25/13 5:05 PM, David M. Cotter wrote: > what must i include in my app package if i'm embedding python? > > i tried including *everything* in the "DLLs" directory, but my app still crashes as soon as i attempt to initialize python. > > this is on a system that does not have python installed, as most of my users won't have it. is it actually a requirement that they first install python? (cuz it does work then) > Have you looked at these docs? http://docs.python.org/2/extending/embedding.html Lots of other hits on Google for ""embedding Python in C app." -- Kevin Walzer Code by Kevin/Mobile Code by Kevin http://www.codebykevin.com http://www.wtmobilesoftware.com From me at davecotter.com Tue Jul 30 16:23:06 2013 From: me at davecotter.com (David M. Cotter) Date: Tue, 30 Jul 2013 13:23:06 -0700 (PDT) Subject: how to package embedded python? In-Reply-To: References: <99b6fbcb-5456-4421-89d6-d0453b722e8c@googlegroups.com> Message-ID: <110caaff-720a-4cc6-a4d4-ad7a7fe612dd@googlegroups.com> yes, i've looked there, and all over google. i'm quite expert at embedding at this point. however nowhere i have looked has had instructions for "this this is how you package up your .exe with all the necessary python modules necessary to actually run on a user's system that does not have python installed". on mac, it's trivial: all macs come with python, there is nothing i need to include with my app and it "just works" on windows: if you don't include the proper DLLs and/or whatnot, then the app will complain about missing DLLs on startup. What DLLs must i include? where are the instructions? From cmpython at gmail.com Tue Jul 30 21:57:49 2013 From: cmpython at gmail.com (CM) Date: Tue, 30 Jul 2013 18:57:49 -0700 (PDT) Subject: how to package embedded python? In-Reply-To: <110caaff-720a-4cc6-a4d4-ad7a7fe612dd@googlegroups.com> References: <99b6fbcb-5456-4421-89d6-d0453b722e8c@googlegroups.com> <110caaff-720a-4cc6-a4d4-ad7a7fe612dd@googlegroups.com> Message-ID: <8505c5aa-0b16-4c32-bab3-17370b4f8f2e@googlegroups.com> On Tuesday, July 30, 2013 4:23:06 PM UTC-4, David M. Cotter wrote: > yes, i've looked there, and all over google. i'm quite expert at embedding at this point. > > > > however nowhere i have looked has had instructions for "this this is how you package up your .exe with all the necessary python modules necessary to actually run on a user's system that does not have python installed". > > > > on mac, it's trivial: all macs come with python, there is nothing i need to include with my app and it "just works" > > > > on windows: if you don't include the proper DLLs and/or whatnot, then the app will complain about missing DLLs on startup. > > > > What DLLs must i include? where are the instructions? I know nothing about embedding, but in terms of packaging up a Python interpreter with an application that needs it, could you use py2exe to do that? If, so is this helpful (gotten from Google's cache since page doesn't appear at the moment...later try embedded Python py2exe search)?: http://webcache.googleusercontent.com/search?q=cache:x3lrdFT5OF0J:www.py2exe.org/index.cgi/ShippingEmbedded+&cd=2&hl=en&ct=clnk&gl=us&client=firefox-a From me at davecotter.com Wed Jul 31 11:47:19 2013 From: me at davecotter.com (David M. Cotter) Date: Wed, 31 Jul 2013 08:47:19 -0700 (PDT) Subject: how to package embedded python? In-Reply-To: <8505c5aa-0b16-4c32-bab3-17370b4f8f2e@googlegroups.com> References: <99b6fbcb-5456-4421-89d6-d0453b722e8c@googlegroups.com> <110caaff-720a-4cc6-a4d4-ad7a7fe612dd@googlegroups.com> <8505c5aa-0b16-4c32-bab3-17370b4f8f2e@googlegroups.com> Message-ID: okay, well that might turn out to be useful, except i don't quite know how to use it, and there are no "from scratch" instructions. i managed to download "py2exe-0.6.9.zip" and unzip it, but how does one "install" this package? (yes, still a newb at that) then, once installed, how do i say "include the entire world" instead of just "mymodule" ? cuz the point of embedding python on my app is that the end-user can run any script at all, not just one module. From yanxiaopei199 at gmail.com Thu Jul 25 22:15:59 2013 From: yanxiaopei199 at gmail.com (Thanatos xiao) Date: Fri, 26 Jul 2013 10:15:59 +0800 Subject: Hello Everyone! A simple questions! Message-ID: >>> values = [0, 1, 2]>>> values[1] = values>>> values[0, [...], 2] why?? -------------- next part -------------- An HTML attachment was scrubbed... URL: From wuwei23 at gmail.com Thu Jul 25 22:45:54 2013 From: wuwei23 at gmail.com (alex23) Date: Fri, 26 Jul 2013 12:45:54 +1000 Subject: Hello Everyone! A simple questions! In-Reply-To: References: Message-ID: On 26/07/2013 12:15 PM, Thanatos xiao wrote: >>>> values = [0, 1, 2] >>>> values[1] = values >>>> values > [0, [...], 2] > > why?? First, it helps if you tell us what you were expecting instead. In your example, you have a list where you have replaced the 2nd element with the list itself. The [...] indicates that it is self-referential; there is no other way for it to display itself, because the contained reference also refers to itself. >>> values[1] [0, [...], 2] >>> values[1][1] [0, [...], 2] >>> values[1][1][1][1][1][1][1][1][1] [0, [...], 2] >>> values[1][1][1][1][1][1][1][1][1] is values True From ben+python at benfinney.id.au Thu Jul 25 23:14:52 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 26 Jul 2013 13:14:52 +1000 Subject: Hello Everyone! A simple questions! References: Message-ID: <7wiozy2es3.fsf@benfinney.id.au> Thanatos xiao writes: > >>> values = [0, 1, 2]>>> values[1] = values>>> values[0, [...], 2] > > why?? Because. :-) Did you have a more specific question? What exactly are you expecting that code to do, and what is the surprise? -- \ ?I was the kid next door's imaginary friend.? ?Emo Philips | `\ | _o__) | Ben Finney From bfloriang at gmail.com Fri Jul 26 01:28:32 2013 From: bfloriang at gmail.com (Florian Baumgartner) Date: Fri, 26 Jul 2013 07:28:32 +0200 Subject: Hello Everyone! A simple questions! In-Reply-To: References: Message-ID: As alex23 already indicated you created a recursive data-structure (by inserting a reference to the list into the second place of the list) and the interpreter handles this gracefully by showing [...]. In case you really want to insert the lists members into the second place you can assign a copy of the list. values = [0,1,2] values[1] = values[:] 2013/7/26 Thanatos xiao > >>> values = [0, 1, 2]>>> values[1] = values>>> values[0, [...], 2] > > why?? > > > -- > http://mail.python.org/mailman/listinfo/python-list > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yanxiaopei199 at gmail.com Fri Jul 26 02:19:25 2013 From: yanxiaopei199 at gmail.com (Thanatos xiao) Date: Fri, 26 Jul 2013 14:19:25 +0800 Subject: Hello Everyone! A simple questions! In-Reply-To: References: Message-ID: Thanks I just surprised by three dot 2013/7/26 Florian Baumgartner > As alex23 already indicated you created a recursive data-structure (by > inserting a reference to the list into the second place of the list) and > the interpreter handles this gracefully by showing [...]. > > In case you really want to insert the lists members into the second place > you can assign a copy of the list. > > values = [0,1,2] > values[1] = values[:] > > > > > > 2013/7/26 Thanatos xiao > >> >>> values = [0, 1, 2]>>> values[1] = values>>> values[0, [...], 2] >> >> why?? >> >> >> -- >> http://mail.python.org/mailman/listinfo/python-list >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From h4ck3rpr0n3 at gmail.com Thu Jul 25 23:16:06 2013 From: h4ck3rpr0n3 at gmail.com (John Doe) Date: Thu, 25 Jul 2013 20:16:06 -0700 (PDT) Subject: How would I do this? Message-ID: Hey guys, I;m working on making a chess program and I hit a wall. I have no idea how to get the position of the chess pieces. I have the chess board 1-8 and A-H for black as well as 1-8 and a-h for white. i just have to figure out how to declare those positions for the actual pieces so like the row of pawns for black would be B1-B8 as well as be able to replace the other coordinates with the piece when they move there like if I moved a black pawn from B1 to C1 B1 would be empty and then C1 would have it. I'm sorry if that's confusing but I can't seem to figure out how to do it... Any help is greatly appreciated and thank you in advance! From ian.g.kelly at gmail.com Thu Jul 25 23:31:58 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Thu, 25 Jul 2013 21:31:58 -0600 Subject: How would I do this? In-Reply-To: References: Message-ID: On Thu, Jul 25, 2013 at 9:16 PM, John Doe wrote: > Hey guys, > > I;m working on making a chess program and I hit a wall. I have no idea how to get the position of the chess pieces. I have the chess board 1-8 and A-H for black as well as 1-8 and a-h for white. i just have to figure out how to declare those positions for the actual pieces so like the row of pawns for black would be B1-B8 as well as be able to replace the other coordinates with the piece when they move there like if I moved a black pawn from B1 to C1 B1 would be empty and then C1 would have it. > > I'm sorry if that's confusing but I can't seem to figure out how to do it... Any help is greatly appreciated and thank you in advance! The usual way to do this would be to represent the board as an 8x8 list of lists, with each item of the inner lists representing one space on the board and denoting its contents. For example, you might have board[1][4] = 'P', indicating that E2 (1, 4 in zero-based row-column coordinates) holds a white pawn. From h4ck3rpr0n3 at gmail.com Thu Jul 25 23:40:07 2013 From: h4ck3rpr0n3 at gmail.com (h4ck3rpr0n3 at gmail.com) Date: Thu, 25 Jul 2013 20:40:07 -0700 (PDT) Subject: How would I do this? In-Reply-To: References: Message-ID: Thank you for the quick reply. Unfortunately I'm still a little confused... How might I implement that into my current code? def Make_Board(): Col = "+--+" Row = "| |" Col2 = "--+" Row2 = " |" print(" ", Col + Col2 + Col2 + Col2 + Col2 + Col2 + Col2 + Col2) print("1", Row + Row2 + Row2 + Row2 + Row2 + Row2 + Row2 + Row2, "1") print(" ", Col + Col2 + Col2 + Col2 + Col2 + Col2 + Col2 + Col2) print("2", Row + Row2 + Row2 + Row2 + Row2 + Row2 + Row2 + Row2, "2") print(" ", Col + Col2 + Col2 + Col2 + Col2 + Col2 + Col2 + Col2) print("3", Row + Row2 + Row2 + Row2 + Row2 + Row2 + Row2 + Row2, "3") print(" ", Col + Col2 + Col2 + Col2 + Col2 + Col2 + Col2 + Col2) print("4", Row + Row2 + Row2 + Row2 + Row2 + Row2 + Row2 + Row2, "4") print(" ", Col + Col2 + Col2 + Col2 + Col2 + Col2 + Col2 + Col2) print("5", Row + Row2 + Row2 + Row2 + Row2 + Row2 + Row2 + Row2, "5") print(" ", Col + Col2 + Col2 + Col2 + Col2 + Col2 + Col2 + Col2) print("6", Row + Row2 + Row2 + Row2 + Row2 + Row2 + Row2 + Row2, "6") print(" ", Col + Col2 + Col2 + Col2 + Col2 + Col2 + Col2 + Col2) print("7", Row + Row2 + Row2 + Row2 + Row2 + Row2 + Row2 + Row2, "7") print(" ", Col + Col2 + Col2 + Col2 + Col2 + Col2 + Col2 + Col2) print("8", Row + Row2 + Row2 + Row2 + Row2 + Row2 + Row2 + Row2, "8") print(" ", Col + Col2 + Col2 + Col2 + Col2 + Col2 + Col2 + Col2) def Col_Label(): Col_Label_Black = " A B C D E F G H" print(Col_Label_Black) def Col_Label2(): Col_Label_White = " A B C D E F G H" Col_Label_White = Col_Label_White.lower() print(Col_Label_White) def Board(): Col_Label() Make_Board() Col_Label2() Board() Thanks again for your help! From ian.g.kelly at gmail.com Fri Jul 26 00:02:18 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Thu, 25 Jul 2013 22:02:18 -0600 Subject: How would I do this? In-Reply-To: References: Message-ID: On Thu, Jul 25, 2013 at 9:40 PM, wrote: > Thank you for the quick reply. Unfortunately I'm still a little confused... How might I implement that into my current code? Where you print spaces indicating an empty space, you need to look up the current square in the board data structure to see what is contained there (whether it's an empty space or not) and print it accordingly. I strongly recommend that you work out how to print the board using nested for loops instead of all those individual print statements. The result will be more compact, easier to work with, and less error-prone. The basic skeleton should look something like this: for row in range(8): for column in range(8): print(board[row][column]) Which you'll need to embellish with the row and column dividers and labels. From tyler at familyrobbins.com Fri Jul 26 01:06:18 2013 From: tyler at familyrobbins.com (tyler at familyrobbins.com) Date: Thu, 25 Jul 2013 22:06:18 -0700 (PDT) Subject: Help Message-ID: <067018ce-ca0c-415b-a5d3-3c2f79f11e8e@googlegroups.com> I'm a bit new to python and I'm trying to create a simple program which adds words and definitions to a list, and then calls them forward when asked to. ------------------------------------------------------------------------------- choice = 0 words = [] entry = 0 definition = 0 escape = 0 finish = 0 memory = 0 dictionary = [] search = 0 def ask_ok(prompt, retries=2, complaint='Yes or no, please!'): while True: ok = input(prompt) if ok in ('y', 'ye', 'yes'): return True if ok in ('n', 'no', 'nop', 'nope'): return False retries = retries - 1 if retries < 0: raise IOError('refusenik user') print(complaint) print("Welcome to Digital Dictionary V1.0!\n\n") while escape < 1: choice = input("Type 'Entry' to add a word to the Dictionary, 'Search' to find a word, and 'Exit' to quit. ") if choice == '`': break if choice == 'Entry' or 'entry': while finish < 1: entry = input("Please type the word you wish to add: ") words.append(entry) definition = input("What does the word mean?\n ") dictionary.append(definition) print(entry, '\n', definition) ask_ok("Is this entry complete? ") if True: finish = 1 entry = 0 definition = 0 elif choice == 'Search' or 'search': search = input("Please type the word you wish to find: ") print(search in words) elif choice == 'Exit' or 'exit': ask_ok("Are you sure you want to exit? ") if True: escape = 1 else: print("Please choose an option from the list.") ------------------------------------------------------------------------------- However, if I run the program using anything but 'entry', the program still runs the 'entry' subroutine. After the 'entry' subroutine is run once, no options work. Ex: ------------------------------------------------------------------------------- >>> Welcome to Digital Dictionary V1.0! Type 'Entry' to add a word to the Dictionary, 'Search' to find a word, and 'Exit' to quit. entry Please type the word you wish to add: computer What does the word mean? a device for computing computer a device for computing Is this entry complete? yes Type 'Entry' to add a word to the Dictionary, 'Search' to find a word, and 'Exit' to quit. search Type 'Entry' to add a word to the Dictionary, 'Search' to find a word, and 'Exit' to quit. exit Type 'Entry' to add a word to the Dictionary, 'Search' to find a word, and 'Exit' to quit. ` >>> ------------------------------------------------------------------------------- The only option which seems to work is the '`' subroutine, which I added to stop the program after running for a while. I believe it works because it was added before the 'entry' subroutine. If anyone can help me figure out why it won't run properly, I'd be really grateful. P.S: I haven't finished the 'search' subroutine, but I just want to get this small problem solved soon so I can From wuwei23 at gmail.com Fri Jul 26 01:21:54 2013 From: wuwei23 at gmail.com (alex23) Date: Fri, 26 Jul 2013 15:21:54 +1000 Subject: Help In-Reply-To: <067018ce-ca0c-415b-a5d3-3c2f79f11e8e@googlegroups.com> References: <067018ce-ca0c-415b-a5d3-3c2f79f11e8e@googlegroups.com> Message-ID: On 26/07/2013 3:06 PM, tyler at familyrobbins.com wrote: > I'm a bit new to python and I'm trying to create a simple program which adds words and definitions to a list, and then calls them forward when asked to. > > ------------------------------------------------------------------------------- > choice = 0 > > words = [] > > entry = 0 > > definition = 0 > > escape = 0 > > finish = 0 > > memory = 0 > > dictionary = [] > > search = 0 > > def ask_ok(prompt, retries=2, complaint='Yes or no, please!'): > while True: > ok = input(prompt) > if ok in ('y', 'ye', 'yes'): > return True > if ok in ('n', 'no', 'nop', 'nope'): > return False > retries = retries - 1 > if retries < 0: > raise IOError('refusenik user') > print(complaint) > > > print("Welcome to Digital Dictionary V1.0!\n\n") > > while escape < 1: > > choice = input("Type 'Entry' to add a word to the Dictionary, 'Search' to find a word, and 'Exit' to quit. ") > > if choice == '`': > break > > if choice == 'Entry' or 'entry': > while finish < 1: > entry = input("Please type the word you wish to add: ") > words.append(entry) > definition = input("What does the word mean?\n ") > dictionary.append(definition) > print(entry, '\n', definition) > ask_ok("Is this entry complete? ") > if True: > finish = 1 > entry = 0 > definition = 0 > > elif choice == 'Search' or 'search': > search = input("Please type the word you wish to find: ") > print(search in words) > > > elif choice == 'Exit' or 'exit': > ask_ok("Are you sure you want to exit? ") > if True: > escape = 1 > > else: > print("Please choose an option from the list.") > ------------------------------------------------------------------------------- > However, if I run the program using anything but 'entry', the program still runs the 'entry' subroutine. After the 'entry' subroutine is run once, no options work. Ex: > ------------------------------------------------------------------------------- >>>> > Welcome to Digital Dictionary V1.0! > > Type 'Entry' to add a word to the Dictionary, 'Search' to find a word, and 'Exit' to quit. entry > Please type the word you wish to add: computer > What does the word mean? > a device for computing > computer > a device for computing > Is this entry complete? yes > Type 'Entry' to add a word to the Dictionary, 'Search' to find a word, and 'Exit' to quit. search > Type 'Entry' to add a word to the Dictionary, 'Search' to find a word, and 'Exit' to quit. exit > Type 'Entry' to add a word to the Dictionary, 'Search' to find a word, and 'Exit' to quit. ` >>>> > ------------------------------------------------------------------------------- > The only option which seems to work is the '`' subroutine, which I added to stop the program after running for a while. I believe it works because it was added before the 'entry' subroutine. > > If anyone can help me figure out why it won't run properly, I'd be really grateful. This doesn't do what you think it does: > if choice == 'Entry' or 'entry': Python interprets this as: if (choice == 'Entry') or 'entry': And as non-empty strings are considered True, it will always succeed and run the associated code block. The same goes for the subsequent conditions, but because you break out of the loop in the 'entry' section, it never reaches them. You do have it right in your `ask_ok` function, so just rewrite the conditions in a similar way: if choice in ('Entry', 'entry'): Or even better, always make the input lower case and then you only have one case to test for: choice = input("Type 'Entry' to etc ... ") choice = choice.lower() ... if choice == 'entry': ... From __peter__ at web.de Fri Jul 26 01:27:55 2013 From: __peter__ at web.de (Peter Otten) Date: Fri, 26 Jul 2013 07:27:55 +0200 Subject: Help References: <067018ce-ca0c-415b-a5d3-3c2f79f11e8e@googlegroups.com> Message-ID: tyler at familyrobbins.com wrote: > I'm a bit new to python and I'm trying to create a simple program which > adds words and definitions to a list, and then calls them forward when > asked to. > while escape < 1: > > choice = input("Type 'Entry' to add a word to the Dictionary, 'Search' > to find a word, and 'Exit' to quit. ") > > if choice == '`': > break > > if choice == 'Entry' or 'entry': > However, if I run the program using anything but 'entry', the program > still runs the 'entry' subroutine. After the 'entry' subroutine is run > once, no options work. Ex: The expression choice == 'Entry' or 'entry' is evaluated by Python as (choice == 'Entry') or 'entry' All strings but "" are True in a boolean context, so this has the same effect as (choice == 'Entry') or True and is always True. Possible fixes: if choice == "Entry" or choice == "entry": if choice in ("Entry", "entry"): if choice.casefold() == "entry".casefold(): # will accept ENTRY, eNtRy etc. From rosuav at gmail.com Fri Jul 26 01:35:41 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 26 Jul 2013 06:35:41 +0100 Subject: Help In-Reply-To: <067018ce-ca0c-415b-a5d3-3c2f79f11e8e@googlegroups.com> References: <067018ce-ca0c-415b-a5d3-3c2f79f11e8e@googlegroups.com> Message-ID: On Fri, Jul 26, 2013 at 6:06 AM, wrote: > I'm a bit new to python and I'm trying to create a simple program which adds words and definitions to a list, and then calls them forward when asked to. One of the most important tidbits of information is: What version of Python are you using? > print("Welcome to Digital Dictionary V1.0!\n\n") This looks like Python 3 style, though it would be valid Python 2 too. > choice = input("Type 'Entry' to add a word to the Dictionary, 'Search' to find a word, and 'Exit' to quit. ") And I sincerely hope that this is Python 3, otherwise it would be doing some very strange and dangerous things. However, relying on us to guess isn't particularly safe, and works only for the major branch of 2.x vs 3.x. The actual version you're using will be helpful. > while finish < 1: > if True: > finish = 1 > entry = 0 > definition = 0 You're doing a lot with sentinel values, and it's getting a bit confusing. Your problem here seems to be that 'finish' is never reset, so the second time through, your loop is completely skipped. I recommend instead that you use 'break' to exit your loops. In a piece of somewhat amusing irony, you're here trying to implement a dictionary. Python has an object type of that name ('dict' is short for dictionary), which will be extremely useful here. I advise you to explore it - it'll replace your words[] and dictionary[] lists. ChrisA From tjreedy at udel.edu Fri Jul 26 01:46:03 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 26 Jul 2013 01:46:03 -0400 Subject: Help In-Reply-To: <067018ce-ca0c-415b-a5d3-3c2f79f11e8e@googlegroups.com> References: <067018ce-ca0c-415b-a5d3-3c2f79f11e8e@googlegroups.com> Message-ID: On 7/26/2013 1:06 AM, tyler at familyrobbins.com wrote: > I'm a bit new to python and I'm trying to create a simple program > which adds words and definitions to a list, and then calls them > forward when asked to. Why are you not putting them in a Python dict, which is made for this? Unless you have a very specialized purpose, it would be more educational about Python programming. -- Terry Jan Reedy From tyler at familyrobbins.com Fri Jul 26 23:40:52 2013 From: tyler at familyrobbins.com (tyler at familyrobbins.com) Date: Fri, 26 Jul 2013 20:40:52 -0700 (PDT) Subject: Help In-Reply-To: <067018ce-ca0c-415b-a5d3-3c2f79f11e8e@googlegroups.com> References: <067018ce-ca0c-415b-a5d3-3c2f79f11e8e@googlegroups.com> Message-ID: <084651f1-439d-40e9-b112-a86b739eb151@googlegroups.com> Thank you everybody who replied. The problem is fixed now and the program is running correctly. I will also try to use your suggestions in the future to make sure I don't make the same mistake. Thanks again :). From me at davecotter.com Fri Jul 26 02:15:28 2013 From: me at davecotter.com (David M. Cotter) Date: Thu, 25 Jul 2013 23:15:28 -0700 (PDT) Subject: embedded python and threading Message-ID: in my app i initialize python on the main thread, then immediately call PyEval_SaveThread() because i do no further python stuff on the main thread. then, for each script i want to run, i use boost::threads to create a new thread, then on that thread i "ensure" the GIL, do my stuff, then release it. so, to test concurrency, on my first background thread, i do an infinite loop that just logs "i'm alive", then calls sleep(0.25) so that thread continues to run forever (with it's GIL ensured) according to the doc: "In order to emulate concurrency of execution, the interpreter regularly tries to switch threads" so i figure i can run another thread that does a single print statement: > ensure gil > print my thing > release gil and this DOES run. however, after releasing it's gil, i guess the interpeter gets back to the first back thread, but then has this error immediately: 9: Traceback (most recent call last): 9: File "", line 70, in ? 9: File "", line 55, in main 9: AttributeError: 'builtin_function_or_method' object has no attribute 'sleep' suddenly the sleep module has been unloaded?? huh? i thought the thread state had been preserved? From stefan_ml at behnel.de Fri Jul 26 05:29:49 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 26 Jul 2013 11:29:49 +0200 Subject: embedded python and threading In-Reply-To: References: Message-ID: David M. Cotter, 26.07.2013 08:15: > in my app i initialize python on the main thread, then immediately call PyEval_SaveThread() because i do no further python stuff on the main thread. > > then, for each script i want to run, i use boost::threads to create a new thread, then on that thread i "ensure" the GIL, do my stuff, then release it. > > so, to test concurrency, on my first background thread, i do an infinite loop that just logs "i'm alive", then calls sleep(0.25) > > so that thread continues to run forever (with it's GIL ensured) > > according to the doc: "In order to emulate concurrency of execution, the interpreter regularly tries to switch threads" > > so i figure i can run another thread that does a single print statement: > >> ensure gil >> print my thing >> release gil > > and this DOES run. however, after releasing it's gil, i guess the interpeter gets back to the first back thread, but then has this error immediately: > > 9: Traceback (most recent call last): > 9: File "", line 70, in ? > 9: File "", line 55, in main > 9: AttributeError: 'builtin_function_or_method' object has no attribute 'sleep' > > suddenly the sleep module has been unloaded?? huh? i thought the thread state had been preserved? You didn't show your code, but as a wild guess, maybe you did from time import time instead of import time somewhere? If not, please provide the exact example code that you are running. Stefan From me at davecotter.com Fri Jul 26 11:57:25 2013 From: me at davecotter.com (David M. Cotter) Date: Fri, 26 Jul 2013 08:57:25 -0700 (PDT) Subject: embedded python and threading In-Reply-To: References: Message-ID: <965b463e-e5bf-4ccd-9a3c-b0cb964b3356@googlegroups.com> okay, i have simplified it: here is the code ========================================== import time def main(): while True: print "i'm alive" time.sleep(0.25) #--------------------------------- if __name__ == "__main__": main() ========================================== the new error is: ========================================== 9: Traceback (most recent call last): 9: File "", line 10, in ? 9: File "", line 6, in main 9: AttributeError: 'builtin_function_or_method' object has no attribute 'sleep' ========================================== From drobinow at gmail.com Fri Jul 26 12:14:39 2013 From: drobinow at gmail.com (David Robinow) Date: Fri, 26 Jul 2013 12:14:39 -0400 Subject: embedded python and threading In-Reply-To: <965b463e-e5bf-4ccd-9a3c-b0cb964b3356@googlegroups.com> References: <965b463e-e5bf-4ccd-9a3c-b0cb964b3356@googlegroups.com> Message-ID: Works for me. Except that if I then do: touch time.py I get the same error as you do. Can you figure out the problem now? On Fri, Jul 26, 2013 at 11:57 AM, David M. Cotter wrote: > okay, i have simplified it: here is the code > > ========================================== > import time > > def main(): > while True: > print "i'm alive" > time.sleep(0.25) > > #--------------------------------- > if __name__ == "__main__": > main() > ========================================== > > the new error is: > > ========================================== > 9: Traceback (most recent call last): > 9: File "", line 10, in ? > 9: File "", line 6, in main > 9: AttributeError: 'builtin_function_or_method' object has no attribute 'sleep' > ========================================== > -- > http://mail.python.org/mailman/listinfo/python-list From gordon at panix.com Fri Jul 26 12:34:38 2013 From: gordon at panix.com (John Gordon) Date: Fri, 26 Jul 2013 16:34:38 +0000 (UTC) Subject: embedded python and threading References: <965b463e-e5bf-4ccd-9a3c-b0cb964b3356@googlegroups.com> Message-ID: In <965b463e-e5bf-4ccd-9a3c-b0cb964b3356 at googlegroups.com> "David M. Cotter" writes: > ========================================== > 9: Traceback (most recent call last): > 9: File "", line 10, in ? > 9: File "", line 6, in main > 9: AttributeError: 'builtin_function_or_method' object has no attribute 'sleep' > ========================================== You must have a file named 'time.py' in the current directory, and the import statement is getting that module instead of the system time module. -- John Gordon A is for Amy, who fell down the stairs gordon at panix.com B is for Basil, assaulted by bears -- Edward Gorey, "The Gashlycrumb Tinies" From me at davecotter.com Fri Jul 26 13:25:07 2013 From: me at davecotter.com (David M. Cotter) Date: Fri, 26 Jul 2013 10:25:07 -0700 (PDT) Subject: embedded python and threading In-Reply-To: References: <965b463e-e5bf-4ccd-9a3c-b0cb964b3356@googlegroups.com> Message-ID: no, there is no "time.py" anywhere (except perhaps as the actual python library originally imported) did you understand that the function works perfectly, looping as it should, up until the time i run a second script on a separate thread? From me at davecotter.com Fri Jul 26 13:28:51 2013 From: me at davecotter.com (David M. Cotter) Date: Fri, 26 Jul 2013 10:28:51 -0700 (PDT) Subject: embedded python and threading In-Reply-To: References: <965b463e-e5bf-4ccd-9a3c-b0cb964b3356@googlegroups.com> Message-ID: <2c80d06f-6946-4624-ba3c-47ddfa9db117@googlegroups.com> DOH! as my second thread, i had been using a sample script that i had copy-pasted without much looking at it. guess what? it prints the time. and yes, it did "from time import time", which explains it all. thanks for the hints here, that helped me figure it out! From stefan_ml at behnel.de Fri Jul 26 14:38:36 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 26 Jul 2013 20:38:36 +0200 Subject: embedded python and threading In-Reply-To: <2c80d06f-6946-4624-ba3c-47ddfa9db117@googlegroups.com> References: <965b463e-e5bf-4ccd-9a3c-b0cb964b3356@googlegroups.com> <2c80d06f-6946-4624-ba3c-47ddfa9db117@googlegroups.com> Message-ID: David M. Cotter, 26.07.2013 19:28: > DOH! as my second thread, i had been using a sample script that i had copy-pasted without much looking at it. guess what? it prints the time. and yes, it did "from time import time", which explains it all. Ah, and you were using the same globals dict for both scripts, I guess? That explains it then. Stefan From rui.maciel at gmail.com Fri Jul 26 05:04:05 2013 From: rui.maciel at gmail.com (Rui Maciel) Date: Fri, 26 Jul 2013 10:04:05 +0100 Subject: Newbie: Python 3 and web applications? Message-ID: I'm currently learning Python, and I've been focusing on Python3. To try to kill two birds with one stone, I would also like to learn the basics of writing small web applications. These web applications don't need to do much more than provide an interface to a small database, and they may not even be required to be accessible outside of a LAN. Does anyone have any tips on what's the best way to start off this adventure? Thanks in advance, Rui Maciel From gordon at panix.com Fri Jul 26 10:41:40 2013 From: gordon at panix.com (John Gordon) Date: Fri, 26 Jul 2013 14:41:40 +0000 (UTC) Subject: Newbie: Python 3 and web applications? References: Message-ID: In Rui Maciel writes: > I'm currently learning Python, and I've been focusing on Python3. To try to > kill two birds with one stone, I would also like to learn the basics of > writing small web applications. > These web applications don't need to do much more than provide an interface > to a small database, and they may not even be required to be accessible > outside of a LAN. > Does anyone have any tips on what's the best way to start off this > adventure? I recommend using a web framework such as Django or Pylons. It's more to learn, but frameworks handle a ton of low-level details for you and make web development overall much easier. -- John Gordon A is for Amy, who fell down the stairs gordon at panix.com B is for Basil, assaulted by bears -- Edward Gorey, "The Gashlycrumb Tinies" From wayne at waynewerner.com Tue Jul 30 17:14:13 2013 From: wayne at waynewerner.com (Wayne Werner) Date: Tue, 30 Jul 2013 16:14:13 -0500 (CDT) Subject: Newbie: Python 3 and web applications? In-Reply-To: References: Message-ID: On Fri, 26 Jul 2013, Rui Maciel wrote: > I'm currently learning Python, and I've been focusing on Python3. To try to > kill two birds with one stone, I would also like to learn the basics of > writing small web applications. > > These web applications don't need to do much more than provide an interface > to a small database, and they may not even be required to be accessible > outside of a LAN. > > Does anyone have any tips on what's the best way to start off this > adventure? Take a look at the Python3 branch of Flask: https://github.com/mitsuhiko/flask.git And the werkzeug webserver: https://github.com/mitsuhiko/werkzeug.git If you download these you can install them with: > python setup.py install (werkzeug first, then flask) Here's the most basic webserver you can create that way: from flask import Flask app = Flask(__name__) @app.route("/") def main(): return "Hello, Web!" if __name__ == "__main__": app.run() And yet flask is highly extensible with a lot of plugins. HTH, W From ron.eggler at gmail.com Fri Jul 26 16:21:47 2013 From: ron.eggler at gmail.com (cerr) Date: Fri, 26 Jul 2013 13:21:47 -0700 (PDT) Subject: dump a multi dimensional dictionary Message-ID: Hi, Can I somehow use pickle.dump() to store a dictionary of lists to a file? I tried this: >>> import pickle >>> mylist = [] >>> mydict = {} >>> mylist = '1','2' >>> mydict['3'] = mylist >>> fhg = open ("test", 'w') >>> pickle.dump(fhg,mydict) Traceback (most recent call last): File "", line 1, in File "/usr/lib/python2.7/pickle.py", line 1370, in dump Pickler(file, protocol).dump(obj) File "/usr/lib/python2.7/pickle.py", line 203, in __init__ self.write = file.write AttributeError: 'dict' object has no attribute 'write' >>> print mydict {'3': ('1', '2')} or should I just write my own dump function that can hanle thiS? Please advise! Thanks, Ron From gordon at panix.com Fri Jul 26 17:00:24 2013 From: gordon at panix.com (John Gordon) Date: Fri, 26 Jul 2013 21:00:24 +0000 (UTC) Subject: dump a multi dimensional dictionary References: Message-ID: In cerr writes: > Can I somehow use pickle.dump() to store a dictionary of lists to a file? > I tried this: > >>> import pickle > >>> mylist = [] > >>> mydict = {} > >>> mylist = '1','2' > >>> mydict['3'] = mylist > >>> fhg = open ("test", 'w') > >>> pickle.dump(fhg,mydict) > Traceback (most recent call last): > File "", line 1, in > File "/usr/lib/python2.7/pickle.py", line 1370, in dump > Pickler(file, protocol).dump(obj) > File "/usr/lib/python2.7/pickle.py", line 203, in __init__ > self.write = file.write > AttributeError: 'dict' object has no attribute 'write' > >>> print mydict > {'3': ('1', '2')} > or should I just write my own dump function that can hanle thiS? I think you have the arguments to pickle.dump() in the wrong order. The data to be dumped should come first, then the file object. -- John Gordon A is for Amy, who fell down the stairs gordon at panix.com B is for Basil, assaulted by bears -- Edward Gorey, "The Gashlycrumb Tinies" From ramit.prasad at jpmorgan.com Fri Jul 26 16:40:54 2013 From: ramit.prasad at jpmorgan.com (Prasad, Ramit) Date: Fri, 26 Jul 2013 20:40:54 +0000 Subject: dump a multi dimensional dictionary In-Reply-To: References: Message-ID: <5B80DD153D7D744689F57F4FB69AF474185E858F@SCACMX008.exchad.jpmchase.net> cerr wrote: > Hi, > > Can I somehow use pickle.dump() to store a dictionary of lists to a file? > I tried this: > > >>> import pickle > >>> mylist = [] > >>> mydict = {} > >>> mylist = '1','2' > >>> mydict['3'] = mylist > >>> fhg = open ("test", 'w') > >>> pickle.dump(fhg,mydict) > Traceback (most recent call last): > File "", line 1, in > File "/usr/lib/python2.7/pickle.py", line 1370, in dump > Pickler(file, protocol).dump(obj) > File "/usr/lib/python2.7/pickle.py", line 203, in __init__ > self.write = file.write > AttributeError: 'dict' object has no attribute 'write' > >>> print mydict > {'3': ('1', '2')} > > or should I just write my own dump function that can hanle thiS? > > Please advise! > > Thanks, > Ron I think you have the parameters for dump backwards. According to API http://docs.python.org/2/library/pickle.html#pickle.dump the format is: pickle.dump(obj, file, protocol=None) Which means you need to use: pickle.dump(mydict, fhg) Ramit This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email. From rosuav at gmail.com Fri Jul 26 19:33:58 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 27 Jul 2013 00:33:58 +0100 Subject: dump a multi dimensional dictionary In-Reply-To: References: Message-ID: On Fri, Jul 26, 2013 at 9:21 PM, cerr wrote: > >>> mylist = [] > >>> mydict = {} > >>> mylist = '1','2' Side point: mylist is no longer a list, it's a tuple. I don't think pickle has problems with tuples, but it's worth noting that difference. ChrisA From steve+comp.lang.python at pearwood.info Fri Jul 26 22:54:28 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 27 Jul 2013 02:54:28 GMT Subject: dump a multi dimensional dictionary References: Message-ID: <51f33663$0$29971$c3e8da3$5496439d@news.astraweb.com> On Fri, 26 Jul 2013 13:21:47 -0700, cerr wrote: > or should I just write my own dump function that can hanle thiS? > > Please advise! Given the choice between reading the documentation to pickle.dump: help(pickle.dump) or spending the next month writing a replacement for pickle, testing it, debugging it, going through revision after revision to try to bring it to the same level of maturity as pickle, which would you prefer? :-) -- Steven From b.krishna2020 at gmail.com Sat Jul 27 01:05:11 2013 From: b.krishna2020 at gmail.com (b.krishna2020 at gmail.com) Date: Fri, 26 Jul 2013 22:05:11 -0700 (PDT) Subject: Best python web framework to build university/academic website Message-ID: <584fc922-a42b-40cb-bd3c-5fa7383dc2b7@googlegroups.com> Hi, I got a chance to build an university website, within very short period of time. I know web2py, little bit of Django, so please suggest me the best to build rapidly. Thanks in advance Raghu From bryanjugglercryptographer at yahoo.com Wed Jul 31 01:23:51 2013 From: bryanjugglercryptographer at yahoo.com (bryanjugglercryptographer at yahoo.com) Date: Tue, 30 Jul 2013 22:23:51 -0700 (PDT) Subject: Best python web framework to build university/academic website In-Reply-To: <584fc922-a42b-40cb-bd3c-5fa7383dc2b7@googlegroups.com> References: <584fc922-a42b-40cb-bd3c-5fa7383dc2b7@googlegroups.com> Message-ID: b.kris... at gmail.com wrote: > I got a chance to build an university website, within very short period of time. > I know web2py, little bit of Django, so please suggest me the best to build rapidly. Web2py rocks like nothing else for getting up fast. If you already know it, problem solved. That said, Django has more available production-quality (free) utility apps. You might might want to check whether someone else has already done your work for you. I'm leaning toward Django because it's ahead of Web2py in Python 3 support. -- --Bryan From devyncjohnson at gmail.com Sat Jul 27 06:58:03 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Sat, 27 Jul 2013 06:58:03 -0400 Subject: Cross-Platform Python3 Equivalent to notify-send Message-ID: <51F3A7BB.4000305@Gmail.com> Linux systems with the proper software can use the "notify-send" command. Is there a cross-platform Python3 equivalent? Mahalo, Devyn Collier Johnson DevynCJohnson at Gmail.com From kwpolska at gmail.com Sat Jul 27 07:30:23 2013 From: kwpolska at gmail.com (=?UTF-8?B?Q2hyaXMg4oCcS3dwb2xza2HigJ0gV2Fycmljaw==?=) Date: Sat, 27 Jul 2013 13:30:23 +0200 Subject: Cross-Platform Python3 Equivalent to notify-send In-Reply-To: <51F3A7BB.4000305@Gmail.com> References: <51F3A7BB.4000305@Gmail.com> Message-ID: On Sat, Jul 27, 2013 at 12:58 PM, Devyn Collier Johnson wrote: > Linux systems with the proper software can use the "notify-send" command. Is > there a cross-platform Python3 equivalent? > > Mahalo, > > Devyn Collier Johnson > DevynCJohnson at Gmail.com > -- > http://mail.python.org/mailman/listinfo/python-list You already asked this on Thursday. And the answer is probably ?no?. Creating Under X11-based systems, you would have to call the dbus notification APIs and pray that the user has something to handle it running (KDE, GNOME Shell, XFCE4?s notification daemon). Under Mac OS X 10.7 and further, you need to work with some system APIs, and that may not be easy, but possible (eg. https://github.com/alloy/terminal-notifier for Ruby). But Windows? GOOD LUCK! The following options exist, none of which is easy to implement, and one of which is not usable with most clients: a) Toast Notifications in Windows 8/Server 2012, which is not a popular platform and may require quite a lot of magic in terms of coding and else (VS2012); b) Create a tray icon and do a balloon (2000 and up?, definitely in XP); c) Create your very own Windows toast notifications framework. -- Chris ?Kwpolska? Warrick PGP: 5EAAEA16 stop html mail | always bottom-post | only UTF-8 makes sense From kwpolska at gmail.com Sat Jul 27 07:31:45 2013 From: kwpolska at gmail.com (=?UTF-8?B?Q2hyaXMg4oCcS3dwb2xza2HigJ0gV2Fycmljaw==?=) Date: Sat, 27 Jul 2013 13:31:45 +0200 Subject: Cross-Platform Python3 Equivalent to notify-send In-Reply-To: References: <51F3A7BB.4000305@Gmail.com> Message-ID: Whoops! On Sat, Jul 27, 2013 at 1:30 PM, Chris ?Kwpolska? Warrick wrote: > You already asked this on Thursday. And the answer is probably ?no?. Creating ?a solution for this is not the easiest thing one can do. -- Chris ?Kwpolska? Warrick PGP: 5EAAEA16 stop html mail | always bottom-post | only UTF-8 makes sense From devyncjohnson at gmail.com Sat Jul 27 08:22:00 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Sat, 27 Jul 2013 08:22:00 -0400 Subject: Cross-Platform Python3 Equivalent to notify-send In-Reply-To: References: <51F3A7BB.4000305@Gmail.com> Message-ID: <51F3BB68.2040307@Gmail.com> On 07/27/2013 07:30 AM, Chris ?Kwpolska? Warrick wrote: > On Sat, Jul 27, 2013 at 12:58 PM, Devyn Collier Johnson > wrote: >> Linux systems with the proper software can use the "notify-send" command. Is >> there a cross-platform Python3 equivalent? >> >> Mahalo, >> >> Devyn Collier Johnson >> DevynCJohnson at Gmail.com >> -- >> http://mail.python.org/mailman/listinfo/python-list > You already asked this on Thursday. And the answer is probably ?no?. Creating > > Under X11-based systems, you would have to call the dbus notification > APIs and pray that the user has something to handle it running (KDE, > GNOME Shell, XFCE4?s notification daemon). Under Mac OS X 10.7 and > further, you need to work with some system APIs, and that may not be > easy, but possible (eg. https://github.com/alloy/terminal-notifier for > Ruby). > > But Windows? GOOD LUCK! The following options exist, none of which > is easy to implement, and one of which is not usable with most > clients: > > a) Toast Notifications in Windows 8/Server 2012, which is not a > popular platform and may require quite a lot of magic in terms of > coding and else (VS2012); > b) Create a tray icon and do a balloon (2000 and up?, definitely in XP); > c) Create your very own Windows toast notifications framework. > That really sucks. I was hoping Python had some way of doing that. All that it needs to do is display a little box at one of the corners of the screen. I thought someone would have implemented something by now. Thank you anyway. Mahalo, DCJ From rosuav at gmail.com Sat Jul 27 08:47:47 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 27 Jul 2013 13:47:47 +0100 Subject: Cross-Platform Python3 Equivalent to notify-send In-Reply-To: <51F3BB68.2040307@Gmail.com> References: <51F3A7BB.4000305@Gmail.com> <51F3BB68.2040307@Gmail.com> Message-ID: On Sat, Jul 27, 2013 at 1:22 PM, Devyn Collier Johnson wrote: > That really sucks. I was hoping Python had some way of doing that. All that > it needs to do is display a little box at one of the corners of the screen. > I thought someone would have implemented something by now. Thank you anyway. Despite the best efforts of a pretty talented core dev team, Python is not yet capable of magic :) If you browse the python-dev archives, you'll see how much of a nightmare cross-platform compatibility can be (eg the recent discussion on cloexec and passing file descriptors to subprocesses); often, what you might think (from a user's point of view) is fairly trivial will turn out to be quite tricky. That said, though, a lot of GUI toolkits will have a means for you to highlight a window. In GTK, it's called "present" (as in, "Lord User, may I present Sir Window and Mrs Window?"). There may be window managers that don't support the feature (and there are certainly those that let the user disable it, which you should respect), but AFAIK all of them should at least accept the command. http://www.pygtk.org/docs/pygtk/class-gtkwindow.html#method-gtkwindow--present So your best bet may be to simply create yourself a small window, then present it. On Windows XP, I think that'll flash the window in the task bar, which is usually enough highlight. On my Debian Wheezy with Xfce, it brings the window to the top of the Z-order, and optionally moves it to the current workspace (user's option, NOT program's). ChrisA From steve+comp.lang.python at pearwood.info Sat Jul 27 11:59:34 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 27 Jul 2013 15:59:34 GMT Subject: Cross-Platform Python3 Equivalent to notify-send References: <51F3A7BB.4000305@Gmail.com> Message-ID: <51f3ee66$0$29971$c3e8da3$5496439d@news.astraweb.com> On Sat, 27 Jul 2013 08:22:00 -0400, Devyn Collier Johnson wrote: > On 07/27/2013 07:30 AM, Chris ?Kwpolska? Warrick wrote: >> On Sat, Jul 27, 2013 at 12:58 PM, Devyn Collier Johnson >> wrote: >>> Linux systems with the proper software can use the "notify-send" >>> command. Is there a cross-platform Python3 equivalent? [snip answer "no"] > That really sucks. I was hoping Python had some way of doing that. All > that it needs to do is display a little box at one of the corners of the > screen. I thought someone would have implemented something by now. Thank > you anyway. If it's that simple, how about you do it? Won't take you more than, oh, ten minutes, right? *wink* Don't underestimate the difficulty of "trivial" code when it has to work on two dozens different platforms with different capabilities: Windows XP, 2000, Vista, 7, 8 ... Mac OS-X FreeBSD, OpenBSD, Linux, running KDE (3 or 4?), Gnome (2 or 3?), Trinity, RatPoison, XFCE, something else... or no window manager at all -- Steven From rosuav at gmail.com Sat Jul 27 12:06:01 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 27 Jul 2013 17:06:01 +0100 Subject: Cross-Platform Python3 Equivalent to notify-send In-Reply-To: <51f3ee66$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <51F3A7BB.4000305@Gmail.com> <51f3ee66$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Sat, Jul 27, 2013 at 4:59 PM, Steven D'Aprano wrote: > On Sat, 27 Jul 2013 08:22:00 -0400, Devyn Collier Johnson wrote: >> That really sucks. I was hoping Python had some way of doing that. All >> that it needs to do is display a little box at one of the corners of the >> screen. I thought someone would have implemented something by now. Thank >> you anyway. > > If it's that simple, how about you do it? Won't take you more than, oh, > ten minutes, right? > > *wink* > > Don't underestimate the difficulty of "trivial" code when it has to work > on two dozens different platforms with different capabilities That doesn't mean the question shouldn't be asked, of course. Steven isn't (at least, I don't think he is!) scorning you for asking. Sometimes the response is quite the opposite - that it's so utterly trivial that there's no NEED for a library function! Never hurts to throw the question out there... ChrisA From devyncjohnson at gmail.com Sat Jul 27 13:03:51 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Sat, 27 Jul 2013 13:03:51 -0400 Subject: Cross-Platform Python3 Equivalent to notify-send In-Reply-To: References: <51F3A7BB.4000305@Gmail.com> <51f3ee66$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51F3FD77.1040702@Gmail.com> On 07/27/2013 12:06 PM, Chris Angelico wrote: > On Sat, Jul 27, 2013 at 4:59 PM, Steven D'Aprano > wrote: >> On Sat, 27 Jul 2013 08:22:00 -0400, Devyn Collier Johnson wrote: >>> That really sucks. I was hoping Python had some way of doing that. All >>> that it needs to do is display a little box at one of the corners of the >>> screen. I thought someone would have implemented something by now. Thank >>> you anyway. >> If it's that simple, how about you do it? Won't take you more than, oh, >> ten minutes, right? >> >> *wink* >> >> Don't underestimate the difficulty of "trivial" code when it has to work >> on two dozens different platforms with different capabilities > That doesn't mean the question shouldn't be asked, of course. Steven > isn't (at least, I don't think he is!) scorning you for asking. > Sometimes the response is quite the opposite - that it's so utterly > trivial that there's no NEED for a library function! Never hurts to > throw the question out there... > > ChrisA I wanted to make a module (boash) that would be useful, but clearly it is not. Here, is a feature that is useful that no one has. This may be the module I decide to make. So, there is no cross-platform for such a command any where? Would this be a useful module? I know Steven is not scorning me. He is making me think and I have an idea. Mahalo, DCJ From devyncjohnson at gmail.com Sat Jul 27 14:08:52 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Sat, 27 Jul 2013 14:08:52 -0400 Subject: Cross-Platform Python3 Equivalent to notify-send In-Reply-To: <51f3ee66$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <51F3A7BB.4000305@Gmail.com> <51f3ee66$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51F40CB4.3060209@Gmail.com> On 07/27/2013 11:59 AM, Steven D'Aprano wrote: > On Sat, 27 Jul 2013 08:22:00 -0400, Devyn Collier Johnson wrote: > >> On 07/27/2013 07:30 AM, Chris ?Kwpolska? Warrick wrote: >>> On Sat, Jul 27, 2013 at 12:58 PM, Devyn Collier Johnson >>> wrote: >>>> Linux systems with the proper software can use the "notify-send" >>>> command. Is there a cross-platform Python3 equivalent? > [snip answer "no"] > >> That really sucks. I was hoping Python had some way of doing that. All >> that it needs to do is display a little box at one of the corners of the >> screen. I thought someone would have implemented something by now. Thank >> you anyway. > If it's that simple, how about you do it? Won't take you more than, oh, > ten minutes, right? > > *wink* > > Don't underestimate the difficulty of "trivial" code when it has to work > on two dozens different platforms with different capabilities: > > Windows XP, 2000, Vista, 7, 8 ... > Mac OS-X > FreeBSD, OpenBSD, Linux, running KDE (3 or 4?), Gnome (2 or 3?), Trinity, > RatPoison, XFCE, something else... or no window manager at all > > > Yeah, good point Steven. It seems like it would be easy, but I know it is not. DCJ From kw at codebykevin.com Sat Jul 27 14:57:48 2013 From: kw at codebykevin.com (Kevin Walzer) Date: Sat, 27 Jul 2013 14:57:48 -0400 Subject: Cross-Platform Python3 Equivalent to notify-send In-Reply-To: References: Message-ID: On 7/27/13 6:58 AM, Devyn Collier Johnson wrote: > Linux systems with the proper software can use the "notify-send" > command. Is there a cross-platform Python3 equivalent? > > Mahalo, > > Devyn Collier Johnson > DevynCJohnson at Gmail.com http://pythonhosted.org/gntp/ ? -- Kevin Walzer Code by Kevin/Mobile Code by Kevin http://www.codebykevin.com http://www.wtmobilesoftware.com From aliencat777 at gmail.com Sat Jul 27 08:01:45 2013 From: aliencat777 at gmail.com (aliencat777 at gmail.com) Date: Sat, 27 Jul 2013 05:01:45 -0700 (PDT) Subject: Best Python book(s) for a pre-teen? In-Reply-To: <20619edc.0302181127.6ad39e68@posting.google.com> References: <20619edc.0302181127.6ad39e68@posting.google.com> Message-ID: Hi, "Start Here: Python 3x Programming Made Fun and Easier" is a very easy to understand beginners book. It also covers a basic introduction to software design, version systems, game logic and design, and packaging your first programs. I wrote this book originally for my two sons when I was home schooling them. You can download the first chapters, "free version" to try before you buy it at www.toonzcat.com/book.html Jody From aliencat777 at gmail.com Sat Jul 27 08:04:54 2013 From: aliencat777 at gmail.com (aliencat777 at gmail.com) Date: Sat, 27 Jul 2013 05:04:54 -0700 (PDT) Subject: Which is the best book to learn python In-Reply-To: <188c0a9f-4e93-4fae-b3f9-e7c84f8030fd@v31g2000pri.googlegroups.com> References: <188c0a9f-4e93-4fae-b3f9-e7c84f8030fd@v31g2000pri.googlegroups.com> Message-ID: <4b5922d2-288a-48d1-9063-71bd87049ac7@googlegroups.com> On Tuesday, January 25, 2011 1:09:31 AM UTC+8, sant wrote: > Hi All, > i am beginner to python please tell me which is the best available > reference for beginner to start from novice A great up to date beginner's book/course is "Start Here: Python 3x Programming, Made Fun and Easier." It introduces software design and makes everything very simple to understand. http://www.quantumsight.mobi Jody From aliencat777 at gmail.com Sat Jul 27 08:11:02 2013 From: aliencat777 at gmail.com (aliencat777 at gmail.com) Date: Sat, 27 Jul 2013 05:11:02 -0700 (PDT) Subject: Q: "Best" book for teaching In-Reply-To: <57e72bda-450b-4bf0-b204-7049ddac1dc2@z14g2000yqa.googlegroups.com> References: <8e3d0032-5e9f-44c2-9380-1d2383552046@u5g2000vbc.googlegroups.com> <57e72bda-450b-4bf0-b204-7049ddac1dc2@z14g2000yqa.googlegroups.com> Message-ID: On Wednesday, April 8, 2009 7:34:55 AM UTC+8, John Yeung wrote: > On Apr 6, 10:37?am, grkunt... at gmail.com wrote: > > I am considering teaching an "introduction to programming" > > course for continuing education adults at a local?community > > college. These would people with no programming experience, > > but I will require a reasonable facility with computers. > > > > What would be a good book to use as the text for the course? > > For an extremely gentle introduction, perhaps take a look at _Python > Programming for the Absolute Beginner_ by Michael Dawson: > > http://www.amazon.com/Python-Programming-Absolute-Beginner-Michael/dp/1592000738 > > A coworker of mine recently bought this book, which is how I found out > about it. Besides assuming no programming experience, it tries to > stay interesting through the use of simple games for its examples > (eventually building up to the use of the popular pygame library). > The writing style is definitely more "fun" than "academic". > > John An up to date book that is great for a short 21 lesson course is; "Start Here: Python 3x Programming Made Fun and Easier" by Jody S. Ginther. This is a good beginning programming course that introduces the entire process of software design including; version systems, planning, logic, and packaging. It takes a newbie from ground zero to making arcade style games complete with sound, music, graphics, and an creating an installation package in only 21 easy to follow lessons. You can find it at http://www.quantum-sight.com Jody From cenkalti at gmail.com Sat Jul 27 12:20:30 2013 From: cenkalti at gmail.com (=?UTF-8?Q?Cenk_Alt=C4=B1?=) Date: Sat, 27 Jul 2013 19:20:30 +0300 Subject: PyPI Notifier watches your requirements.txt files Message-ID: Hello Python, I would like tell you about my project that I have built on my spare time. http://www.pypi-notifier.org/ Basically, you login with your GitHub account and select your Python repositories which have a requirements.txt file. Then, it starts to watch the dependencies listed in requirements.txt files. When a new version of a dependency is released on PyPI you get an email. Enjoy. Cenk Alti -------------- next part -------------- An HTML attachment was scrubbed... URL: From lauraburt4 at icloud.com Sat Jul 27 12:47:55 2013 From: lauraburt4 at icloud.com (Laura Burt) Date: Sun, 28 Jul 2013 02:47:55 +1000 Subject: Oxycodone without a prescription online with overnight delivery | buy Oxycodone no visa online without prescription Message-ID: <3C4527DE-88C4-44A2-B025-9BD0C7AB3E2F@icloud.com> Laura Burt 0400 954 448 -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.b at gmail.com Sat Jul 27 13:48:34 2013 From: m.b at gmail.com (Marco) Date: Sat, 27 Jul 2013 19:48:34 +0200 Subject: Division and multiplication have a different behavior in the overflow case Message-ID: In Python 3, when we hava a division and both the result and at least one operand are too large to convert to float, we get an exception: >>> 2**1028 / 2**-2 Traceback (most recent call last): File "", line 1, in OverflowError: long int too large to convert to float When the result is inside the limits, we get the right value: >>> 2 ** 1025 / 2**10 3.511119404027961e+305 Why the behavior is different in the case of the multiplication? >>> 2 ** 1025 * 2**-10 Traceback (most recent call last): File "", line 1, in OverflowError: long int too large to convert to float I think the multiplication should have the same behavior than the division: * `inf` or `-inf` when the operands are inside the limits, but the result is not * `OverflowError` when the result, and at least one operand, are out of range. -- Marco Buttu From tjreedy at udel.edu Sat Jul 27 16:56:48 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 27 Jul 2013 16:56:48 -0400 Subject: Division and multiplication have a different behavior in the overflow case In-Reply-To: References: Message-ID: On 7/27/2013 1:48 PM, Marco wrote: > In Python 3, when we hava a division and both the result and at least > one operand are too large to convert to float, we get an exception: > > >>> 2**1028 / 2**-2 int/float gets converted to float/float and the int to float conversion is not possible. The result is irrelevant since it never gets calculated. > Traceback (most recent call last): > File "", line 1, in > OverflowError: long int too large to convert to float > > When the result is inside the limits, we get the right value: > > >>> 2 ** 1025 / 2**10 > 3.511119404027961e+305 This must be doing integer division and then converting the result to float, which it can. > Why the behavior is different in the case of the multiplication? > > >>> 2 ** 1025 * 2**-10 Mathematically this is the same thing, but computationally, it is not and you cannot expect to get the same answer. Like the first example, 2**-10 is a float, so 2**1025 must be converted to float to do float multiplication. But that is not possible. > Traceback (most recent call last): > File "", line 1, in > OverflowError: long int too large to convert to float > > I think the multiplication should have the same behavior than the > division: > > * `inf` or `-inf` when the operands are inside the limits, > but the result is not > * `OverflowError` when the result, and at least one operand, > are out of range. -- Terry Jan Reedy From syedk at pacificloud.com Sat Jul 27 19:15:01 2013 From: syedk at pacificloud.com (syed khalid) Date: Sat, 27 Jul 2013 16:15:01 -0700 (PDT) Subject: python import module question Message-ID: I am trying to do a "import shogun" in my python script. I can invoke shogun with a command line with no problem. But I cannot with a python import statement. >>>>>invoking python from a command line........... Syedk at syedk-ThinkPad-T410:~/shogun-2.0.0/src/interfaces/cmdline_static$ shogun | more libshogun (i686/v2.0.0_9c8012f_2012-09-04_09:08_164102447) Copyright (C) 1999-2009 Fraunhofer Institute FIRST Copyright (C) 1999-2011 Max Planck Society Copyright (C) 2009-2011 Berlin Institute of Technology Copyright (C) 2012 Soeren Sonnenburg, Sergey Lisitsyn, Heiko Strathmann Written (W) 1999-2012 Soeren Sonnenburg, Gunnar Raetsch et al. ( configure options: "configure options --interfaces=python_static" compile flag s: "-fPIC -g -Wall -Wno-unused-parameter -Wformat -Wformat-security -Wparenthese s -Wshadow -Wno-deprecated -O9 -fexpensive-optimizations -frerun-cse-after-loop -fcse-follow-jumps -finline-functions -fschedule-insns2 -fthread-jumps -fforce-a ddr -fstrength-reduce -funroll-loops -march=native -mtune=native -pthread" link flags: " -Xlinker --no-undefined" ) ( seeding random number generator with 3656470784 (seed size 256)) determined range for x in log(1+exp(-x)) is:37 ) >>>Trying to call python from a script in the same directory where I invoked the shogun from a command linesyedk at syedk-ThinkPad-T410:~/shogun-2.0.0/src/interfaces/cmdline_static$ python Python 2.7.3 (default, Apr 10 2013, 05:46:21) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import shogun Traceback (most recent call last): File "", line 1, in ImportError: No module named shogun >>> Regards From rosuav at gmail.com Sat Jul 27 19:33:08 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 28 Jul 2013 00:33:08 +0100 Subject: python import module question In-Reply-To: References: Message-ID: On Sun, Jul 28, 2013 at 12:15 AM, syed khalid wrote: > Syedk at syedk-ThinkPad-T410:~/shogun-2.0.0/src/interfaces/cmdline_static$ shogun | more This implies that you have something called 'shogun', without an extension, in your $PATH. Where is the actual script? You may need to install it by a quite different method, to make it available in Python. ChrisA From __peter__ at web.de Sun Jul 28 04:44:00 2013 From: __peter__ at web.de (Peter Otten) Date: Sun, 28 Jul 2013 10:44 +0200 Subject: python import module question References: Message-ID: syed khalid wrote: > I am trying to do a "import shogun" in my python script. I can invoke > shogun with a command line with no problem. But I cannot with a python > import statement. > >>>>>>invoking python from a command line........... > > Syedk at syedk-ThinkPad-T410:~/shogun-2.0.0/src/interfaces/cmdline_static$ > shogun | more libshogun (i686/v2.0.0_9c8012f_2012-09-04_09:08_164102447) > > Copyright (C) 1999-2009 Fraunhofer Institute FIRST > Copyright (C) 1999-2011 Max Planck Society > Copyright (C) 2009-2011 Berlin Institute of Technology > Copyright (C) 2012 Soeren Sonnenburg, Sergey Lisitsyn, Heiko Strathmann > Written (W) 1999-2012 Soeren Sonnenburg, Gunnar Raetsch et al. > > ( configure options: "configure options --interfaces=python_static" > compile flag s: "-fPIC -g -Wall -Wno-unused-parameter -Wformat > -Wformat-security -Wparenthese s -Wshadow -Wno-deprecated -O9 > -fexpensive-optimizations -frerun-cse-after-loop -fcse-follow-jumps > -finline-functions -fschedule-insns2 -fthread-jumps -fforce-a ddr > -fstrength-reduce -funroll-loops -march=native -mtune=native -pthread" > link flags: " -Xlinker --no-undefined" ) ( seeding random number generator > with 3656470784 (seed size 256)) determined range for x in log(1+exp(-x)) > is:37 ) > >>>>Trying to call python from a script in the same directory where I >>>>invoked the shogun from a command >>>>linesyedk at syedk-ThinkPad- T410:~/shogun-2.0.0/src/interfaces/cmdline_static$ >>>>python > Python 2.7.3 (default, Apr 10 2013, 05:46:21) > [GCC 4.6.3] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> import shogun > Traceback (most recent call last): > File "", line 1, in > ImportError: No module named shogun Poking around a bit on the project's website http://shogun-toolbox.org/doc/en/2.0.1/interfaces.html it looks like you are trying to use their "modular" interface when you have only installed the "static" one. I'm guessing that >>> import sg will work. From gouzounakis at hotmail.com Sat Jul 27 19:49:17 2013 From: gouzounakis at hotmail.com (D. Xenakis) Date: Sat, 27 Jul 2013 16:49:17 -0700 (PDT) Subject: PyQt5 and virtualenv problem Message-ID: <39e192a0-779e-4be6-b08f-950ed1c200ac@googlegroups.com> I tried to install SIP and PyQt5 using the pip install command but it didnt work on both cases (i was getting errors), so i finally installed them using the windows installers provided in riverbankcomputing website. My problem though here is that whenever i try to create a new virtualenv enviroment, those packages are not included and i cant import them. How can i add PyQt5 to my new virt enviroment? What is the logic behind this problem so i understand whats going on here? Thx in advance From gouzounakis at hotmail.com Mon Jul 29 09:00:34 2013 From: gouzounakis at hotmail.com (D. Xenakis) Date: Mon, 29 Jul 2013 06:00:34 -0700 (PDT) Subject: PyQt5 and virtualenv problem In-Reply-To: <39e192a0-779e-4be6-b08f-950ed1c200ac@googlegroups.com> References: <39e192a0-779e-4be6-b08f-950ed1c200ac@googlegroups.com> Message-ID: <1e9474e8-c94b-4edd-a9c8-f0fa53690817@googlegroups.com> Answer here: http://stackoverflow.com/questions/1961997/is-it-possible-to-add-pyqt4-pyside-packages-on-a-virtualenv-sandbox From gouzounakis at hotmail.com Mon Jul 29 22:36:08 2013 From: gouzounakis at hotmail.com (D. Xenakis) Date: Mon, 29 Jul 2013 19:36:08 -0700 (PDT) Subject: PyQt5 and virtualenv problem In-Reply-To: <1e9474e8-c94b-4edd-a9c8-f0fa53690817@googlegroups.com> References: <39e192a0-779e-4be6-b08f-950ed1c200ac@googlegroups.com> <1e9474e8-c94b-4edd-a9c8-f0fa53690817@googlegroups.com> Message-ID: Could you help me install PyQt5 properly in my Virtualenv folder and not only globally? I tried installing PyQt5 and sip with the use of pip but i was getting errors all the time (Why is that? Are there any known pip issues along with PyQt5 and sip?), so in the end i had to install it with the installer provided from the official riverbank website. However now, everytime i create a virtual enviroment, PyQT5 or sip package is not included there and i dont know how to solve this problem either. I tried applying this fix: http://stackoverflow.com/questions/1961997/is-it-possible-to-add-pyqt4-pyside-packages-on-a-virtualenv-sandbox , but i do not know if i have done things right. I can import PyQt5 and pip packages from within python console without any errors being showed (both globaly and inside my virtuall enviroment), so i could assume i did all ok. But should this confirmation be enough for me or is there any other way i could confirm that everything is indeed properly installed? I noticed that in start menu - programs, a new folder named "PyQt GPL v5.0 for Python v3.3 (x64)" had been created after the installation, where someone can find view PyQt examples and a PyQT Examples module which starts an application. Big question now.. Everytime i run PyQt Examples module from within global IDLE, everything works ok and then again when i close the application, this appears: Traceback (most recent call last): File "C:\Python33\Lib\site-packages\PyQt5\examples\qtdemo\qtdemo.pyw", line 91, in sys.exit(app.exec_()) SystemExit: 0 However, everytime i run PyQt Examples module from within virtual env IDLE, nothing happends. So as you can undertand, this is why i believe i have not installed properly PyQt5 or sip **(I start virtual env IDLE using this shortcut ""..path..\Python\Python Projects\PriceTAG Grabber\env\Scripts\pythonw.exe" C:\Python33\Lib\idlelib\idle.pyw") I know i may have asked too many questions in just a single topic, but all im trying to achieve here is to be sure that PyQt5 and virtual env are working fine. Thx for your time in advance From dan.h.mcinerney at gmail.com Sat Jul 27 22:29:53 2013 From: dan.h.mcinerney at gmail.com (dan.h.mcinerney at gmail.com) Date: Sat, 27 Jul 2013 19:29:53 -0700 (PDT) Subject: Thread is somehow interfering with a while loop called after the thread is started Message-ID: <416ade59-a3dc-47b1-b8d4-6f845ce1eaae@googlegroups.com> I have a simple scapy + nfqueue dns spoofing script that I want to turn into a thread within a larger program: http://www.bpaste.net/show/HrlfvmUBDA3rjPQdLmdp/ Below is my attempt to thread the program above. Somehow, the only way the while loop actually prints "running" is if the callback function is called consistently. If the callback function isn't started, the script will never print "running". How can that be if the while loop is AFTER the thread was started? Shouldn't the while loop and the thread operate independantly? http://bpaste.net/show/0aCxSsSW7yHcQ7EBLctI/ From irmen.NOSPAM at xs4all.nl Sun Jul 28 07:17:36 2013 From: irmen.NOSPAM at xs4all.nl (Irmen de Jong) Date: Sun, 28 Jul 2013 13:17:36 +0200 Subject: Thread is somehow interfering with a while loop called after the thread is started In-Reply-To: <416ade59-a3dc-47b1-b8d4-6f845ce1eaae@googlegroups.com> References: <416ade59-a3dc-47b1-b8d4-6f845ce1eaae@googlegroups.com> Message-ID: <51f4fdd1$0$15953$e4fe514c@news.xs4all.nl> On 28-7-2013 4:29, dan.h.mcinerney at gmail.com wrote: > I have a simple scapy + nfqueue dns spoofing script that I want to turn into a thread within a larger program: > > http://www.bpaste.net/show/HrlfvmUBDA3rjPQdLmdp/ > > Below is my attempt to thread the program above. Somehow, the only way the while loop actually prints "running" is if the callback function is called consistently. If the callback function isn't started, the script will never print "running". How can that be if the while loop is AFTER the thread was started? Shouldn't the while loop and the thread operate independantly? > > http://bpaste.net/show/0aCxSsSW7yHcQ7EBLctI/ > Try adding sys.stdout.flush() after your print statements, I think you're seeing a stdout buffering issue. Irmen From jaiprakashsingh213 at gmail.com Sun Jul 28 06:32:33 2013 From: jaiprakashsingh213 at gmail.com (Jaiky) Date: Sun, 28 Jul 2013 03:32:33 -0700 (PDT) Subject: Configuraion to run pyhton script on ubuntu 12.04 Message-ID: <2ebe3375-6aa6-45cd-b534-0241bd025eb3@googlegroups.com> want to run a python script which contains simple form of html on firefox browser , but dont know what should be the configuration on ubuntu 12.04 to run this script i.e cgi configuration My code is ubder in /var/www/cgi-bin/forms__.py #!/usr/bin/env python import webapp2 form ="""
""" class MainPage(webapp2.RequestHandler): def get(self): #self.response.headers['Content-Type'] = 'text/plain' self.response.out.write(form) app = webapp2.WSGIApplication([('/', MainPage)], debug=True) From pierre at jaury.eu Sun Jul 28 07:34:43 2013 From: pierre at jaury.eu (Pierre Jaury) Date: Sun, 28 Jul 2013 13:34:43 +0200 Subject: Configuraion to run pyhton script on ubuntu 12.04 References: <2ebe3375-6aa6-45cd-b534-0241bd025eb3@googlegroups.com> Message-ID: <87r4ei5358.fsf@silissia.kaiyou.fr> Jaiky writes: > want to run a python script which contains simple form of html on firefox browser , but dont know what should be the configuration on ubuntu 12.04 to run this script i.e cgi configuration > > > > My code is > ubder > in /var/www/cgi-bin/forms__.py > > > > #!/usr/bin/env python > import webapp2 > > form =""" >
> > > """ > > > class MainPage(webapp2.RequestHandler): > def get(self): > #self.response.headers['Content-Type'] = 'text/plain' > self.response.out.write(form) > > app = webapp2.WSGIApplication([('/', MainPage)], > debug=True) In order for you app to run as cgi, you would have to call the webapp2 run() function. Otherwise, it is going to implement wsgi interfaces. Have a look at: http://webapp-improved.appspot.com/api/webapp2.html#webapp2.WSGIApplication.run As well as: http://httpd.apache.org/docs/current/mod/mod_alias.html#scriptalias -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 489 bytes Desc: not available URL: From jaiprakashsingh213 at gmail.com Sun Jul 28 12:19:08 2013 From: jaiprakashsingh213 at gmail.com (Jaiky) Date: Sun, 28 Jul 2013 09:19:08 -0700 (PDT) Subject: Configuraion to run pyhton script on ubuntu 12.04 In-Reply-To: <87r4ei5358.fsf@silissia.kaiyou.fr> References: <2ebe3375-6aa6-45cd-b534-0241bd025eb3@googlegroups.com> <87r4ei5358.fsf@silissia.kaiyou.fr> Message-ID: Sir i already tried this "Alias" concept I did the following steps =============================================================================== Step 1: added "ScriptAlias /cgi-bin/ /var/www/cgi-bin/" in /etc/apache2/sites-available/default ============================================================================ step 2:- added :- def main(): app.run() if __name__ == '__main__': main() in /var/www/cgi-bin/hello_world.py ================================================================================ Now my Configuration of /etc/apache2/sites-available/default in under ServerAdmin webmaster at localhost DocumentRoot /var/www Options FollowSymLinks AllowOverride None AddHandler mod_python .py PythonHandler mod_python.publisher | .py AddHandler mod_python .psp .psp_ PythonHandler mod_python.psp | .psp .psp ScriptAlias /cgi-bin/ /var/www/cgi-bin/ Options Indexes FollowSymLinks MultiViews ExecCGI AllowOverride None Order allow,deny allow from all AddHandler cgi-script cgi pl #ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/ AllowOverride None Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch AddHandler cgi-script cgi pl Order allow,deny Allow from all ErrorLog ${APACHE_LOG_DIR}/error.log # Possible values include: debug, info, notice, warn, error, crit, # alert, emerg. LogLevel warn CustomLog ${APACHE_LOG_DIR}/access.log combined Alias /doc/ "/usr/share/doc/" Options Indexes MultiViews FollowSymLinks AllowOverride None Order deny,allow Deny from all Allow from 127.0.0.0/255.0.0.0 ::1/128 =============================================================================== my code is under /var/www/cgi-bin/hello_world.py import webapp2 class MainPage(webapp2.RequestHandler): def get(self): self.response.headers['Content-Type'] = 'text/plain' self.response.out.write('Hello, webapp World!') app = webapp2.WSGIApplication([('/', MainPage)], debug=True) def main(): app.run() if __name__ == '__main__': main() ============================================================================= extra thing i did in /etc/apache2/mods-available/mod_python.conf where i created the file "mod_python.conf" AddHandler mod_python .py .psp PythonHandler mod_python.publisher | .py PythonHandler mod_python.psp | .psp ################################################################################ when i run localhost/cgi-bin/hello_world.py error i get Not Found The requested URL /cgi-bin/hello_world.py was not found on this server. Apache/2.2.22 (Ubuntu) Server at localhost Port 80 ############################################################################# From jaiprakashsingh213 at gmail.com Mon Jul 29 06:44:49 2013 From: jaiprakashsingh213 at gmail.com (Jaiky) Date: Mon, 29 Jul 2013 03:44:49 -0700 (PDT) Subject: Configuraion to run pyhton script on ubuntu 12.04 In-Reply-To: <2ebe3375-6aa6-45cd-b534-0241bd025eb3@googlegroups.com> References: <2ebe3375-6aa6-45cd-b534-0241bd025eb3@googlegroups.com> Message-ID: <681351a1-5918-401e-96d0-b6a17c2dfcd9@googlegroups.com> Problem solved regarding cgi configuration on ubuntu 12.04 lts Concept:-) file used:-) /etc/apache2/httpd.conf /etc/apache2/sites-available/default ############################################################################ steps done 1:-) in /etc/apache2/httpd.conf added line #########for link localhost/python2/hello.py ScriptAlias /python2/ /var/www/python2/ AllowOverride None Options +Indexes FollowSymLinks +ExecCGI -MultiViews +SymLinksIfOwnerMatch Order allow,deny Allow from all #######fro runing of python script AddHandler cgi-script cgi pl mod_python .py PythonHandler mod_python.publisher | .py AddHandler mod_python .psp .psp_ PythonHandler mod_python.psp | .psp .psp PythonDebug On ######################################################################### step done 2:-) added line #######fro runing of python script AddHandler cgi-script cgi pl mod_python .py PythonHandler mod_python.publisher | .py AddHandler mod_python .psp .psp_ PythonHandler mod_python.psp | .psp .psp PythonDebug On between ----------------- --------------- HERE I ADDED Line ------------- ----------------- ----------------- --------------- HERE I ADDED Line ------------- ----------------- ############################################################################ Step Done 3:-) added line in /etc/apache2/mods-available/mod_python.conf ######mod_python.conf i created AddHandler mod_python .py .psp PythonHandler mod_python.publisher | .py PythonHandler mod_python.psp | .psp From rosuav at gmail.com Sun Jul 28 13:57:00 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 28 Jul 2013 18:57:00 +0100 Subject: [Savoynet] G&S Opera Co: Pirates of Penzance In-Reply-To: References: Message-ID: On Sun, Jul 28, 2013 at 6:36 PM, David Patterson wrote: > By the way, Chris, I think the book that Ruth brought on was probably > supposed to be Debretts Peerage. I couldn't see the cover clearly but it > would have been a more logical choice in view of the circumstances. Sure. Makes no difference what the book actually is, and I'm not someone who knows these things (people show me caricatures and I have no idea who they're of, largely due to not watching TV). I've edited the post to say Debrett's Peerage, thanks for the tip. ChrisA From ethan at stoneleaf.us Sun Jul 28 19:49:07 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 28 Jul 2013 16:49:07 -0700 Subject: [Savoynet] G&S Opera Co: Pirates of Penzance In-Reply-To: References: Message-ID: <51F5ADF3.2060409@stoneleaf.us> On 07/28/2013 10:57 AM, Chris Angelico wrote: . . . Okay, how did you get confused that this was a Python List question? ;) -- ~Ethan~ From tim at thechases.com Sun Jul 28 20:45:49 2013 From: tim at thechases.com (Tim Chase) Date: Sun, 28 Jul 2013 19:45:49 -0500 Subject: [Savoynet] G&S Opera Co: Pirates of Penzance In-Reply-To: <51F5ADF3.2060409@stoneleaf.us> References: <51F5ADF3.2060409@stoneleaf.us> Message-ID: <20130728194549.04e3128e@bigbox.christie.dr> On 2013-07-28 16:49, Ethan Furman wrote: > On 07/28/2013 10:57 AM, Chris Angelico wrote: > . > . > . > > Okay, how did you get confused that this was a Python List > question? ;) Must have been this ancient thread http://mail.python.org/pipermail/python-list/2006-September/376063.html :-) -tkc From esj at harvee.org Mon Jul 29 01:24:52 2013 From: esj at harvee.org (Eric S. Johansson) Date: Mon, 29 Jul 2013 01:24:52 -0400 Subject: [Savoynet] G&S Opera Co: Pirates of Penzance In-Reply-To: <51F5ADF3.2060409@stoneleaf.us> References: <51F5ADF3.2060409@stoneleaf.us> Message-ID: On Sun, 28 Jul 2013 19:49:07 -0400, Ethan Furman wrote: > On 07/28/2013 10:57 AM, Chris Angelico wrote: > . > . > . > > Okay, how did you get confused that this was a Python List question? ;) got_a_little_list["victim must be found"] = http://www.youtube.com/watch?v=1NLV24qTnlg From rosuav at gmail.com Mon Jul 29 02:49:50 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 29 Jul 2013 07:49:50 +0100 Subject: [Savoynet] G&S Opera Co: Pirates of Penzance In-Reply-To: <51F5ADF3.2060409@stoneleaf.us> References: <51F5ADF3.2060409@stoneleaf.us> Message-ID: On Mon, Jul 29, 2013 at 12:49 AM, Ethan Furman wrote: > On 07/28/2013 10:57 AM, Chris Angelico wrote: > . > . > . > > Okay, how did you get confused that this was a Python List question? ;) *sigh* Because I still haven't gotten around to switching mail clients to one that has a Reply-List feature. The post I was replying to was on Savoynet, which is about Gilbert & Sullivan. ChrisA From jcasale at activenetwerx.com Sun Jul 28 14:32:28 2013 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Sun, 28 Jul 2013 18:32:28 +0000 Subject: sqlite3 version lacks instr Message-ID: I have some queries that utilize instr wrapped by substr but the old version shipped in 2.7.5 doesn't have instr support. Has anyone encountered this and utilized other existing functions within the shipped 3.6.21 sqlite version to accomplish this? Thanks, jlc From jcasale at activenetwerx.com Sun Jul 28 14:53:02 2013 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Sun, 28 Jul 2013 18:53:02 +0000 Subject: sqlite3 version lacks instr In-Reply-To: <13d04b58a1024039b66ea54a9d5bb85b@exch.activenetwerx.com> References: <13d04b58a1024039b66ea54a9d5bb85b@exch.activenetwerx.com> Message-ID: > Has anyone encountered this and utilized other existing functions > within the shipped 3.6.21 sqlite version to accomplish this? Sorry guys, forgot about create_function... From __peter__ at web.de Sun Jul 28 15:14:25 2013 From: __peter__ at web.de (Peter Otten) Date: Sun, 28 Jul 2013 21:14:25 +0200 Subject: sqlite3 version lacks instr References: <13d04b58a1024039b66ea54a9d5bb85b@exch.activenetwerx.com> Message-ID: Joseph L. Casale wrote: >> Has anyone encountered this and utilized other existing functions >> within the shipped 3.6.21 sqlite version to accomplish this? > > Sorry guys, forgot about create_function... Too late, I already did the demo ;) >>> import sqlite3 >>> db = sqlite3.connect(":memory:") >>> cs = db.cursor() >>> cs.execute('select instr("the quick brown fox", "brown")').fetchone()[0] Traceback (most recent call last): File "", line 1, in sqlite3.OperationalError: no such function: instr >>> def instr(a, b): ... return a.find(b) + 1 # add NULL-handling etc. ... >>> db.create_function("instr", 2, instr) >>> cs.execute('select instr("the quick brown fox", "brown")').fetchone()[0] 11 >>> cs.execute('select instr("the quick brown fox", "red")').fetchone()[0] 0 From me at davecotter.com Sun Jul 28 15:21:25 2013 From: me at davecotter.com (David M. Cotter) Date: Sun, 28 Jul 2013 12:21:25 -0700 (PDT) Subject: embedding: how to create an "idle" handler to allow user to kill scripts? Message-ID: <55e03ef8-d1db-4ae0-88a5-7641fb4b1b57@googlegroups.com> in my C++ app, on the main thread i init python, init threads, then call PyEval_SaveThread(), since i'm not going to do any more python on the main thread. then when the user invokes a script, i launch a preemptive thread (boost::threads), and from there, i have this: static int CB_S_Idle(void *in_thiz) { CT_RunScript *thiz((CT_RunScript *)in_thiz); return thiz->Idle(); } int Idle() { int resultI = 0; OSStatus err = noErr; ERR(i_taskRecP->MT_UpdateData(&i_progData)); if (err) { resultI = -1; } ERR(ScheduleIdleCall()); return err; } int ScheduleIdleCall() { int resultI(Py_AddPendingCall(CB_S_Idle, this)); CFAbsoluteTime timeT(CFAbsoluteTimeGetCurrent()); SuperString str; str.Set(timeT, SS_Time_LOG); Logf("$$$ Python idle: (%d) %s\n", resultI, str.utf8Z()); return resultI; } virtual OSStatus operator()(OSStatus err) { ScPyGILState sc; ERR(ScheduleIdleCall()); ERR(PyRun_SimpleString(i_script.utf8Z())); return err; } so, my operator() gets called, and i try to schedule an Idle call, which succeeds, then i run my script. however, the CB_S_Idle() never gets called? the MT_UpdateData() function returns an error if the user had canceled the script must i schedule a run-loop on the main thread or something to get it to be called? From me at davecotter.com Tue Jul 30 15:56:32 2013 From: me at davecotter.com (David M. Cotter) Date: Tue, 30 Jul 2013 12:56:32 -0700 (PDT) Subject: embedding: how to create an "idle" handler to allow user to kill scripts? In-Reply-To: <55e03ef8-d1db-4ae0-88a5-7641fb4b1b57@googlegroups.com> References: <55e03ef8-d1db-4ae0-88a5-7641fb4b1b57@googlegroups.com> Message-ID: Okay, i'm really surprised nobody knows how to do this. and frankly i'm amazed at the utter lack of documentation. but i've figured it out, and it's all working beautifully. if you want the code, go here: http://karaoke.kjams.com/wiki/Python From roy at panix.com Sun Jul 28 15:59:04 2013 From: roy at panix.com (Roy Smith) Date: Sun, 28 Jul 2013 15:59:04 -0400 Subject: collections.Counter surprisingly slow Message-ID: I've been doing an informal "intro to Python" lunchtime series for some co-workers (who are all experienced programmers, in other languages). This week I was going to cover list comprehensions, exceptions, and profiling. So, I did a little demo showing different ways to build a dictionary counting how many times a string appears in some input: test() implements a "look before you leap" python loop exception() uses a try/catch construct in a similar python loop default() uses a defaultdict count() uses a Counter I profiled it, to show how the profiler works. The full code is below. The input is an 8.8 Mbyte file containing about 570,000 lines (11,000 unique strings). Python 2.7.3 on Ubuntu Precise. As I expected, test() is slower than exception(), which is slower than default(). I'm rather shocked to discover that count() is the slowest of all! I expected it to be the fastest. Or, certainly, no slower than default(). The full profiler dump is at the end of this message, but the gist of it is: ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 0.322 0.322 ./stations.py:42(count) 1 0.159 0.159 0.159 0.159 ./stations.py:17(test) 1 0.114 0.114 0.114 0.114 ./stations.py:27(exception) 1 0.097 0.097 0.097 0.097 ./stations.py:36(default) Why is count() [i.e. collections.Counter] so slow? ------------------------------------------------------------- from collections import defaultdict, Counter def main(): lines = open('stations.txt').readlines() d1 = test(lines) d2 = exception(lines) d3 = default(lines) d4 = count(lines) print d1 == d2 print d1 == d3 print d1 == d4 def test(lines): d = {} for station in lines: if station in d: d[station] += 1 else: d[station] = 1 return d def exception(lines): d = {} for station in lines: try: d[station] += 1 except KeyError: d[station] = 1 return d def default(lines): d = defaultdict(int) for station in lines: d[station] += 1 return d def count(lines): d = Counter(lines) return d if __name__ == '__main__': import cProfile import pstats cProfile.run('main()', 'stations.stats') p = pstats.Stats('stations.stats') p.sort_stats('cumulative').print_stats() ------------------------------------------------------------- 570335 function calls (570332 primitive calls) in 0.776 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.023 0.023 0.776 0.776 :1() 1 0.006 0.006 0.753 0.753 ./stations.py:5(main) 1 0.000 0.000 0.322 0.322 ./stations.py:42(count) 1 0.000 0.000 0.322 0.322 /usr/lib/python2.7/collections.py:407(__init__) 1 0.242 0.242 0.322 0.322 /usr/lib/python2.7/collections.py:470(update) 1 0.159 0.159 0.159 0.159 ./stations.py:17(test) 1 0.114 0.114 0.114 0.114 ./stations.py:27(exception) 1 0.097 0.097 0.097 0.097 ./stations.py:36(default) 570285 0.080 0.000 0.080 0.000 {method 'get' of 'dict' objects} 1 0.055 0.055 0.055 0.055 {method 'readlines' of 'file' objects} 1 0.000 0.000 0.000 0.000 {isinstance} 1 0.000 0.000 0.000 0.000 /home/roy/deploy/current/python/lib/python2.7/abc.py:128(__instancecheck__) 2/1 0.000 0.000 0.000 0.000 /home/roy/deploy/current/python/lib/python2.7/abc.py:148(__subclasscheck__) 3/1 0.000 0.000 0.000 0.000 {issubclass} 4 0.000 0.000 0.000 0.000 /home/roy/deploy/current/python/lib/python2.7/_weakrefset.py:58(__iter__) 1 0.000 0.000 0.000 0.000 {open} 2 0.000 0.000 0.000 0.000 /home/roy/deploy/current/python/lib/python2.7/_weakrefset.py:26(__exit__) 2 0.000 0.000 0.000 0.000 /home/roy/deploy/current/python/lib/python2.7/_weakrefset.py:20(__enter__) 2 0.000 0.000 0.000 0.000 /home/roy/deploy/current/python/lib/python2.7/_weakrefset.py:36(__init__) 3 0.000 0.000 0.000 0.000 /home/roy/deploy/current/python/lib/python2.7/_weakrefset.py:68(__contains__) 2 0.000 0.000 0.000 0.000 /home/roy/deploy/current/python/lib/python2.7/_weakrefset.py:52(_commit_removals) 2 0.000 0.000 0.000 0.000 /home/roy/deploy/current/python/lib/python2.7/_weakrefset.py:81(add) 3 0.000 0.000 0.000 0.000 {getattr} 2 0.000 0.000 0.000 0.000 /home/roy/deploy/current/python/lib/python2.7/_weakrefset.py:16(__init__) 2 0.000 0.000 0.000 0.000 {method 'remove' of 'set' objects} 2 0.000 0.000 0.000 0.000 {method '__subclasses__' of 'type' objects} 2 0.000 0.000 0.000 0.000 /home/roy/deploy/current/python/lib/python2.7/_abcoll.py:97(__subclasshook__) 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} 4 0.000 0.000 0.000 0.000 {method 'add' of 'set' objects} From steve+comp.lang.python at pearwood.info Sun Jul 28 16:51:11 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 28 Jul 2013 20:51:11 GMT Subject: collections.Counter surprisingly slow References: Message-ID: <51f5843f$0$29971$c3e8da3$5496439d@news.astraweb.com> On Sun, 28 Jul 2013 15:59:04 -0400, Roy Smith wrote: [...] > I'm rather shocked to discover that count() is the slowest > of all! I expected it to be the fastest. Or, certainly, no slower than > default(). > > The full profiler dump is at the end of this message, but the gist of it > is: > > ncalls tottime percall cumtime percall filename:lineno(function) > 1 0.000 0.000 0.322 0.322 ./stations.py:42(count) > 1 0.159 0.159 0.159 0.159 ./stations.py:17(test) > 1 0.114 0.114 0.114 0.114 ./stations.py:27(exception) > 1 0.097 0.097 0.097 0.097 ./stations.py:36(default) > > Why is count() [i.e. collections.Counter] so slow? It's within a factor of 2 of test, and 3 of exception or default (give or take). I don't think that's surprisingly slow. In 2.7, Counter is written in Python, while defaultdict has an accelerated C version. I expect that has something to do with it. Calling Counter ends up calling essentially this code: for elem in iterable: self[elem] = self.get(elem, 0) + 1 (although micro-optimized), where "iterable" is your data (lines). Calling the get method has higher overhead than dict[key], that will also contribute. -- Steven From roy at panix.com Sun Jul 28 17:57:12 2013 From: roy at panix.com (Roy Smith) Date: Sun, 28 Jul 2013 17:57:12 -0400 Subject: collections.Counter surprisingly slow References: <51f5843f$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: In article <51f5843f$0$29971$c3e8da3$5496439d at news.astraweb.com>, Steven D'Aprano wrote: > > Why is count() [i.e. collections.Counter] so slow? > > It's within a factor of 2 of test, and 3 of exception or default (give or > take). I don't think that's surprisingly slow. It is for a module which describes itself as "High-performance container datatypes" :-) From stefan_ml at behnel.de Mon Jul 29 07:46:58 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 29 Jul 2013 13:46:58 +0200 Subject: collections.Counter surprisingly slow In-Reply-To: <51f5843f$0$29971$c3e8da3$5496439d@news.astraweb.com> References: <51f5843f$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: Steven D'Aprano, 28.07.2013 22:51: > Calling Counter ends up calling essentially this code: > > for elem in iterable: > self[elem] = self.get(elem, 0) + 1 > > (although micro-optimized), where "iterable" is your data (lines). > Calling the get method has higher overhead than dict[key], that will also > contribute. It comes with a C accelerator (at least in Py3.4dev), but it seems like that stumbles a bit over its own feet. The accelerator function special cases the (exact) dict type, but the Counter class is a subtype of dict and thus takes the generic path, which makes it benefit a bit less than possible. Look for _count_elements() in http://hg.python.org/cpython/file/tip/Modules/_collectionsmodule.c Nevertheless, even the generic C code path looks fast enough in general. I think the problem is just that the OP used Python 2.7, which doesn't have this accelerator function. Stefan From joshua at landau.ws Mon Jul 29 08:07:48 2013 From: joshua at landau.ws (Joshua Landau) Date: Mon, 29 Jul 2013 13:07:48 +0100 Subject: collections.Counter surprisingly slow In-Reply-To: References: <51f5843f$0$29971$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 29 July 2013 12:46, Stefan Behnel wrote: > Steven D'Aprano, 28.07.2013 22:51: > > Calling Counter ends up calling essentially this code: > > > > for elem in iterable: > > self[elem] = self.get(elem, 0) + 1 > > > > (although micro-optimized), where "iterable" is your data (lines). > > Calling the get method has higher overhead than dict[key], that will also > > contribute. > > It comes with a C accelerator (at least in Py3.4dev), but it seems like > that stumbles a bit over its own feet. The accelerator function special > cases the (exact) dict type, but the Counter class is a subtype of dict and > thus takes the generic path, which makes it benefit a bit less than > possible. > > Look for _count_elements() in > > http://hg.python.org/cpython/file/tip/Modules/_collectionsmodule.c > > Nevertheless, even the generic C code path looks fast enough in general. I > think the problem is just that the OP used Python 2.7, which doesn't have > this accelerator function. > # _count_elements({}, items), _count_elements(dict_subclass(), items), Counter(items), defaultdict(int) loop with exception handling # "items" is always 1m long with varying levels of repetition >>> for items in randoms: ... helper.timeit(1), helper_subclass.timeit(1), counter.timeit(1), default.timeit(1) ... (0.18816172199876746, 0.4679023139997298, 0.9684444869999425, 0.33518486200046027) (0.2936601179990248, 0.6056111739999324, 1.1316078849995392, 0.46283868699902087) (0.35396358400066674, 0.685048443998312, 1.2120939880005608, 0.5497965239992482) (0.5337620789996436, 0.8658702100001392, 1.4507492869997805, 0.7772859329998028) (0.745282343999861, 1.1455801379997865, 2.116569702000561, 1.3293145009993168) :( I have the helper but Counter is still slow. Is it not getting used for some reason? It's not even as fast as helper on a dict's (direct, no overridden methods) subclass. -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Mon Jul 29 02:25:49 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 29 Jul 2013 09:25:49 +0300 Subject: collections.Counter surprisingly slow In-Reply-To: References: Message-ID: 28.07.13 22:59, Roy Smith ???????(??): > The input is an 8.8 Mbyte file containing about 570,000 lines (11,000 > unique strings). Repeat you tests with totally unique lines. > The full profiler dump is at the end of this message, but the gist of > it is: Profiler affects execution time. In particular it slowdown Counter implementation which uses more function calls. For real world measurement use different approach. > Why is count() [i.e. collections.Counter] so slow? Feel free to contribute a patch which fixes this "wart". Note that Counter shouldn't be slowdowned on mostly unique data. From joshua at landau.ws Mon Jul 29 07:49:53 2013 From: joshua at landau.ws (Joshua Landau) Date: Mon, 29 Jul 2013 12:49:53 +0100 Subject: collections.Counter surprisingly slow In-Reply-To: References: Message-ID: On 29 July 2013 07:25, Serhiy Storchaka wrote: > 28.07.13 22:59, Roy Smith ???????(??): > > The input is an 8.8 Mbyte file containing about 570,000 lines (11,000 >> unique strings). >> > > Repeat you tests with totally unique lines. Counter is about ? the speed of defaultdict in that case (as opposed to ?). > The full profiler dump is at the end of this message, but the gist of >> it is: >> > > Profiler affects execution time. In particular it slowdown Counter > implementation which uses more function calls. For real world measurement > use different approach. Doing some re-times, it seems that his originals for defaultdict, exception and Counter were about right. I haven't timed the other. > Why is count() [i.e. collections.Counter] so slow? >> > > Feel free to contribute a patch which fixes this "wart". Note that Counter > shouldn't be slowdowned on mostly unique data. I find it hard to agree that counter should be optimised for the unique-data case, as surely it's much more oft used when there's a point to counting? Also, couldn't Counter just extend from defaultdict? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ian.g.kelly at gmail.com Mon Jul 29 13:19:22 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 29 Jul 2013 11:19:22 -0600 Subject: collections.Counter surprisingly slow In-Reply-To: References: Message-ID: On Mon, Jul 29, 2013 at 5:49 AM, Joshua Landau wrote: > Also, couldn't Counter just extend from defaultdict? It could, but I expect the C helper function in 3.4 will be faster since it doesn't even need to call __missing__ in the first place. And the cost (both in terms of maintenance and run-time) of adding defaultdict to the Counter MRO just to reuse one tiny __missing__ method seems questionable, especially after taking into account that the constructor signature is incompatible. From storchaka at gmail.com Mon Jul 29 15:37:51 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 29 Jul 2013 22:37:51 +0300 Subject: collections.Counter surprisingly slow In-Reply-To: References: Message-ID: 29.07.13 20:19, Ian Kelly ???????(??): > On Mon, Jul 29, 2013 at 5:49 AM, Joshua Landau wrote: >> Also, couldn't Counter just extend from defaultdict? > > It could, but I expect the C helper function in 3.4 will be faster > since it doesn't even need to call __missing__ in the first place. I'm surprised, but the Counter constructor with commented out import of this accelerator is faster (at least for some data). From stefan_ml at behnel.de Tue Jul 30 02:39:00 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 30 Jul 2013 08:39:00 +0200 Subject: collections.Counter surprisingly slow In-Reply-To: References: Message-ID: Serhiy Storchaka, 29.07.2013 21:37: > 29.07.13 20:19, Ian Kelly ???????(??): >> On Mon, Jul 29, 2013 at 5:49 AM, Joshua Landau wrote: >>> Also, couldn't Counter just extend from defaultdict? >> >> It could, but I expect the C helper function in 3.4 will be faster >> since it doesn't even need to call __missing__ in the first place. > > I'm surprised, but the Counter constructor with commented out import of > this accelerator is faster (at least for some data). Read my post. The accelerator doesn't take the fast path for dicts as Counter is only a subtype of dict, not exactly a dict. That means that it raises and catches a KeyError exception for each new value that it finds, and that is apparently more costly than the overhead of calling get(). So, my expectation is that it's faster for highly repetitive data and slower for mostly unique data. Maybe a "fast_dict_lookup" option for the accelerator that forces the fast path would fix this. The Counter class, just like many (most?) other subtypes of dict, definitely doesn't need the fallback behaviour. Stefan From stefan_ml at behnel.de Tue Jul 30 02:51:48 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 30 Jul 2013 08:51:48 +0200 Subject: collections.Counter surprisingly slow In-Reply-To: References: Message-ID: Stefan Behnel, 30.07.2013 08:39: > Serhiy Storchaka, 29.07.2013 21:37: >> 29.07.13 20:19, Ian Kelly ???????(??): >>> On Mon, Jul 29, 2013 at 5:49 AM, Joshua Landau wrote: >>>> Also, couldn't Counter just extend from defaultdict? >>> >>> It could, but I expect the C helper function in 3.4 will be faster >>> since it doesn't even need to call __missing__ in the first place. >> >> I'm surprised, but the Counter constructor with commented out import of >> this accelerator is faster (at least for some data). > > Read my post. The accelerator doesn't take the fast path for dicts as > Counter is only a subtype of dict, not exactly a dict. That means that it > raises and catches a KeyError exception for each new value that it finds, > and that is apparently more costly than the overhead of calling get(). > > So, my expectation is that it's faster for highly repetitive data and > slower for mostly unique data. > > Maybe a "fast_dict_lookup" option for the accelerator that forces the fast > path would fix this. The Counter class, just like many (most?) other > subtypes of dict, definitely doesn't need the fallback behaviour. Or rather drop the fallback path completely. It's not worth having code duplication if it's not predictable up-front (before looking at the data) if it will help or not. http://bugs.python.org/issue18594 Stefan From storchaka at gmail.com Tue Jul 30 09:04:12 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 30 Jul 2013 16:04:12 +0300 Subject: collections.Counter surprisingly slow In-Reply-To: References: Message-ID: 29.07.13 14:49, Joshua Landau ???????(??): > I find it hard to agree that counter should be optimised for the > unique-data case, as surely it's much more oft used when there's a point > to counting? Different methods are faster for different data. LBYL approach is best for the mostly unique data case, while EAFP approach is best for the mostly repeated data case. In general case a performance of particular method is a function of its performances in this two extreme cases. When it slow for one of extreme case it can be slow in a number of intermediate cases. > Also, couldn't Counter just extend from defaultdict? Unfortunately this only will slowdown it. From rurpy at yahoo.com Sun Jul 28 19:41:27 2013 From: rurpy at yahoo.com (rurpy at yahoo.com) Date: Sun, 28 Jul 2013 16:41:27 -0700 (PDT) Subject: email 8bit encoding Message-ID: <36aa5535-a821-4d7e-8414-1e40820705e4@googlegroups.com> How, using Python-3.3's email module, do I "flatten" (I think that's the right term) a Message object to get utf-8 encoded body with the headers: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit when the message payload was set to a python (unicode) string? From wking at tremily.us Mon Jul 29 02:16:12 2013 From: wking at tremily.us (W. Trevor King) Date: Sun, 28 Jul 2013 23:16:12 -0700 Subject: [python] email 8bit encoding In-Reply-To: <36aa5535-a821-4d7e-8414-1e40820705e4@googlegroups.com> References: <36aa5535-a821-4d7e-8414-1e40820705e4@googlegroups.com> Message-ID: <20130729061612.GA1400@odin.tremily.us> On Sun, Jul 28, 2013 at 04:41:27PM -0700, rurpy at yahoo.com wrote: > How, using Python-3.3's email module, do I "flatten" (I think > that's the right term) a Message object to get utf-8 encoded > body with the headers: > Content-Type: text/plain; charset=UTF-8 > Content-Transfer-Encoding: 8bit > when the message payload was set to a python (unicode) string? I asked about this a while back [1], but never got a response. My current best-guess is here [2]. My fallback flattening works for everything except the 8-bit encoded messages using the UTF-16 charset, but it's pretty ugly. Let me know if you figure out something cleaner :). Cheers, Trevor [1]: http://thread.gmane.org/gmane.comp.python.general/725425 [2]: https://github.com/wking/rss2email/blob/master/rss2email/email.py#L226 -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From antoon.pardon at rece.vub.ac.be Mon Jul 29 16:52:04 2013 From: antoon.pardon at rece.vub.ac.be (Antoon Pardon) Date: Mon, 29 Jul 2013 22:52:04 +0200 Subject: email 8bit encoding In-Reply-To: <36aa5535-a821-4d7e-8414-1e40820705e4@googlegroups.com> References: <36aa5535-a821-4d7e-8414-1e40820705e4@googlegroups.com> Message-ID: <51F6D5F4.7030809@rece.vub.ac.be> Op 29-07-13 01:41, rurpy at yahoo.com schreef: > How, using Python-3.3's email module, do I "flatten" (I think > that's the right term) a Message object to get utf-8 encoded > body with the headers: > Content-Type: text/plain; charset=UTF-8 > Content-Transfer-Encoding: 8bit > when the message payload was set to a python (unicode) string? > I am just trying out some things for my self on python 3.2 so be sure to test this out but you could try the following. msg.set_charset('utf-8') msg['Content-Transfer-Encoding'] = '8bit' -- Antoon Pardon From rurpy at yahoo.com Mon Jul 29 23:56:33 2013 From: rurpy at yahoo.com (rurpy at yahoo.com) Date: Mon, 29 Jul 2013 20:56:33 -0700 (PDT) Subject: [python] email 8bit encoding In-Reply-To: References: <36aa5535-a821-4d7e-8414-1e40820705e4@googlegroups.com> Message-ID: On 07/29/2013 12:16 AM, W. Trevor King wrote: > On Sun, Jul 28, 2013 at 04:41:27PM -0700, rurpy at yahoo.com wrote: >> How, using Python-3.3's email module, do I "flatten" (I think >> that's the right term) a Message object to get utf-8 encoded >> body with the headers: >> Content-Type: text/plain; charset=UTF-8 >> Content-Transfer-Encoding: 8bit >> when the message payload was set to a python (unicode) string? > > I asked about this a while back [1], but never got a response. My > current best-guess is here [2]. My fallback flattening works for > everything except the 8-bit encoded messages using the UTF-16 charset, > but it's pretty ugly. > > Let me know if you figure out something cleaner :). > > Cheers, > Trevor > > [1]: http://thread.gmane.org/gmane.comp.python.general/725425 > [2]: https://github.com/wking/rss2email/blob/master/rss2email/email.py#L226 Thanks, that was very helpful. My code is just a quick utility tool for internal use so I am able to be fairly hackish without shame. :-) Since I am synthesizing the mails from scratch and the encoding requirements fairly simple, I could probably dispense with the email/smtplib packages altogether and just pipe my text directly to sendmail. But I thought using Python's email package would be easier. Silly me. Thanks again. From tocallaghan at gmail.com Sun Jul 28 21:38:10 2013 From: tocallaghan at gmail.com (Tim O'Callaghan) Date: Sun, 28 Jul 2013 18:38:10 -0700 (PDT) Subject: dynamic type returning NameError: Message-ID: Hi, I hope that this hasn't been asked for the millionth time, so my apologies if it has. I have a base class (BaseClass - we'll call it for this example) with an http call that i would like to inherit into a dynamic class at runtime. We'll call that method in BaseClass; 'request'. I have a dictionary(json) of key (class name): value(method) that I would like to create inheriting this 'request' method from the BaseClass. So the derived class would look something like this definition in json: {"Whatever": [{"method1": "Some Default", "async": True},{"method2": "Some Other Default", "async": True}]} Ideally I'd like the class def to look something like this if i were to type it out by hand [excuse the indents] class Whatever(BaseClass): def method1(self): stupid_data = super(Whatever, self).request("method1") return stupid_data def method2(self): stupid_data = super(Whatever, self).request("method1") return stupid_data Now, I've been trying to do this using the python cli, with out success. So, attempting this at runtime I get a plethora of wonderful errors that I suspect has broken my brain. Here is what i've tried: # trying with just an empty object of type BaseClass obj = type("Object", (BaseClass,), {}) whatever = type("WhatEver", (obj,), {"method1": super(WhatEver, self).request("method1")}) but when i try this I get 'NameError: name 'self' is not defined' defining these classes manually works... I hope that this was clear enough, apologies if it wasn't. It's late(ish), I'm tired and borderline frustrated :) But enough about me... Thanks in advance. From tjreedy at udel.edu Sun Jul 28 22:51:57 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 28 Jul 2013 22:51:57 -0400 Subject: dynamic type returning NameError: In-Reply-To: References: Message-ID: On 7/28/2013 9:38 PM, Tim O'Callaghan wrote: > Hi, > > I hope that this hasn't been asked for the millionth time, so my apologies if it has. > > I have a base class (BaseClass - we'll call it for this example) with an http call that i would like to inherit into a dynamic class at runtime. We'll call that method in BaseClass; 'request'. > > I have a dictionary(json) of key (class name): value(method) that I would like to create inheriting this 'request' method from the BaseClass. So the derived class would look something like this > > definition in json: > {"Whatever": [{"method1": "Some Default", "async": True},{"method2": "Some Other Default", "async": True}]} > > Ideally I'd like the class def to look something like this if i were to type it out by hand > > [excuse the indents] > > class Whatever(BaseClass): > def method1(self): > stupid_data = super(Whatever, self).request("method1") > return stupid_data > > def method2(self): > stupid_data = super(Whatever, self).request("method1") > return stupid_data > > Now, I've been trying to do this using the python cli, with out success. > > So, attempting this at runtime I get a plethora of wonderful errors that I suspect has broken my brain. > > Here is what i've tried: > > # trying with just an empty object of type BaseClass > obj = type("Object", (BaseClass,), {}) > > whatever = type("WhatEver", (obj,), {"method1": super(WhatEver, self).request("method1")}) 'method1' has to be mapped to a function object. > but when i try this I get 'NameError: name 'self' is not defined' > > defining these classes manually works... > > I hope that this was clear enough, apologies if it wasn't. It's late(ish), I'm tired and borderline frustrated :) But enough about me... > > Thanks in advance. > -- Terry Jan Reedy From tocallaghan at gmail.com Mon Jul 29 00:14:19 2013 From: tocallaghan at gmail.com (Tim O'Callaghan) Date: Sun, 28 Jul 2013 21:14:19 -0700 (PDT) Subject: dynamic type returning NameError: In-Reply-To: References: Message-ID: <10e1491c-49d1-47ef-8dcd-4e45b87c182a@googlegroups.com> On Sunday, July 28, 2013 10:51:57 PM UTC-4, Terry Reedy wrote: > On 7/28/2013 9:38 PM, Tim O'Callaghan wrote: > > > Hi, > > > > > > I hope that this hasn't been asked for the millionth time, so my apologies if it has. > > > > > > I have a base class (BaseClass - we'll call it for this example) with an http call that i would like to inherit into a dynamic class at runtime. We'll call that method in BaseClass; 'request'. > > > > > > I have a dictionary(json) of key (class name): value(method) that I would like to create inheriting this 'request' method from the BaseClass. So the derived class would look something like this > > > > > > definition in json: > > > {"Whatever": [{"method1": "Some Default", "async": True},{"method2": "Some Other Default", "async": True}]} > > > > > > Ideally I'd like the class def to look something like this if i were to type it out by hand > > > > > > [excuse the indents] > > > > > > class Whatever(BaseClass): > > > def method1(self): > > > stupid_data = super(Whatever, self).request("method1") > > > return stupid_data > > > > > > def method2(self): > > > stupid_data = super(Whatever, self).request("method1") > > > return stupid_data > > > > > > Now, I've been trying to do this using the python cli, with out success. > > > > > > So, attempting this at runtime I get a plethora of wonderful errors that I suspect has broken my brain. > > > > > > Here is what i've tried: > > > > > > # trying with just an empty object of type BaseClass > > > obj = type("Object", (BaseClass,), {}) > > > > > > whatever = type("WhatEver", (obj,), {"method1": super(WhatEver, self).request("method1")}) > > > > 'method1' has to be mapped to a function object. But isn't that what calling super is doing? Calling the function object of the parent class BaseClass? > > > but when i try this I get 'NameError: name 'self' is not defined' > > > > > > defining these classes manually works... > > > > > > I hope that this was clear enough, apologies if it wasn't. It's late(ish), I'm tired and borderline frustrated :) But enough about me... > > > > > > Thanks in advance. > > > > > > > > > -- > > Terry Jan Reedy From steve+comp.lang.python at pearwood.info Mon Jul 29 01:43:39 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 29 Jul 2013 05:43:39 GMT Subject: dynamic type returning NameError: References: Message-ID: <51f6010a$0$30000$c3e8da3$5496439d@news.astraweb.com> On Sun, 28 Jul 2013 18:38:10 -0700, Tim O'Callaghan wrote: > Hi, > > I hope that this hasn't been asked for the millionth time, so my > apologies if it has. [...] > I hope that this was clear enough, apologies if it wasn't. Clear as mud. > It's late(ish), I'm tired and borderline frustrated :) I see your smiley, but perhaps you would get better results by waiting until you can make a better post. It *really* helps if you post actual "working" (even if "working" means "fails in the way I said"), *short*, *simple* code. Often you'll find that trying to simplify the problem gives you the insight to solve the problem yourself. http://www.sscce.org/ I'm going to try to guess what you're attempting, but I may get it completely wrong. Sorry if I do, but hopefully you'll get some insight even from my misunderstandings. > I have a base class (BaseClass - we'll call it for this example) with an > http call that i would like to inherit into a dynamic class at runtime. > We'll call that method in BaseClass; 'request'. If I read this literally, you want to do this: class BaseClass(DynamicParent): def request(self): ... except that DynamicParent isn't known until runtime. Am I close? Obviously the above syntax won't work, but you can use a factory: def make_baseclass(parent): class BaseClass(parent): def request(self): ... return BaseClass class Spam: ... BaseClass = make_baseclass(Spam) Or you can use the type() constructor directly: BaseClass = type('BaseClass', (Spam,), dict_of_methods_and_stuff) which is probably far less convenient. But all this assumes I read you literally, and reading on, I don't think that's what you are after. > I have a dictionary(json) of key (class name): value(method) that I > would like to create inheriting this 'request' method from the > BaseClass. So the derived class would look something like this > > definition in json: > {"Whatever": [{"method1": "Some Default", "async": True},{"method2": > "Some Other Default", "async": True}]} Pure gobbledygook to me. I don't understand what you're talking about, how does a derived class turn into JSON? (Could be worse, it could be XML.) Is BaseClass the "derived class", or are you talking about inheriting from BaseClass? What's "Some Default"? It looks like a string, and it certainly isn't a valid method name, not with a space in it. Where did async and method2 come from? How do these things relate to "request" you talk about above? I think you're too close to the problem and don't realise that others don't sharing your knowledge of the problem. But, moving along, if I've understood you correctly, I don't think inheritance is the right solution here. I think that composition or delegation may be better. Something like this: class BaseClass: def request(self): # Delegate to a method set dynamically, on the instance. return self.some_method() a = BaseClass() a.some_method = one_thing.method1 b = BaseClass() b.some_method = another_thing.method2 Now you have instance a.request calling method1 of another object, and b.request calling method2 of a different object. Does that solve your problem, or am I on a wild-goose chase? > Ideally I'd like the class def to look something like this if i were to > type it out by hand > > [excuse the indents] > > class Whatever(BaseClass): > def method1(self): > stupid_data = super(Whatever, self).request("method1") > return stupid_data > > def method2(self): > stupid_data = super(Whatever, self).request("method1") > return stupid_data Since request is not the method you are currently in, the above is equivalent to: class Whatever(BaseClass): def method1(self): return self.request("method1") def method2(self): return self.request("method2") where "request" is defined by BaseClass, and assuming you don't override it in the subclass. (I assume "method1" in your code above was a typo.) > Now, I've been trying to do this using the python cli, with out success. > > So, attempting this at runtime I get a plethora of wonderful errors that > I suspect has broken my brain. > > Here is what i've tried: > > # trying with just an empty object of type BaseClass > obj = type("Object", (BaseClass,), {}) "obj" here is a class called "Object", inheriting from BaseClass. It overrides no methods. Why does it exist? It doesn't do anything. > whatever = type("WhatEver", (obj,), {"method1": super(WhatEver, > self).request("method1")}) > > but when i try this I get 'NameError: name 'self' is not defined' This is what you are doing: * look up names WhatEver and self, in the current scope (i.e. the scope where you are running this call to type, which is likely the global scope); * pass those objects (if they exist!) to super(), right now; * on the object returned, look up the attribute "request", right now; * call that object with string argument "method1", right now; * take the result of that sequences of calls, let's call it x, and set it in a new dict with key "method1"; * and then create a new type using dict {'method1': x}. Notice that everything you do is done immediately. You should be getting a NameError for WhatEver too, but since you are not, I can only imagine that you have already (accidentally?) defined something, anything, with that name. What you actually want (but probably not what you *need*, see above about delegation) is to delay evaluation of super() and subsequent calls until after your WhatEver class is created and initialised and the method1 *method* is called. That means putting it all those calls inside a function object, for later use, instead of executing them directly *right now*: def method1(self): return super(WhatEver, self).request("method1") WhatEver = type("WhatEver", (BaseClass,), {"method1": method1}) You could alternatively use lambda: WhatEver = type("WhatEver", (BaseClass,), {"method1": lambda self: super(WhatEver, self).request("method1") } ) Note that the class name inside super() *must* match the global name the class is assigned to, "WhatEver". The internal class __name__ -- the first argument to type() -- doesn't have to match, but it should, to avoid confusion. The above should fix the NameError you are getting, and it might even work, but I think delegation is a better solution to this problem. -- Steven From tocallaghan at gmail.com Mon Jul 29 10:34:22 2013 From: tocallaghan at gmail.com (Tim O'Callaghan) Date: Mon, 29 Jul 2013 07:34:22 -0700 (PDT) Subject: dynamic type returning NameError: In-Reply-To: <51f6010a$0$30000$c3e8da3$5496439d@news.astraweb.com> References: <51f6010a$0$30000$c3e8da3$5496439d@news.astraweb.com> Message-ID: <8d5a9112-c511-484d-b934-de7c6970f0ea@googlegroups.com> On Monday, July 29, 2013 1:43:39 AM UTC-4, Steven D'Aprano wrote: > On Sun, 28 Jul 2013 18:38:10 -0700, Tim O'Callaghan wrote: > > > > > Hi, > > > > > > I hope that this hasn't been asked for the millionth time, so my > > > apologies if it has. > > [...] > > > I hope that this was clear enough, apologies if it wasn't. > > > > Clear as mud. > Alright, let me see if I can clear this up. And by the way, thanks for chiming in on this. It's appreciated. I have a 3rd party api definition that I'm using to generate python classes from so that I can access this api using python. The api definition currently changes, so what I've done is saved a local copy of the html (the api definition from the vendor) and screen scraped the categories, and methods for this api. So when the api changes, I can just get a fresh definition from the vendors site, parse the html, and generate the classes again. This screen scape is saved to a json object in the format I originally mentioned: returned from json from screen scrape: {"Whatever": [{"method1": "Some Default", "async": "True"},{"method2": "Some Other Default", "async": "True"}]} **note: "method1": "Some Default" "method2": "Some Other Default" are just dummy values. ** > > > > It's late(ish), I'm tired and borderline frustrated :) > > > > I see your smiley, but perhaps you would get better results by waiting > > until you can make a better post. > > > > It *really* helps if you post actual "working" (even if "working" means > > "fails in the way I said"), *short*, *simple* code. Often you'll find > > that trying to simplify the problem gives you the insight to solve the > > problem yourself. > > > > http://www.sscce.org/ > I would normally post 'working' code, but I'm really not there yet. All I've been doing up until this point is basically proof of concept. > > > I'm going to try to guess what you're attempting, but I may get it > > completely wrong. Sorry if I do, but hopefully you'll get some insight > > even from my misunderstandings. > > > > > > > I have a base class (BaseClass - we'll call it for this example) with an > > > http call that i would like to inherit into a dynamic class at runtime. > > > We'll call that method in BaseClass; 'request'. > > > > If I read this literally, you want to do this: > > > > class BaseClass(DynamicParent): > > def request(self): > > ... > > > > except that DynamicParent isn't known until runtime. Am I close? The parent is the stable/static part. That has the http request method to communicate with the vendor api. The request method in BaseClass creates the request(signs and authorizes the call). Right now the BaseClass.request("api_call_to_vendor") will work and return data, but again I would like to separate each api category into classes with the appropriate methods. > > Obviously the above syntax won't work, but you can use a factory: > > > > def make_baseclass(parent): > > class BaseClass(parent): > > def request(self): > > ... > > return BaseClass > > > > class Spam: ... > > > > BaseClass = make_baseclass(Spam) > > > > > > Or you can use the type() constructor directly: > > > > BaseClass = type('BaseClass', (Spam,), dict_of_methods_and_stuff) > This, on the surface is what I'm after. Except the 'dict_of_methods_and_stuff' call would be something like this: Vendor_API_Cateogry = type("Vendor_API_Category", (BaseClass,), {"api_call_from_vendors_category": "call_supers_request_method_passing_in_vendor_call"}) resulting in a call something like this: vendor_category = Vendor_API_Category() vendor_category.api_call_from_vendors_category() > > which is probably far less convenient. But all this assumes I read you > > literally, and reading on, I don't think that's what you are after. > > > > > > > I have a dictionary(json) of key (class name): value(method) that I > > > would like to create inheriting this 'request' method from the > > > BaseClass. So the derived class would look something like this > > > > > > definition in json: > > > {"Whatever": [{"method1": "Some Default", "async": True},{"method2": > > > "Some Other Default", "async": True}]} > > > > > > Pure gobbledygook to me. I don't understand what you're talking about, > > how does a derived class turn into JSON? (Could be worse, it could be > > XML.) Is BaseClass the "derived class", or are you talking about > > inheriting from BaseClass? What's "Some Default"? It looks like a string, > > and it certainly isn't a valid method name, not with a space in it. > > > > Where did async and method2 come from? How do these things relate to > > "request" you talk about above? I think you're too close to the problem > > and don't realise that others don't sharing your knowledge of the problem. Agreed. > > But, moving along, if I've understood you correctly, I don't think > > inheritance is the right solution here. I think that composition or > > delegation may be better. Something like this: > > > > class BaseClass: > > def request(self): > > # Delegate to a method set dynamically, on the instance. > > return self.some_method() > > > > > > a = BaseClass() > > a.some_method = one_thing.method1 > > > > b = BaseClass() > > b.some_method = another_thing.method2 > > > > > > Now you have instance a.request calling method1 of another object, and > > b.request calling method2 of a different object. Does that solve your > > problem, or am I on a wild-goose chase? > > > > > > > Ideally I'd like the class def to look something like this if i were to > > > type it out by hand > > > > > > [excuse the indents] > > > > > > class Whatever(BaseClass): > > > def method1(self): > > > stupid_data = super(Whatever, self).request("method1") > > > return stupid_data > > > > > > def method2(self): > > > stupid_data = super(Whatever, self).request("method1") > > > return stupid_data > > > > > > Since request is not the method you are currently in, the above is > > equivalent to: > > > > class Whatever(BaseClass): > > def method1(self): > > return self.request("method1") > > def method2(self): > > return self.request("method2") > > > > where "request" is defined by BaseClass, and assuming you don't override > > it in the subclass. (I assume "method1" in your code above was a typo.) > > > > > > > Now, I've been trying to do this using the python cli, with out success. > > > > > > So, attempting this at runtime I get a plethora of wonderful errors that > > > I suspect has broken my brain. > > > > > > Here is what i've tried: > > > > > > # trying with just an empty object of type BaseClass > > > obj = type("Object", (BaseClass,), {}) > > > > "obj" here is a class called "Object", inheriting from BaseClass. It > > overrides no methods. Why does it exist? It doesn't do anything. > > 'obj" was just an example. I was trying to inherit from BaseClass (attempting to get the BaseClass.request method into 'obj') and then try and extend that? > > > > whatever = type("WhatEver", (obj,), {"method1": super(WhatEver, > > > self).request("method1")}) > > > > > > but when i try this I get 'NameError: name 'self' is not defined' > > > > This is what you are doing: > > > > * look up names WhatEver and self, in the current scope (i.e. the scope > > where you are running this call to type, which is likely the global > > scope); > > > > * pass those objects (if they exist!) to super(), right now; > > > > * on the object returned, look up the attribute "request", right now; > > > > * call that object with string argument "method1", right now; > > > > * take the result of that sequences of calls, let's call it x, and set it > > in a new dict with key "method1"; > > > > * and then create a new type using dict {'method1': x}. > > > > > > Notice that everything you do is done immediately. You should be getting > > a NameError for WhatEver too, but since you are not, I can only imagine > > that you have already (accidentally?) defined something, anything, with > > that name. > > > > What you actually want (but probably not what you *need*, see above about > > delegation) is to delay evaluation of super() and subsequent calls until > > after your WhatEver class is created and initialised and the method1 > > *method* is called. That means putting it all those calls inside a > > function object, for later use, instead of executing them directly *right > > now*: > > > > def method1(self): > > return super(WhatEver, self).request("method1") > > > > WhatEver = type("WhatEver", (BaseClass,), {"method1": method1}) > > > > You could alternatively use lambda: > > > > WhatEver = type("WhatEver", (BaseClass,), > > {"method1": > > lambda self: super(WhatEver, self).request("method1") > > } > > ) > > > > > > Note that the class name inside super() *must* match the global name the > > class is assigned to, "WhatEver". The internal class __name__ -- the > > first argument to type() -- doesn't have to match, but it should, to > > avoid confusion. > > > > The above should fix the NameError you are getting, and it might even > > work, but I think delegation is a better solution to this problem. > > Was any of this clear? And Steven, thank you again. > > > > -- > > Steven From ethan at stoneleaf.us Mon Jul 29 03:55:53 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 29 Jul 2013 00:55:53 -0700 Subject: MyOpenID.com no longer supported Message-ID: <51F62009.7080304@stoneleaf.us> Excerpt from http://meta.stackoverflow.com/q/190442/176681: Janrain no longer actively supports MyOpenID, and announced on Twitter that their users should proceed with caution. This decision was made by Janrain, [snip] I know the Python bug tracker allows MyOpenID logins; if that is your only login method you should add another that doesn't depend on MyOpenID. -- ~Ethan~ From solipsis at pitrou.net Mon Jul 29 05:11:35 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 29 Jul 2013 11:11:35 +0200 Subject: MyOpenID.com no longer supported References: <51F62009.7080304@stoneleaf.us> Message-ID: <20130729111135.5978fbe7@pitrou.net> Le Mon, 29 Jul 2013 00:55:53 -0700, Ethan Furman a ?crit : > Excerpt from http://meta.stackoverflow.com/q/190442/176681: > > Janrain no longer actively supports MyOpenID, and announced on > Twitter that their users should proceed with caution. > > This decision was made by Janrain, [snip] > > I know the Python bug tracker allows MyOpenID logins; if that is your > only login method you should add another that doesn't depend on > MyOpenID. I don't understand. The tracker allows any (sufficently compliant) OpenID provider, not just "MyOpenID" or "MyCorporateOpenIDWithTrademarks". Regards Antoine. From ethan at stoneleaf.us Mon Jul 29 11:58:47 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 29 Jul 2013 08:58:47 -0700 Subject: MyOpenID.com no longer supported In-Reply-To: <20130729111135.5978fbe7@pitrou.net> References: <51F62009.7080304@stoneleaf.us> <20130729111135.5978fbe7@pitrou.net> Message-ID: <51F69137.6050008@stoneleaf.us> On 07/29/2013 02:11 AM, Antoine Pitrou wrote: > Le Mon, 29 Jul 2013 00:55:53 -0700, > Ethan Furman a ?crit : >> Excerpt from http://meta.stackoverflow.com/q/190442/176681: >> >> Janrain no longer actively supports MyOpenID, and announced on >> Twitter that their users should proceed with caution. >> >> This decision was made by Janrain, [snip] >> >> I know the Python bug tracker allows MyOpenID logins; if that is your >> only login method you should add another that doesn't depend on >> MyOpenID. > > I don't understand. The tracker allows any (sufficently compliant) > OpenID provider, not just "MyOpenID" or > "MyCorporateOpenIDWithTrademarks". That is true, but if your OpenID provider is MyOpenID (as mine was) then it would be wise to get another. -- ~Ethan~ From robin at reportlab.com Mon Jul 29 06:00:16 2013 From: robin at reportlab.com (Robin Becker) Date: Mon, 29 Jul 2013 11:00:16 +0100 Subject: substituting proxy Message-ID: <51F63D30.2010205@chamonix.reportlab.co.uk> Before attempting to reinvent the wheel has anyone created an http(s) proxy that can replace the content for specific requests. Context: I have access to the client's test site, but a lot of the requests are dynamic and save html complete etc etc doesn't work properly. In addition lots of the content comes from a cloud server which seems to object unless I take great care over spoofing of host names & referrers etc. I would like to debug stuff we supply, but that's hard unless I have the whole kit & kaboodle. I thought a proxy that could substitute for our javascript file(s) might work. Has anyone done this in twisted pymiproxy etc etc? -- Robin Becker From morten.guldager at gmail.com Mon Jul 29 07:41:08 2013 From: morten.guldager at gmail.com (Morten Guldager) Date: Mon, 29 Jul 2013 13:41:08 +0200 Subject: Generating HTML Message-ID: 'Aloha Friends! Still a bit new to python I'm afraid of choosing an obsolete route when it comes to generate some HTML in a Flask based micro web server. I come from the Perl side where I have been using HTML::Element with great success, and now I would like to know if something similar exists for Python? I would like to construct the complete page as a plain python structure and then feed it to a function to get a chunk of HTML ready to send to the client. Something like: table_struct = ['table', ['tr', ['td', {class=>"red"}, "this is red"],['td', {class=>"blue"}, "this is not red"]]] html = struct2html(table_struct) Suggestions? -- /Morten %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From burak.arslan at arskom.com.tr Mon Jul 29 08:14:48 2013 From: burak.arslan at arskom.com.tr (Burak Arslan) Date: Mon, 29 Jul 2013 15:14:48 +0300 Subject: Generating HTML In-Reply-To: References: Message-ID: <51F65CB8.1020506@arskom.com.tr> Hi, On 07/29/13 14:41, Morten Guldager wrote: > Something like: > table_struct = ['table', ['tr', ['td', {class=>"red"}, "this is > red"],['td', {class=>"blue"}, "this is not red"]]] > html = struct2html(table_struct) > > Suggestions? > See: http://lxml.de/lxmlhtml.html#creating-html-with-the-e-factory Python 2.7.5 (default, Jul 16 2013, 00:38:32) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from lxml.html.builder import E >>> e = E.html( ... E.head( ... E.title("Sample Html Page") ... ), ... E.body( ... E.div( ... E.h1("Hello World"), ... E.p( ... "lxml is quite nice when it comes to generating HTML " ... "from scratch" ... ), ... id="container", ... ) ... ) ... ) >>> from lxml.html import tostring >>> print tostring(e, pretty_print=True) Sample Html Page

Hello World

lxml is quite nice when it comes to generating HTML from scratch

From jaiprakashsingh213 at gmail.com Mon Jul 29 08:55:19 2013 From: jaiprakashsingh213 at gmail.com (Jaiky) Date: Mon, 29 Jul 2013 05:55:19 -0700 (PDT) Subject: AssertionError: Headers already set! Status: 500 Internal Server Error Content-Type: text/plain Content-Length: 59 Message-ID: <2cfddec4-58bf-4694-8ac4-de4f4768bc50@googlegroups.com> learning web concpt in python wrote code in /usr/lib/cgi-bin/hello_world.py ########################################################################### #!/usr/bin/env python import webapp2 form ="""
""" class MainPage(webapp2.RequestHandler): def get(self): self.response.out.write(form) class TestHandler(webapp2.RequestHandler): def get(self): q=self.request.get("q") self.response.out.write(q) apps = webapp2.WSGIApplication([('/', MainPage),('/testform',TestHandler)],debug=True) def main(): apps.run() if __name__ == '__main__': main() ############################################################################### when running this code on ubuntu 12.04 using command python hello_world.py error obtained AssertionError: Headers already set! Status: 500 Internal Server Error Content-Type: text/plain Content-Length: 59 ########################################################################## want help please urgent .................... From tjreedy at udel.edu Mon Jul 29 12:39:52 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 29 Jul 2013 12:39:52 -0400 Subject: AssertionError: Headers already set! Status: 500 Internal Server Error Content-Type: text/plain Content-Length: 59 In-Reply-To: <2cfddec4-58bf-4694-8ac4-de4f4768bc50@googlegroups.com> References: <2cfddec4-58bf-4694-8ac4-de4f4768bc50@googlegroups.com> Message-ID: On 7/29/2013 8:55 AM, Jaiky wrote: > learning web concpt in python > wrote code in /usr/lib/cgi-bin/hello_world.py > > ########################################################################### > #!/usr/bin/env python > > import webapp2 This is an external package. > form =""" >
> > > """ > > class MainPage(webapp2.RequestHandler): > def get(self): > self.response.out.write(form) > > class TestHandler(webapp2.RequestHandler): > def get(self): > q=self.request.get("q") > self.response.out.write(q) > > apps = webapp2.WSGIApplication([('/', MainPage),('/testform',TestHandler)],debug=True) > > def main(): > apps.run() > > if __name__ == '__main__': > main() > ############################################################################### > > when running this code on ubuntu 12.04 using command > > python hello_world.py > > error obtained > > AssertionError: Headers already set! This all comes from webapp2. You should never see assertion error. Report to authors or ask on webapp2 mailing list if there is one. > Status: 500 Internal Server Error > Content-Type: text/plain > Content-Length: 59 You should have made subject line something like "Webapp2: AssertionError" as only someone familiar with webapp2 could give you any help. -- Terry Jan Reedy From steve+comp.lang.python at pearwood.info Mon Jul 29 11:43:24 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 29 Jul 2013 15:43:24 GMT Subject: Unexpected results comparing float to Fraction Message-ID: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> Comparing floats to Fractions gives unexpected results: # Python 3.3 py> from fractions import Fraction py> 1/3 == Fraction(1, 3) False but: py> 1/3 == float(Fraction(1, 3)) True I expected that float-to-Fraction comparisons would convert the Fraction to a float, but apparently they do the opposite: they convert the float to a Fraction: py> Fraction(1/3) Fraction(6004799503160661, 18014398509481984) Am I the only one who is surprised by this? Is there a general rule for which way numeric coercions should go when doing such comparisons? -- Steven From ian.g.kelly at gmail.com Mon Jul 29 11:50:34 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 29 Jul 2013 09:50:34 -0600 Subject: Unexpected results comparing float to Fraction In-Reply-To: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Mon, Jul 29, 2013 at 9:43 AM, Steven D'Aprano wrote: > Comparing floats to Fractions gives unexpected results: > > # Python 3.3 > py> from fractions import Fraction > py> 1/3 == Fraction(1, 3) > False > > but: > > py> 1/3 == float(Fraction(1, 3)) > True > > > I expected that float-to-Fraction comparisons would convert the Fraction > to a float, but apparently they do the opposite: they convert the float > to a Fraction: > > py> Fraction(1/3) > Fraction(6004799503160661, 18014398509481984) > > > Am I the only one who is surprised by this? Is there a general rule for > which way numeric coercions should go when doing such comparisons? Any float can be precisely represented as a Fraction. Not so in the other direction. So from that standpoint it makes sense to me to cast to Fraction when comparing. From tjreedy at udel.edu Mon Jul 29 13:08:20 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 29 Jul 2013 13:08:20 -0400 Subject: Unexpected results comparing float to Fraction In-Reply-To: References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 7/29/2013 11:50 AM, Ian Kelly wrote: > On Mon, Jul 29, 2013 at 9:43 AM, Steven D'Aprano > wrote: >> Comparing floats to Fractions gives unexpected results: >> >> # Python 3.3 >> py> from fractions import Fraction >> py> 1/3 == Fraction(1, 3) >> False >> >> but: >> >> py> 1/3 == float(Fraction(1, 3)) >> True >> >> >> I expected that float-to-Fraction comparisons would convert the Fraction >> to a float, but apparently they do the opposite: they convert the float >> to a Fraction: >> >> py> Fraction(1/3) >> Fraction(6004799503160661, 18014398509481984) >> >> >> Am I the only one who is surprised by this? Is there a general rule for >> which way numeric coercions should go when doing such comparisons? > > Any float can be precisely represented as a Fraction. Not so in the > other direction. In other words, there can be multiple unequal Franctions that have the same float value: for instance, Fraction(1,3) and Fraction(6004799503160661, 18014398509481984) > So from that standpoint it makes sense to me to cast to > Fraction when comparing. Otherwise, == becomes non-transitive -- Terry Jan Reedy From steve+comp.lang.python at pearwood.info Mon Jul 29 13:29:48 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 29 Jul 2013 17:29:48 GMT Subject: Unexpected results comparing float to Fraction References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51f6a68c$0$30000$c3e8da3$5496439d@news.astraweb.com> On Mon, 29 Jul 2013 13:08:20 -0400, Terry Reedy wrote: > In other words, there can be multiple unequal Franctions that have the > same float value: for instance, Fraction(1,3) and > Fraction(6004799503160661, 18014398509481984) > > > So from that standpoint it makes sense to me to cast to Fraction when > > comparing. > > Otherwise, == becomes non-transitive This is Python, and we can make __eq__ methods that do anything, including be non-transitive, non-reflexive, and nonsensical if we like :-) But I take your point, and that makes sense. -- Steven From tjreedy at udel.edu Mon Jul 29 16:48:40 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 29 Jul 2013 16:48:40 -0400 Subject: Unexpected results comparing float to Fraction In-Reply-To: <51f6a68c$0$30000$c3e8da3$5496439d@news.astraweb.com> References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> <51f6a68c$0$30000$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 7/29/2013 1:29 PM, Steven D'Aprano wrote: > On Mon, 29 Jul 2013 13:08:20 -0400, Terry Reedy wrote: > >> In other words, there can be multiple unequal Franctions that have the >> same float value: for instance, Fraction(1,3) and >> Fraction(6004799503160661, 18014398509481984) >> >> > So from that standpoint it makes sense to me to cast to Fraction when >> > comparing. >> >> Otherwise, == becomes non-transitive > > This is Python, and we can make __eq__ methods that do anything, > including be non-transitive, non-reflexive, and nonsensical if we like :-) Yes, Python's developers can intentionally introduce bugs, but we try not to. The definitions of sets and dicts and containment assume that == means equality as mathematically defined. As one time, we had 0 == 0.0 and 0 == Decimal(0) but 0.0 != Decimal(0) (and so on for all integral float values. That 'misfeature' was corrected because of the 'problems' it caused. That lesson learned, one of the design requirements for the new enum class (metaclass) was that it not re-introduce non-transitivity. -- Terry Jan Reedy From python at mrabarnett.plus.com Mon Jul 29 12:09:24 2013 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 29 Jul 2013 17:09:24 +0100 Subject: Unexpected results comparing float to Fraction In-Reply-To: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51F693B4.9000201@mrabarnett.plus.com> On 29/07/2013 16:43, Steven D'Aprano wrote: > Comparing floats to Fractions gives unexpected results: > > # Python 3.3 > py> from fractions import Fraction > py> 1/3 == Fraction(1, 3) > False > > but: > > py> 1/3 == float(Fraction(1, 3)) > True > > > I expected that float-to-Fraction comparisons would convert the Fraction > to a float, but apparently they do the opposite: they convert the float > to a Fraction: > > py> Fraction(1/3) > Fraction(6004799503160661, 18014398509481984) > > > Am I the only one who is surprised by this? Is there a general rule for > which way numeric coercions should go when doing such comparisons? > I'm surprised that Fraction(1/3) != Fraction(1, 3); after all, floats are approximate anyway, and the float value 1/3 is more likely to be Fraction(1, 3) than Fraction(6004799503160661, 18014398509481984). From rosuav at gmail.com Mon Jul 29 12:20:27 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 29 Jul 2013 17:20:27 +0100 Subject: Unexpected results comparing float to Fraction In-Reply-To: <51F693B4.9000201@mrabarnett.plus.com> References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F693B4.9000201@mrabarnett.plus.com> Message-ID: On Mon, Jul 29, 2013 at 5:09 PM, MRAB wrote: > I'm surprised that Fraction(1/3) != Fraction(1, 3); after all, floats > are approximate anyway, and the float value 1/3 is more likely to be > Fraction(1, 3) than Fraction(6004799503160661, 18014398509481984). At what point should it become Fraction(1, 3)? >>> Fraction(0.3) Fraction(5404319552844595, 18014398509481984) >>> Fraction(0.33) Fraction(5944751508129055, 18014398509481984) >>> Fraction(0.333) Fraction(5998794703657501, 18014398509481984) >>> Fraction(0.3333333) Fraction(6004798902680711, 18014398509481984) >>> Fraction(0.3333333333) Fraction(6004799502560181, 18014398509481984) >>> Fraction(0.3333333333333) Fraction(6004799503160061, 18014398509481984) >>> Fraction(0.33333333333333333) Fraction(6004799503160661, 18014398509481984) Rounding off like that is a job for a cool library function (one of which was mentioned on this list a little while ago, I believe), but not IMO for the Fraction constructor. ChrisA From python at mrabarnett.plus.com Mon Jul 29 12:48:21 2013 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 29 Jul 2013 17:48:21 +0100 Subject: Unexpected results comparing float to Fraction In-Reply-To: References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F693B4.9000201@mrabarnett.plus.com> Message-ID: <51F69CD5.8060709@mrabarnett.plus.com> On 29/07/2013 17:20, Chris Angelico wrote: > On Mon, Jul 29, 2013 at 5:09 PM, MRAB wrote: >> I'm surprised that Fraction(1/3) != Fraction(1, 3); after all, floats >> are approximate anyway, and the float value 1/3 is more likely to be >> Fraction(1, 3) than Fraction(6004799503160661, 18014398509481984). > > At what point should it become Fraction(1, 3)? > When the error drops below a certain threshold. >>>> Fraction(0.3) > Fraction(5404319552844595, 18014398509481984) >>>> Fraction(0.33) > Fraction(5944751508129055, 18014398509481984) >>>> Fraction(0.333) > Fraction(5998794703657501, 18014398509481984) >>>> Fraction(0.3333333) > Fraction(6004798902680711, 18014398509481984) >>>> Fraction(0.3333333333) > Fraction(6004799502560181, 18014398509481984) >>>> Fraction(0.3333333333333) > Fraction(6004799503160061, 18014398509481984) >>>> Fraction(0.33333333333333333) > Fraction(6004799503160661, 18014398509481984) > > Rounding off like that is a job for a cool library function (one of > which was mentioned on this list a little while ago, I believe), but > not IMO for the Fraction constructor. > From ian.g.kelly at gmail.com Mon Jul 29 12:40:52 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 29 Jul 2013 10:40:52 -0600 Subject: Unexpected results comparing float to Fraction In-Reply-To: References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F693B4.9000201@mrabarnett.plus.com> Message-ID: On Mon, Jul 29, 2013 at 10:20 AM, Chris Angelico wrote: > On Mon, Jul 29, 2013 at 5:09 PM, MRAB wrote: >> I'm surprised that Fraction(1/3) != Fraction(1, 3); after all, floats >> are approximate anyway, and the float value 1/3 is more likely to be >> Fraction(1, 3) than Fraction(6004799503160661, 18014398509481984). > > At what point should it become Fraction(1, 3)? At the point where the float is exactly equal to the value you get from the floating-point division 1/3. If it's some other float then the user didn't get there by entering 1/3, so it's not worth trying to pretend that they did. We do a similar rounding when formatting floats to strings, but in that case one only has to worry about divisors that are powers of 10. I imagine it's going to take more time to find the correct fraction when any pair of relatively prime integers can be a candidate numerator and denominator. Additionally, the string rounding only occurs when the float is being formatted for display; we certainly don't do it as the result of numeric operations where it could result in loss of precision. From python at mrabarnett.plus.com Mon Jul 29 13:04:19 2013 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 29 Jul 2013 18:04:19 +0100 Subject: Unexpected results comparing float to Fraction In-Reply-To: References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F693B4.9000201@mrabarnett.plus.com> Message-ID: <51F6A093.5000803@mrabarnett.plus.com> On 29/07/2013 17:40, Ian Kelly wrote: > On Mon, Jul 29, 2013 at 10:20 AM, Chris Angelico wrote: >> On Mon, Jul 29, 2013 at 5:09 PM, MRAB wrote: >>> I'm surprised that Fraction(1/3) != Fraction(1, 3); after all, floats >>> are approximate anyway, and the float value 1/3 is more likely to be >>> Fraction(1, 3) than Fraction(6004799503160661, 18014398509481984). >> >> At what point should it become Fraction(1, 3)? > > At the point where the float is exactly equal to the value you get > from the floating-point division 1/3. If it's some other float then > the user didn't get there by entering 1/3, so it's not worth trying to > pretend that they did. > I thought that you're not meant to check for equality when using floats. > We do a similar rounding when formatting floats to strings, but in > that case one only has to worry about divisors that are powers of 10. > I imagine it's going to take more time to find the correct fraction > when any pair of relatively prime integers can be a candidate > numerator and denominator. Additionally, the string rounding only > occurs when the float is being formatted for display; we certainly > don't do it as the result of numeric operations where it could result > in loss of precision. > From ian.g.kelly at gmail.com Mon Jul 29 13:35:18 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 29 Jul 2013 11:35:18 -0600 Subject: Unexpected results comparing float to Fraction In-Reply-To: <51F6A093.5000803@mrabarnett.plus.com> References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F693B4.9000201@mrabarnett.plus.com> <51F6A093.5000803@mrabarnett.plus.com> Message-ID: On Mon, Jul 29, 2013 at 11:04 AM, MRAB wrote: > On 29/07/2013 17:40, Ian Kelly wrote: >> At the point where the float is exactly equal to the value you get >> from the floating-point division 1/3. If it's some other float then >> the user didn't get there by entering 1/3, so it's not worth trying to >> pretend that they did. >> > I thought that you're not meant to check for equality when using floats. Equality checking is useful for floats when there is exactly one value that you want to test for as in this case, as opposed to a range of approximate values that are considered to be equal within rounding error. There is exactly one float that the expression 0.1 evaluates to, and although it's not equal to the decimal 0.1, it is the only float that we want to format as "0.1". The immediately preceding and following floats are 0.09999999999999999 and 0.10000000000000002, and they are formatted as such because they're not equal to the float that you get from 0.1. From rosuav at gmail.com Mon Jul 29 13:14:00 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 29 Jul 2013 18:14:00 +0100 Subject: Unexpected results comparing float to Fraction In-Reply-To: References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F693B4.9000201@mrabarnett.plus.com> Message-ID: On Mon, Jul 29, 2013 at 5:40 PM, Ian Kelly wrote: > On Mon, Jul 29, 2013 at 10:20 AM, Chris Angelico wrote: >> On Mon, Jul 29, 2013 at 5:09 PM, MRAB wrote: >>> I'm surprised that Fraction(1/3) != Fraction(1, 3); after all, floats >>> are approximate anyway, and the float value 1/3 is more likely to be >>> Fraction(1, 3) than Fraction(6004799503160661, 18014398509481984). >> >> At what point should it become Fraction(1, 3)? > > At the point where the float is exactly equal to the value you get > from the floating-point division 1/3. If it's some other float then > the user didn't get there by entering 1/3, so it's not worth trying to > pretend that they did. > > We do a similar rounding when formatting floats to strings, but in > that case one only has to worry about divisors that are powers of 10. > I imagine it's going to take more time to find the correct fraction > when any pair of relatively prime integers can be a candidate > numerator and denominator. I imagine it is, and that's where the problem comes in. The true value is somewhere between (X-0.5)/2**n and (X+0.5)/2**n, or whatever the actual range is, and finding a "nice" fraction in that range isn't an instant and direct translation. It's a useful feature, but not IMO necessary for the constructor. ChrisA From invalid at invalid.invalid Mon Jul 29 14:46:33 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Mon, 29 Jul 2013 18:46:33 +0000 (UTC) Subject: Unexpected results comparing float to Fraction References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F693B4.9000201@mrabarnett.plus.com> Message-ID: On 2013-07-29, MRAB wrote: > On 29/07/2013 17:40, Ian Kelly wrote: >> On Mon, Jul 29, 2013 at 10:20 AM, Chris Angelico wrote: >>> On Mon, Jul 29, 2013 at 5:09 PM, MRAB wrote: >>>> I'm surprised that Fraction(1/3) != Fraction(1, 3); after all, floats >>>> are approximate anyway, and the float value 1/3 is more likely to be >>>> Fraction(1, 3) than Fraction(6004799503160661, 18014398509481984). >>> >>> At what point should it become Fraction(1, 3)? >> >> At the point where the float is exactly equal to the value you get >> from the floating-point division 1/3. If it's some other float then >> the user didn't get there by entering 1/3, so it's not worth trying to >> pretend that they did. > > I thought that you're not meant to check for equality when using floats. You check for equality if equality is what you want to check. However much of the time when people _think_ they want to check for FP equality, they're wrong. You'll have to consult with a spiritual advisor to determin what you are "meant" to do... -- Grant Edwards grant.b.edwards Yow! Awright, which one of at you hid my PENIS ENVY? gmail.com From steve+comp.lang.python at pearwood.info Mon Jul 29 16:19:58 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 29 Jul 2013 20:19:58 GMT Subject: Unexpected results comparing float to Fraction References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F693B4.9000201@mrabarnett.plus.com> Message-ID: <51f6ce6d$0$30000$c3e8da3$5496439d@news.astraweb.com> On Mon, 29 Jul 2013 18:04:19 +0100, MRAB wrote: > I thought that you're not meant to check for equality when using floats. Heavens no. You're only meant to *mostly* not check for equality using floats. As Miracle Max might say: "It just so happens that floats are only MOSTLY inexact. There's a big difference between mostly inexact and all inexact. Mostly inexact is slightly exact." Or to be a little more serious, "never check floats for equality" is actually pretty lousy advice. As Professor William Kahan puts it, the word "never" makes it rank superstition. There are three main problems with the general advice "never check floats for equality": 1) there are perfectly fine situations where you can check floats for (in)equality without any problem, e.g.: if x == 0.0: print "Division by zero" else: y = 1/x # perfectly safe 2) most of the time, those giving this advice don't actually say what you should do instead; 3) and when they do, it's often either flat-out wrong, or at least incomplete. For example, you might have been told to instead check whether your float is less than some epsilon: abs(x - y) < eps But chances are you've never been given any sort of reliable, deterministic algorithm for deciding what value eps should have. (Mostly because there is no such algorithm!) Or for deciding when to use absolute differences, like the above, and when to use relative differences. Another issue: if eps is too big, you're taking distinct values and treating them as the same, which is bad. And if eps is too small, you're actually doing what you said you should never do, only slower. E.g. suppose you want to check for a float which equals some value M, I'll pick M = 2199023255552.0 semi-arbitrarily, but it could be any of many numbers. You don't want to use equality, so you pick an epsilon: eps = 1e-6 # seems reasonable... and then test values like this: M = 2199023255552.0 if abs(x - M) <= eps: print("close enough") That is just a longer and slower way of calculating x == M. Why? Because the two floats immediately adjacent to M differ by more than your epsilon: py> M = 2199023255552.0 py> M.hex() '0x1.0000000000000p+41' py> float.fromhex('0x1.0000000000001p+41') - M 0.00048828125 py> M - float.fromhex('0x1.fffffffffffffp+40') 0.000244140625 So in this case, any epsilon below 0.000244140625 is just wasting your time, since it cannot possibly detect any value other than M itself. And that critical cut-off depends on the numbers you are calculating with. Oh, for what it's worth, I don't pretend to know how to choose an epsilon either. I can sometimes recognise a bad epsilon, but not a good one. Floats are *hard*. -- Steven From steve+comp.lang.python at pearwood.info Mon Jul 29 13:27:10 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 29 Jul 2013 17:27:10 GMT Subject: Unexpected results comparing float to Fraction References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F693B4.9000201@mrabarnett.plus.com> Message-ID: <51f6a5ed$0$30000$c3e8da3$5496439d@news.astraweb.com> On Mon, 29 Jul 2013 17:48:21 +0100, MRAB wrote: > On 29/07/2013 17:20, Chris Angelico wrote: >> On Mon, Jul 29, 2013 at 5:09 PM, MRAB >> wrote: >>> I'm surprised that Fraction(1/3) != Fraction(1, 3); after all, floats >>> are approximate anyway, and the float value 1/3 is more likely to be >>> Fraction(1, 3) than Fraction(6004799503160661, 18014398509481984). >> >> At what point should it become Fraction(1, 3)? >> > When the error drops below a certain threshold. Good plan! I pick a threshold of 42.7. Anyone got a better threshold? *wink* -- Steven From sg552 at hotmail.co.uk Mon Jul 29 14:16:45 2013 From: sg552 at hotmail.co.uk (Rotwang) Date: Mon, 29 Jul 2013 19:16:45 +0100 Subject: Unexpected results comparing float to Fraction In-Reply-To: References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F693B4.9000201@mrabarnett.plus.com> Message-ID: On 29/07/2013 17:40, Ian Kelly wrote: > On Mon, Jul 29, 2013 at 10:20 AM, Chris Angelico wrote: >> On Mon, Jul 29, 2013 at 5:09 PM, MRAB wrote: >>> I'm surprised that Fraction(1/3) != Fraction(1, 3); after all, floats >>> are approximate anyway, and the float value 1/3 is more likely to be >>> Fraction(1, 3) than Fraction(6004799503160661, 18014398509481984). >> >> At what point should it become Fraction(1, 3)? > > At the point where the float is exactly equal to the value you get > from the floating-point division 1/3. But the interpreter has no way of knowing that the value 1/3 that's been passed to the Fraction constructor was obtained from the division 1/3, rather than, say, 100000000000000001/300000000000000000 or 6004799503160661/18014398509481984. How do you propose the constructor should decide between the many possible fractions that round to the same float, if not by choosing the one that evaluates to it exactly? Personally the behaviour in the OP is exactly what I would expect. From ian.g.kelly at gmail.com Mon Jul 29 15:33:31 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 29 Jul 2013 13:33:31 -0600 Subject: Unexpected results comparing float to Fraction In-Reply-To: References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F693B4.9000201@mrabarnett.plus.com> Message-ID: On Mon, Jul 29, 2013 at 12:16 PM, Rotwang wrote: > On 29/07/2013 17:40, Ian Kelly wrote: >> At the point where the float is exactly equal to the value you get >> from the floating-point division 1/3. > > > But the interpreter has no way of knowing that the value 1/3 that's been > passed to the Fraction constructor was obtained from the division 1/3, > rather than, say, 100000000000000001/300000000000000000 or > 6004799503160661/18014398509481984. How do you propose the constructor > should decide between the many possible fractions that round to the same > float, if not by choosing the one that evaluates to it exactly? It should choose the fraction with the least terms that rounds to the float. Whether "least" here means least numerator, least denominator, least sum of the two, or whatever is not terribly important. From sg552 at hotmail.co.uk Mon Jul 29 14:20:32 2013 From: sg552 at hotmail.co.uk (Rotwang) Date: Mon, 29 Jul 2013 19:20:32 +0100 Subject: Unexpected results comparing float to Fraction In-Reply-To: References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F693B4.9000201@mrabarnett.plus.com> Message-ID: On 29/07/2013 17:20, Chris Angelico wrote: > On Mon, Jul 29, 2013 at 5:09 PM, MRAB wrote: >> I'm surprised that Fraction(1/3) != Fraction(1, 3); after all, floats >> are approximate anyway, and the float value 1/3 is more likely to be >> Fraction(1, 3) than Fraction(6004799503160661, 18014398509481984). > > At what point should it become Fraction(1, 3)? > >>>> Fraction(0.3) > Fraction(5404319552844595, 18014398509481984) >>>> Fraction(0.33) > Fraction(5944751508129055, 18014398509481984) >>>> Fraction(0.333) > Fraction(5998794703657501, 18014398509481984) >>>> Fraction(0.3333333) > Fraction(6004798902680711, 18014398509481984) >>>> Fraction(0.3333333333) > Fraction(6004799502560181, 18014398509481984) >>>> Fraction(0.3333333333333) > Fraction(6004799503160061, 18014398509481984) >>>> Fraction(0.33333333333333333) > Fraction(6004799503160661, 18014398509481984) > > Rounding off like that is a job for a cool library function (one of > which was mentioned on this list a little while ago, I believe), but > not IMO for the Fraction constructor. How about this? >>> from fractions import Fraction >>> help(Fraction.limit_denominator) Help on function limit_denominator in module fractions: limit_denominator(self, max_denominator=1000000) Closest Fraction to self with denominator at most max_denominator. >>> Fraction('3.141592653589793').limit_denominator(10) Fraction(22, 7) >>> Fraction('3.141592653589793').limit_denominator(100) Fraction(311, 99) >>> Fraction(4321, 8765).limit_denominator(10000) Fraction(4321, 8765) >>> Fraction(1/3).limit_denominator() Fraction(1, 3) From steve+comp.lang.python at pearwood.info Mon Jul 29 15:32:38 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 29 Jul 2013 19:32:38 GMT Subject: Unexpected results comparing float to Fraction References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F693B4.9000201@mrabarnett.plus.com> Message-ID: <51f6c356$0$30000$c3e8da3$5496439d@news.astraweb.com> On Mon, 29 Jul 2013 17:20:27 +0100, Chris Angelico wrote: >>>> Fraction(0.3) > Fraction(5404319552844595, 18014398509481984) >>>> Fraction(0.33) > Fraction(5944751508129055, 18014398509481984) [...] >>>> Fraction(0.33333333333333333) > Fraction(6004799503160661, 18014398509481984) > > Rounding off like that is a job for a cool library function (one of > which was mentioned on this list a little while ago, I believe), but not > IMO for the Fraction constructor. Or: py> Fraction(1/3).limit_denominator(100) Fraction(1, 3) I think it would be useful for Fraction.from_float() to accept an optional second argument, the maximum denominator: Fraction.from_float(x, den=None) => nearest fraction to float 1/3, with denominator no greater than den -- Steven From storchaka at gmail.com Mon Jul 29 15:34:04 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 29 Jul 2013 22:34:04 +0300 Subject: Unexpected results comparing float to Fraction In-Reply-To: <51F693B4.9000201@mrabarnett.plus.com> References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F693B4.9000201@mrabarnett.plus.com> Message-ID: 29.07.13 19:09, MRAB ???????(??): > I'm surprised that Fraction(1/3) != Fraction(1, 3); after all, floats > are approximate anyway, and the float value 1/3 is more likely to be > Fraction(1, 3) than Fraction(6004799503160661, 18014398509481984). >>> def approximate_fraction(f): prev_numer, numer = 0, 1 prev_denom, denom = 1, 0 r = f while True: i = math.floor(r) prev_numer, numer = numer, i * numer + prev_numer prev_denom, denom = denom, i * denom + prev_denom if i == r or numer / denom == f: break r = 1 / (r - i) return Fraction(numer, denom) >>> approximate_fraction(1/3) Fraction(1, 3) >>> approximate_fraction(1e-17) Fraction(1, 100000000000000000) >>> approximate_fraction(math.pi) Fraction(245850922, 78256779) I guess the Fraction constructor is more faster than this function. From ian.g.kelly at gmail.com Mon Jul 29 16:35:21 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 29 Jul 2013 14:35:21 -0600 Subject: Unexpected results comparing float to Fraction In-Reply-To: References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F693B4.9000201@mrabarnett.plus.com> Message-ID: On Jul 29, 2013 1:37 PM, "Serhiy Storchaka" wrote: > > 29.07.13 19:09, MRAB ???????(??): > >> I'm surprised that Fraction(1/3) != Fraction(1, 3); after all, floats >> are approximate anyway, and the float value 1/3 is more likely to be >> Fraction(1, 3) than Fraction(6004799503160661, 18014398509481984). > > > >>> def approximate_fraction(f): > prev_numer, numer = 0, 1 > prev_denom, denom = 1, 0 > r = f > while True: > i = math.floor(r) > prev_numer, numer = numer, i * numer + prev_numer > prev_denom, denom = denom, i * denom + prev_denom > if i == r or numer / denom == f: > break > r = 1 / (r - i) > > return Fraction(numer, denom) > > >>> approximate_fraction(1/3) > Fraction(1, 3) > >>> approximate_fraction(1e-17) > Fraction(1, 100000000000000000) > >>> approximate_fraction(math.pi) > Fraction(245850922, 78256779) > > I guess the Fraction constructor is more faster than this function. You might be able to speed it up a bit with numpy and the observation that the update is a matrix multiplication. But I don't think that you can get away from it being an iterative algorithm. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Tue Jul 30 09:32:04 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Tue, 30 Jul 2013 14:32:04 +0100 Subject: Unexpected results comparing float to Fraction In-Reply-To: <51F693B4.9000201@mrabarnett.plus.com> References: <51f68d9c$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F693B4.9000201@mrabarnett.plus.com> Message-ID: On 29 July 2013 17:09, MRAB wrote: > On 29/07/2013 16:43, Steven D'Aprano wrote: >> >> Comparing floats to Fractions gives unexpected results: You may not have expected these results but as someone who regularly uses the fractions module I do expect them. >> # Python 3.3 >> py> from fractions import Fraction >> py> 1/3 == Fraction(1, 3) >> False >> >> but: >> >> py> 1/3 == float(Fraction(1, 3)) >> True Why would you do the above? You're deliberately trying to create a float with a value that you know is not representable by the float type. The purpose of Fractions is precisely that they can represent all rational values, hence avoiding these problems. When I use Fractions my intention is to perform exact computation. I am very careful to avoid allowing floating point imprecision to sneak into my calculations. Mixing floats and fractions in computation is not IMO a good use of duck-typing. >> I expected that float-to-Fraction comparisons would convert the Fraction >> to a float, but apparently they do the opposite: they convert the float >> to a Fraction: >> >> py> Fraction(1/3) >> Fraction(6004799503160661, 18014398509481984) >> >> Am I the only one who is surprised by this? Is there a general rule for >> which way numeric coercions should go when doing such comparisons? I would say that if type A is a strict superset of type B then the coercion should be to type A. This is the case for float and Fraction since any float can be represented exactly as a Fraction but the converse is not true. > I'm surprised that Fraction(1/3) != Fraction(1, 3); after all, floats > are approximate anyway, and the float value 1/3 is more likely to be > Fraction(1, 3) than Fraction(6004799503160661, 18014398509481984). Refuse the temptation to guess: Fraction(float) should give the exact value of the float. It should not give one of the countably infinite number of other possible rational numbers that would (under a particular rounding scheme and the floating point format in question) round to the same float. If that is the kind of equality you would like to test for in some particular situation then you can do so by coercing to float explicitly. Calling Fraction(1/3) is a misunderstanding of what the fractions module is for and how to use it. The point is to guarantee avoiding floating point errors; this is impossible if you use floating point computations to initialise Fractions. Writing Fraction(1, 3) does look a bit ugly so my preferred way to reduce the boiler-plate in a script that uses lots of Fraction "literals" is to do: from fractions import Fraction as F # 1/3 + 1/9 + 1/27 + ... limit = F('1/3') / (1 - F('1/3')) That's not as good as dedicated syntax but with code highlighting it's still quite readable. Oscar From mattgraves7 at gmail.com Mon Jul 29 12:14:19 2013 From: mattgraves7 at gmail.com (Matt) Date: Mon, 29 Jul 2013 09:14:19 -0700 (PDT) Subject: What do you do when a library is outdated? Message-ID: <4ca2756b-0fae-4284-85cf-264c3f179d4d@googlegroups.com> I'm fairly new to python but have experience in other languages. What do you generally do when a library is outdated? I asked a question on a few forums and everyone has been pointing me to Mechanize, but it will not work with 3.3 What do you do? From rosuav at gmail.com Mon Jul 29 12:34:08 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 29 Jul 2013 17:34:08 +0100 Subject: What do you do when a library is outdated? In-Reply-To: <4ca2756b-0fae-4284-85cf-264c3f179d4d@googlegroups.com> References: <4ca2756b-0fae-4284-85cf-264c3f179d4d@googlegroups.com> Message-ID: On Mon, Jul 29, 2013 at 5:14 PM, Matt wrote: > I'm fairly new to python but have experience in other languages. What do you generally do when a library is outdated? I asked a question on a few forums and everyone has been pointing me to Mechanize, but it will not work with 3.3 > > What do you do? Depends what you mean by "outdated". Lots of things don't _need_ to be up-to-date to be useful, and often, using the very latest version of something just makes it hard to deploy (look at Debian and Red Hat, both of which maintain support for a long time). If there's actually a problem with something not being able to cope with current systems (eg something that's designed to communicate with Windows and can't talk to Win 8), then you go looking for a replacement package that can use the latest, or possibly you write it yourself. But my crystal ball tells me you're not asking about that, but rather about a module that was written for Python 2 and hasn't been ported to Python 3. (Usually there won't be other issues; if something breaks between Py3.2 and Py3.3, it'll be easily fixed.) There are a few options: 1) Talk to the author/maintainer. Explain that you want to use his/her code with Python 3 but can't. Often, the only reason something isn't ported is because of a perceived lack of interest. 2) Run the module code through the 2to3 utility. That might even be all you need to do. 3) Port it yourself. Start with 2to3, and then work through any problems you have. I would recommend getting to know the module on Python 2 first, so you have a chance of knowing what it ought to be doing. You aren't the first to inquire about this. A quick Google search for 'mechanize python 3' brought this up: http://web.cecs.pdx.edu/~adevore/mechanize/ Also, poking around a bit shows recommendations for the lxml and requests modules, which may be able to do what you want. So to answer your general question: Work, sometimes lots of work (though not always). But for Mechanize specifically, Requests may be your best bet. ChrisA From mattgraves7 at gmail.com Mon Jul 29 12:40:37 2013 From: mattgraves7 at gmail.com (Matt) Date: Mon, 29 Jul 2013 09:40:37 -0700 (PDT) Subject: What do you do when a library is outdated? In-Reply-To: References: <4ca2756b-0fae-4284-85cf-264c3f179d4d@googlegroups.com> Message-ID: <3e3a63f4-8e47-47a2-970d-7544e9c169ff@googlegroups.com> On Monday, July 29, 2013 12:34:08 PM UTC-4, Chris Angelico wrote: > On Mon, Jul 29, 2013 at 5:14 PM, Matt wrote: > > > I'm fairly new to python but have experience in other languages. What do you generally do when a library is outdated? I asked a question on a few forums and everyone has been pointing me to Mechanize, but it will not work with 3.3 > > > > > > What do you do? > > > > Depends what you mean by "outdated". Lots of things don't _need_ to be > > up-to-date to be useful, and often, using the very latest version of > > something just makes it hard to deploy (look at Debian and Red Hat, > > both of which maintain support for a long time). If there's actually a > > problem with something not being able to cope with current systems (eg > > something that's designed to communicate with Windows and can't talk > > to Win 8), then you go looking for a replacement package that can use > > the latest, or possibly you write it yourself. > > > > But my crystal ball tells me you're not asking about that, but rather > > about a module that was written for Python 2 and hasn't been ported to > > Python 3. (Usually there won't be other issues; if something breaks > > between Py3.2 and Py3.3, it'll be easily fixed.) There are a few > > options: > > > > 1) Talk to the author/maintainer. Explain that you want to use his/her > > code with Python 3 but can't. Often, the only reason something isn't > > ported is because of a perceived lack of interest. > > 2) Run the module code through the 2to3 utility. That might even be > > all you need to do. > > 3) Port it yourself. Start with 2to3, and then work through any > > problems you have. I would recommend getting to know the module on > > Python 2 first, so you have a chance of knowing what it ought to be > > doing. > > > > You aren't the first to inquire about this. A quick Google search for > > 'mechanize python 3' brought this up: > > http://web.cecs.pdx.edu/~adevore/mechanize/ > > > > Also, poking around a bit shows recommendations for the lxml and > > requests modules, which may be able to do what you want. > > > > So to answer your general question: Work, sometimes lots of work > > (though not always). But for Mechanize specifically, Requests may be > > your best bet. > > > > ChrisA I appreciate this. I did not know of 2to3, and I am going to give that a shot right now. Thank you! From tjreedy at udel.edu Mon Jul 29 13:12:38 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 29 Jul 2013 13:12:38 -0400 Subject: What do you do when a library is outdated? In-Reply-To: <4ca2756b-0fae-4284-85cf-264c3f179d4d@googlegroups.com> References: <4ca2756b-0fae-4284-85cf-264c3f179d4d@googlegroups.com> Message-ID: On 7/29/2013 12:14 PM, Matt wrote: > I'm fairly new to python but have experience in other languages. What > do you generally do when a library is outdated? I asked a question on > a few forums and everyone has been pointing me to Mechanize, but it > will not work with 3.3 > > What do you do? Update it yourself, ask someone else to update it, or use something else. Or regress to an older Python that it will work with. -- Terry Jan Reedy From jtim.arnold at gmail.com Mon Jul 29 15:16:51 2013 From: jtim.arnold at gmail.com (Tim) Date: Mon, 29 Jul 2013 12:16:51 -0700 (PDT) Subject: timing issue: shutil.rmtree and os.makedirs Message-ID: I have the following function (Python2.7 on FreeBSD) that results in an OSError. My intent is to pass it a directory name or path and if it exists, use shutil.rmtree to remove whatever is there (if it isn't a directory, try to unlink it); then use os.makedirs to create a new directory or path: def make_clean_dir(directory): if os.path.exists(directory): if os.path.isdir(directory): shutil.rmtree(directory) else: os.unlink(directory) os.makedirs(directory) The last bit of the traceback is: File "/develop/myproject/helpers/__init__.py", line 35, in make_clean_dir os.makedirs(directory) File "/usr/local/lib/python2.7/os.py", line 157, in makedirs mkdir(name, mode) OSError: [Errno 17] File exists: '/users/tim/testing/testing_html' The directory 'testing_html' existed when I executed the function; So I suppose the directory wasn't finished being removed by the time os.makedirs was invoked. How can avoid this? (A context manager maybe?). thanks, --Tim From rosuav at gmail.com Mon Jul 29 19:52:36 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 30 Jul 2013 00:52:36 +0100 Subject: timing issue: shutil.rmtree and os.makedirs In-Reply-To: References: Message-ID: On Mon, Jul 29, 2013 at 8:16 PM, Tim wrote: > My intent is to pass it a directory name or path and if it exists, use shutil.rmtree to remove whatever is there (if it isn't a directory, try to unlink it); then use os.makedirs to create a new directory or path: > > def make_clean_dir(directory): > if os.path.exists(directory): > if os.path.isdir(directory): > shutil.rmtree(directory) > else: > os.unlink(directory) > os.makedirs(directory) > > The last bit of the traceback is: > File "/develop/myproject/helpers/__init__.py", line 35, in make_clean_dir > os.makedirs(directory) > File "/usr/local/lib/python2.7/os.py", line 157, in makedirs > mkdir(name, mode) > OSError: [Errno 17] File exists: '/users/tim/testing/testing_html' > > The directory 'testing_html' existed when I executed the function; First thing I'd check is: Did rmtree succeed? Try removing the makedirs and test it again; then, when your process has completely finished, see if the directory is there. If it is, the problem is in rmtree - for instance: * You might not have permission to remove everything * There might be a messed-up object in the file system * If the directory is a remote share mount point, the other end might have lied about the removal * Something might have been created inside the directory during the removal * Myriad other possibilities As I understand rmtree's docs, any errors *that it detects* will be raised as exceptions (since you haven't told it to suppress or handle them), but possibly there's an error that it isn't able to detect. Worth a test, anyhow. ChrisA From jtim.arnold at gmail.com Tue Jul 30 09:10:05 2013 From: jtim.arnold at gmail.com (Tim) Date: Tue, 30 Jul 2013 06:10:05 -0700 (PDT) Subject: timing issue: shutil.rmtree and os.makedirs In-Reply-To: References: Message-ID: On Monday, July 29, 2013 7:52:36 PM UTC-4, Chris Angelico wrote: > On Mon, Jul 29, 2013 at 8:16 PM, Tim wrote: > > My intent is to pass it a directory name or path and if it exists, use shutil.rmtree to remove whatever is there (if it isn't a directory, try to unlink it); then use os.makedirs to create a new directory or path: > > def make_clean_dir(directory): > > if os.path.exists(directory): > > if os.path.isdir(directory): > > shutil.rmtree(directory) > > else: > > os.unlink(directory) > > os.makedirs(directory) > > > > The last bit of the traceback is: > > File "/develop/myproject/helpers/__init__.py", line 35, in make_clean_dir > > os.makedirs(directory) > > File "/usr/local/lib/python2.7/os.py", line 157, in makedirs > > mkdir(name, mode) > > OSError: [Errno 17] File exists: '/users/tim/testing/testing_html' > > > > The directory 'testing_html' existed when I executed the function; > > First thing I'd check is: Did rmtree succeed? Try removing the > makedirs and test it again; then, when your process has completely > finished, see if the directory is there. If it is, the problem is in > rmtree - for instance: > * You might not have permission to remove everything > * There might be a messed-up object in the file system > * If the directory is a remote share mount point, the other end might > have lied about the removal > * Something might have been created inside the directory during the removal > * Myriad other possibilities > As I understand rmtree's docs, any errors *that it detects* will be > raised as exceptions (since you haven't told it to suppress or handle > them), but possibly there's an error that it isn't able to detect. > Worth a test, anyhow. > > ChrisA Thanks Chris, but the directory was actually removed on the first run in spite of the traceback; when I run it a second time (immediately after the first time), it runs fine. That's why I thought it was a timing issue. I thought about just putting a sleep in there, but that made me feel dirty. hmm, now that you mention it, this is executing on a remote box with access to the same file system my local calling program is on. That is, there is a local call to an intermediate script that connects to a socket on the remote where the above program actually runs, but the file system is the same place for both local and remote. But even so, since the script that does the rmtree and mkdir is running on the same machine (even though it's remote), I would think the mkdir couldn't execute until the rmtree was completely finished. thanks, --Tim From rosuav at gmail.com Tue Jul 30 09:27:10 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 30 Jul 2013 14:27:10 +0100 Subject: timing issue: shutil.rmtree and os.makedirs In-Reply-To: References: Message-ID: On Tue, Jul 30, 2013 at 2:10 PM, Tim wrote: > hmm, now that you mention it, this is executing on a remote box with access to the same file system my local calling program is on. That is, there is a local call to an intermediate script that connects to a socket on the remote where the above program actually runs, but the file system is the same place for both local and remote. > > But even so, since the script that does the rmtree and mkdir is running on the same machine (even though it's remote), I would think the mkdir couldn't execute until the rmtree was completely finished. Hmm. What system is used for the file system sharing? I know quite a few of them lie about whether something's been completely done or not. Can you use inotify to tell you when the directory's been deleted? Seems stupid though. Worst case, all you need is a quick loop at the bottom, eg: for delay in 100,300,600,1000,3000,5000,10000: if not os.path.exists(directory): break sleep(delay) That'll sleep a maximum of 20 seconds, tune as required. Of course, if there's a way to tune the FS to guarantee that the removal blocks correctly, that would be way better than sleep()! ChrisA From goktug.kayaalp at gmail.com Tue Jul 30 13:09:11 2013 From: goktug.kayaalp at gmail.com (=?UTF-8?B?R8O2a3R1xJ8gS2F5YWFscA==?=) Date: Tue, 30 Jul 2013 20:09:11 +0300 Subject: timing issue: shutil.rmtree and os.makedirs In-Reply-To: References: Message-ID: On Jul 30, 2013 3:29 PM, "Chris Angelico" wrote: > > On Tue, Jul 30, 2013 at 2:10 PM, Tim wrote: > > hmm, now that you mention it, this is executing on a remote box with access to the same file system my local calling program is on. That is, there is a local call to an intermediate script that connects to a socket on the remote where the above program actually runs, but the file system is the same place for both local and remote. > > > > But even so, since the script that does the rmtree and mkdir is running on the same machine (even though it's remote), I would think the mkdir couldn't execute until the rmtree was completely finished. > > Hmm. What system is used for the file system sharing? I know quite a > few of them lie about whether something's been completely done or not. > > Can you use inotify to tell you when the directory's been deleted? > Seems stupid though. Inotify is a linux thing, but there is kqueue for Free?BSD. OP can run the deletion procedure, wait for a NOTE_DELETE event, which would block, and create the fresh directory afterwards. It may require some C hacking though, in case Python lacks a kqueue wrapper. I think that this kind of approach would be more sound than a check-wait-loop approach. (I would elaborate more with pointers to appropriate documentation, but I'm on a silly tablet, please excuse me for that.) -gk -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve+comp.lang.python at pearwood.info Tue Jul 30 10:07:22 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 30 Jul 2013 14:07:22 GMT Subject: timing issue: shutil.rmtree and os.makedirs References: Message-ID: <51f7c89a$0$30000$c3e8da3$5496439d@news.astraweb.com> On Tue, 30 Jul 2013 14:27:10 +0100, Chris Angelico wrote: > for delay in 100,300,600,1000,3000,5000,10000: > if not os.path.exists(directory): break > sleep(delay) > > That'll sleep a maximum of 20 seconds, tune as required. Actually, that will sleep a maximum of 5.55 hours, and a minimum of 1.7 minutes (assuming the directory doesn't get deleted instantaneously). time.sleep() takes an argument in seconds. -- Steven From rosuav at gmail.com Tue Jul 30 10:47:46 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 30 Jul 2013 15:47:46 +0100 Subject: timing issue: shutil.rmtree and os.makedirs In-Reply-To: <51f7c89a$0$30000$c3e8da3$5496439d@news.astraweb.com> References: <51f7c89a$0$30000$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, Jul 30, 2013 at 3:07 PM, Steven D'Aprano wrote: > On Tue, 30 Jul 2013 14:27:10 +0100, Chris Angelico wrote: > >> for delay in 100,300,600,1000,3000,5000,10000: >> if not os.path.exists(directory): break >> sleep(delay) >> >> That'll sleep a maximum of 20 seconds, tune as required. > > Actually, that will sleep a maximum of 5.55 hours, and a minimum of 1.7 > minutes (assuming the directory doesn't get deleted instantaneously). > > time.sleep() takes an argument in seconds. LOL! Whoops. That's what I get for not checking my docs. This is why we have public responses, my errors can be caught by someone else. ChrisA From jtim.arnold at gmail.com Tue Jul 30 11:37:22 2013 From: jtim.arnold at gmail.com (Tim) Date: Tue, 30 Jul 2013 08:37:22 -0700 (PDT) Subject: timing issue: shutil.rmtree and os.makedirs In-Reply-To: References: Message-ID: On Tuesday, July 30, 2013 9:27:10 AM UTC-4, Chris Angelico wrote: > On Tue, Jul 30, 2013 at 2:10 PM, Tim wrote: > > hmm, now that you mention it, this is executing on a remote box with access to the same file system my local calling program is on. That is, there is a local call to an intermediate script that connects to a socket on the remote where the above program actually runs, but the file system is the same place for both local and remote. > > > > But even so, since the script that does the rmtree and mkdir is running on the same machine (even though it's remote), I would think the mkdir couldn't execute until the rmtree was completely finished. > > Hmm. What system is used for the file system sharing? I know quite a > few of them lie about whether something's been completely done or not. > Can you use inotify to tell you when the directory's been deleted? > Seems stupid though. > Worst case, all you need is a quick loop at the bottom, eg: > <> > ChrisA Argg, this isn't the first time I've had troubles with the file system. This is FreeBSD and NFS. I will code up a progressive delay as you mentioned (with Steve's correction). thanks much! --Tim From rosuav at gmail.com Tue Jul 30 12:03:32 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 30 Jul 2013 17:03:32 +0100 Subject: timing issue: shutil.rmtree and os.makedirs In-Reply-To: References: Message-ID: On Tue, Jul 30, 2013 at 4:37 PM, Tim wrote: > Argg, this isn't the first time I've had troubles with the file system. This is FreeBSD and NFS. I will code up a progressive delay as you mentioned (with Steve's correction). I've used several different networked file systems, including NetBIOS/NetBEUI/SMB/Samba/etc, sshfs/cifs, and nfs. Not one of them "feels" as clean as I'd like, though sshfs comes closest. There always seem to be hacks around. ChrisA From devyncjohnson at gmail.com Mon Jul 29 15:43:49 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Mon, 29 Jul 2013 15:43:49 -0400 Subject: PEP8 79 char max Message-ID: <51F6C5F5.5020201@Gmail.com> In Python programming, the PEP8 recommends limiting lines to a maximum of 79 characters because "There are still many devices around that are limited to 80 character lines" (http://www.python.org/dev/peps/pep-0008/#code-lay-out). What devices cannot handle 80 or more characters on a line? Would following this recommendation improve script performance? Mahalo, Devyn Collier Johnson DevynCJohnson at Gmail.com From joel.goldstick at gmail.com Mon Jul 29 16:08:22 2013 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Mon, 29 Jul 2013 16:08:22 -0400 Subject: PEP8 79 char max In-Reply-To: <51F6C5F5.5020201@Gmail.com> References: <51F6C5F5.5020201@Gmail.com> Message-ID: On Mon, Jul 29, 2013 at 3:43 PM, Devyn Collier Johnson wrote: > In Python programming, the PEP8 recommends limiting lines to a maximum of 79 > characters because "There are still many devices around that are limited to > 80 character lines" (http://www.python.org/dev/peps/pep-0008/#code-lay-out). > What devices cannot handle 80 or more characters on a line? well, punch cards ;) Would following > this recommendation improve script performance? Not performance, but human readability > > Mahalo, > > Devyn Collier Johnson > DevynCJohnson at Gmail.com > -- > http://mail.python.org/mailman/listinfo/python-list -- Joel Goldstick http://joelgoldstick.com From devyncjohnson at gmail.com Mon Jul 29 16:24:51 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Mon, 29 Jul 2013 16:24:51 -0400 Subject: PEP8 79 char max In-Reply-To: References: <51F6C5F5.5020201@Gmail.com> Message-ID: <51F6CF93.8030709@Gmail.com> On 07/29/2013 04:08 PM, Joel Goldstick wrote: > On Mon, Jul 29, 2013 at 3:43 PM, Devyn Collier Johnson > wrote: >> In Python programming, the PEP8 recommends limiting lines to a maximum of 79 >> characters because "There are still many devices around that are limited to >> 80 character lines" (http://www.python.org/dev/peps/pep-0008/#code-lay-out). >> What devices cannot handle 80 or more characters on a line? > well, punch cards ;) > Would following >> this recommendation improve script performance? > Not performance, but human readability >> Mahalo, >> >> Devyn Collier Johnson >> DevynCJohnson at Gmail.com >> -- >> http://mail.python.org/mailman/listinfo/python-list > > So, I can have a script with large lines and not negatively influence performance on systems that do not use punch cards? DCJ From cs at zip.com.au Tue Jul 30 17:39:42 2013 From: cs at zip.com.au (Cameron Simpson) Date: Wed, 31 Jul 2013 07:39:42 +1000 Subject: PEP8 79 char max In-Reply-To: <51F6CF93.8030709@Gmail.com> References: <51F6CF93.8030709@Gmail.com> Message-ID: <20130730213942.GA40276@cskk.homeip.net> On 29Jul2013 16:24, Devyn Collier Johnson wrote: | So, I can have a script with large lines and not negatively | influence performance on systems that do not use punch cards? Well, running anything will negatively impact the performance of a system for others...o Please think about what CPython actually executes, and then try to figure out for yourself whether the line length or the source code will affect that execution. Cheers, -- Cameron Simpson "How do you know I'm Mad?" asked Alice. "You must be," said the Cat, "or you wouldn't have come here." From ed at leafe.com Mon Jul 29 16:18:59 2013 From: ed at leafe.com (Ed Leafe) Date: Mon, 29 Jul 2013 15:18:59 -0500 Subject: PEP8 79 char max In-Reply-To: References: <51F6C5F5.5020201@Gmail.com> Message-ID: <66F2B508-0EFB-4226-865E-0FEF854C3C41@leafe.com> On Jul 29, 2013, at 3:08 PM, Joel Goldstick wrote: >> Would following >> this recommendation improve script performance? > > Not performance, but human readability IMO, this isn't always the case. There are many lines of code that are broken up to meet the 79 character limit, and as a result become much less readable. -- Ed Leafe From steve+comp.lang.python at pearwood.info Mon Jul 29 17:34:44 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 29 Jul 2013 21:34:44 GMT Subject: PEP8 79 char max References: <51F6C5F5.5020201@Gmail.com> Message-ID: <51f6dff4$0$30000$c3e8da3$5496439d@news.astraweb.com> On Mon, 29 Jul 2013 15:18:59 -0500, Ed Leafe wrote: > On Jul 29, 2013, at 3:08 PM, Joel Goldstick > wrote: >> Not performance, but human readability > > IMO, this isn't always the case. There are many lines of code that are > broken up to meet the 79 character limit, and as a result become much > less readable. Speaking of readability, what's with the indentation of your post? The leading tab plays havoc with my newsreader's word-wrapping. Breaking lines to fit in 79 characters should almost always be perfectly readable, if you break it at natural code units rather than at random places. E.g. I have a code snippet that looks like this: [....whatever...] else: completer = completer.Completer( bindings=(r'"\C-xo": overwrite-mode', r'"\C-xd": dump-functions', ) ) I'm not entirely happy with the placement of the closing brackets, but by breaking the line at the arguments to Completer, and then putting one binding per line, I think it is perfectly readable. And much more readable than (say) this: else: completer = completer.Completer(bindings= (r'"\C-xo": overwrite-mode', r'"\C-xd": dump-functions',)) As far as I can tell, that's pretty much the longest line I have in my personal code base, possibly excepting unit tests with long lists of data. I simply don't write deeply nested classes and functions unless I absolutely need to. -- Steven From joshua at landau.ws Mon Jul 29 18:24:39 2013 From: joshua at landau.ws (Joshua Landau) Date: Mon, 29 Jul 2013 23:24:39 +0100 Subject: PEP8 79 char max In-Reply-To: <51f6dff4$0$30000$c3e8da3$5496439d@news.astraweb.com> References: <51F6C5F5.5020201@Gmail.com> <51f6dff4$0$30000$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 29 July 2013 22:34, Steven D'Aprano wrote: > On Mon, 29 Jul 2013 15:18:59 -0500, Ed Leafe wrote: > > > On Jul 29, 2013, at 3:08 PM, Joel Goldstick > > wrote: > >> Not performance, but human readability > > > > IMO, this isn't always the case. There are many lines of code > that are > > broken up to meet the 79 character limit, and as a result become > much > > less readable. > > Speaking of readability, what's with the indentation of your post? The > leading tab plays havoc with my newsreader's word-wrapping. > > Breaking lines to fit in 79 characters should almost always be perfectly > readable, if you break it at natural code units rather than at random > places. E.g. I have a code snippet that looks like this: > > [....whatever...] > else: > completer = completer.Completer( > bindings=(r'"\C-xo": overwrite-mode', > r'"\C-xd": dump-functions', > ) > ) > > I'm not entirely happy with the placement of the closing brackets, but by > breaking the line at the arguments to Completer, and then putting one > binding per line, I think it is perfectly readable. And much more > readable than (say) this: > > > else: > completer = completer.Completer(bindings= > (r'"\C-xo": overwrite-mode', r'"\C-xd": dump-functions',)) > But less readable to me than: completer = completer.Completer(bindings=[r'"\C-xo": overwrite-mode', r'"\C-xd": dump-functions']) Personal preference. As far as I can tell, that's pretty much the longest line I have in my > personal code base, possibly excepting unit tests with long lists of > data. I simply don't write deeply nested classes and functions unless I > absolutely need to. I'd go for: completer = completer.Completer(bindings=[ r'"\C-xo": overwrite-mode', r'"\C-xd": dump-functions' ]) although possibly drop the "bindings=" if possible. "[]" is less ambiguous a construct than "()"? and the "balance" of the code is better my way if such ephemeral ideas as that count. Anyway, the point I'm trying to make is that *line length is a personal thing*. There are two rules: 1) Stick with what other people on the team are doing, if relevant 2) Don't be stupid The rest is your choice. Some people like 80 character limits, but I've consistently preferred "whatever you think" as a better rule. ? As in there are fewer possible uses, so it's quicker to know what you're using it for -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve+comp.lang.python at pearwood.info Mon Jul 29 17:42:48 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 29 Jul 2013 21:42:48 GMT Subject: PEP8 79 char max References: <51F6C5F5.5020201@Gmail.com> Message-ID: <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> On Mon, 29 Jul 2013 16:24:51 -0400, Devyn Collier Johnson wrote: > So, I can have a script with large lines and not negatively influence > performance on systems that do not use punch cards? You'll negatively influence anyone who has to read, or edit, your code. Very likely including you. But no, there's no meaningful performance difference based on line length. Interpreting the source code is not meaningfully affected by line length, and by the time the code is compiled and then run, line length is irrelevant. -- Steven From devyncjohnson at gmail.com Mon Jul 29 18:30:43 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Mon, 29 Jul 2013 18:30:43 -0400 Subject: PEP8 79 char max In-Reply-To: <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> Message-ID: <51F6ED13.5010508@Gmail.com> On 07/29/2013 05:42 PM, Steven D'Aprano wrote: > On Mon, 29 Jul 2013 16:24:51 -0400, Devyn Collier Johnson wrote: > >> So, I can have a script with large lines and not negatively influence >> performance on systems that do not use punch cards? > You'll negatively influence anyone who has to read, or edit, your code. > Very likely including you. > > But no, there's no meaningful performance difference based on line > length. Interpreting the source code is not meaningfully affected by line > length, and by the time the code is compiled and then run, line length is > irrelevant. > > > Evidently, it is personal preference. I prefer to read computer code like a book (yes, I am a weirdo (^u^)). The only time I exced 79 characters is when I write a set of commands that perform similar tasks. I do not make too many lines over 79 char. Thanks everyone for the comments and feedback. Mahalo, DCJ From ed at leafe.com Mon Jul 29 18:54:04 2013 From: ed at leafe.com (Ed Leafe) Date: Mon, 29 Jul 2013 17:54:04 -0500 Subject: PEP8 79 char max In-Reply-To: <51F6ED13.5010508@Gmail.com> References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F6ED13.5010508@Gmail.com> Message-ID: On Jul 29, 2013, at 5:30 PM, Devyn Collier Johnson wrote: > Evidently, it is personal preference. I prefer to read computer code like a book (yes, I am a weirdo (^u^)). The only time I exced 79 characters is when I write a set of commands that perform similar tasks. I do not make too many lines over 79 char. Thanks everyone for the comments and feedback. I have heard this statement before, and so I'm wondering: do you read books printed in monospaced typefaces, or do they have proportional fonts? I've yet to come across anything meant to be read as literature that was monospaced, because it is much harder to read. I had read about a developer who switched to using proportional fonts for coding, and somewhat skeptically, tried it out. After a day or so it stopped looking strange, and after a week it seemed so much easier to read. I only switched back because I found I lost productivity switching from vim to a graphical text editor. -- Ed Leafe From vito.detullio at gmail.com Tue Jul 30 13:08:45 2013 From: vito.detullio at gmail.com (Vito De Tullio) Date: Tue, 30 Jul 2013 19:08:45 +0200 Subject: PEP8 79 char max References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F6ED13.5010508@Gmail.com> Message-ID: Ed Leafe wrote: > I had read about a developer who switched to using proportional fonts for > coding, and somewhat skeptically, tried it out. After a day or so it > stopped looking strange, and after a week it seemed so much easier to > read. By my (limited) experience with proportional fonts, they can be useful only with something like elastic tabstops[0]. But, as a general rule, I simply found more "squared" to just use a fixed-width font. [0] http://nickgravgaard.com/elastictabstops/ -- ZeD From joshua at landau.ws Tue Jul 30 13:42:08 2013 From: joshua at landau.ws (Joshua Landau) Date: Tue, 30 Jul 2013 18:42:08 +0100 Subject: PEP8 79 char max In-Reply-To: References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F6ED13.5010508@Gmail.com> Message-ID: On 30 July 2013 18:08, Vito De Tullio wrote: > Ed Leafe wrote: > > > I had read about a developer who switched to using proportional fonts for > > coding, and somewhat skeptically, tried it out. After a day or so it > > stopped looking strange, and after a week it seemed so much easier to > > read. > > By my (limited) experience with proportional fonts, they can be useful only > with something like elastic tabstops[0]. But, as a general rule, I simply > found more "squared" to just use a fixed-width font. > Not if you give up on the whole "aligning" thing. > [0] http://nickgravgaard.com/elastictabstops/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vito.detullio at gmail.com Tue Jul 30 14:00:03 2013 From: vito.detullio at gmail.com (Vito De Tullio) Date: Tue, 30 Jul 2013 20:00:03 +0200 Subject: PEP8 79 char max References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F6ED13.5010508@Gmail.com> Message-ID: Joshua Landau wrote: >> By my (limited) experience with proportional fonts, they can be useful >> only with something like elastic tabstops[0]. But, as a general rule, I >> simply found more "squared" to just use a fixed-width font. > Not if you give up on the whole "aligning" thing. and this is one of the reason why I come back to fixed-width -- By ZeD From invalid at invalid.invalid Tue Jul 30 13:52:31 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Tue, 30 Jul 2013 17:52:31 +0000 (UTC) Subject: PEP8 79 char max References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F6ED13.5010508@Gmail.com> Message-ID: On 2013-07-30, Joshua Landau wrote: > On 30 July 2013 18:08, Vito De Tullio wrote: > >> Ed Leafe wrote: >> >> > I had read about a developer who switched to using proportional fonts for >> > coding, and somewhat skeptically, tried it out. After a day or so it >> > stopped looking strange, and after a week it seemed so much easier to >> > read. >> >> By my (limited) experience with proportional fonts, they can be useful only >> with something like elastic tabstops[0]. But, as a general rule, I simply >> found more "squared" to just use a fixed-width font. >> > > Not if you give up on the whole "aligning" thing. You don't think that Python code at a given level should all be aligned? I find it very helpful when a given block of code is visually left-aligned. I also find intializers for tables of data to be much more easily read and maintained if the columns can be aligned. -- Grant Edwards grant.b.edwards Yow! MMM-MM!! So THIS is at BIO-NEBULATION! gmail.com From joshua at landau.ws Wed Jul 31 02:16:52 2013 From: joshua at landau.ws (Joshua Landau) Date: Wed, 31 Jul 2013 07:16:52 +0100 Subject: PEP8 79 char max In-Reply-To: References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F6ED13.5010508@Gmail.com> Message-ID: On 30 July 2013 18:52, Grant Edwards wrote: > On 2013-07-30, Joshua Landau wrote: > > On 30 July 2013 18:08, Vito De Tullio wrote: > > > >> Ed Leafe wrote: > >> > >> > I had read about a developer who switched to using proportional fonts > for > >> > coding, and somewhat skeptically, tried it out. After a day or so it > >> > stopped looking strange, and after a week it seemed so much easier to > >> > read. > >> > >> By my (limited) experience with proportional fonts, they can be useful > only > >> with something like elastic tabstops[0]. But, as a general rule, I > simply > >> found more "squared" to just use a fixed-width font. > >> > > > > Not if you give up on the whole "aligning" thing. > > You don't think that Python code at a given level should all be > aligned? I find it very helpful when a given block of code is > visually left-aligned. > I don't understand what you mean. My coding practices almost never require anything more than the initial indentation to have things line up -- any other form of alignment is in my opinion overrated. Maybe it helps you, but personally I don't like it. As I've been saying, the whole thing is personal preference and proportional fonts for some people, such as I, are fine. Except in that there are no good proportional fonts at 8px :(. To explain, I tend to take the "HTML" form of alignment by wrapping: open stuff stuff stuff close to open stuff stuff stuff close and thus everything important lines up anyway. Extra non-indentation indents are a burden for me and look worse (again, personal preference). I also find intializers for tables of data to be much more easily read > and maintained if the columns can be aligned. > Why do you have tables in your Python code? -------------- next part -------------- An HTML attachment was scrubbed... URL: From python.list at tim.thechases.com Wed Jul 31 06:56:41 2013 From: python.list at tim.thechases.com (Tim Chase) Date: Wed, 31 Jul 2013 05:56:41 -0500 Subject: PEP8 79 char max In-Reply-To: References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F6ED13.5010508@Gmail.com> Message-ID: <20130731055641.02973f98@bigbox.christie.dr> On 2013-07-31 07:16, Joshua Landau wrote: > On 30 July 2013 18:52, Grant Edwards wrote: >> I also find intializers for tables of data to be much more easily >> read and maintained if the columns can be aligned. > > Why do you have tables in your Python code? I've had occasion to write things like: for name, value, description in ( ("cost", 42, "How much it cost"), ("status", 3141, "Status code from ISO-3.14159"), ... ): do_something(name, value) print(description) I interpret Grant's statement as wanting the "table" to look like for name, value, description in ( ("cost", 42, "How much it cost"), ("status", 3141, "Status code from ISO-3.14159"), ... ): do_something(name, value) print(description) which does give some modest readability benefits, but at a creation cost I personally am unwilling to pay. -tkc From neilc at norwich.edu Wed Jul 31 09:02:39 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 31 Jul 2013 13:02:39 GMT Subject: PEP8 79 char max References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F6ED13.5010508@Gmail.com> Message-ID: On 2013-07-31, Tim Chase wrote: > On 2013-07-31 07:16, Joshua Landau wrote: >> On 30 July 2013 18:52, Grant Edwards wrote: >>> I also find intializers for tables of data to be much more easily >>> read and maintained if the columns can be aligned. >> >> Why do you have tables in your Python code? > > I've had occasion to write things like: > > for name, value, description in ( > ("cost", 42, "How much it cost"), > ("status", 3141, "Status code from ISO-3.14159"), > ... > ): > do_something(name, value) > print(description) > > I interpret Grant's statement as wanting the "table" to look like > > for name, value, description in ( > ("cost", 42, "How much it cost"), > ("status", 3141, "Status code from ISO-3.14159"), > ... > ): > do_something(name, value) > print(description) > > which does give some modest readability benefits, but at a > creation cost I personally am unwilling to pay. I'm actually OK with the creation cost, but not the maintenance cost. -- Neil Cerutti From invalid at invalid.invalid Wed Jul 31 12:35:05 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Wed, 31 Jul 2013 16:35:05 +0000 (UTC) Subject: PEP8 79 char max References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F6ED13.5010508@Gmail.com> Message-ID: On 2013-07-31, Neil Cerutti wrote: > On 2013-07-31, Tim Chase wrote: >> On 2013-07-31 07:16, Joshua Landau wrote: >>> On 30 July 2013 18:52, Grant Edwards wrote: >>>> I also find intializers for tables of data to be much more easily >>>> read and maintained if the columns can be aligned. >>> >>> Why do you have tables in your Python code? >> >> I've had occasion to write things like: >> >> for name, value, description in ( >> ("cost", 42, "How much it cost"), >> ("status", 3141, "Status code from ISO-3.14159"), >> ... >> ): >> do_something(name, value) >> print(description) >> >> I interpret Grant's statement as wanting the "table" to look like >> >> for name, value, description in ( >> ("cost", 42, "How much it cost"), >> ("status", 3141, "Status code from ISO-3.14159"), >> ... >> ): >> do_something(name, value) >> print(description) >> >> which does give some modest readability benefits, but at a >> creation cost I personally am unwilling to pay. > > I'm actually OK with the creation cost, but not the maintenance cost. In my experience, aligning columns in large tables reduces maintence cost by making it much easier/faster to see what you've got and by providing a way to visually "prompt" you for the correct value in the correct place when you add new lines. -- Grant Edwards grant.b.edwards Yow! hubub, hubub, HUBUB, at hubub, hubub, hubub, HUBUB, gmail.com hubub, hubub, hubub. From lists.md at gmail.com Wed Jul 31 13:45:17 2013 From: lists.md at gmail.com (Marcelo MD) Date: Wed, 31 Jul 2013 14:45:17 -0300 Subject: PEP8 79 char max In-Reply-To: References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F6ED13.5010508@Gmail.com> Message-ID: > > In my experience, aligning columns in large tables reduces maintence > cost by making it much easier/faster to see what you've got and by > providing a way to visually "prompt" you for the correct value in the > correct place when you add new lines. > > Works great until one of the values changes in size. Say: bla = ( ....1.0, 1.1, 1.2, ....2.0, 2.1, 2.2, ....3.0, 2.1, 3.2, ) And one day you have to change '2.1' to '2.09999': bla = ( ....1.0, 1.1, 1.2, ....2.0, 2.09999, 2.2, ....3.0, 2.1, 3.2, ) If this happens more than once ( or twice, because I'm patient =)), maintaining the alignment becomes a chore. So I only align columns if I'm typing a table I know won't change. -- Marcelo Mallmann Dias -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Wed Jul 31 14:05:04 2013 From: skip at pobox.com (Skip Montanaro) Date: Wed, 31 Jul 2013 13:05:04 -0500 Subject: PEP8 79 char max In-Reply-To: References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F6ED13.5010508@Gmail.com> Message-ID: => Works great until one of the values changes in size. Slightly off-topic, but still sort of related (talking about the size of things), I started picking 1e+30 as my "really big" some time back because the repr of 1e+99 required more than a glance when it appeared in printed output: >>> repr(1e+30) '1e+30' >>> repr(1e+99) '9.9999999999999997e+98' This problem was fixed in 2.7 (and presumably in 3.something as well), but it used to be a problem. :-) Skip From neilc at norwich.edu Wed Jul 31 13:59:34 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 31 Jul 2013 17:59:34 GMT Subject: PEP8 79 char max References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F6ED13.5010508@Gmail.com> Message-ID: On 2013-07-31, Marcelo MD wrote: >> In my experience, aligning columns in large tables reduces >> maintence cost by making it much easier/faster to see what >> you've got and by providing a way to visually "prompt" you for >> the correct value in the correct place when you add new lines. > > Works great until one of the values changes in size. Say: > bla = ( > ....1.0, 1.1, 1.2, > ....2.0, 2.1, 2.2, > ....3.0, 2.1, 3.2, > ) > > And one day you have to change '2.1' to '2.09999': > bla = ( > ....1.0, 1.1, 1.2, > ....2.0, 2.09999, 2.2, > ....3.0, 2.1, 3.2, > ) > > If this happens more than once ( or twice, because I'm patient =)), > maintaining the alignment becomes a chore. So I only align columns if I'm > typing a table I know won't change. Yes, that was my exprience, too. Besides, after studying The Pragmatic Programmer I removed nearly all the tables from my code and reference them (usually with csv module) instead. -- Neil Cerutti From invalid at invalid.invalid Wed Jul 31 14:16:17 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Wed, 31 Jul 2013 18:16:17 +0000 (UTC) Subject: PEP8 79 char max References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F6ED13.5010508@Gmail.com> Message-ID: On 2013-07-31, Neil Cerutti wrote: > Besides, after studying The Pragmatic Programmer I removed nearly > all the tables from my code and reference them (usually with csv > module) instead. I don't understand. That just moves them to a different file -- doesn't it? You've still got to deal with editing a large table of data (for example when I want to add instructions to your assembler). -- Grant Edwards grant.b.edwards Yow! Spreading peanut at butter reminds me of gmail.com opera!! I wonder why? From neilc at norwich.edu Wed Jul 31 14:37:22 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 31 Jul 2013 18:37:22 GMT Subject: PEP8 79 char max References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F6ED13.5010508@Gmail.com> Message-ID: On 2013-07-31, Grant Edwards wrote: > On 2013-07-31, Neil Cerutti wrote: >> Besides, after studying The Pragmatic Programmer I removed >> nearly all the tables from my code and reference them (usually >> with csv module) instead. > > I don't understand. That just moves them to a different file > -- doesn't it? You've still got to deal with editing a large > table of data (for example when I want to add instructions to > your assembler). Yes, but it is much easier to manipulate and view. I often still edit the tables with Vim, but when I just want to view them I can open them with Excel and get a very attractive display or printout with minimal effort. If it turns out I need to convert the table to some new format, tools are abundant. A couple of big wins: It turned out later that some other entity needed the same data. It has allowed me to add functionality to my program without even editing the program. Wouldn't be cool to add a new instruction by to my assembler, including documentation, merely by editing a csv file? (I admit that would need quite a bit of engineering to work, but it would be cool.) -- Neil Cerutti From invalid at invalid.invalid Wed Jul 31 14:56:48 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Wed, 31 Jul 2013 18:56:48 +0000 (UTC) Subject: PEP8 79 char max References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F6ED13.5010508@Gmail.com> Message-ID: On 2013-07-31, Neil Cerutti wrote: > On 2013-07-31, Grant Edwards wrote: >> On 2013-07-31, Neil Cerutti wrote: >>> Besides, after studying The Pragmatic Programmer I removed >>> nearly all the tables from my code and reference them (usually >>> with csv module) instead. >> >> I don't understand. That just moves them to a different file >> -- doesn't it? You've still got to deal with editing a large >> table of data (for example when I want to add instructions to >> your assembler). > > Yes, but it is much easier to manipulate and view. I often still > edit the tables with Vim, but when I just want to view them I can > open them with Excel and get a very attractive display or > printout with minimal effort. If you're good at Excel. I use a spreadsheet at most a few times a year, and it has been many years since I've used Excel. I find that doing _anything_ with Excel generally involves at least an hour of hairpulling and swearing. Libreoffice isn't much better. > If it turns out I need to convert the table to some new format, > tools are abundant. True. > A couple of big wins: > > It turned out later that some other entity needed the same data. It would save the two seconds it takes to extract the lines from the Python file. > It has allowed me to add functionality to my program without even > editing the program. Now you're playing with semantics. If I have a bunch of lines containing values separated by commas, and I'm editting them, then it makes no difference to me which file they're in -- I'm still adding functionality be editing a table of data. > Wouldn't be cool to add a new instruction by to my assembler, > including documentation, merely by editing a csv file? (I admit > that would need quite a bit of engineering to work, but it would > be cool.) Yes, that's cool, and that's pretty much how it works. Those csv lines just happen to be in the same file as the rest of the assembler source rather than in a second file. -- Grant Edwards grant.b.edwards Yow! Hello? Enema Bondage? at I'm calling because I want gmail.com to be happy, I guess ... From neilc at norwich.edu Wed Jul 31 15:25:09 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 31 Jul 2013 19:25:09 GMT Subject: PEP8 79 char max References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F6ED13.5010508@Gmail.com> Message-ID: On 2013-07-31, Grant Edwards wrote: > On 2013-07-31, Neil Cerutti wrote: >> On 2013-07-31, Grant Edwards wrote: >>> On 2013-07-31, Neil Cerutti wrote: >>>> Besides, after studying The Pragmatic Programmer I removed >>>> nearly all the tables from my code and reference them (usually >>>> with csv module) instead. >>> >>> I don't understand. That just moves them to a different file >>> -- doesn't it? You've still got to deal with editing a large >>> table of data (for example when I want to add instructions to >>> your assembler). >> >> Yes, but it is much easier to manipulate and view. I often still >> edit the tables with Vim, but when I just want to view them I can >> open them with Excel and get a very attractive display or >> printout with minimal effort. > > If you're good at Excel. I use a spreadsheet at most a few times a > year, and it has been many years since I've used Excel. I find that > doing _anything_ with Excel generally involves at least an hour of > hairpulling and swearing. Libreoffice isn't much better. > >> If it turns out I need to convert the table to some new format, >> tools are abundant. > > True. > >> A couple of big wins: >> >> It turned out later that some other entity needed the same >> data. > > It would save the two seconds it takes to extract the lines > from the Python file. ...assuming I'm not creating a maintenance problem by duplicating the table. Most likely you would end up externalizing the table at that point, right? >> It has allowed me to add functionality to my program without >> even editing the program. > > Now you're playing with semantics. If I have a bunch of lines > containing values separated by commas, and I'm editting them, > then it makes no difference to me which file they're in -- I'm > still adding functionality be editing a table of data. The separation of data and program is more distinct in my version, but you're right, of course. An internal representation in addition has the advantage of being able to directly use Python identifiers and expressions, too. But if you take advantage of those features when the time comes to externalize your data it's more work. The only cost you pay more than what I've already spent at that point is whatever time you spent creating the non-external version to begin with. -- Neil Cerutti From ramit.prasad at jpmorgan.com.dmarc.invalid Wed Jul 31 14:33:03 2013 From: ramit.prasad at jpmorgan.com.dmarc.invalid (Prasad, Ramit) Date: Wed, 31 Jul 2013 18:33:03 +0000 Subject: PEP8 79 char max In-Reply-To: References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F6ED13.5010508@Gmail.com> Message-ID: <5B80DD153D7D744689F57F4FB69AF47418603B19@SCACMX008.exchad.jpmchase.net> Grant Edwards wrote: > On 2013-07-31, Neil Cerutti wrote: > > > Besides, after studying The Pragmatic Programmer I removed nearly > > all the tables from my code and reference them (usually with csv > > module) instead. > > I don't understand. That just moves them to a different file -- > doesn't it? You've still got to deal with editing a large table of > data (for example when I want to add instructions to your assembler). > > -- > Grant Edwards grant.b.edwards Yow! Spreading peanut > at butter reminds me of > gmail.com opera!! I wonder why? > -- True, but a CSV file is easy to edit in something like a spreadsheet application (LibreOffice/MS Office); alignment becomes automatic then. Ramit This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email. From invalid at invalid.invalid Wed Jul 31 12:32:30 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Wed, 31 Jul 2013 16:32:30 +0000 (UTC) Subject: PEP8 79 char max References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F6ED13.5010508@Gmail.com> Message-ID: On 2013-07-31, Tim Chase wrote: > On 2013-07-31 07:16, Joshua Landau wrote: >> On 30 July 2013 18:52, Grant Edwards wrote: >>> I also find intializers for tables of data to be much more easily >>> read and maintained if the columns can be aligned. >> >> Why do you have tables in your Python code? For example: if you're writing an assembler, you usually have a table of mnemonics/opcodes/instruction-format/addressing-modes. > I've had occasion to write things like: > > for name, value, description in ( > ("cost", 42, "How much it cost"), > ("status", 3141, "Status code from ISO-3.14159"), > ... > ): > do_something(name, value) > print(description) > > I interpret Grant's statement as wanting the "table" to look like > > for name, value, description in ( > ("cost", 42, "How much it cost"), > ("status", 3141, "Status code from ISO-3.14159"), > ... > ): > do_something(name, value) > print(description) Exactly. When you have more than about 5 columns and 10 rows, having things aligned makes it far, far, easier to maintain. > which does give some modest readability benefits, but at a creation > cost I personally am unwilling to pay. It only gets typed once, it gets read hundreds or thousands of times. Optimize the common case. -- Grant Edwards grant.b.edwards Yow! I am NOT a nut.... at gmail.com From python.list at tim.thechases.com Wed Jul 31 14:19:15 2013 From: python.list at tim.thechases.com (Tim Chase) Date: Wed, 31 Jul 2013 13:19:15 -0500 Subject: PEP8 79 char max In-Reply-To: References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F6ED13.5010508@Gmail.com> Message-ID: <20130731131915.27c3ff19@bigbox.christie.dr> On 2013-07-31 16:32, Grant Edwards wrote: > On 2013-07-31, Tim Chase wrote: > > I interpret Grant's statement as wanting the "table" to look like > > > > for name, value, description in ( > > ("cost", 42, "How much it cost"), > > ("status", 3141, "Status code from ISO-3.14159"), > > ... > > ): > > do_something(name, value) > > print(description) > > Exactly. When you have more than about 5 columns and 10 rows, > having things aligned makes it far, far, easier to maintain. As mentioned by Marcelo, when they can vary in length, maintaining is a pain. Even if your editor supports aligning (like vim with Dr. Chip's align.vim plugin). If I have a table of greater dimensions, I tend to refactor into a more readable+maintainable scheme, whether using dicts or named tuples and then break out the rows onto their own lines: from collections import namedtuple Descriptor = namedtuple("Descriptor", ["name", "value", "description"]) for name, value, description in ( Descriptor( name="cost", value=42, description="How much it cost", ), Descriptor( name="status", value=3141, description="Status code from ISO-3.14159", ), ): do_something(name, value) print(description) Using a namedtuple, if you forget one of the fields (or add an extra, or misspell one), it yells at you: TypeError: __new__() takes exactly 4 arguments (2 given) TypeError: __new__() takes exactly 4 arguments (6 given) TypeError: __new__() got an unexpected keyword argument 'nmae' There is redundancy of the kwarg params, but this can be skipped if you prefer DRY code to more readable code. Doing this also has the benefit that, when diffing, you don't get noise when columns are merely adjusted visually. -tkc From python.list at tim.thechases.com Wed Jul 31 14:24:59 2013 From: python.list at tim.thechases.com (Tim Chase) Date: Wed, 31 Jul 2013 13:24:59 -0500 Subject: PEP8 79 char max In-Reply-To: References: <51F6C5F5.5020201@Gmail.com> <51f6e1d8$0$30000$c3e8da3$5496439d@news.astraweb.com> <51F6ED13.5010508@Gmail.com> Message-ID: <20130731132459.2c4c4b46@bigbox.christie.dr> On 2013-07-31 16:32, Grant Edwards wrote: > On 2013-07-31, Tim Chase wrote: > > I interpret Grant's statement as wanting the "table" to look like > > > > for name, value, description in ( > > ("cost", 42, "How much it cost"), > > ("status", 3141, "Status code from ISO-3.14159"), > > ... > > ): > > do_something(name, value) > > print(description) > > Exactly. When you have more than about 5 columns and 10 rows, > having things aligned makes it far, far, easier to maintain. As mentioned by Marcelo, when they can vary in length, maintaining is a pain. Even if your editor supports aligning (like vim with Dr. Chip's align.vim plugin). If I have a table of greater dimensions, I tend to refactor into a more readable+maintainable scheme, whether using dicts or named tuples and then break out the rows onto their own lines: from collections import namedtuple Descriptor = namedtuple("Descriptor", ["name", "value", "description"]) for name, value, description in ( Descriptor( name="cost", value=42, description="How much it cost", ), Descriptor( name="status", value=3141, description="Status code from ISO-3.14159", ), ): do_something(name, value) print(description) Using a namedtuple, if you forget one of the fields (or add an extra, or misspell one), it yells at you: TypeError: __new__() takes exactly 4 arguments (2 given) TypeError: __new__() takes exactly 4 arguments (6 given) TypeError: __new__() got an unexpected keyword argument 'nmae' There is redundancy of the kwarg params, but this can be skipped if you prefer DRY code to more readable code. Doing this also has the benefit that, when diffing, you don't get noise when columns are merely adjusted visually. -tkc From gordon at panix.com Mon Jul 29 16:30:54 2013 From: gordon at panix.com (John Gordon) Date: Mon, 29 Jul 2013 20:30:54 +0000 (UTC) Subject: PEP8 79 char max References: Message-ID: In Devyn Collier Johnson writes: > (http://www.python.org/dev/peps/pep-0008/#code-lay-out). What devices > cannot handle 80 or more characters on a line? For a start, older fixed-width dumb terminals and printers. And even some very new devices (tablet, smartphone) might have limited screen sizes. And even if you're on a device that can display more than 80 characters, it can be convenient to have several windows display side-to-side. > Would following this recommendation improve script performance? No, but it improves human readability. -- John Gordon A is for Amy, who fell down the stairs gordon at panix.com B is for Basil, assaulted by bears -- Edward Gorey, "The Gashlycrumb Tinies" From steve+comp.lang.python at pearwood.info Mon Jul 29 17:09:10 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 29 Jul 2013 21:09:10 GMT Subject: PEP8 79 char max References: Message-ID: <51f6d9f6$0$30000$c3e8da3$5496439d@news.astraweb.com> On Mon, 29 Jul 2013 15:43:49 -0400, Devyn Collier Johnson wrote: > In Python programming, the PEP8 recommends limiting lines to a maximum > of 79 characters because "There are still many devices around that are > limited to 80 character lines" > (http://www.python.org/dev/peps/pep-0008/#code-lay-out). What devices > cannot handle 80 or more characters on a line? The only one I can think of is actual xterms (Ctrl-Alt-Function key terminals on Unix and Linux). But I think that's actually a red-herring. At least for me, I don't care about devices with 80 character lines. (Smart phones? Or is that more likely to be 40 character lines?) I care about being able to put multiple windows side-by-side, or a single window with code in one pane and a class map at the side. I care about being able to copy and paste code into an email, or Usenet post, without it being mangled. I care about *never* having to scroll left-to-right in order to read a line. And most of all, I care about lines being short enough to read without eye strain and mental fatigue from excessive horizontal width. > Would following this > recommendation improve script performance? No, it is irrelevant to performance, except performance of the reader. -- Steven From rhodri at wildebst.demon.co.uk Mon Jul 29 19:08:06 2013 From: rhodri at wildebst.demon.co.uk (Rhodri James) Date: Tue, 30 Jul 2013 00:08:06 +0100 Subject: PEP8 79 char max References: <51f6d9f6$0$30000$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Mon, 29 Jul 2013 22:09:10 +0100, Steven D'Aprano wrote: > On Mon, 29 Jul 2013 15:43:49 -0400, Devyn Collier Johnson wrote: > >> In Python programming, the PEP8 recommends limiting lines to a maximum >> of 79 characters because "There are still many devices around that are >> limited to 80 character lines" >> (http://www.python.org/dev/peps/pep-0008/#code-lay-out). What devices >> cannot handle 80 or more characters on a line? > > The only one I can think of is actual xterms (Ctrl-Alt-Function key > terminals on Unix and Linux). But I think that's actually a red-herring. > At least for me, I don't care about devices with 80 character lines. > (Smart phones? Or is that more likely to be 40 character lines?) > > I care about being able to put multiple windows side-by-side, or a single > window with code in one pane and a class map at the side. I care about > being able to copy and paste code into an email, or Usenet post, without > it being mangled. I care about *never* having to scroll left-to-right in > order to read a line. > > And most of all, I care about lines being short enough to read without > eye strain and mental fatigue from excessive horizontal width. +1 I'm working on some shonky C code at the moment that inconsistent indentation and very long lines. It is extremely annoying not to be able to put the original code, my "translation" and sundry utilities all side-by-side on the same screen (and it's not a particularly small screen), and having to keep flipping between them slows me down dramatically. Long lines have no effect on the speed of the program, but they can have serious effects on the speed of the programmer. -- Rhodri James *-* Wildebeest Herder to the Masses From joshua at landau.ws Mon Jul 29 20:11:18 2013 From: joshua at landau.ws (Joshua Landau) Date: Tue, 30 Jul 2013 01:11:18 +0100 Subject: PEP8 79 char max In-Reply-To: References: <51f6d9f6$0$30000$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 30 July 2013 00:08, Rhodri James wrote: > On Mon, 29 Jul 2013 22:09:10 +0100, Steven D'Aprano < > steve+comp.lang.python@**pearwood.info> > wrote: > > On Mon, 29 Jul 2013 15:43:49 -0400, Devyn Collier Johnson wrote: >> >> In Python programming, the PEP8 recommends limiting lines to a maximum >>> of 79 characters because "There are still many devices around that are >>> limited to 80 character lines" >>> (http://www.python.org/dev/**peps/pep-0008/#code-lay-out). >>> What devices >>> cannot handle 80 or more characters on a line? >>> >> >> The only one I can think of is actual xterms (Ctrl-Alt-Function key >> terminals on Unix and Linux). But I think that's actually a red-herring. >> At least for me, I don't care about devices with 80 character lines. >> (Smart phones? Or is that more likely to be 40 character lines?) >> >> I care about being able to put multiple windows side-by-side, or a single >> window with code in one pane and a class map at the side. I care about >> being able to copy and paste code into an email, or Usenet post, without >> it being mangled. I care about *never* having to scroll left-to-right in >> order to read a line. >> >> And most of all, I care about lines being short enough to read without >> eye strain and mental fatigue from excessive horizontal width. >> > > +1 > > I'm working on some shonky C code at the moment that inconsistent > indentation and very long lines. It is extremely annoying not to be able > to put the original code, my "translation" and sundry utilities all > side-by-side on the same screen (and it's not a particularly small screen), > and having to keep flipping between them slows me down dramatically. Long > lines have no effect on the speed of the program, but they can have serious > effects on the speed of the programmer. > Then just wrap it. This is a very automatable thing for editors. There might even be a clever hard-wrap somewhere. I just tried pyformat -- that works wonders. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhodri at wildebst.demon.co.uk Mon Jul 29 20:41:14 2013 From: rhodri at wildebst.demon.co.uk (Rhodri James) Date: Tue, 30 Jul 2013 01:41:14 +0100 Subject: PEP8 79 char max References: <51f6d9f6$0$30000$c3e8da3$5496439d@news.astraweb.com> Message-ID: On Tue, 30 Jul 2013 01:11:18 +0100, Joshua Landau wrote: > On 30 July 2013 00:08, Rhodri James wrote: >> I'm working on some shonky C code at the moment that inconsistent >> indentation and very long lines. It is extremely annoying not to be >> able to put the original code, my "translation" and sundry utilities >> all side-by-side on the same screen (and it's not a particularly >> small screen), and having to keep flipping between them slows me >> down dramatically. Long lines have no effect on the speed of the >> program, but they can have serious effects on the speed of the >> programmer. > > Then just wrap it. This is a very automatable thing for editors. There > might even be a clever hard-wrap somewhere. I just tried pyformat -- that > works wonders. I tried that at first, but it actually made matters worse. "Simple" word-wrapping just broke the structural cues from indentation (which I'd already had to instruct my editor to make at least somewhat consistent). I couldn't just take in the code layout at a glance, I had to work at it. -- Rhodri James *-* Wildebeest Herder to the Masses From joshua at landau.ws Mon Jul 29 22:12:29 2013 From: joshua at landau.ws (Joshua Landau) Date: Tue, 30 Jul 2013 03:12:29 +0100 Subject: PEP8 79 char max In-Reply-To: References: <51f6d9f6$0$30000$c3e8da3$5496439d@news.astraweb.com> Message-ID: On 30 July 2013 01:41, Rhodri James wrote: > On Tue, 30 Jul 2013 01:11:18 +0100, Joshua Landau > wrote: > > On 30 July 2013 00:08, Rhodri James wrote: >> >>> I'm working on some shonky C code at the moment that inconsistent >>> indentation and very long lines. It is extremely annoying not to be >>> able to put the original code, my "translation" and sundry utilities >>> all side-by-side on the same screen (and it's not a particularly >>> small screen), and having to keep flipping between them slows me >>> down dramatically. Long lines have no effect on the speed of the >>> program, but they can have serious effects on the speed of the >>> programmer. >>> >> >> Then just wrap it. This is a very automatable thing for editors. There >> might even be a clever hard-wrap somewhere. I just tried pyformat -- that >> works wonders. >> > > I tried that at first, but it actually made matters worse. "Simple" > word-wrapping just broke the structural cues from indentation (which I'd > already had to instruct my editor to make at least somewhat consistent). I > couldn't just take in the code layout at a glance, I had to work at it. 1) pyformat's quite sane, maybe you should try that. It's a few minutes wasted at worst. (sudo pip install pyformat; pyformat [-i for inplace changes]) 2) How does your soft word-wrap work? I know some editors do it terribly, mine does it passably?. I don't know any that do it truly semantically (although it's a feature worthy of implementation). 3) Is the code secret? Let's see a "difficult" snippet if not. ? It either indents to the same indentation as the line's start or an extra indent inwards, depending on context -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Tue Jul 30 17:32:34 2013 From: cs at zip.com.au (Cameron Simpson) Date: Wed, 31 Jul 2013 07:32:34 +1000 Subject: PEP8 79 char max In-Reply-To: References: Message-ID: <20130730213234.GA36719@cskk.homeip.net> On 30Jul2013 01:41, Rhodri James wrote: | On Tue, 30 Jul 2013 01:11:18 +0100, Joshua Landau wrote: | >On 30 July 2013 00:08, Rhodri James wrote: | >>I'm working on some shonky C code at the moment that inconsistent | >>indentation and very long lines. [...] Have you tried the indent(1) command? DESCRIPTION indent is a C program formatter. It reformats the C program in the input_file according to the switches. The switches which can be speci? fied are described below. They may appear before or after the file names. Very handy sometimes. Cheers, -- Cameron Simpson The top three answers: Yes I *am* going to a fire! Oh! We're using *kilometers* per hour now. I have to go that fast to get back to my own time. - Peter Harper From llanitedave at veawb.coop Wed Jul 31 03:30:28 2013 From: llanitedave at veawb.coop (llanitedave) Date: Wed, 31 Jul 2013 00:30:28 -0700 (PDT) Subject: PEP8 79 char max In-Reply-To: References: Message-ID: <8daf84dc-34d9-4fe1-9819-f388694b706f@googlegroups.com> It's not just the number of characters, it's the size and the font. Even fixed-width fonts differ greatly in their readability. I can handle different line widths just fine up til about 120 or so without losing the flow of the program, but some fonts simply make it more difficult at any width. I've tried many, but for some reason I keep coming back to Courier10 at 10 points. I'm almost embarrassed that my choice is such an old and primitive font, but that's how my brain works. In my experience, if code is well-spaced, well-commented, and broken up into logical groups with appropriate blank spaces, line length can be about 3/4 the width of whatever editor is being used. And most editors are wide enough to easily accommodate over 100 characters. From tjreedy at udel.edu Mon Jul 29 17:01:27 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 29 Jul 2013 17:01:27 -0400 Subject: PEP8 79 char max In-Reply-To: <51F6C5F5.5020201@Gmail.com> References: <51F6C5F5.5020201@Gmail.com> Message-ID: On 7/29/2013 3:43 PM, Devyn Collier Johnson wrote: > In Python programming, the PEP8 recommends limiting lines to a maximum > of 79 characters because "There are still many devices around that are > limited to 80 character lines" "plus, limiting windows to 80 characters makes it possible to have several windows side-by-side. " > (http://www.python.org/dev/peps/pep-0008/#code-lay-out). What devices > cannot handle 80 or more characters on a line? PEP 8 is being revised. My understanding is that because such machines are now rare, while the second reason is still operative, the recommended limit will be raised to 100, or at least say that some are using that as the guideline. 200 char lines are harder to read. -- Terry Jan Reedy From skip at pobox.com Mon Jul 29 17:18:45 2013 From: skip at pobox.com (Skip Montanaro) Date: Mon, 29 Jul 2013 16:18:45 -0500 Subject: PEP8 79 char max In-Reply-To: References: <51F6C5F5.5020201@Gmail.com> Message-ID: For the purposes of limiting the length you need to scan between first and last column, I would recommend leaving the recommended line length to ~ 80 columns. Just for grins, I grabbed a non-computer book, Atul Gawande's "Checklist Manifesto," from the pile on my desk and counted the number of characters in a full-width line. 70. Then I grabbed my copy of "Mastering Regular Expressions" and counted the number of characters in a full-width line of text which also included a few special characters. 80. I think the history of printing offers a good gauge for the useful limits to line length. After all, print publishers have been at this for more than a few years. As I typed this, Steven D'Aprano wrote: > No, it is irrelevant to performance, except performance of the reader. Whose performance, I would argue is most important. I would like to hear of books meant to be read with page or column widths of 100 or more characters. I suspect they would be few and far between. Skip From joshua at landau.ws Mon Jul 29 18:07:42 2013 From: joshua at landau.ws (Joshua Landau) Date: Mon, 29 Jul 2013 23:07:42 +0100 Subject: PEP8 79 char max In-Reply-To: References: <51F6C5F5.5020201@Gmail.com> Message-ID: On 29 July 2013 22:18, Skip Montanaro wrote: > For the purposes of limiting the length you need to scan between first > and last column, I would recommend leaving the recommended line length > to ~ 80 columns. > > Just for grins, I grabbed a non-computer book, Atul Gawande's > "Checklist Manifesto," from the pile on my desk and counted the number > of characters in a full-width line. 70. Then I grabbed my copy of > "Mastering Regular Expressions" and counted the number of characters > in a full-width line of text which also included a few special > characters. 80. > > I think the history of printing offers a good gauge for the useful > limits to line length. After all, print publishers have been at this > for more than a few years. > > As I typed this, Steven D'Aprano wrote: > > > No, it is irrelevant to performance, except performance of the reader. > > Whose performance, I would argue is most important. > > I would like to hear of books meant to be read with page or column > widths of 100 or more characters. I suspect they would be few and far > between. > In that gauge I would exclude indentation (you don't count the number of characters the margin takes) and would point out that programming doesn't have generically-wrapped lines -- sometimes the wrapping of a line is more distracting than its length. Written text has a completely different flow to programs; written text is read sequentially where missing the reading of words is a trivial and oft occurrence whereas programming is highly structured. -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Tue Jul 30 10:06:53 2013 From: skip at pobox.com (Skip Montanaro) Date: Tue, 30 Jul 2013 09:06:53 -0500 Subject: PEP8 79 char max In-Reply-To: References: <51F6C5F5.5020201@Gmail.com> Message-ID: > In that gauge I would exclude indentation (you don't count the > number of characters the margin takes) .... I don't think anyone reads the margins. :-) That said, I agree that code and prose are fundamentally different beasts. Still, when reading either and you get to the end of the line, you need to shift your gaze down a line and back to the left margin (or the left margin plus any indentation). That task becomes more difficult as line length increases. As programmers/software engineers, we need to read and write both code and text. I think 80 columns is still a decent compromise. Skip From neilc at norwich.edu Tue Jul 30 10:49:06 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 30 Jul 2013 14:49:06 GMT Subject: PEP8 79 char max References: <51F6C5F5.5020201@Gmail.com> Message-ID: On 2013-07-30, Skip Montanaro wrote: >> In that gauge I would exclude indentation (you don't count the >> number of characters the margin takes) .... > > I don't think anyone reads the margins. :-) > > That said, I agree that code and prose are fundamentally > different beasts. Still, when reading either and you get to > the end of the line, you need to shift your gaze down a line > and back to the left margin (or the left margin plus any > indentation). That task becomes more difficult as line length > increases. Most research about speed of comprehension of different line lengths was based on subjects reading prose. The effect of code line length hasn't been studied extensively. > As programmers/software engineers, we need to read and write > both code and text. I think 80 columns is still a decent > compromise. So rules of thumb, standardizations, and personal preferences are mostly what we have to go by. When code that looks very similar to code you've seen before really *is* similar to code you've seen before, comprehension speed can increase. A study of chess masters' ability to memorize chess positions showed that they were powerfully accurate when shown positions from real games, but no better than the average schmoe when shown randomly positioned pieces. So if everyone basically follows PEP8 we all benefit from playing by the same game rules, as it were. -- Neil Cerutti From skip at pobox.com Tue Jul 30 11:44:12 2013 From: skip at pobox.com (Skip Montanaro) Date: Tue, 30 Jul 2013 10:44:12 -0500 Subject: PEP8 79 char max In-Reply-To: References: <51F6C5F5.5020201@Gmail.com> Message-ID: > So if everyone basically follows PEP8 we all benefit from playing by > the same game rules, as it were. (I think I'm agreeing with you, but nonetheless, I will forge ahead.) To the extent that 80-column window widths have been common for so long, PEP 8 or not (and Python or not), there is a ton of code out there which abides by that convention. More-or-less unilaterally increasing the recommended max line width to 100 (or 99?) columns isn't likely to improve things. People like me (who prefer the status quo) will complain about all the new-fangled code written to a wider standard (and will be tempted to reformat). People who like the new standard will complain about old code wasting all that white space (and will be tempted to reformat). :-) Finally (I promise this is my last word on the topic), most lines don't need to be wrapped as they stand today. See the attached graph for the distribution of line lengths for the current project where I spend most of my time these days (just Python code, blank lines elided, comment lines included). Stretching the max out to 100 columns when most lines are less than 60 columns just wastes screen real estate. Skip -------------- next part -------------- A non-text attachment was scrubbed... Name: square.png Type: image/png Size: 21230 bytes Desc: not available URL: From joshua at landau.ws Tue Jul 30 12:41:22 2013 From: joshua at landau.ws (Joshua Landau) Date: Tue, 30 Jul 2013 17:41:22 +0100 Subject: PEP8 79 char max In-Reply-To: References: <51F6C5F5.5020201@Gmail.com> Message-ID: On 30 July 2013 16:44, Skip Montanaro wrote: > > So if everyone basically follows PEP8 we all benefit from playing by > > the same game rules, as it were. > > (I think I'm agreeing with you, but nonetheless, I will forge ahead.) > > To the extent that 80-column window widths have been common for so > long, PEP 8 or not (and Python or not), there is a ton of code out > there which abides by that convention. More-or-less unilaterally > increasing the recommended max line width to 100 (or 99?) columns > isn't likely to improve things. People like me (who prefer the status > quo) will complain about all the new-fangled code written to a wider > standard (and will be tempted to reformat). People who like the new > standard will complain about old code wasting all that white space > (and will be tempted to reformat). :-) > > Finally (I promise this is my last word on the topic), most lines > don't need to be wrapped as they stand today. See the attached graph > for the distribution of line lengths for the current project where I > spend most of my time these days (just Python code, blank lines > elided, comment lines included). Stretching the max out to 100 > columns when most lines are less than 60 columns just wastes screen > real estate. > Your graph doesn't convince me. Take this line from earlier, which is currently 102 characters and nearing my personal limit. completer = completer.Completer(bindings=[r'"\C-xo": overwrite-mode', r'"\C-xd": dump-functions']) Under rules to wrap to 80 characters (and in this case I'd probably do it anyway), I'd normally wrap to this: completer = completer.Completer(bindings=[ r'"\C-xo": overwrite-mode', r'"\C-xd": dump-functions' ]) of line lengths 46, 36, 35 and 6 respectively. Thus it's impossible to so easily tell how many lines would be "unwrapped" from your graph. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon Jul 29 17:55:37 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 29 Jul 2013 17:55:37 -0400 Subject: PEP8 79 char max In-Reply-To: References: <51F6C5F5.5020201@Gmail.com> Message-ID: On 7/29/2013 5:01 PM, Terry Reedy wrote: > On 7/29/2013 3:43 PM, Devyn Collier Johnson wrote: >> In Python programming, the PEP8 recommends limiting lines to a maximum >> of 79 characters because "There are still many devices around that are >> limited to 80 character lines" > > "plus, limiting windows to 80 characters makes it possible to have > several windows side-by-side. " > >> (http://www.python.org/dev/peps/pep-0008/#code-lay-out). What devices >> cannot handle 80 or more characters on a line? > > PEP 8 is being revised. My understanding is that because such machines > are now rare, while the second reason is still operative, the > recommended limit will be raised to 100, or maybe not, as there is disagreement on the issue also ;-). or at least say that some are > using that as the guideline. 200 char lines are harder to read. > -- Terry Jan Reedy From devyncjohnson at gmail.com Mon Jul 29 15:48:48 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Mon, 29 Jul 2013 15:48:48 -0400 Subject: import syntax Message-ID: <51F6C720.2020404@Gmail.com> The PEP8 recommends importing like this: import os import re not like this: import os, re Why is that? Is there a performance advantage to one of the styles? Mahalo, Devyn Collier Johnson DevynCJohnson at Gmail.com From joel.goldstick at gmail.com Mon Jul 29 16:05:34 2013 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Mon, 29 Jul 2013 16:05:34 -0400 Subject: import syntax In-Reply-To: <51F6C720.2020404@Gmail.com> References: <51F6C720.2020404@Gmail.com> Message-ID: On Mon, Jul 29, 2013 at 3:48 PM, Devyn Collier Johnson wrote: > The PEP8 recommends importing like this: > > import os > import re > > not like this: > > import os, re > > Why is that? Is there a performance advantage to one of the styles? > > > Mahalo, > > Devyn Collier Johnson > DevynCJohnson at Gmail.com > -- > http://mail.python.org/mailman/listinfo/python-list I think its just for clarity -- Joel Goldstick http://joelgoldstick.com From davea at davea.name Mon Jul 29 16:09:18 2013 From: davea at davea.name (Dave Angel) Date: Mon, 29 Jul 2013 16:09:18 -0400 Subject: import syntax In-Reply-To: <51F6C720.2020404@Gmail.com> References: <51F6C720.2020404@Gmail.com> Message-ID: On 07/29/2013 03:48 PM, Devyn Collier Johnson wrote: > The PEP8 recommends importing like this: > > import os > import re > > not like this: > > import os, re > > Why is that? Is there a performance advantage to one of the styles? > > Pep 8 is not about performance, it's about readability. And unless the two libraries are closely related, it's clearer to use separate lines for them. I got a bit further, and if I'm only using a couple of functions from the import, I'll list them in the comment. -- DaveA From python.list at tim.thechases.com Mon Jul 29 16:20:51 2013 From: python.list at tim.thechases.com (Tim Chase) Date: Mon, 29 Jul 2013 15:20:51 -0500 Subject: import syntax In-Reply-To: References: <51F6C720.2020404@Gmail.com> Message-ID: <20130729152051.46003bcf@bigbox.christie.dr> On 2013-07-29 16:09, Dave Angel wrote: > On 07/29/2013 03:48 PM, Devyn Collier Johnson wrote: > > The PEP8 recommends importing like this: > > > > import os > > import re > > > > not like this: > > > > import os, re > > I got a bit further, and if I'm only using a couple of functions > from the import, I'll list them in the comment. If I just plan to use a small subset, I tend to reach for the from sys import stdout, stderr, exit sort of syntax. I find it makes my code read a bit more cleanly than having to type "sys.stderr.write(...)" everywhere but is still pretty readable. -tkc From devyncjohnson at gmail.com Mon Jul 29 16:23:18 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Mon, 29 Jul 2013 16:23:18 -0400 Subject: import syntax In-Reply-To: <20130729152051.46003bcf@bigbox.christie.dr> References: <51F6C720.2020404@Gmail.com> <20130729152051.46003bcf@bigbox.christie.dr> Message-ID: <51F6CF36.7080702@Gmail.com> On 07/29/2013 04:20 PM, Tim Chase wrote: > On 2013-07-29 16:09, Dave Angel wrote: >> On 07/29/2013 03:48 PM, Devyn Collier Johnson wrote: >>> The PEP8 recommends importing like this: >>> >>> import os >>> import re >>> >>> not like this: >>> >>> import os, re >> I got a bit further, and if I'm only using a couple of functions >> from the import, I'll list them in the comment. > If I just plan to use a small subset, I tend to reach for the > > from sys import stdout, stderr, exit > > sort of syntax. I find it makes my code read a bit more cleanly than > having to type "sys.stderr.write(...)" everywhere but is still pretty > readable. > > -tkc > > So, there are no advantages or disadvantages when disregarding readability? DCJ From joshua at landau.ws Mon Jul 29 18:37:37 2013 From: joshua at landau.ws (Joshua Landau) Date: Mon, 29 Jul 2013 23:37:37 +0100 Subject: import syntax In-Reply-To: <51F6CF36.7080702@Gmail.com> References: <51F6C720.2020404@Gmail.com> <20130729152051.46003bcf@bigbox.christie.dr> <51F6CF36.7080702@Gmail.com> Message-ID: On 29 July 2013 21:23, Devyn Collier Johnson wrote: > > On 07/29/2013 04:20 PM, Tim Chase wrote: > >> On 2013-07-29 16:09, Dave Angel wrote: >> >>> On 07/29/2013 03:48 PM, Devyn Collier Johnson wrote: >>> >>>> The PEP8 recommends importing like this: >>>> >>>> import os >>>> import re >>>> >>>> not like this: >>>> >>>> import os, re >>>> >>> I got a bit further, and if I'm only using a couple of functions >>> from the import, I'll list them in the comment. >>> >> If I just plan to use a small subset, I tend to reach for the >> >> from sys import stdout, stderr, exit >> >> sort of syntax. I find it makes my code read a bit more cleanly than >> having to type "sys.stderr.write(...)" everywhere but is still pretty >> readable. >> >> -tkc >> >> >> So, there are no advantages or disadvantages when disregarding > readability? Sure, just as one light is no brighter or dimmer than another when disregarding luminosity. As people have said, it improves diffs as well. It flows quicker into the "from module import things" form (which I oft prefer), too. When asking these questions, ask yourself "why would it *compile* differently? It wouldn't. Plus, premature optimisation is the root of all evil. 1) Write your code 2) If it's slow: 2a) Do you have time? If so: 2b) Is it important to speed up, or is the slowness not worth spending the hours fixing? 2c) Profile it to see what's actually slow 2d) Realise that the slow part is not what you thought it was 2e) Fix the bit that's slow (and nothing else) 2f) Repeat from 2 3) Write some more code -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Jul 29 18:42:27 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 29 Jul 2013 23:42:27 +0100 Subject: import syntax In-Reply-To: References: <51F6C720.2020404@Gmail.com> <20130729152051.46003bcf@bigbox.christie.dr> <51F6CF36.7080702@Gmail.com> Message-ID: On Mon, Jul 29, 2013 at 11:37 PM, Joshua Landau wrote: > 2d) Realise that the slow part is not what you thought it was This step is mandatory. ChrisA From devyncjohnson at gmail.com Mon Jul 29 19:24:01 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Mon, 29 Jul 2013 19:24:01 -0400 Subject: import syntax In-Reply-To: References: <51F6C720.2020404@Gmail.com> <20130729152051.46003bcf@bigbox.christie.dr> <51F6CF36.7080702@Gmail.com> Message-ID: <51F6F991.3050909@Gmail.com> On 07/29/2013 06:37 PM, Joshua Landau wrote: > On 29 July 2013 21:23, Devyn Collier Johnson > wrote: > > > On 07/29/2013 04:20 PM, Tim Chase wrote: > > On 2013-07-29 16:09, Dave Angel wrote: > > On 07/29/2013 03:48 PM, Devyn Collier Johnson wrote: > > The PEP8 recommends importing like this: > > import os > import re > > not like this: > > import os, re > > I got a bit further, and if I'm only using a couple of > functions > from the import, I'll list them in the comment. > > If I just plan to use a small subset, I tend to reach for the > > from sys import stdout, stderr, exit > > sort of syntax. I find it makes my code read a bit more > cleanly than > having to type "sys.stderr.write(...)" everywhere but is still > pretty > readable. > > -tkc > > > So, there are no advantages or disadvantages when disregarding > readability? > > > Sure, just as one light is no brighter or dimmer than another when > disregarding luminosity. > > As people have said, it improves diffs as well. It flows quicker into > the "from module import things" form (which I oft prefer), too. > > When asking these questions, ask yourself "why would it *compile* > differently? It wouldn't. Plus, premature optimisation is the root of > all evil. > > 1) Write your code > 2) If it's slow: > 2a) Do you have time? If so: > 2b) Is it important to speed up, or is the slowness not worth spending > the hours fixing? > 2c) Profile it to see what's actually slow > 2d) Realise that the slow part is not what you thought it was > 2e) Fix the bit that's slow (and nothing else) > 2f) Repeat from 2 > 3) Write some more code Joshua, nice work-flow instructions. Mahalo, DCJ -------------- next part -------------- An HTML attachment was scrubbed... URL: From neilc at norwich.edu Tue Jul 30 09:03:32 2013 From: neilc at norwich.edu (Neil Cerutti) Date: 30 Jul 2013 13:03:32 GMT Subject: import syntax References: <51F6C720.2020404@Gmail.com> <20130729152051.46003bcf@bigbox.christie.dr> <51F6CF36.7080702@Gmail.com> Message-ID: On 2013-07-29, Joshua Landau wrote: > Sure, just as one light is no brighter or dimmer than another > when disregarding luminosity. > > As people have said, it improves diffs as well. It flows > quicker into the "from module import things" form (which I oft > prefer), too. > > When asking these questions, ask yourself "why would it > *compile* differently? It wouldn't. Plus, premature > optimisation is the root of all evil. > > 1) Write your code > 2) If it's slow: > 2a) Do you have time? If so: > 2b) Is it important to speed up, or is the slowness not worth spending the > hours fixing? > 2c) Profile it to see what's actually slow > 2d) Realise that the slow part is not what you thought it was > 2e) Fix the bit that's slow (and nothing else) > 2f) Repeat from 2 > 3) Write some more code 1a) Does it work? 1b) Can you prove it? It's best to at least have some regression tests before you start refactoring and optimizing. -- Neil Cerutti From python.list at tim.thechases.com Mon Jul 29 16:14:45 2013 From: python.list at tim.thechases.com (Tim Chase) Date: Mon, 29 Jul 2013 15:14:45 -0500 Subject: import syntax In-Reply-To: <51F6C720.2020404@Gmail.com> References: <51F6C720.2020404@Gmail.com> Message-ID: <20130729151445.784e975b@bigbox.christie.dr> On 2013-07-29 15:48, Devyn Collier Johnson wrote: > The PEP8 recommends importing like this: > > import os > import re > > not like this: > > import os, re > > Why is that? Is there a performance advantage to one of the styles? While I don't believe there's much of a performance difference (if so, it should be pretty negligible, especially since imports are usually just done at load-time rather than run-time), but 1) it's easier to read one-per-line (IMHO), particularly if you sort them alphabetically 2) it makes for cleaner diffs Which would you rather read and try to figure out what's going on? ===================================== --- before.py 2013-07-29 15:04:38.250996094 -0500 +++ after.py 2013-07-29 15:04:44.026996132 -0500 @@ -1 +1 @@ -import csv, re, sys, os +import csv, re, sys, glob, os ===================================== vs. ===================================== --- before.py 2013-07-29 15:13:13.050997907 -0500 +++ after.py 2013-07-29 15:13:18.434997950 -0500 @@ -1,4 +1,5 @@ import csv +import glob import os import re import sys ===================================== The latter makes it much easier (at least for me) to spot that "glob" was added. And it's more tolerant of merge-conflict resolution. -tkc From devyncjohnson at gmail.com Mon Jul 29 16:56:06 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Mon, 29 Jul 2013 16:56:06 -0400 Subject: Has anyone gotten Pyglet to work Message-ID: <51F6D6E6.2070004@Gmail.com> I tried Pyglet in a Python3 and a Python2 script, but both fail. The error code is below and the script is attached. The 'boot.ogg' file is Ubuntu's default bootup sound. I got my code from this link (http://guzalexander.com/2012/08/17/playing-a-sound-with-python.html). collier at Nacho-Laptop:~$ ./pyglet.py Traceback (most recent call last): File "./pyglet.py", line 2, in import pyglet File "/home/collier/pyglet.py", line 3, in song = pyglet.media.load('./boot.ogg') AttributeError: 'module' object has no attribute 'media' Mahalo, DCJ -------------- next part -------------- A non-text attachment was scrubbed... Name: pyglet.py Type: text/x-python Size: 104 bytes Desc: not available URL: From gary.herron at islandtraining.com Mon Jul 29 17:08:33 2013 From: gary.herron at islandtraining.com (Gary Herron) Date: Mon, 29 Jul 2013 14:08:33 -0700 Subject: Has anyone gotten Pyglet to work In-Reply-To: <51F6D6E6.2070004@Gmail.com> References: <51F6D6E6.2070004@Gmail.com> Message-ID: <51F6D9D1.7080709@islandtraining.com> On 07/29/2013 01:56 PM, Devyn Collier Johnson wrote: > I tried Pyglet in a Python3 and a Python2 script, but both fail. The > error code is below and the script is attached. The 'boot.ogg' file is > Ubuntu's default bootup sound. I got my code from this link > (http://guzalexander.com/2012/08/17/playing-a-sound-with-python.html). > > collier at Nacho-Laptop:~$ ./pyglet.py > Traceback (most recent call last): > File "./pyglet.py", line 2, in > import pyglet > File "/home/collier/pyglet.py", line 3, in > song = pyglet.media.load('./boot.ogg') > AttributeError: 'module' object has no attribute 'media' > > > Mahalo, > > DCJ > > You appear to have confused Python by having a module named pyglet AND a local file named pyglet.py. This when you say import pyglet, you are not getting the pyglet module, but instead your own file pyglet.py, which of course, has nothing named media in it. Rename your file and try again. P.S. It is a common newbie error to hide a system file like this and suffer the consequence. We've all done it -- at least once. :^) ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From devyncjohnson at gmail.com Mon Jul 29 18:22:02 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Mon, 29 Jul 2013 18:22:02 -0400 Subject: Has anyone gotten Pyglet to work In-Reply-To: <51F6D9D1.7080709@islandtraining.com> References: <51F6D6E6.2070004@Gmail.com> <51F6D9D1.7080709@islandtraining.com> Message-ID: <51F6EB0A.7080104@Gmail.com> On 07/29/2013 05:08 PM, Gary Herron wrote: > On 07/29/2013 01:56 PM, Devyn Collier Johnson wrote: >> I tried Pyglet in a Python3 and a Python2 script, but both fail. The >> error code is below and the script is attached. The 'boot.ogg' file >> is Ubuntu's default bootup sound. I got my code from this link >> (http://guzalexander.com/2012/08/17/playing-a-sound-with-python.html). >> >> collier at Nacho-Laptop:~$ ./pyglet.py >> Traceback (most recent call last): >> File "./pyglet.py", line 2, in >> import pyglet >> File "/home/collier/pyglet.py", line 3, in >> song = pyglet.media.load('./boot.ogg') >> AttributeError: 'module' object has no attribute 'media' >> >> >> Mahalo, >> >> DCJ >> >> > > You appear to have confused Python by having a module named pyglet AND > a local file named pyglet.py. > > This when you say import pyglet, you are not getting the pyglet > module, but instead your own file pyglet.py, which of course, has > nothing named media in it. > > Rename your file and try again. > > P.S. It is a common newbie error to hide a system file like this and > suffer the consequence. We've all done it -- at least once. :^) ) > > > Duh, thanks for the tip (^u^), but I still get an error (different error). NOTE: this is all python2.7 code because Pyglet supposedly has issues with Python3. collier at Nacho-Laptop:~$ pip install pyglet Downloading/unpacking pyglet Downloading pyglet-1.1.4.tar.gz (2.9MB): 2.9MB downloaded Running setup.py egg_info for package pyglet .......Blah........ .......Blah........ Installing collected packages: pyglet Running setup.py install for pyglet Successfully installed pyglet Cleaning up... collier at Nacho-Laptop:~/pytest$ ./pymedia.py Traceback (most recent call last): File "./pymedia.py", line 3, in song = pyglet.media.load('./boot.ogg') File "/usr/local/lib/python2.7/dist-packages/pyglet/media/__init__.py", line 1386, in load source = _source_class(filename, file) File "/usr/local/lib/python2.7/dist-packages/pyglet/media/riff.py", line 202, in __init__ 'AVbin is required to decode compressed media') pyglet.media.riff.WAVEFormatException: AVbin is required to decode compressed media AL lib: ReleaseALC: 1 device not closed collier at Nacho-Laptop:~/pytest$ ls boot.ogg pymedia.py Mahalo, DCJ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pyglet.py Type: text/x-python Size: 104 bytes Desc: not available URL: From joshua at landau.ws Mon Jul 29 18:46:26 2013 From: joshua at landau.ws (Joshua Landau) Date: Mon, 29 Jul 2013 23:46:26 +0100 Subject: Has anyone gotten Pyglet to work In-Reply-To: <51F6EB0A.7080104@Gmail.com> References: <51F6D6E6.2070004@Gmail.com> <51F6D9D1.7080709@islandtraining.com> <51F6EB0A.7080104@Gmail.com> Message-ID: On 29 July 2013 23:22, Devyn Collier Johnson wrote: > Duh, thanks for the tip (^u^), but I still get an error (different error). > NOTE: this is all python2.7 code because Pyglet supposedly has issues with > Python3. > > collier at Nacho-Laptop:~$ pip install pyglet > Downloading/unpacking pyglet > Downloading pyglet-1.1.4.tar.gz (2.9MB): 2.9MB downloaded > Running setup.py egg_info for package pyglet > .......Blah........ > .......Blah........ > Installing collected packages: pyglet > Running setup.py install for pyglet > Successfully installed pyglet > Cleaning up... > collier at Nacho-Laptop:~/pytest$ ./pymedia.py > > Traceback (most recent call last): > File "./pymedia.py", line 3, in > > song = pyglet.media.load('./boot.ogg') > File "/usr/local/lib/python2.7/dist-packages/pyglet/media/__init__.py", > line 1386, in load > source = _source_class(filename, file) > File "/usr/local/lib/python2.7/dist-packages/pyglet/media/riff.py", line > 202, in __init__ > 'AVbin is required to decode compressed media') > pyglet.media.riff.WAVEFormatException: AVbin is required to decode > compressed media > AL lib: ReleaseALC: 1 device not closed > collier at Nacho-Laptop:~/pytest$ ls > boot.ogg pymedia.py > This line: > pyglet.media.riff.WAVEFormatException: AVbin is required to decode compressed media You should read your exceptions very carefully, they're normally there for a very good reason. Then Google that line. Then get http://stackoverflow.com/questions/10302873/python-pyglet-avbin-how-to-install-avbin (first link for me). As someone who knows nothing about this, that's your best bet. If not, Google's other links looked promising. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve+comp.lang.python at pearwood.info Mon Jul 29 18:40:10 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 29 Jul 2013 22:40:10 GMT Subject: Has anyone gotten Pyglet to work References: <51F6D6E6.2070004@Gmail.com> <51F6D9D1.7080709@islandtraining.com> Message-ID: <51f6ef4a$0$30000$c3e8da3$5496439d@news.astraweb.com> On Mon, 29 Jul 2013 18:22:02 -0400, Devyn Collier Johnson wrote: > Duh, thanks for the tip (^u^), but I still get an error (different > error). NOTE: this is all python2.7 code because Pyglet supposedly has > issues with Python3. [...] > Traceback (most recent call last): > File "./pymedia.py", line 3, in > song = pyglet.media.load('./boot.ogg') > File > "/usr/local/lib/python2.7/dist-packages/pyglet/media/__init__.py", line > 1386, in load > source = _source_class(filename, file) > File "/usr/local/lib/python2.7/dist-packages/pyglet/media/riff.py", > line 202, in __init__ > 'AVbin is required to decode compressed media') > pyglet.media.riff.WAVEFormatException: AVbin is required to decode > compressed media You have a media file (boot.ogg) which pyglet is trying to play using AVbin, but you don't have AVbin. As far as I know, pip is incapable of handling non-Python dependencies. I suggest you try your system's package manager: Debian/Ubuntu: sudo aptitude install AVbin -or- sudo apt-get install AVbin RedHat/Fedora/Centos: sudo yum install AVbin and hopefully your distro will already support the latest, or at least working, version. -- Steven From devyncjohnson at gmail.com Mon Jul 29 19:42:59 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Mon, 29 Jul 2013 19:42:59 -0400 Subject: Has anyone gotten Pyglet to work In-Reply-To: <51F6D9D1.7080709@islandtraining.com> References: <51F6D6E6.2070004@Gmail.com> <51F6D9D1.7080709@islandtraining.com> Message-ID: <51F6FE03.9090904@Gmail.com> Thanks everyone for your help. I installed AVbin v10. I no longer get Python errors, but no sound is produced. This is now an AVbin problem that I must figure out. At least I am done with the Python issues. For those of you that need AVbin: http://avbin.github.io/AVbin/Home/Home.html Mahalo, DCJ From missive at hotmail.com Mon Jul 29 17:34:56 2013 From: missive at hotmail.com (Lee Harr) Date: Tue, 30 Jul 2013 02:04:56 +0430 Subject: Has anyone gotten Pyglet to work Message-ID: > $ ./pyglet.py > Traceback (most recent call last): > ? ?File "./pyglet.py", line 2, in > ? ? ?import pyglet > ? ?File "/home/collier/pyglet.py", line 3, in > ? ? ?song = pyglet.media.load('./boot.ogg') > AttributeError: 'module' object has no attribute 'media' Name your program something other than "pyglet.py" import pyglet is importing your module, instead of the pyglet module. From steve+comp.lang.python at pearwood.info Mon Jul 29 17:41:03 2013 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: 29 Jul 2013 21:41:03 GMT Subject: Has anyone gotten Pyglet to work References: Message-ID: <51f6e16f$0$30000$c3e8da3$5496439d@news.astraweb.com> On Mon, 29 Jul 2013 16:56:06 -0400, Devyn Collier Johnson wrote: > collier at Nacho-Laptop:~$ ./pyglet.py Here you are running a module called pyglet, which I assume you wrote. > Traceback (most recent call last): > File "./pyglet.py", line 2, in > import pyglet And here you try to import the third-party module called pyglet, except it is shadowed by your module, and you get your own module instead. > File "/home/collier/pyglet.py", line 3, in > song = pyglet.media.load('./boot.ogg') > AttributeError: 'module' object has no attribute 'media' Since your module has no attribute 'media', that error is correct. The lesson here is, never name your own files the same as library files. Unfortunately, that's easier said than done. I think everyone has made the same mistake at least once in their life as a Python programmer. -- Steven From devyncjohnson at gmail.com Mon Jul 29 17:25:14 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Mon, 29 Jul 2013 17:25:14 -0400 Subject: Bitwise Operations Message-ID: <51F6DDBA.7030208@Gmail.com> On Python3, how can I perform bitwise operations? For instance, I want something that will 'and', 'or', and 'xor' a binary integer. Mahalo, DevynCJohnson at Gmail.com From invalid at invalid.invalid Mon Jul 29 17:53:33 2013 From: invalid at invalid.invalid (Grant Edwards) Date: Mon, 29 Jul 2013 21:53:33 +0000 (UTC) Subject: Bitwise Operations References: Message-ID: On 2013-07-29, Devyn Collier Johnson wrote: > On Python3, how can I perform bitwise operations? For instance, I want > something that will 'and', 'or', and 'xor' a binary integer. http://www.google.com/search?q=python+bitwise+operations -- Grant Edwards grant.b.edwards Yow! I have the power to at HALT PRODUCTION on all gmail.com TEENAGE SEX COMEDIES!! From devyncjohnson at gmail.com Mon Jul 29 19:34:00 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Mon, 29 Jul 2013 19:34:00 -0400 Subject: Bitwise Operations In-Reply-To: References: Message-ID: <51F6FBE8.8000106@Gmail.com> On 07/29/2013 05:53 PM, Grant Edwards wrote: > On 2013-07-29, Devyn Collier Johnson wrote: > >> On Python3, how can I perform bitwise operations? For instance, I want >> something that will 'and', 'or', and 'xor' a binary integer. > http://www.google.com/search?q=python+bitwise+operations > I understand the symbols. I want to know how to perform the task in a script or terminal. I have searched Google, but I never saw a command. Typing "101 & 010" or "x = (int(101, 2) & int(010, 2))" only gives errors. Mahalo, DCJ From ethan at stoneleaf.us Mon Jul 29 19:41:38 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 29 Jul 2013 16:41:38 -0700 Subject: Bitwise Operations In-Reply-To: <51F6FBE8.8000106@Gmail.com> References: <51F6FBE8.8000106@Gmail.com> Message-ID: <51F6FDB2.9040307@stoneleaf.us> On 07/29/2013 04:34 PM, Devyn Collier Johnson wrote: > > On 07/29/2013 05:53 PM, Grant Edwards wrote: >> On 2013-07-29, Devyn Collier Johnson wrote: >> >>> On Python3, how can I perform bitwise operations? For instance, I want >>> something that will 'and', 'or', and 'xor' a binary integer. >> http://www.google.com/search?q=python+bitwise+operations >> > I understand the symbols. I want to know how to perform the task in a script or terminal. I have searched Google, but I > never saw a command. Typing "101 & 010" or "x = (int(101, 2) & int(010, 2))" only gives errors. x = (int('101', 2) & int('010', 2)) Notice the quotes. In the future you'll better answers quicker if you tell us what you did (such as your example above) as well as the errors. -- ~Ethan~ From devyncjohnson at gmail.com Mon Jul 29 19:48:34 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Mon, 29 Jul 2013 19:48:34 -0400 Subject: Bitwise Operations In-Reply-To: <51F6FDB2.9040307@stoneleaf.us> References: <51F6FBE8.8000106@Gmail.com> <51F6FDB2.9040307@stoneleaf.us> Message-ID: <51F6FF52.4040806@Gmail.com> On 07/29/2013 07:41 PM, Ethan Furman wrote: > On 07/29/2013 04:34 PM, Devyn Collier Johnson wrote: >> >> On 07/29/2013 05:53 PM, Grant Edwards wrote: >>> On 2013-07-29, Devyn Collier Johnson wrote: >>> >>>> On Python3, how can I perform bitwise operations? For instance, I want >>>> something that will 'and', 'or', and 'xor' a binary integer. >>> http://www.google.com/search?q=python+bitwise+operations >>> >> I understand the symbols. I want to know how to perform the task in a >> script or terminal. I have searched Google, but I >> never saw a command. Typing "101 & 010" or "x = (int(101, 2) & >> int(010, 2))" only gives errors. > > x = (int('101', 2) & int('010', 2)) > > Notice the quotes. > > In the future you'll better answers quicker if you tell us what you > did (such as your example above) as well as the errors. > > -- > ~Ethan~ Thanks Ethan for the code help and the tip. I need to get out of that habit of not including errors. This code works well, thanks! I cannot believe I was that close to the solution! Now here is something that confuses me, the binary numbers are numbers not strings, so why are they put in quotes as if they are strings? Mahalo, DCJ From rosuav at gmail.com Mon Jul 29 20:07:46 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 30 Jul 2013 01:07:46 +0100 Subject: Bitwise Operations In-Reply-To: <51F6FF52.4040806@Gmail.com> References: <51F6FBE8.8000106@Gmail.com> <51F6FDB2.9040307@stoneleaf.us> <51F6FF52.4040806@Gmail.com> Message-ID: On Tue, Jul 30, 2013 at 12:48 AM, Devyn Collier Johnson wrote: > Now here is something that confuses me, the binary numbers are numbers not > strings, so why are they put in quotes as if they are strings? They aren't numbers at that point, they're strings of digits. A number is represented in various forms: >>> 1234, 0x4d2, 0o2322, 0b10011010010 (1234, 1234, 1234, 1234) The two-argument form of int() takes a string of digits and a base: >>> int("ya",36) 1234 In base 36, y and a are digits. But in Python's base syntax, you can't use them that way, so there's no way to render the number other than as a string. For what you're doing, I think the 0b notation is the best. It's an int literal written in binary. ChrisA From rosuav at gmail.com Mon Jul 29 19:44:19 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 30 Jul 2013 00:44:19 +0100 Subject: Bitwise Operations In-Reply-To: <51F6FBE8.8000106@Gmail.com> References: <51F6FBE8.8000106@Gmail.com> Message-ID: On Tue, Jul 30, 2013 at 12:34 AM, Devyn Collier Johnson wrote: > > I understand the symbols. I want to know how to perform the task in a script > or terminal. I have searched Google, but I never saw a command. Typing "101 > & 010" or "x = (int(101, 2) & int(010, 2))" only gives errors. Your problem here isn't in the bitwise operators, but in your binary literals. Python deliberately and consciously rejects 010 as a literal, because it might be interpreted either as decimal 10, or as octal (decimal 8), the latter being C's interpretation. Fixing that shows up a more helpful error: >>> x = (int(101, 2) & int(10, 2)) Traceback (most recent call last): File "", line 1, in x = (int(101, 2) & int(10, 2)) TypeError: int() can't convert non-string with explicit base The int() call isn't doing what you think it is, because 101 is already an int. The obvious solution now is to quote the values: >>> x = (int("101", 2) & int("010", 2)) >>> x 0 But there's an easier way: >>> x = 0b101 & 0b010 >>> x 0 I think that might do what you want. Also check out the bin() function, which will turn an integer into a string of digits. ChrisA From tjreedy at udel.edu Tue Jul 30 01:46:58 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 30 Jul 2013 01:46:58 -0400 Subject: Bitwise Operations In-Reply-To: References: <51F6FBE8.8000106@Gmail.com> Message-ID: On 7/29/2013 7:44 PM, Chris Angelico wrote: > But there's an easier way: > >>>> x = 0b101 & 0b010 >>>> x > 0 > > I think that might do what you want. Also check out the bin() > function, which will turn an integer into a string of digits. >>> bin(0b101 | 0b010) '0b111' Now you are set to go. Have fun. -- Terry Jan Reedy From python at mrabarnett.plus.com Mon Jul 29 21:55:51 2013 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 30 Jul 2013 02:55:51 +0100 Subject: Bitwise Operations In-Reply-To: <51F6FBE8.8000106@Gmail.com> References: <51F6FBE8.8000106@Gmail.com> Message-ID: <51F71D27.3040001@mrabarnett.plus.com> On 30/07/2013 00:34, Devyn Collier Johnson wrote: > > On 07/29/2013 05:53 PM, Grant Edwards wrote: >> On 2013-07-29, Devyn Collier Johnson wrote: >> >>> On Python3, how can I perform bitwise operations? For instance, I want >>> something that will 'and', 'or', and 'xor' a binary integer. >> http://www.google.com/search?q=python+bitwise+operations >> > I understand the symbols. I want to know how to perform the task in a > script or terminal. I have searched Google, but I never saw a command. > Typing "101 & 010" or "x = (int(101, 2) & int(010, 2))" only gives errors. > In Python 2, an integer with a leading 0, such as 0101, was octal (base 8). This was a feature borrowed from C but often confused newbies because it looked like decimal ("Why does 0101 == 101 return False?"). In Python 3, octal is indicated by a leading 0o, such as 0o101 (== 1*64+0*8+1==65) and the old style raises an exception so that those who have switched from Python 2 will get a clear message that something has changed. For binary you need a leading 0b and for hexadecimal you need a leading 0x, so doing something similar for octal makes sense. 0b101 == 1*4+0*2+0 == 5 0o101 == 1*64+0*8+1 == 65 0x101 == 1*256+0*16+1 == 257 From ulrich.eckhardt at dominolaser.com Tue Jul 30 02:20:06 2013 From: ulrich.eckhardt at dominolaser.com (Ulrich Eckhardt) Date: Tue, 30 Jul 2013 08:20:06 +0200 Subject: Bitwise Operations In-Reply-To: References: Message-ID: Am 30.07.2013 01:34, schrieb Devyn Collier Johnson: > Typing "101 & 010" or "x = (int(101, 2) & int(010, 2))" only gives errors. What errors? Check out Eric Raymond's essay on asking smart questions, it's a real eye-opener! ;) That said, use "0b" as prefix for binary number literals (0b1000 is eight, for example). Cheers! Uli From __peter__ at web.de Tue Jul 30 01:06:44 2013 From: __peter__ at web.de (Peter Otten) Date: Tue, 30 Jul 2013 07:06:44 +0200 Subject: Bitwise Operations References: <51F6DDBA.7030208@Gmail.com> Message-ID: Devyn Collier Johnson wrote: > On Python3, how can I perform bitwise operations? For instance, I want > something that will 'and', 'or', and 'xor' a binary integer. >>> 0b1010 | 0b1100 14 >>> bin(_) '0b1110' >>> 0b1010 & 0b1100 8 >>> bin(_) '0b1000' >>> 0b1010 ^ 0b1100 6 >>> bin(_) '0b110' From syedk at pacificloud.com Mon Jul 29 17:57:08 2013 From: syedk at pacificloud.com (syed khalid) Date: Mon, 29 Jul 2013 11:57:08 -1000 Subject: importing modules Message-ID: I am attempting to import modules from Shogun to python from a non-standard python directory ie from my /home/xxx directory. is there a way on ubuntu to selectively some modules, scripts, data from one directory and others modules, scripts from another directory. In other words, is there a file(s) that provide pointers to where these modules are located. regards Sy -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From davea at davea.name Tue Jul 30 11:37:48 2013 From: davea at davea.name (Dave Angel) Date: Tue, 30 Jul 2013 11:37:48 -0400 Subject: importing modules In-Reply-To: References: Message-ID: On 07/29/2013 05:57 PM, syed khalid wrote: > I am attempting to import modules from Shogun to python from a non-standard > python directory ie from my /home/xxx directory. is there a way on ubuntu > to selectively some modules, scripts, data from one directory and others > modules, scripts from another directory. In other words, is there a file(s) > that provide pointers to where these modules are located. > Your question is confusing, but the short answer is that import will search the sys.path list for you. So in your main script, you can add a directory to that list, before doing the imports. It can also be done with an environment variable, or with the site file, but I don't bother. -- DaveA From john_ladasky at sbcglobal.net Mon Jul 29 18:04:18 2013 From: john_ladasky at sbcglobal.net (John Ladasky) Date: Mon, 29 Jul 2013 15:04:18 -0700 (PDT) Subject: Pyglet on Python3.x, problems Message-ID: Hi folks, For whatever reason, the pyglet package is getting a lot of attention on c.l.python these past few days. I am guilty of generating some of that potentially off-topic conversation myself. At the end of my last thread, I reported that I had found the pyglet-users newsgroup, and would direct my questions there. https://groups.google.com/d/msg/comp.lang.python/ARtI0GC9RHc/_6URRrhz7nUJ Well, I joined pyglet-users. I have waited for about 36 hours for the moderator to approve my first post asking for help. Several other posts have appeared, but not mine. I don't think that I was impolite. Apologies to everyone, but I'm going to ask my questions here. As you may recall, I'm teaching some adolescent computer programming students. I need a Python 3-compatible GUI, preferably one oriented towards games. I might have that GUI in hand, but I need to test it myself first. I've been researching this issue for over a week. I have a student tomorrow, so I'm facing a bit of a deadline. (Yeah, I could teach him about B-trees instead, but... you know, kids.) In my own defense, I'm not entirely certain that the problems I have encountered are specific to pyglet. They might have to do with the limitations of the 2to3 program. I'm starting with my own Ubuntu 13.04 system. I downloaded pyglet-1.2alpha1, unzipped it and executed "sudo python3 setup.py install", as I have done with many a package. Then I tried running various programs in two directories (these programs are also on the pyglet.org web site). pyglet-1.2alpha1/examples/programming_guide/hello_world.py runs fine. pyglet-1.2alpha1/examples/programming_guide/image_viewer.py also runs fine. pyglet-1.2alpha1/examples/programming_guide/animation.py produces an error. Here's the traceback. ======================================================= Traceback (most recent call last): File "/usr/local/lib/python3.3/dist-packages/pyglet/resource.py", line 538, in animation identity = self._cached_animations[name] File "/usr/lib/python3.3/weakref.py", line 69, in __getitem__ o = self.data[key]() KeyError: 'dinosaur.gif' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "animation.py", line 62, in animation = pyglet.resource.animation('dinosaur.gif') File "/usr/local/lib/python3.3/dist-packages/pyglet/resource.py", line 540, in animation animation = pyglet.image.load_animation(name, self.file(name)) File "/usr/local/lib/python3.3/dist-packages/pyglet/image/__init__.py", line 2425, in load_animation raise first_exception File "/usr/local/lib/python3.3/dist-packages/pyglet/image/__init__.py", line 2417, in load_animation image = decoder.decode_animation(file, filename) File "/usr/local/lib/python3.3/dist-packages/pyglet/image/codecs/gdkpixbuf2.py", line 121, in decode_animation gif_stream = gif.read(file) File "/usr/local/lib/python3.3/dist-packages/pyglet/image/codecs/gif.py", line 85, in read raise ImageDecodeException('Not a GIF stream') pyglet.image.codecs.ImageDecodeException: Not a GIF stream ======================================================= >From earlier discussions, I know that distutils automatically executes 2to3 on the pyglet core module, when you invoke pyglet's setup.py file from Python 3. The setup script does not appear to run 2to3 on the code outside of the package itself. And while I doubted that the KeyError I saw above was a Python 2/3 compatibility issue, I tried running 2to3 on animation.py anyway. 2to3 seems to agree with me, reporting back "RefactoringTool: No files need to be modified." Finally, I got errors when trying to run pyglet-1.2alpha1/tests/test.py. At first, it looked like the fix would be easy: ======================================================= File "tests/test.py", line 274 print '-' * 78 ^ SyntaxError: invalid syntax ======================================================= Clearly this was a problem for 2to3, so I ran it on test.py. It corrected that one print statement, and nothing else. Then I ran test.py again. This time I got a deeper problem: ======================================================= Traceback (most recent call last): File "test.py", line 215, in import tests.regression File "../tests/regression/__init__.py", line 11, in from pyglet.image import get_buffer_manager File "../pyglet/__init__.py", line 276 print '[%d] %s%s %s' % (thread, indent, name, location) ^ SyntaxError: invalid syntax ======================================================= This error suggests that, during the build and install process, 2to3 left some code uncorrected in pyglet/__init__.py. This has me worried. Could there be other code that 2to3 failed to correct? I don't want to offer my students a tool, only to have it crash on them. So, there you have it, the two errors that I am encountering. If anyone has any advice, I would appreciate it. Thanks! From joshua at landau.ws Mon Jul 29 19:14:09 2013 From: joshua at landau.ws (Joshua Landau) Date: Tue, 30 Jul 2013 00:14:09 +0100 Subject: Pyglet on Python3.x, problems In-Reply-To: References: Message-ID: On 29 July 2013 23:04, John Ladasky wrote: > For whatever reason, the pyglet package is getting a lot of attention on > c.l.python these past few days. I am guilty of generating some of that > potentially off-topic conversation myself. At the end of my last thread, I > reported that I had found the pyglet-users newsgroup, and would direct my > questions there. > > https://groups.google.com/d/msg/comp.lang.python/ARtI0GC9RHc/_6URRrhz7nUJ I suggest you redirect to the archives from http://mail.python.org/pipermail/python-list/. > Well, I joined pyglet-users. I have waited for about 36 hours for the > moderator to approve my first post asking for help. Several other posts > have appeared, but not mine. I don't think that I was impolite. > > Apologies to everyone, but I'm going to ask my questions here. As you may > recall, I'm teaching some adolescent computer programming students. I need > a Python 3-compatible GUI, preferably one oriented towards games. I might > have that GUI in hand, but I need to test it myself first. I've been > researching this issue for over a week. I have a student tomorrow, so I'm > facing a bit of a deadline. (Yeah, I could teach him about B-trees > instead, but... you know, kids.) > > In my own defense, I'm not entirely certain that the problems I have > encountered are specific to pyglet. They might have to do with the > limitations of the 2to3 program. > > I'm starting with my own Ubuntu 13.04 system. I downloaded > pyglet-1.2alpha1, unzipped it and executed "sudo python3 setup.py install", > as I have done with many a package. Then I tried running various programs > in two directories (these programs are also on the pyglet.org web site). > > pyglet-1.2alpha1/examples/programming_guide/hello_world.py runs fine. > > pyglet-1.2alpha1/examples/programming_guide/image_viewer.py also runs fine. > > pyglet-1.2alpha1/examples/programming_guide/animation.py produces an > error. Here's the traceback. > ... > From earlier discussions, I know that distutils automatically executes > 2to3 on the pyglet core module, when you invoke pyglet's setup.py file from > Python 3. The setup script does not appear to run 2to3 on the code outside > of the package itself. And while I doubted that the KeyError I saw above > was a Python 2/3 compatibility issue, I tried running 2to3 on animation.py > anyway. 2to3 seems to agree with me, reporting back "RefactoringTool: No > files need to be modified." > > Finally, I got errors when trying to run pyglet-1.2alpha1/tests/test.py. > At first, it looked like the fix would be easy: > ... > Clearly this was a problem for 2to3, so I ran it on test.py. It corrected > that one print statement, and nothing else. Then I ran test.py again. > This time I got a deeper problem: > ... > This error suggests that, during the build and install process, 2to3 left > some code uncorrected in pyglet/__init__.py. This has me worried. Could > there be other code that 2to3 failed to correct? I don't want to offer my > students a tool, only to have it crash on them. > > So, there you have it, the two errors that I am encountering. If anyone > has any advice, I would appreciate it. Thanks! > Seems relevant: https://code.google.com/p/pyglet/issues/detail?id=615&colspec=ID%20Stars%20StatusType%20OpSys%20Modified%20Summary If https://code.google.com/p/pyglet/issues/detail?id=630&colspec=ID%20Stars%20StatusType%20OpSys%20Modified%20Summary is your problem then, for me: >>> import pyglet >>> pyglet.window >>> pyglet.window.Window() Window pops up. It works! Sorry. What happens when you help(pyglet) and help(pyglet.window) and help(pyglet.window.Window)? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ian.g.kelly at gmail.com Mon Jul 29 19:44:18 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 29 Jul 2013 17:44:18 -0600 Subject: Pyglet on Python3.x, problems In-Reply-To: References: Message-ID: On Mon, Jul 29, 2013 at 4:04 PM, John Ladasky wrote: > pyglet-1.2alpha1/examples/programming_guide/hello_world.py runs fine. > > pyglet-1.2alpha1/examples/programming_guide/image_viewer.py also runs fine. > > pyglet-1.2alpha1/examples/programming_guide/animation.py produces an error. Here's the traceback. > > ======================================================= > > Traceback (most recent call last): > File "/usr/local/lib/python3.3/dist-packages/pyglet/resource.py", line 538, in animation > identity = self._cached_animations[name] > File "/usr/lib/python3.3/weakref.py", line 69, in __getitem__ > o = self.data[key]() > KeyError: 'dinosaur.gif' > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "animation.py", line 62, in > animation = pyglet.resource.animation('dinosaur.gif') > File "/usr/local/lib/python3.3/dist-packages/pyglet/resource.py", line 540, in animation > animation = pyglet.image.load_animation(name, self.file(name)) > File "/usr/local/lib/python3.3/dist-packages/pyglet/image/__init__.py", line 2425, in load_animation > raise first_exception > File "/usr/local/lib/python3.3/dist-packages/pyglet/image/__init__.py", line 2417, in load_animation > image = decoder.decode_animation(file, filename) > File "/usr/local/lib/python3.3/dist-packages/pyglet/image/codecs/gdkpixbuf2.py", line 121, in decode_animation > gif_stream = gif.read(file) > File "/usr/local/lib/python3.3/dist-packages/pyglet/image/codecs/gif.py", line 85, in read > raise ImageDecodeException('Not a GIF stream') > pyglet.image.codecs.ImageDecodeException: Not a GIF stream > > ======================================================= > > From earlier discussions, I know that distutils automatically executes 2to3 on the pyglet core module, when you invoke pyglet's setup.py file from Python 3. The setup script does not appear to run 2to3 on the code outside of the package itself. And while I doubted that the KeyError I saw above was a Python 2/3 compatibility issue, I tried running 2to3 on animation.py anyway. 2to3 seems to agree with me, reporting back "RefactoringTool: No files need to be modified." >From the error message my first guess would be that the example is broken because the 'dinosaur.gif' file is either missing or corrupt. Have you tried running the same example in Python 2? > Finally, I got errors when trying to run pyglet-1.2alpha1/tests/test.py. At first, it looked like the fix would be easy: > > ======================================================= > > File "tests/test.py", line 274 > print '-' * 78 > ^ > SyntaxError: invalid syntax > > ======================================================= > > Clearly this was a problem for 2to3, so I ran it on test.py. It corrected that one print statement, and nothing else. Then I ran test.py again. This time I got a deeper problem: This is in the directory that you extracted the files into, not the installation directory, correct? If so, I'm not surprised that the files in that location haven't been ported by the installer. The question then would be "what's the proper way to run the tests under Python 3", which I don't know the answer to. > ======================================================= > > Traceback (most recent call last): > File "test.py", line 215, in > import tests.regression > File "../tests/regression/__init__.py", line 11, in > from pyglet.image import get_buffer_manager > File "../pyglet/__init__.py", line 276 > print '[%d] %s%s %s' % (thread, indent, name, location) > ^ > SyntaxError: invalid syntax > > ======================================================= > > This error suggests that, during the build and install process, 2to3 left some code uncorrected in pyglet/__init__.py. This has me worried. Could there be other code that 2to3 failed to correct? I don't want to offer my students a tool, only to have it crash on them. The traceback is using ../pyglet, not your system installation of pyglet, so as with the above it looks like this hasn't been ported because it's the raw copy in the installation directory. If that code had not been ported in the system installation then you would not even be able to "import pyglet" without getting an error, which you clearly are able to do since you said the first two examples worked for you. From john_ladasky at sbcglobal.net Mon Jul 29 20:34:39 2013 From: john_ladasky at sbcglobal.net (John Ladasky) Date: Mon, 29 Jul 2013 17:34:39 -0700 (PDT) Subject: Pyglet on Python3.x, problems In-Reply-To: References: Message-ID: <1f999589-e446-483e-af77-f74c9ddba2a7@googlegroups.com> Thanks for your reply, Joshua. >From the interpreter, I too can import pyglet, instantiate a pyglet.window.Window, and have it pop up (although, following your directions, now I can't CLOSE it because you didn't assign a name to it! :^]). I can get all the help information as well. It looks like Ian is on the right track (see below). If I remove tests folder from within the pyglet-1.2alpha1 folder and then try to run test.py, it starts. From john_ladasky at sbcglobal.net Mon Jul 29 20:47:40 2013 From: john_ladasky at sbcglobal.net (John Ladasky) Date: Mon, 29 Jul 2013 17:47:40 -0700 (PDT) Subject: Pyglet on Python3.x, problems In-Reply-To: References: Message-ID: <7a49a3ab-070f-4c5f-baea-9faf4fc3a6f7@googlegroups.com> Grrrr. Ian, thank you, you gave me a clue. I thought I was being careful about avoiding local imports. I just removed the tests directory from inside the pyglet-1.2alpha1 directory and tried running it from its new location. That got rid of the error message which was displaying uncorrected Python 2.x code, and now I am running tests. I'm getting one problem. After a few tests run, I can't close a window. I am normally closing each interactive test with the ESC key. But when that fails I try clicking with the mouse. This also fails. This broken behavior appears to be triggered when the window created by pyglet overlaps my terminal window and thereby steals the focus. Any idea why that might be happening? The GIF problem in animation.py persists in Python 3.3. I tried it in Python 2 as you suggested, and it works there. Odd. From rosuav at gmail.com Mon Jul 29 21:38:13 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 30 Jul 2013 02:38:13 +0100 Subject: Pyglet on Python3.x, problems In-Reply-To: <7a49a3ab-070f-4c5f-baea-9faf4fc3a6f7@googlegroups.com> References: <7a49a3ab-070f-4c5f-baea-9faf4fc3a6f7@googlegroups.com> Message-ID: On Tue, Jul 30, 2013 at 1:47 AM, John Ladasky wrote: > I'm getting one problem. After a few tests run, I can't close a window. I am normally closing each interactive test with the ESC key. But when that fails I try clicking with the mouse. This also fails. This broken behavior appears to be triggered when the window created by pyglet overlaps my terminal window and thereby steals the focus. Any idea why that might be happening? When you say "steals the focus", what exactly do you mean? Do the windows normally get created without focus, but occasionally focus is set to that window? Seems odd. If that's the case, is there any difference if you click outside the window (eg on your terminal) and then back in? Maybe there's something happening on gotfocus that should happen on create. ChrisA From cmpython at gmail.com Mon Jul 29 22:02:29 2013 From: cmpython at gmail.com (CM) Date: Mon, 29 Jul 2013 19:02:29 -0700 (PDT) Subject: SQLite logic error or missing database Message-ID: (Posted to SQLite users list first; 3 views so far, and no answers, so trying here, thinking that perhaps a Python user would have some clues; I hope that is OK) I am using SQLite through either Python 2.5 or 2.7, which is the sqlite3 module. In a desktop application, every now and then, and in a fairly irreproducible way, when committing to the database I get this error: "sqlite3.OperationalError: SQL logic error or missing database" I thought this was a PySqlite generated error, but now I see the same error is seen with Ruby, PHP, C++ and other languages, so now I think it is generated by SQLite itself...but I really don't know. If I try additional commits in that same instance of my app being open, it gives me the same error every time. If I close the app and re-open it, it does not give me this error, with the same or very similar data being written in the same routines. So I "know" that the code as written is correct (a significant--greater than 90%?--of the time I don't see this error). In terms of what is causing this, I don't know. But I've noticed that on the occasions that this has happened my computer's RAM was pretty bogged down and my computer is pretty laggy. That said, I've had other times when my RAM was hogged just as much and it didn't do this. This error might go away if I used a newer/cleaner/more RAM computer, but I want to "stress test" my application for those who may be using similarly clunky computers--I want to try to avoid it even for older model computers. Any advice appreciated. From rosuav at gmail.com Mon Jul 29 22:17:25 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 30 Jul 2013 03:17:25 +0100 Subject: SQLite logic error or missing database In-Reply-To: References: Message-ID: On Tue, Jul 30, 2013 at 3:02 AM, CM wrote: > If I try additional commits in that same instance of my app being open, > it gives me the same error every time. If I close the app and re-open > it, it does not give me this error, with the same or very similar data > being written in the same routines. So I "know" that the code as > written is correct (a significant--greater than 90%?--of the time I > don't see this error). Is it a race between two concurrent instances of the app? I don't know sqlite but that seems like something to consider, at least. ChrisA From quartese at gmail.com Tue Jul 30 04:57:06 2013 From: quartese at gmail.com (quartese at gmail.com) Date: Tue, 30 Jul 2013 01:57:06 -0700 (PDT) Subject: =?windows-1252?Q?Modeling_life_on_Earth_=96=2D_an_object=2Doriented_=28Pyth?= =?windows-1252?Q?on=3F=29_challenge?= Message-ID: Dear List, I have to start this email by saying that I have recently attended EuroPython in Florence, and it was the best and better organized conference I have ever attended in 14 years of international meetings. I apologize if this is off topic, but I read in the list's description that ?[p]retty much anything Python-related is fair game for discussion?. Although I am not a Python developer, I decided to attend EuroPython in search for a programmer interested in collaborating in the Python project I briefly describe below. I use ecosystem models implemented with a procedural paradigm in a language different from Python (Pascal, for the records). I would like to migrate these ecosystem models (and code) to an object-oriented paradigm using Python, as I have come to believe its expressiveness would help a lot get the semantics right, rather than simply split procedural code into objects corresponding to ecological elements. What's more, our models use physiological analogies among the different levels of the food chain or web, and this makes them amenable to an even higher level of object-oriented abstraction given adequate expressiveness. The goal is to go beyond the currently (mostly) formal implementation of the object-oriented paradigm in ecological models. To do that, I would need help from an expert Python programmer (who also has some math skills, knows English, and can work in the Rome area, or at least central Italy). I need help because I am a Python beginner with limited programming experience in general, and hence my contribution will mainly be the ecosystem modeling insight. At EuroPython, I gave a lightning talk about the project that can be found on YouTube http://youtu.be/iUNbgNuN0qY?t=31m50s As I already made some very promising contacts at EuroPyton with developers that are interested and willing to help, and many people shared their views and provided useful insight into the issue (thanks!), this post is meant to get further feedback on my idea and possibly reach other interested developers. Kindly contact me if you have any interest in the idea and time to devote it, as it is becoming a funded project. Kind regards, thanks for any hint, and apologies for the many inaccuracies, Luigi From dwightdhutto at gmail.com Tue Jul 30 07:23:02 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Tue, 30 Jul 2013 07:23:02 -0400 Subject: =?windows-1252?Q?Re=3A_Modeling_life_on_Earth_=96=2D_an_object=2Doriented_=28?= =?windows-1252?Q?Python=3F=29_challenge?= In-Reply-To: References: Message-ID: Never used pascal, and python might not be the fastest way to implement a program such as this. In a previous discussion, this was taken place by someone using a predator prey brain class.. The simulation will vary, until a full refinement of forecast is above a certainty percentage level. Visualization is needed as well. Collaboration is, of course , the best possible route. However you need to start with certain statistics, and know there will be an Uncerrtainty Principle rule applied. The algorithm for such massive amounts of data analysis in a simulation forecast, will involve HD space and RAM . You will also want to collaborate with certain databases in order to refine the accuracy of your models. This is kind of what I would consider being a Dune(Frank Herbert) planetary engineer. It also takes in other db data such as tagging marks of animals percentiles of bacterias/viruses/etc....SO it's not as simple as it sounds, and python would be more of a prototyping language, and later translated into another language for faster maneuvering of data. On Tue, Jul 30, 2013 at 4:57 AM, wrote: > Dear List, > > I have to start this email by saying that I have recently attended > EuroPython in Florence, and it was the best and better organized conference > I have ever attended in 14 years of international meetings. > > I apologize if this is off topic, but I read in the list's description > that ?[p]retty much anything Python-related is fair game for discussion?. > > Although I am not a Python developer, I decided to attend EuroPython in > search for a programmer interested in collaborating in the Python project I > briefly describe below. > > I use ecosystem models implemented with a procedural paradigm in a > language different from Python (Pascal, for the records). I would like to > migrate these ecosystem models (and code) to an object-oriented paradigm > using Python, as I have come to believe its expressiveness would help a lot > get the semantics right, rather than simply split procedural code into > objects corresponding to ecological elements. What's more, our models use > physiological analogies among the different levels of the food chain or > web, and this makes them amenable to an even higher level of > object-oriented abstraction given adequate expressiveness. > > The goal is to go beyond the currently (mostly) formal implementation of > the object-oriented paradigm in ecological models. To do that, I would need > help from an expert Python programmer (who also has some math skills, knows > English, and can work in the Rome area, or at least central Italy). I need > help because I am a Python beginner with limited programming experience in > general, and hence my contribution will mainly be the ecosystem modeling > insight. > > At EuroPython, I gave a lightning talk about the project that can be found > on YouTube > http://youtu.be/iUNbgNuN0qY?t=31m50s > > As I already made some very promising contacts at EuroPyton with > developers that are interested and willing to help, and many people shared > their views and provided useful insight into the issue (thanks!), this post > is meant to get further feedback on my idea and possibly reach other > interested developers. > > Kindly contact me if you have any interest in the idea and time to devote > it, as it is becoming a funded project. > > Kind regards, thanks for any hint, and apologies for the many inaccuracies, > > Luigi > -- > http://mail.python.org/mailman/listinfo/python-list > -- Best Regards, David Hutto *CEO:* *http://www.hitwebdevelopment.com* -------------- next part -------------- An HTML attachment was scrubbed... URL: From quartese at gmail.com Wed Jul 31 01:31:48 2013 From: quartese at gmail.com (Luigi Ponti) Date: Wed, 31 Jul 2013 07:31:48 +0200 Subject: =?windows-1252?Q?Re=3A_Modeling_life_on_Earth_=96=2D_an_object=2Doriented_=28?= =?windows-1252?Q?Python=3F=29_challenge?= In-Reply-To: References: Message-ID: [...forgot to reply to the list...] Dear David, Thanks for your feedback -- you got right to the point: ...python would be more of a prototyping language, and later translated > into another language for faster maneuvering of data > exactly! I was hoping that, since the modeling framework is conceptually well developed (i.e., books, papers, analysis, etc. in 35+ years), most of the work would be towards getting the code up to the same conceptual (i.e., abstraction) level. Hence, I was thinking Python would be a good tool for that. Performance can be taken care of at a later stage, if needed. Please do not hesitate to drop a further line. Kind regards, Luigi On Tue, Jul 30, 2013 at 1:23 PM, David Hutto wrote: > Never used pascal, and python might not be the fastest way to implement a > program such as this. > > In a previous discussion, this was taken place by someone using a predator > prey brain class.. > > The simulation will vary, until a full refinement of forecast is above a > certainty percentage level. > > Visualization is needed as well. > > Collaboration is, of course > , the best possible route. However you need to start with certain > statistics, and know there will be an Uncerrtainty Principle rule applied. > > The algorithm for such massive amounts of data analysis in a simulation > forecast, will involve HD space and RAM > . > > You will also want to collaborate with certain databases in order to > refine the accuracy of your models. > > This is kind of what I would consider being a Dune(Frank Herbert) > planetary engineer. It also takes in other db data such as tagging marks of > animals percentiles of bacterias/viruses/etc....SO it's not as simple as it > sounds, and python would be more of a prototyping language, and later > translated into another language for faster maneuvering of data. > > > > On Tue, Jul 30, 2013 at 4:57 AM, wrote: > >> Dear List, >> >> I have to start this email by saying that I have recently attended >> EuroPython in Florence, and it was the best and better organized conference >> I have ever attended in 14 years of international meetings. >> >> I apologize if this is off topic, but I read in the list's description >> that ?[p]retty much anything Python-related is fair game for discussion?. >> >> Although I am not a Python developer, I decided to attend EuroPython in >> search for a programmer interested in collaborating in the Python project I >> briefly describe below. >> >> I use ecosystem models implemented with a procedural paradigm in a >> language different from Python (Pascal, for the records). I would like to >> migrate these ecosystem models (and code) to an object-oriented paradigm >> using Python, as I have come to believe its expressiveness would help a lot >> get the semantics right, rather than simply split procedural code into >> objects corresponding to ecological elements. What's more, our models use >> physiological analogies among the different levels of the food chain or >> web, and this makes them amenable to an even higher level of >> object-oriented abstraction given adequate expressiveness. >> >> The goal is to go beyond the currently (mostly) formal implementation of >> the object-oriented paradigm in ecological models. To do that, I would need >> help from an expert Python programmer (who also has some math skills, knows >> English, and can work in the Rome area, or at least central Italy). I need >> help because I am a Python beginner with limited programming experience in >> general, and hence my contribution will mainly be the ecosystem modeling >> insight. >> >> At EuroPython, I gave a lightning talk about the project that can be >> found on YouTube >> http://youtu.be/iUNbgNuN0qY?t=31m50s >> >> As I already made some very promising contacts at EuroPyton with >> developers that are interested and willing to help, and many people shared >> their views and provided useful insight into the issue (thanks!), this post >> is meant to get further feedback on my idea and possibly reach other >> interested developers. >> >> Kindly contact me if you have any interest in the idea and time to devote >> it, as it is becoming a funded project. >> >> Kind regards, thanks for any hint, and apologies for the many >> inaccuracies, >> >> Luigi >> -- >> http://mail.python.org/mailman/listinfo/python-list >> > > > > -- > Best Regards, > David Hutto > *CEO:* *http://www.hitwebdevelopment.com* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From karthiksridhar1990 at gmail.com Tue Jul 30 06:40:10 2013 From: karthiksridhar1990 at gmail.com (karthik sridhar) Date: Tue, 30 Jul 2013 03:40:10 -0700 (PDT) Subject: How to parse the starting and ending of a loop statements in python Message-ID: <3559218c-8ebd-494c-a144-97399e6bef12@googlegroups.com> My objective is to find the line numbers of the start and the end of a loop statement in python. Example scenario #A.py Line1: a=0 Line2: while a<5: Line3: print a Line4: a=a+1 Desired output: Start of a loop Line2 End of a loop Line4 Current parser code #parser.py with open(a) as f: tree = ast.parse(f.read()) taskline=[] for node in ast.walk(tree): if isinstance(node, (ast.For)) or isinstance(node,(ast.While)): print node.lineno-1 <-- This give line number on for the start of a loop I wanted to achieve the above output. I use AST to parse a given file and determine the occurrence of loops. With AST parsing i am able to find line number for the start of the loop but the line number for ending of the loop is yet to be determined. Is there any way i could parse an entire loop statement and determine its starting and ending line number ? From christoph.wruck at gmail.com Tue Jul 30 09:04:14 2013 From: christoph.wruck at gmail.com (CWr) Date: Tue, 30 Jul 2013 06:04:14 -0700 (PDT) Subject: Python descriptor protocol (for more or less structured data) Message-ID: <040f33f6-6435-4aea-98ae-eabf8c16b167@googlegroups.com> Hi together, Some years ago I started a small WSGI project at my university. Since then the project was grown up every year. Some classes have more than 600 lines of code with (incl. boiler-plates mostly in descriptors/properties). Many of these properties are similar or have depencies among themselves. The idea is to grouping similar properties like: new style: ---------- >>>m = MyClass(...) >>>m.attr = 'some; complex:data#string' >>>m.attr.value 'some' >>>m.attr.extras {'complex':('data','string')} I wrote this descriptor: class Descr: def __init__(self, value): self.attribute = self.__class__.__name__ self.__set__(None, value) def __get__(self, obj, Type=None): return getattr(obj, self.attribute, self) def __set__(self, obj, value): if obj is None: # descripting yourself # do something here ... self.value = value else: if hasattr(obj, self.attribute): self.__get__(obj).__set__(None, value) else: setattr(obj, self.attribute, type(self)(value)) This works fine as long as the value attribute of Descr is read-only and the user have to use the descriptor interface e.g. __get__/__set__. Because it's not guaranteed that the user gets a seperated instance of Descr which will be living in obj.__dict__. If obj is None the descriptor will be returned themselves. But I would like that the user can use the following statement: >>>m = MyClass(...) >>>m.attr = 'some; complex:data#string' >>>m.attr.value 'some' >>>m.attr.value = 'some other' >>>m.attr.value 'some other' But this usage will be problematic. If the descriptor returned themselves (default case) and the user modified the value, he modified the default value without to create a seperated instance attribute. >>>class C: >>> def __init__(self, value): >>> if not value is None: >>> self.attr = value >>> attr = Descr('default value') >>># explicit default usage (no problem): >>>C.attr.value 'default value' >>>a = C() >>>a.attr.value 'default value' The following is the main Problem: >>>a.attr.value = 'other' >>>C.attr.value 'other' The user could think that a new instance based value will be created. But it isn't. It will works fine only if I assign a value explicitly. >>>m = MyClass(value='test') >>>m.__dict__ >>>{'Descr':} Has anyone had a similar problem in the past? Or I am on the wrong way. Kind Regards, Chris Sorry for my terrible english ... From __peter__ at web.de Tue Jul 30 10:35:41 2013 From: __peter__ at web.de (Peter Otten) Date: Tue, 30 Jul 2013 16:35:41 +0200 Subject: Python descriptor protocol (for more or less structured data) References: <040f33f6-6435-4aea-98ae-eabf8c16b167@googlegroups.com> Message-ID: CWr wrote: > Some years ago I started a small WSGI project at my university. Since then > the project was grown up every year. Some classes have more than 600 lines > of code with (incl. boiler-plates mostly in descriptors/properties). > > Many of these properties are similar or have depencies among themselves. > The idea is to grouping similar properties like: > > new style: > ---------- >>>>m = MyClass(...) >>>>m.attr = 'some; complex:data#string' > >>>>m.attr.value > 'some' >>>>m.attr.extras > {'complex':('data','string')} > > I wrote this descriptor: > > class Descr: > > def __init__(self, value): > self.attribute = self.__class__.__name__ > self.__set__(None, value) > > def __get__(self, obj, Type=None): > return getattr(obj, self.attribute, self) > > def __set__(self, obj, value): > if obj is None: # descripting yourself > # do something here ... > self.value = value > else: > if hasattr(obj, self.attribute): > self.__get__(obj).__set__(None, value) > else: > setattr(obj, self.attribute, type(self)(value)) You must not store per-object data in the descriptor. I suggest a naming convention (the internal data for obj.attr is stored in obj._attr) together with a value class that handles breaking of the string into attributes of an instance of itself: class StructuredAttribute: def __init__(self, name, make_default): self.name = name self.make_default = make_default def __get__(self, obj, type=None): if obj is None: return self _name = "_" + self.name try: return getattr(obj, _name) except AttributeError: setattr(obj, _name, self.make_default()) return getattr(obj, _name) def __set__(self, obj, value): self.__get__(obj).update(value) class Value: def __init__(self, value): self.update(value) def update(self, value): if isinstance(value, str): self.value, sep, rest = value.partition(";") self.extras = dict(item.partition("#")[::2] for item in rest.split()) else: self.value = value.value self.extras = value.extras def __repr__(self): return repr("{}; {}".format(self.value, " ".join("{}: {}".format(*item) for item in self.extras.items()))) def make_default_value(): return Value("some; complex:data#string") class A: attr = StructuredAttribute("alpha", make_default_value) def show(obj): print("attr:", obj.attr) print("attr.value:", obj.attr.value) print("attr.extras:", obj.attr.extras) a = A() show(a) newvalue = "whatever" print("updating value to", newvalue) a.attr.value = newvalue show(a) That's the general idea if you want "setattr polymorphism". Personally I would go with simpler standard attributes: class A: def __init__(self): self.attr = Value(...) a = A() a.value = Value(...) a.value.extras = ... From christoph.wruck at gmail.com Wed Jul 31 05:16:08 2013 From: christoph.wruck at gmail.com (CWr) Date: Wed, 31 Jul 2013 02:16:08 -0700 (PDT) Subject: Python descriptor protocol (for more or less structured data) In-Reply-To: References: <040f33f6-6435-4aea-98ae-eabf8c16b167@googlegroups.com> Message-ID: <7ba674c2-9645-4644-8254-4407905e3270@googlegroups.com> Peter, thanks for your response. Sure, you are right when you say that's easier to use standard attribute assigning via __init__. But my intention was: - reducing the complexiticity of __init__ - avoiding boiler-plates (mostly property descriptors inside of the main class) - creating instances (for complex data strings) only if they will be needed, otherwise use default instances (descriptors) - make it prossible that the data structure can be used in static context - like MyClass.attr - to get default values Standard procedure: >>>class C: >>> def __init__(self, one, two=None, three=None, four=None, five=None, ...): >>> if not two is None: >>> self.two = Value(two) >>> else: >>> self.two = Value(self.DEFAULT_4_TWO) >>> ... vs: >>>class C: >>> >>> two = MyDescriptor('default for two') >>> >>> def __init__(self, one, two=None, three=None, four=None, five=None, ...): >>> self.one = one >>> if not two is None: >>> self.two = two >>> ... Probably it will be necessary to set the attribute at first access. Alternatively it may be possible to observe the descriptor til an attribute will be setted e.g. instance.attr.value = 'whatever'. At this point a new instance (like Value) should be created on obj.__dict__. It's the procedure what I'm looking for. ;) Kind Regards, Chris From tjreedy at udel.edu Wed Jul 31 13:44:15 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 31 Jul 2013 13:44:15 -0400 Subject: Python descriptor protocol (for more or less structured data) In-Reply-To: <7ba674c2-9645-4644-8254-4407905e3270@googlegroups.com> References: <040f33f6-6435-4aea-98ae-eabf8c16b167@googlegroups.com> <7ba674c2-9645-4644-8254-4407905e3270@googlegroups.com> Message-ID: On 7/31/2013 5:16 AM, CWr wrote: > Peter, thanks for your response. > Sure, you are right when you say that's easier to use standard attribute assigning via __init__. > > But my intention was: > - reducing the complexiticity of __init__ > - avoiding boiler-plates (mostly property descriptors inside of the main class) > - creating instances (for complex data strings) only if they will be needed, otherwise use default instances (descriptors) > - make it prossible that the data structure can be used in static context - like MyClass.attr - to get default values > > Standard procedure: > >>>> class C: DEFAULT_4_TWO = # for following code to work >>>> def __init__(self, one, two=None, three=None, four=None, five=None, ...): >>>> if not two is None: "if two is not None:" reads better and is the preferred form. 'is not' is a single comparison operator, just like '!=' and 'not in'. The current CPython AST or peephole optimizer happens to notice that the 'is' operator followed by the 'not' operator can be replaced by the 'is not' operator, but this is not guaranteed for all implementations. >>>> self.two = Value(two) >>>> else: >>>> self.two = Value(self.DEFAULT_4_TWO) self.two = Value(two if two is not None else self.DEFAULT_4_TWO) There is no need to introduce the new name DEFAULT_4_TWO. It is a symptom of using the wrong namespace to get the default. class C: two = def __init__(self, two=None): self.two = Value(two if two is not None else C.two) -- Terry Jan Reedy From cool1574 at gmail.com Tue Jul 30 10:49:04 2013 From: cool1574 at gmail.com (cool1574 at gmail.com) Date: Tue, 30 Jul 2013 07:49:04 -0700 (PDT) Subject: Python script help Message-ID: <4566d0e7-2576-4d09-83f5-fca3b370710a@googlegroups.com> Hello, I am looking for a script that will be able to search an online document (by giving the script the URL) and find all the downloadable links in the document and then download them automatically. I appreciate your help, Thank you. From rosuav at gmail.com Tue Jul 30 11:38:57 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 30 Jul 2013 16:38:57 +0100 Subject: Python script help In-Reply-To: <4566d0e7-2576-4d09-83f5-fca3b370710a@googlegroups.com> References: <4566d0e7-2576-4d09-83f5-fca3b370710a@googlegroups.com> Message-ID: On Tue, Jul 30, 2013 at 3:49 PM, wrote: > Hello, I am looking for a script that will be able to search an online document (by giving the script the URL) and find all the downloadable links in the document and then download them automatically. > I appreciate your help, > Thank you. baseurl = "http://........" options = "....." os.system("wget "+options+" "+baseurl) Sometimes the right tool for the job isn't Python. ChrisA From cool1574 at gmail.com Tue Jul 30 11:49:11 2013 From: cool1574 at gmail.com (cool1574 at gmail.com) Date: Tue, 30 Jul 2013 08:49:11 -0700 (PDT) Subject: Python script help In-Reply-To: References: <4566d0e7-2576-4d09-83f5-fca3b370710a@googlegroups.com> Message-ID: I know but I think using Python in this situation is good...is that the full script? From rosuav at gmail.com Tue Jul 30 11:58:14 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 30 Jul 2013 16:58:14 +0100 Subject: Python script help In-Reply-To: References: <4566d0e7-2576-4d09-83f5-fca3b370710a@googlegroups.com> Message-ID: On Tue, Jul 30, 2013 at 4:49 PM, wrote: > I know but I think using Python in this situation is good...is that the full script? That script just drops out to the system and lets wget do it. So don't bother with it. ChrisA From cool1574 at gmail.com Tue Jul 30 12:10:34 2013 From: cool1574 at gmail.com (cool1574 at gmail.com) Date: Tue, 30 Jul 2013 09:10:34 -0700 (PDT) Subject: Python script help In-Reply-To: References: <4566d0e7-2576-4d09-83f5-fca3b370710a@googlegroups.com> Message-ID: What if I want to use only Python? is that possible? using lib and lib2? From cool1574 at gmail.com Tue Jul 30 12:12:50 2013 From: cool1574 at gmail.com (cool1574 at gmail.com) Date: Tue, 30 Jul 2013 09:12:50 -0700 (PDT) Subject: Python script help In-Reply-To: References: <4566d0e7-2576-4d09-83f5-fca3b370710a@googlegroups.com> Message-ID: <7e9a50d5-3340-4cf0-a3b8-ef1109837529@googlegroups.com> ** urlib, urlib2 From cs at zip.com.au Tue Jul 30 17:47:59 2013 From: cs at zip.com.au (Cameron Simpson) Date: Wed, 31 Jul 2013 07:47:59 +1000 Subject: Python script help In-Reply-To: <7e9a50d5-3340-4cf0-a3b8-ef1109837529@googlegroups.com> References: <7e9a50d5-3340-4cf0-a3b8-ef1109837529@googlegroups.com> Message-ID: <20130730214759.GA47935@cskk.homeip.net> On 30Jul2013 09:12, cool1574 at gmail.com wrote: | ** urlib, urlib2 Sure. And I'd use BeautifulSoup to do the parse. You'll need to fetch that. So: urllib[2] to fetch the document and BS to parse it for links, then urllib[2] to fetch the links you want. http://www.crummy.com/software/BeautifulSoup/bs4/download/4.0/ Cheers, -- Cameron Simpson You can be psychotic and still be competent. - John L. Young, American Academy of Psychiatry and the Law on Ted Kaczynski, and probably most internet users From joshua at landau.ws Wed Jul 31 02:24:13 2013 From: joshua at landau.ws (Joshua Landau) Date: Wed, 31 Jul 2013 07:24:13 +0100 Subject: Python script help In-Reply-To: <20130730214759.GA47935@cskk.homeip.net> References: <7e9a50d5-3340-4cf0-a3b8-ef1109837529@googlegroups.com> <20130730214759.GA47935@cskk.homeip.net> Message-ID: On 30 July 2013 22:47, Cameron Simpson wrote: > On 30Jul2013 09:12, cool1574 at gmail.com wrote: > | ** urlib, urlib2 > > Sure. And I'd use BeautifulSoup to do the parse. You'll need to fetch that. > So: urllib[2] to fetch the document and BS to parse it for links, > then urllib[2] to fetch the links you want. > > http://www.crummy.com/software/BeautifulSoup/bs4/download/4.0/ Personally BeautifulSoup + requests is a great combination. Maybe I'm just lazy ;). -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Jul 30 12:22:11 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 30 Jul 2013 17:22:11 +0100 Subject: Python script help In-Reply-To: References: <4566d0e7-2576-4d09-83f5-fca3b370710a@googlegroups.com> Message-ID: On Tue, Jul 30, 2013 at 5:10 PM, wrote: > What if I want to use only Python? is that possible? using lib and lib2? > -- > http://mail.python.org/mailman/listinfo/python-list Sure, anything's possible. And a lot easier if you quote context in your posts. But why do it? wget is exactly what you need. ChrisA From vincent.vandevyvre at swing.be Tue Jul 30 12:58:43 2013 From: vincent.vandevyvre at swing.be (Vincent Vande Vyvre) Date: Tue, 30 Jul 2013 18:58:43 +0200 Subject: Python script help In-Reply-To: References: <4566d0e7-2576-4d09-83f5-fca3b370710a@googlegroups.com> Message-ID: <51F7F0C3.5080209@swing.be> Le 30/07/2013 18:10, cool1574 at gmail.com a ?crit : > What if I want to use only Python? is that possible? using lib and lib2? Have a look here: http://bazaar.launchpad.net/~vincent-vandevyvre/qarte/trunk/view/head:/parsers.py This script get a web page and parse it to find downloadable objects. -- Vincent V.V. Oqapy . Qarte . PaQager From ulrich.eckhardt at dominolaser.com Tue Jul 30 11:32:33 2013 From: ulrich.eckhardt at dominolaser.com (Ulrich Eckhardt) Date: Tue, 30 Jul 2013 17:32:33 +0200 Subject: Python script help In-Reply-To: <4566d0e7-2576-4d09-83f5-fca3b370710a@googlegroups.com> References: <4566d0e7-2576-4d09-83f5-fca3b370710a@googlegroups.com> Message-ID: Am 30.07.2013 16:49, schrieb cool1574 at gmail.com: > Hello, I am looking for a script that will be able to search an > online document (by giving the script the URL) and find all the > downloadable links in the document and then download them > automatically. Well, that's actually pretty simple. Using the URL, download the document. Then, parse it in order to extract embedded URLs and finally download the resulting URLs. If you have specific problems, please provide more info which part exactly you're having problems with, along with what you already tried etc. In short, show some effort yourself. In the meantime, I'd suggest reading a Python tutorial and Eric Raymonds essay on asking smart questions. Greetings! Uli From denismfmcmahon at gmail.com Wed Jul 31 01:08:47 2013 From: denismfmcmahon at gmail.com (Denis McMahon) Date: Wed, 31 Jul 2013 05:08:47 +0000 (UTC) Subject: Python script help References: <4566d0e7-2576-4d09-83f5-fca3b370710a@googlegroups.com> Message-ID: On Tue, 30 Jul 2013 07:49:04 -0700, cool1574 wrote: > Hello, I am looking for a script that will be able to search an online > document (by giving the script the URL) and find all the downloadable > links in the document and then download them automatically. > I appreciate your help, Why use Python? Just: wget -m url -- Denis McMahon, denismfmcmahon at gmail.com From cool1574 at gmail.com Wed Jul 31 04:15:12 2013 From: cool1574 at gmail.com (cool1574 at gmail.com) Date: Wed, 31 Jul 2013 01:15:12 -0700 (PDT) Subject: Python script help In-Reply-To: <4566d0e7-2576-4d09-83f5-fca3b370710a@googlegroups.com> References: <4566d0e7-2576-4d09-83f5-fca3b370710a@googlegroups.com> Message-ID: Here are some scripts, how do I put them together to create the script I want? (to search a online document and download all the links in it) p.s: can I set a destination folder for the downloads? urllib.urlopen("http://....") possible_urls = re.findall(r'\S+:\S+', text) import urllib2 response = urllib2.urlopen('http://www.example.com/') html = response.read() From wuwei23 at gmail.com Wed Jul 31 20:57:01 2013 From: wuwei23 at gmail.com (alex23) Date: Thu, 01 Aug 2013 10:57:01 +1000 Subject: Python script help In-Reply-To: References: <4566d0e7-2576-4d09-83f5-fca3b370710a@googlegroups.com> Message-ID: On 31/07/2013 6:15 PM, cool1574 at gmail.com wrote: > Here are some scripts, how do I put them together to create the script I want? (to search a online document and download all the links in it) 1. Think about the requirements. 2. Write some code. 3. Test it. 4. Repeat until requirements are met. > p.s: can I set a destination folder for the downloads? Yes. Show us you're actively trying to solve this yourself rather than just asking us to write the code for you. From devyncjohnson at gmail.com Tue Jul 30 10:06:25 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Tue, 30 Jul 2013 10:06:25 -0400 Subject: Share Code: Laptop Lid State Message-ID: <51F7C861.3060208@Gmail.com> Aloha everyone! I attached a script that I thought I could share with everyone for your help. This Python3 script only works on Unix systems. It prints the current state of the lid. This can be used to make a script that performs some action when the lid is closed or open. The script is licensed under LGPLv3 and I will soon upload it to my Launchpad account. Enjoy! Mahalo, DevynCJohnson at Gmail.com -------------- next part -------------- #!/usr/bin/env python3 #Made by Devyn Collier Johnson, NCLA, Linux+, LPIC-1, DCTS # LGPLv3 - 2013. (Devyn Collier Johnson, NCLA, Linux+, LPIC-1, DCTS)? #This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as #published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. # This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. #You should have received a copy of the GNU General Public License along with this program. If not, see . import subprocess; print(subprocess.getoutput('cat /proc/acpi/button/lid/LID/state | awk \'{ print $2 }\'')) From rosuav at gmail.com Tue Jul 30 12:00:04 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 30 Jul 2013 17:00:04 +0100 Subject: Share Code: Laptop Lid State In-Reply-To: <51F7C861.3060208@Gmail.com> References: <51F7C861.3060208@Gmail.com> Message-ID: On Tue, Jul 30, 2013 at 3:06 PM, Devyn Collier Johnson wrote: > Aloha everyone! > > I attached a script that I thought I could share with everyone for your > help. This Python3 script only works on Unix systems. It prints the current > state of the lid. This can be used to make a script that performs some > action when the lid is closed or open. The script is licensed under LGPLv3 > and I will soon upload it to my Launchpad account. Enjoy! There's... no Python code in that. Why not simply open("/proc/acpi/button/lid/LID/state") and read from it, instead of using cat and awk? ChrisA From ian.g.kelly at gmail.com Tue Jul 30 14:14:35 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Tue, 30 Jul 2013 12:14:35 -0600 Subject: Share Code: Laptop Lid State In-Reply-To: References: <51F7C861.3060208@Gmail.com> Message-ID: On Jul 30, 2013 10:06 AM, "Chris Angelico" wrote: > > On Tue, Jul 30, 2013 at 3:06 PM, Devyn Collier Johnson > wrote: > > Aloha everyone! > > > > I attached a script that I thought I could share with everyone for your > > help. This Python3 script only works on Unix systems. It prints the current > > state of the lid. This can be used to make a script that performs some > > action when the lid is closed or open. The script is licensed under LGPLv3 > > and I will soon upload it to my Launchpad account. Enjoy! > > There's... no Python code in that. Why not simply > open("/proc/acpi/button/lid/LID/state") and read from it, instead of > using cat and awk? Or for that matter, why not just make it a bash script instead of Python? It's kind of pointless to go to all the trouble of starting a Python interpreter just to have it start a subprocess. -------------- next part -------------- An HTML attachment was scrubbed... URL: From devyncjohnson at gmail.com Tue Jul 30 23:05:05 2013 From: devyncjohnson at gmail.com (Devyn Collier Johnson) Date: Tue, 30 Jul 2013 23:05:05 -0400 Subject: Share Code: Laptop Lid State In-Reply-To: References: <51F7C861.3060208@Gmail.com> Message-ID: <51F87EE1.50206@Gmail.com> On 07/30/2013 12:00 PM, Chris Angelico wrote: > On Tue, Jul 30, 2013 at 3:06 PM, Devyn Collier Johnson > wrote: >> Aloha everyone! >> >> I attached a script that I thought I could share with everyone for your >> help. This Python3 script only works on Unix systems. It prints the current >> state of the lid. This can be used to make a script that performs some >> action when the lid is closed or open. The script is licensed under LGPLv3 >> and I will soon upload it to my Launchpad account. Enjoy! > There's... no Python code in that. Why not simply > open("/proc/acpi/button/lid/LID/state") and read from it, instead of > using cat and awk? > > ChrisA The script returns either "open" or "close" instead of printing the whole file contents. I thought some people would find it useful (^_^;). DCJ From rosuav at gmail.com Wed Jul 31 03:14:46 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 31 Jul 2013 08:14:46 +0100 Subject: Share Code: Laptop Lid State In-Reply-To: <51F87EE1.50206@Gmail.com> References: <51F7C861.3060208@Gmail.com> <51F87EE1.50206@Gmail.com> Message-ID: On Wed, Jul 31, 2013 at 4:05 AM, Devyn Collier Johnson wrote: > > On 07/30/2013 12:00 PM, Chris Angelico wrote: >> >> On Tue, Jul 30, 2013 at 3:06 PM, Devyn Collier Johnson >> wrote: >>> >>> Aloha everyone! >>> >>> I attached a script that I thought I could share with everyone for >>> your >>> help. This Python3 script only works on Unix systems. It prints the >>> current >>> state of the lid. This can be used to make a script that performs some >>> action when the lid is closed or open. The script is licensed under >>> LGPLv3 >>> and I will soon upload it to my Launchpad account. Enjoy! >> >> There's... no Python code in that. Why not simply >> open("/proc/acpi/button/lid/LID/state") and read from it, instead of >> using cat and awk? >> >> ChrisA > > The script returns either "open" or "close" instead of printing the whole > file contents. I thought some people would find it useful (^_^;). Not having a Linux laptop handy I can't test it, but my point is that text parsing of that nature can be done directly by Python. You can snip out the "open" or "close" easily with one, maybe two lines of code at the most, and that without dropping to a shell, a completely superfluous 'cat' process, and awk. You then capture the STDOUT of that and promptly print it to your own STDOUT. Why not either do it truly in Python, or do it directly in a shell script and skip the Python interpreter? ChrisA From Bas.vdWulp at gmail.com Tue Jul 30 14:18:16 2013 From: Bas.vdWulp at gmail.com (Bas van der Wulp) Date: Tue, 30 Jul 2013 20:18:16 +0200 Subject: OrderedEnum examples Message-ID: <51f8036c$0$2995$6d4158fb@reader.xsnews.nl> Using the enum34 0.9.13 package from PyPi in Python 2.7.3, the examples for OrderedEnum seem to be broken. The example in the package documentation reads: class OrderedEnum(Enum): def __ge__(self, other): if self.__class__ is other.__class__: return self._value >= other._value return NotImplemented def __gt__(self, other): if self.__class__ is other.__class__: return self._value > other._value return NotImplemented def __le__(self, other): if self.__class__ is other.__class__: return self._value <= other._value return NotImplemented def __lt__(self, other): if self.__class__ is other.__class__: return self._value < other._value return NotImplemented class Grade(OrderedEnum): __ordered__ = 'A B C D F' A = 5 B = 4 C = 3 D = 2 F = 1 Grade.C < Grade.A to which Python replies with: Traceback (most recent call last): File "test.py", line 35, in print Grade.C < Grade.A File "test.py", line 23, in __lt__ return self._value < other._value AttributeError: 'Grade' object has no attribute '_value' Also, in the example in the Python 3.4 library documentation (section 8.17.2) has the __ordered__ attribute removed (presumably because, in contrast to Python 2.x, Python 3 will respect the order of attribute definition). This example gives the same ValueErrror when using the enum34 package in Python 2.7.3. It is the same example, after all. Replacing each occurrence of self._value with either self._value_ or self.value in the examples seems to make them work as expected. Are both examples incorrect, or not intended to work in Python 2.x? -- S. van der Wulp From ian.g.kelly at gmail.com Tue Jul 30 14:58:27 2013 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Tue, 30 Jul 2013 12:58:27 -0600 Subject: OrderedEnum examples In-Reply-To: <51f8036c$0$2995$6d4158fb@reader.xsnews.nl> References: <51f8036c$0$2995$6d4158fb@reader.xsnews.nl> Message-ID: On Tue, Jul 30, 2013 at 12:18 PM, Bas van der Wulp wrote: > Replacing each occurrence of self._value with either self._value_ or > self.value in the examples seems to make them work as expected. > > Are both examples incorrect, or not intended to work in Python 2.x? The _value attribute was renamed _value_ in: http://hg.python.org/cpython/rev/511c4daac102 It looks like the example wasn't updated to match. You should probably just use self.value here since the name of the private attribute is an implementation detail. From ethan at stoneleaf.us Tue Jul 30 15:30:04 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 30 Jul 2013 12:30:04 -0700 Subject: OrderedEnum examples In-Reply-To: References: <51f8036c$0$2995$6d4158fb@reader.xsnews.nl> Message-ID: <51F8143C.309@stoneleaf.us> On 07/30/2013 11:58 AM, Ian Kelly wrote: > On Tue, Jul 30, 2013 at 12:18 PM, Bas van der Wulp wrote: >> Replacing each occurrence of self._value with either self._value_ or >> self.value in the examples seems to make them work as expected. >> >> Are both examples incorrect, or not intended to work in Python 2.x? > > The _value attribute was renamed _value_ in: > > http://hg.python.org/cpython/rev/511c4daac102 > > It looks like the example wasn't updated to match. You should > probably just use self.value here since the name of the private > attribute is an implementation detail. In `__new__` it has to be `_value_`, but in the other methods `.value` works fine. Updated the 3.4 example with `.value`. -- ~Ethan~ From Bas.vdWulp at gmail.com Tue Jul 30 18:30:28 2013 From: Bas.vdWulp at gmail.com (Bas van der Wulp) Date: Wed, 31 Jul 2013 00:30:28 +0200 Subject: OrderedEnum examples In-Reply-To: References: <51f8036c$0$2995$6d4158fb@reader.xsnews.nl> Message-ID: <51f83e89$0$21818$6d4158fb@reader.xsnews.nl> On 30-7-2013 21:30, Ethan Furman wrote: > On 07/30/2013 11:58 AM, Ian Kelly wrote: >> On Tue, Jul 30, 2013 at 12:18 PM, Bas van der Wulp >> wrote: >>> Replacing each occurrence of self._value with either self._value_ or >>> self.value in the examples seems to make them work as expected. >>> >>> Are both examples incorrect, or not intended to work in Python 2.x? >> >> The _value attribute was renamed _value_ in: >> >> http://hg.python.org/cpython/rev/511c4daac102 >> >> It looks like the example wasn't updated to match. You should >> probably just use self.value here since the name of the private >> attribute is an implementation detail. > > In `__new__` it has to be `_value_`, but in the other methods `.value` > works fine. Updated the 3.4 example with `.value`. > > -- > ~Ethan~ That was quick! Thanks Ethan and Ian. Regards, Bas From ethan at stoneleaf.us Tue Jul 30 14:38:30 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 30 Jul 2013 11:38:30 -0700 Subject: OrderedEnum examples In-Reply-To: <51f8036c$0$2995$6d4158fb@reader.xsnews.nl> References: <51f8036c$0$2995$6d4158fb@reader.xsnews.nl> Message-ID: <51F80826.1080407@stoneleaf.us> On 07/30/2013 11:18 AM, Bas van der Wulp wrote: > Using the enum34 0.9.13 package from PyPi in Python 2.7.3, the examples for OrderedEnum seem to be broken. Thanks for catching that, I'll get it fixed asap. > Also, in the example in the Python 3.4 library documentation (section 8.17.2) has the __ordered__ attribute removed > (presumably because, in contrast to Python 2.x, Python 3 will respect the order of attribute definition). Correct. In 3.4 __ordered__ never came into being as it was not necessary. I added that purely so that 2.x could be ordered if desired. -- ~Ethan~ From ethan at stoneleaf.us Tue Jul 30 15:16:37 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 30 Jul 2013 12:16:37 -0700 Subject: OrderedEnum examples In-Reply-To: <51F80826.1080407@stoneleaf.us> References: <51f8036c$0$2995$6d4158fb@reader.xsnews.nl> <51F80826.1080407@stoneleaf.us> Message-ID: <51F81115.2060003@stoneleaf.us> On 07/30/2013 11:38 AM, Ethan Furman wrote: > > Thanks for catching that, I'll get it fixed asap. Latest code is on PyPI. -- ~Ethan~ From neng.zhou at gmail.com Tue Jul 30 14:33:35 2013 From: neng.zhou at gmail.com (neng.zhou at gmail.com) Date: Tue, 30 Jul 2013 11:33:35 -0700 (PDT) Subject: Announcing Picat, the next scripting language after Python Message-ID: <8353fcd1-5947-4c88-a8ae-b8ee5b63097b@googlegroups.com> We are pleased to announce the launch of the Picat system on picat-lang.org. Picat is a simple, and yet powerful, logic-based multi-paradigm programming language aimed for general-purpose applications. Picat is a rule-based language, in which predicates, functions, and actors are defined with pattern-matching rules. Picat incorporates many declarative language features for better productivity of software development, including explicit non-determinism, explicit unification, functions, list comprehensions, constraints, and tabling. Picat also provides imperative language constructs, such as assignments and loops, for programming everyday things. The Picat implementation, which is based on a well-designed virtual machine and incorporates a memory manager that garbage-collects and expands the stacks and data areas when needed, is efficient and scalable. Picat can be used for not only symbolic computations, which is a traditional application domain of declarative languages, but also for scripting and modeling tasks. Example 1: The following predicate, input_data(Tri), reads rows of integers from the text file "triangle.txt" into an array. This is the first part of a Picat solution for the Euler Project, problem #67 (picat-lang.org/projects.html). import util. input_data(Tri) => Lines = read_file_lines("triangle.txt"), Tri = new_array(Lines.length), I = 1, foreach(Line in Lines) Tri[I] = Line.split().map(to_integer).to_array(), I := I+1 end. The function read_file_lines/1, which is imported by default from the io module, reads all of the lines from a file as a list of strings. For each Line in Lines, the foreach loop splits Line into tokens (using the function split/1, which is imported from the util module), maps the tokens to integers (map(to_integer)), and converts the list to an array (to_array). As illustrated in this example, Picat, as a scripting language, is as powerful as Python and Ruby. Example 2: Given a triangle stored in an array, the following tabled predicate finds the maximum total sum from top to bottom. This is the second part of the Picat solution for the Euler Project, problem #67. table (+,+,max,nt) path(Row,Col,Sum,Tri),Row==Tri.length => Sum=Tri[Row,Col]. path(Row,Col,Sum,Tri) ?=> path(Row+1,Col,Sum1,Tri), Sum = Sum1+Tri[Row,Col]. path(Row,Col,Sum,Tri) => path(Row+1,Col+1,Sum1,Tri), Sum = Sum1+Tri[Row,Col]. Sum = Sum1+Tri[Row,Col]. The first line is a table mode declaration that instructs the system about how to table the calls and answers: '+' means that the argument is tabled, 'max' means that the argument should be maximized, and 'nt' means that the argument is not tabled. This predicate searches for a path with the maximum total sum. If the current row is at the bottom of the triangle, then the leaf value is returned. Otherwise, it makes a non-deterministic choice between two branches, one going straight down and the other going down to the adjacent number. This program is not only compact, but also runs fast. For the 100-row triangle that is provided by the Euler project, this program finds the answer in only 0.01 second. Example 3: The following example models the N-queens problem by using three all_different constraints. import cp. queens3(N, Q) => Q = new_list(N), Q in 1..N, all_different(Q), all_different([$Q[I]-I : I in 1..N]), all_different([$Q[I]+I : I in 1..N]), solve([ff],Q). List comprehensions are used to specify lists. The expressions that are preceded with a dollar sign denote terms rather than function calls. This program uses the CP solver. If the sat module is imported instead of cp, then the SAT solver will be used (and the ff option will be ignored). As demonstrated by the three examples, Picat offers many advantages over other languages. Compared with functional and scripting languages, the support of explicit unification, explicit non-determinism, tabling, and constraints makes Picat more suitable for symbolic computations. Compared with Prolog, Picat is arguably more expressive and scalable: it is not rare to find problems for which Picat requires an order of magnitude fewer lines of code to describe than Prolog and Picat can be significantly faster than Prolog because pattern-matching facilitates indexing of rules. Picat can be used for any fair purpose, including commercial applications. The C source code will be made available to registered developers and users free of charge. Please keep tuned. Sincerely, Neng-Fa Zhou Jonathan Fruhman Hakan Kjellerstrand From ron.eggler at gmail.com Tue Jul 30 16:29:49 2013 From: ron.eggler at gmail.com (cerr) Date: Tue, 30 Jul 2013 13:29:49 -0700 (PDT) Subject: binary key in dictionary Message-ID: <9004a556-958f-4d1d-81a7-4d1b731348c5@googlegroups.com> Hi, In my application I have followingf lines: print curr_mac print hexlify(buf) binmac = unhexlify(curr_mac) tmpgndict[binmac] += buf curr_mac being a 3Byte MAVC address in ASCII and I want to populate a dictionary where the value(buf) is indexed by binary mac. I get this in my code: Traceback (most recent call last): File "gateway.py", line 2485, in main() File "gateway.py", line 2459, in main cloud_check() File "gateway.py", line 770, in cloud_check gnstr_dict[src] = gn_from_cloud(curr_mac) File "gateway.py", line 2103, in gn_from_cloud tmpgndict[binmac] += "HELLO" KeyError: '\x04\xeeu' but then again, the following works fine in the python interpreter: >>> mac = '04ee75' >>> dat = '2a0001016d03c400040001000a' >>> mydict = {} >>> mydict[unhexlify(mac)]=dat >>> print mydict {'\x04\xeeu': '2a0001016d03c400040001000a'} I really seem to do something wrong and can't see what it is. Can anyone help me further here? Thank you very much! Ron From gary.herron at islandtraining.com Tue Jul 30 16:44:47 2013 From: gary.herron at islandtraining.com (Gary Herron) Date: Tue, 30 Jul 2013 13:44:47 -0700 Subject: binary key in dictionary In-Reply-To: <9004a556-958f-4d1d-81a7-4d1b731348c5@googlegroups.com> References: <9004a556-958f-4d1d-81a7-4d1b731348c5@googlegroups.com> Message-ID: <51F825BF.3010008@islandtraining.com> On 07/30/2013 01:29 PM, cerr wrote: > Hi, > > In my application I have followingf lines: > print curr_mac > print hexlify(buf) > binmac = unhexlify(curr_mac) > tmpgndict[binmac] += buf > curr_mac being a 3Byte MAVC address in ASCII and I want to populate a dictionary where the value(buf) is indexed by binary mac. > > I get this in my code: > > Traceback (most recent call last): > File "gateway.py", line 2485, in > main() > File "gateway.py", line 2459, in main > cloud_check() > File "gateway.py", line 770, in cloud_check > gnstr_dict[src] = gn_from_cloud(curr_mac) > File "gateway.py", line 2103, in gn_from_cloud > tmpgndict[binmac] += "HELLO" > KeyError: '\x04\xeeu' > > but then again, the following works fine in the python interpreter: >>>> mac = '04ee75' >>>> dat = '2a0001016d03c400040001000a' >>>> mydict = {} >>>> mydict[unhexlify(mac)]=dat >>>> print mydict > {'\x04\xeeu': '2a0001016d03c400040001000a'} > > I really seem to do something wrong and can't see what it is. Can anyone help me further here? > > Thank you very much! > Ron You are confusing the problem with excess code. Examine the following simpler example which illustrates the problem: >>> d = {} >>> d[1] = 99 >>> d[2] += 98 Traceback (most recent call last): File "", line 1, in KeyError: 2 >>> The line d[1] = 99 creates a key-value pair in the dictionary, but the line d[2] += 98 tries to add 98 to an already existing value at d[2], But there is no value at d[2] until you set it: d[2] = 0 # for instance You may want to look at defaultdict from the collections module. Gary Herron From gordon at panix.com Wed Jul 31 10:26:27 2013 From: gordon at panix.com (John Gordon) Date: Wed, 31 Jul 2013 14:26:27 +0000 (UTC) Subject: binary key in dictionary References: <9004a556-958f-4d1d-81a7-4d1b731348c5@googlegroups.com> Message-ID: In <9004a556-958f-4d1d-81a7-4d1b731348c5 at googlegroups.com> cerr writes: > Traceback (most recent call last): > File "gateway.py", line 2485, in > main() > File "gateway.py", line 2459, in main > cloud_check() > File "gateway.py", line 770, in cloud_check > gnstr_dict[src] = gn_from_cloud(curr_mac) > File "gateway.py", line 2103, in gn_from_cloud > tmpgndict[binmac] += "HELLO" > KeyError: '\x04\xeeu' You're not assigning to tmpgndict[binmac]; you're appending to it. This requires that tmpgndict[binmac] already exists, which it does not. Make sure that tmpgndict[binmac] exists before you try appending to it. -- John Gordon A is for Amy, who fell down the stairs gordon at panix.com B is for Basil, assaulted by bears -- Edward Gorey, "The Gashlycrumb Tinies" From pjenvey at underboss.org Tue Jul 30 16:39:13 2013 From: pjenvey at underboss.org (Philip Jenvey) Date: Tue, 30 Jul 2013 13:39:13 -0700 Subject: PyPy3 2.1 beta 1 released Message-ID: ================ PyPy3 2.1 beta 1 ================ We're pleased to announce the first beta of the upcoming 2.1 release of PyPy3. This is the first release of PyPy which targets Python 3 (3.2.3) compatibility. We would like to thank all of the people who donated_ to the `py3k proposal`_ for supporting the work that went into this and future releases. You can download the PyPy3 2.1 beta 1 release here: http://pypy.org/download.html#pypy3-2-1-beta-1 Highlights ========== * The first release of PyPy3: support for Python 3, targetting CPython 3.2.3! - There are some `known issues`_ including performance regressions (issues `#1540`_ & `#1541`_) slated to be resolved before the final release. What is PyPy? ============== PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7.3 or 3.2.3. It's fast due to its integrated tracing JIT compiler. This release supports x86 machines running Linux 32/64, Mac OS X 64 or Windows 32. Also this release supports ARM machines running Linux 32bit - anything with ``ARMv6`` (like the Raspberry Pi) or ``ARMv7`` (like Beagleboard, Chromebook, Cubieboard, etc.) that supports ``VFPv3`` should work. Cheers, the PyPy team From musicdenotation at gmail.com Tue Jul 30 23:45:36 2013 From: musicdenotation at gmail.com (Musical Notation) Date: Wed, 31 Jul 2013 10:45:36 +0700 Subject: Script that converts between indentation and curly braces in Python code Message-ID: <09DBD544-FA04-48DC-8FEA-1E788FE1C6A6@gmail.com> Is there any script that converts indentation in Python code to curly braces? The indentation is sometime lost when I copy my code to an application or a website. From joel.goldstick at gmail.com Wed Jul 31 08:37:55 2013 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Wed, 31 Jul 2013 08:37:55 -0400 Subject: Script that converts between indentation and curly braces in Python code In-Reply-To: <09DBD544-FA04-48DC-8FEA-1E788FE1C6A6@gmail.com> References: <09DBD544-FA04-48DC-8FEA-1E788FE1C6A6@gmail.com> Message-ID: On Tue, Jul 30, 2013 at 11:45 PM, Musical Notation wrote: > Is there any script that converts indentation in Python code to curly braces? The indentation is sometime lost when I copy my code to an application or a website. I guess you could google that. What do you mean that indentation is 'sometimes' lost when copying. I think you should walk down that path to understand why you are losing your indentation. It could be you are using tabs instead of spaces for indentation. It could also be that you are inserting your code directly into HTML which of course will eat the extra whitespace. That problem can be solved with 
 tags, or maybe even  (but
I am not sure about that)

If you are copying code to a website like stack exchange you need to
precede each line of code with  (2 I think!) spaces.  They use
markdown.

If you wanted to add in curly braces, I am guesses the only purpose
for this would be that you would convert them back to proper
indentation in the copied to environment.  So then you have to figure
out a way to do that.



> --
> http://mail.python.org/mailman/listinfo/python-list



-- 
Joel Goldstick
http://joelgoldstick.com


From beth.mcnany at gmail.com  Wed Jul 31 08:39:45 2013
From: beth.mcnany at gmail.com (Beth McNany)
Date: Wed, 31 Jul 2013 08:39:45 -0400
Subject: Script that converts between indentation and curly braces in
	Python code
In-Reply-To: <09DBD544-FA04-48DC-8FEA-1E788FE1C6A6@gmail.com>
References: <09DBD544-FA04-48DC-8FEA-1E788FE1C6A6@gmail.com>
Message-ID: 

from __future__ import braces ;)

ok, ok, if you *really* want it, you could keep track of how many leading
spaces there are (you are using spaces, right?), and insert an open bracket
where that number increases and a closing bracket where it decreases.  Of
course, as with all parsing problems, this is oversimplification... if you
have multi-line statements you'll need to check for those (if line starts
with """ or ''', ends with \, or if there's an unclosed bracket or
paren...) - but that'd be a reasonable place to start if you're only doing
short code snippets.

Also, I have to ask, how to you intend this to be used?  If there's no
indentation it will still be difficult to read, and your readers will then
have the problem of figuring out which curly braces to remove (for simple
code this might be easy, but for longer things you'd probably want an
automated way to re-indent, and a simple search-and-replace will screw up
any dictionaries, for instance).

-beth

On Tue, Jul 30, 2013 at 11:45 PM, Musical Notation <
musicdenotation at gmail.com> wrote:

> Is there any script that converts indentation in Python code to curly
> braces? The indentation is sometime lost when I copy my code to an
> application or a website.
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 

From rosuav at gmail.com  Wed Jul 31 09:55:51 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 31 Jul 2013 14:55:51 +0100
Subject: Script that converts between indentation and curly braces in
	Python code
In-Reply-To: 
References: <09DBD544-FA04-48DC-8FEA-1E788FE1C6A6@gmail.com>
	
Message-ID: 

On Wed, Jul 31, 2013 at 1:39 PM, Beth McNany  wrote:
> ok, ok, if you *really* want it, you could keep track of how many leading
> spaces there are (you are using spaces, right?), and insert an open bracket
> where that number increases and a closing bracket where it decreases.  Of
> course, as with all parsing problems, this is oversimplification... if you
> have multi-line statements you'll need to check for those (if line starts
> with """ or ''', ends with \, or if there's an unclosed bracket or paren...)
> - but that'd be a reasonable place to start if you're only doing short code
> snippets.
>

Since the braced version won't run anyway, how about a translation like this:

def foo():
    print("""Hello,
world!""")
    for i in range(5):
        foo()
    return 42

-->

0-def foo():
4-print("""Hello,
0-world!""")
4-for i in range(5):
8-foo()
4-return 42

That's a simple translation that guarantees safe round-tripping, and
you can probably do it with a one-liner fwiw... let's see...

# Assumes spaces OR tabs but not both
# Can't see an easy way to count leading spaces other than:
# len(s)-len(s.lstrip())
code = '\n'.join("%d-%s"%(len(s)-len(s.lstrip()),s.lstrip()) for s in
code.split('\n'))

# Recreates with spaces, choose tabs for the multiplication if you prefer
code = '\n'.join(' '*int(s.split('-',1)[0])+s.split('-',1)[1] for s in
code.split('\n'))

These would be better done in a couple of lines, but I like doing
one-liners just for fun. :)

ChrisA


From joel.goldstick at gmail.com  Wed Jul 31 10:19:12 2013
From: joel.goldstick at gmail.com (Joel Goldstick)
Date: Wed, 31 Jul 2013 10:19:12 -0400
Subject: Script that converts between indentation and curly braces in
	Python code
In-Reply-To: 
References: <09DBD544-FA04-48DC-8FEA-1E788FE1C6A6@gmail.com>
	
	
Message-ID: 

So, why do you want to do this?  As has been pointed out, its a
difficult and likely sizable task to build such a parser.  In the end
you get something that isn't a computer language -- even tho it looks
like one.  And it also is probably just as big a job to convert it
back to python.  So, what is the point?

Pardon me for making assumptions, but this seems to be a variation on
the theme of 'I don't like python, it doesn't have curly braces'.  So
if you need curly braces, you have many other languages to choose
from.  Python isn't one of them.


From sg552 at hotmail.co.uk  Wed Jul 31 10:07:05 2013
From: sg552 at hotmail.co.uk (Rotwang)
Date: Wed, 31 Jul 2013 15:07:05 +0100
Subject: Script that converts between indentation and curly braces in
	Python code
In-Reply-To: 
References: <09DBD544-FA04-48DC-8FEA-1E788FE1C6A6@gmail.com>
	
	
Message-ID: 

On 31/07/2013 14:55, Chris Angelico wrote:
> [...]
>
>
> Since the braced version won't run anyway, how about a translation like this:
>
> def foo():
>      print("""Hello,
> world!""")
>      for i in range(5):
>          foo()
>      return 42
>
> -->
>
> 0-def foo():
> 4-print("""Hello,
> 0-world!""")
> 4-for i in range(5):
> 8-foo()
> 4-return 42
>
> That's a simple translation that guarantees safe round-tripping, and
> you can probably do it with a one-liner fwiw... let's see...
>
> # Assumes spaces OR tabs but not both
> # Can't see an easy way to count leading spaces other than:
> # len(s)-len(s.lstrip())

How about len(s.expandtabs()) - len(s.lstrip()) instead?


From rosuav at gmail.com  Wed Jul 31 10:23:54 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 31 Jul 2013 15:23:54 +0100
Subject: Script that converts between indentation and curly braces in
	Python code
In-Reply-To: 
References: <09DBD544-FA04-48DC-8FEA-1E788FE1C6A6@gmail.com>
	
	
	
Message-ID: 

On Wed, Jul 31, 2013 at 3:07 PM, Rotwang  wrote:
>> # Assumes spaces OR tabs but not both
>> # Can't see an easy way to count leading spaces other than:
>> # len(s)-len(s.lstrip())
>
>
> How about len(s.expandtabs()) - len(s.lstrip()) instead?

Still comes to the same thing. The only diff is that tabs get treated
as eight spaces instead of one (and the bug that a tab elsewhere in
the line will result in indentation, which is fixed by lstripping the
tab-expanded form). It won't perfectly round-trip with a mixture of
tabs and spaces; as it is, you can pick one or the other and run with
it. Anyway, the main point is that indentation will work. Sure you
might have ugly narrow code, but it'll run with one-space indents.

ChrisA


From nhodgson at iinet.net.au  Wed Jul 31 09:20:01 2013
From: nhodgson at iinet.net.au (Neil Hodgson)
Date: Wed, 31 Jul 2013 23:20:01 +1000
Subject: Script that converts between indentation and curly braces in
	Python code
In-Reply-To: 
References: 
Message-ID: 

Musical Notation:

> Is there any script that converts indentation in Python code to curly braces?
> The indentation is sometime lost when I copy my code to an application or a website.

    pindent.py in the Tools/Scripts directory of Python installations 
does something similar by adding or removing comments that look like

# end if

    Neil


From storchaka at gmail.com  Wed Jul 31 09:31:45 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 31 Jul 2013 16:31:45 +0300
Subject: Script that converts between indentation and curly braces in
	Python code
In-Reply-To: <09DBD544-FA04-48DC-8FEA-1E788FE1C6A6@gmail.com>
References: <09DBD544-FA04-48DC-8FEA-1E788FE1C6A6@gmail.com>
Message-ID: 

31.07.13 06:45, Musical Notation ???????(??):
> Is there any script that converts indentation in Python code to curly braces? The indentation is sometime lost when I copy my code to an application or a website.

Look at the pindent.py script.



From ishish at domhain.de  Wed Jul 31 09:13:53 2013
From: ishish at domhain.de (IshIsh)
Date: Wed, 31 Jul 2013 14:13:53 +0100
Subject: Script that converts between indentation and curly braces in
	Python code
Message-ID: <51F90D91.2090800@domhain.de>

from __future__ import braces is just an easter egg... ...and shouldn't 
it better state from __past__ import braces ;-)

Anyway, as far as I know the IPython interpreter can recognize lines 
ending in ?:? and indent the next line, while also un-indenting 
automatically after ?raise? or ?return?.

=> sas


From musicdenotation at gmail.com  Wed Jul 31 09:40:38 2013
From: musicdenotation at gmail.com (Musical Notation)
Date: Wed, 31 Jul 2013 20:40:38 +0700
Subject: Script that converts between indentation and curly braces in
	Python code
In-Reply-To: <51F902A3.7030900@domhain.de>
References: <09DBD544-FA04-48DC-8FEA-1E788FE1C6A6@gmail.com>
	<51F902A3.7030900@domhain.de>
Message-ID: <3E67969D-5462-4DEF-88B4-8F97F366C466@gmail.com>

On Jul 31, 2013, at 19:27, IshIsh  wrote:

> Try from __future__ import braces as the first line of a source file (or typing it in an interactive     session), and watch the interpreter's response...
"SyntaxError: not a chance" I already know that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 

From musicdenotation at gmail.com  Wed Jul 31 02:53:26 2013
From: musicdenotation at gmail.com (Musical Notation)
Date: Wed, 31 Jul 2013 13:53:26 +0700
Subject: Lambda function Turing completeness
Message-ID: <5CB71036-C359-4211-8B3B-62B17AACF88E@gmail.com>

Is it possible to write a Turing-complete lambda function (which does not depend on named functions) in Python?

From js at globe.de  Wed Jul 31 09:19:29 2013
From: js at globe.de (Schneider)
Date: Wed, 31 Jul 2013 15:19:29 +0200
Subject: Lambda function Turing completeness
In-Reply-To: <5CB71036-C359-4211-8B3B-62B17AACF88E@gmail.com>
References: <5CB71036-C359-4211-8B3B-62B17AACF88E@gmail.com>
Message-ID: <51F90EE1.4060406@globe.de>

On Wed 31 Jul 2013 08:53:26 AM CEST, Musical Notation wrote:
> Is it possible to write a Turing-complete lambda function (which does not depend on named functions) in Python?

what should a sinlge Turing-complete lambda function be?
For me, a programming language can be Turing-complete or a function can 
be universal,  e.g. like an interpreter for a  programming language.

bg,
Johannes

--
GLOBE Development GmbH
K?nigsberger Strasse 260
48157 M?nsterGLOBE Development GmbH
K?nigsberger Strasse 260
48157 M?nster
0251/5205 390


From ian.g.kelly at gmail.com  Wed Jul 31 13:07:45 2013
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Wed, 31 Jul 2013 11:07:45 -0600
Subject: Lambda function Turing completeness
In-Reply-To: <5CB71036-C359-4211-8B3B-62B17AACF88E@gmail.com>
References: <5CB71036-C359-4211-8B3B-62B17AACF88E@gmail.com>
Message-ID: 

On Wed, Jul 31, 2013 at 12:53 AM, Musical Notation
 wrote:
> Is it possible to write a Turing-complete lambda function (which does not depend on named functions) in Python?

Yes, lambda functions are Turing-complete.  You can get anonymous
recursion by defining the function to take a recursive function
argument and then passing it to itself.  For example, this will
(inefficiently) give you the 13th Fibonacci number:

(lambda f, n: f(f, n))(lambda f, n: n if n < 2 else f(f, n-2) + f(f, n-1), 13)


From frank at chagford.com  Wed Jul 31 05:44:01 2013
From: frank at chagford.com (Frank Millman)
Date: Wed, 31 Jul 2013 11:44:01 +0200
Subject: Problem with psycopg2, bytea, and memoryview
Message-ID: 

Hi all

I don't know if this question is more appropriate for the psycopg2 list, but 
I thought I would ask here first.

I have some binary data (a gzipped xml object) that I want to store in a 
database. For PostgreSQL I use a column with datatype 'bytea', which is 
their recommended way of storing binary strings.

I use psycopg2 to access the database. It returns binary data in the form of 
a python 'memoryview'.

My problem is that, after a roundtrip to the database and back, the object 
no longer compares equal to the original.

>>> memoryview(b'abcdef') == b'abcdef'
True
>>> cur.execute('create table fmtemp (code int, xml bytea)')
>>> cur.execute('insert into fmtemp values (%s, %s)', (1, b'abcdef'))
>>> cur.execute('select * from fmtemp where code =1')
>>> row = cur.fetchone()
>>> row
(1, )
>>> row[1] == b'abcdef'
False
>>> row[1].tobytes() == b'abcdef'
True
>>>

Using MS SQL Server and pyodbc, it returns a byte string, not a memoryview, 
and it does compare equal with the original.

I can hack my program to use tobytes(), but it would add complication, and 
it would be database-specific. I would prefer a cleaner solution.

Does anyone have any suggestions?

Versions - Python: 3.3.2  PostgreSQL: 9.2.4  psycopg2: 2.5

Frank Millman





From solipsis at pitrou.net  Wed Jul 31 05:50:37 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 31 Jul 2013 09:50:37 +0000 (UTC)
Subject: Problem with psycopg2, bytea, and memoryview
References: 
Message-ID: 

Frank Millman  chagford.com> writes:
> 
> I have some binary data (a gzipped xml object) that I want to store in a 
> database. For PostgreSQL I use a column with datatype 'bytea', which is 
> their recommended way of storing binary strings.
> 
> I use psycopg2 to access the database. It returns binary data in the form of 
> a python 'memoryview'.
> 
[...]
> 
> Using MS SQL Server and pyodbc, it returns a byte string, not a memoryview, 
> and it does compare equal with the original.
> 
> I can hack my program to use tobytes(), but it would add complication, and 
> it would be database-specific. I would prefer a cleaner solution.

Just cast the result to bytes (`bytes(row[1])`). It will work both with bytes
and memoryview objcts.

Regards

Antoine.




From frank at chagford.com  Wed Jul 31 07:43:05 2013
From: frank at chagford.com (Frank Millman)
Date: Wed, 31 Jul 2013 13:43:05 +0200
Subject: Problem with psycopg2, bytea, and memoryview
References: 
	
Message-ID: 


"Antoine Pitrou"  wrote in message 
news:loom.20130731T114936-455 at post.gmane.org...
> Frank Millman  chagford.com> writes:
>>
>> I have some binary data (a gzipped xml object) that I want to store in a
>> database. For PostgreSQL I use a column with datatype 'bytea', which is
>> their recommended way of storing binary strings.
>>
>> I use psycopg2 to access the database. It returns binary data in the form 
>> of
>> a python 'memoryview'.
>>
> [...]
>>
>> Using MS SQL Server and pyodbc, it returns a byte string, not a 
>> memoryview,
>> and it does compare equal with the original.
>>
>> I can hack my program to use tobytes(), but it would add complication, 
>> and
>> it would be database-specific. I would prefer a cleaner solution.
>
> Just cast the result to bytes (`bytes(row[1])`). It will work both with 
> bytes
> and memoryview objcts.
>
> Regards
>
> Antoine.
>

Thanks for that, Antoine. It is an improvement over tobytes(), but i am 
afraid it is still not ideal for my purposes.

At present, I loop over a range of columns, comparing 'before' and 'after' 
values, without worrying about their types. Strings are returned as str, 
integers are returned as int, etc. Now I will have to check the type of each 
column before deciding whether to cast to 'bytes'.

Can anyone explain *why* the results do not compare equal? If I understood 
the problem, I might be able to find a workaround.

Frank





From solipsis at pitrou.net  Wed Jul 31 09:07:33 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 31 Jul 2013 13:07:33 +0000 (UTC)
Subject: Problem with psycopg2, bytea, and memoryview
References: 
	
	
Message-ID: 

Frank Millman  chagford.com> writes:
> 
> Thanks for that, Antoine. It is an improvement over tobytes(), but i am 
> afraid it is still not ideal for my purposes.

I would suggest asking the psycopg2 project why they made this choice, and
if they would reconsider. Returning a memoryview doesn't make much sense IMHO.

For example, the standard sqlite3 module returns bytes for BLOB columns,
and str for TEXT columns:
http://docs.python.org/3.4/library/sqlite3.html#introduction

> Can anyone explain *why* the results do not compare equal? If I understood 
> the problem, I might be able to find a workaround.

Well, under recent Python versions, they should compare equal:

Python 3.2.3 (default, Oct 19 2012, 19:53:16) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> memoryview(b"abc") == b"abc"
True


Regards

Antoine.




From frank at chagford.com  Wed Jul 31 09:41:41 2013
From: frank at chagford.com (Frank Millman)
Date: Wed, 31 Jul 2013 15:41:41 +0200
Subject: Problem with psycopg2, bytea, and memoryview
References: 
	
Message-ID: 


"Antoine Pitrou"  wrote in message 
news:loom.20130731T150154-520 at post.gmane.org...
> Frank Millman  chagford.com> writes:
>>
>> Thanks for that, Antoine. It is an improvement over tobytes(), but i am
>> afraid it is still not ideal for my purposes.
>
> I would suggest asking the psycopg2 project why they made this choice, and
> if they would reconsider. Returning a memoryview doesn't make much sense 
> IMHO.
>

I'll try it, and see what they say.

> For example, the standard sqlite3 module returns bytes for BLOB columns,
> and str for TEXT columns:
> http://docs.python.org/3.4/library/sqlite3.html#introduction
>
>> Can anyone explain *why* the results do not compare equal? If I 
>> understood
>> the problem, I might be able to find a workaround.
>
> Well, under recent Python versions, they should compare equal:
>
> Python 3.2.3 (default, Oct 19 2012, 19:53:16)
> [GCC 4.7.2] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> memoryview(b"abc") == b"abc"
> True
>

I am using Python 3.3.2.

If I try your example above, it does work.

However, for some reason, after a round-trip to the server, they do not 
compare equal.

See my original post for a full example.

Frank





From tjreedy at udel.edu  Wed Jul 31 13:59:20 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 31 Jul 2013 13:59:20 -0400
Subject: Problem with psycopg2, bytea, and memoryview
In-Reply-To: 
References: 
	
	
	
Message-ID: 

On 7/31/2013 9:07 AM, Antoine Pitrou wrote:
> Frank Millman  chagford.com> writes:
>>
>> Thanks for that, Antoine. It is an improvement over tobytes(), but i am
>> afraid it is still not ideal for my purposes.
>
> I would suggest asking the psycopg2 project why they made this choice, and
> if they would reconsider. Returning a memoryview doesn't make much sense IMHO.

I agree.
"memoryview objects allow Python code to access the internal data of an 
object that supports the buffer protocol without copying."
Example: the binary image data of an image object.
They are not intended to be a standalone objects when there is an 
obvious alternative (in this case, bytes).

-- 
Terry Jan Reedy



From neilc at norwich.edu  Wed Jul 31 10:08:44 2013
From: neilc at norwich.edu (Neil Cerutti)
Date: 31 Jul 2013 14:08:44 GMT
Subject: Problem with psycopg2, bytea, and memoryview
References: 
	
	
Message-ID: 

On 2013-07-31, Frank Millman  wrote:
>
> "Antoine Pitrou"  wrote in message 
> news:loom.20130731T114936-455 at post.gmane.org...
>> Frank Millman  chagford.com> writes:
>>>
>>> I have some binary data (a gzipped xml object) that I want to store in a
>>> database. For PostgreSQL I use a column with datatype 'bytea', which is
>>> their recommended way of storing binary strings.
>>>
>>> I use psycopg2 to access the database. It returns binary data
>>> in the form of a python 'memoryview'.
>>>
>> [...]
>>>
>>> Using MS SQL Server and pyodbc, it returns a byte string, not
>>> a memoryview, and it does compare equal with the original.
>>>
>>> I can hack my program to use tobytes(), but it would add
>>> complication, and it would be database-specific. I would
>>> prefer a cleaner solution.
>>
>> Just cast the result to bytes (`bytes(row[1])`). It will work
>> both with bytes and memoryview objcts.
>
> Thanks for that, Antoine. It is an improvement over tobytes(),
> but i am afraid it is still not ideal for my purposes.
>
> At present, I loop over a range of columns, comparing 'before'
> and 'after' values, without worrying about their types. Strings
> are returned as str, integers are returned as int, etc. Now I
> will have to check the type of each column before deciding
> whether to cast to 'bytes'.
>
> Can anyone explain *why* the results do not compare equal? If I
> understood the problem, I might be able to find a workaround.

A memoryview will compare equal to another object that supports
the buffer protocol when the format and shape are also equal. The
database must be returning chunks of binary data in a different
shape or format than you are writing it.

Perhaps psycopg2 is returning a chunk of ints when you have
written a chunk of bytes. Check the .format and .shape members of
the return value to see.

>>> x = memoryview(b"12345")
>>> x.format
'B'
>>> x.shape
(5,)
>>> x == b"12345"
True

My guess is you're getting format "I" from psycopg2. Hopefully
there's a way to coerce your desired "B" format interpretation of
the raw data using psycopg2's API.

-- 
Neil Cerutti


From urbangabo at gmail.com  Wed Jul 31 06:53:48 2013
From: urbangabo at gmail.com (Gabor Urban)
Date: Wed, 31 Jul 2013 12:53:48 +0200
Subject: Repository of non-standard modules.
Message-ID: 

Hi,

I am to start a new free-time project in the next couple of weeks. I am
ready to use open accessible Python modules not wanting to reinvent the
weel :-)
Is there any repository where I can find Python modules not being part of
the standard distribution?

I have some hits by Google but that seems to be far from complete.

Thanks in advance,

Gabor Urban

-- 
Urb?n G?bor

Linux is like a wigwam: no Gates, no Windows and an Apache inside.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 

From rosuav at gmail.com  Wed Jul 31 07:02:04 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 31 Jul 2013 12:02:04 +0100
Subject: Repository of non-standard modules.
In-Reply-To: 
References: 
Message-ID: 

On Wed, Jul 31, 2013 at 11:53 AM, Gabor Urban  wrote:
> Hi,
>
> I am to start a new free-time project in the next couple of weeks. I am
> ready to use open accessible Python modules not wanting to reinvent the weel
> :-)
> Is there any repository where I can find Python modules not being part of
> the standard distribution?

Check out the Python Package Index:

http://pypi.python.org/

Lots of stuff there, not all of it actively developed.

ChrisA


From ian.g.kelly at gmail.com  Wed Jul 31 12:54:46 2013
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Wed, 31 Jul 2013 10:54:46 -0600
Subject: Repository of non-standard modules.
In-Reply-To: 
References: 
	
Message-ID: 

On Wed, Jul 31, 2013 at 5:02 AM, Chris Angelico  wrote:
> On Wed, Jul 31, 2013 at 11:53 AM, Gabor Urban  wrote:
>> Hi,
>>
>> I am to start a new free-time project in the next couple of weeks. I am
>> ready to use open accessible Python modules not wanting to reinvent the weel
>> :-)
>> Is there any repository where I can find Python modules not being part of
>> the standard distribution?
>
> Check out the Python Package Index:
>
> http://pypi.python.org/

There's also the somewhat prettier crate.io.


From wachkama at gmail.com  Wed Jul 31 11:33:25 2013
From: wachkama at gmail.com (wachkama at gmail.com)
Date: Wed, 31 Jul 2013 08:33:25 -0700 (PDT)
Subject: script to Login a website
Message-ID: 


I have created a script to log in a website. It gets its username and password from two files, then log's in with this credentials. My code is not showing me what username it is using to login from the file. And I am not sure if it is even opening up the url and prompting for login. I am stuck can someone help me ?





    import urllib, urllib2

    user = open ('users.txt' , 'r')
    password = open ('password.txt' , 'r')

    for users in user:
        password.seek(0)
        for pass_list in password:
            login_data = users + '\n' + pass_list
            print login_data
           
    base_url = 'http://mysite.com'       
    #login action we want to post data to 
    response = urllib2.Request(base_url)     
    login_action = '/auth/login'   
    login_action = base_url + login_action
    response = urllib2.urlopen(login_action, login_data)
    response.read()
    print response.headers
    print response.getcode()



From gordon at panix.com  Wed Jul 31 12:21:59 2013
From: gordon at panix.com (John Gordon)
Date: Wed, 31 Jul 2013 16:21:59 +0000 (UTC)
Subject: script to Login a website
References: 
Message-ID: 

In  wachkama at gmail.com writes:

> I have created a script to log in a website. It gets its username and
> password from two files, then log's in with this credentials. My code is
> not showing me what username it is using to login from the file. And I am
> not sure if it is even opening up the url and prompting for login. I am
> stuck can someone help me ?

How is the data in 'users.txt' and 'password.txt' organized?  Given the
filenames, I would expect that 'users.txt' contains one username on each
line, and 'password.txt' contains one password on each line, with the
first username belonging with the first password, the second username
belonging with the second password, and so on.

Is this correct?

If so, that raises the question of how to handle multiple usernames and
passwords.  Do you just want to use one, or are you supposed to use them
all somehow?

Anyway, to begin to solve your problem, I'd copy just the file-reading code
into a separate program and add plenty of print statements to make sure it
works correctly.  Once you have that fixed, then you can worry about the
web login stuff.

And when you read the contents of each file, be aware that the newlines
at the end of each line are included.  If you don't want these, be sure
to call the rstrip() method to remove traling whitespace.

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon at panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"



From wachkama at gmail.com  Wed Jul 31 12:50:45 2013
From: wachkama at gmail.com (wachkama at gmail.com)
Date: Wed, 31 Jul 2013 09:50:45 -0700 (PDT)
Subject: script to Login a website
In-Reply-To: 
References: 
	
Message-ID: <0bf56e63-a6d3-4c7a-a4e9-642351081311@googlegroups.com>

On Wednesday, July 31, 2013 12:21:59 PM UTC-4, John Gordon wrote:
> In  wachkama at gmail.com writes:
> 
> 
> 
> > I have created a script to log in a website. It gets its username and
> 
> > password from two files, then log's in with this credentials. My code is
> 
> > not showing me what username it is using to login from the file. And I am
> 
> > not sure if it is even opening up the url and prompting for login. I am
> 
> > stuck can someone help me ?
> 
> 
> 
> How is the data in 'users.txt' and 'password.txt' organized?  Given the
> 
> filenames, I would expect that 'users.txt' contains one username on each
> 
> line, and 'password.txt' contains one password on each line, with the
> 
> first username belonging with the first password, the second username
> 
> belonging with the second password, and so on.
> 
> 
> 
> Is this correct?
> 
> 
> 
> If so, that raises the question of how to handle multiple usernames and
> 
> passwords.  Do you just want to use one, or are you supposed to use them
> 
> all somehow?
> 
> 
> 
> Anyway, to begin to solve your problem, I'd copy just the file-reading code
> 
> into a separate program and add plenty of print statements to make sure it
> 
> works correctly.  Once you have that fixed, then you can worry about the
> 
> web login stuff.
> 
> 
> 
> And when you read the contents of each file, be aware that the newlines
> 
> at the end of each line are included.  If you don't want these, be sure
> 
> to call the rstrip() method to remove traling whitespace.
> 
> 
> 
> -- 
> 
> John Gordon                   A is for Amy, who fell down the stairs
> 
> gordon at panix.com              B is for Basil, assaulted by bears
> 
>                                 -- Edward Gorey, "The Gashlycrumb Tinies"

Hi John 
let me answer your questions 
the user.txt file has one user name on each line and so does the password.txt. with this in mind each user will attempt to log in with all the passwords on password.txt file. when it gets to the end of the line it will go to the next user in users.txt and do the same i.e attempt to log in with all the passwords in the file.
So to your second question I will use all the users one after the other attempting to log in.
I am able to get each user to use "x" number of passwords in the password.txt that part works fine. The login section is where I am stuck ?

Sam


From joel.goldstick at gmail.com  Wed Jul 31 16:39:27 2013
From: joel.goldstick at gmail.com (Joel Goldstick)
Date: Wed, 31 Jul 2013 16:39:27 -0400
Subject: script to Login a website
In-Reply-To: 
References: 
Message-ID: 

On Wed, Jul 31, 2013 at 11:33 AM,   wrote:
>
> I have created a script to log in a website. It gets its username and password from two files, then log's in with this credentials. My code is not showing me what username it is using to login from the file. And I am not sure if it is even opening up the url and prompting for login. I am stuck can someone help me ?
>
>
>
>
>
>     import urllib, urllib2
>
>     user = open ('users.txt' , 'r')
>     password = open ('password.txt' , 'r')
>
>     for users in user:
>         password.seek(0)
>         for pass_list in password:
>             login_data = users + '\n' + pass_list
>             print login_data
>
I think you will note that login_data is overwritten each password
loop.  In the end of all of the above you have the last users followed
by a newline, followed by the last pass_list
You might want to think about putting the code below in a function
that can be called after print login_data above if you want to check
each username and password combination.


>     base_url = 'http://mysite.com'
>     #login action we want to post data to
>     response = urllib2.Request(base_url)
>     login_action = '/auth/login'
>     login_action = base_url + login_action
>     response = urllib2.urlopen(login_action, login_data)

I don't think the above line provides login_data as specified by the
spec:   http://docs.python.org/2/library/urllib2.html#module-urllib2

It looks like data needs to be tuples

>     response.read()
>     print response.headers
>     print response.getcode()
>
> --

Once you correct the top of your code I recommend Requests module
since its easier to understand, simpler, and better documented than
the standard url stuff.  You can find it at
http://docs.python-requests.org/en/latest/

> http://mail.python.org/mailman/listinfo/python-list



-- 
Joel Goldstick
http://joelgoldstick.com


From jaiprakashsingh213 at gmail.com  Wed Jul 31 13:51:52 2013
From: jaiprakashsingh213 at gmail.com (Jaiky)
Date: Wed, 31 Jul 2013 10:51:52 -0700 (PDT)
Subject: ImportError: No module named appengine.ext
Message-ID: <719f0bd8-cddc-4b28-97ee-08b56d359ec6@googlegroups.com>

hey learning python 

problem facing is under when typing on interpreter


>>> from google.appengine.ext import db
Traceback (most recent call last):
  File "", line 1, in 
ImportError: No module named appengine.ext


what to do please help .....


From gordon at panix.com  Wed Jul 31 14:20:39 2013
From: gordon at panix.com (John Gordon)
Date: Wed, 31 Jul 2013 18:20:39 +0000 (UTC)
Subject: ImportError: No module named appengine.ext
References: <719f0bd8-cddc-4b28-97ee-08b56d359ec6@googlegroups.com>
Message-ID: 

In <719f0bd8-cddc-4b28-97ee-08b56d359ec6 at googlegroups.com> Jaiky  writes:

> >>> from google.appengine.ext import db
> Traceback (most recent call last):
>   File "", line 1, in 
> ImportError: No module named appengine.ext

> what to do please help .....

Has the Google App Engine library been installed on your system?

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon at panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"



From jaiprakashsingh213 at gmail.com  Wed Jul 31 14:34:13 2013
From: jaiprakashsingh213 at gmail.com (Jaiky)
Date: Wed, 31 Jul 2013 11:34:13 -0700 (PDT)
Subject: ImportError: No module named appengine.ext
In-Reply-To: 
References: <719f0bd8-cddc-4b28-97ee-08b56d359ec6@googlegroups.com>
	
Message-ID: <8d670cc6-03a9-42b2-8b86-eecf0b573d93@googlegroups.com>

you mean to say SDK for python ?????????????/



From kwpolska at gmail.com  Wed Jul 31 14:28:16 2013
From: kwpolska at gmail.com (=?UTF-8?B?Q2hyaXMg4oCcS3dwb2xza2HigJ0gV2Fycmljaw==?=)
Date: Wed, 31 Jul 2013 20:28:16 +0200
Subject: ImportError: No module named appengine.ext
In-Reply-To: <719f0bd8-cddc-4b28-97ee-08b56d359ec6@googlegroups.com>
References: <719f0bd8-cddc-4b28-97ee-08b56d359ec6@googlegroups.com>
Message-ID: 

On Wed, Jul 31, 2013 at 7:51 PM, Jaiky  wrote:
> hey learning python
>
> problem facing is under when typing on interpreter
>
>
>>>> from google.appengine.ext import db
> Traceback (most recent call last):
>   File "", line 1, in 
> ImportError: No module named appengine.ext
>
>
> what to do please help .....
> --
> http://mail.python.org/mailman/listinfo/python-list

You do not have the Google AppEngine packages installed on your
system.  You may want to get the AppEngine SDK or the AppEngine
development server.
-- 
Chris ?Kwpolska? Warrick 
PGP: 5EAAEA16
stop html mail | always bottom-post | only UTF-8 makes sense


From memilanuk at gmail.com  Wed Jul 31 14:35:44 2013
From: memilanuk at gmail.com (memilanuk)
Date: Wed, 31 Jul 2013 11:35:44 -0700
Subject: Using system python vs. updated/current version
Message-ID: 

Hello there,

What would be considered the correct/best way to run a current release
of python locally vs. the installed system version?  On openSUSE 12.3,
the repos currently have 2.7.3 and 3.3.0.  As far as I know, I'm not
really hitting any limitations with the existing versions - my skills
just aren't that far along - so its not a burning 'need' but I'm still
curious/interested in the topic.

Also... in some places in the 'Net I see references to installing
everything 'locally' via pip, etc. in virtualenvs and not touching the
system installed version of python... yet most linux distros seem to
have many/most such packages available in their package repos, which
seems like it'd be easier to install via the package manager and let it
keep things updated.  Could someone touch on what they feel the pros and
cons would be either way?

Thanks,

Monte



From ramit.prasad at jpmorgan.com.dmarc.invalid  Wed Jul 31 15:17:34 2013
From: ramit.prasad at jpmorgan.com.dmarc.invalid (Prasad, Ramit)
Date: Wed, 31 Jul 2013 19:17:34 +0000
Subject: Using system python vs. updated/current version
In-Reply-To: 
References: 
Message-ID: <5B80DD153D7D744689F57F4FB69AF47418603BE1@SCACMX008.exchad.jpmchase.net>

memilanuk wrote:
> Hello there,
> 
> What would be considered the correct/best way to run a current release
> of python locally vs. the installed system version?  On openSUSE 12.3,
> the repos currently have 2.7.3 and 3.3.0.  As far as I know, I'm not
> really hitting any limitations with the existing versions - my skills
> just aren't that far along - so its not a burning 'need' but I'm still
> curious/interested in the topic.

You should be able to install both Python 2 and 3 in most modern
Linux distributions (at the same time). I would not change the system
Python version.

If you are not blocked from running Python 3 by some necessary
dependency then you should use it. Otherwise, use the most
recent Python version you can. 

> 
> Also... in some places in the 'Net I see references to installing
> everything 'locally' via pip, etc. in virtualenvs and not touching the
> system installed version of python... yet most linux distros seem to
> have many/most such packages available in their package repos, which
> seems like it'd be easier to install via the package manager and let it
> keep things updated.  Could someone touch on what they feel the pros and
> cons would be either way?

Virtual envs are great if you work on multiple projects and want to 
keep each project's dependencies separate and manageable. This
will let you change dependencies to a newer version based on project 
rather than having to change the dependency for all projects.

Not to mention this also allows you to install packages on hosts
that you do not have access to install them to the system
packages directory.

On a personal machine, I would install some things like pip/virtualenv 
(/numpy maybe) to system packages directory but keep most packages in 
a project specific directory (i.e. virtualenv).


> 
> Thanks,
> 
> Monte



This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email.  


From memilanuk at gmail.com  Wed Jul 31 16:19:53 2013
From: memilanuk at gmail.com (memilanuk)
Date: Wed, 31 Jul 2013 13:19:53 -0700
Subject: Using system python vs. updated/current version
In-Reply-To: <5B80DD153D7D744689F57F4FB69AF47418603BE1@SCACMX008.exchad.jpmchase.net>
References: 
	<5B80DD153D7D744689F57F4FB69AF47418603BE1@SCACMX008.exchad.jpmchase.net>
Message-ID: 

On 07/31/2013 12:17 PM, Prasad, Ramit wrote:
> You should be able to install both Python 2 and 3 in most modern
> Linux distributions (at the same time). I would not change the system
> Python version.

I hadn't really planned on mucking with the system python... I recall
from a long while back (on Mac OSX) as that being a Bad Thing ;)

But that is kind of (one of) the question(s) here... I presume it is
'possible' to run a local version of python, installed in the user's
home directory... just curious if its worth the hassle.

> If you are not blocked from running Python 3 by some necessary
> dependency then you should use it. Otherwise, use the most
> recent Python version you can. 

Are there any significant flaws with v.3.3.0 that would necessitate
upgrading to the most recent version (3.3.2?)

The only 'dependency' I have as far as 2.7.x is concerned is that I've
become rather accustomed to using spyder (IDE)... which at this point
doesn't support python3 - definitely a bummer.

> On a personal machine, I would install some things like pip/virtualenv 
> (/numpy maybe) to system packages directory but keep most packages in 
> a project specific directory (i.e. virtualenv).

What about larger gui toolkits like PyQt?  Some material I browsed
(skimmed) indicated that it wasn't quite as simple to run straight from
a virtualenv...?

How much of a pain are virtualenvs when working from an IDE?




From tjreedy at udel.edu  Wed Jul 31 17:43:20 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 31 Jul 2013 17:43:20 -0400
Subject: Using system python vs. updated/current version
In-Reply-To: 
References: 
	<5B80DD153D7D744689F57F4FB69AF47418603BE1@SCACMX008.exchad.jpmchase.net>
	
Message-ID: 

On 7/31/2013 4:19 PM, memilanuk wrote:

> Are there any significant flaws with v.3.3.0 that would necessitate
> upgrading to the most recent version (3.3.2?)

Go to the overview page http://docs.python.org/3/index.html
and click on  'What's new in Python 3.3' to get to
http://docs.python.org/3/whatsnew/3.3.html
Click the 'changelog' link at the end of the first sentence.
http://docs.python.org/3.3/whatsnew/changelog.html
Browse through the 3.3.1 and 3.3.2 sections for bug fixes.
Decide if any of the entries are relevant to you.

-- 
Terry Jan Reedy



From tjreedy at udel.edu  Wed Jul 31 16:25:28 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 31 Jul 2013 16:25:28 -0400
Subject: Using system python vs. updated/current version
In-Reply-To: 
References: 
Message-ID: 

On 7/31/2013 2:35 PM, memilanuk wrote:
> Hello there,
>
> What would be considered the correct/best way to run a current release
> of python locally vs. the installed system version?  On openSUSE 12.3,
> the repos currently have 2.7.3  and 3.3.0

released April 2012. 2.7.5 100+?? bug fixes.
and released Sept 2012, fewer bug fixes.

>  As far as I know, I'm not
> really hitting any limitations with the existing versions

Quite possible. However, anyone using Idle should get the latest 
versions since there have been many Idle fixes since last September.

-- 
Terry Jan Reedy



From ben at benlast.com  Wed Jul 31 20:17:20 2013
From: ben at benlast.com (Ben Last)
Date: Thu, 1 Aug 2013 08:17:20 +0800
Subject: Using system python vs. updated/current version
Message-ID: 

> From: memilanuk 
> How much of a pain are virtualenvs when working from an IDE?
>

That depends on the IDE. I use PyCharm, and it has support for setting up
multiple Python virtualenvs, and associated different projects with
different virtualenvs, as well as tracking the packages installed per
virtualenv, etc.

I personally never touch the system Python install (I run ubuntu) because I
can never be sure what other OS systems rely on them. I use virtualenvs for
all development.

b
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 

From wuwei23 at gmail.com  Wed Jul 31 20:44:21 2013
From: wuwei23 at gmail.com (alex23)
Date: Thu, 01 Aug 2013 10:44:21 +1000
Subject: Using system python vs. updated/current version
In-Reply-To: 
References: 
Message-ID: 

On 1/08/2013 4:35 AM, memilanuk wrote:
> Also... in some places in the 'Net I see references to installing
> everything 'locally' via pip, etc. in virtualenvs and not touching the
> system installed version of python... yet most linux distros seem to
> have many/most such packages available in their package repos, which
> seems like it'd be easier to install via the package manager and let it
> keep things updated.  Could someone touch on what they feel the pros and
> cons would be either way?

Generally, if your OS installs a version of Python by default you should 
leave it alone because the OS itself is dependent on it. Updating to 
newer versions of Python or installed libraries can introduce version 
conflict errors in system-level apps, which is a bad thing.

Similarly, using the system install & libraries ties you to those 
versions. This may not be an issue if you're just scripting a few helper 
tools for your system, but it's an unnecessary hinderance if you're 
developing independent applications.

Tools like virtualenv or zc.buildout provide a handy way of sandboxing 
the dependencies of individual applications. They let you build more 
than one app in parallel and not let the dependencies of one interfere 
with the others. Of equal importance is their use in deploying to other 
machines. With virtualenv, you can create a list of installed libraries 
with:

     pip freeze > requirements.txt

To ensure a target machine has all of the dependencies your application 
needs you can then do:

     pin install -r requirements.txt

So: for simple scripts, just go with the system install. For serious 
development work, I highly recommend using virtualenv or zc.buildout to 
contain each development environment.


From rosuav at gmail.com  Wed Jul 31 20:55:03 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 1 Aug 2013 01:55:03 +0100
Subject: Using system python vs. updated/current version
In-Reply-To: 
References: 
Message-ID: 

On Wed, Jul 31, 2013 at 7:35 PM, memilanuk  wrote:
> Also... in some places in the 'Net I see references to installing
> everything 'locally' via pip, etc. in virtualenvs and not touching the
> system installed version of python... yet most linux distros seem to
> have many/most such packages available in their package repos, which
> seems like it'd be easier to install via the package manager and let it
> keep things updated.  Could someone touch on what they feel the pros and
> cons would be either way?

I personally like to compile some things from source (CPython, Pike,
etc - though not everything, I use a prepackaged PostgreSQL, for
instance). There's no harm in installing a new CPython on a Linux box
- just type 'sudo make altinstall' (or however you become root), and
it'll give you a binary called python3.4 or whatever version, without
touching your system Python. That lets you run as many versions as you
like, in parallel, though you may have issues running 3.3.0 and 3.3.2
(but there should be no reason to do so - just use 3.3.2).

ChrisA


From skip at pobox.com  Wed Jul 31 14:39:29 2013
From: skip at pobox.com (Skip Montanaro)
Date: Wed, 31 Jul 2013 13:39:29 -0500
Subject: Editing tabular data [was: PEP8 79 char max]
Message-ID: 

> I don't understand.  That just moves them to a different file --
> doesn't it?  You've still got to deal with editing a large table of
> data (for example when I want to add instructions to your assembler).

My guess is it would be more foolproof to edit that stuff with a spreadsheet.

Skip


From invalid at invalid.invalid  Wed Jul 31 15:02:35 2013
From: invalid at invalid.invalid (Grant Edwards)
Date: Wed, 31 Jul 2013 19:02:35 +0000 (UTC)
Subject: Editing tabular data [was: PEP8 79 char max]
References: 
Message-ID: 

On 2013-07-31, Skip Montanaro  wrote:
>> I don't understand.  That just moves them to a different file --
>> doesn't it?  You've still got to deal with editing a large table of
>> data (for example when I want to add instructions to your assembler).
>
> My guess is it would be more foolproof to edit that stuff with a
> spreadsheet.

Many years ago, I worked with somebody who used a spreadsheet like
that.  I tried it and found it to be way too cumbersome. The overhead
involved of putting tables in to slew of different files and starting
up LibreOffice to edit/view them is huge compared to just editing them
with emacs in a file along with the source code.  Maybe my computer is
too old/slow.  Maybe it's just due to how bad I am at Excel/LibreOffice...

-- 
Grant Edwards               grant.b.edwards        Yow! I haven't been married
                                  at               in over six years, but we
                              gmail.com            had sexual counseling every
                                                   day from Oral Roberts!!


From skip at pobox.com  Wed Jul 31 15:35:24 2013
From: skip at pobox.com (Skip Montanaro)
Date: Wed, 31 Jul 2013 14:35:24 -0500
Subject: Editing tabular data [was: PEP8 79 char max]
In-Reply-To: 
References: 
	
Message-ID: 

>> My guess is it would be more foolproof to edit that stuff with a
>> spreadsheet.
>
> Many years ago, I worked with somebody who used a spreadsheet like
> that.

I really love Emacs, however...  One of the traders here where I work
(who shall not be named) had a space-delimited data file with hundreds
of rows and 50 or so columns.  I could never get him to edit it in any
kind of spreadsheet or put it in a database (expecting him to master
SQL would have been pointless - I would have had to write a GUI tool
for him).  He always modified it in Emacs, and would delete columns,
add extra spaces, fragmentary rows, etc.  He'd edit this file late at
night, the automated processes the next morning would crap out, and I
would scramble to try and find and fix the problem before the market
opened.

This is clearly a case where choosing the proper tool is important.  I
agree that using a spreadsheet to edit a 3x5 CSV file is likely
overkill (might just as well use Notepad or TextEdit), but tabular
data are tabular data, no matter how they might be delimited, and if
there are many of those little data critters, there are better tools
than a text editor (or Python IDE) for maintaining them.

Skip


From rosuav at gmail.com  Wed Jul 31 20:50:13 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 1 Aug 2013 01:50:13 +0100
Subject: Editing tabular data [was: PEP8 79 char max]
In-Reply-To: 
References: 
	
Message-ID: 

On Wed, Jul 31, 2013 at 8:02 PM, Grant Edwards  wrote:
> On 2013-07-31, Skip Montanaro  wrote:
>>> I don't understand.  That just moves them to a different file --
>>> doesn't it?  You've still got to deal with editing a large table of
>>> data (for example when I want to add instructions to your assembler).
>>
>> My guess is it would be more foolproof to edit that stuff with a
>> spreadsheet.
>
> Many years ago, I worked with somebody who used a spreadsheet like
> that.  I tried it and found it to be way too cumbersome. The overhead
> involved of putting tables in to slew of different files and starting
> up LibreOffice to edit/view them is huge compared to just editing them
> with emacs in a file along with the source code.  Maybe my computer is
> too old/slow.  Maybe it's just due to how bad I am at Excel/LibreOffice...

I'm glad someone else feels that way!

At work, we have a number of CSV files (at my boss's insistence; I
would much rather they be either embedded in the source, or in some
clearer and simpler format) which I like to manipulate in SciTE,
rather than OO/LibreOffice. (I'll not distinguish those two. Far as
I'm concerned, they're one product with two names.) My boss can't
understand why I do this. I can't understand why he objects to having
to edit code files to alter internal data. I have pointed him to [1]
but to no avail.

The one thing I would do, though, is align with tabs rather than
spaces. That gives you an 8:1 (if you keep your tabs at eight, which I
do) improvement in maintainability, because edits that don't cross a
boundary don't require fiddling with the layout.

[1] http://thedailywtf.com/Articles/Soft_Coding.aspx

ChrisA


From rhodri at wildebst.demon.co.uk  Wed Jul 31 15:03:44 2013
From: rhodri at wildebst.demon.co.uk (Rhodri James)
Date: Wed, 31 Jul 2013 20:03:44 +0100
Subject: Editing tabular data [was: PEP8 79 char max]
References: 
Message-ID: 

On Wed, 31 Jul 2013 19:39:29 +0100, Skip Montanaro  wrote:

>> I don't understand.  That just moves them to a different file --
>> doesn't it?  You've still got to deal with editing a large table of
>> data (for example when I want to add instructions to your assembler).
>
> My guess is it would be more foolproof to edit that stuff with a  
> spreadsheet.

There's nothing foolproof about using a spreadsheet!

-- 
Rhodri James *-* Wildebeest Herder to the Masses


From neilc at norwich.edu  Wed Jul 31 15:07:47 2013
From: neilc at norwich.edu (Neil Cerutti)
Date: 31 Jul 2013 19:07:47 GMT
Subject: Editing tabular data [was: PEP8 79 char max]
References: 
	
Message-ID: 

On 2013-07-31, Rhodri James  wrote:
> On Wed, 31 Jul 2013 19:39:29 +0100, Skip Montanaro  wrote:
>
>>> I don't understand.  That just moves them to a different file --
>>> doesn't it?  You've still got to deal with editing a large table of
>>> data (for example when I want to add instructions to your assembler).
>>
>> My guess is it would be more foolproof to edit that stuff with a  
>> spreadsheet.
>
> There's nothing foolproof about using a spreadsheet!

I edit csv files using Excel all the time. But I don't use it to
edit my hand-created data files. It does too much meddling.

-- 
Neil Cerutti


From wanderer at dialup4less.com  Wed Jul 31 16:37:32 2013
From: wanderer at dialup4less.com (Wanderer)
Date: Wed, 31 Jul 2013 13:37:32 -0700 (PDT)
Subject: Editing tabular data [was: PEP8 79 char max]
In-Reply-To: 
References: 
Message-ID: <8face294-5199-4ec6-8732-ec9ec653a52a@googlegroups.com>

On Wednesday, July 31, 2013 2:39:29 PM UTC-4, Skip Montanaro wrote:
> > I don't understand.  That just moves them to a different file --
> 
> > doesn't it?  You've still got to deal with editing a large table of
> 
> > data (for example when I want to add instructions to your assembler).
> 
> 
> 
> My guess is it would be more foolproof to edit that stuff with a spreadsheet.
> 
> 
> 
> Skip

Has anyone tried Pyspread?

http://manns.github.io/pyspread/


From skip.montanaro at gmail.com  Wed Jul 31 19:40:14 2013
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Wed, 31 Jul 2013 18:40:14 -0500
Subject: Editing tabular data [was: PEP8 79 char max]
In-Reply-To: <8face294-5199-4ec6-8732-ec9ec653a52a@googlegroups.com>
References: 
	<8face294-5199-4ec6-8732-ec9ec653a52a@googlegroups.com>
Message-ID: 

> Has anyone tried Pyspread?

I have not.

I have a fundamental problem with spreadsheets, the extremely narrow view
of the workspace. There was a piece on NPR the other day about some errors
in some modeling applications. I missed most of it (does someone have a
link? I'm on my phone right now), but the expert commentator was saying
that they are working on standard structures for these sorts of complex
modeling simulations. I think they will need more than that. Something like
pyspread might allow you to mix structured and object oriented programming
with the convenience if a spreadsheet.

Skip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 

From samaneh.yahyapour at gmail.com  Wed Jul 31 15:12:16 2013
From: samaneh.yahyapour at gmail.com (sam319)
Date: Wed, 31 Jul 2013 12:12:16 -0700 (PDT)
Subject: pcurl and network connection's problem
Message-ID: <2271ec9a-ec36-485b-9776-3c7157cedae1@googlegroups.com>

I am having problems with pycurl in my threads , when i run it , it does correctly but some times the connection has been established but nothing will be downloaded and the threads stay alive without doing any thing (especially when the network's speed is slow and has aborted status) .

i can't use TIMEOUT  because i don't have the max time for downloading 

how can i solve this problem in python


From ramit.prasad at jpmorgan.com.dmarc.invalid  Wed Jul 31 15:32:46 2013
From: ramit.prasad at jpmorgan.com.dmarc.invalid (Prasad, Ramit)
Date: Wed, 31 Jul 2013 19:32:46 +0000
Subject: pcurl and network connection's problem
In-Reply-To: <2271ec9a-ec36-485b-9776-3c7157cedae1@googlegroups.com>
References: <2271ec9a-ec36-485b-9776-3c7157cedae1@googlegroups.com>
Message-ID: <5B80DD153D7D744689F57F4FB69AF47418603C23@SCACMX008.exchad.jpmchase.net>

sam319 wrote:
> I am having problems with pycurl in my threads , when i run it , it does correctly but some times the
> connection has been established but nothing will be downloaded and the threads stay alive without
> doing any thing (especially when the network's speed is slow and has aborted status) .
> 
> i can't use TIMEOUT  because i don't have the max time for downloading
> 
> how can i solve this problem in python

So you want to wait hours/days/years for a failed download? You should always set a 
timeout that is sane. If you anticipate network slowness, then set your timeout to 
something larger like 2-3 minutes. IIRC, the timeout only affects connection
establishment so that will not help out your use case but I want to emphasize
that you always want a timeout to be set. 

Pycurl has an option for LOW_SPEED_LIMIT/LOW_SPEED_TIME which should allow you to set 
a minimum transfer speed which will abort the transfer. This should solve your problem
where the connection is created but nothing is being downloaded.

See: http://stackoverflow.com/questions/4960021/handle-pycurl-hang-on-twitter-streaming-api


Ramit



This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email.