From list1 at tompassin.net Sun Jan 1 08:11:02 2023 From: list1 at tompassin.net (Thomas Passin) Date: Sun, 1 Jan 2023 08:11:02 -0500 Subject: NoneType List In-Reply-To: <00e601d91d8f$a4e59b20$eeb0d160$@gmail.com> References: <2b238dce-3e9f-9bee-27a0-d163cb6c7cef@DancesWithMice.info> <006501d91d68$6538dd10$2faa9730$@gmail.com> <00e601d91d8f$a4e59b20$eeb0d160$@gmail.com> Message-ID: <71caabcd-65a0-8b3f-a971-79cd3f4415e1@tompassin.net> On 12/31/2022 10:17 PM, avi.e.gross at gmail.com wrote: > Agreed, there are lots of pro/con arguments and the feature is what it is > historically and not trivial to change. Inline changes to an object make > sense to just be done "silently" and if there are errors, they propagate the > usual way. > > As Guido was a major influence at that time, one view was seen as more > important and prevailed. > > Had a language like that been created today, I wonder if some designs might > have looked a bit different so that some functions could be called with > optional arguments that specified what the user wanted returned. Guido had been working on the ABC language for some years before he developed Python. ABC was intended mainly as a teaching and prototyping language. Guido probably had a good sense of what things worked well and what didn't in that usage. IIRC, Python did not originally have classes or instances, and a "fluent" style of programming probably wasn't in use yet. Having an object return itself after an operation is useful (mostly, perhaps) for fluent programming (that is, the style where one can write X.method1().method2().method3() ...). > In particular, besides a function where you add a value returning > nothing/None, there may be room for other choices and any choice hard-wired > in would eleicit complaints from others. > > lst.add("Value") > > could also return one of several other things such as a pointer to the > upgraded object but also of a pointer to the object as it looked before the > change (not a serious proposal) or a True/False value specifying if the > change was able to be completed (such as running out of memory, or the > addition not being something you can put in the object) or even return what > was added or how many objects are now in the object at the top level, or how > many times the method was called so far! > > I suspect, at the expense of some overhead, you could just add an argument > to many kinds of methods or even functions such as returning='option' that > guides what you want returned, with the default often being what Guido and > others currently have set. > > Python already allows functions to return anything they feel like, so this > probably would not break many things. > > Of course there are other paths in that direction, such as setting an > attribute of the list/object first that affects how things get returned but > that seems more cumbersome and makes all kinds of errors more likely. Still, > that is a path often used by some Python modules where objects are created > and then tweaked to behave in various ways when later methods are invoked. > > But is any of it needed? The current system generally works fine and we have > seen many ways to get other results without tampering with the current > implementation. > > This may be yet another example of people who come to python with > pre-existing bias, such as insisting it work like another language they have > used, or wanting the computer to do what they MEANT rather than what they > explicitly or implicitly programmed! > > > -----Original Message----- > From: Python-list On > Behalf Of Greg Ewing > Sent: Saturday, December 31, 2022 7:21 PM > To: python-list at python.org > Subject: Re: NoneType List > > On 1/01/23 11:36 am, avi.e.gross at gmail.com wrote: >> And, of course, we had the philosophical question of why the feature >> was designed to not return anything ... rather than return the changed >> object. > > My understanding is that Guido designed it that way to keep a clear > separation between mutating and non-mutating methods, and to help catch > mistakes resulting from mixing them up. > > -- > Greg > -- > https://mail.python.org/mailman/listinfo/python-list > From goranikac65 at gmail.com Sun Jan 1 03:16:01 2023 From: goranikac65 at gmail.com (Goran Ikac) Date: Sun, 1 Jan 2023 09:16:01 +0100 Subject: NoneType List Message-ID: Thank you, guys, thank you very much! I'm amazed by the time and effort so many of you have put into the answers. And at this time of year! Not only is the issue clear to me now, but I learned a lot (and I'm still to learn a lot more) from your answers and the links you provided. And not only about the issue that bothered me when I wrote the question! I've read many times that one of the main features of Python was the large community ready to help and discuss. Well, I'm just beginning to see the true meaning of that, and I'm - well - amazed. Thanks, guys, and Happy New Year! From hjp-python at hjp.at Sun Jan 1 12:47:31 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 1 Jan 2023 18:47:31 +0100 Subject: Possible re bug when using ".*" In-Reply-To: <014c86f6-d181-ec5c-3908-88dabd52a966@mrabarnett.plus.com> References: <014c86f6-d181-ec5c-3908-88dabd52a966@mrabarnett.plus.com> Message-ID: <20230101174731.rxcr3zfqy4r4ge4b@hjp.at> On 2022-12-28 19:07:06 +0000, MRAB wrote: > On 2022-12-28 18:42, Alexander Richert - NOAA Affiliate via Python-list > wrote: > > print(re.sub(".*", "replacement", "pattern")) > > yields the output "replacementreplacement". [...] > It's not a bug, it's a change in behaviour to bring it more into line with > other regex implementations in other languages. Interesting. Perl does indeed behave that way, too. Never noticed that in 28 years of using it. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From list1 at tompassin.net Sun Jan 1 13:03:24 2023 From: list1 at tompassin.net (Thomas Passin) Date: Sun, 1 Jan 2023 13:03:24 -0500 Subject: NoneType List In-Reply-To: References: <2b238dce-3e9f-9bee-27a0-d163cb6c7cef@DancesWithMice.info> <006501d91d68$6538dd10$2faa9730$@gmail.com> <00e601d91d8f$a4e59b20$eeb0d160$@gmail.com> <71caabcd-65a0-8b3f-a971-79cd3f4415e1@tompassin.net> Message-ID: <582ee874-bc37-9933-216c-b32921dce153@tompassin.net> On 1/1/2023 8:47 AM, Stefan Ram wrote: > Thomas Passin writes: >> Guido had been working on the ABC language for some years before he >> developed Python. ABC was intended mainly as a teaching and prototyping >> language. > > In those days, there used to be a language called "Pascal". > Pascal had a dichotomy between "functions" and "procedures". > A call to a function was intended to have a value. > A call to a procedure was intended to have an effect. Wirth developed Pascal as a teaching language. IIRC, originally it was taught to students before there were any implementations. I did most of my programming with Turbo Pascal for many years. Just to clarify what you wrote above, in Pascal a "procedure" does not return anything while a "function" does. I really liked (Turbo) Pascal and I hated C back then. No wonder I like Python so much. It must be something about how my mind works. > For some beginners, the difference between a value and > and effect can be hard to grasp. So, Pascal's distinction > helps to hammer that home. > > Experienced programmers know the difference and do no longer > require the effort of the language to teach it to them. > > The time when someone is a beginner and still struggles > to understand the difference between values and effects > usually is significantly shorter than the later time > where he has understood it and is programming productively, > so it might be better when the language is adapted to > people who already have understood the difference. > > From avi.e.gross at gmail.com Sun Jan 1 13:17:05 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Sun, 1 Jan 2023 13:17:05 -0500 Subject: NoneType List In-Reply-To: References: <2b238dce-3e9f-9bee-27a0-d163cb6c7cef@DancesWithMice.info> <006501d91d68$6538dd10$2faa9730$@gmail.com> <00e601d91d8f$a4e59b20$eeb0d160$@gmail.com> <71caabcd-65a0-8b3f-a971-79cd3f4415e1@tompassin.net> Message-ID: <005b01d91e0d$4127b630$c3772290$@gmail.com> Several of you have mentioned the role of history in the development of languages and what the founders of a language were trying to improve. As noted with PASCAL, some earlier languages strived to be different things and in a certain sense, their procedures were perhaps seen as a safer and better way to do things that languages like BASIC hade done with a GOSUB. On top of that, there were concerns in a compiled language such as not leaving an unused result on the stack when a function returned and even efficiency considerations of several kinds. If a function returns a value, that value has to persist beyond the lifetime of the function. It has to be in a kind of memory different than where you can store values that go away. In a procedure call, the compiler could use the fact that nothing is returned other than through external variables. As noted, Guido started various languages, including Python, in a historical context where he was not yet seeing things as "objects" in the way we often do now. And he was trying to avoid some features often used in languages like C for various reasons while keeping or extending others. I am not sure about timing, but there have been many debates in the field about things like programming without side effects and some insist that changing things IN PLACE is a poor way to do things and instead things should be copied when modified and so on. So it well may be that allowing an inline sort or addition to a list was added carefully in a way that made it very clear what you were doing and seeing it more as a procedure call than a function that returned a value of itself as Stefan points out. But Python evolved and keeps evolving and like many biological organisms, to some extent, ontogeny recapitulates phylogeny. It is really hard to get rid of some early design choices even when a brand new design might allow a cleaner design with more advantages. Python is becoming more like other languages as it borrows from them and they borrow from it, and all borrow or adapt from yet others. The result, frankly, can be a tad hard to understand as the languages both converge and diverge only to converge again and some simply go extinct or get totally re-done to adapt ... -----Original Message----- From: Python-list On Behalf Of Stefan Ram Sent: Sunday, January 1, 2023 8:47 AM To: python-list at python.org Subject: Re: NoneType List Thomas Passin writes: >Guido had been working on the ABC language for some years before he >developed Python. ABC was intended mainly as a teaching and >prototyping language. In those days, there used to be a language called "Pascal". Pascal had a dichotomy between "functions" and "procedures". A call to a function was intended to have a value. A call to a procedure was intended to have an effect. For some beginners, the difference between a value and and effect can be hard to grasp. So, Pascal's distinction helps to hammer that home. Experienced programmers know the difference and do no longer require the effort of the language to teach it to them. The time when someone is a beginner and still struggles to understand the difference between values and effects usually is significantly shorter than the later time where he has understood it and is programming productively, so it might be better when the language is adapted to people who already have understood the difference. -- https://mail.python.org/mailman/listinfo/python-list From goranikac65 at gmail.com Sun Jan 1 13:20:22 2023 From: goranikac65 at gmail.com (Goran Ikac) Date: Sun, 1 Jan 2023 19:20:22 +0100 Subject: NoneType List In-Reply-To: <005b01d91e0d$4127b630$c3772290$@gmail.com> References: <2b238dce-3e9f-9bee-27a0-d163cb6c7cef@DancesWithMice.info> <006501d91d68$6538dd10$2faa9730$@gmail.com> <00e601d91d8f$a4e59b20$eeb0d160$@gmail.com> <71caabcd-65a0-8b3f-a971-79cd3f4415e1@tompassin.net> <005b01d91e0d$4127b630$c3772290$@gmail.com> Message-ID: Thank you, guys, thank you very much! I'm amazed by the time and effort so many of you have put into the answers. And at this time of year! Not only is the issue clear to me now, but I learned a lot (and I'm still to learn a lot more) from your answers and the links you provided. And not only about the issue that bothered me when I wrote the question! I've read many times that one of the main features of Python was the large community ready to help and discuss. Well, I'm just beginning to see the true meaning of that, and I'm - well - amazed. Thanks, guys, and Happy New Year! ned, 1. sij 2023. u 19:17 napisao je: > Several of you have mentioned the role of history in the development of > languages and what the founders of a language were trying to improve. > > As noted with PASCAL, some earlier languages strived to be different things > and in a certain sense, their procedures were perhaps seen as a safer and > better way to do things that languages like BASIC hade done with a GOSUB. > On > top of that, there were concerns in a compiled language such as not leaving > an unused result on the stack when a function returned and even efficiency > considerations of several kinds. If a function returns a value, that value > has to persist beyond the lifetime of the function. It has to be in a kind > of memory different than where you can store values that go away. In a > procedure call, the compiler could use the fact that nothing is returned > other than through external variables. > > As noted, Guido started various languages, including Python, in a > historical > context where he was not yet seeing things as "objects" in the way we often > do now. And he was trying to avoid some features often used in languages > like C for various reasons while keeping or extending others. I am not sure > about timing, but there have been many debates in the field about things > like programming without side effects and some insist that changing things > IN PLACE is a poor way to do things and instead things should be copied > when > modified and so on. > > So it well may be that allowing an inline sort or addition to a list was > added carefully in a way that made it very clear what you were doing and > seeing it more as a procedure call than a function that returned a value of > itself as Stefan points out. > > But Python evolved and keeps evolving and like many biological organisms, > to > some extent, ontogeny recapitulates phylogeny. It is really hard to get rid > of some early design choices even when a brand new design might allow a > cleaner design with more advantages. Python is becoming more like other > languages as it borrows from them and they borrow from it, and all borrow > or > adapt from yet others. The result, frankly, can be a tad hard to understand > as the languages both converge and diverge only to converge again and some > simply go extinct or get totally re-done to adapt ... > > -----Original Message----- > From: Python-list > On > Behalf Of Stefan Ram > Sent: Sunday, January 1, 2023 8:47 AM > To: python-list at python.org > Subject: Re: NoneType List > > Thomas Passin writes: > >Guido had been working on the ABC language for some years before he > >developed Python. ABC was intended mainly as a teaching and > >prototyping language. > > In those days, there used to be a language called "Pascal". > Pascal had a dichotomy between "functions" and "procedures". > A call to a function was intended to have a value. > A call to a procedure was intended to have an effect. > > For some beginners, the difference between a value and > and effect can be hard to grasp. So, Pascal's distinction > helps to hammer that home. > > Experienced programmers know the difference and do no longer > require the effort of the language to teach it to them. > > The time when someone is a beginner and still struggles > to understand the difference between values and effects > usually is significantly shorter than the later time > where he has understood it and is programming productively, > so it might be better when the language is adapted to > people who already have understood the difference. > > > -- > https://mail.python.org/mailman/listinfo/python-list > > -- > https://mail.python.org/mailman/listinfo/python-list > From avi.e.gross at gmail.com Sun Jan 1 21:14:50 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Sun, 1 Jan 2023 21:14:50 -0500 Subject: NoneType List In-Reply-To: <582ee874-bc37-9933-216c-b32921dce153@tompassin.net> References: <2b238dce-3e9f-9bee-27a0-d163cb6c7cef@DancesWithMice.info> <006501d91d68$6538dd10$2faa9730$@gmail.com> <00e601d91d8f$a4e59b20$eeb0d160$@gmail.com> <71caabcd-65a0-8b3f-a971-79cd3f4415e1@tompassin.net> <582ee874-bc37-9933-216c-b32921dce153@tompassin.net> Message-ID: <00b501d91e4f$ff0704c0$fd150e40$@gmail.com> Thomas, I used PASCAL before C and I felt like I was wearing a straitjacket at times in PASCAL when I was trying to write encryption/decryption functions and had to find ways to fiddle with bits. Similar things were easy in C, and are even easier in many more recent languages such as Python. The distinction between teaching a first language, or one that is so cautious as to catch and prevent all mistakes it can, is not for people willing to be bolder or work faster or write routines that can be used more generally. What has not been mentioned is that languages like python go a step further and allow a function to return many arguments and even a varying number of arguments, as well as none at all. To do anything like that in PASCAL took some thought on how to make some structure you could fill out then return as a single value that the receiving code had to sort of decode and perhaps deal with parts that could hold a union of several things. Can a compiler or interpreter easily verify you did something reasonable, as compared to say python that allows something like: (res1, res2, _, res4, _, rest) = f(x) The above tells the interpreter you expect perhaps 6 or more results and what to do with them. -----Original Message----- From: Python-list On Behalf Of Thomas Passin Sent: Sunday, January 1, 2023 1:03 PM To: python-list at python.org Subject: Re: NoneType List On 1/1/2023 8:47 AM, Stefan Ram wrote: > Thomas Passin writes: >> Guido had been working on the ABC language for some years before he >> developed Python. ABC was intended mainly as a teaching and >> prototyping language. > > In those days, there used to be a language called "Pascal". > Pascal had a dichotomy between "functions" and "procedures". > A call to a function was intended to have a value. > A call to a procedure was intended to have an effect. Wirth developed Pascal as a teaching language. IIRC, originally it was taught to students before there were any implementations. I did most of my programming with Turbo Pascal for many years. Just to clarify what you wrote above, in Pascal a "procedure" does not return anything while a "function" does. I really liked (Turbo) Pascal and I hated C back then. No wonder I like Python so much. It must be something about how my mind works. > For some beginners, the difference between a value and > and effect can be hard to grasp. So, Pascal's distinction > helps to hammer that home. > > Experienced programmers know the difference and do no longer > require the effort of the language to teach it to them. > > The time when someone is a beginner and still struggles > to understand the difference between values and effects > usually is significantly shorter than the later time > where he has understood it and is programming productively, > so it might be better when the language is adapted to > people who already have understood the difference. > > -- https://mail.python.org/mailman/listinfo/python-list From dvl at psu.edu Sun Jan 1 19:30:51 2023 From: dvl at psu.edu (Christman, Roger Graydon) Date: Mon, 2 Jan 2023 00:30:51 +0000 Subject: Python-list Digest, Vol 232, Issue 1 In-Reply-To: References: Message-ID: Re: Nonetype List In my introductory programming course, I have drawn some attention to this behavior regarding mutating lists. Indeed, Python is very consistent with its behavior: Any function that mutates a list parameter does not return that list as a return value. For one thing, there is no need to return that value, because the caller still owns the list parameter that has been modified. But secondly, (and what I find especially important), is that returning the modified list would lead too many program bugs or misunderstandings. For example, if append did return the list, you might see this: x = [1,2,3] y = x.append(4) z = y.append(5) The principal of 'least surprise' would cause a casual reader to believe that x retains the value of [1,2,3], y would have the value of [1,2,3,4], and z would contain [1,2,3,4,5]. So it would be very surprising indeed to discover that x contains [1,2,3,4,5], especially after that statement that makes no reference to x. Since append modifies the list in place, returning that list would make x, y, and z all aliases of each other, and aliasing is a source of many bugs that are very hard to find. So a recommendation that I make to my class (and which coincides with Python behavior), is to NEVER return a modified list as a return value, but only to return lists that were newly created within the function. So to support this principal of 'least surprise', the append method above would necessarily create new lists for y and z that are not aliases to x. Why Python does not do that is a very obvious cases of run-time efficiency (constant time to append vs. linear to recreate a new list). And as another observation, I have my students review all of the methods defined for the list object, and they are all very consistent. Most of them either define a return value, or modify the list parameter, but almost none do both. The sole exception is the pop() function that modified a list and returns a value, but that returned value still is not the modified list, so the aliasing problem will never arise. So, I am very happy with this Python language decision -- it allows for the most efficient means to modify a list in place and also very much reduce the danger of aliasing bugs. Roger Christman Pennsylvania State University ________________________________ From: Python-list on behalf of python-list-request at python.org Sent: Sunday, January 1, 2023 12:00 PM To: python-list at python.org Subject: Python-list Digest, Vol 232, Issue 1 Send Python-list mailing list submissions to python-list at python.org To subscribe or unsubscribe via the World Wide Web, visit https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-list&data=05%7C01%7Cdvl%40psu.edu%7C744c83fc485a4b1c79db08daec19a436%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638081892123929669%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=etYqO01OszhEpqgjLeKQTMC9b3wT0sc%2FcN8oJo9eEhk%3D&reserved=0 or, via email, send a message with subject or body 'help' to python-list-request at python.org You can reach the person managing the list at python-list-owner at python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Python-list digest..." Today's Topics: 1. Re: NoneType List (Thomas Passin) 2. Re: NoneType List (MRAB) 3. Re: NoneType List (dn) 4. RE: NoneType List (avi.e.gross at gmail.com) 5. Re: NoneType List (Thomas Passin) 6. Re: NoneType List (Greg Ewing) 7. RE: NoneType List (avi.e.gross at gmail.com) 8. Re: NoneType List (Chris Angelico) 9. RE: NoneType List (avi.e.gross at gmail.com) 10. Re: NoneType List (Chris Angelico) 11. Re: NoneType List (Thomas Passin) ---------------------------------------------------------------------- Message: 1 Date: Sat, 31 Dec 2022 12:07:25 -0500 From: Thomas Passin To: python-list at python.org Subject: Re: NoneType List Message-ID: <3eb7480c-88f7-72cf-af66-c0072928bf3f at tompassin.net> Content-Type: text/plain; charset=UTF-8; format=flowed Oops, my reply got lost somehow. Here it is: Everyone's answer to date has been too complicated. What is going on is that list.append() changes the list in place. It returns nothing. If you want to append an item and then assign the result to a new list, you have to do just that: l1.append(item) # If we want a *copy* of the appended list: l2 = l1[:] # Changes to l2 will not change l1 # If we want another name for the appended list: l2 = l1 # Changes to l2 will change l1 since they are the same object list.sort() also operates in place. There is a function sorted() that returns the sorted list (without changing the original list). The same thing is true of set.add(). The set is changed in place, and nothing is returned. On 12/31/2022 10:50 AM, Thomas Passin wrote: > Happy New Year, everybody! > I'm new in the Python List, new in Python world, and new in coding. > A few days (weeks?) ago, I faced a problem trying to write a program for an > exercise. I asked for help and nobody answered. > In the meantime, I found a part of the solution, but a part still remains a > mystery for me. Please run this small snippet, and help. > Thanks > > a = [1, 2] > print() > > print("a ==", a) > print("type(a) is", type(a)) > > b = a.append(3) > print("\nb =", "a.append(3)") > print("b ==", b) > print("type(b) is", type(b)) > > c = ['1', '2'] > print("\nc ==", c) > print("type(c) is", type(c)) > > d = c.append('3') > print("\nd =", "c.append('3')") > print("d ==", d) > print("type(d) is", type(d)) > > """ > I mean: why b = a.append(something) is the None type, and how to make a new > list > that contains all the items from a and some new items? > I wasn't able to solve it in a few hours :( > Now I know: > """ > > crta = '=' > print('\n', 4 * ' ', crta * (len('The solution:')), sep='') > print(3 * ' ', 'The solution:') > print(4 * ' ', crta * (len('The solution:')), sep='') > print('\nThe solution is the slice, an element of Python syntax that > allows') > print('to make a brand new copy of a list, or parts of a list.') > print('The slice actually copies the list\'s contents, not the list\'s > name:') > > print() > a = [1, 2] > print("a ==", a) > print("type(a) is", type(a)) > > b = a[:] > print("\nb = a[:]") > print("b ==", b) > b.append(3) > print("\nb =", "b.append(3)") > print("b ==", b) > print("type(b) is", type(b)) > print("\na ==", a) > > print('But I still don't know why "b = a.append(something)" is the None > type.') > print('Is there anybody out there?!') ------------------------------ Message: 2 Date: Sat, 31 Dec 2022 17:37:39 +0000 From: MRAB To: python-list at python.org Subject: Re: NoneType List Message-ID: <2321d9bc-5eb3-c1b9-0425-020b54dd39e8 at mrabarnett.plus.com> Content-Type: text/plain; charset=UTF-8; format=flowed On 2022-12-31 05:45, Goran Ikac wrote: > Happy New Year, everybody! > I'm new in the Python List, new in Python world, and new in coding. > A few days (weeks?) ago, I faced a problem trying to write a program for an > exercise. I asked for help and nobody answered. > In the meantime, I found a part of the solution, but a part still remains a > mystery for me. Please run this small snippet, and help. > Thanks > > a = [1, 2] > print() > > print("a ==", a) > print("type(a) is", type(a)) > > b = a.append(3) > print("\nb =", "a.append(3)") > print("b ==", b) > print("type(b) is", type(b)) > > c = ['1', '2'] > print("\nc ==", c) > print("type(c) is", type(c)) > > d = c.append('3') > print("\nd =", "c.append('3')") > print("d ==", d) > print("type(d) is", type(d)) > > """ > I mean: why b = a.append(something) is the None type, and how to make a new > list > that contains all the items from a and some new items? > I wasn't able to solve it in a few hours :( > Now I know: > """ > > crta = '=' > print('\n', 4 * ' ', crta * (len('The solution:')), sep='') > print(3 * ' ', 'The solution:') > print(4 * ' ', crta * (len('The solution:')), sep='') > print('\nThe solution is the slice, an element of Python syntax that > allows') > print('to make a brand new copy of a list, or parts of a list.') > print('The slice actually copies the list\'s contents, not the list\'s > name:') > > print() > a = [1, 2] > print("a ==", a) > print("type(a) is", type(a)) > > b = a[:] > print("\nb = a[:]") > print("b ==", b) > b.append(3) > print("\nb =", "b.append(3)") > print("b ==", b) > print("type(b) is", type(b)) > print("\na ==", a) > > print('But I still don't know why "b = a.append(something)" is the None > type.') > print('Is there anybody out there?!') Methods that modify in-place usually return None. "a.append(something)" modifies (appends to) the list 'a' and returns None. If you want to a new line with something at the end try "b = a + [something]". ------------------------------ Message: 3 Date: Sun, 1 Jan 2023 09:58:35 +1300 From: dn To: python-list at python.org Subject: Re: NoneType List Message-ID: <2b238dce-3e9f-9bee-27a0-d163cb6c7cef at DancesWithMice.info> Content-Type: text/plain; charset=UTF-8; format=flowed On 31/12/2022 18.45, Goran Ikac wrote: ... > A few days (weeks?) ago, I faced a problem trying to write a program for an > exercise. I asked for help and nobody answered. Looking back over the last six months of List-Archives, your name does not appear against a single post. This may explain why "nobody answered". However, ten hours after the above/first message, you posted again. This time as "Thomas Passin". That was followed an hour-or-so later, with a reply-to-self saying: "Everyone's answer to date has been too complicated", and then basically repeating the information previously-provided by a number of contributors. It then goes on to talk about copying, sorting, adding, and even sets. Much of which you had earlier said: "I know". Was there a further question in there? Which part of which answer did you find "too complicated"? Did you try the experiments suggested, and read the references-provided? Please ask a further question detailing what you have understood, and what is still mystifying. Folk here endeavor to be helpful (see also, earlier reference to the Python-Tutor list). When asking a question here, please try to reduce the problem to its simplest form, so that you're not asking volunteers to read multiple-screens of irrelevant code. It will help to ask one question at a time, or to carefully separate multiple questions. Yes, this can be difficult when one is learning. That said, we all started somewhere and are happy to help you to become a valued colleague! -- Regards, =dn ------------------------------ Message: 4 Date: Sat, 31 Dec 2022 17:36:59 -0500 From: To: Subject: RE: NoneType List Message-ID: <006501d91d68$6538dd10$2faa9730$@gmail.com> Content-Type: text/plain; charset="us-ascii" It depends on what people consider too complicated. I find it a tad complicated when someone posts using two different ID, and then wonders ... The question related to taking a list and extending it and using the result in an assignment statement. There were several inter-related questions people responded to. One was asking why something did not work as expected. Several answers pointed out it was because it was not designed the way expected and reading the manual page or other reference material would help explain that you need to use code the way it was intended and not as you wish it. Other aspects of similar code that do what you expect were shown and you could pick any you liked. A second question some were answering is various ways to get the functionality wanted. Some were simple enough like using "+" and others did seem complex as they showed many variations on copying an object or how to make a subclass that effectively substitutes a method that returns the internally changed object. And, of course, we had the philosophical question of why the feature was designed to not return anything (well, NULL) rather than return the changed object. Returning nothing is arguably slightly more efficient in the many cases where the return value is simply ignored and thus tossed. But as mentioned, it would be nice for some purposes, including chaining, to be able to write something like result = lst.add_ret("item").sort_ret() As mentioned, this can still be easily done using something like: result = sorted(lst + "item") -----Original Message----- From: Python-list On Behalf Of dn Sent: Saturday, December 31, 2022 3:59 PM To: python-list at python.org Subject: Re: NoneType List On 31/12/2022 18.45, Goran Ikac wrote: ... > A few days (weeks?) ago, I faced a problem trying to write a program > for an exercise. I asked for help and nobody answered. Looking back over the last six months of List-Archives, your name does not appear against a single post. This may explain why "nobody answered". However, ten hours after the above/first message, you posted again. This time as "Thomas Passin". That was followed an hour-or-so later, with a reply-to-self saying: "Everyone's answer to date has been too complicated", and then basically repeating the information previously-provided by a number of contributors. It then goes on to talk about copying, sorting, adding, and even sets. Much of which you had earlier said: "I know". Was there a further question in there? Which part of which answer did you find "too complicated"? Did you try the experiments suggested, and read the references-provided? Please ask a further question detailing what you have understood, and what is still mystifying. Folk here endeavor to be helpful (see also, earlier reference to the Python-Tutor list). When asking a question here, please try to reduce the problem to its simplest form, so that you're not asking volunteers to read multiple-screens of irrelevant code. It will help to ask one question at a time, or to carefully separate multiple questions. Yes, this can be difficult when one is learning. That said, we all started somewhere and are happy to help you to become a valued colleague! -- Regards, =dn -- https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-list&data=05%7C01%7Cdvl%40psu.edu%7C744c83fc485a4b1c79db08daec19a436%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638081892123929669%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=etYqO01OszhEpqgjLeKQTMC9b3wT0sc%2FcN8oJo9eEhk%3D&reserved=0 ------------------------------ Message: 5 Date: Sat, 31 Dec 2022 18:32:04 -0500 From: Thomas Passin To: python-list at python.org Subject: Re: NoneType List Message-ID: <11625e0f-8975-b929-a2ca-e14ed74a19b0 at tompassin.net> Content-Type: text/plain; charset=UTF-8; format=flowed On 12/31/2022 3:58 PM, dn wrote: > On 31/12/2022 18.45, Goran Ikac wrote: > ... >> A few days (weeks?) ago, I faced a problem trying to write a program >> for an >> exercise. I asked for help and nobody answered. > > Looking back over the last six months of List-Archives, your name does > not appear against a single post. This may explain why "nobody answered". > > However, ten hours after the above/first message, you posted again. This > time as "Thomas Passin". That message was probably a mistaken one from me. I had composed a reply but through some mental glitch had to re-do it. I managed to send it with the only quoted thread but not the reply I had wanted to include. So I added my reply and sent it again. It was probably confusing, and I'm sorry about that. > That was followed an hour-or-so later, with a > reply-to-self saying: "Everyone's answer to date has been too > complicated", and then basically repeating the information > previously-provided by a number of contributors. > > It then goes on to talk about copying, sorting, adding, and even sets. > Much of which you had earlier said: "I know". Was there a further > question in there? > > Which part of which answer did you find "too complicated"? > > Did you try the experiments suggested, and read the references-provided? > > Please ask a further question detailing what you have understood, and > what is still mystifying. Folk here endeavor to be helpful (see also, > earlier reference to the Python-Tutor list). > > When asking a question here, please try to reduce the problem to its > simplest form, so that you're not asking volunteers to read > multiple-screens of irrelevant code. It will help to ask one question at > a time, or to carefully separate multiple questions. > > Yes, this can be difficult when one is learning. That said, we all > started somewhere and are happy to help you to become a valued colleague! > ------------------------------ Message: 6 Date: Sun, 1 Jan 2023 13:21:05 +1300 From: Greg Ewing To: python-list at python.org Subject: Re: NoneType List Message-ID: Content-Type: text/plain; charset=UTF-8; format=flowed On 1/01/23 11:36 am, avi.e.gross at gmail.com wrote: > And, of course, we had the philosophical question of why the feature was > designed to not return anything ... rather than return the changed > object. My understanding is that Guido designed it that way to keep a clear separation between mutating and non-mutating methods, and to help catch mistakes resulting from mixing them up. -- Greg ------------------------------ Message: 7 Date: Sat, 31 Dec 2022 22:17:56 -0500 From: To: "'Greg Ewing'" , Subject: RE: NoneType List Message-ID: <00e601d91d8f$a4e59b20$eeb0d160$@gmail.com> Content-Type: text/plain; charset="us-ascii" Agreed, there are lots of pro/con arguments and the feature is what it is historically and not trivial to change. Inline changes to an object make sense to just be done "silently" and if there are errors, they propagate the usual way. As Guido was a major influence at that time, one view was seen as more important and prevailed. Had a language like that been created today, I wonder if some designs might have looked a bit different so that some functions could be called with optional arguments that specified what the user wanted returned. In particular, besides a function where you add a value returning nothing/None, there may be room for other choices and any choice hard-wired in would eleicit complaints from others. lst.add("Value") could also return one of several other things such as a pointer to the upgraded object but also of a pointer to the object as it looked before the change (not a serious proposal) or a True/False value specifying if the change was able to be completed (such as running out of memory, or the addition not being something you can put in the object) or even return what was added or how many objects are now in the object at the top level, or how many times the method was called so far! I suspect, at the expense of some overhead, you could just add an argument to many kinds of methods or even functions such as returning='option' that guides what you want returned, with the default often being what Guido and others currently have set. Python already allows functions to return anything they feel like, so this probably would not break many things. Of course there are other paths in that direction, such as setting an attribute of the list/object first that affects how things get returned but that seems more cumbersome and makes all kinds of errors more likely. Still, that is a path often used by some Python modules where objects are created and then tweaked to behave in various ways when later methods are invoked. But is any of it needed? The current system generally works fine and we have seen many ways to get other results without tampering with the current implementation. This may be yet another example of people who come to python with pre-existing bias, such as insisting it work like another language they have used, or wanting the computer to do what they MEANT rather than what they explicitly or implicitly programmed! -----Original Message----- From: Python-list On Behalf Of Greg Ewing Sent: Saturday, December 31, 2022 7:21 PM To: python-list at python.org Subject: Re: NoneType List On 1/01/23 11:36 am, avi.e.gross at gmail.com wrote: > And, of course, we had the philosophical question of why the feature > was designed to not return anything ... rather than return the changed > object. My understanding is that Guido designed it that way to keep a clear separation between mutating and non-mutating methods, and to help catch mistakes resulting from mixing them up. -- Greg -- https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-list&data=05%7C01%7Cdvl%40psu.edu%7C744c83fc485a4b1c79db08daec19a436%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638081892123929669%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=etYqO01OszhEpqgjLeKQTMC9b3wT0sc%2FcN8oJo9eEhk%3D&reserved=0 ------------------------------ Message: 8 Date: Sun, 1 Jan 2023 14:23:13 +1100 From: Chris Angelico To: python-list at python.org Subject: Re: NoneType List Message-ID: Content-Type: text/plain; charset="UTF-8" On Sun, 1 Jan 2023 at 14:19, wrote: > Had a language like that been created today, I wonder if some designs might > have looked a bit different so that some functions could be called with > optional arguments that specified what the user wanted returned. Frankly, I doubt it. While you can argue "returning self is more useful" vs "returning None is more clear if you get it wrong", having options does NOT improve things, and just makes everything more complicated, slower, harder to comprehend, and generally worse to work with. A language should have some sort of consistent behaviour and stick to it. If that's not possible, an object type should at least have that. ChrisA ------------------------------ Message: 9 Date: Sat, 31 Dec 2022 23:16:10 -0500 From: To: "'Chris Angelico'" , Subject: RE: NoneType List Message-ID: <00fd01d91d97$c78994d0$569cbe70$@gmail.com> Content-Type: text/plain; charset="us-ascii" Chris, There is much to say about consistent behavior as compared to flexibility and convenience. I have seen other languages provide functions for example, where the result can vary and often cause confusion. R had a function that would sometimes notice the result could be simplified and return that. Guess what? It caused lots of problems and an option was added that said it should NOT simplify and always return the same kind of thing. Consider what happens if a calculation that returned a matrix would decide that is there was only one columns, it would return a vector/array/whatever as a one-dimensional result and if the calculation produced a 1 by 1 matrix, it would simply return a scalar value! But it may be a feature you want in some cases, albeit rarely when the results of the function are fed into something that is not expecting it. Various forms of polymorphism can be helpful but can also be very confusing. Some solutions are what I have described in earlier messages and some languages blur distinctions. Again, in R, there may not be a scalar as all you have is a vector of length one. But matrices even of 1-D are not vectors and I have had to interconvert between them to make some code run. I am not making comparisons in the sense that nothing other languages choose to do is binding on what Python should do. Still, I think it is wrong to suggest that it does not often do partially ambiguous things from some perspective. Many functions will take a variety of arguments and return something reasonable but different each time. As a dumb example, what does a simple function like max() return if fed just integers, just floating point or a combination of both? It seems to return whatever type the maximum indicates. But it also accepts characters and sorts them appropriately returning type 'str' and probably accepts and returns all kinds of objects if you specify a key function. Is that so much different than we are discussing in that there isn't any absolute consistency and things can go various ways in terms of return value? Heck, I can even make max() return None! -----Original Message----- From: Python-list On Behalf Of Chris Angelico Sent: Saturday, December 31, 2022 10:23 PM To: python-list at python.org Subject: Re: NoneType List On Sun, 1 Jan 2023 at 14:19, wrote: > Had a language like that been created today, I wonder if some designs > might have looked a bit different so that some functions could be > called with optional arguments that specified what the user wanted returned. Frankly, I doubt it. While you can argue "returning self is more useful" vs "returning None is more clear if you get it wrong", having options does NOT improve things, and just makes everything more complicated, slower, harder to comprehend, and generally worse to work with. A language should have some sort of consistent behaviour and stick to it. If that's not possible, an object type should at least have that. ChrisA -- https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-list&data=05%7C01%7Cdvl%40psu.edu%7C744c83fc485a4b1c79db08daec19a436%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638081892123929669%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=etYqO01OszhEpqgjLeKQTMC9b3wT0sc%2FcN8oJo9eEhk%3D&reserved=0 ------------------------------ Message: 10 Date: Sun, 1 Jan 2023 15:23:59 +1100 From: Chris Angelico To: python-list at python.org Subject: Re: NoneType List Message-ID: Content-Type: text/plain; charset="UTF-8" On Sun, 1 Jan 2023 at 15:16, wrote: > > Chris, > > There is much to say about consistent behavior as compared to flexibility > and convenience. > > I have seen other languages provide functions for example, where the result > can vary and often cause confusion. R had a function that would sometimes > notice the result could be simplified and return that. Guess what? It caused > lots of problems and an option was added that said it should NOT simplify > and always return the same kind of thing. The option was presumably added because backward compatibility is important, but it would have been better to not need the option in the first place. > As a dumb example, what does a simple function like max() return if fed just > integers, just floating point or a combination of both? It seems to return > whatever type the maximum indicates. But it also accepts characters and > sorts them appropriately returning type 'str' and probably accepts and > returns all kinds of objects if you specify a key function. > > Is that so much different than we are discussing in that there isn't any > absolute consistency and things can go various ways in terms of return > value? Heck, I can even make max() return None! > ... yes? So? It still always returns the largest item in the list, for some definition of "largest". You'll never have that value returned wrapped up in a single-element list, though. ChrisA ------------------------------ Message: 11 Date: Sun, 1 Jan 2023 08:11:02 -0500 From: Thomas Passin To: python-list at python.org Subject: Re: NoneType List Message-ID: <71caabcd-65a0-8b3f-a971-79cd3f4415e1 at tompassin.net> Content-Type: text/plain; charset=UTF-8; format=flowed On 12/31/2022 10:17 PM, avi.e.gross at gmail.com wrote: > Agreed, there are lots of pro/con arguments and the feature is what it is > historically and not trivial to change. Inline changes to an object make > sense to just be done "silently" and if there are errors, they propagate the > usual way. > > As Guido was a major influence at that time, one view was seen as more > important and prevailed. > > Had a language like that been created today, I wonder if some designs might > have looked a bit different so that some functions could be called with > optional arguments that specified what the user wanted returned. Guido had been working on the ABC language for some years before he developed Python. ABC was intended mainly as a teaching and prototyping language. Guido probably had a good sense of what things worked well and what didn't in that usage. IIRC, Python did not originally have classes or instances, and a "fluent" style of programming probably wasn't in use yet. Having an object return itself after an operation is useful (mostly, perhaps) for fluent programming (that is, the style where one can write X.method1().method2().method3() ...). > In particular, besides a function where you add a value returning > nothing/None, there may be room for other choices and any choice hard-wired > in would eleicit complaints from others. > > lst.add("Value") > > could also return one of several other things such as a pointer to the > upgraded object but also of a pointer to the object as it looked before the > change (not a serious proposal) or a True/False value specifying if the > change was able to be completed (such as running out of memory, or the > addition not being something you can put in the object) or even return what > was added or how many objects are now in the object at the top level, or how > many times the method was called so far! > > I suspect, at the expense of some overhead, you could just add an argument > to many kinds of methods or even functions such as returning='option' that > guides what you want returned, with the default often being what Guido and > others currently have set. > > Python already allows functions to return anything they feel like, so this > probably would not break many things. > > Of course there are other paths in that direction, such as setting an > attribute of the list/object first that affects how things get returned but > that seems more cumbersome and makes all kinds of errors more likely. Still, > that is a path often used by some Python modules where objects are created > and then tweaked to behave in various ways when later methods are invoked. > > But is any of it needed? The current system generally works fine and we have > seen many ways to get other results without tampering with the current > implementation. > > This may be yet another example of people who come to python with > pre-existing bias, such as insisting it work like another language they have > used, or wanting the computer to do what they MEANT rather than what they > explicitly or implicitly programmed! > > > -----Original Message----- > From: Python-list On > Behalf Of Greg Ewing > Sent: Saturday, December 31, 2022 7:21 PM > To: python-list at python.org > Subject: Re: NoneType List > > On 1/01/23 11:36 am, avi.e.gross at gmail.com wrote: >> And, of course, we had the philosophical question of why the feature >> was designed to not return anything ... rather than return the changed >> object. > > My understanding is that Guido designed it that way to keep a clear > separation between mutating and non-mutating methods, and to help catch > mistakes resulting from mixing them up. > > -- > Greg > -- > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-list&data=05%7C01%7Cdvl%40psu.edu%7C744c83fc485a4b1c79db08daec19a436%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638081892123929669%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=etYqO01OszhEpqgjLeKQTMC9b3wT0sc%2FcN8oJo9eEhk%3D&reserved=0 > ------------------------------ Subject: Digest Footer -- https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-list&data=05%7C01%7Cdvl%40psu.edu%7C744c83fc485a4b1c79db08daec19a436%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638081892123929669%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=etYqO01OszhEpqgjLeKQTMC9b3wT0sc%2FcN8oJo9eEhk%3D&reserved=0 ------------------------------ End of Python-list Digest, Vol 232, Issue 1 ******************************************* From list1 at tompassin.net Sun Jan 1 22:14:45 2023 From: list1 at tompassin.net (Thomas Passin) Date: Sun, 1 Jan 2023 22:14:45 -0500 Subject: NoneType List In-Reply-To: <00b501d91e4f$ff0704c0$fd150e40$@gmail.com> References: <2b238dce-3e9f-9bee-27a0-d163cb6c7cef@DancesWithMice.info> <006501d91d68$6538dd10$2faa9730$@gmail.com> <00e601d91d8f$a4e59b20$eeb0d160$@gmail.com> <71caabcd-65a0-8b3f-a971-79cd3f4415e1@tompassin.net> <582ee874-bc37-9933-216c-b32921dce153@tompassin.net> <00b501d91e4f$ff0704c0$fd150e40$@gmail.com> Message-ID: On 1/1/2023 9:14 PM, avi.e.gross at gmail.com wrote: > Thomas, > > I used PASCAL before C and I felt like I was wearing a straitjacket > at times in PASCAL when I was trying to write encryption/decryption > functions and had to find ways to fiddle with bits. Similar things > were easy in C, and are even easier in many more recent languages > such as Python. PASCAL was not the first language I learned. I won't pretend I had to do anything very complicated, or do much bit-twiddling. It was, though, the first one (except probably for FORTH) I enjoyed programming with more than I disliked the boiler-plate formalities. > The distinction between teaching a first language, or one that is so > cautious as to catch and prevent all mistakes it can, is not for > people willing to be bolder or work faster or write routines that can > be used more generally. > > What has not been mentioned is that languages like python go a step > further and allow a function to return many arguments and even a > varying number of arguments, as well as none at all. To do anything > like that in PASCAL (or C, for that matter) > took some thought on how to make some structure you could fill out > then return as a single value that the receiving code had to sort of > decode and perhaps deal with parts that could hold a union of several > things. Can a compiler or interpreter easily verify you did something > reasonable, as compared to say python that allows something like: > > (res1, res2, _, res4, _, rest) = f(x) Yes, that's one of the good things about Python, how it makes working with tuples so easy and natural. OTOH, harking back to PASCAL for a minute, it had enumerations and sets long before Python got them. > The above tells the interpreter you expect perhaps 6 or more results > and what to do with them. > > > > -----Original Message----- From: Python-list > On Behalf Of > Thomas Passin Sent: Sunday, January 1, 2023 1:03 PM To: > python-list at python.org Subject: Re: NoneType List > > On 1/1/2023 8:47 AM, Stefan Ram wrote: >> Thomas Passin writes: >>> Guido had been working on the ABC language for some years before >>> he developed Python. ABC was intended mainly as a teaching and >>> prototyping language. >> >> In those days, there used to be a language called "Pascal". Pascal >> had a dichotomy between "functions" and "procedures". A call to a >> function was intended to have a value. A call to a procedure was >> intended to have an effect. > > Wirth developed Pascal as a teaching language. IIRC, originally it > was taught to students before there were any implementations. I did > most of my programming with Turbo Pascal for many years. Just to > clarify what you wrote above, in Pascal a "procedure" does not return > anything while a "function" does. > > I really liked (Turbo) Pascal and I hated C back then. No wonder I > like Python so much. It must be something about how my mind works. > >> For some beginners, the difference between a value and and effect >> can be hard to grasp. So, Pascal's distinction helps to hammer that >> home. >> >> Experienced programmers know the difference and do no longer >> require the effort of the language to teach it to them. >> >> The time when someone is a beginner and still struggles to >> understand the difference between values and effects usually is >> significantly shorter than the later time where he has understood >> it and is programming productively, so it might be better when the >> language is adapted to people who already have understood the >> difference. >> >> > > -- https://mail.python.org/mailman/listinfo/python-list > From avi.e.gross at gmail.com Mon Jan 2 00:19:49 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Mon, 2 Jan 2023 00:19:49 -0500 Subject: NoneType List In-Reply-To: References: <2b238dce-3e9f-9bee-27a0-d163cb6c7cef@DancesWithMice.info> <006501d91d68$6538dd10$2faa9730$@gmail.com> <00e601d91d8f$a4e59b20$eeb0d160$@gmail.com> <71caabcd-65a0-8b3f-a971-79cd3f4415e1@tompassin.net> <582ee874-bc37-9933-216c-b32921dce153@tompassin.net> <00b501d91e4f$ff0704c0$fd150e40$@gmail.com> Message-ID: <010201d91e69$d6221ee0$82665ca0$@gmail.com> Not to wax poetic about our pasts, Thomas, but I do did not start with PASCAL and used quite a few languages before and plenty after. At the time it had interesting contrasts to languages like BASIC, FORTRAN and LISP and I tended to use whatever was available on the machines I was using. My first computer job (other than in grad school itself) was using OMSI PASCAL. I even wrote my thesis as a PASCAL program containing comments that included typesetting instructions in a language called RUNOFF while the PASCAL program itself was a set of comments as far as RUNOFF was concerned. So it compiled or was typeset and printed from the same source. But my next job after graduation was for Bell Labs and I had to forget all the mainframe or DEC systems and adapt to UNIX, and of course C and later C++ and the various other little languages that came with that such as AWK and PERL ... There is no one right language but there often is a small set of right languages given your circumstances, such as what your employer has everyone using and especially in group projects. To be fair, languages like Python and R seem to keep having parts rewritten in C or C++ to make some things run faster and can include libraries written ages ago in languages like FORTRAN that have been finely tuned. Under the hood, these languages often hide parts so that developing a new python (or R or ...) module/package can start by writing it all in that language and then taking some functionality and rewriting it in the other language for critical regions where a slower interpreted method may be speeded up. But for prototyping, or when speed is not a big deal, I really prefer Python to ... -----Original Message----- From: Python-list On Behalf Of Thomas Passin Sent: Sunday, January 1, 2023 10:15 PM To: python-list at python.org Subject: Re: NoneType List On 1/1/2023 9:14 PM, avi.e.gross at gmail.com wrote: > Thomas, > > I used PASCAL before C and I felt like I was wearing a straitjacket at > times in PASCAL when I was trying to write encryption/decryption > functions and had to find ways to fiddle with bits. Similar things > were easy in C, and are even easier in many more recent languages such > as Python. PASCAL was not the first language I learned. I won't pretend I had to do anything very complicated, or do much bit-twiddling. It was, though, the first one (except probably for FORTH) I enjoyed programming with more than I disliked the boiler-plate formalities. > The distinction between teaching a first language, or one that is so > cautious as to catch and prevent all mistakes it can, is not for > people willing to be bolder or work faster or write routines that can > be used more generally. > > What has not been mentioned is that languages like python go a step > further and allow a function to return many arguments and even a > varying number of arguments, as well as none at all. To do anything > like that in PASCAL (or C, for that matter) > took some thought on how to make some structure you could fill out > then return as a single value that the receiving code had to sort of > decode and perhaps deal with parts that could hold a union of several > things. Can a compiler or interpreter easily verify you did something > reasonable, as compared to say python that allows something like: > > (res1, res2, _, res4, _, rest) = f(x) Yes, that's one of the good things about Python, how it makes working with tuples so easy and natural. OTOH, harking back to PASCAL for a minute, it had enumerations and sets long before Python got them. > The above tells the interpreter you expect perhaps 6 or more results > and what to do with them. > > > > -----Original Message----- From: Python-list > On Behalf Of > Thomas Passin Sent: Sunday, January 1, 2023 1:03 PM To: > python-list at python.org Subject: Re: NoneType List > > On 1/1/2023 8:47 AM, Stefan Ram wrote: >> Thomas Passin writes: >>> Guido had been working on the ABC language for some years before he >>> developed Python. ABC was intended mainly as a teaching and >>> prototyping language. >> >> In those days, there used to be a language called "Pascal". Pascal >> had a dichotomy between "functions" and "procedures". A call to a >> function was intended to have a value. A call to a procedure was >> intended to have an effect. > > Wirth developed Pascal as a teaching language. IIRC, originally it was > taught to students before there were any implementations. I did most > of my programming with Turbo Pascal for many years. Just to clarify > what you wrote above, in Pascal a "procedure" does not return anything > while a "function" does. > > I really liked (Turbo) Pascal and I hated C back then. No wonder I > like Python so much. It must be something about how my mind works. > >> For some beginners, the difference between a value and and effect can >> be hard to grasp. So, Pascal's distinction helps to hammer that home. >> >> Experienced programmers know the difference and do no longer require >> the effort of the language to teach it to them. >> >> The time when someone is a beginner and still struggles to understand >> the difference between values and effects usually is significantly >> shorter than the later time where he has understood it and is >> programming productively, so it might be better when the language is >> adapted to people who already have understood the difference. >> >> > > -- https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list From learn2program at gmail.com Mon Jan 2 03:01:23 2023 From: learn2program at gmail.com (Alan Gauld) Date: Mon, 2 Jan 2023 08:01:23 +0000 Subject: NoneType List In-Reply-To: <00b501d91e4f$ff0704c0$fd150e40$@gmail.com> References: <2b238dce-3e9f-9bee-27a0-d163cb6c7cef@DancesWithMice.info> <006501d91d68$6538dd10$2faa9730$@gmail.com> <00e601d91d8f$a4e59b20$eeb0d160$@gmail.com> <71caabcd-65a0-8b3f-a971-79cd3f4415e1@tompassin.net> <582ee874-bc37-9933-216c-b32921dce153@tompassin.net> <00b501d91e4f$ff0704c0$fd150e40$@gmail.com> Message-ID: On 02/01/2023 02:14, avi.e.gross at gmail.com wrote: > I used PASCAL before C and I felt like I was wearing a straitjacket at times > in PASCAL when I was trying to write encryption/decryption functions and had > to find ways to fiddle with bits. Similar things were easy in C, and are > even easier in many more recent languages such as Python. That's true of pure Pascal. But Thomas was talking about Turbo Pascal which had extra functions and features for all those "real world" type things. (And you could insert some inline assembler if all else failed) It also relaxed the ludicrously strict typing slightly. Turbo Pascal made Pascal a joy and I still use Delphi for Windows programming today. TP also introduced classes to Pascal (although Apple had already done so for the Mac and Borland basically ported the syntax to the PC). -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From list1 at tompassin.net Mon Jan 2 11:46:40 2023 From: list1 at tompassin.net (Thomas Passin) Date: Mon, 2 Jan 2023 11:46:40 -0500 Subject: NoneType List In-Reply-To: References: <2b238dce-3e9f-9bee-27a0-d163cb6c7cef@DancesWithMice.info> <006501d91d68$6538dd10$2faa9730$@gmail.com> <00e601d91d8f$a4e59b20$eeb0d160$@gmail.com> <71caabcd-65a0-8b3f-a971-79cd3f4415e1@tompassin.net> <582ee874-bc37-9933-216c-b32921dce153@tompassin.net> <00b501d91e4f$ff0704c0$fd150e40$@gmail.com> Message-ID: On 1/2/2023 3:01 AM, Alan Gauld wrote: > On 02/01/2023 02:14, avi.e.gross at gmail.com wrote: >> I used PASCAL before C and I felt like I was wearing a straitjacket at times >> in PASCAL when I was trying to write encryption/decryption functions and had >> to find ways to fiddle with bits. Similar things were easy in C, and are >> even easier in many more recent languages such as Python. > > That's true of pure Pascal. But Thomas was talking about Turbo Pascal > which had extra functions and features for all those "real world" type > things. (And you could insert some inline assembler if all else failed) > It also relaxed the ludicrously strict typing slightly. Turbo Pascal > made Pascal a joy and I still use Delphi for Windows programming today. If Python weren't around, I might be doing that myself. > TP also introduced classes to Pascal (although Apple had already done > so for the Mac and Borland basically ported the syntax to the PC). That was V5.5, wasn't it? Classes were a hugely useful extension to TP. I used a version with classes (a later version that had a graphics primitives module) to write a window manager (in DOS graphics mode, all 350 X 200 pixels of it). From ramyam at jnnce.ac.in Mon Jan 2 00:27:46 2023 From: ramyam at jnnce.ac.in (Ramya M) Date: Mon, 2 Jan 2023 10:57:46 +0530 Subject: Fwd: About the Python In-Reply-To: References: Message-ID: ---------- Forwarded message --------- From: Ramya M Date: Mon, Jan 2, 2023, 9:58 AM Subject: About the Python To: This is from JNN College of Engineering, Shimoga. we are facing some problems while using python. Please can you resolve this issue. We are using python 3.11.1 (64 bit) for windows 10. but while installing the "numpy and matplotlib" packages we are getting errors. In some cases after installation of the above said packages we are getting errors while working. So, Please can you people resolve this issue as the python labs are going on and students are facing some difficulties. Thanking you Regards Ramya M Instructor, ECE Dept. JNNCE, Shimoga From pallavikannor83 at gmail.com Mon Jan 2 01:48:21 2023 From: pallavikannor83 at gmail.com (pallavi Kannor) Date: Mon, 2 Jan 2023 12:18:21 +0530 Subject: NoneType List In-Reply-To: <010201d91e69$d6221ee0$82665ca0$@gmail.com> References: <2b238dce-3e9f-9bee-27a0-d163cb6c7cef@DancesWithMice.info> <006501d91d68$6538dd10$2faa9730$@gmail.com> <00e601d91d8f$a4e59b20$eeb0d160$@gmail.com> <71caabcd-65a0-8b3f-a971-79cd3f4415e1@tompassin.net> <582ee874-bc37-9933-216c-b32921dce153@tompassin.net> <00b501d91e4f$ff0704c0$fd150e40$@gmail.com> <010201d91e69$d6221ee0$82665ca0$@gmail.com> Message-ID: Thanks a lot all for your suggestions, I am clear now. On Mon, Jan 2, 2023 at 10:50 AM wrote: > Not to wax poetic about our pasts, Thomas, but I do did not start with > PASCAL and used quite a few languages before and plenty after. At the time > it had interesting contrasts to languages like BASIC, FORTRAN and LISP and > I > tended to use whatever was available on the machines I was using. My first > computer job (other than in grad school itself) was using OMSI PASCAL. I > even wrote my thesis as a PASCAL program containing comments that included > typesetting instructions in a language called RUNOFF while the PASCAL > program itself was a set of comments as far as RUNOFF was concerned. So it > compiled or was typeset and printed from the same source. > > But my next job after graduation was for Bell Labs and I had to forget all > the mainframe or DEC systems and adapt to UNIX, and of course C and later > C++ and the various other little languages that came with that such as AWK > and PERL ... > > There is no one right language but there often is a small set of right > languages given your circumstances, such as what your employer has everyone > using and especially in group projects. > > To be fair, languages like Python and R seem to keep having parts rewritten > in C or C++ to make some things run faster and can include libraries > written > ages ago in languages like FORTRAN that have been finely tuned. Under the > hood, these languages often hide parts so that developing a new python (or > R > or ...) module/package can start by writing it all in that language and > then > taking some functionality and rewriting it in the other language for > critical regions where a slower interpreted method may be speeded up. > > But for prototyping, or when speed is not a big deal, I really prefer > Python > to ... > > > -----Original Message----- > From: Python-list > On > Behalf Of Thomas Passin > Sent: Sunday, January 1, 2023 10:15 PM > To: python-list at python.org > Subject: Re: NoneType List > > On 1/1/2023 9:14 PM, avi.e.gross at gmail.com wrote: > > Thomas, > > > > I used PASCAL before C and I felt like I was wearing a straitjacket at > > times in PASCAL when I was trying to write encryption/decryption > > functions and had to find ways to fiddle with bits. Similar things > > were easy in C, and are even easier in many more recent languages such > > as Python. > > PASCAL was not the first language I learned. I won't pretend I had to do > anything very complicated, or do much bit-twiddling. It was, though, the > first one (except probably for FORTH) I enjoyed programming with more than > I > disliked the boiler-plate formalities. > > > The distinction between teaching a first language, or one that is so > > cautious as to catch and prevent all mistakes it can, is not for > > people willing to be bolder or work faster or write routines that can > > be used more generally. > > > > What has not been mentioned is that languages like python go a step > > further and allow a function to return many arguments and even a > > varying number of arguments, as well as none at all. To do anything > > like that in PASCAL > > (or C, for that matter) > > > took some thought on how to make some structure you could fill out > > then return as a single value that the receiving code had to sort of > > decode and perhaps deal with parts that could hold a union of several > > things. Can a compiler or interpreter easily verify you did something > > reasonable, as compared to say python that allows something like: > > > > (res1, res2, _, res4, _, rest) = f(x) > > Yes, that's one of the good things about Python, how it makes working with > tuples so easy and natural. OTOH, harking back to PASCAL for a minute, it > had enumerations and sets long before Python got them. > > > The above tells the interpreter you expect perhaps 6 or more results > > and what to do with them. > > > > > > > > -----Original Message----- From: Python-list > > On Behalf Of > > Thomas Passin Sent: Sunday, January 1, 2023 1:03 PM To: > > python-list at python.org Subject: Re: NoneType List > > > > On 1/1/2023 8:47 AM, Stefan Ram wrote: > >> Thomas Passin writes: > >>> Guido had been working on the ABC language for some years before he > >>> developed Python. ABC was intended mainly as a teaching and > >>> prototyping language. > >> > >> In those days, there used to be a language called "Pascal". Pascal > >> had a dichotomy between "functions" and "procedures". A call to a > >> function was intended to have a value. A call to a procedure was > >> intended to have an effect. > > > > Wirth developed Pascal as a teaching language. IIRC, originally it was > > taught to students before there were any implementations. I did most > > of my programming with Turbo Pascal for many years. Just to clarify > > what you wrote above, in Pascal a "procedure" does not return anything > > while a "function" does. > > > > I really liked (Turbo) Pascal and I hated C back then. No wonder I > > like Python so much. It must be something about how my mind works. > > > >> For some beginners, the difference between a value and and effect can > >> be hard to grasp. So, Pascal's distinction helps to hammer that home. > >> > >> Experienced programmers know the difference and do no longer require > >> the effort of the language to teach it to them. > >> > >> The time when someone is a beginner and still struggles to understand > >> the difference between values and effects usually is significantly > >> shorter than the later time where he has understood it and is > >> programming productively, so it might be better when the language is > >> adapted to people who already have understood the difference. > >> > >> > > > > -- https://mail.python.org/mailman/listinfo/python-list > > > > -- > https://mail.python.org/mailman/listinfo/python-list > > -- > https://mail.python.org/mailman/listinfo/python-list > From avi.e.gross at gmail.com Mon Jan 2 01:13:58 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Mon, 2 Jan 2023 01:13:58 -0500 Subject: Python-list Digest, Vol 232, Issue 1 In-Reply-To: References: Message-ID: <012001d91e71$66974e30$33c5ea90$@gmail.com> Well explained, Roger. Your explanation reminds me why some languages very deliberately do not want the C operators of pre/post increment/decrement. Similar to your argument, code in C like: Y = X++ Or Y = ++X And similarly the -- versions, have a sort of side effect of changing X either before or after the "value" is used. Code like: X += 1 Or X = X + 1 Followed by Y = X Can be longer but a tad clearer, as is code like: Y = X X -= 1 I have seen code in languages that support this that can take some head scratching to figure out when combinations using parentheses and multiple instances of ++/-- are used pre and/or post. Then again, Python added the Walrus operator recently that also allows a kind of change within an expression that can be quite useful but can be viewed as a sort of change to an object and a return of the new value. >>> x = 1 >>> y = (x := 5) >>> y 5 >>> z = (x := x * 2) + (x := x + 1) >>> z 21 >>> x 11 I posit constructs such as the above may have similarities to changing an object and returning the new updated object. Maybe not the same but ... Avi -----Original Message----- From: Python-list On Behalf Of Christman, Roger Graydon Sent: Sunday, January 1, 2023 7:31 PM To: python-list at python.org Subject: Re: Python-list Digest, Vol 232, Issue 1 Re: Nonetype List In my introductory programming course, I have drawn some attention to this behavior regarding mutating lists. Indeed, Python is very consistent with its behavior: Any function that mutates a list parameter does not return that list as a return value. For one thing, there is no need to return that value, because the caller still owns the list parameter that has been modified. But secondly, (and what I find especially important), is that returning the modified list would lead too many program bugs or misunderstandings. For example, if append did return the list, you might see this: x = [1,2,3] y = x.append(4) z = y.append(5) The principal of 'least surprise' would cause a casual reader to believe that x retains the value of [1,2,3], y would have the value of [1,2,3,4], and z would contain [1,2,3,4,5]. So it would be very surprising indeed to discover that x contains [1,2,3,4,5], especially after that statement that makes no reference to x. Since append modifies the list in place, returning that list would make x, y, and z all aliases of each other, and aliasing is a source of many bugs that are very hard to find. So a recommendation that I make to my class (and which coincides with Python behavior), is to NEVER return a modified list as a return value, but only to return lists that were newly created within the function. So to support this principal of 'least surprise', the append method above would necessarily create new lists for y and z that are not aliases to x. Why Python does not do that is a very obvious cases of run-time efficiency (constant time to append vs. linear to recreate a new list). And as another observation, I have my students review all of the methods defined for the list object, and they are all very consistent. Most of them either define a return value, or modify the list parameter, but almost none do both. The sole exception is the pop() function that modified a list and returns a value, but that returned value still is not the modified list, so the aliasing problem will never arise. So, I am very happy with this Python language decision -- it allows for the most efficient means to modify a list in place and also very much reduce the danger of aliasing bugs. Roger Christman Pennsylvania State University ________________________________ From: Python-list on behalf of python-list-request at python.org Sent: Sunday, January 1, 2023 12:00 PM To: python-list at python.org Subject: Python-list Digest, Vol 232, Issue 1 Send Python-list mailing list submissions to python-list at python.org To subscribe or unsubscribe via the World Wide Web, visit https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.pytho n.org%2Fmailman%2Flistinfo%2Fpython-list&data=05%7C01%7Cdvl%40psu.edu%7C744c 83fc485a4b1c79db08daec19a436%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C63 8081892123929669%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzI iLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=etYqO01OszhEpqgjLeKQTM C9b3wT0sc%2FcN8oJo9eEhk%3D&reserved=0 or, via email, send a message with subject or body 'help' to python-list-request at python.org You can reach the person managing the list at python-list-owner at python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Python-list digest..." Today's Topics: 1. Re: NoneType List (Thomas Passin) 2. Re: NoneType List (MRAB) 3. Re: NoneType List (dn) 4. RE: NoneType List (avi.e.gross at gmail.com) 5. Re: NoneType List (Thomas Passin) 6. Re: NoneType List (Greg Ewing) 7. RE: NoneType List (avi.e.gross at gmail.com) 8. Re: NoneType List (Chris Angelico) 9. RE: NoneType List (avi.e.gross at gmail.com) 10. Re: NoneType List (Chris Angelico) 11. Re: NoneType List (Thomas Passin) ---------------------------------------------------------------------- Message: 1 Date: Sat, 31 Dec 2022 12:07:25 -0500 From: Thomas Passin To: python-list at python.org Subject: Re: NoneType List Message-ID: <3eb7480c-88f7-72cf-af66-c0072928bf3f at tompassin.net> Content-Type: text/plain; charset=UTF-8; format=flowed Oops, my reply got lost somehow. Here it is: Everyone's answer to date has been too complicated. What is going on is that list.append() changes the list in place. It returns nothing. If you want to append an item and then assign the result to a new list, you have to do just that: l1.append(item) # If we want a *copy* of the appended list: l2 = l1[:] # Changes to l2 will not change l1 # If we want another name for the appended list: l2 = l1 # Changes to l2 will change l1 since they are the same object list.sort() also operates in place. There is a function sorted() that returns the sorted list (without changing the original list). The same thing is true of set.add(). The set is changed in place, and nothing is returned. On 12/31/2022 10:50 AM, Thomas Passin wrote: > Happy New Year, everybody! > I'm new in the Python List, new in Python world, and new in coding. > A few days (weeks?) ago, I faced a problem trying to write a program > for an exercise. I asked for help and nobody answered. > In the meantime, I found a part of the solution, but a part still > remains a mystery for me. Please run this small snippet, and help. > Thanks > > a = [1, 2] > print() > > print("a ==", a) > print("type(a) is", type(a)) > > b = a.append(3) > print("\nb =", "a.append(3)") > print("b ==", b) > print("type(b) is", type(b)) > > c = ['1', '2'] > print("\nc ==", c) > print("type(c) is", type(c)) > > d = c.append('3') > print("\nd =", "c.append('3')") > print("d ==", d) > print("type(d) is", type(d)) > > """ > I mean: why b = a.append(something) is the None type, and how to make > a new list that contains all the items from a and some new items? > I wasn't able to solve it in a few hours :( Now I know: > """ > > crta = '=' > print('\n', 4 * ' ', crta * (len('The solution:')), sep='') > print(3 * ' ', 'The solution:') > print(4 * ' ', crta * (len('The solution:')), sep='') print('\nThe > solution is the slice, an element of Python syntax that > allows') > print('to make a brand new copy of a list, or parts of a list.') > print('The slice actually copies the list\'s contents, not the list\'s > name:') > > print() > a = [1, 2] > print("a ==", a) > print("type(a) is", type(a)) > > b = a[:] > print("\nb = a[:]") > print("b ==", b) > b.append(3) > print("\nb =", "b.append(3)") > print("b ==", b) > print("type(b) is", type(b)) > print("\na ==", a) > > print('But I still don't know why "b = a.append(something)" is the > None > type.') > print('Is there anybody out there?!') ------------------------------ Message: 2 Date: Sat, 31 Dec 2022 17:37:39 +0000 From: MRAB To: python-list at python.org Subject: Re: NoneType List Message-ID: <2321d9bc-5eb3-c1b9-0425-020b54dd39e8 at mrabarnett.plus.com> Content-Type: text/plain; charset=UTF-8; format=flowed On 2022-12-31 05:45, Goran Ikac wrote: > Happy New Year, everybody! > I'm new in the Python List, new in Python world, and new in coding. > A few days (weeks?) ago, I faced a problem trying to write a program > for an exercise. I asked for help and nobody answered. > In the meantime, I found a part of the solution, but a part still > remains a mystery for me. Please run this small snippet, and help. > Thanks > > a = [1, 2] > print() > > print("a ==", a) > print("type(a) is", type(a)) > > b = a.append(3) > print("\nb =", "a.append(3)") > print("b ==", b) > print("type(b) is", type(b)) > > c = ['1', '2'] > print("\nc ==", c) > print("type(c) is", type(c)) > > d = c.append('3') > print("\nd =", "c.append('3')") > print("d ==", d) > print("type(d) is", type(d)) > > """ > I mean: why b = a.append(something) is the None type, and how to make > a new list that contains all the items from a and some new items? > I wasn't able to solve it in a few hours :( Now I know: > """ > > crta = '=' > print('\n', 4 * ' ', crta * (len('The solution:')), sep='') > print(3 * ' ', 'The solution:') > print(4 * ' ', crta * (len('The solution:')), sep='') print('\nThe > solution is the slice, an element of Python syntax that > allows') > print('to make a brand new copy of a list, or parts of a list.') > print('The slice actually copies the list\'s contents, not the list\'s > name:') > > print() > a = [1, 2] > print("a ==", a) > print("type(a) is", type(a)) > > b = a[:] > print("\nb = a[:]") > print("b ==", b) > b.append(3) > print("\nb =", "b.append(3)") > print("b ==", b) > print("type(b) is", type(b)) > print("\na ==", a) > > print('But I still don't know why "b = a.append(something)" is the > None > type.') > print('Is there anybody out there?!') Methods that modify in-place usually return None. "a.append(something)" modifies (appends to) the list 'a' and returns None. If you want to a new line with something at the end try "b = a + [something]". ------------------------------ Message: 3 Date: Sun, 1 Jan 2023 09:58:35 +1300 From: dn To: python-list at python.org Subject: Re: NoneType List Message-ID: <2b238dce-3e9f-9bee-27a0-d163cb6c7cef at DancesWithMice.info> Content-Type: text/plain; charset=UTF-8; format=flowed On 31/12/2022 18.45, Goran Ikac wrote: ... > A few days (weeks?) ago, I faced a problem trying to write a program > for an exercise. I asked for help and nobody answered. Looking back over the last six months of List-Archives, your name does not appear against a single post. This may explain why "nobody answered". However, ten hours after the above/first message, you posted again. This time as "Thomas Passin". That was followed an hour-or-so later, with a reply-to-self saying: "Everyone's answer to date has been too complicated", and then basically repeating the information previously-provided by a number of contributors. It then goes on to talk about copying, sorting, adding, and even sets. Much of which you had earlier said: "I know". Was there a further question in there? Which part of which answer did you find "too complicated"? Did you try the experiments suggested, and read the references-provided? Please ask a further question detailing what you have understood, and what is still mystifying. Folk here endeavor to be helpful (see also, earlier reference to the Python-Tutor list). When asking a question here, please try to reduce the problem to its simplest form, so that you're not asking volunteers to read multiple-screens of irrelevant code. It will help to ask one question at a time, or to carefully separate multiple questions. Yes, this can be difficult when one is learning. That said, we all started somewhere and are happy to help you to become a valued colleague! -- Regards, =dn ------------------------------ Message: 4 Date: Sat, 31 Dec 2022 17:36:59 -0500 From: To: Subject: RE: NoneType List Message-ID: <006501d91d68$6538dd10$2faa9730$@gmail.com> Content-Type: text/plain; charset="us-ascii" It depends on what people consider too complicated. I find it a tad complicated when someone posts using two different ID, and then wonders ... The question related to taking a list and extending it and using the result in an assignment statement. There were several inter-related questions people responded to. One was asking why something did not work as expected. Several answers pointed out it was because it was not designed the way expected and reading the manual page or other reference material would help explain that you need to use code the way it was intended and not as you wish it. Other aspects of similar code that do what you expect were shown and you could pick any you liked. A second question some were answering is various ways to get the functionality wanted. Some were simple enough like using "+" and others did seem complex as they showed many variations on copying an object or how to make a subclass that effectively substitutes a method that returns the internally changed object. And, of course, we had the philosophical question of why the feature was designed to not return anything (well, NULL) rather than return the changed object. Returning nothing is arguably slightly more efficient in the many cases where the return value is simply ignored and thus tossed. But as mentioned, it would be nice for some purposes, including chaining, to be able to write something like result = lst.add_ret("item").sort_ret() As mentioned, this can still be easily done using something like: result = sorted(lst + "item") -----Original Message----- From: Python-list On Behalf Of dn Sent: Saturday, December 31, 2022 3:59 PM To: python-list at python.org Subject: Re: NoneType List On 31/12/2022 18.45, Goran Ikac wrote: ... > A few days (weeks?) ago, I faced a problem trying to write a program > for an exercise. I asked for help and nobody answered. Looking back over the last six months of List-Archives, your name does not appear against a single post. This may explain why "nobody answered". However, ten hours after the above/first message, you posted again. This time as "Thomas Passin". That was followed an hour-or-so later, with a reply-to-self saying: "Everyone's answer to date has been too complicated", and then basically repeating the information previously-provided by a number of contributors. It then goes on to talk about copying, sorting, adding, and even sets. Much of which you had earlier said: "I know". Was there a further question in there? Which part of which answer did you find "too complicated"? Did you try the experiments suggested, and read the references-provided? Please ask a further question detailing what you have understood, and what is still mystifying. Folk here endeavor to be helpful (see also, earlier reference to the Python-Tutor list). When asking a question here, please try to reduce the problem to its simplest form, so that you're not asking volunteers to read multiple-screens of irrelevant code. It will help to ask one question at a time, or to carefully separate multiple questions. Yes, this can be difficult when one is learning. That said, we all started somewhere and are happy to help you to become a valued colleague! -- Regards, =dn -- https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.pytho n.org%2Fmailman%2Flistinfo%2Fpython-list&data=05%7C01%7Cdvl%40psu.edu%7C744c 83fc485a4b1c79db08daec19a436%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C63 8081892123929669%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzI iLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=etYqO01OszhEpqgjLeKQTM C9b3wT0sc%2FcN8oJo9eEhk%3D&reserved=0 ------------------------------ Message: 5 Date: Sat, 31 Dec 2022 18:32:04 -0500 From: Thomas Passin To: python-list at python.org Subject: Re: NoneType List Message-ID: <11625e0f-8975-b929-a2ca-e14ed74a19b0 at tompassin.net> Content-Type: text/plain; charset=UTF-8; format=flowed On 12/31/2022 3:58 PM, dn wrote: > On 31/12/2022 18.45, Goran Ikac wrote: > ... >> A few days (weeks?) ago, I faced a problem trying to write a program >> for an exercise. I asked for help and nobody answered. > > Looking back over the last six months of List-Archives, your name does > not appear against a single post. This may explain why "nobody answered". > > However, ten hours after the above/first message, you posted again. > This time as "Thomas Passin". That message was probably a mistaken one from me. I had composed a reply but through some mental glitch had to re-do it. I managed to send it with the only quoted thread but not the reply I had wanted to include. So I added my reply and sent it again. It was probably confusing, and I'm sorry about that. > That was followed an hour-or-so later, with a reply-to-self saying: > "Everyone's answer to date has been too complicated", and then > basically repeating the information previously-provided by a number of > contributors. > > It then goes on to talk about copying, sorting, adding, and even sets. > Much of which you had earlier said: "I know". Was there a further > question in there? > > Which part of which answer did you find "too complicated"? > > Did you try the experiments suggested, and read the references-provided? > > Please ask a further question detailing what you have understood, and > what is still mystifying. Folk here endeavor to be helpful (see also, > earlier reference to the Python-Tutor list). > > When asking a question here, please try to reduce the problem to its > simplest form, so that you're not asking volunteers to read > multiple-screens of irrelevant code. It will help to ask one question > at a time, or to carefully separate multiple questions. > > Yes, this can be difficult when one is learning. That said, we all > started somewhere and are happy to help you to become a valued colleague! > ------------------------------ Message: 6 Date: Sun, 1 Jan 2023 13:21:05 +1300 From: Greg Ewing To: python-list at python.org Subject: Re: NoneType List Message-ID: Content-Type: text/plain; charset=UTF-8; format=flowed On 1/01/23 11:36 am, avi.e.gross at gmail.com wrote: > And, of course, we had the philosophical question of why the feature > was designed to not return anything ... rather than return the changed > object. My understanding is that Guido designed it that way to keep a clear separation between mutating and non-mutating methods, and to help catch mistakes resulting from mixing them up. -- Greg ------------------------------ Message: 7 Date: Sat, 31 Dec 2022 22:17:56 -0500 From: To: "'Greg Ewing'" , Subject: RE: NoneType List Message-ID: <00e601d91d8f$a4e59b20$eeb0d160$@gmail.com> Content-Type: text/plain; charset="us-ascii" Agreed, there are lots of pro/con arguments and the feature is what it is historically and not trivial to change. Inline changes to an object make sense to just be done "silently" and if there are errors, they propagate the usual way. As Guido was a major influence at that time, one view was seen as more important and prevailed. Had a language like that been created today, I wonder if some designs might have looked a bit different so that some functions could be called with optional arguments that specified what the user wanted returned. In particular, besides a function where you add a value returning nothing/None, there may be room for other choices and any choice hard-wired in would eleicit complaints from others. lst.add("Value") could also return one of several other things such as a pointer to the upgraded object but also of a pointer to the object as it looked before the change (not a serious proposal) or a True/False value specifying if the change was able to be completed (such as running out of memory, or the addition not being something you can put in the object) or even return what was added or how many objects are now in the object at the top level, or how many times the method was called so far! I suspect, at the expense of some overhead, you could just add an argument to many kinds of methods or even functions such as returning='option' that guides what you want returned, with the default often being what Guido and others currently have set. Python already allows functions to return anything they feel like, so this probably would not break many things. Of course there are other paths in that direction, such as setting an attribute of the list/object first that affects how things get returned but that seems more cumbersome and makes all kinds of errors more likely. Still, that is a path often used by some Python modules where objects are created and then tweaked to behave in various ways when later methods are invoked. But is any of it needed? The current system generally works fine and we have seen many ways to get other results without tampering with the current implementation. This may be yet another example of people who come to python with pre-existing bias, such as insisting it work like another language they have used, or wanting the computer to do what they MEANT rather than what they explicitly or implicitly programmed! -----Original Message----- From: Python-list On Behalf Of Greg Ewing Sent: Saturday, December 31, 2022 7:21 PM To: python-list at python.org Subject: Re: NoneType List On 1/01/23 11:36 am, avi.e.gross at gmail.com wrote: > And, of course, we had the philosophical question of why the feature > was designed to not return anything ... rather than return the changed > object. My understanding is that Guido designed it that way to keep a clear separation between mutating and non-mutating methods, and to help catch mistakes resulting from mixing them up. -- Greg -- https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.pytho n.org%2Fmailman%2Flistinfo%2Fpython-list&data=05%7C01%7Cdvl%40psu.edu%7C744c 83fc485a4b1c79db08daec19a436%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C63 8081892123929669%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzI iLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=etYqO01OszhEpqgjLeKQTM C9b3wT0sc%2FcN8oJo9eEhk%3D&reserved=0 ------------------------------ Message: 8 Date: Sun, 1 Jan 2023 14:23:13 +1100 From: Chris Angelico To: python-list at python.org Subject: Re: NoneType List Message-ID: Content-Type: text/plain; charset="UTF-8" On Sun, 1 Jan 2023 at 14:19, wrote: > Had a language like that been created today, I wonder if some designs > might have looked a bit different so that some functions could be > called with optional arguments that specified what the user wanted returned. Frankly, I doubt it. While you can argue "returning self is more useful" vs "returning None is more clear if you get it wrong", having options does NOT improve things, and just makes everything more complicated, slower, harder to comprehend, and generally worse to work with. A language should have some sort of consistent behaviour and stick to it. If that's not possible, an object type should at least have that. ChrisA ------------------------------ Message: 9 Date: Sat, 31 Dec 2022 23:16:10 -0500 From: To: "'Chris Angelico'" , Subject: RE: NoneType List Message-ID: <00fd01d91d97$c78994d0$569cbe70$@gmail.com> Content-Type: text/plain; charset="us-ascii" Chris, There is much to say about consistent behavior as compared to flexibility and convenience. I have seen other languages provide functions for example, where the result can vary and often cause confusion. R had a function that would sometimes notice the result could be simplified and return that. Guess what? It caused lots of problems and an option was added that said it should NOT simplify and always return the same kind of thing. Consider what happens if a calculation that returned a matrix would decide that is there was only one columns, it would return a vector/array/whatever as a one-dimensional result and if the calculation produced a 1 by 1 matrix, it would simply return a scalar value! But it may be a feature you want in some cases, albeit rarely when the results of the function are fed into something that is not expecting it. Various forms of polymorphism can be helpful but can also be very confusing. Some solutions are what I have described in earlier messages and some languages blur distinctions. Again, in R, there may not be a scalar as all you have is a vector of length one. But matrices even of 1-D are not vectors and I have had to interconvert between them to make some code run. I am not making comparisons in the sense that nothing other languages choose to do is binding on what Python should do. Still, I think it is wrong to suggest that it does not often do partially ambiguous things from some perspective. Many functions will take a variety of arguments and return something reasonable but different each time. As a dumb example, what does a simple function like max() return if fed just integers, just floating point or a combination of both? It seems to return whatever type the maximum indicates. But it also accepts characters and sorts them appropriately returning type 'str' and probably accepts and returns all kinds of objects if you specify a key function. Is that so much different than we are discussing in that there isn't any absolute consistency and things can go various ways in terms of return value? Heck, I can even make max() return None! -----Original Message----- From: Python-list On Behalf Of Chris Angelico Sent: Saturday, December 31, 2022 10:23 PM To: python-list at python.org Subject: Re: NoneType List On Sun, 1 Jan 2023 at 14:19, wrote: > Had a language like that been created today, I wonder if some designs > might have looked a bit different so that some functions could be > called with optional arguments that specified what the user wanted returned. Frankly, I doubt it. While you can argue "returning self is more useful" vs "returning None is more clear if you get it wrong", having options does NOT improve things, and just makes everything more complicated, slower, harder to comprehend, and generally worse to work with. A language should have some sort of consistent behaviour and stick to it. If that's not possible, an object type should at least have that. ChrisA -- https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.pytho n.org%2Fmailman%2Flistinfo%2Fpython-list&data=05%7C01%7Cdvl%40psu.edu%7C744c 83fc485a4b1c79db08daec19a436%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C63 8081892123929669%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzI iLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=etYqO01OszhEpqgjLeKQTM C9b3wT0sc%2FcN8oJo9eEhk%3D&reserved=0 ------------------------------ Message: 10 Date: Sun, 1 Jan 2023 15:23:59 +1100 From: Chris Angelico To: python-list at python.org Subject: Re: NoneType List Message-ID: Content-Type: text/plain; charset="UTF-8" On Sun, 1 Jan 2023 at 15:16, wrote: > > Chris, > > There is much to say about consistent behavior as compared to > flexibility and convenience. > > I have seen other languages provide functions for example, where the > result can vary and often cause confusion. R had a function that would > sometimes notice the result could be simplified and return that. Guess > what? It caused lots of problems and an option was added that said it > should NOT simplify and always return the same kind of thing. The option was presumably added because backward compatibility is important, but it would have been better to not need the option in the first place. > As a dumb example, what does a simple function like max() return if > fed just integers, just floating point or a combination of both? It > seems to return whatever type the maximum indicates. But it also > accepts characters and sorts them appropriately returning type 'str' > and probably accepts and returns all kinds of objects if you specify a key function. > > Is that so much different than we are discussing in that there isn't > any absolute consistency and things can go various ways in terms of > return value? Heck, I can even make max() return None! > ... yes? So? It still always returns the largest item in the list, for some definition of "largest". You'll never have that value returned wrapped up in a single-element list, though. ChrisA ------------------------------ Message: 11 Date: Sun, 1 Jan 2023 08:11:02 -0500 From: Thomas Passin To: python-list at python.org Subject: Re: NoneType List Message-ID: <71caabcd-65a0-8b3f-a971-79cd3f4415e1 at tompassin.net> Content-Type: text/plain; charset=UTF-8; format=flowed On 12/31/2022 10:17 PM, avi.e.gross at gmail.com wrote: > Agreed, there are lots of pro/con arguments and the feature is what it > is historically and not trivial to change. Inline changes to an object > make sense to just be done "silently" and if there are errors, they > propagate the usual way. > > As Guido was a major influence at that time, one view was seen as > more important and prevailed. > > Had a language like that been created today, I wonder if some designs > might have looked a bit different so that some functions could be > called with optional arguments that specified what the user wanted returned. Guido had been working on the ABC language for some years before he developed Python. ABC was intended mainly as a teaching and prototyping language. Guido probably had a good sense of what things worked well and what didn't in that usage. IIRC, Python did not originally have classes or instances, and a "fluent" style of programming probably wasn't in use yet. Having an object return itself after an operation is useful (mostly, perhaps) for fluent programming (that is, the style where one can write X.method1().method2().method3() ...). > In particular, besides a function where you add a value returning > nothing/None, there may be room for other choices and any choice > hard-wired in would eleicit complaints from others. > > lst.add("Value") > > could also return one of several other things such as a pointer to the > upgraded object but also of a pointer to the object as it looked > before the change (not a serious proposal) or a True/False value > specifying if the change was able to be completed (such as running out > of memory, or the addition not being something you can put in the > object) or even return what was added or how many objects are now in > the object at the top level, or how many times the method was called so far! > > I suspect, at the expense of some overhead, you could just add an > argument to many kinds of methods or even functions such as > returning='option' that guides what you want returned, with the > default often being what Guido and others currently have set. > > Python already allows functions to return anything they feel like, so > this probably would not break many things. > > Of course there are other paths in that direction, such as setting an > attribute of the list/object first that affects how things get > returned but that seems more cumbersome and makes all kinds of errors > more likely. Still, that is a path often used by some Python modules > where objects are created and then tweaked to behave in various ways when later methods are invoked. > > But is any of it needed? The current system generally works fine and > we have seen many ways to get other results without tampering with the > current implementation. > > This may be yet another example of people who come to python with > pre-existing bias, such as insisting it work like another language > they have used, or wanting the computer to do what they MEANT rather > than what they explicitly or implicitly programmed! > > > -----Original Message----- > From: Python-list > On Behalf Of > Greg Ewing > Sent: Saturday, December 31, 2022 7:21 PM > To: python-list at python.org > Subject: Re: NoneType List > > On 1/01/23 11:36 am, avi.e.gross at gmail.com wrote: >> And, of course, we had the philosophical question of why the feature >> was designed to not return anything ... rather than return the >> changed object. > > My understanding is that Guido designed it that way to keep a clear > separation between mutating and non-mutating methods, and to help > catch mistakes resulting from mixing them up. > > -- > Greg > -- > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail > .python.org%2Fmailman%2Flistinfo%2Fpython-list&data=05%7C01%7Cdvl%40ps > u.edu%7C744c83fc485a4b1c79db08daec19a436%7C7cf48d453ddb4389a9c1c115526 > eb52e%7C0%7C0%7C638081892123929669%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4 > wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7 > C&sdata=etYqO01OszhEpqgjLeKQTMC9b3wT0sc%2FcN8oJo9eEhk%3D&reserved=0 > ------------------------------ Subject: Digest Footer -- https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.pytho n.org%2Fmailman%2Flistinfo%2Fpython-list&data=05%7C01%7Cdvl%40psu.edu%7C744c 83fc485a4b1c79db08daec19a436%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C63 8081892123929669%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzI iLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=etYqO01OszhEpqgjLeKQTM C9b3wT0sc%2FcN8oJo9eEhk%3D&reserved=0 ------------------------------ End of Python-list Digest, Vol 232, Issue 1 ******************************************* -- https://mail.python.org/mailman/listinfo/python-list From avi.e.gross at gmail.com Mon Jan 2 12:31:06 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Mon, 2 Jan 2023 12:31:06 -0500 Subject: NoneType List In-Reply-To: References: <2b238dce-3e9f-9bee-27a0-d163cb6c7cef@DancesWithMice.info> <006501d91d68$6538dd10$2faa9730$@gmail.com> <00e601d91d8f$a4e59b20$eeb0d160$@gmail.com> <71caabcd-65a0-8b3f-a971-79cd3f4415e1@tompassin.net> <582ee874-bc37-9933-216c-b32921dce153@tompassin.net> <00b501d91e4f$ff0704c0$fd150e40$@gmail.com> Message-ID: <002e01d91ecf$fed6bdd0$fc843970$@gmail.com> Alan, I stand corrected as my path never led me back to any form of PASCAL. And, frankly, my path rarely led me again to having to do what we describe as twiddling bits with the minor exception when doing something like setting the bits needed to specify what permissions should be associated with a file in the UNIX world. What you say is largely true of programming languages in general. Many are created with some paradigm in mind that sounds like an idea until it meets reality and compromises and adjustments are made. When new languages follow that have the ability to gain from prior experience, they may indeed come up with a more nuanced paradigm. I have been studying JavaScript and Node.js lately, for no special reason, and see that as an example of sorts. The former was largely created to run inside a browser and NOT do anything harmful to the user's machine. Lots of parts normally included in other programming languages such as Python were not only deliberately left out but the code often monitors some things to make sure you do not try anything sneaky. But the language took off and evolved and at some point seemed to people to be a good tool to use on their servers too, and especially since the two sides could exchange relatively live objects such as with JSON. The paradigm had to change as most such programs written in what is now Node.js or other names, actually had to do things to the server including reading and writing files. So they had to add quite a bit and struggled at times to keep other parts of the languages similar as they evolved semi-independently. In my opinion, it remains a futile work in progress. Has Python managed the version 2 versus version 3 schism to the point where enough users have migrated their code and new users avoid version 2? Other languages have had to split when big enough changes were made. -----Original Message----- From: Alan Gauld Sent: Monday, January 2, 2023 3:01 AM To: avi.e.gross at gmail.com; python-list at python.org Subject: Re: RE: NoneType List On 02/01/2023 02:14, avi.e.gross at gmail.com wrote: > I used PASCAL before C and I felt like I was wearing a straitjacket at > times in PASCAL when I was trying to write encryption/decryption > functions and had to find ways to fiddle with bits. Similar things > were easy in C, and are even easier in many more recent languages such as Python. That's true of pure Pascal. But Thomas was talking about Turbo Pascal which had extra functions and features for all those "real world" type things. (And you could insert some inline assembler if all else failed) It also relaxed the ludicrously strict typing slightly. Turbo Pascal made Pascal a joy and I still use Delphi for Windows programming today. TP also introduced classes to Pascal (although Apple had already done so for the Mac and Borland basically ported the syntax to the PC). -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From list1 at tompassin.net Mon Jan 2 12:32:56 2023 From: list1 at tompassin.net (Thomas Passin) Date: Mon, 2 Jan 2023 12:32:56 -0500 Subject: Fwd: About the Python In-Reply-To: References: Message-ID: <5f15b214-edbe-3eb3-0175-174f4bb2b7aa@tompassin.net> Since you have an immediate need to have working installations, I suggest that you downgrade to an earlier version of Python. V3.11.1 is new and some binary libraries (such as numpy) may not be working correctly yet with the latest version. Your students will not need any of the new features that 3.11 may provide. On 1/2/2023 12:27 AM, Ramya M wrote: > ---------- Forwarded message --------- > From: Ramya M > Date: Mon, Jan 2, 2023, 9:58 AM > Subject: About the Python > To: > > > This is from JNN College of Engineering, Shimoga. we are facing some > problems while using python. Please can you resolve this issue. > > We are using python 3.11.1 (64 bit) for windows 10. but while installing > the "numpy and matplotlib" packages we are getting errors. In some cases > after installation of the above said packages we are getting errors while > working. > > So, Please can you people resolve this issue as the python labs are going > on and students are facing some difficulties. > > Thanking you > > > Regards > Ramya M > Instructor, ECE Dept. > JNNCE, Shimoga From mats at wichmann.us Mon Jan 2 14:16:56 2023 From: mats at wichmann.us (Mats Wichmann) Date: Mon, 2 Jan 2023 12:16:56 -0700 Subject: Fwd: About the Python In-Reply-To: <5f15b214-edbe-3eb3-0175-174f4bb2b7aa@tompassin.net> References: <5f15b214-edbe-3eb3-0175-174f4bb2b7aa@tompassin.net> Message-ID: On 1/2/23 10:32, Thomas Passin wrote: > Since you have an immediate need to have working installations, I > suggest that you downgrade to an earlier version of Python. V3.11.1 is > new and some binary libraries (such as numpy) may not be working > correctly yet with the latest version.? Your students will not need any > of the new features that 3.11 may provide. > > On 1/2/2023 12:27 AM, Ramya M wrote: >> ---------- Forwarded message --------- >> From: Ramya M >> Date: Mon, Jan 2, 2023, 9:58 AM >> Subject: About the Python >> To: >> >> >> This is from JNN College of Engineering, Shimoga. we are facing some >> problems while using python. Please can you resolve this issue. >> >> We are using python 3.11.1 (64 bit) for windows 10. but while installing >> the "numpy and matplotlib" packages we are getting errors. In some cases >> after installation of the above said packages we are getting errors while >> working. >> >> So, Please can you people resolve this issue as the python labs are going >> on and students are facing some difficulties. >> >> Thanking you >> >> >> Regards >> Ramya M >> Instructor, ECE Dept. >> JNNCE, Shimoga Second these comments... you don't have to be in a rush to upgrade to a new Python if there are a lot of complex dependencies. There's a site where you can do a quick check for some of the more popular addons: https://pyreadiness.org That said, the packages you mention should be available for 3.11 - though there could be some dependencies in your chain causing problems. You haven't given any actionable information. "Getting errors" is a meaningless statement without details. People minded to help aren't going to be able to do so. From grant.b.edwards at gmail.com Mon Jan 2 16:59:03 2023 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Mon, 02 Jan 2023 13:59:03 -0800 (PST) Subject: NoneType List References: <2b238dce-3e9f-9bee-27a0-d163cb6c7cef@DancesWithMice.info> <006501d91d68$6538dd10$2faa9730$@gmail.com> <00e601d91d8f$a4e59b20$eeb0d160$@gmail.com> <71caabcd-65a0-8b3f-a971-79cd3f4415e1@tompassin.net> <582ee874-bc37-9933-216c-b32921dce153@tompassin.net> <00b501d91e4f$ff0704c0$fd150e40$@gmail.com> Message-ID: <63b353a7.050a0220.3b4b4.c9b8@mx.google.com> On 2023-01-02, wrote: > I used PASCAL before C and I felt like I was wearing a straitjacket at times > in PASCAL when I was trying to write encryption/decryption functions and had > to find ways to fiddle with bits. Similar things were easy in C, and are > even easier in many more recent languages such as Python. Yonks ago (early 80s), I used Pascal to write an RTOS kernel and call processing firmware for a cell-site radio running a 16-bit microprocessor (Z8000). It worked great. However, that Pascal compiler had a few extensions designed specifically to support embedded systems use on bare metal -- things like specific length signed/unsigned interger types (with binary operations), pointers to raw memory and configurable bounds-checking. Even with wide use of those extensions it was easier to write code that "worked the first time" than it was with C. This was in the early 80's, and C compilers didn't hold your hand as much as they do today. -- Grant From torriem at gmail.com Mon Jan 2 17:00:20 2023 From: torriem at gmail.com (Michael Torrie) Date: Mon, 2 Jan 2023 15:00:20 -0700 Subject: Fwd: About the Python In-Reply-To: References: Message-ID: On 1/1/23 22:27, Ramya M wrote: > This is from JNN College of Engineering, Shimoga. we are facing some > problems while using python. Please can you resolve this issue. Without any further information on your part, we can only guess at what the problems might be. Crystal ball says that Thomas' suggestion will probably solve your problem. But since we don't know what the problems even are, this is just a wild guess. > We are using python 3.11.1 (64 bit) for windows 10. but while installing > the "numpy and matplotlib" packages we are getting errors. In some cases > after installation of the above said packages we are getting errors while > working. You'll need to provide more information than that. What are these errors? Copy and paste the text into your message. Attachments are not allowed on this list. From grant.b.edwards at gmail.com Mon Jan 2 17:01:16 2023 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Mon, 02 Jan 2023 14:01:16 -0800 (PST) Subject: NoneType List References: <2b238dce-3e9f-9bee-27a0-d163cb6c7cef@DancesWithMice.info> <006501d91d68$6538dd10$2faa9730$@gmail.com> <00e601d91d8f$a4e59b20$eeb0d160$@gmail.com> <71caabcd-65a0-8b3f-a971-79cd3f4415e1@tompassin.net> <582ee874-bc37-9933-216c-b32921dce153@tompassin.net> <00b501d91e4f$ff0704c0$fd150e40$@gmail.com> Message-ID: <63b3542c.020a0220.add4f.c056@mx.google.com> On 2023-01-02, Alan Gauld wrote: > On 02/01/2023 02:14, avi.e.gross at gmail.com wrote: >> I used PASCAL before C and I felt like I was wearing a straitjacket at times >> in PASCAL when I was trying to write encryption/decryption functions and had >> to find ways to fiddle with bits. Similar things were easy in C, and are >> even easier in many more recent languages such as Python. > > That's true of pure Pascal. But Thomas was talking about Turbo Pascal > which had extra functions and features for all those "real world" type > things. (And you could insert some inline assembler if all else failed) > It also relaxed the ludicrously strict typing slightly. Turbo Pascal > made Pascal a joy and I still use Delphi for Windows programming today. > > TP also introduced classes to Pascal (although Apple had already done > so for the Mac and Borland basically ported the syntax to the PC). TP was indeed a joy to work with. It made it trivial to do simple graphics on an IBM-PC and it was FAST -- both to write in, to debug, and in raw execution speed. -- Grant From list1 at tompassin.net Mon Jan 2 18:22:35 2023 From: list1 at tompassin.net (Thomas Passin) Date: Mon, 2 Jan 2023 18:22:35 -0500 Subject: NoneType List In-Reply-To: <63b3542c.020a0220.add4f.c056@mx.google.com> References: <2b238dce-3e9f-9bee-27a0-d163cb6c7cef@DancesWithMice.info> <006501d91d68$6538dd10$2faa9730$@gmail.com> <00e601d91d8f$a4e59b20$eeb0d160$@gmail.com> <71caabcd-65a0-8b3f-a971-79cd3f4415e1@tompassin.net> <582ee874-bc37-9933-216c-b32921dce153@tompassin.net> <00b501d91e4f$ff0704c0$fd150e40$@gmail.com> <63b3542c.020a0220.add4f.c056@mx.google.com> Message-ID: On 1/2/2023 5:01 PM, Grant Edwards wrote: > On 2023-01-02, Alan Gauld wrote: >> On 02/01/2023 02:14, avi.e.gross at gmail.com wrote: >>> I used PASCAL before C and I felt like I was wearing a straitjacket at times >>> in PASCAL when I was trying to write encryption/decryption functions and had >>> to find ways to fiddle with bits. Similar things were easy in C, and are >>> even easier in many more recent languages such as Python. >> >> That's true of pure Pascal. But Thomas was talking about Turbo Pascal >> which had extra functions and features for all those "real world" type >> things. (And you could insert some inline assembler if all else failed) >> It also relaxed the ludicrously strict typing slightly. Turbo Pascal >> made Pascal a joy and I still use Delphi for Windows programming today. >> >> TP also introduced classes to Pascal (although Apple had already done >> so for the Mac and Borland basically ported the syntax to the PC). > > TP was indeed a joy to work with. It made it trivial to do simple > graphics on an IBM-PC and it was FAST -- both to write in, to debug, > and in raw execution speed. FAST, yes, even with 256K memory and only floppy drives - my first PC (compatible) machine. From bowman at montana.com Mon Jan 2 22:52:10 2023 From: bowman at montana.com (rbowman) Date: 3 Jan 2023 03:52:10 GMT Subject: NoneType List References: <2b238dce-3e9f-9bee-27a0-d163cb6c7cef@DancesWithMice.info> <006501d91d68$6538dd10$2faa9730$@gmail.com> <00e601d91d8f$a4e59b20$eeb0d160$@gmail.com> <71caabcd-65a0-8b3f-a971-79cd3f4415e1@tompassin.net> <582ee874-bc37-9933-216c-b32921dce153@tompassin.net> <00b501d91e4f$ff0704c0$fd150e40$@gmail.com> <63b353a7.050a0220.3b4b4.c9b8@mx.google.com> Message-ID: On Mon, 02 Jan 2023 13:59:03 -0800 (PST), Grant Edwards wrote: > Yonks ago (early 80s), I used Pascal to write an RTOS kernel and call > processing firmware for a cell-site radio running a 16-bit > microprocessor (Z8000). It worked great. However, that Pascal compiler > had a few extensions designed specifically to support embedded systems > use on bare metal -- things like specific length signed/unsigned > interger types (with binary operations), pointers to raw memory and > configurable bounds-checking. Around that time I had a source of income from writing extensions. My client tended to hire engineers fro the University of Maine where Pascal was the didactic language. Wirth's Pascal was characterized as a language very good at telling secrets to itself. Controlling robotic arms, gathering information from process controllers, and so forth wasn't on the menu. As for Turbo Pascal, I had been using the BDS C subset on a CP/M system. I installed TP, did the requisite hello world, and I thought something was broken. I had to convince myself the compiler had spit out an executable that fast and hadn't crashed. From c.buhtz at posteo.jp Tue Jan 3 10:35:23 2023 From: c.buhtz at posteo.jp (c.buhtz at posteo.jp) Date: Tue, 3 Jan 2023 15:35:23 +0000 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? Message-ID: <4NmcJ022hFz6tpm@submission01.posteo.de> Hello, this posting isn't about asking for a technical solution. My intention is to understand the design decision Python's core developers made in context of that topic. The logging module write everything to stderr no matter which logging level is used. The argparse module does it more differentiated. If arguments are mandatory but none are given then argparse produce a short version of the usage info; on stderr. If the user request the usage info via "-h" it goes to stdout. This is the behavior I would expect. Why does logging behave different? DEBUG and INFO imho should go to stdout not stderr. Of course I could modify the handlers etc. to workaround this. But I assume that there was something in mind of the Python developers when they decided that. My goal is not to divide between the use of print() or logging.info() in my own code. This would mess up a lot. From list1 at tompassin.net Tue Jan 3 11:26:52 2023 From: list1 at tompassin.net (Thomas Passin) Date: Tue, 3 Jan 2023 11:26:52 -0500 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: <4NmcJ022hFz6tpm@submission01.posteo.de> References: <4NmcJ022hFz6tpm@submission01.posteo.de> Message-ID: <5baf3919-dbe3-f04c-8b4c-62ed0fd5a736@tompassin.net> On 1/3/2023 10:35 AM, c.buhtz at posteo.jp wrote: > Hello, > > this posting isn't about asking for a technical solution. My intention > is to understand the design decision Python's core developers made in > context of that topic. > > The logging module write everything to stderr no matter which logging > level is used. > > The argparse module does it more differentiated. If arguments are > mandatory but none are given then argparse produce a short version of > the usage info; on stderr. If the user request the usage info via "-h" > it goes to stdout. This is the behavior I would expect. > > Why does logging behave different? DEBUG and INFO imho should go to > stdout not stderr. I do not have any special insight into the developers' minds, but here is my own reaction. Logging output is normally not the primary output of a program. For example, you might want to capture the program's output and do some processing on it. In that case you don't want to have to parse out the logging text so as to avoid processing it. If not to stdout, then where should logging output go? The only two possibilities seem to be stderr, or a known file. Also, I think it would be confusing to sometimes have logging output go to stdout and other times go to stderr. In the case of argparse, the messages are the intended program output, so to me it makes sense that error announcements should go to stderr and requests for help information should go to stdout. There is probably no single right way to do it, but the way it's being done makes sense to me, FWIW. > > Of course I could modify the handlers etc. to workaround this. But I > assume that there was something in mind of the Python developers when > they decided that. > > My goal is not to divide between the use of print() or logging.info() > in my own code. This would mess up a lot. From python at mrabarnett.plus.com Tue Jan 3 11:36:53 2023 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 3 Jan 2023 16:36:53 +0000 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: <4NmcJ022hFz6tpm@submission01.posteo.de> References: <4NmcJ022hFz6tpm@submission01.posteo.de> Message-ID: <81c9582a-e8fd-006d-6963-0ae73a63de32@mrabarnett.plus.com> On 2023-01-03 15:35, c.buhtz at posteo.jp wrote: > Hello, > > this posting isn't about asking for a technical solution. My intention > is to understand the design decision Python's core developers made in > context of that topic. > > The logging module write everything to stderr no matter which logging > level is used. > > The argparse module does it more differentiated. If arguments are > mandatory but none are given then argparse produce a short version of > the usage info; on stderr. If the user request the usage info via "-h" > it goes to stdout. This is the behavior I would expect. > > Why does logging behave different? DEBUG and INFO imho should go to > stdout not stderr. > > Of course I could modify the handlers etc. to workaround this. But I > assume that there was something in mind of the Python developers when > they decided that. > > My goal is not to divide between the use of print() or logging.info() > in my own code. This would mess up a lot. The purpose of stderr is to display status messages, logging and error messages, even user prompts, and not mess up the program's actual output. This is important on a *nix system where you might be piping the output of one program into the input of another. From Richard at damon-family.org Tue Jan 3 11:40:33 2023 From: Richard at damon-family.org (Richard Damon) Date: Tue, 3 Jan 2023 11:40:33 -0500 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? Message-ID: ? > On Jan 3, 2023, at 10:38 AM, c.buhtz at posteo.jp wrote: > ?Hello, > > this posting isn't about asking for a technical solution. My intention > is to understand the design decision Python's core developers made in > context of that topic. > > The logging module write everything to stderr no matter which logging > level is used. > > The argparse module does it more differentiated. If arguments are > mandatory but none are given then argparse produce a short version of > the usage info; on stderr. If the user request the usage info via "-h" > it goes to stdout. This is the behavior I would expect. > > Why does logging behave different? DEBUG and INFO imho should go to > stdout not stderr. > > Of course I could modify the handlers etc. to workaround this. But I > assume that there was something in mind of the Python developers when > they decided that. > > My goal is not to divide between the use of print() or logging.info() > in my own code. This would mess up a lot. > -- > https://mail.python.org/mailman/listinfo/python-list My thought are, that a ?Log? should be a COMPLETE output of what is logged. Logged messages are NOT intended to be a message to the current user, but for a post operation analysis. Remember, a message sent by logging.info may never be seen, as that level of logging may be turned off. (You are providing ways to control how much is being set to the log, especially if going to the ?screen?). As such, stderr is a often better spot to send the log than stdout, as the program may well be using stdout to interact with the user, and you don?t want the interaction cluttered with the logging. Also, I am fairly sure you can configure the logger to do what you want by just configuring two different logging handlers, one for stderr that does just errors, and one for stdout that shows all messages. If you don?t want the errors on stderr and stdout, just log the errors to a different logger which is on an error.* hierarchy and don?t let that hierarchy propagate up to the root logger. From gweatherby at uchc.edu Tue Jan 3 11:52:24 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Tue, 3 Jan 2023 16:52:24 +0000 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: <81c9582a-e8fd-006d-6963-0ae73a63de32@mrabarnett.plus.com> References: <4NmcJ022hFz6tpm@submission01.posteo.de> <81c9582a-e8fd-006d-6963-0ae73a63de32@mrabarnett.plus.com> Message-ID: Really doesn?t have much to do with Python and very much with standard streams, which go back as far as the 1950s. https://en.wikipedia.org/wiki/Standard_streams When a user types -h / --help to a Python argparse script, the output of the script is the help message. The standard behavior is to print that and exit, regardless of any other arguments that are passed. So its behavior is consistent with the concept of standard out. From: Python-list on behalf of MRAB Date: Tuesday, January 3, 2023 at 11:41 AM To: python-list at python.org Subject: Re: What should go to stdout/stderr and why Python logging write everything to stderr? *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** On 2023-01-03 15:35, c.buhtz at posteo.jp wrote: > Hello, > > this posting isn't about asking for a technical solution. My intention > is to understand the design decision Python's core developers made in > context of that topic. > > The logging module write everything to stderr no matter which logging > level is used. > > The argparse module does it more differentiated. If arguments are > mandatory but none are given then argparse produce a short version of > the usage info; on stderr. If the user request the usage info via "-h" > it goes to stdout. This is the behavior I would expect. > > Why does logging behave different? DEBUG and INFO imho should go to > stdout not stderr. > > Of course I could modify the handlers etc. to workaround this. But I > assume that there was something in mind of the Python developers when > they decided that. > > My goal is not to divide between the use of print() or logging.info() > in my own code. This would mess up a lot. The purpose of stderr is to display status messages, logging and error messages, even user prompts, and not mess up the program's actual output. This is important on a *nix system where you might be piping the output of one program into the input of another. -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!jni-tegcOXxLEkj6ohH0wbjz0Gd2OmHirJln0S1hX5z3nT6MUnqw4-ZMgaJGkzT228O9zSczGws4tyRERCMntNGR$ From eryksun at gmail.com Tue Jan 3 13:31:38 2023 From: eryksun at gmail.com (Eryk Sun) Date: Tue, 3 Jan 2023 12:31:38 -0600 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: <4NmcJ022hFz6tpm@submission01.posteo.de> References: <4NmcJ022hFz6tpm@submission01.posteo.de> Message-ID: On 1/3/23, c.buhtz at posteo.jp wrote: > > If the user request the usage info via "-h" it goes to stdout. The standard file for application output is sys.stdout. Note that sys.stdout may be buffered, particularly if it isn't a tty. When a file is buffered, writes are aggregated and only written to the OS file when the buffer fills up or is manually flushed. > Why does logging behave different? DEBUG and INFO imho should go to > stdout not stderr. The standard file for error messages and other diagnostic information is sys.stderr. This file should never be buffered. From Keith.S.Thompson+u at gmail.com Tue Jan 3 13:45:46 2023 From: Keith.S.Thompson+u at gmail.com (Keith Thompson) Date: Tue, 03 Jan 2023 10:45:46 -0800 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? References: <4NmcJ022hFz6tpm@submission01.posteo.de> <81c9582a-e8fd-006d-6963-0ae73a63de32@mrabarnett.plus.com> Message-ID: <87sfgro3s5.fsf@nosuchdomain.example.com> MRAB writes: [...] > The purpose of stderr is to display status messages, logging and error > messages, even user prompts, and not mess up the program's actual > output. This is important on a *nix system where you might be piping > the output of one program into the input of another. I would expect user prompts to be written to stdout, or perhaps to some system-specific stream like the current tty, not to stderr. If a program has user prompts, it probably doesn't make sense to pipe its output to the input of another. -- Keith Thompson (The_Other_Keith) Keith.S.Thompson+u at gmail.com Working, but not speaking, for XCOM Labs void Void(void) { Void(); } /* The recursive call of the void */ From list1 at tompassin.net Tue Jan 3 14:33:52 2023 From: list1 at tompassin.net (Thomas Passin) Date: Tue, 3 Jan 2023 14:33:52 -0500 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: References: Message-ID: <849f9af8-8d55-d753-42fb-477a7ce912bc@tompassin.net> On 1/3/2023 11:51 AM, Stefan Ram wrote: > Thomas Passin writes: >> On 1/3/2023 10:35 AM, c.buhtz at posteo.jp wrote: >> Also, I think it would be confusing to sometimes have logging output go >> to stdout and other times go to stderr. > > In UNIX, the output of a program can be redirected, > so error messages written to the output might get appended > to some result file and might not be seen by the user. Yes, and in Windows, too. Note that stderr messages can be written or redirected to a file, too, and then would not be seen on the terminal. > Therefore, UNIX and C provide the unbuffered "stderr" stream > that can be redirected separately. > > The following example shows how logging might write to stdout. > > import logging > import sys > logging.getLogger().addHandler( logging.StreamHandler( sys.stdout )) > logging.error( 'example' ) This is starting to look too much like java. Yuck! From gweatherby at uchc.edu Tue Jan 3 14:52:07 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Tue, 3 Jan 2023 19:52:07 +0000 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: References: <4NmcJ022hFz6tpm@submission01.posteo.de> Message-ID: If sys.stdout is a tty, it typically flushes on newline. e. g. !/usr/bin/env python3 import time import sys print("No flush",end='',file=sys.stdout) time.sleep(2) print("flushed",file=sys.stdout) time.sleep(5) will print the ?flushed? 5 seconds before the script ends From: Python-list on behalf of Eryk Sun Date: Tuesday, January 3, 2023 at 1:33 PM To: c.buhtz at posteo.jp Cc: python-list at python.org Subject: Re: What should go to stdout/stderr and why Python logging write everything to stderr? *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** On 1/3/23, c.buhtz at posteo.jp wrote: > > If the user request the usage info via "-h" it goes to stdout. The standard file for application output is sys.stdout. Note that sys.stdout may be buffered, particularly if it isn't a tty. When a file is buffered, writes are aggregated and only written to the OS file when the buffer fills up or is manually flushed. > Why does logging behave different? DEBUG and INFO imho should go to > stdout not stderr. The standard file for error messages and other diagnostic information is sys.stderr. This file should never be buffered. -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!jGBVNrRuUThZCKMmShbuvBgggwv7FBDL9h2vW-vvehPnBHdfkrJUhohhZCgsCAqlRrDluk9c526jABrLjg$ From eryksun at gmail.com Tue Jan 3 15:29:44 2023 From: eryksun at gmail.com (Eryk Sun) Date: Tue, 3 Jan 2023 14:29:44 -0600 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: References: <4NmcJ022hFz6tpm@submission01.posteo.de> Message-ID: On 1/3/23, Weatherby,Gerard wrote: > If sys.stdout is a tty, it typically flushes on newline. e. g. Sorry if I wasn't clear. Yes, a tty file will be buffered, but it's line buffering, which isn't an issue as long as lines are written to stdout. I was referring to full buffering of pipe and disk files. This can be a problem when printing diagnostics. We went the information written to the file immediately, not whenever the buffer happens to fill up. Thus sys.stderr is unbuffered. From c.buhtz at posteo.jp Tue Jan 3 16:24:20 2023 From: c.buhtz at posteo.jp (c.buhtz at posteo.jp) Date: Tue, 03 Jan 2023 21:24:20 +0000 Subject: What should go to stdout/stderr and why Python logging write everything to =?UTF-8?Q?stderr=3F?= In-Reply-To: References: Message-ID: Am 03.01.2023 17:51 schrieb ram at zedat.fu-berlin.de: > logging.getLogger().addHandler( logging.StreamHandler( sys.stdout )) But I don't want to make all log levels go to stdout. Just DEBUG and INFO. But this would be a workaround. The main question here is why does Python deciecded to make all logs go to stderr? Maybe I totally misunderstood the intention of logging.info()?! Isn't this the "usual applicaton output"? If not, what messages should go into logging.info()? Can you name me some examples? From cs at cskk.id.au Tue Jan 3 16:29:39 2023 From: cs at cskk.id.au (Cameron Simpson) Date: Wed, 4 Jan 2023 08:29:39 +1100 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: References: Message-ID: On Jan 3, 2023, at 10:38 AM, c.buhtz at posteo.jp wrote: >> this posting isn't about asking for a technical solution. My >> intention is to understand the design decision Python's core developers made in >> context of that topic. >> >> The logging module write everything to stderr no matter which logging >> level is used. Sure. They're logs. >> The argparse module does it more differentiated. If arguments are >> mandatory but none are given then argparse produce a short version of >> the usage info; on stderr. If the user request the usage info via "-h" >> it goes to stdout. This is the behavior I would expect. Error messages to stderr. _Requested output_ to stdout. Good with me. >> Why does logging behave different? DEBUG and INFO imho should go to >> stdout not stderr. I entirely disagree. Consider a shell pipeline: cmd1 | cmd2 | cmd3 where `cmd2` and `cmd3` parse data coming down the pipes. Letting logging info into those data streams would produce garbage. This separation between core programme output and error/logging output is the basic reason that the two streams exist, and the reason logging goes to stderr by default. You're conflating "normal output" with INFO, which is not a substitute for eg `print()`. >> Of course I could modify the handlers etc. to workaround this. But I >> assume that there was something in mind of the Python developers when >> they decided that. Basicly, logging data is not the normal output. So it does not get sent to stdout. The log level doesn't change that characteristic. >> My goal is not to divide between the use of print() or logging.info() >> in my own code. This would mess up a lot. I think you should, or you should fiddle the handlers as you previous resist doing if it is your goal. But keep in mind that if you do that globally (eg in the root logger) it will affect _all_ logging calls, not merely your own, and other libraries will assume they can log at whatever level and not pollute stdout with their logs. Basicly, logging isn't "output". Cheers, Cameron Simpson From rosuav at gmail.com Tue Jan 3 16:59:42 2023 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 Jan 2023 08:59:42 +1100 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: References: Message-ID: On Wed, 4 Jan 2023 at 08:25, wrote: > > Am 03.01.2023 17:51 schrieb ram at zedat.fu-berlin.de: > > logging.getLogger().addHandler( logging.StreamHandler( sys.stdout )) > > But I don't want to make all log levels go to stdout. Just DEBUG and > INFO. But this would be a workaround. But why are DEBUG logs part of stdout? That doesn't make any sense. > The main question here is why does Python deciecded to make all logs go > to stderr? > Maybe I totally misunderstood the intention of logging.info()?! Isn't > this the "usual applicaton output"? > > If not, what messages should go into logging.info()? Can you name me > some examples? Reports of what the program is doing. ChrisA From torriem at gmail.com Tue Jan 3 17:16:22 2023 From: torriem at gmail.com (Michael Torrie) Date: Tue, 3 Jan 2023 15:16:22 -0700 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: <87sfgro3s5.fsf@nosuchdomain.example.com> References: <4NmcJ022hFz6tpm@submission01.posteo.de> <81c9582a-e8fd-006d-6963-0ae73a63de32@mrabarnett.plus.com> <87sfgro3s5.fsf@nosuchdomain.example.com> Message-ID: <321a0a02-357d-ce09-683e-0ec6d9192280@gmail.com> On 1/3/23 11:45, Keith Thompson wrote: > MRAB writes: > [...] >> The purpose of stderr is to display status messages, logging and error >> messages, even user prompts, and not mess up the program's actual >> output. This is important on a *nix system where you might be piping >> the output of one program into the input of another. > > I would expect user prompts to be written to stdout, or perhaps to some > system-specific stream like the current tty, not to stderr. If a > program has user prompts, it probably doesn't make sense to pipe its > output to the input of another. I can't think of a specific example, but I know I have piped the output of a program while at the same time interacting with a prompt on stderr. A rare thing, though. Maybe some day an interface and shell syntax will be developed to interact with an arbitrary number of standard streams. Current piping syntax really only works well with one stream and even trying to use stderr and stdout with pipes and filters in a shell is awkward. From 2QdxY4RzWzUUiLuE at potatochowder.com Tue Jan 3 17:03:41 2023 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Tue, 3 Jan 2023 17:03:41 -0500 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: References: Message-ID: On 2023-01-03 at 21:24:20 +0000, c.buhtz at posteo.jp wrote: > The main question here is why does Python deciecded to make all logs > go to stderr? It makes sense to send all logging to one destination, and stderr is that place. > Maybe I totally misunderstood the intention of logging.info()?! Isn't > this the "usual applicaton output"? Aha. This seems to be the misconception. "Usual" in the sense of "the application is running as usual," but not in the sense of "the output the users usually see." Logging is the process where applications write details that ordinarily aren't visible to the user. Logs are for forensics, after the fact. There is an entire universe of software for managing (writing, reading, archiving, retrieving, searching, etc.) log files. At my last job, we wrote gigabytes of compressed logs every hour, and read roughly two or three out of a million entries to track down questions from our users. Users wouldn't have understood 99% of what was in the logs anyway. I also worked a lot in embedded systems, like telephone switches. Users (i.e., the people with the telephones) didn't have access to the switches, but when things went wrong, I could look at the switches' logs to get the their view of what happened. > If not, what messages should go into logging.info()? Can you name me some > examples? INFO level messages usually track "regular" happy-path progress through your application: reading config files, calculating results, writing report. Again, not meant for users (who would have their own progress meters, textual or graphical), but for tracking down problems when there are problems. For one example, the name of the config file, even if you only tell the users that you found a config file. HTH From rosuav at gmail.com Tue Jan 3 17:40:59 2023 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 Jan 2023 09:40:59 +1100 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: <321a0a02-357d-ce09-683e-0ec6d9192280@gmail.com> References: <4NmcJ022hFz6tpm@submission01.posteo.de> <81c9582a-e8fd-006d-6963-0ae73a63de32@mrabarnett.plus.com> <87sfgro3s5.fsf@nosuchdomain.example.com> <321a0a02-357d-ce09-683e-0ec6d9192280@gmail.com> Message-ID: On Wed, 4 Jan 2023 at 09:17, Michael Torrie wrote: > > On 1/3/23 11:45, Keith Thompson wrote: > > MRAB writes: > > [...] > >> The purpose of stderr is to display status messages, logging and error > >> messages, even user prompts, and not mess up the program's actual > >> output. This is important on a *nix system where you might be piping > >> the output of one program into the input of another. > > > > I would expect user prompts to be written to stdout, or perhaps to some > > system-specific stream like the current tty, not to stderr. If a > > program has user prompts, it probably doesn't make sense to pipe its > > output to the input of another. > > I can't think of a specific example, but I know I have piped the output > of a program while at the same time interacting with a prompt on stderr. > A rare thing, though. That can certainly happen, but you may more often see things that go directly to the TTY rather than using the standard streams at all. For example, ssh and sudo will prompt on the terminal if they need a password, to avoid interfering with the standard streams (which can and often will be redirected). > Maybe some day an interface and shell syntax will be developed to > interact with an arbitrary number of standard streams. Current piping > syntax really only works well with one stream and even trying to use > stderr and stdout with pipes and filters in a shell is awkward. Yeah, maybe some day. And then maybe someone will use a time machine to backport it into bash. You have at least ten available, with more being possible but potentially causing conflicts: https://www.gnu.org/software/bash/manual/html_node/Redirections.html ChrisA From grant.b.edwards at gmail.com Tue Jan 3 17:51:29 2023 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Tue, 03 Jan 2023 14:51:29 -0800 (PST) Subject: What should go to stdout/stderr and why Python logging write everything to stderr? References: <4NmcJ022hFz6tpm@submission01.posteo.de> <81c9582a-e8fd-006d-6963-0ae73a63de32@mrabarnett.plus.com> <87sfgro3s5.fsf@nosuchdomain.example.com> <321a0a02-357d-ce09-683e-0ec6d9192280@gmail.com> Message-ID: <63b4b171.020a0220.32a56.c350@mx.google.com> On 2023-01-03, Michael Torrie wrote: > On 1/3/23 11:45, Keith Thompson wrote: >> MRAB writes: >> [...] >>> The purpose of stderr is to display status messages, logging and error >>> messages, even user prompts, and not mess up the program's actual >>> output. This is important on a *nix system where you might be piping >>> the output of one program into the input of another. >> >> I would expect user prompts to be written to stdout, or perhaps to some >> system-specific stream like the current tty, not to stderr. If a >> program has user prompts, it probably doesn't make sense to pipe its >> output to the input of another. > > I can't think of a specific example, but I know I have piped the output > of a program while at the same time interacting with a prompt on stderr. > A rare thing, though. Programs that ask for passwords often do it by reading/writing to fd 0 or /dev/tty so that stdout is unaffected. -- Grant From rosuav at gmail.com Tue Jan 3 21:26:00 2023 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 Jan 2023 13:26:00 +1100 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: <63b4b171.020a0220.32a56.c350@mx.google.com> References: <4NmcJ022hFz6tpm@submission01.posteo.de> <81c9582a-e8fd-006d-6963-0ae73a63de32@mrabarnett.plus.com> <87sfgro3s5.fsf@nosuchdomain.example.com> <321a0a02-357d-ce09-683e-0ec6d9192280@gmail.com> <63b4b171.020a0220.32a56.c350@mx.google.com> Message-ID: On Wed, 4 Jan 2023 at 09:52, Grant Edwards wrote: > > On 2023-01-03, Michael Torrie wrote: > > On 1/3/23 11:45, Keith Thompson wrote: > >> MRAB writes: > >> [...] > >>> The purpose of stderr is to display status messages, logging and error > >>> messages, even user prompts, and not mess up the program's actual > >>> output. This is important on a *nix system where you might be piping > >>> the output of one program into the input of another. > >> > >> I would expect user prompts to be written to stdout, or perhaps to some > >> system-specific stream like the current tty, not to stderr. If a > >> program has user prompts, it probably doesn't make sense to pipe its > >> output to the input of another. > > > > I can't think of a specific example, but I know I have piped the output > > of a program while at the same time interacting with a prompt on stderr. > > A rare thing, though. > > Programs that ask for passwords often do it by reading/writing to fd 0 > or /dev/tty so that stdout is unaffected. > Reading/writing the FD is the same as using stdout (technically you'd write to fd 1 and read from fd 0), but yes, can use /dev/tty to reach for the console directly. ChrisA From rosuav at gmail.com Tue Jan 3 21:28:08 2023 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 Jan 2023 13:28:08 +1100 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: References: Message-ID: On Wed, 4 Jan 2023 at 10:28, Stefan Ram wrote: > > c.buhtz at posteo.jp writes: > >But I don't want to make all log levels go to stdout. Just DEBUG and > >INFO. > > The following was stripped down from something some guy > posted on a web site, maybe it was "rkachach". Yet another shining example of bad code that does what someone asks, but will make life harder for the next person to come along and try to work with that program. It's like if someone asked for a way to make True equal to -1 instead of 1, for compatibility with BASIC. Bad idea. ChrisA From eryksun at gmail.com Tue Jan 3 23:11:10 2023 From: eryksun at gmail.com (Eryk Sun) Date: Tue, 3 Jan 2023 22:11:10 -0600 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: References: <4NmcJ022hFz6tpm@submission01.posteo.de> <81c9582a-e8fd-006d-6963-0ae73a63de32@mrabarnett.plus.com> <87sfgro3s5.fsf@nosuchdomain.example.com> <321a0a02-357d-ce09-683e-0ec6d9192280@gmail.com> <63b4b171.020a0220.32a56.c350@mx.google.com> Message-ID: On 1/3/23, Chris Angelico wrote: > > writing the FD is the same as using stdout Except stdout may be buffered. One should probably flush the buffer before each raw write to the file descriptor. > use /dev/tty to reach for the console directly. On Windows, open "CON" (read or write), "CONIN$" (read-write), or "CONOUT$" (read-write). Or use the more explicit device paths "\\.\CON", "\\.\CONIN$", and "\\.\CONOUT$". If the process has no console, one can call WinAPI AllocConsole() or AttachConsole(process_id) to obtain one. From sskeen0002 at gmail.com Tue Jan 3 21:21:46 2023 From: sskeen0002 at gmail.com (September Skeen) Date: Tue, 3 Jan 2023 18:21:46 -0800 (PST) Subject: hello can I be in your group? Message-ID: <76135434-c4e1-4228-a20d-6dfe7d506f59n@googlegroups.com> I was wondering if I could be in your group From aapost at idontexist.club Tue Jan 3 22:57:02 2023 From: aapost at idontexist.club (aapost) Date: Tue, 3 Jan 2023 22:57:02 -0500 Subject: Python - working with xml/lxml/objectify/schemas, datatypes, and assignments Message-ID: I am trying to wrap my head around how one goes about working with and editing xml elements since it feels more complicated than it seems it should be.. Just to get some feedback on how others might approach it and see if I am missing anything obvious that I haven't discovered yet, since maybe I am wandering off in a wrong way of thinking.. I am looking to interact with elements directly, loaded from a template, editing them, then ultimately submitting them to an API as a modified xml document. Consider the following: from lxml import objectify, etree schema = etree.XMLSchema(file="path_to_my_xsd_schema_file") parser = objectify.makeparser(schema=schema, encoding="UTF-8") xml_obj = objectify.parse("path_to_my_xml_file", parser=parser) xml_root = xml_obj.getroot() let's say I have a Version element, that is defined simply as a string in a 3rd party provided xsd schema and is set to a number 2342 in my document The xml file loads with the above code successfully against the schema But lxml objectify decides the element type is Int, and the pytype is int.. Version Version.pyval Let's say I want this loaded into a UI with a variety of dynamically loaded entry widgets so I can edit a large number of values like this and of many other different types. I can assign in one of two ways (both resulting the same) xml_root.Version = xml_root['Version'] = (if there is some other more kosher way of assignment, let me know) I can assign "2342" and the element suddenly becomes a I can assign 1.4 and the element suddenly becomes a The schema does not check during this assignment, it could be invalid, like assigning "abc" to a xs:dateTime and it does so any way. The original value is lost. The only way I see to verify against the schema again is to do so explicitly against the whole root. schema.validate(xml_root) This returns False because of the added xmlns:py, py:pytype stuff, I can strip those with: objectify.deannotate(xml_root[etree.QName(xml_root.Version.tag).localname], cleanup_namespaces=True) and get back to schema.validate(xml_root) validating True. BUT, it validates True whether the element is a String, Int, Float, etc (so long as it 'could' potentially be a string or something..).. So let's say a Version is 322.1121000, should be a string, validates against the schema as string, but is now 322.1121 (much more relevant for something like a product identification number) If it is a case where the validate remains False, I then have to manually look at the error log via schema.error_log for something like this: api_files/Basic:0:0:ERROR:SCHEMASV:SCHEMAV_CVC_DATATYPE_VALID_1_2_1: Element '{nsstuff}StartTime': 'asdfasdfa' is not a valid value of the atomic type 'xs:dateTime'. Then I have to consider how I should reject the users input.. From a UI design standpoint it just seems like a lot of added steps, and redundant work on top of a object layer that doesn't really do anything other than give me a thumbs up on the way in and a thumbs up on a way out. Rather than interacting with an object that can say your change is schema approved or not from the get-go, I instead seem to have to parse 100000+ lines of xsd and design UI interaction much more situationally and granularly to assert types and corner cases and preserve original values in duplicate structures, etc.. My original assumptions when hearing about xml features doesn't seem to exist from what I have found so far. Where schema should be the law, if my schema says something should be loaded as a string, it should be a string (or something close enough, definitely not an int or float), then attempting to assign something to it that doesn't match schema should be denied or throw an error. I am sure under the hood it would probably have performance draw backs or something.. Oh well.. Back to contemplating and tinkering.. From grant.b.edwards at gmail.com Wed Jan 4 00:20:10 2023 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Tue, 03 Jan 2023 21:20:10 -0800 (PST) Subject: What should go to stdout/stderr and why Python logging write everything to stderr? References: <4NmcJ022hFz6tpm@submission01.posteo.de> <81c9582a-e8fd-006d-6963-0ae73a63de32@mrabarnett.plus.com> <87sfgro3s5.fsf@nosuchdomain.example.com> <321a0a02-357d-ce09-683e-0ec6d9192280@gmail.com> <63b4b171.020a0220.32a56.c350@mx.google.com> Message-ID: <63b50c8a.c80a0220.69d3.36f5@mx.google.com> On 2023-01-04, Chris Angelico wrote: > On Wed, 4 Jan 2023 at 09:52, Grant Edwards wrote: >> >>> I can't think of a specific example, but I know I have piped the output >>> of a program while at the same time interacting with a prompt on stderr. >>> A rare thing, though. >> >> Programs that ask for passwords often do it by reading/writing to >> fd 0 or /dev/tty so that stdout is unaffected. > > Reading/writing the FD is the same as using stdout. No, stdout is fd 1. > (technically you'd write to fd 1 and read from fd 0), No, the whole point is _not_ to write to stdout (which would corrupt the program's output). The way I remember it was that you wrote the prompt to fd 0, and then read the password from fd 0. That way it worked even when fd 1 (stdout) was redirected. It still failed if stdin was redirected... IIRC, you can't usually write to the stdin stream in C (or Python), so you had to write(0,promptstr,strlen(promptstr)) instead of printf("%s",promptstr) or fputs(promptstr,stdin). > but yes, can use /dev/tty to reach for the console directly. That's definitely a better option, but I'm pretty sure I've seen multiple examples over the decades where fd 0 was used instead. Was /dev/tty a "recent" enough invention that I would have seen application code that was written before it existed? Or maybe I was just looking at code written by people who weren't aware of /dev/tty? -- Grant From rosuav at gmail.com Wed Jan 4 00:46:37 2023 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 Jan 2023 16:46:37 +1100 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: References: <4NmcJ022hFz6tpm@submission01.posteo.de> <81c9582a-e8fd-006d-6963-0ae73a63de32@mrabarnett.plus.com> <87sfgro3s5.fsf@nosuchdomain.example.com> <321a0a02-357d-ce09-683e-0ec6d9192280@gmail.com> <63b4b171.020a0220.32a56.c350@mx.google.com> Message-ID: On Wed, 4 Jan 2023 at 15:11, Eryk Sun wrote: > > On 1/3/23, Chris Angelico wrote: > > > > writing the FD is the same as using stdout > > Except stdout may be buffered. One should probably flush the buffer > before each raw write to the file descriptor. FDs can also be buffered. If it's buffering you want to avoid, don't mess around with exactly which one you're writing to, just flush. > > use /dev/tty to reach for the console directly. > > On Windows, open "CON" (read or write), "CONIN$" (read-write), or > "CONOUT$" (read-write). Or use the more explicit device paths > "\\.\CON", "\\.\CONIN$", and "\\.\CONOUT$". If the process has no > console, one can call WinAPI AllocConsole() or > AttachConsole(process_id) to obtain one. Yeah, /dev/tty is the Unix option, Windows has its own way, but whatever platform you're on, there's a way to directly reach the console. ChrisA From rosuav at gmail.com Wed Jan 4 00:53:44 2023 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 Jan 2023 16:53:44 +1100 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: <63b50c8a.c80a0220.69d3.36f5@mx.google.com> References: <4NmcJ022hFz6tpm@submission01.posteo.de> <81c9582a-e8fd-006d-6963-0ae73a63de32@mrabarnett.plus.com> <87sfgro3s5.fsf@nosuchdomain.example.com> <321a0a02-357d-ce09-683e-0ec6d9192280@gmail.com> <63b4b171.020a0220.32a56.c350@mx.google.com> <63b50c8a.c80a0220.69d3.36f5@mx.google.com> Message-ID: On Wed, 4 Jan 2023 at 16:21, Grant Edwards wrote: > > On 2023-01-04, Chris Angelico wrote: > > On Wed, 4 Jan 2023 at 09:52, Grant Edwards wrote: > >> > >>> I can't think of a specific example, but I know I have piped the output > >>> of a program while at the same time interacting with a prompt on stderr. > >>> A rare thing, though. > >> > >> Programs that ask for passwords often do it by reading/writing to > >> fd 0 or /dev/tty so that stdout is unaffected. > > > > Reading/writing the FD is the same as using stdout. > > No, stdout is fd 1. > > > (technically you'd write to fd 1 and read from fd 0), I assumed that you were using a shorthand. My bad. > No, the whole point is _not_ to write to stdout (which would corrupt > the program's output). The way I remember it was that you wrote the > prompt to fd 0, and then read the password from fd 0. That way it > worked even when fd 1 (stdout) was redirected. It still failed if > stdin was redirected... Writing to FD 0 assumes that (a) it was originally connected to the TTY, and (b) it hasn't been redirected. Not a reliable assumption. rosuav at sikorsky:~/tmp$ cat testfds.py import os for fd in range(3): os.write(fd, b"This is on FD %d\n" % fd) rosuav at sikorsky:~/tmp$ python3 testfds.py This is on FD 0 This is on FD 1 This is on FD 2 rosuav at sikorsky:~/tmp$ echo | python3 testfds.py Traceback (most recent call last): File "/home/rosuav/tmp/testfds.py", line 3, in os.write(fd, b"This is on FD %d\n" % fd) OSError: [Errno 9] Bad file descriptor Writing to fd 0 might work if stdout is redirected, but it won't work if stdin is redirected, so honestly, I don't think this hack is at all reliable. > > but yes, can use /dev/tty to reach for the console directly. > > That's definitely a better option, but I'm pretty sure I've seen > multiple examples over the decades where fd 0 was used instead. Was > /dev/tty a "recent" enough invention that I would have seen > application code that was written before it existed? Or maybe I was > just looking at code written by people who weren't aware of /dev/tty? No idea. Probably. It's definitely possible that someone tries it with stdout redirected and goes "oh hey, writing to fd 0 still works", and never tries it with stdin redirected. It's very common for programmers to see that something works without knowing WHY it works :) By the way, I can reinstate the behaviour of "FD 0 writes to the TTY" in bash by redirecting it: rosuav at sikorsky:~/tmp$ python3 testfds.py >/dev/null 0<>/dev/tty This is on FD 0 This is on FD 2 which is another neat demonstration that bash has more power than you will ever need... (though redirecting FDs above 2 in read/write mode can be *incredibly* useful.) ChrisA From eryksun at gmail.com Wed Jan 4 01:19:30 2023 From: eryksun at gmail.com (Eryk Sun) Date: Wed, 4 Jan 2023 00:19:30 -0600 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: <63b50c8a.c80a0220.69d3.36f5@mx.google.com> References: <4NmcJ022hFz6tpm@submission01.posteo.de> <81c9582a-e8fd-006d-6963-0ae73a63de32@mrabarnett.plus.com> <87sfgro3s5.fsf@nosuchdomain.example.com> <321a0a02-357d-ce09-683e-0ec6d9192280@gmail.com> <63b4b171.020a0220.32a56.c350@mx.google.com> <63b50c8a.c80a0220.69d3.36f5@mx.google.com> Message-ID: On 1/3/23, Grant Edwards wrote: > > That's definitely a better option, but I'm pretty sure I've seen > multiple examples over the decades where fd 0 was used instead. Was > /dev/tty a "recent" enough invention that I would have seen > application code that was written before it existed? Or maybe I was > just looking at code written by people who weren't aware of /dev/tty? POSIX has required "/dev/tty" for 20 years, or longer. It's in the 2004 edition of the spec, which was a minor update to the 2001 edition. https://pubs.opengroup.org/onlinepubs/000095399/basedefs/xbd_chap10.html > No, the whole point is _not_ to write to stdout (which would corrupt > the program's output). The way I remember it was that you wrote the > prompt to fd 0, and then read the password from fd 0. That way it > worked even when fd 1 (stdout) was redirected. It still failed if > stdin was redirected... FYI, that wouldn't work on Windows. The "\\.\CONIN$" and "\\.\CONOUT$" files can be opened with read-write access, but it's for a different purpose. A console input buffer can be read from via read() or WinAPI ReadFile(). It can also be read from using IOCTLs that work with 16-bit wide-character strings, which is the basis of WinAPI ReadConsoleW() and ReadConsoleInputW(). A console input buffer can be *written to* via the IOCTL-based function WriteConsoleInputW(). A console screen buffer can be written to via write() or WinAPI WriteFile(). It can also be written to using IOCTLs that work with 16-bit wide-character strings, which is the basis of WriteConsoleW() and WriteConsoleOutputW(). A console screen buffer can be *read from* via the IOCTL-based function ReadConsoleOutputW(). From eryksun at gmail.com Wed Jan 4 01:26:06 2023 From: eryksun at gmail.com (Eryk Sun) Date: Wed, 4 Jan 2023 00:26:06 -0600 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: References: <4NmcJ022hFz6tpm@submission01.posteo.de> <81c9582a-e8fd-006d-6963-0ae73a63de32@mrabarnett.plus.com> <87sfgro3s5.fsf@nosuchdomain.example.com> <321a0a02-357d-ce09-683e-0ec6d9192280@gmail.com> <63b4b171.020a0220.32a56.c350@mx.google.com> Message-ID: On 1/3/23, Chris Angelico wrote: > > FDs can also be buffered. If it's buffering you want to avoid, don't > mess around with exactly which one you're writing to, just flush. I meant to flush a C FILE stream or Python file before writing directly to the file descriptor, in order to avoid out-of-sequence and interlaced writes that make no sense. Any OS buffering on the file doesn't matter in this regard. From rosuav at gmail.com Wed Jan 4 01:46:26 2023 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 Jan 2023 17:46:26 +1100 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: References: <4NmcJ022hFz6tpm@submission01.posteo.de> <81c9582a-e8fd-006d-6963-0ae73a63de32@mrabarnett.plus.com> <87sfgro3s5.fsf@nosuchdomain.example.com> <321a0a02-357d-ce09-683e-0ec6d9192280@gmail.com> <63b4b171.020a0220.32a56.c350@mx.google.com> Message-ID: On Wed, 4 Jan 2023 at 17:26, Eryk Sun wrote: > > On 1/3/23, Chris Angelico wrote: > > > > FDs can also be buffered. If it's buffering you want to avoid, don't > > mess around with exactly which one you're writing to, just flush. > > I meant to flush a C FILE stream or Python file before writing > directly to the file descriptor, in order to avoid out-of-sequence and > interlaced writes that make no sense. Any OS buffering on the file > doesn't matter in this regard. True, that can help. If you're about to prompt for a password, though, OS buffering can most certainly make a difference. And it doesn't hurt to explicitly flush when you need it flushed. I've known some systems to have a trigger of "reading on FD 0 flushes FD 1", but that's not going to work when you use certain keyboard-input routines (ISTR readline was one of the few things that *didn't* have that problem, but I may be misremembering), so the general strategy was: print your prompt, flush stdout, THEN request input. ChrisA From gweatherby at uchc.edu Wed Jan 4 07:02:55 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Wed, 4 Jan 2023 12:02:55 +0000 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: <321a0a02-357d-ce09-683e-0ec6d9192280@gmail.com> References: <4NmcJ022hFz6tpm@submission01.posteo.de> <81c9582a-e8fd-006d-6963-0ae73a63de32@mrabarnett.plus.com> <87sfgro3s5.fsf@nosuchdomain.example.com> <321a0a02-357d-ce09-683e-0ec6d9192280@gmail.com> Message-ID: Dealing with stdout / stderr is bash is just syntax. I don?t always remember it off the top of my head but ? stackoverflow.com. On Linux at least it?s possible to pipe to arbitrary streams, it seems. The following uses bash to write ?Hi? to the file ?third? opened by Python. I determined the file was 5 empirically. import os import subprocess command= 'echo Hi >/dev/fd/5' fd = os.open("third",os.O_WRONLY|os.O_CREAT) os.set_inheritable(fd,True) subprocess.run(('/usr/bin/bash','-c',command),close_fds=False) From: Python-list on behalf of Michael Torrie Date: Tuesday, January 3, 2023 at 5:18 PM To: python-list at python.org Subject: Re: What should go to stdout/stderr and why Python logging write everything to stderr? *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** Maybe some day an interface and shell syntax will be developed to interact with an arbitrary number of standard streams. Current piping syntax really only works well with one stream and even trying to use stderr and stdout with pipes and filters in a shell is awkward. -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!j8qV9yx1DJI_G2F-q1fQz2LfnVYoMi40Qpk_h8bxrOcw50rVXpwScpFJSyZ212Tm9rj6T7vKgJjaIEgLRw$ From 2QdxY4RzWzUUiLuE at potatochowder.com Wed Jan 4 07:20:08 2023 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Wed, 4 Jan 2023 07:20:08 -0500 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: References: <4NmcJ022hFz6tpm@submission01.posteo.de> <81c9582a-e8fd-006d-6963-0ae73a63de32@mrabarnett.plus.com> <87sfgro3s5.fsf@nosuchdomain.example.com> <321a0a02-357d-ce09-683e-0ec6d9192280@gmail.com> Message-ID: On 2023-01-04 at 12:02:55 +0000, "Weatherby,Gerard" wrote: > Dealing with stdout / stderr is bash is just syntax. I don?t always > remember it off the top of my head but ? stackoverflow.com. https://tldp.org/LDP/abs/html/io-redirection.html > On Linux at least it?s possible to pipe to arbitrary streams, it > seems. The following uses bash to write ?Hi? to the file ?third? > opened by Python. I determined the file was 5 empirically. Empirically? > import os > import subprocess > command= 'echo Hi >/dev/fd/5' > fd = os.open("third",os.O_WRONLY|os.O_CREAT) command = f"echo Hi >/dev/fd/{fd}" Or: command = f"echo Hi >&{fd}" > os.set_inheritable(fd,True) > > subprocess.run(('/usr/bin/bash','-c',command),close_fds=False) By the time I'm that far down that path, I usually write a separate function to call fork, open, close, and exec myself (old habits die hard). In the end, there's no reward for writing and maintaining shell programs in Python. From c.buhtz at posteo.jp Wed Jan 4 08:53:23 2023 From: c.buhtz at posteo.jp (c.buhtz at posteo.jp) Date: Wed, 4 Jan 2023 13:53:23 +0000 Subject: No solution for "--verbose" (on stdout) output in Pythonds standard library? Message-ID: <4Nn9zq3cv0z6tmG@submission01.posteo.de> Hello, first I have to say that in my current and fresh humble opinion the often seen "--verbose" switch in command line applications should affect only the messages given to the users. This means messages on "stdout". That is what this question is about. The logging module is not an option because it works on stderr and is not intended to offer messages for the user but just for the system and its admins via logs (log-files, syslog, stderr redirection, ...). Using logging handlers redirecting to stdout are considered as workarounds by me and also not an option. This is not only my opinion but also part of the Python documentation: https://docs.python.org/3/howto/logging.html#when-to-use-logging I'm so detailed here about that difference between stdout and stderr because in the wild (e.g. on StackOverflow) you often find "use logging log levels" as a solution for that problem, which IMHO isn't one. Now the question: From my research on the docs it seems there is no feature in the standard library which could help me to implement "--verbose" or multiple verbosity levels like "-vvv"? I found some workarounds/hacks. https://stackoverflow.com/q/5980042/4865723 But my experience with Python as a Swiss knife is that there is always a standard solution for such basic and often reinvented things. I won't believe that each Python developer writes its own verbose feature. ;) From rosuav at gmail.com Wed Jan 4 08:59:37 2023 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 Jan 2023 00:59:37 +1100 Subject: No solution for "--verbose" (on stdout) output in Pythonds standard library? In-Reply-To: <4Nn9zq3cv0z6tmG@submission01.posteo.de> References: <4Nn9zq3cv0z6tmG@submission01.posteo.de> Message-ID: On Thu, 5 Jan 2023 at 00:54, wrote: > > Hello, > > first I have to say that in my current and fresh humble opinion the > often seen "--verbose" switch in command line applications should > affect only the messages given to the users. This means messages on > "stdout". That is what this question is about. > > The logging module is not an option because it works on stderr and is > not intended to offer messages for the user but just for the system and > its admins via logs (log-files, syslog, stderr redirection, ...). We've already explained in the other thread that these kinds of logs - which are EXACTLY what a --verbose switch usually covers - belong on stderr. I don't understand the problem here. But have you looked into argparse? I mean, you did say you researched the docs, so surely you must have found it. ChrisA From gweatherby at uchc.edu Wed Jan 4 09:46:56 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Wed, 4 Jan 2023 14:46:56 +0000 Subject: No solution for "--verbose" (on stdout) output in Pythonds standard library? In-Reply-To: <4Nn9zq3cv0z6tmG@submission01.posteo.de> References: <4Nn9zq3cv0z6tmG@submission01.posteo.de> Message-ID: A couple options. The -vvv is more of Linux thing rather than Pythonic, to my way of thinking. import argparse parser = argparse.ArgumentParser() parser.add_argument('-v',action='store_true') parser.add_argument('-vv',action='store_true') parser.add_argument('-vvv',action='store_true') args = parser.parse_args() parser = argparse.ArgumentParser() parser.add_argument('-v','--verbose',type=int,default=0,nargs='?',help="verbose level") args = parser.parse_args() level = args.verbose if args.verbose is not None else 0 print(level) Personally I just do: import argparse import logging logging.basicConfig() parser = argparse.ArgumentParser() parser.add_argument('-l', '--loglevel', default='WARN', help="Python logging level") args = parser.parse_args() logger = logging.getLogger(__name__) logger.setLevel(getattr(logging,args.loglevel)) From: Python-list on behalf of c.buhtz at posteo.jp Date: Wednesday, January 4, 2023 at 8:55 AM To: python-list at python.org Subject: No solution for "--verbose" (on stdout) output in Pythonds standard library? *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** Hello, first I have to say that in my current and fresh humble opinion the often seen "--verbose" switch in command line applications should affect only the messages given to the users. This means messages on "stdout". That is what this question is about. The logging module is not an option because it works on stderr and is not intended to offer messages for the user but just for the system and its admins via logs (log-files, syslog, stderr redirection, ...). Using logging handlers redirecting to stdout are considered as workarounds by me and also not an option. This is not only my opinion but also part of the Python documentation: https://urldefense.com/v3/__https://docs.python.org/3/howto/logging.html*when-to-use-logging__;Iw!!Cn_UX_p3!gctSoo2C-FjIeI4wfVetRVylYZ-X1te71-Q05ylEpJ_2XICGGoFbXFjrm02smi-UKx0H2EbiEXiJLfNcgTsq$ I'm so detailed here about that difference between stdout and stderr because in the wild (e.g. on StackOverflow) you often find "use logging log levels" as a solution for that problem, which IMHO isn't one. Now the question: >From my research on the docs it seems there is no feature in the standard library which could help me to implement "--verbose" or multiple verbosity levels like "-vvv"? I found some workarounds/hacks. https://urldefense.com/v3/__https://stackoverflow.com/q/5980042/4865723__;!!Cn_UX_p3!gctSoo2C-FjIeI4wfVetRVylYZ-X1te71-Q05ylEpJ_2XICGGoFbXFjrm02smi-UKx0H2EbiEXiJLU6LbUZG$ But my experience with Python as a Swiss knife is that there is always a standard solution for such basic and often reinvented things. I won't believe that each Python developer writes its own verbose feature. ;) -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!gctSoo2C-FjIeI4wfVetRVylYZ-X1te71-Q05ylEpJ_2XICGGoFbXFjrm02smi-UKx0H2EbiEXiJLYWBuS4g$ From dieter at handshake.de Wed Jan 4 09:42:28 2023 From: dieter at handshake.de (Dieter Maurer) Date: Wed, 4 Jan 2023 15:42:28 +0100 Subject: Python - working with xml/lxml/objectify/schemas, datatypes, and assignments In-Reply-To: References: Message-ID: <25525.36948.543184.270051@ixdm.fritz.box> aapost wrote at 2023-1-3 22:57 -0500: > ... >Consider the following: > >from lxml import objectify, etree >schema = etree.XMLSchema(file="path_to_my_xsd_schema_file") >parser = objectify.makeparser(schema=schema, encoding="UTF-8") >xml_obj = objectify.parse("path_to_my_xml_file", parser=parser) >xml_root = xml_obj.getroot() > >let's say I have a Version element, that is defined simply as a string >in a 3rd party provided xsd schema > > Does your schema include the third party schema? You might have a look at `PyXB`, too. It tries hard to enforce schema restrictions in Python code. From eryksun at gmail.com Wed Jan 4 10:02:48 2023 From: eryksun at gmail.com (Eryk Sun) Date: Wed, 4 Jan 2023 09:02:48 -0600 Subject: No solution for "--verbose" (on stdout) output in Pythonds standard library? In-Reply-To: <4Nn9zq3cv0z6tmG@submission01.posteo.de> References: <4Nn9zq3cv0z6tmG@submission01.posteo.de> Message-ID: On 1/4/23, c.buhtz at posteo.jp wrote: > > often seen "--verbose" switch in command line applications should > affect only the messages given to the users. This means messages on > "stdout". That is what this question is about. Is this additional context information such as help and definitions? If it's anything to do with diagnostics and telemetry, then logging to stderr is normal. In my experience the verbosity level typically applies to the latter. I might step it up as I delve deeper into a problem and need to examine specific details. From Richard at damon-family.org Wed Jan 4 10:04:33 2023 From: Richard at damon-family.org (Richard Damon) Date: Wed, 4 Jan 2023 10:04:33 -0500 Subject: No solution for "--verbose" (on stdout) output in Pythonds standard library? In-Reply-To: <4Nn9zq3cv0z6tmG@submission01.posteo.de> References: <4Nn9zq3cv0z6tmG@submission01.posteo.de> Message-ID: <44238DC1-A653-4EAC-BB3C-AF77004C901A@damon-family.org> > On Jan 4, 2023, at 8:56 AM, c.buhtz at posteo.jp wrote: > > ?Hello, > > first I have to say that in my current and fresh humble opinion the > often seen "--verbose" switch in command line applications should > affect only the messages given to the users. This means messages on > "stdout". That is what this question is about. > > The logging module is not an option because it works on stderr and is > not intended to offer messages for the user but just for the system and > its admins via logs (log-files, syslog, stderr redirection, ...). > > Using logging handlers redirecting to stdout are considered as > workarounds by me and also not an option. > > This is not only my opinion but also part of the Python documentation: > https://docs.python.org/3/howto/logging.html#when-to-use-logging > > I'm so detailed here about that difference between stdout and stderr > because in the wild (e.g. on StackOverflow) you often find "use logging > log levels" as a solution for that problem, which IMHO isn't one. > > Now the question: > From my research on the docs it seems there is no feature in the > standard library which could help me to implement "--verbose" or > multiple verbosity levels like "-vvv"? > I found some workarounds/hacks. > https://stackoverflow.com/q/5980042/4865723 > But my experience with Python as a Swiss knife is that there is always > a standard solution for such basic and often reinvented things. I won't > believe that each Python developer writes its own verbose feature. ;) > -- > https://mail.python.org/mailman/listinfo/python-list First, I would say you are incorrect that Payton ALWAYS has a standard solution for ?basic? problems, as some problems (like this) aren?t actual that easy to just provide a solution. ?Verbosity? levels, need to be defined by an application, so can?t just be provided by Python, but WILL need some effort on by the programmer. Second, Stack Overflow is NOT a good source of ?Truth? about things, it is just a source of what people on Stack Overflow think is truth, which is up to the reader to decide if it is actually usable. Now, it turns out that you CAN use argparse (or similar libraries) and logging (or similar libraries) to implement a form of verbosity. Start off by default to NOT have logging enabled (or only enable for ?High? level messages). Different levels of Verbosity can enable lower levels of logging, and perhaps move the logging from the ?default? of stderr to stdout, all of these are possible with these libraries. Whether this is an ?abuse? of the logging library or not is up to you are the programmer to decide, no one is making you do it that way, or saying that is the way it should be done, just a way it COULD be done. Personally, I might consider the logging module for this, or I might just add print statements conditioned on a verbosity level set by the argument parsing module I was using. From barry at barrys-emacs.org Wed Jan 4 10:29:35 2023 From: barry at barrys-emacs.org (Barry Scott) Date: Wed, 4 Jan 2023 15:29:35 +0000 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: References: <4NmcJ022hFz6tpm@submission01.posteo.de> <81c9582a-e8fd-006d-6963-0ae73a63de32@mrabarnett.plus.com> <87sfgro3s5.fsf@nosuchdomain.example.com> <321a0a02-357d-ce09-683e-0ec6d9192280@gmail.com> <63b4b171.020a0220.32a56.c350@mx.google.com> Message-ID: On 04/01/2023 02:26, Chris Angelico wrote: > Reading/writing the FD is the same as using stdout (technically you'd > write to fd 1 and read from fd 0), but yes, can use /dev/tty to reach > for the console directly. I think the logic is more like checking that stdin is a tty then using the tty it to read and write the tty. This works with stdout redirected. Also stdin is likely only open with read access. For example the way to prevent ssh-add from prompting in the terminal is this: $ ssh-add ~/.ssh/id_rsa References: <4NmcJ022hFz6tpm@submission01.posteo.de> <81c9582a-e8fd-006d-6963-0ae73a63de32@mrabarnett.plus.com> <87sfgro3s5.fsf@nosuchdomain.example.com> <321a0a02-357d-ce09-683e-0ec6d9192280@gmail.com> <63b4b171.020a0220.32a56.c350@mx.google.com> Message-ID: <2cd35053-62fc-2468-a74f-944f615f2f18@barrys-emacs.org> On 04/01/2023 06:46, Chris Angelico wrote: > I've known some systems to have a trigger of "reading on FD 0 flushes > FD 1" C++ has this feature: Quote from https://en.cppreference.com/w/cpp/io/cin "Once |std::cin| is constructed, std::cin.tie() returns &std::cout , and likewise, std::wcin.tie() returns &std::wcout . This means that any formatted input operation on |std::cin| forces a call to std::cout .flush() if any characters are pending for output." Barry From hjp-python at hjp.at Wed Jan 4 11:14:51 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Wed, 4 Jan 2023 17:14:51 +0100 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: References: <4NmcJ022hFz6tpm@submission01.posteo.de> <81c9582a-e8fd-006d-6963-0ae73a63de32@mrabarnett.plus.com> <87sfgro3s5.fsf@nosuchdomain.example.com> <321a0a02-357d-ce09-683e-0ec6d9192280@gmail.com> <63b4b171.020a0220.32a56.c350@mx.google.com> Message-ID: <20230104161451.jr2zwa6aamjzbpuh@hjp.at> On 2023-01-04 16:46:37 +1100, Chris Angelico wrote: > On Wed, 4 Jan 2023 at 15:11, Eryk Sun wrote: > > On 1/3/23, Chris Angelico wrote: > > > writing the FD is the same as using stdout > > > > Except stdout may be buffered. Maybe I'm overly lax in my terminology, but I frequently use the term "stdout" as a shorthand for "standard output" which refers to file descriptor 1, not just to the C FILE* of that name. Or Python's sys.stdout which is a different thing again (but also is a frontend to FD 1). > > One should probably flush the buffer > > before each raw write to the file descriptor. > > FDs can also be buffered. That depends on what you mean by "buffered". A write to an fd referring to a file on a disk will not usually affect the disk, true. But it will be immediately visible to other processes reading that file. A write to a terminal attached via a serial cable will also be written to a buffer first (in the OS or maybe inside the UART), but it will be sent to the terminal as fast as the hardware allows. So. sure, the writes are "buffered", but is that meaningful in this context? Reading from a terminal also often involves a buffer, but in a very different sense: Here the OS buffers each line to implement line editing functions. If the kernel didn't do that each application which could possibly read from a terminal would have to implement that itself. and it was decided back in the 1970's that this wasn't a good use of resources. (Of course these days many programs which didn't do that originally (like shells) do implement line editing (often via readline).) > If it's buffering you want to avoid, don't > mess around with exactly which one you're writing to, just flush. I don't think fsync() will have an effect on an FD connected to a terminal (or a pipe or a socket). hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From hjp-python at hjp.at Wed Jan 4 11:21:54 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Wed, 4 Jan 2023 17:21:54 +0100 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: References: Message-ID: <20230104162154.uzljittbs6xwtx6y@hjp.at> On 2023-01-03 21:24:20 +0000, c.buhtz at posteo.jp wrote: > The main question here is why does Python deciecded to make all logs go to > stderr? It doesn't. The Python logging system provides a plethora of handlers to log to lots of different places. Only the StreamHandler defaults to writing to stderr, and even that can log to any stream - just pass it as an argument. If none of existing handlers does what you want you can always write your own. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From list1 at tompassin.net Wed Jan 4 12:32:40 2023 From: list1 at tompassin.net (Thomas Passin) Date: Wed, 4 Jan 2023 12:32:40 -0500 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: <4NmcJ022hFz6tpm@submission01.posteo.de> References: <4NmcJ022hFz6tpm@submission01.posteo.de> Message-ID: <2f1602f1-9717-6ec6-0d5b-ac40046fdd91@tompassin.net> On 1/3/2023 10:35 AM, c.buhtz at posteo.jp wrote: > The logging module write everything to stderr no matter which logging > level is used. The OP wrote this, but it's not so by default. There is a HOW-TO page that explains this and how to get the logging package to log everything to a file, along with many other useful bits of information - https://docs.python.org/3/howto/logging.html From barry at barrys-emacs.org Wed Jan 4 12:49:57 2023 From: barry at barrys-emacs.org (Barry Scott) Date: Wed, 4 Jan 2023 17:49:57 +0000 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: References: Message-ID: On 03/01/2023 21:24, c.buhtz at posteo.jp wrote: > Am 03.01.2023 17:51 schrieb ram at zedat.fu-berlin.de: >> logging.getLogger().addHandler( logging.StreamHandler( sys.stdout )) > > But I don't want to make all log levels go to stdout. Just DEBUG and > INFO. But this would be a workaround. > > The main question here is why does Python deciecded to make all logs > go to stderr? > Maybe I totally misunderstood the intention of logging.info()?! Isn't > this the "usual applicaton output"? > > If not, what messages should go into logging.info()? Can you name me > some examples? It is up to you, the designer of an application, to decide how it works. You will take into account conventions that your users will expect you to follow. If the logging module helps you then use it, but you are not forced by logging to design your app is a particular way. The default behavior of the logging module is a generally useful default, but its only a default. You are free to setup logging to meet your needs. Barry From barry at barrys-emacs.org Wed Jan 4 12:55:45 2023 From: barry at barrys-emacs.org (Barry Scott) Date: Wed, 4 Jan 2023 17:55:45 +0000 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: References: Message-ID: <679194c9-c287-261e-dd9a-82a6106f4377@barrys-emacs.org> On 03/01/2023 21:24, c.buhtz at posteo.jp wrote: > Am 03.01.2023 17:51 schrieb ram at zedat.fu-berlin.de: >> logging.getLogger().addHandler( logging.StreamHandler( sys.stdout )) > > But I don't want to make all log levels go to stdout. Just DEBUG and > INFO. But this would be a workaround. > > The main question here is why does Python deciecded to make all logs > go to stderr? > Maybe I totally misunderstood the intention of logging.info()?! Isn't > this the "usual applicaton output"? > > If not, what messages should go into logging.info()? Can you name me > some examples? Example: write an app that prints the contents of a file. The application output is the contents of the file. The logging might be this when all is working: INFO About to open INFO Wrote bytes from The logging might be this when there is a problem: INFO About to open ERROR Failed to open - Does that help? Barry From rosuav at gmail.com Wed Jan 4 13:44:50 2023 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 Jan 2023 05:44:50 +1100 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: <679194c9-c287-261e-dd9a-82a6106f4377@barrys-emacs.org> References: <679194c9-c287-261e-dd9a-82a6106f4377@barrys-emacs.org> Message-ID: On Thu, 5 Jan 2023 at 05:06, Barry Scott wrote: > > > On 03/01/2023 21:24, c.buhtz at posteo.jp wrote: > > Am 03.01.2023 17:51 schrieb ram at zedat.fu-berlin.de: > >> logging.getLogger().addHandler( logging.StreamHandler( sys.stdout )) > > > > But I don't want to make all log levels go to stdout. Just DEBUG and > > INFO. But this would be a workaround. > > > > The main question here is why does Python deciecded to make all logs > > go to stderr? > > Maybe I totally misunderstood the intention of logging.info()?! Isn't > > this the "usual applicaton output"? > > > > If not, what messages should go into logging.info()? Can you name me > > some examples? > > Example: > > write an app that prints the contents of a file. > > The application output is the contents of the file. > > The logging might be this when all is working: > > INFO About to open > INFO Wrote bytes from > > The logging might be this when there is a problem: > > INFO About to open > ERROR Failed to open - > > Does that help? Notably, the info lines can provide context for an error. For example: INFO: Loading module XYZ WARN: Unrecognized module option frobnicate=1 INFO: Using default spamination of 4 ERROR: Insufficient ham to spaminate This tells a hypothetical (but only slightly hypothetical) story of a module that was given an incorrect or misspelled option, and which then failed to load as a consequence. Without the INFO lines, it would be harder to figure out what the problem was, but they are still part of logging, not output, and belong in the same place as the warnings and errors. ChrisA From aapost at idontexist.club Wed Jan 4 12:13:47 2023 From: aapost at idontexist.club (aapost) Date: Wed, 4 Jan 2023 12:13:47 -0500 Subject: Python - working with xml/lxml/objectify/schemas, datatypes, and assignments References: <25525.36948.543184.270051@ixdm.fritz.box> Message-ID: On 1/4/23 09:42, Dieter Maurer wrote: > aapost wrote at 2023-1-3 22:57 -0500: >> ... >> Consider the following: >> >>from lxml import objectify, etree >> schema = etree.XMLSchema(file="path_to_my_xsd_schema_file") >> parser = objectify.makeparser(schema=schema, encoding="UTF-8") >> xml_obj = objectify.parse("path_to_my_xml_file", parser=parser) >> xml_root = xml_obj.getroot() >> >> let's say I have a Version element, that is defined simply as a string >> in a 3rd party provided xsd schema >> >> > > Does your schema include the third party schema? > > You might have a look at `PyXB`, too. > It tries hard to enforce schema restrictions in Python code. Yes, to clarify, they provide the schema, which is what we use, downloaded locally. Basically just trying to remain compliant with their structures that they already define without reinventing the wheel for numerous calls and custom types, and in a way that feels more live rather than just checking validity at the end of the edits as if I were modifying the XML manually. Thank you for the suggestion, PyXB works much more like how I envisioned working with xml in my head: >>> xml_root.Version = 1231.32000 pyxb.exceptions_.SimpleTypeValueError: Type {http://www.w3.org/2001/XMLSchema}string cannot be created from: 1231.32 >>> xml_root.Version = "1231.32000" I will have to do some more testing to see how smooth the transition back to a formatted document goes, since it creates a variable for all possible fields defined in the type, even if they are optional and not there in the situational template. Thanks From hjp-python at hjp.at Thu Jan 5 06:27:49 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Thu, 5 Jan 2023 12:27:49 +0100 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: <2f1602f1-9717-6ec6-0d5b-ac40046fdd91@tompassin.net> References: <4NmcJ022hFz6tpm@submission01.posteo.de> <2f1602f1-9717-6ec6-0d5b-ac40046fdd91@tompassin.net> Message-ID: <20230105112749.cvbjo2jxcnqic5lo@hjp.at> On 2023-01-04 12:32:40 -0500, Thomas Passin wrote: > On 1/3/2023 10:35 AM, c.buhtz at posteo.jp wrote: > > The logging module write everything to stderr no matter which logging > > level is used. > > The OP wrote this, but it's not so by default. By default it is - sort of. That is all log messages go to stderr, but not all log levels are logged. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From list1 at tompassin.net Thu Jan 5 08:31:40 2023 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 5 Jan 2023 08:31:40 -0500 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: <20230105112749.cvbjo2jxcnqic5lo@hjp.at> References: <4NmcJ022hFz6tpm@submission01.posteo.de> <2f1602f1-9717-6ec6-0d5b-ac40046fdd91@tompassin.net> <20230105112749.cvbjo2jxcnqic5lo@hjp.at> Message-ID: <09dda310-56c4-a5a1-637a-33b4f143aafe@tompassin.net> On 1/5/2023 6:27 AM, Peter J. Holzer wrote: > On 2023-01-04 12:32:40 -0500, Thomas Passin wrote: >> On 1/3/2023 10:35 AM, c.buhtz at posteo.jp wrote: >>> The logging module write everything to stderr no matter which logging >>> level is used. >> >> The OP wrote this, but it's not so by default. > > By default it is - sort of. > > That is all log messages go to stderr, but not all log levels are > logged. Actually, I misread something else that made me think that non-warning messages would go to stdout, but I was mistaken. As it happens, the page in the Python docs only mention stderr twice, both in the same paragraph: "logging.lastResort A ?handler of last resort? is available through this attribute. This is a StreamHandler writing to sys.stderr with a level of WARNING, and is used to handle logging events in the absence of any logging configuration. The end result is to just print the message to sys.stderr." Nowhere else on this page is the printed output destination mentioned, except that it gets printed on the console, which could be either stderr or stdout. The logging system is so configurable that a user could set a different destination for each level of logging. So it seems that the O.P.'s original question about why the package's developers choose stderr for all levels can be answered: "They didn't". From hjp-python at hjp.at Thu Jan 5 14:18:00 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Thu, 5 Jan 2023 20:18:00 +0100 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: <09dda310-56c4-a5a1-637a-33b4f143aafe@tompassin.net> References: <4NmcJ022hFz6tpm@submission01.posteo.de> <2f1602f1-9717-6ec6-0d5b-ac40046fdd91@tompassin.net> <20230105112749.cvbjo2jxcnqic5lo@hjp.at> <09dda310-56c4-a5a1-637a-33b4f143aafe@tompassin.net> Message-ID: <20230105191800.fmmih56mtadxyji5@hjp.at> On 2023-01-05 08:31:40 -0500, Thomas Passin wrote: > The logging system is so configurable that a user could set a different > destination for each level of logging. So it seems that the O.P.'s original > question about why the package's developers choose stderr for all levels can > be answered: "They didn't". Which is almost exactly what I wrote in <20230104162154.uzljittbs6xwtx6y at hjp.at> ;-) hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From list1 at tompassin.net Thu Jan 5 14:23:58 2023 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 5 Jan 2023 14:23:58 -0500 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: <20230105191800.fmmih56mtadxyji5@hjp.at> References: <4NmcJ022hFz6tpm@submission01.posteo.de> <2f1602f1-9717-6ec6-0d5b-ac40046fdd91@tompassin.net> <20230105112749.cvbjo2jxcnqic5lo@hjp.at> <09dda310-56c4-a5a1-637a-33b4f143aafe@tompassin.net> <20230105191800.fmmih56mtadxyji5@hjp.at> Message-ID: <52a362a5-3938-3d65-fded-b3fe136e34fc@tompassin.net> On 1/5/2023 2:18 PM, Peter J. Holzer wrote: > On 2023-01-05 08:31:40 -0500, Thomas Passin wrote: >> The logging system is so configurable that a user could set a different >> destination for each level of logging. So it seems that the O.P.'s original >> question about why the package's developers choose stderr for all levels can >> be answered: "They didn't". > > Which is almost exactly what I wrote in > <20230104162154.uzljittbs6xwtx6y at hjp.at> ;-) Yup, two minds are better than one :) From grant.b.edwards at gmail.com Thu Jan 5 15:29:33 2023 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Thu, 05 Jan 2023 12:29:33 -0800 (PST) Subject: What should go to stdout/stderr and why Python logging write everything to stderr? References: <4NmcJ022hFz6tpm@submission01.posteo.de> <2f1602f1-9717-6ec6-0d5b-ac40046fdd91@tompassin.net> <20230105112749.cvbjo2jxcnqic5lo@hjp.at> <09dda310-56c4-a5a1-637a-33b4f143aafe@tompassin.net> Message-ID: <63b7332d.920a0220.77e00.fc12@mx.google.com> On 2023-01-05, Thomas Passin wrote: > The logging system is so configurable that... I find it almost impossible to use unless I copy a working example I find somewhere. ;) I'm not at all surprised that the OP didn't understand how it works. I've been writing Python programs for over 20 years, and it's beyond me. -- Grant From list1 at tompassin.net Thu Jan 5 16:09:00 2023 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 5 Jan 2023 16:09:00 -0500 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: <63b7332d.920a0220.77e00.fc12@mx.google.com> References: <4NmcJ022hFz6tpm@submission01.posteo.de> <2f1602f1-9717-6ec6-0d5b-ac40046fdd91@tompassin.net> <20230105112749.cvbjo2jxcnqic5lo@hjp.at> <09dda310-56c4-a5a1-637a-33b4f143aafe@tompassin.net> <63b7332d.920a0220.77e00.fc12@mx.google.com> Message-ID: On 1/5/2023 3:29 PM, Grant Edwards wrote: > On 2023-01-05, Thomas Passin wrote: > >> The logging system is so configurable that... > > I find it almost impossible to use unless I copy a working example I > find somewhere. ;) > > I'm not at all surprised that the OP didn't understand how it > works. I've been writing Python programs for over 20 years, and it's > beyond me. I know what you mean! I have never used the Python logging package, but I inherited a java project that uses Log4j. Talk about obscure! The python docs are a wonder of clarity beside Log4j. So most of my Log4j configuration has been by copy-pasta, with occasional surprises and Log4j bug bites. Like a handler that deleted its own previous log file when it rolled over at midnight - a "feature" that wasn't advertised. I will say, though, that having used a bit of Log4j must have been helpful, since much of what is in the Python logging package docs seemed somewhat familiar and even seemed to make sense. BTW, I was able to re-create that java project using Jython for about 90% of it. Way better! From gweatherby at uchc.edu Thu Jan 5 16:28:28 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Thu, 5 Jan 2023 21:28:28 +0000 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: <63b7332d.920a0220.77e00.fc12@mx.google.com> References: <4NmcJ022hFz6tpm@submission01.posteo.de> <2f1602f1-9717-6ec6-0d5b-ac40046fdd91@tompassin.net> <20230105112749.cvbjo2jxcnqic5lo@hjp.at> <09dda310-56c4-a5a1-637a-33b4f143aafe@tompassin.net> <63b7332d.920a0220.77e00.fc12@mx.google.com> Message-ID: logging.basicConfig() logging.info(?Nice to know?) logging.debug(?Details for when things are funky?) logging.warn(?Trouble is brewing?) From: Python-list on behalf of Grant Edwards Date: Thursday, January 5, 2023 at 3:31 PM To: python-list at python.org Subject: Re: What should go to stdout/stderr and why Python logging write everything to stderr? *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** On 2023-01-05, Thomas Passin wrote: > The logging system is so configurable that... I find it almost impossible to use unless I copy a working example I find somewhere. ;) I'm not at all surprised that the OP didn't understand how it works. I've been writing Python programs for over 20 years, and it's beyond me. -- Grant -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!l6vSXQFKppEuLS0R5gYeLYaiHyVFfs2Rapqm1oPGEtvnZ5ivQyApZcnJyNTnnH9zEVY80ajNb-HfYkNwgw8fMtsnlSOT$ From grant.b.edwards at gmail.com Thu Jan 5 16:59:23 2023 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Thu, 05 Jan 2023 13:59:23 -0800 (PST) Subject: What should go to stdout/stderr and why Python logging write everything to stderr? References: <4NmcJ022hFz6tpm@submission01.posteo.de> <2f1602f1-9717-6ec6-0d5b-ac40046fdd91@tompassin.net> <20230105112749.cvbjo2jxcnqic5lo@hjp.at> <09dda310-56c4-a5a1-637a-33b4f143aafe@tompassin.net> <63b7332d.920a0220.77e00.fc12@mx.google.com> =?utf-8?q?=3CSJ0PR14MB5864E46B?= =?utf-8?q?1AA0F95FCC5F2FB9B9FA9=40SJ0PR14MB5864=2Enamprd14=2Eprod=2Eoutlook?= =?utf-8?q?=2Ecom=3E?= Message-ID: <63b7483b.920a0220.5b8ba.0b83@mx.google.com> On 2023-01-05, Weatherby,Gerard wrote: > logging.basicConfig() > logging.info(?Nice to know?) > logging.debug(?Details for when things are funky?) > logging.warn(?Trouble is brewing?) I always seem to need something slightly more complex. Usually something like: * Specify a log level on the command line. * Allow logging to a file as specified on the command line. * Optional timestamps for log messages. I know, without doubt, that it can easily do all those things. I just never seem to be able to figure out how unless I can find example code to look at. -- Grant From list1 at tompassin.net Thu Jan 5 17:31:37 2023 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 5 Jan 2023 17:31:37 -0500 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: References: <4NmcJ022hFz6tpm@submission01.posteo.de> <2f1602f1-9717-6ec6-0d5b-ac40046fdd91@tompassin.net> <20230105112749.cvbjo2jxcnqic5lo@hjp.at> <09dda310-56c4-a5a1-637a-33b4f143aafe@tompassin.net> <63b7332d.920a0220.77e00.fc12@mx.google.com> Message-ID: On 1/5/2023 4:28 PM, Weatherby,Gerard wrote: > logging.basicConfig() > logging.info(?Nice to know?) > logging.debug(?Details for when things are funky?) > logging.warn(?Trouble is brewing?) Not quite - >>> import logging >>> logging.basicConfig() >>> logging.info("Nice to know") >>> logging.debug("Details for when things are funky") >>> logging.warn("Trouble is brewing") :1: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead WARNING:root:Trouble is brewing Only warning level messages are displayed, unless you do some configuration, as mentioned in the logging Howto page: "The default level is WARNING, which means that only events of this level and above will be tracked, unless the logging package is configured to do otherwise." So following the Howto: >>> logging.basicConfig(level=logging.DEBUG) >>> logging.info('So should this') INFO:root:So should this : Finally! Not quite straightforward, though it is in the Howto. Now about those handlers ... > From: Python-list on behalf of Grant Edwards > Date: Thursday, January 5, 2023 at 3:31 PM > To: python-list at python.org > Subject: Re: What should go to stdout/stderr and why Python logging write everything to stderr? > *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** > > On 2023-01-05, Thomas Passin wrote: > >> The logging system is so configurable that... > > I find it almost impossible to use unless I copy a working example I > find somewhere. ;) > > I'm not at all surprised that the OP didn't understand how it > works. I've been writing Python programs for over 20 years, and it's > beyond me. > > -- > Grant > > > > > -- > https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!l6vSXQFKppEuLS0R5gYeLYaiHyVFfs2Rapqm1oPGEtvnZ5ivQyApZcnJyNTnnH9zEVY80ajNb-HfYkNwgw8fMtsnlSOT$ From list1 at tompassin.net Thu Jan 5 17:41:40 2023 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 5 Jan 2023 17:41:40 -0500 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: References: <4NmcJ022hFz6tpm@submission01.posteo.de> <2f1602f1-9717-6ec6-0d5b-ac40046fdd91@tompassin.net> <20230105112749.cvbjo2jxcnqic5lo@hjp.at> <09dda310-56c4-a5a1-637a-33b4f143aafe@tompassin.net> <63b7332d.920a0220.77e00.fc12@mx.google.com> Message-ID: On 1/5/2023 4:24 PM, Stefan Ram wrote: > You often can replace threads in tkinter by coroutines using > asyncio when you write a replacement for the mainloop of > tkinter that uses asyncio. Now, try to read only the official > documentation of asyncio and tkinter and figure out only from > this how to get such a program to work! Cool! Can we have a link, please? From list1 at tompassin.net Thu Jan 5 22:44:40 2023 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 5 Jan 2023 22:44:40 -0500 Subject: asyncio and tkinter (was: What should go to stdout/stderr and why Python logging write everything to stderr?) In-Reply-To: References: <4NmcJ022hFz6tpm@submission01.posteo.de> <2f1602f1-9717-6ec6-0d5b-ac40046fdd91@tompassin.net> <20230105112749.cvbjo2jxcnqic5lo@hjp.at> <09dda310-56c4-a5a1-637a-33b4f143aafe@tompassin.net> <63b7332d.920a0220.77e00.fc12@mx.google.com> Message-ID: <6168558b-cbfb-f58b-bbb4-9e9bfc4d85bb@tompassin.net> On 1/5/2023 7:52 PM, Stefan Ram wrote: > Thomas Passin writes: >> On 1/5/2023 4:24 PM, Stefan Ram wrote: >>> You often can replace threads in tkinter by coroutines using >>> asyncio when you write a replacement for the mainloop of >>> tkinter that uses asyncio. Now, try to read only the official >>> documentation of asyncio and tkinter and figure out only from >>> this how to get such a program to work! >> Cool! Can we have a link, please? > > I do not post links, but tried to create a minimal example app. Thanks! That's not exactly obvious, is it? Congratulations on getting it working. > import asyncio > import logging > import tkinter > > # This program was tested on Windows, it is experimental. > # It should open a tkinter root window. > # Ctrl-E should create a task. You should be able to create > # a new task even while one task is already running. > # Each task will end after about 10 seconds. > # Ctrl-X should exit the process. > # Or, use the menu "Action". > > # I (Stefan Ram) designed and wrote the program, but especially > # the code in "app_async_mainloop" and "terminate_tasks" was > # written following educational material from the World-Wide Web. > > class app_class( tkinter.Tk ): > def __init__( self, *args, **kwargs ): > self.is_running = 1_000_000 > super().__init__( *args, **kwargs ) > root = self > root.protocol( "WM_DELETE_WINDOW", self.app_exit_macro ) > async def app_example_task( self, example_label ): > try: > for i in range( 10 ): > example_label[ 'text' ]=str( i ) > await asyncio.sleep( 1 ) > except asyncio.CancelledError: > pass > example_label.destroy() > def app_example_macro( self ): > root = self > example_label=tkinter.Label( root ) > example_label.pack() > asyncio.get_running_loop().create_task\ > ( self.app_example_task( example_label )) > root = self > def terminate_tasks( self ): > loop = asyncio.get_running_loop() > pending = asyncio.all_tasks( loop=loop ) > label_tasks = [] > for task in pending: > if 'app_example_task' in str( task ): > label_tasks.append( task ) > task.cancel() > group = asyncio.gather( *label_tasks, return_exceptions=True ) > try: > loop.run_until_complete( group ) > except AssertionError: > # ignoring an assertion error on Windows I do not understand > pass > def app_exit_macro( self ): > self.terminate_tasks() > self.is_running = 99 > root = self > root.destroy() > def app_add_basemenu( self, example=False ): > root = self > menubar = tkinter.Menu( root ) > menu = tkinter.Menu( menubar, tearoff=0 ) > name = "Actions"; menu = tkinter.Menu( menubar, tearoff=0 ) > if example: > text = "Example"; > callback = self.app_example_macro; menu.add_command\ > ( label=text, underline=0, command=callback, > accelerator="Control-e" ); > root.bind\ > ( "", lambda event, callback=callback:callback() ) > text = "Exit"; > callback = self.app_exit_macro > menu.add_command\ > ( label=text, underline=1, > command=callback, accelerator="Control-x" ) > root.bind\ > ( "", lambda event,callback=callback: callback() ) > menubar.add_cascade( label=name, underline=0, menu=menu ) > root.config( menu=menubar ) > async def app_async_mainloop( self ): > root = self > root.willdispatch() > root.tk.dooneevent( tkinter._tkinter.DONT_WAIT ) > while self.is_running > 0: > if self.is_running < 1_000_000: self.is_running -= 1 > try: > await asyncio.sleep\ > ( tkinter._tkinter.getbusywaitinterval() / 10_000 ) > root.tk.dooneevent( tkinter._tkinter.DONT_WAIT ) > root.update() > except tkinter.TclError: > break > async def app_async_main( self ): > try: > await self.app_async_mainloop() > except asyncio.CancelledError: > logging.debug( f'asyncio.CancelledError' ) > def app_async_run( self ): > asyncio.run( self.app_async_main() ) > > app = app_class() > app.app_add_basemenu(example=True) > app.app_async_run() > > > From jacob.kruger.work at gmail.com Fri Jan 6 03:18:24 2023 From: jacob.kruger.work at gmail.com (jacob kruger) Date: Fri, 6 Jan 2023 10:18:24 +0200 Subject: Recommendations in terms of threading, multi-threading and/or asynchronous processes/programming? - Sent Mail - Mozilla Thunderbird Message-ID: <599c7ea1-3dec-551c-e565-e0583f4eab9b@gmail.com> I am just trying to make up my mind with regards to what I should look into working with/making use of in terms of what have put in subject line? As in, if want to be able to trigger multiple/various threads/processes to run in the background, possibly monitoring their states, either via interface, or via global variables, but, possibly while processing other forms of user interaction via the normal/main process, what would be recommended? As in, for example, the following page mentions some possibilities, like threading, asyncio, etc., but, without going into too much detail: https://itnext.io/practical-guide-to-async-threading-multiprocessing-958e57d7bbb8 And, have played around with threading in the past, and, was looking into asyncio now, but, thought would rather first ask for recommendations/suggestions here? For reference, am currently working with python 3.11, or might roll back to 3.10 if relevant, but, main thing is just want to get an idea of what's simplest to make use of in this context? Thanks in advance -- Jacob Kruger +2782 413 4791 "Resistance is futile...but, acceptance is versatile..." From info at egenix.com Fri Jan 6 07:04:03 2023 From: info at egenix.com (eGenix Team) Date: Fri, 6 Jan 2023 13:04:03 +0100 Subject: =?UTF-8?Q?ANN=3a_Python_Meeting_D=c3=bcsseldorf_-_18=2e01=2e2023?= Message-ID: <808f6a23-4e45-5a21-92bf-88eac413bb87@egenix.com> /This announcement is in German since it targets a local user group//meeting in D?sseldorf, Germany/ Ank?ndigung Python Meeting D?sseldorf - Januar 2023 Ein Treffen von Python Enthusiasten und Interessierten in ungezwungener Atmosph?re. *18.01.2023, 18:00 Uhr* Raum 1, 2.OG im B?rgerhaus Stadtteilzentrum Bilk D?sseldorfer Arcaden , Bachstr. 145, 40217 D?sseldorf Programm Bereits angemeldete Vortr?ge: Marc-Andre Lemburg: ??? ??? "Einf?hrung in Event Driven Architecture (EDA)" Alexey Kuznetsov: ??? ??? "Python und Neo4j" Ioannis Georgiou: ??? ??? "A gentle introduction to AWS using CDK" Jo-fai Chow: ??? ??? "Supercharging Jupyter Notebooks for Effective Storytelling" Weitere Vortr?ge k?nnen gerne noch angemeldet werden. Bei Interesse, bitte unter info at pyddf.de melden. Startzeit und Ort Wir treffen uns um 18:00 Uhr im B?rgerhaus in den D?sseldorfer Arcaden. Das B?rgerhaus teilt sich den Eingang mit dem Schwimmbad und befindet sich an der Seite der Tiefgarageneinfahrt der D?sseldorfer Arcaden. ?ber dem Eingang steht ein gro?es "Schwimm? in Bilk" Logo. Hinter der T?r direkt links zu den zwei Aufz?gen, dann in den 2. Stock hochfahren. Der Eingang zum Raum 1 liegt direkt links, wenn man aus dem Aufzug kommt. >>> Eingang in Google Street View Corona Die Corona Einschr?nkungen sind mittlerweile aufgehoben worden. Vorsicht ist zwar immer noch geboten, aber jetzt jedem selbst ?berlassen. *?? Wichtig*: Bitte nur dann anmelden, wenn ihr absolut sicher seid, dass ihr auch kommt. Angesichts der begrenzten Anzahl Pl?tze, haben wir kein Verst?ndnis f?r kurzfristige Absagen oder No-Shows. Einleitung Das Python Meeting D?sseldorf ist eine regelm??ige Veranstaltung in D?sseldorf, die sich an Python Begeisterte aus der Region wendet. Einen guten ?berblick ?ber die Vortr?ge bietet unser PyDDF YouTube-Kanal , auf dem wir Videos der Vortr?ge nach den Meetings ver?ffentlichen. Veranstaltet wird das Meeting von der eGenix.com GmbH , Langenfeld, in Zusammenarbeit mit Clark Consulting & Research , D?sseldorf: Format Das Python Meeting D?sseldorf nutzt eine Mischung aus (Lightning) Talks und offener Diskussion. Vortr?ge k?nnen vorher angemeldet werden, oder auch spontan w?hrend des Treffens eingebracht werden. Ein Beamer mit XGA Aufl?sung steht zur Verf?gung. (Lightning) Talk Anmeldung bitte formlos per EMail an info at pyddf.de Kostenbeteiligung Das Python Meeting D?sseldorf wird von Python Nutzern f?r Python Nutzer veranstaltet. Da Tagungsraum, Beamer, Internet und Getr?nke Kosten produzieren, bitten wir die Teilnehmer um einen Beitrag in H?he von EUR 10,00 inkl. 19% Mwst. Sch?ler und Studenten zahlen EUR 5,00 inkl. 19% Mwst. Wir m?chten alle Teilnehmer bitten, den Betrag in bar mitzubringen. Anmeldung Da wir nur 25 Personen in dem angemieteten Raum empfangen k?nnen, m?chten wir bitten, sich vorher anzumelden. *Meeting Anmeldung* bitte per Meetup Weitere Informationen Weitere Informationen finden Sie auf der Webseite des Meetings: https://pyddf.de/ Viel Spa? ! -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Oct 18 2022) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ From hjp-python at hjp.at Fri Jan 6 13:19:30 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Fri, 6 Jan 2023 19:19:30 +0100 Subject: Recommendations in terms of threading, multi-threading and/or asynchronous processes/programming? - Sent Mail - Mozilla Thunderbird In-Reply-To: <599c7ea1-3dec-551c-e565-e0583f4eab9b@gmail.com> References: <599c7ea1-3dec-551c-e565-e0583f4eab9b@gmail.com> Message-ID: <20230106181930.dhzlh6msan7s4i7c@hjp.at> On 2023-01-06 10:18:24 +0200, jacob kruger wrote: > I am just trying to make up my mind with regards to what I should look into > working with/making use of in terms of what have put in subject line? > > > As in, if want to be able to trigger multiple/various threads/processes to > run in the background, possibly monitoring their states, either via > interface, or via global variables, but, possibly while processing other > forms of user interaction via the normal/main process, what would be > recommended? This depends very much on what you want to do and what the constraints and requirements are and is completely impossible to answer in the abstract. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From hjp-python at hjp.at Fri Jan 6 13:53:15 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Fri, 6 Jan 2023 19:53:15 +0100 Subject: What should go to stdout/stderr and why Python logging write everything to stderr? In-Reply-To: <63b7332d.920a0220.77e00.fc12@mx.google.com> References: <4NmcJ022hFz6tpm@submission01.posteo.de> <2f1602f1-9717-6ec6-0d5b-ac40046fdd91@tompassin.net> <20230105112749.cvbjo2jxcnqic5lo@hjp.at> <09dda310-56c4-a5a1-637a-33b4f143aafe@tompassin.net> <63b7332d.920a0220.77e00.fc12@mx.google.com> Message-ID: <20230106185315.eydqjkgbwpcnrmfv@hjp.at> On 2023-01-05 12:29:33 -0800, Grant Edwards wrote: > On 2023-01-05, Thomas Passin wrote: > > > The logging system is so configurable that... > > I find it almost impossible to use unless I copy a working example I > find somewhere. ;) I usually copy the working example from my previous project ;-). I think the general structure is clear enough. You need a formatter to format your log messages, a handler to actually write them and finally a logger to determine what goes where. I can look up the details in the docs but generally the logging is always the same (except for path names and log levels), so I can just copy the config from last time and adjust those. So I might have a config.py like this: # ... logging = { "version": 1, "disable_existing_loggers": False, "formatters": { "standard": { "format": "%(asctime)s %(levelname)s %(name)s %(funcName)s %(lineno)d | %(message)s" } }, "handlers": { "file": { "class": "switchinglogfilehandlers.TimeoutSwitchingFileHandler", "formatter": "standard", "filename": "/var/log/www/XXX.hjp.at/XXX.", }, }, "loggers": { "": { "handlers": ["file"], "level": "INFO" } } } # ... And then my application would start like this: import logging import logging.config import config logging.config.dictConfig(config.logging) log = logging.getLogger(__name__) Plus typically every other source file contains import logging log = logging.getLogger(__name__) somewhere near the start. Then I can just scatter my log.debug(...) or whatever whereever I want. When I decide that I need debug output from one module, I'll just add a logger. Or if some library is too chatty I'll add another logger to shut it up - no programming, just a few extra lines in a dict. (Instead of a config.py I might use a json file, especially if I expect the config to change often.) > I'm not at all surprised that the OP didn't understand how it > works. It probably helps to have worked with log4j before that. The structure is very similar, although I find Python logging easier to use (but then I never did much Java programming so that's probably just a matter of familiarity. hp PS: The TimeoutSwitchingFileHandler mentioned above is one I wrote myself. You can find it on PyPI, but be warned that the documentation is (still) quite lacking. -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From rosuav at gmail.com Fri Jan 6 14:19:33 2023 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 7 Jan 2023 06:19:33 +1100 Subject: Recommendations in terms of threading, multi-threading and/or asynchronous processes/programming? - Sent Mail - Mozilla Thunderbird In-Reply-To: <599c7ea1-3dec-551c-e565-e0583f4eab9b@gmail.com> References: <599c7ea1-3dec-551c-e565-e0583f4eab9b@gmail.com> Message-ID: On Sat, 7 Jan 2023 at 04:54, jacob kruger wrote: > > I am just trying to make up my mind with regards to what I should look > into working with/making use of in terms of what have put in subject line? > > > As in, if want to be able to trigger multiple/various threads/processes > to run in the background, possibly monitoring their states, either via > interface, or via global variables, but, possibly while processing other > forms of user interaction via the normal/main process, what would be > recommended? > Any. All. Whatever suits your purpose. They all have different goals, different tradeoffs. Threads are great for I/O bound operations; they're easy to work with (especially in Python), behave pretty much like just having multiple things running concurrently, and generally are the easiest to use. But you'll run into limits as your thread count climbs (with a simple test, I started seeing delays at about 10,000 threads, with more serious problems at 100,000), so it's not well-suited for huge scaling. Also, only one thread at a time can run Python code, which limits them to I/O-bound tasks like networking. Multiple processes take a lot more management. You have to carefully define your communication channels (for instance, a multiprocessing.Queue() to collect results), but they can do CPU-bound tasks in parallel. So multiprocessing is a good way to saturate all of your CPU cores. Big downsides include it being much harder to share information between the processes, and much MUCH higher resource usage than threads (with the same test as the above, I ran into limitations at just over 500 processes - way fewer than the 10,000 threads!). Asynchronous I/O runs a single thread in a single process. So like multithreading, it's only good for I/O bound tasks like networking. It's harder to work with, though, since you have to be very careful to include proper await points, and you can stall out the entire event loop with one mistake (common culprits being synchronous disk I/O, and gethostbyname). But the upside is that you get near-infinite tasks, basically just limited by available memory (or other resources). Use whichever one is right for your needs. ChrisA From gweatherby at uchc.edu Fri Jan 6 14:21:48 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Fri, 6 Jan 2023 19:21:48 +0000 Subject: Python logging (was Re: What should go to stdout/stderr and why Python logging write everything to stderr?) In-Reply-To: <20230106185315.eydqjkgbwpcnrmfv@hjp.at> References: <4NmcJ022hFz6tpm@submission01.posteo.de> <2f1602f1-9717-6ec6-0d5b-ac40046fdd91@tompassin.net> <20230105112749.cvbjo2jxcnqic5lo@hjp.at> <09dda310-56c4-a5a1-637a-33b4f143aafe@tompassin.net> <63b7332d.920a0220.77e00.fc12@mx.google.com> <20230106185315.eydqjkgbwpcnrmfv@hjp.at> Message-ID: FWIW, it wasn?t too difficult to build a logging handler to send messages to Slack. https://pypi.org/project/slack-webclient-logger/ From Keith.S.Thompson+u at gmail.com Fri Jan 6 20:02:15 2023 From: Keith.S.Thompson+u at gmail.com (Keith Thompson) Date: Fri, 06 Jan 2023 17:02:15 -0800 Subject: hello can I be in your group? References: <76135434-c4e1-4228-a20d-6dfe7d506f59n@googlegroups.com> Message-ID: <878rifnomg.fsf@nosuchdomain.example.com> September Skeen writes: > I was wondering if I could be in your group This is an unmoderated Usenet newsgroup. It doesn't have members, just people who post to it. If you want to discuss Python, just post. (Take a look around first to get an idea how how thinks work.) If you see a response from "Manosh Manosh", I recommend ignoring it. He appears to be a spammer. -- Keith Thompson (The_Other_Keith) Keith.S.Thompson+u at gmail.com Working, but not speaking, for XCOM Labs void Void(void) { Void(); } /* The recursive call of the void */ From dieter at handshake.de Sat Jan 7 13:46:23 2023 From: dieter at handshake.de (Dieter Maurer) Date: Sat, 7 Jan 2023 19:46:23 +0100 Subject: hello can I be in your group? In-Reply-To: <878rifnomg.fsf@nosuchdomain.example.com> References: <76135434-c4e1-4228-a20d-6dfe7d506f59n@googlegroups.com> <878rifnomg.fsf@nosuchdomain.example.com> Message-ID: <25529.48639.801221.593@ixdm.fritz.box> Keith Thompson wrote at 2023-1-6 17:02 -0800: >September Skeen writes: >> I was wondering if I could be in your group > >This is an unmoderated Usenet newsgroup. In fact, there are several access channels, Usenet newsgroup is one of them. Another channel is the python-list mailing list. You can subscribe to it on "python.org" -- look for community/support --> mailing lists. The maling list tends to have fewer spam then the newsgroup. From c.buhtz at posteo.jp Sun Jan 8 05:49:40 2023 From: c.buhtz at posteo.jp (c.buhtz at posteo.jp) Date: Sun, 8 Jan 2023 10:49:40 +0000 Subject: subprocess equivalent for "os.execvp()"? Message-ID: <4NqYjz4fQQz9rxL@submission02.posteo.de> Hello, is there an equivalent in the subprocess module for "os.execvp()" to replace the current process with the new called one? Kind Christian From rosuav at gmail.com Sun Jan 8 05:52:10 2023 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 8 Jan 2023 21:52:10 +1100 Subject: subprocess equivalent for "os.execvp()"? In-Reply-To: <4NqYjz4fQQz9rxL@submission02.posteo.de> References: <4NqYjz4fQQz9rxL@submission02.posteo.de> Message-ID: On Sun, 8 Jan 2023 at 21:51, wrote: > > Hello, > > is there an equivalent in the subprocess module for "os.execvp()" to > replace the current process with the new called one? It won't make a subprocess, so no. It's in the os module - under the name execvp. You found it already :) ChrisA From jacob.kruger.work at gmail.com Sun Jan 8 06:49:38 2023 From: jacob.kruger.work at gmail.com (jacob kruger) Date: Sun, 8 Jan 2023 13:49:38 +0200 Subject: Recommendations in terms of threading, multi-threading and/or asynchronous processes/programming? - Sent Mail - Mozilla Thunderbird In-Reply-To: References: <599c7ea1-3dec-551c-e565-e0583f4eab9b@gmail.com> Message-ID: <6d9c8996-dd8b-4e4f-fb29-0d3a76fcaaab@gmail.com> Ok, the specific usage case right now is that I need to set up a process pulling contents of e-mail messages from an IMAP protocol mail server, which I then populate into a postgresql database, and, since this is the inbox of a relatively large-scale CRM/support system, there are currently over 2.5 million e-mails in the inbox, but, it can grow by over 50000 per day. I already have the basic process operating, using imap_tools, but, wanted to enable you to query the process during run-time, without needing to either check logs, or query the database itself while it is on-the-go - even if this is just for initial population time-period, since later on I will just set up code to run under a form of cron job, or handling time-based repeats itself on a separate machine. Also wanted to offer the ability to either pause, or terminate processes while it's busy batch processing large chunks of e-mail messages - either send a message to the thread, or set a global variable to tell it to end the run after the current process item has finished off, just in case. So, I think that for now, threading is probably the simplest to look into. Later on, was also considering forms of low-level monitoring for UI elements, but, this is not really related to initial task, but, could almost relate to forms of non-visual gaming interfaces, for blind/VI individuals - I am myself 100% blind, but, that's not really relevant in this context. Stay well Jacob Kruger +2782 413 4791 "Resistance is futile...but, acceptance is versatile..." On 2023/01/06 21:19, Chris Angelico wrote: > On Sat, 7 Jan 2023 at 04:54, jacob kruger wrote: >> I am just trying to make up my mind with regards to what I should look >> into working with/making use of in terms of what have put in subject line? >> >> >> As in, if want to be able to trigger multiple/various threads/processes >> to run in the background, possibly monitoring their states, either via >> interface, or via global variables, but, possibly while processing other >> forms of user interaction via the normal/main process, what would be >> recommended? >> > Any. All. Whatever suits your purpose. > > They all have different goals, different tradeoffs. Threads are great > for I/O bound operations; they're easy to work with (especially in > Python), behave pretty much like just having multiple things running > concurrently, and generally are the easiest to use. But you'll run > into limits as your thread count climbs (with a simple test, I started > seeing delays at about 10,000 threads, with more serious problems at > 100,000), so it's not well-suited for huge scaling. Also, only one > thread at a time can run Python code, which limits them to I/O-bound > tasks like networking. > > Multiple processes take a lot more management. You have to carefully > define your communication channels (for instance, a > multiprocessing.Queue() to collect results), but they can do CPU-bound > tasks in parallel. So multiprocessing is a good way to saturate all of > your CPU cores. Big downsides include it being much harder to share > information between the processes, and much MUCH higher resource usage > than threads (with the same test as the above, I ran into limitations > at just over 500 processes - way fewer than the 10,000 threads!). > > Asynchronous I/O runs a single thread in a single process. So like > multithreading, it's only good for I/O bound tasks like networking. > It's harder to work with, though, since you have to be very careful to > include proper await points, and you can stall out the entire event > loop with one mistake (common culprits being synchronous disk I/O, and > gethostbyname). But the upside is that you get near-infinite tasks, > basically just limited by available memory (or other resources). > > Use whichever one is right for your needs. > > ChrisA From eryksun at gmail.com Sun Jan 8 11:22:33 2023 From: eryksun at gmail.com (Eryk Sun) Date: Sun, 8 Jan 2023 10:22:33 -0600 Subject: subprocess equivalent for "os.execvp()"? In-Reply-To: <4NqYjz4fQQz9rxL@submission02.posteo.de> References: <4NqYjz4fQQz9rxL@submission02.posteo.de> Message-ID: On 1/8/23, c.buhtz at posteo.jp wrote: > > is there an equivalent in the subprocess module for "os.execvp()" to > replace the current process with the new called one? A note for Windows users Avoid using any of the `os.exec*` functions on Windows. There's no support for replacing a Windows process image, so the `exec*()` functions simply spawn a child process and terminate the current one. This is a mess in general because a waiting parent isn't aware of the spawned descendant. It's particularly bad for a console process that inherits the console session, since two processes will try to use console I/O simultaneously. From hjp-python at hjp.at Sun Jan 8 12:11:40 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 8 Jan 2023 18:11:40 +0100 Subject: Recommendations in terms of threading, multi-threading and/or asynchronous processes/programming? - Sent Mail - Mozilla Thunderbird In-Reply-To: <6d9c8996-dd8b-4e4f-fb29-0d3a76fcaaab@gmail.com> References: <599c7ea1-3dec-551c-e565-e0583f4eab9b@gmail.com> <6d9c8996-dd8b-4e4f-fb29-0d3a76fcaaab@gmail.com> Message-ID: <20230108171140.hikmkmew2bijtwuq@hjp.at> On 2023-01-08 13:49:38 +0200, jacob kruger wrote: > Ok, the specific usage case right now is that I need to set up a process > pulling contents of e-mail messages from an IMAP protocol mail server, which > I then populate into a postgresql database, and, since this is the inbox of > a relatively large-scale CRM/support system, there are currently over 2.5 > million e-mails in the inbox, but, it can grow by over 50000 per day. This is probably I/O-bound. You will likely spend much more time waiting for the IMAP server or the database than parsing the messages. So you probably don't need multi-processing just to utilize all your cores. On the other hand you have some nicely separated task which can be parallelized, so multi-threading should help (async probably would work just as well or as badly as multi-threading but I find that harder to understand so I would discard it at this point). I might be mistaken, though: Depending on how much processing you need to do on these messages it might be worth it split the work across multiple processes. Check the CPU-usage of your process: If it's close to 100% you will probably gain significantly from multi-processing. > I already have the basic process operating, using imap_tools, but, wanted to > enable you to query the process during run-time, without needing to either > check logs, or query the database itself while it is on-the-go [...] > Also wanted to offer the ability to either pause, or terminate processes > while it's busy batch processing large chunks of e-mail messages So that would be an http (or other socket-based) interface? Should also be possible to add as an additional thread (or process). > So, I think that for now, threading is probably the simplest to look into. I agree with that assessment. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From raphael.santiago.53 at gmail.com Sun Jan 8 05:44:30 2023 From: raphael.santiago.53 at gmail.com (Raphael Santiago) Date: Sun, 8 Jan 2023 12:44:30 +0200 Subject: Suggestion: Regex string specifier like r and f Message-ID: Maybe something like re"" It should behave exactly like a raw string but would be useful for syntax highlighting and debugging. Perhaps also for type hinting expected regex input (don't know if this is feasible). From angitolol36 at gmail.com Sun Jan 8 07:54:11 2023 From: angitolol36 at gmail.com (Angitolol36) Date: Sun, 8 Jan 2023 13:54:11 +0100 Subject: Question. Message-ID: <85529512-AE98-4DE2-8AB4-872176E935A0@hxcore.ol> Hello, i installed phyton in Windows 10 22H2 and i can?t find the program. I used the repair that doesnt work. ? ? Enviado desde [1]Correo para Windows ? [2][IMG] Libre de virus.[3]www.avast.com References Visible links 1. https://go.microsoft.com/fwlink/?LinkId=550986 2. https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient 3. https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient From barry at barrys-emacs.org Sun Jan 8 16:46:18 2023 From: barry at barrys-emacs.org (Barry) Date: Sun, 8 Jan 2023 21:46:18 +0000 Subject: Question. In-Reply-To: <85529512-AE98-4DE2-8AB4-872176E935A0@hxcore.ol> References: <85529512-AE98-4DE2-8AB4-872176E935A0@hxcore.ol> Message-ID: <1DEBBDE4-A7C6-4523-BB57-D8D090A6F2EB@barrys-emacs.org> > On 8 Jan 2023, at 21:20, Angitolol36 wrote: > > ? Hello, i installed phyton in Windows 10 22H2 and i can?t find the program. > I used the repair that doesnt work. Does this help? https://docs.python.org/3/using/windows.html Barry > > > > > > Enviado desde [1]Correo para Windows > > > > [2][IMG] Libre de virus.[3]www.avast.com > > References > > Visible links > 1. https://go.microsoft.com/fwlink/?LinkId=550986 > 2. https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient > 3. https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient > -- > https://mail.python.org/mailman/listinfo/python-list From cs at cskk.id.au Sun Jan 8 16:47:56 2023 From: cs at cskk.id.au (Cameron Simpson) Date: Mon, 9 Jan 2023 08:47:56 +1100 Subject: Suggestion: Regex string specifier like r and f In-Reply-To: References: Message-ID: On 08Jan2023 12:44, Raphael Santiago wrote: >Maybe something like re"" >It should behave exactly like a raw string but would be useful for syntax >highlighting and debugging. Perhaps also for type hinting expected regex >input (don't know if this is feasible). A nice idea. (Though I'm personally reluctant to make regexps any "easier" to use than they are already, a discussion for another thread.) But the IDE/syntax help is cool. There was discussion of essentially this idea just recently here: https://discuss.python.org/t/allow-for-arbitrary-string-prefix-of-strings/19740 which you will find interesting. I think it fizzled for 2 main reasons: - string prefixes current have semantic meaning for the Python parser/interpreter, while an artbitrary prefix ... doesn't - "too hard"? to get IDEs to use it? Of these, I only find the former compelling, and one could argue that type annotations already form a "has no effect" syntax extension to Python, so we're already in that playground :-) Cheers, Cameron Simpson From barry at barrys-emacs.org Sun Jan 8 16:50:08 2023 From: barry at barrys-emacs.org (Barry) Date: Sun, 8 Jan 2023 21:50:08 +0000 Subject: Suggestion: Regex string specifier like r and f In-Reply-To: References: Message-ID: > On 8 Jan 2023, at 21:16, Raphael Santiago wrote: > > ?Maybe something like re"" > It should behave exactly like a raw string but would be useful for syntax > highlighting and debugging. Perhaps also for type hinting expected regex > input (don't know if this is feasible). This is unlikely to be implemented. See https://discuss.python.org/t/allow-for-arbitrary-string-prefix-of-strings/19740/12 for related idea discussion. Barry > -- > https://mail.python.org/mailman/listinfo/python-list > From list1 at tompassin.net Sun Jan 8 17:27:01 2023 From: list1 at tompassin.net (Thomas Passin) Date: Sun, 8 Jan 2023 17:27:01 -0500 Subject: Question. In-Reply-To: <85529512-AE98-4DE2-8AB4-872176E935A0@hxcore.ol> References: <85529512-AE98-4DE2-8AB4-872176E935A0@hxcore.ol> Message-ID: On 1/8/2023 7:54 AM, Angitolol36 wrote: > Hello, i installed phyton in Windows 10 22H2 and i can?t find the program. > I used the repair that doesnt work. This is as if you had said "I bought a car and it doesn't work". Please tell us what you did and noticed that caused you to say "i can?t find the program". > Enviado desde [1]Correo para Windows > > > > [2][IMG] Libre de virus.[3]www.avast.com > > References > > Visible links > 1. https://go.microsoft.com/fwlink/?LinkId=550986 > 2. https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient > 3. https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient From c.buhtz at posteo.jp Mon Jan 9 03:36:21 2023 From: c.buhtz at posteo.jp (c.buhtz at posteo.jp) Date: Mon, 09 Jan 2023 08:36:21 +0000 Subject: subprocess equivalent for =?UTF-8?Q?=22os=2Eexecvp=28=29=22?= =?UTF-8?Q?=3F?= In-Reply-To: References: <4NqYjz4fQQz9rxL@submission02.posteo.de> Message-ID: Dear Eryk, Am 08.01.2023 17:22 schrieb Eryk Sun: > Avoid using any of the `os.exec*` functions on Windows. There's no > support for replacing a Windows process image, so the `exec*()` > functions simply spawn a child process and terminate the current one. Thanks for bringing this up. On Python for Windows what is the appropriate way how a process can call itself again? Let me give you an example [1]: There is a project "bitcli" having two entry points [project.scripts] bitcli = "bitcli.__main__:main" bitcli-root = "bitcli.__main__:run_main_as_root_via_policykit" The first is usual. But the second does call "bitcli" via "pkexec" to give it some root rights. This application is intended to be run as user or root by the user himself. def run_main_as_root_via_policykit(): cmd = ['pkexec', '--disable-internal-agent', 'bitcli'] # See https://github.com/python/cpython/issues/39569 os.execvp(cmd[0], cmd) Is there a better way to achiev this? [1] -- From eryksun at gmail.com Mon Jan 9 05:32:16 2023 From: eryksun at gmail.com (Eryk Sun) Date: Mon, 9 Jan 2023 04:32:16 -0600 Subject: subprocess equivalent for "os.execvp()"? In-Reply-To: References: <4NqYjz4fQQz9rxL@submission02.posteo.de> Message-ID: On 1/9/23, c.buhtz at posteo.jp wrote: > > On Python for Windows what is the appropriate way how a process can call > itself again? > > Let me give you an example [1]: > There is a project "bitcli" having two entry points > > [project.scripts] > bitcli = "bitcli.__main__:main" > bitcli-root = "bitcli.__main__:run_main_as_root_via_policykit" > > The first is usual. > > But the second does call "bitcli" via "pkexec" to give it some root > rights. > > This application is intended to be run as user or root by the user > himself. > > def run_main_as_root_via_policykit(): > cmd = ['pkexec', '--disable-internal-agent', 'bitcli'] > > # See https://github.com/python/cpython/issues/39569 > os.execvp(cmd[0], cmd) The nearest equivalent on Windows, if the current process doesn't have administrator access and high integrity level, would be to spawn a process with administrator access and elevated integrity level by calling ShellExecuteExW() with the "runas" operation. The original process should wait on the spawned process and proxy its exit status. It should also add the spawned process to a kill-on-close, silent-breakaway job, such that terminating the original process also terminates the spawned process. The standard library doesn't support calling ShellExecuteExW() or working with job objects, except via ctypes. Or use the PyWin32 package. From stephen_tucker at sil.org Mon Jan 9 09:34:30 2023 From: stephen_tucker at sil.org (Stephen Tucker) Date: Mon, 9 Jan 2023 14:34:30 +0000 Subject: file.read Method Documentation (Python 2.7.10) Message-ID: Dear Python-list, Yes, I know that Python 2.x is no longer supported. I have found that the documentation for this method is misleading when the file being read is UTF-8-encoded: Instead of reading *size* bytes, the method reads *size *UTF-8 byte *sequences*. Has this error been corrected in the Python 3.x documentation? Stephen Tucker. From euyhegnrh at gmail.com Mon Jan 9 09:40:52 2023 From: euyhegnrh at gmail.com (om om) Date: Mon, 9 Jan 2023 16:40:52 +0200 Subject: =?UTF-8?B?Tm8gbW9kdWxlIG5hbWVkICdwbGF5c291bmQn4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP?= =?UTF-8?B?4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP?= Message-ID: I'm installing playsound pip install playsound but it keeps saying No module named playsound and this error occurs on other packages From python at mrabarnett.plus.com Mon Jan 9 10:25:50 2023 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 9 Jan 2023 15:25:50 +0000 Subject: =?UTF-8?B?UmU6IE5vIG1vZHVsZSBuYW1lZCAncGxheXNvdW5kJ+KAj+KAj+KAjw==?= =?UTF-8?B?4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP?= In-Reply-To: References: Message-ID: On 2023-01-09 14:40, om om wrote: > I'm installing playsound pip install playsound > but it keeps saying No module named playsound > and this error occurs on other packages What are you typing on the command line? Which operating system are you using? Please copy and post what you're typing (with the command prompt) and the complete error message. From list1 at tompassin.net Mon Jan 9 10:30:33 2023 From: list1 at tompassin.net (Thomas Passin) Date: Mon, 9 Jan 2023 10:30:33 -0500 Subject: =?UTF-8?B?UmU6IE5vIG1vZHVsZSBuYW1lZCAncGxheXNvdW5kJ+KAj+KAj+KAjw==?= =?UTF-8?B?4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP?= In-Reply-To: References: Message-ID: <1b0ff5f4-ef1d-865e-17bb-613d1c8cd5f4@tompassin.net> On 1/9/2023 9:40 AM, om om wrote: > I'm installing playsound pip install playsound > but it keeps saying No module named playsound > and this error occurs on other packages Did the installation by pip succeed? if not, what was the error message? "It keeps saying ...". What keeps saying that (it's not a standard error message from pip)? What is the code or command that causes this error message? You would get such a message if you tried to import a package but it had not been installed. If you used pip to install a package and it succeeded but you could not import the package (or module), the most likely reason is that it got installed by a different version of python than the one that used pip to install it. You may have several different versions of Python on your computer. You can check versions by running pip -V Then use the command you normally use to run Python (i.e., python3, py, etc.) and run py_x -m pip -V (don't type "py_x", use the command name you normally use to run python) If the two outputs are different, then you will know for sure that your packages got installed to a different version of Python. Re-install them using "python -m pip install" instead of "pip install" - again, make sure that you use the actual name for the python command that you normally use. From barry at barrys-emacs.org Mon Jan 9 11:56:36 2023 From: barry at barrys-emacs.org (Barry Scott) Date: Mon, 9 Jan 2023 16:56:36 +0000 Subject: file.read Method Documentation (Python 2.7.10) In-Reply-To: References: Message-ID: <05bd9cc9-09e0-51c9-f964-f6b807d19243@barrys-emacs.org> On 09/01/2023 14:34, Stephen Tucker wrote: > Dear Python-list, > > Yes, I know that Python 2.x is no longer supported. > > I have found that the documentation for this method is misleading when the > file being read is UTF-8-encoded: > > Instead of reading *size* bytes, the method reads *size *UTF-8 byte > *sequences*. > > Has this error been corrected in the Python 3.x documentation? Please read the python 3 docs and let us know if you think its correct now. Barry From mats at wichmann.us Mon Jan 9 11:59:09 2023 From: mats at wichmann.us (Mats Wichmann) Date: Mon, 9 Jan 2023 09:59:09 -0700 Subject: =?UTF-8?B?UmU6IE5vIG1vZHVsZSBuYW1lZCAncGxheXNvdW5kJ+KAj+KAj+KAjw==?= =?UTF-8?B?4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP?= In-Reply-To: <1b0ff5f4-ef1d-865e-17bb-613d1c8cd5f4@tompassin.net> References: <1b0ff5f4-ef1d-865e-17bb-613d1c8cd5f4@tompassin.net> Message-ID: <98a2fa2d-25a8-7709-c7b3-f805949e5d6e@wichmann.us> On 1/9/23 08:30, Thomas Passin wrote: > On 1/9/2023 9:40 AM, om om wrote: >> I'm installing playsound pip install playsound >> but it keeps saying No module named playsound >> and this error occurs on other packages > > Did the installation by pip succeed? if not, what was the error message? > > "It keeps saying ...". What keeps saying that (it's not a standard error > message from pip)? What is the code or command that causes this error > message? > > You would get such a message if you tried to import a package but it had > not been installed.? If you used pip to install a package and it > succeeded but you could not import the package (or module), the most > likely reason is that it got installed by a different version of python > than the one that used pip to install it.? You may have several > different versions of Python on your computer. > > You can check versions by running > > pip -V As described in this reply, but to be more explicit: if a package installed wtihout error, and then you can't import it in Python, it's basically always a path problem. Invoking pip the same way as you did to install, do: pip show playsound This will tell you where it put it. If that's not a place your Python is looking, it won't find it. > > Then use the command you normally use to run Python (i.e., python3, py, > etc.) and run > > py_x -m pip -V > > (don't type "py_x", use the command name you normally use to run python) > > If the two outputs are different, then you will know for sure that your > packages got installed to a different version of Python.? Re-install > them using "python -m pip install" instead of "pip install" - again, > make sure that you use the actual name for the python command that you > normally use. From euyhegnrh at gmail.com Mon Jan 9 12:29:21 2023 From: euyhegnrh at gmail.com (om om) Date: Mon, 9 Jan 2023 19:29:21 +0200 Subject: I'm installing playsound but it keeps saying No module named playsound Message-ID: I'm installing playsound and its saying but it keeps saying No module named playsound and this error occurs on other packages when I install it saying Requirement already satisfied: playsound in c:\users\omrio\appdata\local\programs\python\python39\lib\site-packages From rosuav at gmail.com Mon Jan 9 13:25:07 2023 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 10 Jan 2023 05:25:07 +1100 Subject: file.read Method Documentation (Python 2.7.10) In-Reply-To: References: Message-ID: On Tue, 10 Jan 2023 at 01:36, Stephen Tucker wrote: > > Dear Python-list, > > Yes, I know that Python 2.x is no longer supported. > > I have found that the documentation for this method is misleading when the > file being read is UTF-8-encoded: > > Instead of reading *size* bytes, the method reads *size *UTF-8 byte > *sequences*. > > Has this error been corrected in the Python 3.x documentation? > What documentation is this? The builtin 'file' type doesn't know anything about encodings, and only ever returns bytes. ChrisA From list1 at tompassin.net Mon Jan 9 12:11:29 2023 From: list1 at tompassin.net (Thomas Passin) Date: Mon, 9 Jan 2023 12:11:29 -0500 Subject: =?UTF-8?B?UmU6IE5vIG1vZHVsZSBuYW1lZCAncGxheXNvdW5kJ+KAj+KAj+KAjw==?= =?UTF-8?B?4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP?= In-Reply-To: <98a2fa2d-25a8-7709-c7b3-f805949e5d6e@wichmann.us> References: <1b0ff5f4-ef1d-865e-17bb-613d1c8cd5f4@tompassin.net> <98a2fa2d-25a8-7709-c7b3-f805949e5d6e@wichmann.us> Message-ID: <576f5250-cef2-cf14-ef10-aa83ebc3c162@tompassin.net> I wrote >> "the most likely reason is that it got installed by a different >> version of python than the one that used pip to install it." That's not quite what I meant to say. Better: "the most likely reason is that you are trying to import it using a different version of python than the one that used pip to install it." On 1/9/2023 11:59 AM, Mats Wichmann wrote: > On 1/9/23 08:30, Thomas Passin wrote: >> On 1/9/2023 9:40 AM, om om wrote: >>> I'm installing playsound pip install playsound >>> but it keeps saying No module named playsound >>> and this error occurs on other packages >> >> Did the installation by pip succeed? if not, what was the error message? >> >> "It keeps saying ...". What keeps saying that (it's not a standard >> error message from pip)? What is the code or command that causes this >> error message? >> >> You would get such a message if you tried to import a package but it >> had not been installed.? If you used pip to install a package and it >> succeeded but you could not import the package (or module), the most >> likely reason is that it got installed by a different version of >> python than the one that used pip to install it.? You may have several >> different versions of Python on your computer. >> >> You can check versions by running >> >> pip -V > > As described in this reply, but to be more explicit: if a package > installed wtihout error, and then you can't import it in Python, it's > basically always a path problem. > > Invoking pip the same way as you did to install, do: > > pip show playsound > > This will tell you where it put it. If that's not a place your Python is > looking, it won't find it. > >> >> Then use the command you normally use to run Python (i.e., python3, >> py, etc.) and run >> >> py_x -m pip -V >> >> (don't type "py_x", use the command name you normally use to run python) >> >> If the two outputs are different, then you will know for sure that >> your packages got installed to a different version of Python. >> Re-install them using "python -m pip install" instead of "pip install" >> - again, make sure that you use the actual name for the python command >> that you normally use. > > From python at mrabarnett.plus.com Mon Jan 9 14:10:28 2023 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 9 Jan 2023 19:10:28 +0000 Subject: =?UTF-8?B?UmU6IE5vIG1vZHVsZSBuYW1lZCAncGxheXNvdW5kJ+KAj+KAj+KAjw==?= =?UTF-8?B?4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP?= In-Reply-To: <576f5250-cef2-cf14-ef10-aa83ebc3c162@tompassin.net> References: <1b0ff5f4-ef1d-865e-17bb-613d1c8cd5f4@tompassin.net> <98a2fa2d-25a8-7709-c7b3-f805949e5d6e@wichmann.us> <576f5250-cef2-cf14-ef10-aa83ebc3c162@tompassin.net> Message-ID: <5dd80518-fe7d-cd47-0736-509b9838b2ce@mrabarnett.plus.com> On 2023-01-09 17:11, Thomas Passin wrote: > > I wrote > >>> "the most likely reason is that it got installed by a different >>> version of python than the one that used pip to install it." > > That's not quite what I meant to say. Better: > > "the most likely reason is that you are trying to import it using a > different version of python than the one that used pip to install it." > On Windows it's best to use pip via the Python Launcher: py -m pip show playsound > > On 1/9/2023 11:59 AM, Mats Wichmann wrote: >> On 1/9/23 08:30, Thomas Passin wrote: >>> On 1/9/2023 9:40 AM, om om wrote: >>>> I'm installing playsound pip install playsound >>>> but it keeps saying No module named playsound >>>> and this error occurs on other packages >>> >>> Did the installation by pip succeed? if not, what was the error message? >>> >>> "It keeps saying ...". What keeps saying that (it's not a standard >>> error message from pip)? What is the code or command that causes this >>> error message? >>> >>> You would get such a message if you tried to import a package but it >>> had not been installed.? If you used pip to install a package and it >>> succeeded but you could not import the package (or module), the most >>> likely reason is that it got installed by a different version of >>> python than the one that used pip to install it.? You may have several >>> different versions of Python on your computer. >>> >>> You can check versions by running >>> >>> pip -V >> >> As described in this reply, but to be more explicit: if a package >> installed wtihout error, and then you can't import it in Python, it's >> basically always a path problem. >> >> Invoking pip the same way as you did to install, do: >> >> pip show playsound >> >> This will tell you where it put it. If that's not a place your Python is >> looking, it won't find it. >> >>> >>> Then use the command you normally use to run Python (i.e., python3, >>> py, etc.) and run >>> >>> py_x -m pip -V >>> >>> (don't type "py_x", use the command name you normally use to run python) >>> >>> If the two outputs are different, then you will know for sure that >>> your packages got installed to a different version of Python. >>> Re-install them using "python -m pip install" instead of "pip install" >>> - again, make sure that you use the actual name for the python command >>> that you normally use. >> >> > From eryksun at gmail.com Mon Jan 9 15:00:02 2023 From: eryksun at gmail.com (Eryk Sun) Date: Mon, 9 Jan 2023 14:00:02 -0600 Subject: =?UTF-8?B?UmU6IE5vIG1vZHVsZSBuYW1lZCAncGxheXNvdW5kJ+KAj+KAj+KAj+KAj+KAj+KAj+KAjw==?= =?UTF-8?B?4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP?= In-Reply-To: <5dd80518-fe7d-cd47-0736-509b9838b2ce@mrabarnett.plus.com> References: <1b0ff5f4-ef1d-865e-17bb-613d1c8cd5f4@tompassin.net> <98a2fa2d-25a8-7709-c7b3-f805949e5d6e@wichmann.us> <576f5250-cef2-cf14-ef10-aa83ebc3c162@tompassin.net> <5dd80518-fe7d-cd47-0736-509b9838b2ce@mrabarnett.plus.com> Message-ID: On 1/9/23, MRAB wrote: > > On Windows it's best to use pip via the Python Launcher: > > py -m pip show playsound Python's app distribution on the Microsoft Store doesn't include the py launcher, and we don't (but should) have a standalone app or desktop version of the launcher. Unlike the desktop distribution, however, the app distribution installs a versioned name as a link, such as "python3.11". (Note that a venv virtual environment only has a "python" command on Windows.) If "pip" is in PATH, a versioned name such as "pip3.11" should also be available. If multiple versions of the app distribution are installed, and for some reason the "python" and "pip" links are mapped to different versions, one can match up the versioned names "pip3.11" and "python3.11" to ensure consistency. Or simply run "python -m pip". From list1 at tompassin.net Mon Jan 9 16:21:45 2023 From: list1 at tompassin.net (Thomas Passin) Date: Mon, 9 Jan 2023 16:21:45 -0500 Subject: =?UTF-8?B?UmU6IE5vIG1vZHVsZSBuYW1lZCAncGxheXNvdW5kJ+KAj+KAj+KAjw==?= =?UTF-8?B?4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP?= In-Reply-To: <5dd80518-fe7d-cd47-0736-509b9838b2ce@mrabarnett.plus.com> References: <1b0ff5f4-ef1d-865e-17bb-613d1c8cd5f4@tompassin.net> <98a2fa2d-25a8-7709-c7b3-f805949e5d6e@wichmann.us> <576f5250-cef2-cf14-ef10-aa83ebc3c162@tompassin.net> <5dd80518-fe7d-cd47-0736-509b9838b2ce@mrabarnett.plus.com> Message-ID: <99d00f23-2113-6fa3-6486-b50201d09059@tompassin.net> On 1/9/2023 2:10 PM, MRAB wrote: > On Windows it's best to use pip via the Python Launcher: > > py -m pip show playsound Sure - I just didn't want to complicate the post any more, though I did mention it in passing. py is definitely the best way. I wonder how many people know that py can launch other versions of Python too? From list1 at tompassin.net Mon Jan 9 16:26:45 2023 From: list1 at tompassin.net (Thomas Passin) Date: Mon, 9 Jan 2023 16:26:45 -0500 Subject: I'm installing playsound but it keeps saying No module named playsound In-Reply-To: References: Message-ID: On 1/9/2023 12:29 PM, om om wrote: > I'm installing playsound and its saying > but it keeps saying No module named playsound > and this error occurs on other packages > > when I install it saying Requirement already satisfied: playsound in > c:\users\omrio\appdata\local\programs\python\python39\lib\site-packages This doesn't tell us anything new. Please say exactly what you did and saw when you got the error messages. Please tell us the result of running "pip -V" and "python -m pip -V". No one will be able to help if you don't tell us important details. From PythonList at DancesWithMice.info Mon Jan 9 17:33:24 2023 From: PythonList at DancesWithMice.info (dn) Date: Tue, 10 Jan 2023 11:33:24 +1300 Subject: Mailing-Lists (pointer) In-Reply-To: References: Message-ID: <7c170ef9-179e-1029-19ca-24459fddb659@DancesWithMice.info> On 10/01/2023 08.46, Stefan Ram wrote: > If anyone is interested: In "comp.misc", there's a discussion > about the use of mailing lists in software development. > Subject: An objective criteria for deprecating community platforms > (I did not create this subject!) (and I don't read comp.misc) There is an increasingly relevant question though: how do we 'reach' as many people as possible, without diluting the (community) value of responses? At one time, if you wanted to talk/hear certain folk you felt compelled to join Twitter (see also AOL, MySpace, Facebook, ...). Recently many more people have realised that a single, centralised, (and corporately-owned) 'service' has its down-sides. If there are too many channels for communication, it increases the difficulty for any one person to 'keep up', eg python-list and python-forum. On the other hand, by splitting the community, eg python-list and python-tutor, there are benefits of relevance and focus. What some don't seem to appreciate is that whilst a conversation such as "how do I ..." seems to have the objective of (quickly) solving that (one) person's problem, at that one point in time; it also (potentially) forms a 'knowledgebase' for people who have the same problem, at some later time. For those brought-up with, or in the mode of, 'instant messaging', there is no concept of future-value. This is evidenced by the many folk who fail to scan the mailing-list archives before posting a question 'here' (who may not even realise that archives are kept, and for that very purpose). This in-turn, feeds the notion of splitting 'learners' from 'core developers' (for example). How many times have you seen a question about how to use Python for the very first time after installing on Windows? See also the wisdom of enabling comp.lang.python and python-list as 'mirrors', enabling those who prefer one mechanism/client to another, yet maintaining a single 'community'. -- Regards, =dn From list1 at tompassin.net Mon Jan 9 16:23:37 2023 From: list1 at tompassin.net (Thomas Passin) Date: Mon, 9 Jan 2023 16:23:37 -0500 Subject: =?UTF-8?B?UmU6IE5vIG1vZHVsZSBuYW1lZCAncGxheXNvdW5kJ+KAj+KAj+KAjw==?= =?UTF-8?B?4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP4oCP?= In-Reply-To: References: <1b0ff5f4-ef1d-865e-17bb-613d1c8cd5f4@tompassin.net> <98a2fa2d-25a8-7709-c7b3-f805949e5d6e@wichmann.us> <576f5250-cef2-cf14-ef10-aa83ebc3c162@tompassin.net> <5dd80518-fe7d-cd47-0736-509b9838b2ce@mrabarnett.plus.com> Message-ID: On 1/9/2023 3:00 PM, Eryk Sun wrote: > On 1/9/23, MRAB wrote: >> >> On Windows it's best to use pip via the Python Launcher: >> >> py -m pip show playsound > > Python's app distribution on the Microsoft Store doesn't include the > py launcher, and we don't (but should) have a standalone app or > desktop version of the launcher. Unlike the desktop distribution, > however, the app distribution installs a versioned name as a link, > such as "python3.11". (Note that a venv virtual environment only has a > "python" command on Windows.) > > If "pip" is in PATH, a versioned name such as "pip3.11" should also be > available. If multiple versions of the app distribution are installed, > and for some reason the "python" and "pip" links are mapped to > different versions, one can match up the versioned names "pip3.11" and > "python3.11" to ensure consistency. Or simply run "python -m pip". It better be "pythonx.xx -m pip", though! From rosuav at gmail.com Mon Jan 9 17:41:46 2023 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 10 Jan 2023 09:41:46 +1100 Subject: Mailing-Lists (pointer) In-Reply-To: <7c170ef9-179e-1029-19ca-24459fddb659@DancesWithMice.info> References: <7c170ef9-179e-1029-19ca-24459fddb659@DancesWithMice.info> Message-ID: On Tue, 10 Jan 2023 at 09:37, dn wrote: > > On 10/01/2023 08.46, Stefan Ram wrote: > > If anyone is interested: In "comp.misc", there's a discussion > > about the use of mailing lists in software development. > > Subject: An objective criteria for deprecating community platforms > > (I did not create this subject!) > > (and I don't read comp.misc) > > > There is an increasingly relevant question though: how do we 'reach' as > many people as possible, without diluting the (community) value of > responses? > > At one time, if you wanted to talk/hear certain folk you felt compelled > to join Twitter (see also AOL, MySpace, Facebook, ...). Recently many > more people have realised that a single, centralised, (and > corporately-owned) 'service' has its down-sides. > Relevant XKCD: https://xkcd.com/1810/ ChrisA From news at milanglacier.com Tue Jan 10 03:35:11 2023 From: news at milanglacier.com (Milan Glacier) Date: Tue, 10 Jan 2023 03:35:11 -0500 Subject: Mailing-Lists (pointer) In-Reply-To: <7c170ef9-179e-1029-19ca-24459fddb659@DancesWithMice.info> References: <7c170ef9-179e-1029-19ca-24459fddb659@DancesWithMice.info> Message-ID: <20230110083511.4vap5num37wg7fb6@zoho.com> On 01/10/23 11:33, dn wrote: >On 10/01/2023 08.46, Stefan Ram wrote: >> If anyone is interested: In "comp.misc", there's a discussion >> about the use of mailing lists in software development. >> Subject: An objective criteria for deprecating community platforms >> (I did not create this subject!) > >(and I don't read comp.misc) > > >There is an increasingly relevant question though: how do we 'reach' >as many people as possible, without diluting the (community) value of >responses? > >At one time, if you wanted to talk/hear certain folk you felt >compelled to join Twitter (see also AOL, MySpace, Facebook, ...). >Recently many more people have realised that a single, centralised, >(and corporately-owned) 'service' has its down-sides. > >If there are too many channels for communication, it increases the >difficulty for any one person to 'keep up', eg python-list and >python-forum. I remember there was once a hot thread in this python-list discussing about abandoning this mailing list and move all the discussion to the forum. Has anyone known about any status quo about the decision? I personally strongly preferred mailing list. It is open-format, open-archive and easy to download and retrive information using your preferred indexing tools and homemade scripts. From cl at isbd.net Tue Jan 10 03:45:58 2023 From: cl at isbd.net (Chris Green) Date: Tue, 10 Jan 2023 08:45:58 +0000 Subject: Mailing-Lists (pointer) References: <7c170ef9-179e-1029-19ca-24459fddb659@DancesWithMice.info> Message-ID: <67lv8j-anpe1.ln1@esprimo.zbmc.eu> dn wrote: [snip] > See also the wisdom of enabling comp.lang.python and python-list as > 'mirrors', enabling those who prefer one mechanism/client to another, > yet maintaining a single 'community'. > Yes, this is important I think. Plus, if possible, if it's decided to move to a forum format make that accessible by E-Mail. -- Chris Green ? From abskumar798 at gmail.com Tue Jan 10 09:57:57 2023 From: abskumar798 at gmail.com (Abhay Singh) Date: Tue, 10 Jan 2023 06:57:57 -0800 (PST) Subject: Tkinter GUI freezing, used Thread then encountered RuntimeError: threads can only be started once Message-ID: <5938bcdd-da2c-41af-81a7-2274e75de5fdn@googlegroups.com> Here is the entire code snippet of the same. Please help def change_flag(top_frame, bottom_frame, button1, button2, button3, button4, controller): global counter, canvas, my_image, chosen, flag, directory canvas.delete('all') button5['state'] = DISABLED counter += 1 chosen, options_text = function_options() right_answer_flag = get_right_answer_flag(chosen, options_text) #pdb.set_trace() try: location = directory + chosen + format_image except: controller.show_frame(PlayAgainExit) my_image = PhotoImage(file=location) canvas.create_image(160, 100, anchor=CENTER, image=my_image) button1["text"] = options_text[0] button2["text"] = options_text[1] button3["text"] = options_text[2] button4["text"] = options_text[3] button1['state'] = NORMAL button2['state'] = NORMAL button3['state'] = NORMAL button4['state'] = NORMAL ############## button5 = Button( next_frame, width=20, text="next", fg="black", #command=lambda: change_flag(top_frame,bottom_frame,button1,button2,button3,button4,controller)) command=Thread(target=change_flag, args =(top_frame,bottom_frame,button1,button2,button3,button4,controller)).start) button5.pack(side=RIGHT, padx=5, pady=5) Thanks, Abhay From python at mrabarnett.plus.com Tue Jan 10 13:32:32 2023 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 10 Jan 2023 18:32:32 +0000 Subject: Tkinter GUI freezing, used Thread then encountered RuntimeError: threads can only be started once In-Reply-To: <5938bcdd-da2c-41af-81a7-2274e75de5fdn@googlegroups.com> References: <5938bcdd-da2c-41af-81a7-2274e75de5fdn@googlegroups.com> Message-ID: On 2023-01-10 14:57, Abhay Singh wrote: > Here is the entire code snippet of the same. > > Please help > > def change_flag(top_frame, bottom_frame, button1, button2, button3, button4, controller): global counter, canvas, my_image, chosen, flag, directory canvas.delete('all') button5['state'] = DISABLED counter += 1 > > chosen, options_text = function_options() > right_answer_flag = get_right_answer_flag(chosen, options_text) > #pdb.set_trace() > > try: > location = directory + chosen + format_image > except: > controller.show_frame(PlayAgainExit) > > my_image = PhotoImage(file=location) > canvas.create_image(160, 100, anchor=CENTER, image=my_image) > > button1["text"] = options_text[0] > button2["text"] = options_text[1] > button3["text"] = options_text[2] > button4["text"] = options_text[3] > > button1['state'] = NORMAL > button2['state'] = NORMAL > button3['state'] = NORMAL > button4['state'] = NORMAL > ############## > > button5 = Button( > next_frame, > width=20, > text="next", > fg="black", > #command=lambda: change_flag(top_frame,bottom_frame,button1,button2,button3,button4,controller)) > command=Thread(target=change_flag, args =(top_frame,bottom_frame,button1,button2,button3,button4,controller)).start) > > button5.pack(side=RIGHT, padx=5, pady=5) > The formatting is messed up, which doesn't help. Some points: You have a 'bare' except, i.e. "except:". Don't do that. It swallows _all_ exceptions and can hide bugs. I don't like how you're passing Thread...start as an argument. IMHO, it would be better/cleaner to pass a plain function, even if the only thing that function does is to start the thread. I can't tell what 'change_flag' is doing because of the formatting issue. Is it doing GUI stuff? In a thread? If yes, don't do that. The GUI doesn't like that. Only the main thread should do GUI stuff. From jenkris at tutanota.com Tue Jan 10 15:03:03 2023 From: jenkris at tutanota.com (Jen Kris) Date: Tue, 10 Jan 2023 21:03:03 +0100 (CET) Subject: To clarify how Python handles two equal objects Message-ID: I am writing a spot speedup in assembly language for a short but computation-intensive Python loop, and I discovered something about Python array handling that I would like to clarify.? For a simplified example, I created a matrix mx1 and assigned the array arr1 to the third row of the matrix: mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ] arr1 = mx1[2] The pointers to these are now the same: ida = id(mx1[2]) - 140260325306880 idb = id(arr1) - 140260325306880 That?s great because when I encounter this in assembly or C, I can just borrow the pointer to row 3 for the array arr1, on the assumption that they will continue to point to the same object.? Then when I do any math operations in arr1 it will be reflected in both arrays because they are now pointing to the same array: arr1[0] += 2 print(mx1[2]) - [9, 8, 9] print(arr1) - [9, 8, 9] Now mx1 looks like this: [ 1, 2, 3 ] [ 4, 5, 6 ] [ 9, 8, 9 ] and it stays that way for remaining iterations.?? But on the next iteration we assign arr1 to something else: arr1 = [ 10, 11, 12 ] idc = id(arr1) ? 140260325308160 idd = id(mx1[2]) ? 140260325306880 Now arr1 is no longer equal to mx1[2], and any subsequent operations in arr1 will not affect mx1.? So where I?m rewriting some Python code in a low level language, I can?t assume that the two objects are equal because that equality will not remain if either is reassigned.? So if I do some operation on one array I have to conform the two arrays for as long as they remain equal, I can?t just do it in one operation because I can?t rely on the objects remaining equal.? Is my understanding of this correct?? Is there anything I?m missing?? Thanks very much.? Jen From rosuav at gmail.com Tue Jan 10 15:28:49 2023 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 11 Jan 2023 07:28:49 +1100 Subject: To clarify how Python handles two equal objects In-Reply-To: References: Message-ID: On Wed, 11 Jan 2023 at 07:14, Jen Kris via Python-list wrote: > > I am writing a spot speedup in assembly language for a short but computation-intensive Python loop, and I discovered something about Python array handling that I would like to clarify. > > For a simplified example, I created a matrix mx1 and assigned the array arr1 to the third row of the matrix: > > mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ] > arr1 = mx1[2] > > The pointers to these are now the same: > > ida = id(mx1[2]) - 140260325306880 > idb = id(arr1) - 140260325306880 > > That?s great because when I encounter this in assembly or C, I can just borrow the pointer to row 3 for the array arr1, on the assumption that they will continue to point to the same object. Then when I do any math operations in arr1 it will be reflected in both arrays because they are now pointing to the same array: > That's not an optimization; what you've done is set arr1 to be a reference to that object. > But on the next iteration we assign arr1 to something else: > > arr1 = [ 10, 11, 12 ] > idc = id(arr1) ? 140260325308160 > idd = id(mx1[2]) ? 140260325306880 > > Now arr1 is no longer equal to mx1[2], and any subsequent operations in arr1 will not affect mx1. Yep, you have just set arr1 to be a completely different object. > So where I?m rewriting some Python code in a low level language, I can?t assume that the two objects are equal because that equality will not remain if either is reassigned. So if I do some operation on one array I have to conform the two arrays for as long as they remain equal, I can?t just do it in one operation because I can?t rely on the objects remaining equal. > > Is my understanding of this correct? Is there anything I?m missing? > Assignment in Python is a matter of object references. It's not "conform them as long as they remain equal". You'll have to think in terms of object references the entire way. ChrisA From jenkris at tutanota.com Tue Jan 10 15:41:06 2023 From: jenkris at tutanota.com (Jen Kris) Date: Tue, 10 Jan 2023 21:41:06 +0100 (CET) Subject: To clarify how Python handles two equal objects In-Reply-To: References: Message-ID: Thanks for your comments.? I'd like to make one small point.? You say: "Assignment in Python is a matter of object references. It's not "conform them as long as they remain equal". You'll have to think in terms of object references the entire way." But where they have been set to the same object, an operation on one will affect the other as long as they are equal (in Python).? So I will have to conform them in those cases because Python will reflect any math operation in both the array and the matrix.? Jan 10, 2023, 12:28 by rosuav at gmail.com: > On Wed, 11 Jan 2023 at 07:14, Jen Kris via Python-list > wrote: > >> >> I am writing a spot speedup in assembly language for a short but computation-intensive Python loop, and I discovered something about Python array handling that I would like to clarify. >> >> For a simplified example, I created a matrix mx1 and assigned the array arr1 to the third row of the matrix: >> >> mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ] >> arr1 = mx1[2] >> >> The pointers to these are now the same: >> >> ida = id(mx1[2]) - 140260325306880 >> idb = id(arr1) - 140260325306880 >> >> That?s great because when I encounter this in assembly or C, I can just borrow the pointer to row 3 for the array arr1, on the assumption that they will continue to point to the same object. Then when I do any math operations in arr1 it will be reflected in both arrays because they are now pointing to the same array: >> > > That's not an optimization; what you've done is set arr1 to be a > reference to that object. > >> But on the next iteration we assign arr1 to something else: >> >> arr1 = [ 10, 11, 12 ] >> idc = id(arr1) ? 140260325308160 >> idd = id(mx1[2]) ? 140260325306880 >> >> Now arr1 is no longer equal to mx1[2], and any subsequent operations in arr1 will not affect mx1. >> > > Yep, you have just set arr1 to be a completely different object. > >> So where I?m rewriting some Python code in a low level language, I can?t assume that the two objects are equal because that equality will not remain if either is reassigned. So if I do some operation on one array I have to conform the two arrays for as long as they remain equal, I can?t just do it in one operation because I can?t rely on the objects remaining equal. >> >> Is my understanding of this correct? Is there anything I?m missing? >> > > Assignment in Python is a matter of object references. It's not > "conform them as long as they remain equal". You'll have to think in > terms of object references the entire way. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > From rosuav at gmail.com Tue Jan 10 15:45:18 2023 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 11 Jan 2023 07:45:18 +1100 Subject: To clarify how Python handles two equal objects In-Reply-To: References: Message-ID: On Wed, 11 Jan 2023 at 07:41, Jen Kris wrote: > > > Thanks for your comments. I'd like to make one small point. You say: > > "Assignment in Python is a matter of object references. It's not > "conform them as long as they remain equal". You'll have to think in > terms of object references the entire way." > > But where they have been set to the same object, an operation on one will affect the other as long as they are equal (in Python). So I will have to conform them in those cases because Python will reflect any math operation in both the array and the matrix. > It's not that "an operation on one will affect the other" - it's that, no matter how you refer to that object, you're affecting *that one single object*. It's like when you're washing a window; the inside and outside of the window are the exact same window, so regardless of where you're looking at it from, it's the same single window and changes affect it equally. So you shouldn't have to replicate any changes. What should be happening is that you find the right object to mutate, and mutate that. For example: stuff = [[1, 2, 3], [4, 5, 6]] stuff.append(stuff[0]) print(stuff) You now have two references to the same list, inside another list. Any change to stuff[0] is a change to stuff[2], because they're the exact same list. When you append "a reference to this list over here" (which you found by asking for stuff[0]) to the outer list, you get that list. That's Python's object model, and trying to cheat it by copying changes is just going to cause untold nightmares of desynchronization. ChrisA From thomas at python.org Tue Jan 10 15:24:48 2023 From: thomas at python.org (Thomas Wouters) Date: Tue, 10 Jan 2023 21:24:48 +0100 Subject: Python 3.12.0 alpha 4 released Message-ID: I'm pleased to announce the release of Python 3.12 alpha 4. https://www.python.org/downloads/release/python-3120a4/ *This is an early developer preview of Python 3.12*. Major new features of the 3.12 series, compared to 3.11 Python 3.12 is still in development. This release, 3.12.0a4 is the fourth of seven planned alpha releases. Alpha releases are intended to make it easier to test the current state of new features and bug fixes and to test the release process. During the alpha phase, features may be added up until the start of the beta phase (2023-05-08) and, if necessary, may be modified or deleted up until the release candidate phase (2023-07-31). Please keep in mind that this is a preview release and its use is *not *recommended for production environments. Many new features for Python 3.12 are still being planned and written. Among the new major new features and changes so far: - Even more improved error messages. More exceptions potentially caused by typos now make suggestions to the user. - Support for the Linux perf profiler to report Python function names in traces. - The deprecated wstr and wstr_length members of the C implementation of unicode objects were removed, per PEP 623 . - In the unittest module, a number of long deprecated methods and classes were removed. (They had been deprecated since Python 3.1 or 3.2). - The deprecated smtpd and distutils modules have been removed (see PEP 594 and PEP 632 ). The setuptools package (installed by default in virtualenvs and many other places) continues to provide the distutils module. - A number of other old, broken and deprecated functions, classes and methods have been removed. - (Hey,* fellow core developer*, if a feature you find important is missing from this list, let Thomas know .) For more details on the changes to Python 3.12, see What's new in Python 3.12 . The next pre-release of Python 3.12 will be 3.12.0a5, currently scheduled for 2023-02-06. More resources Online Documentation . PEP 693 , the Python 3.12 Release Schedule. Report bugs via GitHub Issues . Help fund Python and its community . And now for something completely different Two haikus apt, as Python's development springs ever forward. I write, erase, rewrite > Erase again, and then > A poppy blooms. Haiku by Katsushika Hokusai. O snail > Climb Mount Fuji, > But slowly, slowly! Haiku by Kobayashi Issa. Enjoy the new releases Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. Regards from chilly Amsterdam, Your release team, Thomas Wouters Ned Deily Steve Dower -- Thomas Wouters From dieter at handshake.de Tue Jan 10 13:11:27 2023 From: dieter at handshake.de (Dieter Maurer) Date: Tue, 10 Jan 2023 19:11:27 +0100 Subject: Mailing-Lists (pointer) In-Reply-To: <67lv8j-anpe1.ln1@esprimo.zbmc.eu> References: <7c170ef9-179e-1029-19ca-24459fddb659@DancesWithMice.info> <67lv8j-anpe1.ln1@esprimo.zbmc.eu> Message-ID: <25533.43599.426707.882744@ixdm.fritz.box> Chris Green wrote at 2023-1-10 08:45 +0000: > ... >Yes, this is important I think. Plus, if possible, if it's decided to >move to a forum format make that accessible by E-Mail. I much prefer a mailing list over an http based service. With mailing lists, all interesting messages arrive in my email reader, i.e. at a central place; with http based services, I have to visit the various sites to learn whether there is relevant new information. From gweatherby at uchc.edu Tue Jan 10 16:03:23 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Tue, 10 Jan 2023 21:03:23 +0000 Subject: To clarify how Python handles two equal objects In-Reply-To: References: Message-ID: For clarification, equality is not identity in Python. e.g. x = 7 y = 7.0 print(x == y) print(x is y) Will return True False Full explanation at https://docs.python.org/3/reference/expressions.html#comparisons From: Python-list on behalf of Chris Angelico Date: Tuesday, January 10, 2023 at 3:47 PM To: Python List Subject: Re: To clarify how Python handles two equal objects *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** On Wed, 11 Jan 2023 at 07:41, Jen Kris wrote: > > > Thanks for your comments. I'd like to make one small point. You say: > > "Assignment in Python is a matter of object references. It's not > "conform them as long as they remain equal". You'll have to think in > terms of object references the entire way." > > But where they have been set to the same object, an operation on one will affect the other as long as they are equal (in Python). So I will have to conform them in those cases because Python will reflect any math operation in both the array and the matrix. > It's not that "an operation on one will affect the other" - it's that, no matter how you refer to that object, you're affecting *that one single object*. It's like when you're washing a window; the inside and outside of the window are the exact same window, so regardless of where you're looking at it from, it's the same single window and changes affect it equally. So you shouldn't have to replicate any changes. What should be happening is that you find the right object to mutate, and mutate that. For example: stuff = [[1, 2, 3], [4, 5, 6]] stuff.append(stuff[0]) print(stuff) You now have two references to the same list, inside another list. Any change to stuff[0] is a change to stuff[2], because they're the exact same list. When you append "a reference to this list over here" (which you found by asking for stuff[0]) to the outer list, you get that list. That's Python's object model, and trying to cheat it by copying changes is just going to cause untold nightmares of desynchronization. ChrisA -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!iORRBJFui8Kn7WzY0ZRPfSdGKcmnDV81UffITsv7ExEAbBXEtv86qC3BOvGaDXCAY708Q4QbDXh0_wo7$ From roel at roelschroeven.net Tue Jan 10 16:07:08 2023 From: roel at roelschroeven.net (Roel Schroeven) Date: Tue, 10 Jan 2023 22:07:08 +0100 Subject: To clarify how Python handles two equal objects In-Reply-To: References: Message-ID: <275b81e4-12f1-8d0a-2c47-bb89eb981ddb@roelschroeven.net> Jen Kris via Python-list schreef op 10/01/2023 om 21:41: > But where they have been set to the same object, an operation on one will affect the other as long as they are equal (in Python). As long as they are *identical*, not equal. Identical as in having the same identity as Python defines it. I advise you to read Ned Batchelder's explanation about names and values in Python, or watch his presentation, to get a good understanding. See https://nedbatchelder.com/text/names1.html -- "Don't Panic." -- Douglas Adams, The Hitchhiker's Guide to the Galaxy From python at mrabarnett.plus.com Tue Jan 10 16:10:44 2023 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 10 Jan 2023 21:10:44 +0000 Subject: To clarify how Python handles two equal objects In-Reply-To: References: Message-ID: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> On 2023-01-10 20:41, Jen Kris via Python-list wrote: > > Thanks for your comments.? I'd like to make one small point.? You say: > > "Assignment in Python is a matter of object references. It's not > "conform them as long as they remain equal". You'll have to think in > terms of object references the entire way." > > But where they have been set to the same object, an operation on one will affect the other as long as they are equal (in Python).? So I will have to conform them in those cases because Python will reflect any math operation in both the array and the matrix. > It's not a 2D matrix, it's a 1D list containing references to 1D lists, each of which contains references to Python ints. In CPython, references happen to be pointers, but that's just an implementation detail. > > > Jan 10, 2023, 12:28 by rosuav at gmail.com: > >> On Wed, 11 Jan 2023 at 07:14, Jen Kris via Python-list >> wrote: >> >>> >>> I am writing a spot speedup in assembly language for a short but computation-intensive Python loop, and I discovered something about Python array handling that I would like to clarify. >>> >>> For a simplified example, I created a matrix mx1 and assigned the array arr1 to the third row of the matrix: >>> >>> mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ] >>> arr1 = mx1[2] >>> >>> The pointers to these are now the same: >>> >>> ida = id(mx1[2]) - 140260325306880 >>> idb = id(arr1) - 140260325306880 >>> >>> That?s great because when I encounter this in assembly or C, I can just borrow the pointer to row 3 for the array arr1, on the assumption that they will continue to point to the same object. Then when I do any math operations in arr1 it will be reflected in both arrays because they are now pointing to the same array: >>> >> >> That's not an optimization; what you've done is set arr1 to be a >> reference to that object. >> >>> But on the next iteration we assign arr1 to something else: >>> >>> arr1 = [ 10, 11, 12 ] >>> idc = id(arr1) ? 140260325308160 >>> idd = id(mx1[2]) ? 140260325306880 >>> >>> Now arr1 is no longer equal to mx1[2], and any subsequent operations in arr1 will not affect mx1. >>> >> >> Yep, you have just set arr1 to be a completely different object. >> >>> So where I?m rewriting some Python code in a low level language, I can?t assume that the two objects are equal because that equality will not remain if either is reassigned. So if I do some operation on one array I have to conform the two arrays for as long as they remain equal, I can?t just do it in one operation because I can?t rely on the objects remaining equal. >>> >>> Is my understanding of this correct? Is there anything I?m missing? >>> >> >> Assignment in Python is a matter of object references. It's not >> "conform them as long as they remain equal". You'll have to think in >> terms of object references the entire way. >> >> ChrisA >> -- >> https://mail.python.org/mailman/listinfo/python-list >> > From cs at cskk.id.au Tue Jan 10 16:37:19 2023 From: cs at cskk.id.au (Cameron Simpson) Date: Wed, 11 Jan 2023 08:37:19 +1100 Subject: Mailing-Lists (pointer) In-Reply-To: <67lv8j-anpe1.ln1@esprimo.zbmc.eu> References: <67lv8j-anpe1.ln1@esprimo.zbmc.eu> Message-ID: On 10Jan2023 08:45, Chris Green wrote: >dn wrote: >> See also the wisdom of enabling comp.lang.python and python-list as >> 'mirrors', enabling those who prefer one mechanism/client to another, >> yet maintaining a single 'community'. >> >Yes, this is important I think. Plus, if possible, if it's decided to >move to a forum format make that accessible by E-Mail. There's a Discourse forum over at discuss.python.org. I use it in "mailing list mode" and do almost all my interactions via email, exactly as I do for python-list. Posts come to me and land in the same local mail folder I use for python-list. My replies land on the forum as expected (and of course also go by email to those members who have turned that mode on). So I'm using both the new forum and the currently mailing list still, and broadly in exactly the same way. Cheers, Cameron Simpson From list1 at tompassin.net Tue Jan 10 16:59:59 2023 From: list1 at tompassin.net (Thomas Passin) Date: Tue, 10 Jan 2023 16:59:59 -0500 Subject: To clarify how Python handles two equal objects In-Reply-To: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> Message-ID: <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> Just to add a possibly picky detail to what others have said, Python does not have an "array" type. It has a "list" type, as well as some other, not necessarily mutable, sequence types. If you want to speed up list and matrix operations, you might use NumPy. Its arrays and matrices are heavily optimized for fast processing and provide many useful operations on them. No use calling out to C code yourself when NumPy has been refining that for many years. On 1/10/2023 4:10 PM, MRAB wrote: > On 2023-01-10 20:41, Jen Kris via Python-list wrote: >> >> Thanks for your comments.? I'd like to make one small point.? You say: >> >> "Assignment in Python is a matter of object references. It's not >> "conform them as long as they remain equal". You'll have to think in >> terms of object references the entire way." >> >> But where they have been set to the same object, an operation on one >> will affect the other as long as they are equal (in Python).? So I >> will have to conform them in those cases because Python will reflect >> any math operation in both the array and the matrix. >> > It's not a 2D matrix, it's a 1D list containing references to 1D lists, > each of which contains references to Python ints. > > In CPython, references happen to be pointers, but that's just an > implementation detail. > >> >> >> Jan 10, 2023, 12:28 by rosuav at gmail.com: >> >>> On Wed, 11 Jan 2023 at 07:14, Jen Kris via Python-list >>> wrote: >>> >>>> >>>> I am writing a spot speedup in assembly language for a short but >>>> computation-intensive Python loop, and I discovered something about >>>> Python array handling that I would like to clarify. >>>> >>>> For a simplified example, I created a matrix mx1 and assigned the >>>> array arr1 to the third row of the matrix: >>>> >>>> mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ] >>>> arr1 = mx1[2] >>>> >>>> The pointers to these are now the same: >>>> >>>> ida = id(mx1[2]) - 140260325306880 >>>> idb = id(arr1) - 140260325306880 >>>> >>>> That?s great because when I encounter this in assembly or C, I can >>>> just borrow the pointer to row 3 for the array arr1, on the >>>> assumption that they will continue to point to the same object. >>>> Then when I do any math operations in arr1 it will be reflected in >>>> both arrays because they are now pointing to the same array: >>>> >>> >>> That's not an optimization; what you've done is set arr1 to be a >>> reference to that object. >>> >>>> But on the next iteration we assign arr1 to something else: >>>> >>>> arr1 = [ 10, 11, 12 ] >>>> idc = id(arr1) ? 140260325308160 >>>> idd = id(mx1[2]) ? 140260325306880 >>>> >>>> Now arr1 is no longer equal to mx1[2], and any subsequent operations >>>> in arr1 will not affect mx1. >>>> >>> >>> Yep, you have just set arr1 to be a completely different object. >>> >>>> So where I?m rewriting some Python code in a low level language, I >>>> can?t assume that the two objects are equal because that equality >>>> will not remain if either is reassigned.? So if I do some operation >>>> on one array I have to conform the two arrays for as long as they >>>> remain equal, I can?t just do it in one operation because I can?t >>>> rely on the objects remaining equal. >>>> >>>> Is my understanding of this correct?? Is there anything I?m missing? >>>> >>> >>> Assignment in Python is a matter of object references. It's not >>> "conform them as long as they remain equal". You'll have to think in >>> terms of object references the entire way. >>> >>> ChrisA >>> -- >>> https://mail.python.org/mailman/listinfo/python-list >>> >> > From rosuav at gmail.com Tue Jan 10 17:11:04 2023 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 11 Jan 2023 09:11:04 +1100 Subject: To clarify how Python handles two equal objects In-Reply-To: <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> Message-ID: On Wed, 11 Jan 2023 at 09:08, Thomas Passin wrote: > > Just to add a possibly picky detail to what others have said, Python > does not have an "array" type. It has a "list" type, as well as some > other, not necessarily mutable, sequence types. Just to be even pickier, Python DOES have an array type, but it's not the one the OP was using :) https://docs.python.org/3/library/array.html ChrisA From jenkris at tutanota.com Tue Jan 10 17:21:34 2023 From: jenkris at tutanota.com (Jen Kris) Date: Tue, 10 Jan 2023 23:21:34 +0100 (CET) Subject: To clarify how Python handles two equal objects In-Reply-To: <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> Message-ID: There are cases where NumPy would be the best choice, but that wasn?t the case here with what the loop was doing.? To sum up what I learned from this post, where one object derives from another object (a = b[0], for example), any operation that would alter one will alter the other.? When either is assigned to something else, then they no longer point to the same memory location and they?re once again independent.?? I hope the word "derives" sidesteps the semantic issue of whether they are "equal." ?? Thanks to all who replied to this post.? Jen Jan 10, 2023, 13:59 by list1 at tompassin.net: > Just to add a possibly picky detail to what others have said, Python does not have an "array" type. It has a "list" type, as well as some other, not necessarily mutable, sequence types. > > If you want to speed up list and matrix operations, you might use NumPy. Its arrays and matrices are heavily optimized for fast processing and provide many useful operations on them. No use calling out to C code yourself when NumPy has been refining that for many years. > > On 1/10/2023 4:10 PM, MRAB wrote: > >> On 2023-01-10 20:41, Jen Kris via Python-list wrote: >> >>> >>> Thanks for your comments.? I'd like to make one small point.? You say: >>> >>> "Assignment in Python is a matter of object references. It's not >>> "conform them as long as they remain equal". You'll have to think in >>> terms of object references the entire way." >>> >>> But where they have been set to the same object, an operation on one will affect the other as long as they are equal (in Python).? So I will have to conform them in those cases because Python will reflect any math operation in both the array and the matrix. >>> >> It's not a 2D matrix, it's a 1D list containing references to 1D lists, each of which contains references to Python ints. >> >> In CPython, references happen to be pointers, but that's just an implementation detail. >> >>> >>> >>> Jan 10, 2023, 12:28 by rosuav at gmail.com: >>> >>>> On Wed, 11 Jan 2023 at 07:14, Jen Kris via Python-list >>>> wrote: >>>> >>>>> >>>>> I am writing a spot speedup in assembly language for a short but computation-intensive Python loop, and I discovered something about Python array handling that I would like to clarify. >>>>> >>>>> For a simplified example, I created a matrix mx1 and assigned the array arr1 to the third row of the matrix: >>>>> >>>>> mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ] >>>>> arr1 = mx1[2] >>>>> >>>>> The pointers to these are now the same: >>>>> >>>>> ida = id(mx1[2]) - 140260325306880 >>>>> idb = id(arr1) - 140260325306880 >>>>> >>>>> That?s great because when I encounter this in assembly or C, I can just borrow the pointer to row 3 for the array arr1, on the assumption that they will continue to point to the same object. Then when I do any math operations in arr1 it will be reflected in both arrays because they are now pointing to the same array: >>>>> >>>> >>>> That's not an optimization; what you've done is set arr1 to be a >>>> reference to that object. >>>> >>>>> But on the next iteration we assign arr1 to something else: >>>>> >>>>> arr1 = [ 10, 11, 12 ] >>>>> idc = id(arr1) ? 140260325308160 >>>>> idd = id(mx1[2]) ? 140260325306880 >>>>> >>>>> Now arr1 is no longer equal to mx1[2], and any subsequent operations in arr1 will not affect mx1. >>>>> >>>> >>>> Yep, you have just set arr1 to be a completely different object. >>>> >>>>> So where I?m rewriting some Python code in a low level language, I can?t assume that the two objects are equal because that equality will not remain if either is reassigned.? So if I do some operation on one array I have to conform the two arrays for as long as they remain equal, I can?t just do it in one operation because I can?t rely on the objects remaining equal. >>>>> >>>>> Is my understanding of this correct?? Is there anything I?m missing? >>>>> >>>> >>>> Assignment in Python is a matter of object references. It's not >>>> "conform them as long as they remain equal". You'll have to think in >>>> terms of object references the entire way. >>>> >>>> ChrisA >>>> -- >>>> https://mail.python.org/mailman/listinfo/python-list >>>> > > -- > https://mail.python.org/mailman/listinfo/python-list > From list1 at tompassin.net Tue Jan 10 18:15:08 2023 From: list1 at tompassin.net (Thomas Passin) Date: Tue, 10 Jan 2023 18:15:08 -0500 Subject: To clarify how Python handles two equal objects In-Reply-To: References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> Message-ID: <720cea38-85e0-404e-0c31-706975d08544@tompassin.net> On 1/10/2023 5:11 PM, Chris Angelico wrote: > On Wed, 11 Jan 2023 at 09:08, Thomas Passin wrote: >> >> Just to add a possibly picky detail to what others have said, Python >> does not have an "array" type. It has a "list" type, as well as some >> other, not necessarily mutable, sequence types. > > Just to be even pickier, Python DOES have an array type, but it's not > the one the OP was using :) > > https://docs.python.org/3/library/array.html > Ha! And here all this time I thought you got them from an add-on package. From ethan at stoneleaf.us Tue Jan 10 18:29:23 2023 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 10 Jan 2023 15:29:23 -0800 Subject: To clarify how Python handles two equal objects In-Reply-To: References: Message-ID: <8f4400ff-36ee-567d-7e70-433e7a4fe250@stoneleaf.us> On 1/10/23 12:03, Jen Kris via Python-list wrote: > I am writing a spot speedup in assembly language for a short but computation-intensive Python > loop, and I discovered something about Python array handling that I would like to clarify. > But on the next iteration we assign arr1 to something else: > > arr1 = [ 10, 11, 12 ] > idc = id(arr1) ? 140260325308160 > idd = id(mx1[2]) ? 140260325306880 > > Now arr1 is no longer equal to mx1[2]... If you want to have `arr1` to still be `mx1[2]` (and consequently for `mx1[2]` to now be `[10, 11, 12]` you need to mutate `arr1` instead of reassigning it: arr1[:] = [10, 11, 12] -- ~Ethan~ From list1 at tompassin.net Tue Jan 10 18:28:35 2023 From: list1 at tompassin.net (Thomas Passin) Date: Tue, 10 Jan 2023 18:28:35 -0500 Subject: To clarify how Python handles two equal objects In-Reply-To: References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> Message-ID: <766042fd-3808-7e11-f079-371f3fd4572e@tompassin.net> On 1/10/2023 5:21 PM, Jen Kris wrote: > There are cases where NumPy would be the best choice, but that wasn?t > the case here with what the loop was doing. > > To sum up what I learned from this post, where one object derives from > another object (a = b[0], for example), any operation that would alter > one will alter the other. Let's make sure we're clear here. The way you were doing it, it *looks like* "one alters the other". But in reality, both are the same thing, and so when that thing gets altered in some way, both variables will show the change because they are in fact references to the same object. As an analogy, if you dye your hair purple and look in a mirror, you will see your image with the new purple hair. If you look in a different mirror, you will also see your image with the new purple hair. They are both reflections of the same object, namely you with your new purple hair. The point about the identity of objects contained within other objects is echoed by the copy and deepcopy operations. copy() copies the references, deepcopy() makes new objects that are equal to the original ones. After making a deep copy of a list and assigning it to new variable, changes in one will no longer show up in the other because the elements are are no longer the same elements. > When either is assigned to something else, > then they no longer point to the same memory location and they?re once > again independent. This is right except that in Python, it's better not to think about their memory locations, because that would basically be an implementation detail (well, except that if you are going to access them with C you will possibly need actual locations). Their logical identity would be better to think about. > I hope the word "derives" sidesteps the semantic > issue of whether they are "equal." > > Thanks to all who replied to this post. > > Jen > > > Jan 10, 2023, 13:59 by list1 at tompassin.net: > > Just to add a possibly picky detail to what others have said, Python > does not have an "array" type. It has a "list" type, as well as some > other, not necessarily mutable, sequence types. > > If you want to speed up list and matrix operations, you might use > NumPy. Its arrays and matrices are heavily optimized for fast > processing and provide many useful operations on them. No use > calling out to C code yourself when NumPy has been refining that for > many years. > > On 1/10/2023 4:10 PM, MRAB wrote: > > On 2023-01-10 20:41, Jen Kris via Python-list wrote: > > > Thanks for your comments.? I'd like to make one small > point.? You say: > > "Assignment in Python is a matter of object references. It's not > "conform them as long as they remain equal". You'll have to > think in > terms of object references the entire way." > > But where they have been set to the same object, an > operation on one will affect the other as long as they are > equal (in Python).? So I will have to conform them in those > cases because Python will reflect any math operation in both > the array and the matrix. > > It's not a 2D matrix, it's a 1D list containing references to 1D > lists, each of which contains references to Python ints. > > In CPython, references happen to be pointers, but that's just an > implementation detail. > > > > Jan 10, 2023, 12:28 by rosuav at gmail.com: > > On Wed, 11 Jan 2023 at 07:14, Jen Kris via Python-list > wrote: > > > I am writing a spot speedup in assembly language for > a short but computation-intensive Python loop, and I > discovered something about Python array handling > that I would like to clarify. > > For a simplified example, I created a matrix mx1 and > assigned the array arr1 to the third row of the matrix: > > mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ] > arr1 = mx1[2] > > The pointers to these are now the same: > > ida = id(mx1[2]) - 140260325306880 > idb = id(arr1) - 140260325306880 > > That?s great because when I encounter this in > assembly or C, I can just borrow the pointer to row > 3 for the array arr1, on the assumption that they > will continue to point to the same object. Then when > I do any math operations in arr1 it will be > reflected in both arrays because they are now > pointing to the same array: > > > That's not an optimization; what you've done is set arr1 > to be a > reference to that object. > > But on the next iteration we assign arr1 to > something else: > > arr1 = [ 10, 11, 12 ] > idc = id(arr1) ? 140260325308160 > idd = id(mx1[2]) ? 140260325306880 > > Now arr1 is no longer equal to mx1[2], and any > subsequent operations in arr1 will not affect mx1. > > > Yep, you have just set arr1 to be a completely different > object. > > So where I?m rewriting some Python code in a low > level language, I can?t assume that the two objects > are equal because that equality will not remain if > either is reassigned.? So if I do some operation on > one array I have to conform the two arrays for as > long as they remain equal, I can?t just do it in one > operation because I can?t rely on the objects > remaining equal. > > Is my understanding of this correct?? Is there > anything I?m missing? > > > Assignment in Python is a matter of object references. > It's not > "conform them as long as they remain equal". You'll have > to think in > terms of object references the entire way. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > > > -- > https://mail.python.org/mailman/listinfo/python-list > > From python at mrabarnett.plus.com Tue Jan 10 18:36:37 2023 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 10 Jan 2023 23:36:37 +0000 Subject: To clarify how Python handles two equal objects In-Reply-To: References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> Message-ID: On 2023-01-10 22:21, Jen Kris via Python-list wrote: > There are cases where NumPy would be the best choice, but that wasn?t the case here with what the loop was doing. > > To sum up what I learned from this post, where one object derives from another object (a = b[0], for example), any operation that would alter one will alter the other.? When either is assigned to something else, then they no longer point to the same memory location and they?re once again independent.?? I hope the word "derives" sidesteps the semantic issue of whether they are "equal." > [snip] In C terms (and in CPython), a 'list' is a resizable array of pointers to objects, so after "a=b[0]", the name "a" will point to the same object that b[0] points to. That object might or might not be mutable. From cs at cskk.id.au Tue Jan 10 19:13:42 2023 From: cs at cskk.id.au (Cameron Simpson) Date: Wed, 11 Jan 2023 11:13:42 +1100 Subject: Tkinter GUI freezing, used Thread then encountered RuntimeError: threads can only be started once In-Reply-To: References: Message-ID: On 10Jan2023 18:32, MRAB wrote: >I don't like how you're passing Thread...start as an argument. IMHO, it >would be better/cleaner to pass a plain function, even if the only >thing that function does is to start the thread. Yes, and this is likely the thing causing the cited exception "threads can only be started once". Your setup of the button with the action defined as: Thread(....).start creates a _single_ new Thread _when you define the button_, and makes hte button callback try to start it. On the second and following callback, you're trying to start the _same_ single Thread again. Do as MRAB suggests and have the callback create-and-start a Thread instead of just starting an _existing_ Thread. Also, for simple quick things there's no need to use a Thread at all. If the final version of the programme is going to do something long running at that point, then sure. >I can't tell what 'change_flag' is doing because of the formatting >issue. Is it doing GUI stuff? In a thread? If yes, don't do that. The >GUI doesn't like that. Only the main thread should do GUI stuff. Aye. This is very important in almost all GUI toolkits. Bit me very badly with Qt once, badly in that the segfaults (yes! segfaults! in a Python app!) were erratic and very timing dependent, making them hard to reproduce and understand. It wasn't until I _realised_ it was thread/concurrency related that I could fix it. Note that in Tk you can have a callback do GUI work, just not in a separate thread. Cheers, Cameron Simpson From python at mrabarnett.plus.com Tue Jan 10 20:20:53 2023 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 11 Jan 2023 01:20:53 +0000 Subject: Tkinter GUI freezing, used Thread then encountered RuntimeError: threads can only be started once In-Reply-To: References: Message-ID: On 2023-01-11 00:13, Cameron Simpson wrote: > On 10Jan2023 18:32, MRAB wrote: >>I don't like how you're passing Thread...start as an argument. IMHO, it >>would be better/cleaner to pass a plain function, even if the only >>thing that function does is to start the thread. > > Yes, and this is likely the thing causing the cited exception "threads > can only be started once". Your setup of the button with the action > defined as: > > Thread(....).start > > creates a _single_ new Thread _when you define the button_, and makes > hte button callback try to start it. On the second and following > callback, you're trying to start the _same_ single Thread again. > You're right! I missed that detail. :-( > Do as MRAB suggests and have the callback create-and-start a Thread > instead of just starting an _existing_ Thread. > > Also, for simple quick things there's no need to use a Thread at all. If > the final version of the programme is going to do something long running > at that point, then sure. > >>I can't tell what 'change_flag' is doing because of the formatting >>issue. Is it doing GUI stuff? In a thread? If yes, don't do that. The >>GUI doesn't like that. Only the main thread should do GUI stuff. > > Aye. This is very important in almost all GUI toolkits. > > Bit me very badly with Qt once, badly in that the segfaults (yes! > segfaults! in a Python app!) were erratic and very timing dependent, > making them hard to reproduce and understand. It wasn't until I > _realised_ it was thread/concurrency related that I could fix it. > > Note that in Tk you can have a callback do GUI work, just not in a > separate thread. > From greg.ewing at canterbury.ac.nz Tue Jan 10 20:31:43 2023 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 11 Jan 2023 14:31:43 +1300 Subject: To clarify how Python handles two equal objects In-Reply-To: References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> Message-ID: On 11/01/23 11:21 am, Jen Kris wrote: > where one object derives from another object (a = b[0], for example), any operation that would alter one will alter the other. I think you're still confused. In C terms, after a = b[0], a and b[0] are pointers to the same block of memory. If you change that block of memory, then of course you will see the change through either pointer. Here's a rough C translation of some of your Python code: /* mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ] */ int **mx1 = (int **)malloc(3 * sizeof(int *)); mx1[0] = (int *)malloc(3 * sizeof(int)); mx1[0][0] = 1; mx1[0][1] = 2; mx1[0][2] = 3; mx1[1] = (int *)malloc(3 * sizeof(int)); mx1[1][0] = 4; mx1[1][1] = 5; mx1[1][2] = 6; mx1[2] = (int *)malloc(3 * sizeof(int)); mx1[2][0] = 7; mx1[2][1] = 8; mx1[2][2] = 9; /* arr1 = mx1[2] */ int *arr1 = mx[2]; /* arr1 = [ 10, 11, 12 ] */ arr1 = (int *)malloc(3 * sizeof(int)); arr1[0] = 10; arr1[1] = 11; arr1[2] = 12; Does that help your understanding? -- Greg From aapost at idontexist.club Tue Jan 10 22:15:07 2023 From: aapost at idontexist.club (aapost) Date: Tue, 10 Jan 2023 22:15:07 -0500 Subject: Python - working with xml/lxml/objectify/schemas, datatypes, and assignments References: <25525.36948.543184.270051@ixdm.fritz.box> Message-ID: On 1/4/23 12:13, aapost wrote: > On 1/4/23 09:42, Dieter Maurer wrote: >> aapost wrote at 2023-1-3 22:57 -0500: >>> ... >>> Consider the following: >>> >>> from lxml import objectify, etree >>> schema = etree.XMLSchema(file="path_to_my_xsd_schema_file") >>> parser = objectify.makeparser(schema=schema, encoding="UTF-8") >>> xml_obj = objectify.parse("path_to_my_xml_file", parser=parser) >>> xml_root = xml_obj.getroot() >>> >>> let's say I have a Version element, that is defined simply as a string >>> in a 3rd party provided xsd schema >>> >>> >> >> Does your schema include the third party schema? >> >> You might have a look at `PyXB`, too. >> It tries hard to enforce schema restrictions in Python code. > > > Yes, to clarify, they provide the schema, which is what we use, > downloaded locally. Basically just trying to remain compliant with their > structures that they already define without reinventing the wheel for > numerous calls and custom types, and in a way that feels more live > rather than just checking validity at the end of the edits as if I were > modifying the XML manually. > > Thank you for the suggestion, PyXB works much more like how I envisioned > working with xml in my head: > > >>> xml_root.Version = 1231.32000 > pyxb.exceptions_.SimpleTypeValueError: Type > {http://www.w3.org/2001/XMLSchema}string cannot be created from: 1231.32 > >>> xml_root.Version = "1231.32000" > > I will have to do some more testing to see how smooth the transition > back to a formatted document goes, since it creates a variable for all > possible fields defined in the type, even if they are optional and not > there in the situational template. > > Thanks Unfortunately picking it apart for a while and diving deeper in to a rabbit hole, PyXB looks to be a no-go. PyXB while interesting, and I respect it's complexity and depth, is lacking in design consistency in how it operates if you are trying to modify and work with the resulting structure intuitively. It was developed on Python2 14 years ago, made compatible with python3 late, seems like it was trying to maintain vast version compatibility rather than getting a needed overhaul, before being abandoned in 2017 after the author moved on to more interesting work... I don't blame him, lol.. The community forks are just minor bug fixes currently. There are no setValue()/_setValue() functions for SimpleTypes (the bulk of your objects) so you can't change their values directly. Assigning to them appears to work if they are nested inside a parent that has __setattr__ overloaded (as a default resulting structure does when you first load a document), but it is a rats nest as far as what happens from there. Sometimes it calls .Factory(), sometimes it goes through a series of __init__s, but nothing is really clear on what is or is not a kosher approach to managing value changes, and my attempts have failed so far to see if I could figure out how to encompass those paths in to a single _setValue() call. Then there are ComplexTypes, with a value called _IsSimpleContent, which indicates whether it is a wrapper for a custom SimpleType, or something that does not contain SimpleType data. These DO have _setValue() functions IF it contains SimpleType data, where the SimpleType is stored in a __content member variable. Assignment on these also appears to work but the results aren't good, you need to use _setValue(), or you lose things like attributes. It would have been nicer if the structure of the ComplexType was called something else and wrapped all objects with a common set of functions. The validate functions do not work how one would assume, like they do for other libraries, where they go back and verify the data. I believe they only function on the way in, because if the data becomes invalid through some manual messing with it after the fact, they indicate that the data is still valid even when it's not. It seems like what I am probably looking for may reside in java with JAXB, but nothing really beyond that. generateDS, doesn't really offer anything I need from what I could tell in messing with it, and by the looks of it, I might have to bite the bullet and use the xmlschema library, work with dicts, handle many more corner cases on my side, and just let the result be a lot clunkier than I was hoping. Unless I find 8 years to redesign the wheel myself, not sure I am granted that ability though. Oh well. lol From stephen_tucker at sil.org Wed Jan 11 05:31:31 2023 From: stephen_tucker at sil.org (Stephen Tucker) Date: Wed, 11 Jan 2023 10:31:31 +0000 Subject: file.read Method Documentation (Python 2.7.10) In-Reply-To: References: Message-ID: Chris - In the Python 2.7.10 documentation, I am referring to section 5. Built-in Types, subsection 5.9 File Objects. In that subsection, I have the following paragraph: file.read([*size*]) Read at most *size* bytes from the file (less if the read hits EOF before obtaining *size* bytes). If the *size* argument is negative or omitted, read all data until EOF is reached. The bytes are returned as a string object. An empty string is returned when EOF is encountered immediately. (For certain files, like ttys, it makes sense to continue reading after an EOF is hit.) Note that this method may call the underlying C function fread() more than once in an effort to acquire as close to *size* bytes as possible. Also note that when in non-blocking mode, less data than was requested may be returned, even if no *size* parameter was given. Note This function is simply a wrapper for the underlying fread() C function, and will behave the same in corner cases, such as whether the EOF value is cached. Stephen. On Mon, Jan 9, 2023 at 6:25 PM Chris Angelico wrote: > On Tue, 10 Jan 2023 at 01:36, Stephen Tucker > wrote: > > > > Dear Python-list, > > > > Yes, I know that Python 2.x is no longer supported. > > > > I have found that the documentation for this method is misleading when > the > > file being read is UTF-8-encoded: > > > > Instead of reading *size* bytes, the method reads *size *UTF-8 byte > > *sequences*. > > > > Has this error been corrected in the Python 3.x documentation? > > > > What documentation is this? The builtin 'file' type doesn't know > anything about encodings, and only ever returns bytes. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > From rosuav at gmail.com Wed Jan 11 06:00:21 2023 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 11 Jan 2023 22:00:21 +1100 Subject: file.read Method Documentation (Python 2.7.10) In-Reply-To: References: Message-ID: On Wed, 11 Jan 2023 at 21:31, Stephen Tucker wrote: > > Chris - > > In the Python 2.7.10 documentation, I am referring to section 5. Built-in Types, subsection 5.9 File Objects. > > In that subsection, I have the following paragraph: > > file.read([size]) > > Read at most size bytes from the file (less if the read hits EOF before obtaining size bytes). If the size argument is negative or omitted, read all data until EOF is reached. The bytes are returned as a string object. An empty string is returned when EOF is encountered immediately. (For certain files, like ttys, it makes sense to continue reading after an EOF is hit.) Note that this method may call the underlying C function fread() more than once in an effort to acquire as close to size bytes as possible. Also note that when in non-blocking mode, less data than was requested may be returned, even if no size parameter was given. > Yes, so it should be that number of bytes, which is what it does, isn't it? ChrisA From jenkris at tutanota.com Wed Jan 11 10:33:50 2023 From: jenkris at tutanota.com (Jen Kris) Date: Wed, 11 Jan 2023 16:33:50 +0100 (CET) Subject: To clarify how Python handles two equal objects In-Reply-To: References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> Message-ID: Yes, I did understand that.? In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they?re the same memory block).? My issue in Python came up because Python can dynamically change one or the other to a different object (memory block) so I have to be aware of that when handing this kind of situation.? Jan 10, 2023, 17:31 by greg.ewing at canterbury.ac.nz: > On 11/01/23 11:21 am, Jen Kris wrote: > >> where one object derives from another object (a = b[0], for example), any operation that would alter one will alter the other. >> > > I think you're still confused. In C terms, after a = b[0], a and b[0] > are pointers to the same block of memory. If you change that block of > memory, then of course you will see the change through either pointer. > > Here's a rough C translation of some of your Python code: > > /* mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ] */ > int **mx1 = (int **)malloc(3 * sizeof(int *)); > mx1[0] = (int *)malloc(3 * sizeof(int)); > mx1[0][0] = 1; > mx1[0][1] = 2; > mx1[0][2] = 3; > mx1[1] = (int *)malloc(3 * sizeof(int)); > mx1[1][0] = 4; > mx1[1][1] = 5; > mx1[1][2] = 6; > mx1[2] = (int *)malloc(3 * sizeof(int)); > mx1[2][0] = 7; > mx1[2][1] = 8; > mx1[2][2] = 9; > > /* arr1 = mx1[2] */ > int *arr1 = mx[2]; > > /* arr1 = [ 10, 11, 12 ] */ > arr1 = (int *)malloc(3 * sizeof(int)); > arr1[0] = 10; > arr1[1] = 11; > arr1[2] = 12; > > Does that help your understanding? > > -- > Greg > -- > https://mail.python.org/mailman/listinfo/python-list > From roel at roelschroeven.net Wed Jan 11 12:13:44 2023 From: roel at roelschroeven.net (Roel Schroeven) Date: Wed, 11 Jan 2023 18:13:44 +0100 Subject: To clarify how Python handles two equal objects In-Reply-To: References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> Message-ID: <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list: > Yes, I did understand that.? In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they?re the same memory block). Sorry if you feel I'm being overly pedantic, but your explanation "an operation on one is an operation on the other (because they?re the same memory block)" still feels a bit misguided. "One" and "other" still make it sound like there are two objects, and "an operation on one" and "an operation on the other" make it sound like there are two operations. Sometimes it doesn't matter if we're a bit sloppy for sake of simplicity or convenience, sometimes we really need to be precise. I think this is a case where we need to be precise. So, to be precise: there is only one object, with possible multiple names to it. We can change the object, using one of the names. That is one and only one operation on one and only one object. Since the different names refer to the same object, that change will of course be visible through all of them. Note that 'name' in that sentence doesn't just refer to variables (mx1, arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...), loop variables, function arguments. The correct mental model is important here, and I do think you're on track or very close to it, but the way you phrase things does give me that nagging feeling that you still might be just a bit off. -- "Peace cannot be kept by force. It can only be achieved through understanding." -- Albert Einstein From stephen_tucker at sil.org Wed Jan 11 12:31:26 2023 From: stephen_tucker at sil.org (Stephen Tucker) Date: Wed, 11 Jan 2023 17:31:26 +0000 Subject: file.read Method Documentation (Python 2.7.10) In-Reply-To: References: Message-ID: Chris, Thanks for your reply. I hope the evidence below (taken from IDLE) clarifies my issue: Stephen. ====================== 1. Create BOM.txt ----------------- >>> myfil = open ("BOM.txt", "wb") >>> myfil.write ("\xef" + "\xbb" + "\xbf") >>> myfil.close() 2. Input three bytes at once from BOM.txt and print them -------------------------------------------------------- >>> myfil = open ("BOM.txt", "rb") >>> myBOM = myfil.read (3) >>> myBOM '\xef\xbb\xbf' >>> myfil.close() 3. Input three bytes one at a time from BOM.txt and print them -------------------------------------------------------------- >>> myfil = open ("BOM.txt", "rb") >>> myBOM_1 = myfil.read (1) >>> myBOM_2 = myfil.read (1) >>> myBOM_3 = myfil.read (1) >>> myBOM_1 '\xef' >>> myBOM_2 '\xbb' >>> myBOM_3 '\xbf' >>> myfil.close() 4. Input three bytes at once from BOM.txt and print them -------------------------------------------------------- >>> import codecs >>> myfil = codecs.open ("BOM.txt", mode="rb", encoding="UTF-8") >>> myBOM = unicode (myfil.read (3)) >>> myBOM u'\ufeff' >>> myfil.close () 5. Attempt to input three bytes one at a time from BOM.txt and print them ------------------------------------------------------------------------- >>> myfil = codecs.open ("BOM.txt", mode="rb", encoding="UTF-8") >>> myBOM_4 = myfil.read (1) >>> myBOM_5 = myfil.read (1) >>> myBOM_6 = myfil.read (1) >>> myBOM_4 u'\ufeff' >>> myBOM_5 u'' >>> myBOM_6 u'' >>> myfil.close() Notes A. The attempt at Part 5 actually inputs all three bytes when we ask it to input just the first one! B. The outcome from Part 5 shows that, actually, the request to input text in Part 4 brought about a response from the program something like this: Input the UTF-8-encoded character as the first "byte"; As expected, after reaching the end of the file, continue supplying an empty string for each of the requested extra bytes. ====================== On Wed, Jan 11, 2023 at 11:00 AM Chris Angelico wrote: > On Wed, 11 Jan 2023 at 21:31, Stephen Tucker > wrote: > > > > Chris - > > > > In the Python 2.7.10 documentation, I am referring to section 5. > Built-in Types, subsection 5.9 File Objects. > > > > In that subsection, I have the following paragraph: > > > > file.read([size]) > > > > Read at most size bytes from the file (less if the read hits EOF before > obtaining size bytes). If the size argument is negative or omitted, read > all data until EOF is reached. The bytes are returned as a string object. > An empty string is returned when EOF is encountered immediately. (For > certain files, like ttys, it makes sense to continue reading after an EOF > is hit.) Note that this method may call the underlying C function fread() > more than once in an effort to acquire as close to size bytes as possible. > Also note that when in non-blocking mode, less data than was requested may > be returned, even if no size parameter was given. > > > > Yes, so it should be that number of bytes, which is what it does, isn't it? > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > From rosuav at gmail.com Wed Jan 11 12:36:30 2023 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 12 Jan 2023 04:36:30 +1100 Subject: file.read Method Documentation (Python 2.7.10) In-Reply-To: References: Message-ID: On Thu, 12 Jan 2023 at 04:31, Stephen Tucker wrote: > 1. Create BOM.txt > 2. Input three bytes at once from BOM.txt and print them > 3. Input three bytes one at a time from BOM.txt and print them All of these correctly show that a file, in binary mode, reads and writes bytes. > 4. Input three bytes at once from BOM.txt and print them > >>> import codecs > >>> myfil = codecs.open ("BOM.txt", mode="rb", encoding="UTF-8") This is now a codecs file, NOT a vanilla file object. See its docs here: https://docs.python.org/2.7/library/codecs.html#codecs.open The output is "codec-dependent" but I would assume that UTF-8 will yield Unicode text strings. > 5. Attempt to input three bytes one at a time from BOM.txt and print them > ------------------------------------------------------------------------- > > >>> myfil = codecs.open ("BOM.txt", mode="rb", encoding="UTF-8") > >>> myBOM_4 = myfil.read (1) > >>> myBOM_4 > u'\ufeff' > A. The attempt at Part 5 actually inputs all three bytes when we ask it to input just the first one! On the contrary; you asked it for one *character* and it read one character. Where were you seeing documentation that disagreed with this? ChrisA From roel at roelschroeven.net Wed Jan 11 12:49:39 2023 From: roel at roelschroeven.net (Roel Schroeven) Date: Wed, 11 Jan 2023 18:49:39 +0100 Subject: file.read Method Documentation (Python 2.7.10) In-Reply-To: References: Message-ID: <6fbb23b6-c070-764e-6262-d0dc1d1d546d@roelschroeven.net> Chris Angelico schreef op 11/01/2023 om 18:36: > On Thu, 12 Jan 2023 at 04:31, Stephen Tucker wrote: > > 1. Create BOM.txt > > 2. Input three bytes at once from BOM.txt and print them > > 3. Input three bytes one at a time from BOM.txt and print them > > All of these correctly show that a file, in binary mode, reads and writes bytes. > > > 4. Input three bytes at once from BOM.txt and print them > > >>> import codecs > > >>> myfil = codecs.open ("BOM.txt", mode="rb", encoding="UTF-8") > > This is now a codecs file, NOT a vanilla file object. See its docs here: > > https://docs.python.org/2.7/library/codecs.html#codecs.open > > The output is "codec-dependent" but I would assume that UTF-8 will > yield Unicode text strings. > > > 5. Attempt to input three bytes one at a time from BOM.txt and print them > > ------------------------------------------------------------------------- > > > > >>> myfil = codecs.open ("BOM.txt", mode="rb", encoding="UTF-8") > > >>> myBOM_4 = myfil.read (1) > > >>> myBOM_4 > > u'\ufeff' > > > A. The attempt at Part 5 actually inputs all three bytes when we ask it to input just the first one! > > On the contrary; you asked it for one *character* and it read one character. Not exactly. You're right of course that things opened with codecs.open() behave differently from vanilla file objects. codecs.open() returns a StreamReaderWriter instance, which combines StreamReader and StreamWriter. For read(), StreamReader is what matters (documented at https://docs.python.org/3.11/library/codecs.html#codecs.StreamReader). It's read() method is: read(size=- 1, chars=- 1, firstline=False) _size_ indicates the approximate maximum number of encoded bytes or code points to read for decoding. The decoder can modify this setting as appropriate. _chars_ indicates the number of decoded code points or bytes to return. The read() method will never return more data than requested, but it might return less, if there is not enough available. When only one parameter is provided, without name, it's _size_. So myfil.read(1) asks to read enough bytes to decode 1 code point (approximately). That's totally consistent with the observer behavior. -- "Peace cannot be kept by force. It can only be achieved through understanding." -- Albert Einstein From cl at isbd.net Wed Jan 11 04:52:58 2023 From: cl at isbd.net (Chris Green) Date: Wed, 11 Jan 2023 09:52:58 +0000 Subject: Mailing-Lists (pointer) References: <67lv8j-anpe1.ln1@esprimo.zbmc.eu> Message-ID: Cameron Simpson wrote: > On 10Jan2023 08:45, Chris Green wrote: > >dn wrote: > >> See also the wisdom of enabling comp.lang.python and python-list as > >> 'mirrors', enabling those who prefer one mechanism/client to another, > >> yet maintaining a single 'community'. > >> > >Yes, this is important I think. Plus, if possible, if it's decided to > >move to a forum format make that accessible by E-Mail. > > There's a Discourse forum over at discuss.python.org. I use it in > "mailing list mode" and do almost all my interactions via email, exactly > as I do for python-list. Posts come to me and land in the same local > mail folder I use for python-list. My replies land on the forum as > expected (and of course also go by email to those members who have > turned that mode on). > > So I'm using both the new forum and the currently mailing list still, > and broadly in exactly the same way. > Yes, Discourse is one of the few web forums that also provides full E-Mail access. -- Chris Green ? From dieter at handshake.de Wed Jan 11 13:10:46 2023 From: dieter at handshake.de (Dieter Maurer) Date: Wed, 11 Jan 2023 19:10:46 +0100 Subject: Mailing-Lists (pointer) In-Reply-To: References: <67lv8j-anpe1.ln1@esprimo.zbmc.eu> Message-ID: <25534.64422.128942.642832@ixdm.fritz.box> Cameron Simpson wrote at 2023-1-11 08:37 +1100: > ... >There's a Discourse forum over at discuss.python.org. I use it in >"mailing list mode" and do almost all my interactions via email, exactly >as I do for python-list. Posts come to me and land in the same local >mail folder I use for python-list. My replies land on the forum as >expected (and of course also go by email to those members who have >turned that mode on). I am also using the Plone `Discourse` forum in "mailing list mode". It now works quite well but it took some years before reaching this state. For a very long time, my mail replies did not reach the forum reliably. My latest complaint (more than half a year ago): when I had visited the forum via `http` (I did this occasionally to verify my reply has reached the forum), it sometimes thought, I had seen a new message and did not inform me about it via mail. Meanwhile, all replies seem to arrive reliably and I no longer use `http` for access. Therefore, I do not know whether the behavior described above still persists. From dieter at handshake.de Wed Jan 11 13:21:06 2023 From: dieter at handshake.de (Dieter Maurer) Date: Wed, 11 Jan 2023 19:21:06 +0100 Subject: Python - working with xml/lxml/objectify/schemas, datatypes, and assignments In-Reply-To: References: <25525.36948.543184.270051@ixdm.fritz.box> Message-ID: <25534.65042.617254.20506@ixdm.fritz.box> aapost wrote at 2023-1-10 22:15 -0500: >On 1/4/23 12:13, aapost wrote: >> On 1/4/23 09:42, Dieter Maurer wrote: >> ... >>> You might have a look at `PyXB`, too. >>> It tries hard to enforce schema restrictions in Python code. >> ... >Unfortunately picking it apart for a while and diving deeper in to a >rabbit hole, PyXB looks to be a no-go. > >PyXB while interesting, and I respect it's complexity and depth, is >lacking in design consistency in how it operates if you are trying to >modify and work with the resulting structure intuitively. > ... problem with simple types ... I use `PyXB` in `dm.saml2` and `dm.zope.saml2`, i.e. with the SAML2 schema definitions (which include those of XML signature and XML encryption). I had no problems with simple types. I just assign them to attributes of the Python objects representing the XML elements. `PyXB` does the right thing when it serializes those objects into XML. From jenkris at tutanota.com Wed Jan 11 13:28:36 2023 From: jenkris at tutanota.com (Jen Kris) Date: Wed, 11 Jan 2023 19:28:36 +0100 (CET) Subject: To clarify how Python handles two equal objects In-Reply-To: <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> Message-ID: Thanks for your comments.? After all, I asked for clarity so it?s not pedantic to be precise, and you?re helping to clarify.? Going back to my original post, mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ] arr1 = mx1[2] Now if I write "arr1[1] += 5" then both arr1 and mx1[2][1] will be changed because while they are different names, they are the assigned same memory location (pointer).? Similarly, if I write "mx1[2][1] += 5" then again both names will be updated.? That?s what I meant by "an operation on one is an operation on the other."? To be more precise, an operation on one name will be reflected in the other name.? The difference is in the names,? not the pointers.? Each name has the same pointer in my example, but operations can be done in Python using either name.? Jan 11, 2023, 09:13 by roel at roelschroeven.net: > Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list: > >> Yes, I did understand that.? In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they?re the same memory block). >> > > Sorry if you feel I'm being overly pedantic, but your explanation "an operation on one is an operation on the other (because they?re the same memory block)" still feels a bit misguided. "One" and "other" still make it sound like there are two objects, and "an operation on one" and "an operation on the other" make it sound like there are two operations. > Sometimes it doesn't matter if we're a bit sloppy for sake of simplicity or convenience, sometimes we really need to be precise. I think this is a case where we need to be precise. > > So, to be precise: there is only one object, with possible multiple names to it. We can change the object, using one of the names. That is one and only one operation on one and only one object. Since the different names refer to the same object, that change will of course be visible through all of them. > Note that 'name' in that sentence doesn't just refer to variables (mx1, arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...), loop variables, function arguments. > > The correct mental model is important here, and I do think you're on track or very close to it, but the way you phrase things does give me that nagging feeling that you still might be just a bit off. > > -- > "Peace cannot be kept by force. It can only be achieved through understanding." > -- Albert Einstein > > -- > https://mail.python.org/mailman/listinfo/python-list > From Richard at Damon-Family.org Wed Jan 11 13:55:48 2023 From: Richard at Damon-Family.org (Richard Damon) Date: Wed, 11 Jan 2023 13:55:48 -0500 Subject: To clarify how Python handles two equal objects In-Reply-To: References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> Message-ID: <7a5a4443-faf4-ecfe-f25e-eaa6e599abd3@Damon-Family.org> I think the key point is that "the operation" doesn't act on "names" but on "objects" (which are different sort of things), and thus there isn't an "the other" when talking about the object being operated on. Thinking of an operation being on a "name" is the mental model error. The only operations that operate on "names" are assignment operations. Augmented assignment operations are more complicatd, as they can either work on the object the name points on, if that object is mutable, or rebinds the name to a new object if it isn't. Thus a += b is NOT neccessarilily the same as a = a + b, as a += b might just mutate the object that a is bound to or might rebind in the manner of a = a + b; Thus: a = [ 1, 2, 3] b = a a += [4] will change the single list that a and b are bound to into [1, 2, 3, 4], while a = "foo" b = a a += "bar" will change a to be bound to the string object "foobar" but not b, since the string object "foo" wasn't mutable. Brings up the point, that you need to be careful with augmented assignment operators. On 1/11/23 1:28 PM, Jen Kris via Python-list wrote: > Thanks for your comments.? After all, I asked for clarity so it?s not pedantic to be precise, and you?re helping to clarify. > > Going back to my original post, > > mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ] > arr1 = mx1[2] > > Now if I write "arr1[1] += 5" then both arr1 and mx1[2][1] will be changed because while they are different names, they are the assigned same memory location (pointer).? Similarly, if I write "mx1[2][1] += 5" then again both names will be updated. > > That?s what I meant by "an operation on one is an operation on the other."? To be more precise, an operation on one name will be reflected in the other name.? The difference is in the names,? not the pointers.? Each name has the same pointer in my example, but operations can be done in Python using either name. > > > > > Jan 11, 2023, 09:13 by roel at roelschroeven.net: > >> Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list: >> >>> Yes, I did understand that.? In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they?re the same memory block). >>> >> Sorry if you feel I'm being overly pedantic, but your explanation "an operation on one is an operation on the other (because they?re the same memory block)" still feels a bit misguided. "One" and "other" still make it sound like there are two objects, and "an operation on one" and "an operation on the other" make it sound like there are two operations. >> Sometimes it doesn't matter if we're a bit sloppy for sake of simplicity or convenience, sometimes we really need to be precise. I think this is a case where we need to be precise. >> >> So, to be precise: there is only one object, with possible multiple names to it. We can change the object, using one of the names. That is one and only one operation on one and only one object. Since the different names refer to the same object, that change will of course be visible through all of them. >> Note that 'name' in that sentence doesn't just refer to variables (mx1, arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...), loop variables, function arguments. >> >> The correct mental model is important here, and I do think you're on track or very close to it, but the way you phrase things does give me that nagging feeling that you still might be just a bit off. >> >> -- >> "Peace cannot be kept by force. It can only be achieved through understanding." >> -- Albert Einstein >> >> -- >> https://mail.python.org/mailman/listinfo/python-list >> -- Richard Damon From wlfraed at ix.netcom.com Wed Jan 11 13:55:09 2023 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Wed, 11 Jan 2023 13:55:09 -0500 Subject: To clarify how Python handles two equal objects References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> Message-ID: <9f1urhdjteo73ei30bhicg69hvl5cc9tni@4ax.com> On Tue, 10 Jan 2023 16:59:59 -0500, Thomas Passin declaimed the following: >Just to add a possibly picky detail to what others have said, Python >does not have an "array" type. It has a "list" type, as well as some >other, not necessarily mutable, sequence types. > However, it has long had https://docs.python.org/3/library/array.html -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From hjp-python at hjp.at Wed Jan 11 15:26:22 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Wed, 11 Jan 2023 21:26:22 +0100 Subject: To clarify how Python handles two equal objects In-Reply-To: References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> Message-ID: <20230111202622.s42zndlcskauzyih@hjp.at> On 2023-01-11 18:49:14 +0000, Stefan Ram wrote: > Jen Kris writes: > >Each name has the same pointer > > ... from the C programmer's point of view. > > From the Python programmer's point of view, there are no "pointers". That's just window dressing. Pointers are evil, so we can't have pointers. So we'll just call them by a different name. > Instead, names or other targets are /assigned/ or /bound/ to objects. This is the wrong way around. The name isn't assigned to the object. The object is assigned to the name. As you quoted: > |... the sequence is asked to assign the assigned object to > | its item with that index ... Also, while the word "bind" is used in this manner in the official docs > |... the name is bound to the object ... and some members of this list, I think it is really misleading. It sounds like the name is now an attribute of the object. But it isn't. There is no way to the name (or the names) from the object. But there is a way to get from the name the object. I can't think of a snappy, unambiguous single verb to describe what's happening here, but do we need one? We already have words like "assign", "refer", "point" with fairly standardized meaning in IT. We can write that "a reference to an object is assigned to a variable", and that after the assignment "the variable refers to the object". Or we can even use the P-word. And since we know that variables in Python can only contain references and never values, we can abbreviate "a reference to an object" as "an object" in most contexts. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From cs at cskk.id.au Wed Jan 11 15:45:05 2023 From: cs at cskk.id.au (Cameron Simpson) Date: Thu, 12 Jan 2023 07:45:05 +1100 Subject: Mailing-Lists (pointer) In-Reply-To: <25534.64422.128942.642832@ixdm.fritz.box> References: <25534.64422.128942.642832@ixdm.fritz.box> Message-ID: On 11Jan2023 19:10, Dieter Maurer wrote: >Cameron Simpson wrote at 2023-1-11 08:37 +1100: >> ... >>There's a Discourse forum over at discuss.python.org. I use it in >>"mailing list mode" and do almost all my interactions via email, exactly >>as I do for python-list. [...] > >I am also using the Plone `Discourse` forum in "mailing list mode". >It now works quite well but it took some years before reaching this state. Some of this kind of thing will be because the Doscourse devs, as you might imagine, are forum/web-first people while you and I are email-first people. So they won't notice email shortcomings as readily. That said, they do seem very engaged and willing to chase and fix bugs if they can be identified. >For a very long time, my mail replies did not reach the forum reliably. >My latest complaint (more than half a year ago): when I had visited >the forum via `http` (I did this occasionally to verify >my reply has reached the forum), it sometimes thought, I had >seen a new message and did not inform me about it via mail. There's certainly still an issue where some messages are not reliably sent via email when the inbound needs-spam-review filter flags a message/post, particularly the first post; they're fixing that right now :-) >Meanwhile, all replies seem to arrive reliably and I no longer >use `http` for access. Therefore, I do not know whether >the behavior described above still persists. One day everything will be perfect! Cheers, Cameron Simpson From stephen_tucker at sil.org Thu Jan 12 05:25:11 2023 From: stephen_tucker at sil.org (Stephen Tucker) Date: Thu, 12 Jan 2023 10:25:11 +0000 Subject: file.read Method Documentation (Python 2.7.10) In-Reply-To: References: Message-ID: Chris, Thanks for this clarification. I have not found documentation that disagrees with you. I simply observe that the documentation that I have alluded to earlier in this chain (section 5.9 File Objects) could have been made clearer by the addition of a note along the lines that the behaviour of a file's read method (in particular, what the unit of information is that it reads (that is, "byte", "UTF-8 encoded character", or whatever)) depends on the way in which the file has been opened. Thank you, Chris (and others) for your attention to my request. I consider this enquiry closed. Stephen. On Wed, Jan 11, 2023 at 5:36 PM Chris Angelico wrote: > On Thu, 12 Jan 2023 at 04:31, Stephen Tucker > wrote: > > 1. Create BOM.txt > > 2. Input three bytes at once from BOM.txt and print them > > 3. Input three bytes one at a time from BOM.txt and print them > > All of these correctly show that a file, in binary mode, reads and writes > bytes. > > > 4. Input three bytes at once from BOM.txt and print them > > >>> import codecs > > >>> myfil = codecs.open ("BOM.txt", mode="rb", encoding="UTF-8") > > This is now a codecs file, NOT a vanilla file object. See its docs here: > > https://docs.python.org/2.7/library/codecs.html#codecs.open > > The output is "codec-dependent" but I would assume that UTF-8 will > yield Unicode text strings. > > > 5. Attempt to input three bytes one at a time from BOM.txt and print them > > ------------------------------------------------------------------------- > > > > >>> myfil = codecs.open ("BOM.txt", mode="rb", encoding="UTF-8") > > >>> myBOM_4 = myfil.read (1) > > >>> myBOM_4 > > u'\ufeff' > > > A. The attempt at Part 5 actually inputs all three bytes when we ask it > to input just the first one! > > On the contrary; you asked it for one *character* and it read one > character. > > Where were you seeing documentation that disagreed with this? > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > From rosuav at gmail.com Thu Jan 12 06:49:54 2023 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 12 Jan 2023 22:49:54 +1100 Subject: file.read Method Documentation (Python 2.7.10) In-Reply-To: References: Message-ID: On Thu, 12 Jan 2023 at 21:25, Stephen Tucker wrote: > > Chris, > > Thanks for this clarification. > > I have not found documentation that disagrees with you. I simply observe that the documentation that I have alluded to earlier in this chain (section 5.9 File Objects) That's specifically the plain file objects. Other types of objects behave differently. > could have been made clearer by the addition of a note along the lines that the behaviour of a file's read method (in particular, what the unit of information is that it reads (that is, "byte", "UTF-8 encoded character", or whatever)) depends on the way in which the file has been opened. > > Thank you, Chris (and others) for your attention to my request. I consider this enquiry closed. > Cool cool! ChrisA From avi.e.gross at gmail.com Thu Jan 12 23:51:00 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Thu, 12 Jan 2023 23:51:00 -0500 Subject: To clarify how Python handles two equal objects In-Reply-To: References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> Message-ID: <00d801d9270a$a2597c30$e70c7490$@gmail.com> Jen, It is dangerous territory you are treading as there are times all or parts of objects are copied, or changed in place or the method you use to make a view is not doing quite what you want. As an example, you can create a named slice such as: middle_by_two = slice(5, 10, 2) The above is not in any sense pointing at anything yet. But given a long enough list or other such objects, it will take items (starting at index 0) starting with item that are at indices 5 then 7 then 9 as in this: nums = [n for n in range(12)] nums[middle_by_two] [5, 7, 9] The same slice will work on anything else: list('abcdefghijklmnopqrstuvwxyz')[middle_by_two] ['f', 'h', 'j'] So although you may think the slice is bound to something, it is not. It is an object that only later is briefly connected to whatever you want to apply it to. If I later change nums, above, like this: nums = [-3, -2, -1] + nums nums [-3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] nums[middle_by_two] [2, 4, 6] In the example, you can forget about whether we are talking about pointers directly or indirectly or variable names and so on. Your "view" remains valid ONLY as long as you do not change either the slice or the underlying object you are applying to -- at least not the items you want to extract. Since my example inserted three new items at the start using negative numbers for illustration, you would need to adjust the slice by making a new slice designed to fit your new data. The example below created an adjusted slice that adds 3 to the start and stop settings of the previous slice while copying the step value and then it works on the elongated object: middle_by_two_adj = slice(middle_by_two.start + 3, middle_by_two.stop + 3, middle_by_two.step) nums[middle_by_two_adj] [5, 7, 9] A suggestion is that whenever you are not absolutely sure that the contents of some data structure might change without your participation, then don't depend on various kinds of aliases to keep the contents synchronized. Make a copy, perhaps a deep copy and make sure the only thing ever changing it is your code and later, if needed, copy the result back to any other data structure. Of course, if anything else is accessing the result in the original in between, it won't work. Just FYI, a similar analysis applies to uses of the numpy and pandas and other modules if you get some kind of object holding indices to a series such as integers or Booleans and then later try using it after the number of items or rows or columns have changed. Your indices no longer match. Avi -----Original Message----- From: Python-list On Behalf Of Jen Kris via Python-list Sent: Wednesday, January 11, 2023 1:29 PM To: Roel Schroeven Cc: python-list at python.org Subject: Re: To clarify how Python handles two equal objects Thanks for your comments. After all, I asked for clarity so it?s not pedantic to be precise, and you?re helping to clarify. Going back to my original post, mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ] arr1 = mx1[2] Now if I write "arr1[1] += 5" then both arr1 and mx1[2][1] will be changed because while they are different names, they are the assigned same memory location (pointer). Similarly, if I write "mx1[2][1] += 5" then again both names will be updated. That?s what I meant by "an operation on one is an operation on the other." To be more precise, an operation on one name will be reflected in the other name. The difference is in the names, not the pointers. Each name has the same pointer in my example, but operations can be done in Python using either name. Jan 11, 2023, 09:13 by roel at roelschroeven.net: > Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list: > >> Yes, I did understand that. In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they?re the same memory block). >> > > Sorry if you feel I'm being overly pedantic, but your explanation "an operation on one is an operation on the other (because they?re the same memory block)" still feels a bit misguided. "One" and "other" still make it sound like there are two objects, and "an operation on one" and "an operation on the other" make it sound like there are two operations. > Sometimes it doesn't matter if we're a bit sloppy for sake of simplicity or convenience, sometimes we really need to be precise. I think this is a case where we need to be precise. > > So, to be precise: there is only one object, with possible multiple names to it. We can change the object, using one of the names. That is one and only one operation on one and only one object. Since the different names refer to the same object, that change will of course be visible through all of them. > Note that 'name' in that sentence doesn't just refer to variables (mx1, arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...), loop variables, function arguments. > > The correct mental model is important here, and I do think you're on track or very close to it, but the way you phrase things does give me that nagging feeling that you still might be just a bit off. > > -- > "Peace cannot be kept by force. It can only be achieved through understanding." > -- Albert Einstein > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list From jenkris at tutanota.com Fri Jan 13 10:57:45 2023 From: jenkris at tutanota.com (Jen Kris) Date: Fri, 13 Jan 2023 16:57:45 +0100 (CET) Subject: To clarify how Python handles two equal objects In-Reply-To: <00d801d9270a$a2597c30$e70c7490$@gmail.com> References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> <00d801d9270a$a2597c30$e70c7490$@gmail.com> Message-ID: Avi, Thanks for your comments.? You make a good point.? Going back to my original question, and using your slice() example:? middle_by_two = slice(5, 10, 2) nums = [n for n in range(12)] q = nums[middle_by_two] x = id(q) b = q y = id(b) If I assign "b" to "q", then x and y match ? they point to the same memory until "b" OR "q" are? reassigned to something else.? If "q" changes during the lifetime of "b" then it?s not safe to use the pointer to "q" for "b", as in: nums = [n for n in range(2, 14)] q = nums[middle_by_two] x = id(q) y = id(b) Now "x" and "y" are different, as we would expect.? So when writing a spot speed up in a compiled language, you can see in the Python source if either is reassigned, so you?ll know how to handle it.? The motivation behind my question was that in a compiled extension it?s faster to borrow a pointer than to move an entire array if it?s possible, but special care must be taken.? Jen Jan 12, 2023, 20:51 by avi.e.gross at gmail.com: > Jen, > > It is dangerous territory you are treading as there are times all or parts of objects are copied, or changed in place or the method you use to make a view is not doing quite what you want. > > As an example, you can create a named slice such as: > > middle_by_two = slice(5, 10, 2) > > The above is not in any sense pointing at anything yet. But given a long enough list or other such objects, it will take items (starting at index 0) starting with item that are at indices 5 then 7 then 9 as in this: > > nums = [n for n in range(12)] > nums[middle_by_two] > > [5, 7, 9] > > The same slice will work on anything else: > > list('abcdefghijklmnopqrstuvwxyz')[middle_by_two] > ['f', 'h', 'j'] > > So although you may think the slice is bound to something, it is not. It is an object that only later is briefly connected to whatever you want to apply it to. > > If I later change nums, above, like this: > > nums = [-3, -2, -1] + nums > nums > [-3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] > nums[middle_by_two] > [2, 4, 6] > > In the example, you can forget about whether we are talking about pointers directly or indirectly or variable names and so on. Your "view" remains valid ONLY as long as you do not change either the slice or the underlying object you are applying to -- at least not the items you want to extract. > > Since my example inserted three new items at the start using negative numbers for illustration, you would need to adjust the slice by making a new slice designed to fit your new data. The example below created an adjusted slice that adds 3 to the start and stop settings of the previous slice while copying the step value and then it works on the elongated object: > > middle_by_two_adj = slice(middle_by_two.start + 3, middle_by_two.stop + 3, middle_by_two.step) > nums[middle_by_two_adj] > [5, 7, 9] > > A suggestion is that whenever you are not absolutely sure that the contents of some data structure might change without your participation, then don't depend on various kinds of aliases to keep the contents synchronized. Make a copy, perhaps a deep copy and make sure the only thing ever changing it is your code and later, if needed, copy the result back to any other data structure. Of course, if anything else is accessing the result in the original in between, it won't work. > > Just FYI, a similar analysis applies to uses of the numpy and pandas and other modules if you get some kind of object holding indices to a series such as integers or Booleans and then later try using it after the number of items or rows or columns have changed. Your indices no longer match. > > Avi > > -----Original Message----- > From: Python-list On Behalf Of Jen Kris via Python-list > Sent: Wednesday, January 11, 2023 1:29 PM > To: Roel Schroeven > Cc: python-list at python.org > Subject: Re: To clarify how Python handles two equal objects > > Thanks for your comments. After all, I asked for clarity so it?s not pedantic to be precise, and you?re helping to clarify. > > Going back to my original post, > > mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ] > arr1 = mx1[2] > > Now if I write "arr1[1] += 5" then both arr1 and mx1[2][1] will be changed because while they are different names, they are the assigned same memory location (pointer). Similarly, if I write "mx1[2][1] += 5" then again both names will be updated. > > That?s what I meant by "an operation on one is an operation on the other." To be more precise, an operation on one name will be reflected in the other name. The difference is in the names, not the pointers. Each name has the same pointer in my example, but operations can be done in Python using either name. > > > > > Jan 11, 2023, 09:13 by roel at roelschroeven.net: > >> Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list: >> >>> Yes, I did understand that. In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they?re the same memory block). >>> >> >> Sorry if you feel I'm being overly pedantic, but your explanation "an operation on one is an operation on the other (because they?re the same memory block)" still feels a bit misguided. "One" and "other" still make it sound like there are two objects, and "an operation on one" and "an operation on the other" make it sound like there are two operations. >> Sometimes it doesn't matter if we're a bit sloppy for sake of simplicity or convenience, sometimes we really need to be precise. I think this is a case where we need to be precise. >> >> So, to be precise: there is only one object, with possible multiple names to it. We can change the object, using one of the names. That is one and only one operation on one and only one object. Since the different names refer to the same object, that change will of course be visible through all of them. >> Note that 'name' in that sentence doesn't just refer to variables (mx1, arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...), loop variables, function arguments. >> >> The correct mental model is important here, and I do think you're on track or very close to it, but the way you phrase things does give me that nagging feeling that you still might be just a bit off. >> >> -- >> "Peace cannot be kept by force. It can only be achieved through understanding." >> -- Albert Einstein >> >> -- >> https://mail.python.org/mailman/listinfo/python-list >> > > -- > https://mail.python.org/mailman/listinfo/python-list > From bob at mellowood.ca Fri Jan 13 11:45:42 2023 From: bob at mellowood.ca (Bob van der Poel) Date: Fri, 13 Jan 2023 09:45:42 -0700 Subject: To clarify how Python handles two equal objects In-Reply-To: References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> <00d801d9270a$a2597c30$e70c7490$@gmail.com> Message-ID: It seems to me that the the entire concept of relying on python's idea of where an object is stored is just plain dangerous. A most simple example might be: >>> a=1 >>> b=1 >>> a is b True >>> a=1234 >>> b=1234 >>> a is b False Not sure what happens if you manipulate the data referenced by 'b' in the first example thinking you are changing something referred to by 'a' ... but you might be smart to NOT think that you know. On Fri, Jan 13, 2023 at 9:00 AM Jen Kris via Python-list < python-list at python.org> wrote: > > Avi, > > Thanks for your comments. You make a good point. > > Going back to my original question, and using your slice() example: > > middle_by_two = slice(5, 10, 2) > nums = [n for n in range(12)] > q = nums[middle_by_two] > x = id(q) > b = q > y = id(b) > > If I assign "b" to "q", then x and y match ? they point to the same memory > until "b" OR "q" are reassigned to something else. If "q" changes during > the lifetime of "b" then it?s not safe to use the pointer to "q" for "b", > as in: > > nums = [n for n in range(2, 14)] > q = nums[middle_by_two] > x = id(q) > y = id(b) > > Now "x" and "y" are different, as we would expect. So when writing a spot > speed up in a compiled language, you can see in the Python source if either > is reassigned, so you?ll know how to handle it. The motivation behind my > question was that in a compiled extension it?s faster to borrow a pointer > than to move an entire array if it?s possible, but special care must be > taken. > > Jen > > > > Jan 12, 2023, 20:51 by avi.e.gross at gmail.com: > > > Jen, > > > > It is dangerous territory you are treading as there are times all or > parts of objects are copied, or changed in place or the method you use to > make a view is not doing quite what you want. > > > > As an example, you can create a named slice such as: > > > > middle_by_two = slice(5, 10, 2) > > > > The above is not in any sense pointing at anything yet. But given a long > enough list or other such objects, it will take items (starting at index 0) > starting with item that are at indices 5 then 7 then 9 as in this: > > > > nums = [n for n in range(12)] > > nums[middle_by_two] > > > > [5, 7, 9] > > > > The same slice will work on anything else: > > > > list('abcdefghijklmnopqrstuvwxyz')[middle_by_two] > > ['f', 'h', 'j'] > > > > So although you may think the slice is bound to something, it is not. It > is an object that only later is briefly connected to whatever you want to > apply it to. > > > > If I later change nums, above, like this: > > > > nums = [-3, -2, -1] + nums > > nums > > [-3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] > > nums[middle_by_two] > > [2, 4, 6] > > > > In the example, you can forget about whether we are talking about > pointers directly or indirectly or variable names and so on. Your "view" > remains valid ONLY as long as you do not change either the slice or the > underlying object you are applying to -- at least not the items you want to > extract. > > > > Since my example inserted three new items at the start using negative > numbers for illustration, you would need to adjust the slice by making a > new slice designed to fit your new data. The example below created an > adjusted slice that adds 3 to the start and stop settings of the previous > slice while copying the step value and then it works on the elongated > object: > > > > middle_by_two_adj = slice(middle_by_two.start + 3, middle_by_two.stop + > 3, middle_by_two.step) > > nums[middle_by_two_adj] > > [5, 7, 9] > > > > A suggestion is that whenever you are not absolutely sure that the > contents of some data structure might change without your participation, > then don't depend on various kinds of aliases to keep the contents > synchronized. Make a copy, perhaps a deep copy and make sure the only > thing ever changing it is your code and later, if needed, copy the result > back to any other data structure. Of course, if anything else is accessing > the result in the original in between, it won't work. > > > > Just FYI, a similar analysis applies to uses of the numpy and pandas and > other modules if you get some kind of object holding indices to a series > such as integers or Booleans and then later try using it after the number > of items or rows or columns have changed. Your indices no longer match. > > > > Avi > > > > -----Original Message----- > > From: Python-list > On Behalf Of Jen Kris via Python-list > > Sent: Wednesday, January 11, 2023 1:29 PM > > To: Roel Schroeven > > Cc: python-list at python.org > > Subject: Re: To clarify how Python handles two equal objects > > > > Thanks for your comments. After all, I asked for clarity so it?s not > pedantic to be precise, and you?re helping to clarify. > > > > Going back to my original post, > > > > mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ] > > arr1 = mx1[2] > > > > Now if I write "arr1[1] += 5" then both arr1 and mx1[2][1] will be > changed because while they are different names, they are the assigned same > memory location (pointer). Similarly, if I write "mx1[2][1] += 5" then > again both names will be updated. > > > > That?s what I meant by "an operation on one is an operation on the > other." To be more precise, an operation on one name will be reflected in > the other name. The difference is in the names, not the pointers. Each > name has the same pointer in my example, but operations can be done in > Python using either name. > > > > > > > > > > Jan 11, 2023, 09:13 by roel at roelschroeven.net: > > > >> Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list: > >> > >>> Yes, I did understand that. In your example, "a" and "b" are the same > pointer, so an operation on one is an operation on the other (because > they?re the same memory block). > >>> > >> > >> Sorry if you feel I'm being overly pedantic, but your explanation "an > operation on one is an operation on the other (because they?re the same > memory block)" still feels a bit misguided. "One" and "other" still make it > sound like there are two objects, and "an operation on one" and "an > operation on the other" make it sound like there are two operations. > >> Sometimes it doesn't matter if we're a bit sloppy for sake of > simplicity or convenience, sometimes we really need to be precise. I think > this is a case where we need to be precise. > >> > >> So, to be precise: there is only one object, with possible multiple > names to it. We can change the object, using one of the names. That is one > and only one operation on one and only one object. Since the different > names refer to the same object, that change will of course be visible > through all of them. > >> Note that 'name' in that sentence doesn't just refer to variables (mx1, > arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...), > loop variables, function arguments. > >> > >> The correct mental model is important here, and I do think you're on > track or very close to it, but the way you phrase things does give me that > nagging feeling that you still might be just a bit off. > >> > >> -- > >> "Peace cannot be kept by force. It can only be achieved through > understanding." > >> -- Albert Einstein > >> > >> -- > >> https://mail.python.org/mailman/listinfo/python-list > >> > > > > -- > > https://mail.python.org/mailman/listinfo/python-list > > > > -- > https://mail.python.org/mailman/listinfo/python-list > -- **** Listen to my FREE CD at http://www.mellowood.ca/music/cedars **** Bob van der Poel ** Wynndel, British Columbia, CANADA ** EMAIL: bob at mellowood.ca WWW: http://www.mellowood.ca From mail at axel-reichert.de Fri Jan 13 03:21:52 2023 From: mail at axel-reichert.de (Axel Reichert) Date: Fri, 13 Jan 2023 09:21:52 +0100 Subject: To clarify how Python handles two equal objects References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> <00d801d9270a$a2597c30$e70c7490$@gmail.com> Message-ID: <875ydax2sf.fsf@axel-reichert.de> writes: > As an example, you can create a named slice such as: > > middle_by_two = slice(5, 10, 2) > > The above is not in any sense pointing at anything yet. >From a functional programming point of view this just looks like a partially applied function, and with this in mind the behaviour to me seems to be completely as expected. No surprises here, or do I miss something? Best regards Axel From ethan at stoneleaf.us Fri Jan 13 13:00:01 2023 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 13 Jan 2023 10:00:01 -0800 Subject: The Zen of D.E.K. In-Reply-To: References: Message-ID: On 1/13/23 09:06, Stefan Ram wrote: > "Beautiful is better than ugly." - The Zen of Python > > This says nothing. You have to sacrifice something that > really has /value/! > > "[A]esthetics are more important than efficiency." - Donald E. Knuth [okay, falling for the troll bait] Those two things do not say the same thing; further, in Python at least, and depending on the situation, aesthetics may /not/ be more important than efficiency. -- ~Ethan~ From jenkris at tutanota.com Fri Jan 13 13:24:22 2023 From: jenkris at tutanota.com (Jen Kris) Date: Fri, 13 Jan 2023 19:24:22 +0100 (CET) Subject: To clarify how Python handles two equal objects In-Reply-To: References: <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> <00d801d9270a$a2597c30$e70c7490$@gmail.com> Message-ID: Bob, Your examples show a and b separately defined.? My example is where the definition is a=1; b = a.? But I'm only interested in arrays.? I would not rely on this for integers, and there's not likely to be any real cost savings there. ? Jan 13, 2023, 08:45 by bob at mellowood.ca: > It seems to me that the the entire concept of relying on python's idea of where an object is stored is just plain dangerous. A most simple example might be: > ?? >>> a=1 > ?? >>> b=1 > ?? >>> a is b > ? True > ? >>> a=1234 > ? >>> b=1234 > ? >>> a is b > ? False > > Not sure what happens if you manipulate the data referenced by 'b' in the first example thinking you are changing something referred to by 'a' ... but you might be smart to NOT think that you know. > > > > On Fri, Jan 13, 2023 at 9:00 AM Jen Kris via Python-list <> python-list at python.org> > wrote: > >> >> Avi, >> >> Thanks for your comments.? You make a good point.? >> >> Going back to my original question, and using your slice() example:? >> >> middle_by_two = slice(5, 10, 2) >> nums = [n for n in range(12)] >> q = nums[middle_by_two] >> x = id(q) >> b = q >> y = id(b) >> >> If I assign "b" to "q", then x and y match ? they point to the same memory until "b" OR "q" are? reassigned to something else.? If "q" changes during the lifetime of "b" then it?s not safe to use the pointer to "q" for "b", as in: >> >> nums = [n for n in range(2, 14)] >> q = nums[middle_by_two] >> x = id(q) >> y = id(b) >> >> Now "x" and "y" are different, as we would expect.? So when writing a spot speed up in a compiled language, you can see in the Python source if either is reassigned, so you?ll know how to handle it.? The motivation behind my question was that in a compiled extension it?s faster to borrow a pointer than to move an entire array if it?s possible, but special care must be taken.? >> >> Jen >> >> >> >> Jan 12, 2023, 20:51 by >> avi.e.gross at gmail.com>> : >> >> > Jen, >> > >> > It is dangerous territory you are treading as there are times all or parts of objects are copied, or changed in place or the method you use to make a view is not doing quite what you want. >> > >> > As an example, you can create a named slice such as: >> > >> >? middle_by_two = slice(5, 10, 2) >> > >> > The above is not in any sense pointing at anything yet. But given a long enough list or other such objects, it will take items (starting at index 0) starting with item that are at indices 5 then 7 then 9? as in this: >> > >> >? nums = [n for n in range(12)] >> >? nums[middle_by_two] >> > >> > [5, 7, 9] >> > >> > The same slice will work on anything else: >> > >> >? list('abcdefghijklmnopqrstuvwxyz')[middle_by_two] >> > ['f', 'h', 'j'] >> > >> > So although you may think the slice is bound to something, it is not. It is an object that only later is briefly connected to whatever you want to apply it to. >> > >> > If I later change nums, above, like this: >> > >> >? nums = [-3, -2, -1] + nums >> >? nums >> > [-3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] >> >? nums[middle_by_two] >> > [2, 4, 6] >> > >> > In the example, you can forget about whether we are talking about pointers directly or indirectly or variable names and so on. Your "view" remains valid ONLY as long as you do not change either the slice or the underlying object you are applying to -- at least not the items you want to extract. >> > >> > Since my example inserted three new items at the start using negative numbers for illustration, you would need to adjust the slice by making a new slice designed to fit your new data. The example below created an adjusted slice that adds 3 to the start and stop settings of the previous slice while copying the step value and then it works on the elongated object: >> > >> >? middle_by_two_adj = slice(middle_by_two.start + 3, middle_by_two.stop + 3, middle_by_two.step) >> >? nums[middle_by_two_adj] >> > [5, 7, 9] >> > >> > A suggestion is? that whenever you are not absolutely sure that the contents of some data structure might change without your participation, then don't depend on various kinds of aliases to keep the contents synchronized. Make a copy, perhaps? a deep copy and make sure the only thing ever changing it is your code and later, if needed, copy the result back to any other data structure. Of course, if anything else is accessing the result in the original in between, it won't work. >> > >> > Just FYI, a similar analysis applies to uses of the numpy and pandas and other modules if you get some kind of object holding indices to a series such as integers or Booleans and then later try using it after the number of items or rows or columns have changed. Your indices no longer match. >> > >> > Avi >> > >> > -----Original Message----- >> > From: Python-list > gmail.com at python.org>> > On Behalf Of Jen Kris via Python-list >> > Sent: Wednesday, January 11, 2023 1:29 PM >> > To: Roel Schroeven <>> roel at roelschroeven.net>> > >> > Cc: >> python-list at python.org >> > Subject: Re: To clarify how Python handles two equal objects >> > >> > Thanks for your comments.? After all, I asked for clarity so it?s not pedantic to be precise, and you?re helping to clarify. >> > >> > Going back to my original post, >> > >> > mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ] >> > arr1 = mx1[2] >> > >> > Now if I write "arr1[1] += 5" then both arr1 and mx1[2][1] will be changed because while they are different names, they are the assigned same memory location (pointer).? Similarly, if I write "mx1[2][1] += 5" then again both names will be updated. >> > >> > That?s what I meant by "an operation on one is an operation on the other."? To be more precise, an operation on one name will be reflected in the other name.? The difference is in the names,? not the pointers.? Each name has the same pointer in my example, but operations can be done in Python using either name. >> > >> > >> > >> > >> > Jan 11, 2023, 09:13 by >> roel at roelschroeven.net>> : >> > >> >> Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list: >> >> >> >>> Yes, I did understand that.? In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they?re the same memory block). >> >>> >> >> >> >> Sorry if you feel I'm being overly pedantic, but your explanation "an operation on one is an operation on the other (because they?re the same memory block)" still feels a bit misguided. "One" and "other" still make it sound like there are two objects, and "an operation on one" and "an operation on the other" make it sound like there are two operations. >> >> Sometimes it doesn't matter if we're a bit sloppy for sake of simplicity or convenience, sometimes we really need to be precise. I think this is a case where we need to be precise. >> >> >> >> So, to be precise: there is only one object, with possible multiple names to it. We can change the object, using one of the names. That is one and only one operation on one and only one object. Since the different names refer to the same object, that change will of course be visible through all of them. >> >> Note that 'name' in that sentence doesn't just refer to variables (mx1, arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...), loop variables, function arguments. >> >> >> >> The correct mental model is important here, and I do think you're on track or very close to it, but the way you phrase things does give me that nagging feeling that you still might be just a bit off. >> >> >> >> -- >> >> "Peace cannot be kept by force. It can only be achieved through understanding." >> >>? -- Albert Einstein >> >> >> >> -- >> >> >> https://mail.python.org/mailman/listinfo/python-list >> >> >> > >> > -- >> > >> https://mail.python.org/mailman/listinfo/python-list >> > >> >> -- >> >> https://mail.python.org/mailman/listinfo/python-list >> > > > -- > > **** Listen to my FREE CD at > http://www.mellowood.ca/music/cedars> **** > Bob van der Poel ** Wynndel, British Columbia, CANADA ** > EMAIL: > bob at mellowood.ca > WWW: ? > http://www.mellowood.ca > From hjp-python at hjp.at Fri Jan 13 14:38:49 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Fri, 13 Jan 2023 20:38:49 +0100 Subject: To clarify how Python handles two equal objects In-Reply-To: References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> <00d801d9270a$a2597c30$e70c7490$@gmail.com> Message-ID: <20230113193849.axvpjhzrkb5pizi2@hjp.at> On 2023-01-13 16:57:45 +0100, Jen Kris via Python-list wrote: > Thanks for your comments.? You make a good point.? > > Going back to my original question, and using your slice() example:? > > middle_by_two = slice(5, 10, 2) > nums = [n for n in range(12)] > q = nums[middle_by_two] > x = id(q) > b = q > y = id(b) > > If I assign "b" to "q", You don't asssign b to q. You assign q to b. Assignment is not commutative, the direction matters. > then x and y match ? they point to the same memory until "b" OR "q" > are? reassigned to something else. Correct. >? If "q" changes during the lifetime of "b" then it?s not safe to use >the pointer to "q" for "b", as in: There is no pointer to q[1]. q is a pointer to something (an object of type list with 3 elements). b is a pointer to the same object at this point. Of course, if you assign a different pointer to q, then q will point to the new object from that point on while b will continue to point to the original object (until it is also re-assigned). > nums = [n for n in range(2, 14)] > q = nums[middle_by_two] > x = id(q) > y = id(b) > > Now "x" and "y" are different, as we would expect. > So when writing a spot speed up in a compiled language, If you are writing a Python compiler you certainly will have to adjust for the fact that the same variable may hold pointers to very different objects over its lifetime. But as far as I understand it, you aren't trying to write a Python compiler, just an extension, so that shouldn't concern you. > The motivation behind my question was that in a compiled extension > it?s faster to borrow a pointer than to move an entire array if it?s > possible, but special care must be taken.? I still don't understand what you mean by that. hp [1] Not quite true. The run-time system must keep track of the variables, so there likely is a pointer to q somewhere. But it's (IMHO) irrelevant to this discussion, unless the extension is trying to look up variables by name or something like that. -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From hjp-python at hjp.at Fri Jan 13 14:46:57 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Fri, 13 Jan 2023 20:46:57 +0100 Subject: The Zen of D.E.K. In-Reply-To: References: Message-ID: <20230113194657.ofbbymosgcmmj5lk@hjp.at> On 2023-01-13 10:00:01 -0800, Ethan Furman wrote: > On 1/13/23 09:06, Stefan Ram wrote: > > "Beautiful is better than ugly." - The Zen of Python > > > > This says nothing. You have to sacrifice something that > > really has /value/! Time? Making something beautiful takes time. > > "[A]esthetics are more important than efficiency." - Donald E. Knuth > > [okay, falling for the troll bait] > > Those two things do not say the same thing; further, in Python at least, and > depending on the situation, aesthetics may /not/ be more important than > efficiency. This is true independent of language. Aesthetics /may/ be more important than efficiency. Aesthetics /may/ not be more important than efficiency. It depends on the application. Although I would deposit that if you care about run-time efficiency (as opposed to developer-time efficiency), Python might not be your language of choice at least for the part where efficiency is crucial. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From avi.e.gross at gmail.com Fri Jan 13 21:41:05 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Fri, 13 Jan 2023 21:41:05 -0500 Subject: To clarify how Python handles two equal objects In-Reply-To: References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> Message-ID: <00a801d927c1$a6b80730$f4281590$@gmail.com> Jen, This may not be on target but I was wondering about your needs in this category. Are all your data in a form where all in a cluster are the same object type, such as floating point? Python has features designed to allow you to get multiple views on such objects such as memoryview that can be used to say see an array as a matrix of n rows by m columns, or m x n, or any other combo. And of course the fuller numpy package has quite a few features. However, as you note, there is no guarantee that any reference to the data may not shift away from it unless you build fairly convoluted logic or data structures such as having an object that arranges to do something when you try to remove it, such as tinkering with the __del__ method as well as whatever method is used to try to set it to a new value. I guess that might make sense for something like asynchronous programming including when setting locks so multiple things cannot overlap when being done. Anyway, some of the packages like numpy are optimized in many ways but if you want to pass a subset of sorts to make processing faster, I suspect you could do things like pass a memoryview but it might not be faster than what you build albeit probably more reliable and portable. I note another odd idea that others may have mentioned, with caution. If you load the sys module, you can CAREFULLY use code like this. a="Something Unique" sys.getrefcount(a) 2 Note if a==1 you will get some huge number of references and this is meaningless. The 2 above is because asking about how many references also references it. So save what ever number you have and see what happens when you make a second reference or a third, and what happens if you delete or alter a reference: a="Something Unique" sys.getrefcount(a) 2 b = a sys.getrefcount(a) 3 sys.getrefcount(b) 3 c = b d = a sys.getrefcount(a) 5 sys.getrefcount(d) 5 del(a) sys.getrefcount(d) 4 b = "something else" sys.getrefcount(d) 3 So, in theory, you could carefully write your code to CHECK the reference count had not changed but there remain edge cases where a removed reference is replaced by yet another new reference and you would have no idea. Avi -----Original Message----- From: Python-list On Behalf Of Jen Kris via Python-list Sent: Wednesday, January 11, 2023 1:29 PM To: Roel Schroeven Cc: python-list at python.org Subject: Re: To clarify how Python handles two equal objects Thanks for your comments. After all, I asked for clarity so it?s not pedantic to be precise, and you?re helping to clarify. Going back to my original post, mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ] arr1 = mx1[2] Now if I write "arr1[1] += 5" then both arr1 and mx1[2][1] will be changed because while they are different names, they are the assigned same memory location (pointer). Similarly, if I write "mx1[2][1] += 5" then again both names will be updated. That?s what I meant by "an operation on one is an operation on the other." To be more precise, an operation on one name will be reflected in the other name. The difference is in the names, not the pointers. Each name has the same pointer in my example, but operations can be done in Python using either name. Jan 11, 2023, 09:13 by roel at roelschroeven.net: > Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list: > >> Yes, I did understand that. In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they?re the same memory block). >> > > Sorry if you feel I'm being overly pedantic, but your explanation "an operation on one is an operation on the other (because they?re the same memory block)" still feels a bit misguided. "One" and "other" still make it sound like there are two objects, and "an operation on one" and "an operation on the other" make it sound like there are two operations. > Sometimes it doesn't matter if we're a bit sloppy for sake of simplicity or convenience, sometimes we really need to be precise. I think this is a case where we need to be precise. > > So, to be precise: there is only one object, with possible multiple names to it. We can change the object, using one of the names. That is one and only one operation on one and only one object. Since the different names refer to the same object, that change will of course be visible through all of them. > Note that 'name' in that sentence doesn't just refer to variables (mx1, arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...), loop variables, function arguments. > > The correct mental model is important here, and I do think you're on track or very close to it, but the way you phrase things does give me that nagging feeling that you still might be just a bit off. > > -- > "Peace cannot be kept by force. It can only be achieved through understanding." > -- Albert Einstein > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list From avi.e.gross at gmail.com Fri Jan 13 21:56:48 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Fri, 13 Jan 2023 21:56:48 -0500 Subject: To clarify how Python handles two equal objects In-Reply-To: References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> <00d801d9270a$a2597c30$e70c7490$@gmail.com> Message-ID: <00c001d927c3$d88b15c0$89a14140$@gmail.com> Jen, Can a compiler, or spot compiler, always know if something that was a reference is changed? Obvious examples may be if a change happens in non-deterministic ways such as in a fork chosen at random or from user input but also sometimes levels of indirection such as deleting an object that internally contains a reference the other, perhaps even more indirectly. I know programmers often have their code overhauled and their assumptions go away such as someone deciding to create a variable name in an inner scope and thus hiding the same variable they were using in an outer scope. So even if your code is currently valid, after it changes, a later compiler if it detected some change might not want to do your speedup. I think the subject line of the message we keep exchanging is now a bit misleading. It is not about two objects nor really about how python handles them. There seem to be one object and possibly multiple views of it and you may not want to pass the entire object around or manipulate it a certain way. I am not so certain your methods necessarily speed things up as certain views simply do calculations on the many places they need to read or change to supply what you want. From: Jen Kris Sent: Friday, January 13, 2023 10:58 AM To: avi.e.gross at gmail.com Cc: python-list at python.org Subject: RE: To clarify how Python handles two equal objects Avi, Thanks for your comments. You make a good point. Going back to my original question, and using your slice() example: middle_by_two = slice(5, 10, 2) nums = [n for n in range(12)] q = nums[middle_by_two] x = id(q) b = q y = id(b) If I assign "b" to "q", then x and y match ? they point to the same memory until "b" OR "q" are reassigned to something else. If "q" changes during the lifetime of "b" then it?s not safe to use the pointer to "q" for "b", as in: nums = [n for n in range(2, 14)] q = nums[middle_by_two] x = id(q) y = id(b) Now "x" and "y" are different, as we would expect. So when writing a spot speed up in a compiled language, you can see in the Python source if either is reassigned, so you?ll know how to handle it. The motivation behind my question was that in a compiled extension it?s faster to borrow a pointer than to move an entire array if it?s possible, but special care must be taken. Jen Jan 12, 2023, 20:51 by avi.e.gross at gmail.com : Jen, It is dangerous territory you are treading as there are times all or parts of objects are copied, or changed in place or the method you use to make a view is not doing quite what you want. As an example, you can create a named slice such as: middle_by_two = slice(5, 10, 2) The above is not in any sense pointing at anything yet. But given a long enough list or other such objects, it will take items (starting at index 0) starting with item that are at indices 5 then 7 then 9 as in this: nums = [n for n in range(12)] nums[middle_by_two] [5, 7, 9] The same slice will work on anything else: list('abcdefghijklmnopqrstuvwxyz')[middle_by_two] ['f', 'h', 'j'] So although you may think the slice is bound to something, it is not. It is an object that only later is briefly connected to whatever you want to apply it to. If I later change nums, above, like this: nums = [-3, -2, -1] + nums nums [-3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] nums[middle_by_two] [2, 4, 6] In the example, you can forget about whether we are talking about pointers directly or indirectly or variable names and so on. Your "view" remains valid ONLY as long as you do not change either the slice or the underlying object you are applying to -- at least not the items you want to extract. Since my example inserted three new items at the start using negative numbers for illustration, you would need to adjust the slice by making a new slice designed to fit your new data. The example below created an adjusted slice that adds 3 to the start and stop settings of the previous slice while copying the step value and then it works on the elongated object: middle_by_two_adj = slice(middle_by_two.start + 3, middle_by_two.stop + 3, middle_by_two.step) nums[middle_by_two_adj] [5, 7, 9] A suggestion is that whenever you are not absolutely sure that the contents of some data structure might change without your participation, then don't depend on various kinds of aliases to keep the contents synchronized. Make a copy, perhaps a deep copy and make sure the only thing ever changing it is your code and later, if needed, copy the result back to any other data structure. Of course, if anything else is accessing the result in the original in between, it won't work. Just FYI, a similar analysis applies to uses of the numpy and pandas and other modules if you get some kind of object holding indices to a series such as integers or Booleans and then later try using it after the number of items or rows or columns have changed. Your indices no longer match. Avi -----Original Message----- From: Python-list > On Behalf Of Jen Kris via Python-list Sent: Wednesday, January 11, 2023 1:29 PM To: Roel Schroeven > Cc: python-list at python.org Subject: Re: To clarify how Python handles two equal objects Thanks for your comments. After all, I asked for clarity so it?s not pedantic to be precise, and you?re helping to clarify. Going back to my original post, mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ] arr1 = mx1[2] Now if I write "arr1[1] += 5" then both arr1 and mx1[2][1] will be changed because while they are different names, they are the assigned same memory location (pointer). Similarly, if I write "mx1[2][1] += 5" then again both names will be updated. That?s what I meant by "an operation on one is an operation on the other." To be more precise, an operation on one name will be reflected in the other name. The difference is in the names, not the pointers. Each name has the same pointer in my example, but operations can be done in Python using either name. Jan 11, 2023, 09:13 by roel at roelschroeven.net : Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list: Yes, I did understand that. In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they?re the same memory block). Sorry if you feel I'm being overly pedantic, but your explanation "an operation on one is an operation on the other (because they?re the same memory block)" still feels a bit misguided. "One" and "other" still make it sound like there are two objects, and "an operation on one" and "an operation on the other" make it sound like there are two operations. Sometimes it doesn't matter if we're a bit sloppy for sake of simplicity or convenience, sometimes we really need to be precise. I think this is a case where we need to be precise. So, to be precise: there is only one object, with possible multiple names to it. We can change the object, using one of the names. That is one and only one operation on one and only one object. Since the different names refer to the same object, that change will of course be visible through all of them. Note that 'name' in that sentence doesn't just refer to variables (mx1, arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...), loop variables, function arguments. The correct mental model is important here, and I do think you're on track or very close to it, but the way you phrase things does give me that nagging feeling that you still might be just a bit off. -- "Peace cannot be kept by force. It can only be achieved through understanding." -- Albert Einstein -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list From avi.e.gross at gmail.com Fri Jan 13 22:12:37 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Fri, 13 Jan 2023 22:12:37 -0500 Subject: To clarify how Python handles two equal objects In-Reply-To: <875ydax2sf.fsf@axel-reichert.de> References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> <00d801d9270a$a2597c30$e70c7490$@gmail.com> <875ydax2sf.fsf@axel-reichert.de> Message-ID: <00d701d927c6$0e702a20$2b507e60$@gmail.com> Axel and others, I can appreciate the comparison to a partially applied function but not in this case. Not that it matters, but this example is more like creating an object in something like machine learning and initializing parameters without adding data. Only when you ad data and call upon some transforms and so on, does it do something. This case is even more general. You create an object that does NOTHING. It simply holds a start/end/step set of up to three values. Lots of other functions will take this object as an argument. It can be used and reused any number of times. Strictly speaking, code like name[5:10:1] just creates a transient slice object and then uses that to get the answer. It is not delayed or partial as much as making one does nothing. Stefan mentioned functools.partial and that does create a bit of a curried function that wraps the data and holds on to it so invoking it sort of wakes the function up, with some or all data already accessible. A slice does not do that and needs some other functionality to use IT alongside whatever object you want to see a slice of. No special behavior was intended by me. I was illustrating how some methods of providing a selected view of an object are equally sensitive to the underlying data changing. A partially applied function that still takes an argument later, would have a similar problem if underlying data outside the what is stored within the function, changed, or if the saved was a reference to something that changed. But this is really far from unique. In the example given of creating a partial call, what if you made a second copy to that call then the first variable to the partial function was re-defined. -----Original Message----- From: Python-list On Behalf Of Axel Reichert Sent: Friday, January 13, 2023 3:22 AM To: python-list at python.org Subject: Re: To clarify how Python handles two equal objects writes: > As an example, you can create a named slice such as: > > middle_by_two = slice(5, 10, 2) > > The above is not in any sense pointing at anything yet. >From a functional programming point of view this just looks like a partially applied function, and with this in mind the behaviour to me seems to be completely as expected. No surprises here, or do I miss something? Best regards Axel -- https://mail.python.org/mailman/listinfo/python-list From avi.e.gross at gmail.com Sat Jan 14 12:16:57 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Sat, 14 Jan 2023 12:16:57 -0500 Subject: The Zen of D.E.K. In-Reply-To: References: Message-ID: <003f01d9283c$02331030$06993090$@gmail.com> I can appreciate a beautiful piece of code but I can also appreciate another piece of code that does things in another pleasing way so there is quite a bit of subjectivity here. And, in yet another computer language, the implementation of what seems to be the same algorithm is somewhat jarring as it does not quite fit the environment. Some people consider the symmetry of a language that ends an IF statement with FI to be sort of pleasing. Others feel that way about matched opposing braces and yet others like having things the same symbol such as an unadorned double quote or slash to be both the beginning and end. It goes way deeper than that but I think there is plenty of subjectivity in what people find pleasing. Some adore it if an algorithm is a very curt and mysterious one-liner while others like when code is lined up just so on multiple lines, perhaps using a nice color scheme in their editor. Some adore copious detailed comments while others find they get in the way. Efficiency is another matter but again has some subjectivity and variations. The same algorithm can be much more efficient in one language/implementation than another but also in other ways can be less. If an algorithm must sort a billion items, the algorithm may dominate the resources used. But to sort a small number of items, the overhead of invoking and loading an external module that has a faster method than the built-in way, may be much slower if used only once. In the real world, there are other candidates for what is in some sense better to do. One example is how fast it can be designed and implemented and another might be if it tends to generate fewer bugs and glitches. A big one is if it saves the company money in creating and maintaining it or at runtime. And, of course, a good algorithm implementation is one that others, perhaps less extremely educated than you, can later read your code and understand it well enough to modify it, or perhaps port it to another language with different ways than the one you wrote it in. Efficiency keeps being relative as languages evolve. A change in the interpreter may add features that end up making the feature you chose to slow a bit. Replacing some functionality with a version written in a language like C that is compiled, can often speed it up. Changing an algorithm from using a list to a numpy array can have dramatic differences even as the skeleton of the algorithm remains the same in terms of aesthetics. Amusingly, I have been reading about ideas of Aesthetics and sort of beauty by Mathematicians and Physicists in how it guides them in their work. Knuth and others in C.S. are arguably doing similar things. -----Original Message----- From: Python-list On Behalf Of Ethan Furman Sent: Friday, January 13, 2023 1:00 PM To: python-list at python.org Subject: Re: The Zen of D.E.K. On 1/13/23 09:06, Stefan Ram wrote: > "Beautiful is better than ugly." - The Zen of Python > > This says nothing. You have to sacrifice something that > really has /value/! > > "[A]esthetics are more important than efficiency." - Donald E. Knuth [okay, falling for the troll bait] Those two things do not say the same thing; further, in Python at least, and depending on the situation, aesthetics may /not/ be more important than efficiency. -- ~Ethan~ -- https://mail.python.org/mailman/listinfo/python-list From jenkris at tutanota.com Sat Jan 14 18:30:48 2023 From: jenkris at tutanota.com (Jen Kris) Date: Sun, 15 Jan 2023 00:30:48 +0100 (CET) Subject: To clarify how Python handles two equal objects In-Reply-To: <00a801d927c1$a6b80730$f4281590$@gmail.com> References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> <00a801d927c1$a6b80730$f4281590$@gmail.com> Message-ID: Avi, Your comments go farther afield than my original question, but you made some interesting additional points.? For example, I sometimes work with the C API and sys.getrefcount may be helpful in deciding when to INCREF and DECREF.? But that?s another issue.? The situation I described in my original post is limited to a case such as x = y where both "x" and "y" are arrays ? whether they are lists in Python, or from the array module ? and the question in a compiled C extension is whether the assignment can be done simply by "x" taking the pointer to "y" rather than moving all the data from "y" into the memory buffer for "x" which, for a wide array, would be much more time consuming than just moving a pointer.? The other advantage to doing it that way is if, as in my case, we perform a math operation on any element in "x" then Python expects that the same change to be reflected in "y."? If I don?t use the same pointers then I would have to perform that operation twice ? once for "x" and once? for "y" ? in addition to the expense of moving all the data.? The answers I got from this post confirmed that it I can use the pointer if "y" is not re-defined to something else during the lifespan of "x."? If it is then "x" has to be restored to its original pointer.? I did it that way, and helpfully the compiler did not overrule me.? Jan 13, 2023, 18:41 by avi.e.gross at gmail.com: > Jen, > > This may not be on target but I was wondering about your needs in this category. Are all your data in a form where all in a cluster are the same object type, such as floating point? > > Python has features designed to allow you to get multiple views on such objects such as memoryview that can be used to say see an array as a matrix of n rows by m columns, or m x n, or any other combo. And of course the fuller numpy package has quite a few features. > > However, as you note, there is no guarantee that any reference to the data may not shift away from it unless you build fairly convoluted logic or data structures such as having an object that arranges to do something when you try to remove it, such as tinkering with the __del__ method as well as whatever method is used to try to set it to a new value. I guess that might make sense for something like asynchronous programming including when setting locks so multiple things cannot overlap when being done. > > Anyway, some of the packages like numpy are optimized in many ways but if you want to pass a subset of sorts to make processing faster, I suspect you could do things like pass a memoryview but it might not be faster than what you build albeit probably more reliable and portable. > > I note another odd idea that others may have mentioned, with caution. > > If you load the sys module, you can CAREFULLY use code like this. > > a="Something Unique" > sys.getrefcount(a) > 2 > > Note if a==1 you will get some huge number of references and this is meaningless. The 2 above is because asking about how many references also references it. > > So save what ever number you have and see what happens when you make a second reference or a third, and what happens if you delete or alter a reference: > > a="Something Unique" > sys.getrefcount(a) > 2 > b = a > sys.getrefcount(a) > 3 > sys.getrefcount(b) > 3 > c = b > d = a > sys.getrefcount(a) > 5 > sys.getrefcount(d) > 5 > del(a) > sys.getrefcount(d) > 4 > b = "something else" > sys.getrefcount(d) > 3 > > So, in theory, you could carefully write your code to CHECK the reference count had not changed but there remain edge cases where a removed reference is replaced by yet another new reference and you would have no idea. > > Avi > > > -----Original Message----- > From: Python-list On Behalf Of Jen Kris via Python-list > Sent: Wednesday, January 11, 2023 1:29 PM > To: Roel Schroeven > Cc: python-list at python.org > Subject: Re: To clarify how Python handles two equal objects > > Thanks for your comments. After all, I asked for clarity so it?s not pedantic to be precise, and you?re helping to clarify. > > Going back to my original post, > > mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ] > arr1 = mx1[2] > > Now if I write "arr1[1] += 5" then both arr1 and mx1[2][1] will be changed because while they are different names, they are the assigned same memory location (pointer). Similarly, if I write "mx1[2][1] += 5" then again both names will be updated. > > That?s what I meant by "an operation on one is an operation on the other." To be more precise, an operation on one name will be reflected in the other name. The difference is in the names, not the pointers. Each name has the same pointer in my example, but operations can be done in Python using either name. > > > > > Jan 11, 2023, 09:13 by roel at roelschroeven.net: > >> Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list: >> >>> Yes, I did understand that. In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they?re the same memory block). >>> >> >> Sorry if you feel I'm being overly pedantic, but your explanation "an operation on one is an operation on the other (because they?re the same memory block)" still feels a bit misguided. "One" and "other" still make it sound like there are two objects, and "an operation on one" and "an operation on the other" make it sound like there are two operations. >> Sometimes it doesn't matter if we're a bit sloppy for sake of simplicity or convenience, sometimes we really need to be precise. I think this is a case where we need to be precise. >> >> So, to be precise: there is only one object, with possible multiple names to it. We can change the object, using one of the names. That is one and only one operation on one and only one object. Since the different names refer to the same object, that change will of course be visible through all of them. >> Note that 'name' in that sentence doesn't just refer to variables (mx1, arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...), loop variables, function arguments. >> >> The correct mental model is important here, and I do think you're on track or very close to it, but the way you phrase things does give me that nagging feeling that you still might be just a bit off. >> >> -- >> "Peace cannot be kept by force. It can only be achieved through understanding." >> -- Albert Einstein >> >> -- >> https://mail.python.org/mailman/listinfo/python-list >> > > -- > https://mail.python.org/mailman/listinfo/python-list > From rosuav at gmail.com Sat Jan 14 18:51:30 2023 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 15 Jan 2023 10:51:30 +1100 Subject: To clarify how Python handles two equal objects In-Reply-To: References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> <00a801d927c1$a6b80730$f4281590$@gmail.com> Message-ID: On Sun, 15 Jan 2023 at 10:32, Jen Kris via Python-list wrote: > The situation I described in my original post is limited to a case such as x = y ... the assignment can be done simply by "x" taking the pointer to "y" rather than moving all the data from "y" into the memory buffer for "x" > It's not simply whether it *can* be done. It, in fact, *MUST* be done that way. The ONLY meaning of "x = y" is that you now have a name "x" which refers to whatever object is currently found under the name "y". This is not an optimization, it is a fundamental of Python's object model. This is true regardless of what kind of object this is; every object must behave this way. ChrisA From jenkris at tutanota.com Sat Jan 14 19:38:20 2023 From: jenkris at tutanota.com (Jen Kris) Date: Sun, 15 Jan 2023 01:38:20 +0100 (CET) Subject: To clarify how Python handles two equal objects In-Reply-To: References: <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> <00a801d927c1$a6b80730$f4281590$@gmail.com> Message-ID: Yes, in fact I asked my original question ? "I discovered something about Python array handling that I would like to clarify" -- because I saw that Python did it that way.? Jan 14, 2023, 15:51 by rosuav at gmail.com: > On Sun, 15 Jan 2023 at 10:32, Jen Kris via Python-list > wrote: > >> The situation I described in my original post is limited to a case such as x = y ... the assignment can be done simply by "x" taking the pointer to "y" rather than moving all the data from "y" into the memory buffer for "x" >> > > It's not simply whether it *can* be done. It, in fact, *MUST* be done > that way. The ONLY meaning of "x = y" is that you now have a name "x" > which refers to whatever object is currently found under the name "y". > This is not an optimization, it is a fundamental of Python's object > model. This is true regardless of what kind of object this is; every > object must behave this way. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > From rosuav at gmail.com Sat Jan 14 19:41:12 2023 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 15 Jan 2023 11:41:12 +1100 Subject: To clarify how Python handles two equal objects In-Reply-To: References: <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> <00a801d927c1$a6b80730$f4281590$@gmail.com> Message-ID: On Sun, 15 Jan 2023 at 11:38, Jen Kris wrote: > > Yes, in fact I asked my original question ? "I discovered something about Python array handling that I would like to clarify" -- because I saw that Python did it that way. > Yep. This is not specific to arrays; it is true of all Python objects. Also, I suspect you're still thinking about things backwards, and am trying to lead you to a completely different way of thinking that actually does align with Python's object model. ChrisA From roel at roelschroeven.net Sat Jan 14 21:36:29 2023 From: roel at roelschroeven.net (Roel Schroeven) Date: Sun, 15 Jan 2023 03:36:29 +0100 Subject: To clarify how Python handles two equal objects In-Reply-To: References: <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> <00a801d927c1$a6b80730$f4281590$@gmail.com> Message-ID: <415e4a55-2ce6-f4fc-7c2e-df4fd9e1e301@roelschroeven.net> Chris Angelico schreef op 15/01/2023 om 1:41: > On Sun, 15 Jan 2023 at 11:38, Jen Kris wrote: > > > > Yes, in fact I asked my original question ? "I discovered something about Python array handling that I would like to clarify" -- because I saw that Python did it that way. > > > > Yep. This is not specific to arrays; it is true of all Python objects. > Also, I suspect you're still thinking about things backwards, and am > trying to lead you to a completely different way of thinking that > actually does align with Python's object model. Indeen, I also still have the impression that Jen is thinking in terms of variables that are possible aliased such as you can have in a language like C, instead of objects with one or more names like we have in Python. Jens, in the Python model you really have to think of the objects largely independently of the names that are or are not referencing the objects. -- "Ever since I learned about confirmation bias, I've been seeing it everywhere." -- Jon Ronson From frank at chagford.com Sun Jan 15 00:46:44 2023 From: frank at chagford.com (Frank Millman) Date: Sun, 15 Jan 2023 07:46:44 +0200 Subject: To clarify how Python handles two equal objects In-Reply-To: <415e4a55-2ce6-f4fc-7c2e-df4fd9e1e301@roelschroeven.net> References: <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> <00a801d927c1$a6b80730$f4281590$@gmail.com> <415e4a55-2ce6-f4fc-7c2e-df4fd9e1e301@roelschroeven.net> Message-ID: <2f398a79-f6e7-fa75-4462-e7ba838b375b@chagford.com> On 2023-01-15 4:36 AM, Roel Schroeven wrote: > > > Chris Angelico schreef op 15/01/2023 om 1:41: >> On Sun, 15 Jan 2023 at 11:38, Jen Kris wrote: >> > >> > Yes, in fact I asked my original question ? "I discovered something >> about Python array handling that I would like to clarify" -- because I >> saw that Python did it that way. >> > >> >> Yep. This is not specific to arrays; it is true of all Python objects. >> Also, I suspect you're still thinking about things backwards, and am >> trying to lead you to a completely different way of thinking that >> actually does align with Python's object model. > Indeen, I also still have the impression that Jen is thinking in terms > of variables that are possible aliased such as you can have in a > language like C, instead of objects with one or more names like we have > in Python. Jens, in the Python model you really have to think of the > objects largely independently of the names that are or are not > referencing the objects. > My 'aha' moment came when I understood that a python object has only three properties - a type, an id, and a value. It does *not* have a name. Frank Millman From dino at no.spam.ar Sat Jan 14 23:26:27 2023 From: dino at no.spam.ar (Dino) Date: Sat, 14 Jan 2023 23:26:27 -0500 Subject: Fast lookup of bulky "table" Message-ID: Hello, I have built a PoC service in Python Flask for my work, and - now that the point is made - I need to make it a little more performant (to be honest, chances are that someone else will pick up from where I left off, and implement the same service from scratch in a different language (GoLang? .Net? Java?) but I am digressing). Anyway, my Flask service initializes by loading a big "table" of 100k rows and 40 columns or so (memory footprint: order of 300 Mb) and then accepts queries through a REST endpoint. Columns are strings, enums, and numbers. Once initialized, the table is read only. The endpoint will parse the query and match it against column values (equality, inequality, greater than, etc.) Finally, it will return a (JSON) list of all rows that satisfy all conditions in the query. As you can imagine, this is not very performant in its current form, but performance was not the point of the PoC - at least initially. Before I deliver the PoC to a more experienced software architect who will look at my code, though, I wouldn't mind to look a bit less lame and do something about performance in my own code first, possibly by bringing the average time for queries down from where it is now (order of 1 to 4 seconds per query on my laptop) to 1 or 2 milliseconds on average). To be honest, I was already able to bring the time down to a handful of microseconds thanks to a rudimentary cache that will associate the "signature" of a query to its result, and serve it the next time the same query is received, but this may not be good enough: 1) queries might be many and very different from one another each time, AND 2) I am not sure the server will have a ton of RAM if/when this thing - or whatever is derived from it - is placed into production. How can I make my queries generally more performant, ideally also in case of a new query? Here's what I have been considering: 1. making my cache more "modular", i.e. cache the result of certain (wide) queries. When a complex query comes in, I may be able to restrict my search to a subset of the rows (as determined by a previously cached partial query). This should keep the memory footprint under control. 2. Load my data into a numpy.array and use numpy.array operations to slice and dice my data. 3. load my data into sqlite3 and use SELECT statement to query my table. I have never used sqllite, plus there's some extra complexity as comparing certain colum requires custom logic, but I wonder if this architecture would work well also when dealing with a 300Mb database. 4. Other ideas? Hopefully I made sense. Thank you for your attention Dino From lal at solute.de Sun Jan 15 03:17:03 2023 From: lal at solute.de (Lars Liedtke) Date: Sun, 15 Jan 2023 09:17:03 +0100 Subject: Fast lookup of bulky "table" In-Reply-To: References: Message-ID: <68e30710-b177-2301-4228-6b3216c1145b@solute.de> Hey, before you start optimizing. I would suggest, that you measure response times and query times, data search times and so on. In order to save time, you have to know where you "loose" time. Does your service really have to load the whole table at once? Yes that might lead to quicker response times on requests, but databases are often very good with caching themselves, so that the first request might be slower than following requests, with similar parameters. Do you use a database, or are you reading from a file? Are you maybe looping through your whole dataset on every request? Instead of asking for the specific data? Before you start introducing a cache and its added complexity, do you really need that cache? You are talking about saving microseconds, that sounds a bit as if you might be ?overdoing? it. How many requests will you have in the future? At least in which magnitude and how quick do they have to be? You write about 1-4 seconds on your laptop. But that does not really tell you that much, because most probably the service will run on a server. I am not saying that you should get a server or a cloud-instance to test against, but to talk with your architect about that. I totally understand your impulse to appear as good as can be, but you have to know where you really need to debug and optimize. It will not be advantageous for you, if you start to optimize for optimizing's sake. Additionally if you service is a PoC, optimizing now might be not the first thing you have to worry about, but about that you made everything as simple and readable as possible and that you do not spend too much time for just showing how it could work. But of course, I do not know the tasks given to you and the expectations you have to fulfil. All I am trying to say is to reconsider where you really could improve and how far you have to improve. Lars Liedtke Software Entwickler [Tel.] +49 721 98993- [Fax] +49 721 98993- [E-Mail] lal at solute.de solute GmbH Zeppelinstra?e 15 76185 Karlsruhe Germany [Logo Solute] Marken der solute GmbH | brands of solute GmbH [Marken] [Advertising Partner] Gesch?ftsf?hrer | Managing Director: Dr. Thilo Gans, Bernd Vermaaten Webseite | www.solute.de Sitz | Registered Office: Karlsruhe Registergericht | Register Court: Amtsgericht Mannheim Registernummer | Register No.: HRB 110579 USt-ID | VAT ID: DE234663798 Informationen zum Datenschutz | Information about privacy policy https://www.solute.de/ger/datenschutz/grundsaetze-der-datenverarbeitung.php Am 15.01.23 um 05:26 schrieb Dino: Hello, I have built a PoC service in Python Flask for my work, and - now that the point is made - I need to make it a little more performant (to be honest, chances are that someone else will pick up from where I left off, and implement the same service from scratch in a different language (GoLang? .Net? Java?) but I am digressing). Anyway, my Flask service initializes by loading a big "table" of 100k rows and 40 columns or so (memory footprint: order of 300 Mb) and then accepts queries through a REST endpoint. Columns are strings, enums, and numbers. Once initialized, the table is read only. The endpoint will parse the query and match it against column values (equality, inequality, greater than, etc.) Finally, it will return a (JSON) list of all rows that satisfy all conditions in the query. As you can imagine, this is not very performant in its current form, but performance was not the point of the PoC - at least initially. Before I deliver the PoC to a more experienced software architect who will look at my code, though, I wouldn't mind to look a bit less lame and do something about performance in my own code first, possibly by bringing the average time for queries down from where it is now (order of 1 to 4 seconds per query on my laptop) to 1 or 2 milliseconds on average). To be honest, I was already able to bring the time down to a handful of microseconds thanks to a rudimentary cache that will associate the "signature" of a query to its result, and serve it the next time the same query is received, but this may not be good enough: 1) queries might be many and very different from one another each time, AND 2) I am not sure the server will have a ton of RAM if/when this thing - or whatever is derived from it - is placed into production. How can I make my queries generally more performant, ideally also in case of a new query? Here's what I have been considering: 1. making my cache more "modular", i.e. cache the result of certain (wide) queries. When a complex query comes in, I may be able to restrict my search to a subset of the rows (as determined by a previously cached partial query). This should keep the memory footprint under control. 2. Load my data into a numpy.array and use numpy.array operations to slice and dice my data. 3. load my data into sqlite3 and use SELECT statement to query my table. I have never used sqllite, plus there's some extra complexity as comparing certain colum requires custom logic, but I wonder if this architecture would work well also when dealing with a 300Mb database. 4. Other ideas? Hopefully I made sense. Thank you for your attention Dino From hjp-python at hjp.at Sun Jan 15 06:14:24 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 15 Jan 2023 12:14:24 +0100 Subject: Fast lookup of bulky "table" In-Reply-To: References: Message-ID: <20230115111424.xbd732bkmqumycno@hjp.at> On 2023-01-14 23:26:27 -0500, Dino wrote: > Hello, I have built a PoC service in Python Flask for my work, and - now > that the point is made - I need to make it a little more performant (to be > honest, chances are that someone else will pick up from where I left off, > and implement the same service from scratch in a different language (GoLang? > .Net? Java?) but I am digressing). > > Anyway, my Flask service initializes by loading a big "table" of 100k rows > and 40 columns or so (memory footprint: order of 300 Mb) 300 MB is large enough that you should at least consider putting that into a database (Sqlite is probably simplest. Personally I would go with PostgreSQL because I'm most familiar with it and Sqlite is a bit of an outlier). The main reason for putting it into a database is the ability to use indexes, so you don't have to scan all 100 k rows for each query. You may be able to do that for your Python data structures, too: Can you set up dicts which map to subsets you need often? There are some specialized in-memory bitmap implementations which can be used for filtering. I've used [Judy bitmaps](https://judy.sourceforge.net/doc/Judy1_3x.htm) in the past (mostly in Perl). These days [Roaring Bitmaps](https://www.roaringbitmap.org/) is probably the most popular. I see several packages on PyPI - but I haven't used any of them yet, so no recommendation from me. Numpy might also help. You will still have linear scans, but it is more compact and many of the searches can probably be done in C and not in Python. > As you can imagine, this is not very performant in its current form, but > performance was not the point of the PoC - at least initially. For performanc optimization it is very important to actually measure performance, and a good profiler helps very much in identifying hot spots. Unfortunately until recently Python was a bit deficient in this area, but [Scalene](https://pypi.org/project/scalene/) looks promising. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From list1 at tompassin.net Sun Jan 15 10:38:22 2023 From: list1 at tompassin.net (Thomas Passin) Date: Sun, 15 Jan 2023 10:38:22 -0500 Subject: Fast lookup of bulky "table" In-Reply-To: <20230115111424.xbd732bkmqumycno@hjp.at> References: <20230115111424.xbd732bkmqumycno@hjp.at> Message-ID: <499510f2-c5e9-9ed7-d09e-4207488d87fc@tompassin.net> On 1/15/2023 6:14 AM, Peter J. Holzer wrote: > On 2023-01-14 23:26:27 -0500, Dino wrote: >> Hello, I have built a PoC service in Python Flask for my work, and - now >> that the point is made - I need to make it a little more performant (to be >> honest, chances are that someone else will pick up from where I left off, >> and implement the same service from scratch in a different language (GoLang? >> .Net? Java?) but I am digressing). >> >> Anyway, my Flask service initializes by loading a big "table" of 100k rows >> and 40 columns or so (memory footprint: order of 300 Mb) > > 300 MB is large enough that you should at least consider putting that > into a database (Sqlite is probably simplest. Personally I would go with > PostgreSQL because I'm most familiar with it and Sqlite is a bit of an > outlier). > > The main reason for putting it into a database is the ability to use > indexes, so you don't have to scan all 100 k rows for each query. I have an (inherited) server program that uses about 30 MB of data in a MySQL database. It services queries received over the network. It too had performance problems, to which adding indexes and smarter joins helped but not enough. I changed the program so that at startup it imports much of the data into Python dictionaries that are structured to support the kinds of queries that need the help. Response time to queries dropped dramatically. Some kinds of queries needed more help, and I collected auxiliary collections of (usually highly pre-processed) data into ordinary files, and those too get imported into dictionaries during startup. Note that these dictionaries do not always match the table structures. Some of them change the structure to make queries easier to process. You may be able to do that with Python code, or by creating SQL views in the database and importing directly from the views (database views take almost no database memory). The drawback is that all that data is now stored in memory while the program is running. In my case, many hundreds of MB. But if it would be too much memory for you - you would need to prototype it to know - you should let the database engine do the work. It is more highly optimized and efficient for searches than your code could ever be. But there will be a price to pay. The price is in denormalizing the database table design. This means to include redundant data, organized to match the kinds of queries that will be made. No more 3rd normal form! Your sql queries will need to be designed to take advantage of this new structure. This will be a cost because the database will be larger, but also because the redundancies will make it much harder to update the data correctly. Fortunately you do not need to do that during normal operation (my program's data was also static like yours). PostgreSQL would probably be a better choice than Sqlite, since it supports features such as foreign keys, and has a function definition capability. From avi.e.gross at gmail.com Sun Jan 15 13:13:09 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Sun, 15 Jan 2023 13:13:09 -0500 Subject: To clarify how Python handles two equal objects In-Reply-To: <2f398a79-f6e7-fa75-4462-e7ba838b375b@chagford.com> References: <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> <00a801d927c1$a6b80730$f4281590$@gmail.com> <415e4a55-2ce6-f4fc-7c2e-df4fd9e1e301@roelschroeven.net> <2f398a79-f6e7-fa75-4462-e7ba838b375b@chagford.com> Message-ID: <003a01d9290d$065fb6f0$131f24d0$@gmail.com> <<< Frank Millman>>> My 'aha' moment came when I understood that a python object has only three properties - a type, an id, and a value. It does *not* have a name. Yes, Frank, it is a bit like how some people need to wrap their minds around a concept like an anonymous function. It has no name and for many purposes needs no name, but at the same time can be bound to one or more names if needed. Some people will encounter a situation where you need to pass a function to be called such as a key function for determining the sort order and they will create a new function first the usual way with a name and then pass it along by name. That works fine but the function then persists while not being used again and even choosing a name can have consequences if the same name is already in use and so on. Experienced programmers might use some kind of lambda expression in-line as the function as an object is then created, and passed along as a reference to be used and probably discarded once nothing refers to it. Similar things can happen if an object is created as part of a larger complex such as a list or deque of them. Each item can be referenced unambiguously without having a formal name. You can pop off the next one an use it or ask to take the fifth. It may acquire and lose names as the program runs but sometimes can have many names or none. Getting people to see that an object exists whether it has no name and to speak about them with that understanding can be a problem. We as humans seem to think we are our names. But I know my names have changed over the years partially because I moved between countries and I have several additional names used just for special purposes, and yet for some purposes I am simply a number. If I raise my class in a lecture hall where my name is not known, they may point at me or ask the guy in the blue short to talk. At the Department of Motor Vehicles, my name, like everyone else, is NEXT! The real point is what Python does, not what other situations require. Two objects are equal does not always mean what you think or want it to mean. What we are discussing here is two objects loosely of type "name" that are being compared not for whether THEY are themselves equal as in the same symbols composing them and perhaps living in the same namespace. The question was about two objects that contained references to another object or even to different parts of another object. As it happens, that can be a deeper question in which some parts are not a required aspect of the language and may be part of one of many possible implementations. And, as noted, the names can be detached from that reference too so a link is not permanent. Have we beaten this one to death yet? -----Original Message----- From: Python-list On Behalf Of Frank Millman Sent: Sunday, January 15, 2023 12:47 AM To: python-list at python.org Subject: Re: To clarify how Python handles two equal objects -- https://mail.python.org/mailman/listinfo/python-list From nntp.mbourne at spamgourmet.com Sun Jan 15 08:14:50 2023 From: nntp.mbourne at spamgourmet.com (Mark Bourne) Date: Sun, 15 Jan 2023 13:14:50 +0000 Subject: To clarify how Python handles two equal objects References: <31d2ed67-d710-cbb9-5e44-0872a07747c5@mrabarnett.plus.com> <59d349a8-508c-cc7d-c833-d309d888d4c2@tompassin.net> <3f5458ee-2084-dfef-a10a-e6a9ce959482@roelschroeven.net> <00a801d927c1$a6b80730$f4281590$@gmail.com> Message-ID: Jen Kris wrote: > Avi, > > Your comments go farther afield than my original question, but you made some interesting additional points.? For example, I sometimes work with the C API and sys.getrefcount may be helpful in deciding when to INCREF and DECREF.? But that?s another issue. > > The situation I described in my original post is limited to a case such as x = y where both "x" and "y" are arrays ? whether they are lists in Python, or from the array module ? and the question in a compiled C extension is whether the assignment can be done simply by "x" taking the pointer to "y" rather than moving all the data from "y" into the memory buffer for "x" which, for a wide array, would be much more time consuming than just moving a pointer.? The other advantage to doing it that way is if, as in my case, we perform a math operation on any element in "x" then Python expects that the same change to be reflected in "y."? If I don?t use the same pointers then I would have to perform that operation twice ? once for "x" and once? for "y" ? in addition to the expense of moving all the data. > > The answers I got from this post confirmed that it I can use the pointer if "y" is not re-defined to something else during the lifespan of "x."? If it is then "x" has to be restored to its original pointer.? I did it that way, and helpfully the compiler did not overrule me. I haven't done much with C extensions, but I don't think you'd need to do anything with "x" in that case. If something else is assigned to "y", "x" would still be a reference to the original object - why would it need to be "restored" to anything? Unless I've misunderstood what's going on here... -- Mark. From dino at no.spam.ar Sun Jan 15 08:20:01 2023 From: dino at no.spam.ar (Dino) Date: Sun, 15 Jan 2023 08:20:01 -0500 Subject: Fast lookup of bulky "table" References: <68e30710-b177-2301-4228-6b3216c1145b@solute.de> Message-ID: Thank you for your answer, Lars. Just a clarification: I am already doing a rough measuring of my queries. A fresh query without any caching: < 4s. Cached full query: < 5 micro-s (i.e. 6 orders of magnitude faster) Desired speed for my POC: 10 Hey, > > before you start optimizing. I would suggest, that you measure response > times and query times, data search times and so on. In order to save > time, you have to know where you "loose" time. > > Does your service really have to load the whole table at once? Yes that > might lead to quicker response times on requests, but databases are > often very good with caching themselves, so that the first request might > be slower than following requests, with similar parameters. Do you use a > database, or are you reading from a file? Are you maybe looping through > your whole dataset on every request? Instead of asking for the specific > data? > > Before you start introducing a cache and its added complexity, do you > really need that cache? > > You are talking about saving microseconds, that sounds a bit as if you > might be ?overdoing? it. How many requests will you have in the future? > At least in which magnitude and how quick do they have to be? You write > about 1-4 seconds on your laptop. But that does not really tell you that > much, because most probably the service will run on a server. I am not > saying that you should get a server or a cloud-instance to test against, > but to talk with your architect about that. > > I totally understand your impulse to appear as good as can be, but you > have to know where you really need to debug and optimize. It will not be > advantageous for you, if you start to optimize for optimizing's sake. > Additionally if you service is a PoC, optimizing now might be not the > first thing you have to worry about, but about that you made everything > as simple and readable as possible and that you do not spend too much > time for just showing how it could work. > > But of course, I do not know the tasks given to you and the expectations > you have to fulfil. All I am trying to say is to reconsider where you > really could improve and how far you have to improve. > > From dino at no.spam.ar Sun Jan 15 08:27:29 2023 From: dino at no.spam.ar (Dino) Date: Sun, 15 Jan 2023 08:27:29 -0500 Subject: Fast lookup of bulky "table" References: <20230115111424.xbd732bkmqumycno@hjp.at> Message-ID: Thank you, Peter. Yes, setting up my own indexes is more or less the idea of the modular cache that I was considering. Seeing others think in the same direction makes it look more viable. About Scalene, thank you for the pointer. I'll do some research. Do you have any idea about the speed of a SELECT query against a 100k rows / 300 Mb Sqlite db? Dino On 1/15/2023 6:14 AM, Peter J. Holzer wrote: > On 2023-01-14 23:26:27 -0500, Dino wrote: >> Hello, I have built a PoC service in Python Flask for my work, and - now >> that the point is made - I need to make it a little more performant (to be >> honest, chances are that someone else will pick up from where I left off, >> and implement the same service from scratch in a different language (GoLang? >> .Net? Java?) but I am digressing). >> >> Anyway, my Flask service initializes by loading a big "table" of 100k rows >> and 40 columns or so (memory footprint: order of 300 Mb) > > 300 MB is large enough that you should at least consider putting that > into a database (Sqlite is probably simplest. Personally I would go with > PostgreSQL because I'm most familiar with it and Sqlite is a bit of an > outlier). > > The main reason for putting it into a database is the ability to use > indexes, so you don't have to scan all 100 k rows for each query. > > You may be able to do that for your Python data structures, too: Can you > set up dicts which map to subsets you need often? > > There are some specialized in-memory bitmap implementations which can be > used for filtering. I've used > [Judy bitmaps](https://judy.sourceforge.net/doc/Judy1_3x.htm) in the > past (mostly in Perl). > These days [Roaring Bitmaps](https://www.roaringbitmap.org/) is probably > the most popular. I see several packages on PyPI - but I haven't used > any of them yet, so no recommendation from me. > > Numpy might also help. You will still have linear scans, but it is more > compact and many of the searches can probably be done in C and not in > Python. > >> As you can imagine, this is not very performant in its current form, but >> performance was not the point of the PoC - at least initially. > > For performanc optimization it is very important to actually measure > performance, and a good profiler helps very much in identifying hot > spots. Unfortunately until recently Python was a bit deficient in this > area, but [Scalene](https://pypi.org/project/scalene/) looks promising. > > hp > From gweatherby at uchc.edu Sun Jan 15 14:23:40 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Sun, 15 Jan 2023 19:23:40 +0000 Subject: Fast lookup of bulky "table" In-Reply-To: References: <68e30710-b177-2301-4228-6b3216c1145b@solute.de> Message-ID: That?s about what I got using a Python dictionary on random data on a high memory machine. https://github.com/Gerardwx/database_testing.git It?s not obvious to me how to get it much faster than that. From: Python-list on behalf of Dino Date: Sunday, January 15, 2023 at 1:29 PM To: python-list at python.org Subject: Re: Fast lookup of bulky "table" *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** Thank you for your answer, Lars. Just a clarification: I am already doing a rough measuring of my queries. A fresh query without any caching: < 4s. Cached full query: < 5 micro-s (i.e. 6 orders of magnitude faster) Desired speed for my POC: 10 Hey, > > before you start optimizing. I would suggest, that you measure response > times and query times, data search times and so on. In order to save > time, you have to know where you "loose" time. > > Does your service really have to load the whole table at once? Yes that > might lead to quicker response times on requests, but databases are > often very good with caching themselves, so that the first request might > be slower than following requests, with similar parameters. Do you use a > database, or are you reading from a file? Are you maybe looping through > your whole dataset on every request? Instead of asking for the specific > data? > > Before you start introducing a cache and its added complexity, do you > really need that cache? > > You are talking about saving microseconds, that sounds a bit as if you > might be ?overdoing? it. How many requests will you have in the future? > At least in which magnitude and how quick do they have to be? You write > about 1-4 seconds on your laptop. But that does not really tell you that > much, because most probably the service will run on a server. I am not > saying that you should get a server or a cloud-instance to test against, > but to talk with your architect about that. > > I totally understand your impulse to appear as good as can be, but you > have to know where you really need to debug and optimize. It will not be > advantageous for you, if you start to optimize for optimizing's sake. > Additionally if you service is a PoC, optimizing now might be not the > first thing you have to worry about, but about that you made everything > as simple and readable as possible and that you do not spend too much > time for just showing how it could work. > > But of course, I do not know the tasks given to you and the expectations > you have to fulfil. All I am trying to say is to reconsider where you > really could improve and how far you have to improve. > > -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!npizb3UAz-jPUnhlimB3_lctLibK5EW4zJwjZVmQ41yV_-2WSm2eQ5cTi8vzOEuCfsdNTjIvIhFcakrX$ From hjp-python at hjp.at Sun Jan 15 14:39:14 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 15 Jan 2023 20:39:14 +0100 Subject: Fast lookup of bulky "table" In-Reply-To: <499510f2-c5e9-9ed7-d09e-4207488d87fc@tompassin.net> References: <20230115111424.xbd732bkmqumycno@hjp.at> <499510f2-c5e9-9ed7-d09e-4207488d87fc@tompassin.net> Message-ID: <20230115193914.ba64o3tniup3uco2@hjp.at> On 2023-01-15 10:38:22 -0500, Thomas Passin wrote: > On 1/15/2023 6:14 AM, Peter J. Holzer wrote: > > On 2023-01-14 23:26:27 -0500, Dino wrote: > > > Anyway, my Flask service initializes by loading a big "table" of 100k rows > > > and 40 columns or so (memory footprint: order of 300 Mb) > > > > 300 MB is large enough that you should at least consider putting that > > into a database (Sqlite is probably simplest. Personally I would go with > > PostgreSQL because I'm most familiar with it and Sqlite is a bit of an > > outlier). > > > > The main reason for putting it into a database is the ability to use > > indexes, so you don't have to scan all 100 k rows for each query. > > I have an (inherited) server program that uses about 30 MB of data in a > MySQL database. It services queries received over the network. It too had > performance problems, to which adding indexes and smarter joins helped but > not enough. > > I changed the program so that at startup it imports much of the data into > Python dictionaries that are structured to support the kinds of queries that > need the help. Response time to queries dropped dramatically. This is to be expected: Firstly, because you don't have disk accesses any more, secondly because you don't have network latency any more and thirdly, because you structured the data to fit the queries. The thing to keep in mind is that the relational database model was invented to have a uniform and simple way to model all data, and that RDBMSs are designed to handle all workloads (from a single tiny table to thousands of tables with hundreds of terabytes) reasonably well. For any given application you can always find a more efficient solution than using an RDBMS. Sometimes it's simple (just load all the data into a dict and serve from there), sometimes it's a major research project. The nice thing about RDBMSs isn't that they are the optimal solution for anything but that they are a "good enough" solution for a large class of problems. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From gweatherby at uchc.edu Sun Jan 15 14:36:53 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Sun, 15 Jan 2023 19:36:53 +0000 Subject: Fast lookup of bulky "table" In-Reply-To: References: <68e30710-b177-2301-4228-6b3216c1145b@solute.de> Message-ID: I think any peformance improvements would have to come from a language change or better indexing of the data. From: Python-list on behalf of Weatherby,Gerard Date: Sunday, January 15, 2023 at 2:25 PM To: Dino , python-list at python.org Subject: Re: Fast lookup of bulky "table" That?s about what I got using a Python dictionary on random data on a high memory machine. https://urldefense.com/v3/__https://github.com/Gerardwx/database_testing.git__;!!Cn_UX_p3!keHKWsb1LGR6u_6BQA04MyEJlnzICq04FNdn8z9BnnjG8NopVu3KiL0k3rMiowxtp87xBUi6OcavBQIqksBjbd9v$ It?s not obvious to me how to get it much faster than that. From: Python-list on behalf of Dino Date: Sunday, January 15, 2023 at 1:29 PM To: python-list at python.org Subject: Re: Fast lookup of bulky "table" *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** Thank you for your answer, Lars. Just a clarification: I am already doing a rough measuring of my queries. A fresh query without any caching: < 4s. Cached full query: < 5 micro-s (i.e. 6 orders of magnitude faster) Desired speed for my POC: 10 Hey, > > before you start optimizing. I would suggest, that you measure response > times and query times, data search times and so on. In order to save > time, you have to know where you "loose" time. > > Does your service really have to load the whole table at once? Yes that > might lead to quicker response times on requests, but databases are > often very good with caching themselves, so that the first request might > be slower than following requests, with similar parameters. Do you use a > database, or are you reading from a file? Are you maybe looping through > your whole dataset on every request? Instead of asking for the specific > data? > > Before you start introducing a cache and its added complexity, do you > really need that cache? > > You are talking about saving microseconds, that sounds a bit as if you > might be ?overdoing? it. How many requests will you have in the future? > At least in which magnitude and how quick do they have to be? You write > about 1-4 seconds on your laptop. But that does not really tell you that > much, because most probably the service will run on a server. I am not > saying that you should get a server or a cloud-instance to test against, > but to talk with your architect about that. > > I totally understand your impulse to appear as good as can be, but you > have to know where you really need to debug and optimize. It will not be > advantageous for you, if you start to optimize for optimizing's sake. > Additionally if you service is a PoC, optimizing now might be not the > first thing you have to worry about, but about that you made everything > as simple and readable as possible and that you do not spend too much > time for just showing how it could work. > > But of course, I do not know the tasks given to you and the expectations > you have to fulfil. All I am trying to say is to reconsider where you > really could improve and how far you have to improve. > > -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!npizb3UAz-jPUnhlimB3_lctLibK5EW4zJwjZVmQ41yV_-2WSm2eQ5cTi8vzOEuCfsdNTjIvIhFcakrX$ -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!keHKWsb1LGR6u_6BQA04MyEJlnzICq04FNdn8z9BnnjG8NopVu3KiL0k3rMiowxtp87xBUi6OcavBQIqkvzm3bP5$ From list1 at tompassin.net Sun Jan 15 14:58:12 2023 From: list1 at tompassin.net (Thomas Passin) Date: Sun, 15 Jan 2023 14:58:12 -0500 Subject: Fast lookup of bulky "table" In-Reply-To: <20230115193914.ba64o3tniup3uco2@hjp.at> References: <20230115111424.xbd732bkmqumycno@hjp.at> <499510f2-c5e9-9ed7-d09e-4207488d87fc@tompassin.net> <20230115193914.ba64o3tniup3uco2@hjp.at> Message-ID: On 1/15/2023 2:39 PM, Peter J. Holzer wrote: > On 2023-01-15 10:38:22 -0500, Thomas Passin wrote: >> On 1/15/2023 6:14 AM, Peter J. Holzer wrote: >>> On 2023-01-14 23:26:27 -0500, Dino wrote: >>>> Anyway, my Flask service initializes by loading a big "table" of 100k rows >>>> and 40 columns or so (memory footprint: order of 300 Mb) >>> >>> 300 MB is large enough that you should at least consider putting that >>> into a database (Sqlite is probably simplest. Personally I would go with >>> PostgreSQL because I'm most familiar with it and Sqlite is a bit of an >>> outlier). >>> >>> The main reason for putting it into a database is the ability to use >>> indexes, so you don't have to scan all 100 k rows for each query. >> >> I have an (inherited) server program that uses about 30 MB of data in a >> MySQL database. It services queries received over the network. It too had >> performance problems, to which adding indexes and smarter joins helped but >> not enough. >> >> I changed the program so that at startup it imports much of the data into >> Python dictionaries that are structured to support the kinds of queries that >> need the help. Response time to queries dropped dramatically. > > This is to be expected: Firstly, because you don't have disk accesses > any more, secondly because you don't have network latency any more and > thirdly, because you structured the data to fit the queries. Of course: that's exactly why I made those changes. The tradeoff is using more memory for your program, sometimes a lot more. > The thing to keep in mind is that the relational database model was > invented to have a uniform and simple way to model all data, and that > RDBMSs are designed to handle all workloads (from a single tiny table to > thousands of tables with hundreds of terabytes) reasonably well. For any > given application you can always find a more efficient solution than > using an RDBMS. Sometimes it's simple (just load all the data into a > dict and serve from there), sometimes it's a major research project. > The nice thing about RDBMSs isn't that they are the optimal solution for > anything but that they are a "good enough" solution for a large class of > problems. Often the solution is careful (and not very normalized) table design to support your queries. In the case I'm discussing, it was easier for me to make Python do the work, and I could afford the memory load. In other cases, you have to put in the work on the database side. Often for slow queries, disk latency and I/O are not the limiting factors, but you have to put in the work and do the testing to make sure. From PythonList at DancesWithMice.info Sun Jan 15 15:12:30 2023 From: PythonList at DancesWithMice.info (dn) Date: Mon, 16 Jan 2023 09:12:30 +1300 Subject: Fast lookup of bulky "table" In-Reply-To: References: <68e30710-b177-2301-4228-6b3216c1145b@solute.de> Message-ID: <911c6267-7e6c-f2c2-fa88-1073070bbbb5@DancesWithMice.info> On 16/01/2023 08.36, Weatherby,Gerard wrote: > I think any peformance improvements would have to come from a language change or better indexing of the data. Exactly! Expanding on @Peter's post: databases (relational or not) are best organised according to use. Some must accept rapid insert/updates. Others are about look-ups (data-retrieval). A basic RDBMS, just as a Python dict, may only offer a single key for efficient retrieval. Postgres and MySQL (for example) enable the establishment of multiple and sophisticated indices/indexes, and the aptly-named "views" of data. If the queries can be grouped according to the manner in which the data must be accessed, a view could be built for each. At which time, even if every row must be accessed, the retrieval will be made more efficient and/or the response better-organised. Thus, if we have a DB of people. Many retrievals are likely to utilise an index on 'name'. However, if at times interest is limited to place or suburb, an index and view of such will speed things from O(n). Similarly, if a query is only to return people with a dog license. Some programmers don't realise that SQL can also be used for calculations, eg the eponymous COUNT(), which saves (CPU-time and coding-effort) over post-processing in Python. If there are many way to group/examine the data, then this may not be possible, but that depends upon the count of views cf the value of speedy-response - and bearing-in-mind that the demands for response-time may vary by type of query/data-access. So many variables to consider ... -- Regards, =dn From bowman at montana.com Sun Jan 15 15:54:18 2023 From: bowman at montana.com (rbowman) Date: 15 Jan 2023 20:54:18 GMT Subject: Fast lookup of bulky "table" References: <20230115111424.xbd732bkmqumycno@hjp.at> Message-ID: On Sun, 15 Jan 2023 08:27:29 -0500, Dino wrote: > Do you have any idea about the speed of a SELECT query against a 100k > rows / 300 Mb Sqlite db? https://www.sqlite.org/speed.html The site is old but has a number of comparisons. I have not used SQLite with Python yet but with both C and C# I've been impressed with the speed versus Postgres or MSSQL. One thing to watch is insertions. By default each insertion is a transaction. There is a dramatic speed increase for multiple insertions if you explicitly start a transaction, do the inserts, and end the transaction. My usage is a result of ESRI's dropping their C++/C# Engine API. My new approach uses queries against the AcrGIS Server REST interface for much of the functionality but some spatial queries can be replaced with predetermined tabular data rather than runtime spatial queries. For example, for a given dataset you can determine the coordinates of every intersection beforehand and the intersection of PINE ST and MAPLE AVE becomes a simple search in the SLQite database. ESRI's ArcPy is the closest replacement for the legacy C++ API so I assume in the future I will be using it in conjunction with SQLite. The actual geodata will still need to be in a spatially aware RDMBS like SQL Server or PostgreSQL/PostGIS but SQLite so far is the fastest and easiest to implement for non-spatial data. Also, it is in the public domain which avoids the complexities of MySQL and its derivatives for commercial applications. From gweatherby at uchc.edu Sun Jan 15 17:31:17 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Sun, 15 Jan 2023 22:31:17 +0000 Subject: Fast lookup of bulky "table" In-Reply-To: References: <68e30710-b177-2301-4228-6b3216c1145b@solute.de> <911c6267-7e6c-f2c2-fa88-1073070bbbb5@DancesWithMice.info> Message-ID: With Postgresql, one can also do pre-processing in Python. https://www.postgresql.org/docs/15/plpython.html While it?s not as convenient to develop as client-side Python, it can be used to implement complicated constraints or implement filtering on the server side, which reduces the amount of data that has to be sent back to the client. From: Python-list on behalf of Stefan Ram Date: Sunday, January 15, 2023 at 5:03 PM To: python-list at python.org Subject: Re: Fast lookup of bulky "table" *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** dn writes: >Some programmers don't realise that SQL can also be used for >calculations, eg the eponymous COUNT(), which saves (CPU-time and >coding-effort) over post-processing in Python. Yes, I second that! Sometimes, people only re-invent things in Python because they don't know SQL well enough, or they do not normalize their tables because they have not properly learned how to do this. I'd always start out with normalized tables and do as many operations in SQL as possible. I would then hesitate to de-normalize anything or transfer data operations into the programming language unless I am very sure that this is really advantageous. Once I had the task of writing VBA code to query and analyze data from a Jet engine (i.e., Microsoft Access). I ended up writing 90 % of the code in SQL and a thin layer of 10 % in VBA. And it was fast. -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!kAIZWRJ3oqrlkixX-iwrGeG9VVWjooBvzuMirfp44VTP32cELWf8Dk6MkPQwK2QwWzuUT9eNPNTlN152b23eFcM$ From list1 at tompassin.net Sun Jan 15 18:06:36 2023 From: list1 at tompassin.net (Thomas Passin) Date: Sun, 15 Jan 2023 18:06:36 -0500 Subject: Fast lookup of bulky "table" In-Reply-To: References: <68e30710-b177-2301-4228-6b3216c1145b@solute.de> <911c6267-7e6c-f2c2-fa88-1073070bbbb5@DancesWithMice.info> Message-ID: <92a3381c-5101-b946-8e2c-1decc5cd0817@tompassin.net> On 1/15/2023 4:49 PM, Stefan Ram wrote: > dn writes: >> Some programmers don't realise that SQL can also be used for >> calculations, eg the eponymous COUNT(), which saves (CPU-time and >> coding-effort) over post-processing in Python. > > Yes, I second that! Sometimes, people only re-invent things > in Python because they don't know SQL well enough, or they > do not normalize their tables because they have not properly > learned how to do this. > > I'd always start out with normalized tables and do as many > operations in SQL as possible. I would then hesitate to > de-normalize anything or transfer data operations into > the programming language unless I am very sure that this > is really advantageous. Yes, if you get the indexes and joins right, sometimes you can get a very large speed-up. It takes some experimenting and use of EXPLAIN, but it's worth doing. You especially want to avoid letting the database engine do full-table scans over and over. And you never want to send a lot of rows to Python and do post-filtering on them if you can avoid it. Use WHERE instead of HAVING if possible (HAVING works post-scan, WHERE works during row retrieval). > Once I had the task of writing VBA code to query and analyze > data from a Jet engine (i.e., Microsoft Access). I ended up > writing 90 % of the code in SQL and a thin layer of 10 % in VBA. > And it was fast. > > From greg.ewing at canterbury.ac.nz Sun Jan 15 17:54:02 2023 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 16 Jan 2023 11:54:02 +1300 Subject: Fast lookup of bulky "table" In-Reply-To: References: <20230115111424.xbd732bkmqumycno@hjp.at> Message-ID: On 16/01/23 2:27 am, Dino wrote: > Do you have any idea about the speed of a SELECT query against a 100k > rows / 300 Mb Sqlite db? That depends entirely on the nature of the query and how the data is indexed. If it's indexed in a way that allows sqlite to home in directly on the wanted data, it will be very fast. If it has to fall back on a linear search, it probably won't be significantly faster than your existing Python implementation. -- Greg From aapost at idontexist.club Sun Jan 15 20:13:35 2023 From: aapost at idontexist.club (aapost) Date: Sun, 15 Jan 2023 20:13:35 -0500 Subject: Python - working with xml/lxml/objectify/schemas, datatypes, and assignments References: Message-ID: On 1/3/23 22:57, aapost wrote: > I am trying to wrap my head around how one goes about working with and > editing xml elements ... Back to > contemplating and tinkering.. For anyone in a similar situation, xmlschema is actually quite nice. It didn't have the features I was looking for out of the box, but it does have a to_objects function and I have learned quite a bit while picking it apart. I am able to patch it to be good enough for my requirements. Below is the patch for anyone interested: # # Contribution for the xmlschema & elementpath python modules which are # Copyright (c), 2016-2020, SISSA (International School for Advanced Studies). # All rights reserved. # # This file is distributed under the terms of the MIT License. # See the file 'LICENSE' in the root directory of the present # distribution, or http://opensource.org/licenses/MIT. # # Patching and expansion of the xmlschema.dataobjects.DataElement object features # to get the best demonstration, change schema variable to your .xsd, and xmlobj to your .xml files # then run this as $ python -i filename.py from typing import Any, Optional, Union, Tuple #from types import MethodType class ValueLockedError(Exception): def __init__(self, obj, variable_name): self.message = "Can't set ." + variable_name + \ "\nThe object:\n" + str(obj) + \ "\nis Locked (._locked is set to True)" super().__init__(self.message) # importing in order necessary for intended monkey patch import elementpath.etree as ep_etree # Monkey patching additional static functions to the import of elementpath.etree # for namespace management of xml.etree.ElementTree code paths (which use # the global variable register_namespace._namespace_map for namespace registering) def etree_remove_registered_namespace(elem: ep_etree.ElementProtocol, uri: str = '') -> None: etree_module: Any if not ep_etree.is_etree_element(elem): raise TypeError(f"{elem!r} is not an Element") elif isinstance(elem, ep_etree.PyElementTree.Element): etree_module = ep_etree.PyElementTree elif not hasattr(elem, 'nsmap'): etree_module = ep_etree.ElementTree else: import lxml.etree as etree_module # type: ignore[no-redef] if not hasattr(elem, 'nsmap'): if uri in etree_module.register_namespace._namespace_map: del etree_module.register_namespace._namespace_map[uri] else: # TODO research this for better understanding # _namespace_map is uri->prefix # DataElement.nsmap prefix->uri # lxml etree .nsmap ?->? # not using lxml anyway so not really an issue as # this condition shouldn't be met for key, value in elem.nsmap.items(): # research - can there be multiple instances of uri to prefix?.. # or are they intended to be 1:1?.. if value == uri: if key in elem.nsmap: del elem.nsmap[key] #patching setattr(ep_etree, "etree_remove_registered_namespace", staticmethod(etree_remove_registered_namespace)) # for namespace management of xml.etree.ElementTree code paths (which use # the global variable register_namespace._namespace_map for namespace registering) def etree_get_registered_namespaces(elem: ep_etree.ElementProtocol) -> dict: etree_module: Any if not ep_etree.is_etree_element(elem): raise TypeError(f"{elem!r} is not an Element") elif isinstance(elem, ep_etree.PyElementTree.Element): etree_module = ep_etree.PyElementTree elif not hasattr(elem, 'nsmap'): etree_module = ep_etree.ElementTree else: import lxml.etree as etree_module # type: ignore[no-redef] if not hasattr(elem, 'nsmap'): return etree_module.register_namespace._namespace_map else: return elem.nsmap # shouldn't be met #patching setattr(ep_etree, "etree_get_registered_namespaces", staticmethod(etree_get_registered_namespaces)) # for namespace management of xml.etree.ElementTree code paths (which use # the global variable register_namespace._namespace_map for namespace registering) def etree_register_namespace(elem: ep_etree.ElementProtocol, prefix: str = None, uri: str = None) -> None: etree_module: Any if not ep_etree.is_etree_element(elem): raise TypeError(f"{elem!r} is not an Element") elif isinstance(elem, ep_etree.PyElementTree.Element): etree_module = ep_etree.PyElementTree elif not hasattr(elem, 'nsmap'): etree_module = ep_etree.ElementTree else: import lxml.etree as etree_module # type: ignore[no-redef] if prefix != None and uri != None: if not hasattr(elem, 'nsmap'): etree_module.register_namespace(prefix, uri) else: # TODO research this for better understanding # _namespace_map is uri->prefix # DataElement.nsmap prefix->uri # lxml etree .nsmap ?->? # not using lxml anyway so not really an issue as # this condition shouldn't be met elem.nsmap[prefix] = uri #patching setattr(ep_etree, "etree_register_namespace", staticmethod(etree_register_namespace)) # importing in order necessary for intended monkey patch import xmlschema # Monkey patching additional instance functions to the import of xmlschema # specifically xmlschema.dataobjects.DataElement # Instance functions so DataElement object can use above elementpath.etree namespace functions def register_namespace(self, prefix: str = None, uri: str = None) -> None: #root = self.encode(validation='strict') root, errors = self.encode(validation='lax') if prefix != None and uri != None: ep_etree.etree_register_namespace(root, prefix, uri) #patching setattr(xmlschema.dataobjects.DataElement, "register_namespace", register_namespace) def remove_registered_namespace(self, uri: str = '') -> None: #root = self.encode(validation='strict') root, errors = self.encode(validation='lax') ep_etree.etree_remove_registered_namespace(root, uri) #patching setattr(xmlschema.dataobjects.DataElement, "remove_registered_namespace", remove_registered_namespace) def get_registered_namespaces(self) -> dict: #root = self.encode(validation='strict') root, errors = self.encode(validation='lax') return ep_etree.etree_get_registered_namespaces(root) #patching setattr(xmlschema.dataobjects.DataElement, "get_registered_namespaces", get_registered_namespaces) # replacing .validate() & .is_valid() on DataElement so that namespaces from the DataElement # get set to the xml.etree.ElementTree register_namespace._namespace_map global when used def validate(self, use_defaults: bool = True, namespaces: Optional[xmlschema.aliases.NamespacesType] = None, max_depth: Optional[int] = None) -> None: """ Validates the XML data object. :raises: :exc:`XMLSchemaValidationError` if XML data object is not valid. :raises: :exc:`XMLSchemaValueError` if the instance has no schema bindings. """ if (self.nsmap and namespaces == None): #added code namespaces = self.nsmap #added code for error in self.iter_errors(use_defaults, namespaces, max_depth): raise error #patching setattr(xmlschema.dataobjects.DataElement, "validate", validate) def is_valid(self, use_defaults: bool = True, namespaces: Optional[xmlschema.aliases.NamespacesType] = None, max_depth: Optional[int] = None) -> bool: """ Like :meth:`validate` except it does not raise an exception on validation error but returns ``True`` if the XML data object is valid, ``False`` if it's invalid. :raises: :exc:`XMLSchemaValueError` if the instance has no schema bindings. :raises: :exc:`XMLSchemaValueError` if the instance has no schema bindings. """ if (self.nsmap and namespaces == None): #added code namespaces = self.nsmap #added code error = next(self.iter_errors(use_defaults, namespaces, max_depth), None) return error is None #patching setattr(xmlschema.dataobjects.DataElement, "is_valid", is_valid) # replace .tostring() on DataElement to allow for xml_declaration/encoding support # TODO research more, will likely customize a bit further def tostring(self, namespaces: Optional[xmlschema.aliases.NamespacesType] = None, indent: str = '', max_lines: Optional[int] = None, spaces_for_tab: Optional[int] = None, xml_declaration: Optional[bool] = None, encoding: str = 'unicode', method: str = 'xml') -> Any: if (self.nsmap and namespaces == None): namespaces = self.nsmap # Serializes the data element tree to an XML source string. # root, errors = self.encode(validation='lax') root = self.encode(validation="strict") #prefer strict on my output just in case.. return ep_etree.etree_tostring( root, namespaces, indent, max_lines, spaces_for_tab, xml_declaration, encoding, method) #patching setattr(xmlschema.dataobjects.DataElement, "tostring", tostring) # add get_value function - paired with set_value def get_value(self) -> Any: print(type(self)) return self.value #patching setattr(xmlschema.dataobjects.DataElement, "get_value", get_value) # add set_value function # assures change meets XMLSchema # reverts back on error # assumes data meets Schema to begin with, will remain unchanged in the end if it is not # :raises: :exc:`XMLSchemaValidationError` if XML data object is not valid after attempted change # :raises: :exc:`XMLSchemaValueError` if the instance has no schema bindings. # :raises: :exc:`ValueLockedError` if using ._locked and set to True def set_value(self, value: Any, use_defaults: bool = True, namespaces: Optional[xmlschema.aliases.NamespacesType] = None, max_depth: Optional[int] = None) -> None: if hasattr(self, "_locked") and self._locked == True: raise ValueLockedError(self, variable_name='value') else: if hasattr(self, "_locked"): self._locked = True self._set_value_temp_value = self.value self.value = value if (self.nsmap and namespaces == None): namespaces = self.nsmap for error in self.iter_errors(use_defaults, namespaces, max_depth): self.value = self._set_value_temp_value # revert value back to original del self._set_value_temp_value #clean up if hasattr(self, "_locked"): # unlock before raising if using/exists self._locked = False raise error # raise error # no errors del self._set_value_temp_value # clean up if hasattr(self, "_locked"): self._locked = False # unlock before returning if using/exists #patching setattr(xmlschema.dataobjects.DataElement, "set_value", set_value) # add get_attrib function - paired with set_attrib # remove added logic from .get(), requiring explicit matches only def get_attrib(self, key: str) -> Any: return self.attrib[key] #patching setattr(xmlschema.dataobjects.DataElement, "get_attrib", get_attrib) # add set_attrib function # assures change meets XMLSchema # reverts back on error # assumes data meets Schema to begin with, will remain unchanged in the end if it is not # :raises: :exc:`XMLSchemaValidationError` if XML data object is not valid after attempted change # :raises: :exc:`XMLSchemaValueError` if the instance has no schema bindings. # :raises: :exc:`ValueLockedError` if using ._locked and set to True def set_attrib(self, key: str, value: Any, use_defaults: bool = True, namespaces: Optional[xmlschema.aliases.NamespacesType] = None, max_depth: Optional[int] = None) -> Union[bool, Optional[Tuple[bool, str]]]: if hasattr(self, "_locked") and self._locked == True: raise ValueLockedError(self, variable_name='attrib[' + key + ']') else: if hasattr(self, "_locked"): self._locked = True if key in self.attrib: self._set_attrib_temp_value = self.attrib[key] # save original value if exists else: self._set_attrib_value_did_not_exist = True # or note if it doesn't exist self.attrib[key] = value if (self.nsmap and namespaces == None): namespaces = self.nsmap for error in self.iter_errors(use_defaults, namespaces, max_depth): if hasattr(self, '_set_attrib_temp_value'): self.attrib[key] = self._set_attrib_temp_value # revert value back to original if existed del self._set_attrib_temp_value elif hasattr(self, '_set_attrib_value_did_not_exist'): del self.attrib[key] # or just delete if it didn't del self._set_attrib_value_did_not_exist if hasattr(self, "_locked"): self._locked = False raise error # no errors if hasattr(self, '_set_attrib_temp_value'): del self._set_attrib_temp_value # clean up elif hasattr(self, '_set_attrib_value_did_not_exist'): del self._set_attrib_value_did_not_exist # clean up # TODO research @property / some or some type of better variable binding? # self._expand_xDE_attrib_prefix exists if expand_xmlschema_DataElement is run if hasattr(self, '_expand_xDE_attrib_prefix'): setattr(self, self._expand_xDE_attrib_prefix + key, value) if hasattr(self, "_locked"): self._locked = False # unlock before returning if using/exists #patching setattr(xmlschema.dataobjects.DataElement, "set_attrib", set_attrib) # add del_attrib function # assures change meets XMLSchema # reverts back on error # assumes data meets Schema to begin with, will remain unchanged in the end if it is not # :raises: :exc:`XMLSchemaValidationError` if XML data object is not valid after attempted change # :raises: :exc:`XMLSchemaValueError` if the instance has no schema bindings. # :raises: :exc:`ValueLockedError` if using ._locked and set to True # :raises: :exc:`KeyError` if xml tag attribute (.attrib[key]) doesn't exist def del_attrib(self, key: str, use_defaults: bool = True, namespaces: Optional[xmlschema.aliases.NamespacesType] = None, max_depth: Optional[int] = None) -> Union[bool, Optional[Tuple[bool, str]]]: if hasattr(self, "_locked") and self._locked == True: raise ValueLockedError(self, variable_name='attrib[' + key + ']') else: if hasattr(self, "_locked"): self._locked = True if key in self.attrib: self._del_attrib_temp_value = self.attrib[key] # save original value if exists else: if hasattr(self, "_locked"): self._locked = False raise KeyError("'" + key + "' Attribute does not exist, nothing to do") del self.attrib[key] if (self.nsmap and namespaces == None): namespaces = self.nsmap for error in self.iter_errors(use_defaults, namespaces, max_depth): if hasattr(self, '_del_attrib_temp_value'): self.attrib[key] = self._del_attrib_temp_value # attribute required, recreate value back to original del self._del_attrib_temp_value if hasattr(self, "_locked"): self._locked = False # append informational message to error output if hasattr(error, "message"): error.message += ":\n\nThe attribute value was returned to original state due to error" \ "\n\nThis error represents the state of this element IF the attribute were removed" raise error # no errors if hasattr(self, '_del_attrib_temp_value'): del self._del_attrib_temp_value # clean up # TODO research @property / some or some type of better variable binding? # self._expand_xDE_attrib_prefix exists if expand_xmlschema_DataElement is run if hasattr(self, '_expand_xDE_attrib_prefix'): delattr(self, self._expand_xDE_attrib_prefix + key) if hasattr(self, "_locked"): self._locked = False # unlock before returning if using/exists #patching setattr(xmlschema.dataobjects.DataElement, "del_attrib", del_attrib) # Monkey patching some class methods helpful for learning / troubleshooting @classmethod def _show_me_mro(cls): return cls.mro() setattr(xmlschema.validators.schemas.XsdValidator, "_show_me_mro", classmethod(_show_me_mro)) setattr(xmlschema.dataobjects.DataElement, "_show_me_mro", classmethod(_show_me_mro)) schema = xmlschema.XMLSchema("path/to/your.xsd", converter=xmlschema.JsonMLConverter) xmlobj = schema.to_objects("path/to/your.xml") # creates dot notation naming for all children recursively # c_ default prefix for child, a_ default prefix for tag attribute # _# numbered suffix for all children starting at 0 # increases from there if more than 1 child with same name def expand_xmlschema_DataElement(xsobj: xmlschema.dataobjects.DataElement, child_prefix: str = 'c_', attrib_prefix: str = 'a_') -> None: xsobj._expand_xDE_child_prefix = child_prefix xsobj._expand_xDE_attrib_prefix = attrib_prefix # _locked just an idea at the moment, may or may not use this in the end setattr(xsobj, "_locked", False) # set a class attribute for each xml tag attribute # DO NOT change these directly, use set_attrib on the parent class which changes .attrib first # These are currently just a copy of what is in the .attrib dict # Validation has no knowledge of their existence if they are changed outside of design # TODO research @property / or some type of better variable binding? if (xsobj.attrib): #print(xsobj.local_name + " has attributes") for key in xsobj.attrib.keys(): setattr(xsobj, xsobj._expand_xDE_attrib_prefix + key, xsobj.attrib[key]) # set a class attribute for each child for each in xsobj.iterchildren(): expand_xmlschema_DataElement(each) count = 0 while(True): if hasattr(xsobj, xsobj._expand_xDE_child_prefix + each.local_name + "_" + str(count)): count += 1 else: setattr(xsobj, xsobj._expand_xDE_child_prefix + each.local_name + "_" + str(count), each) break expand_xmlschema_DataElement(xmlobj) From aapost at idontexist.club Sun Jan 15 20:56:21 2023 From: aapost at idontexist.club (aapost) Date: Sun, 15 Jan 2023 20:56:21 -0500 Subject: Python - working with xml/lxml/objectify/schemas, datatypes, and assignments References: <25525.36948.543184.270051@ixdm.fritz.box> <25534.65042.617254.20506@ixdm.fritz.box> Message-ID: On 1/11/23 13:21, Dieter Maurer wrote: > aapost wrote at 2023-1-10 22:15 -0500: >> On 1/4/23 12:13, aapost wrote: >>> On 1/4/23 09:42, Dieter Maurer wrote: >>> ... >>>> You might have a look at `PyXB`, too. >>>> It tries hard to enforce schema restrictions in Python code. >>> ... >> Unfortunately picking it apart for a while and diving deeper in to a >> rabbit hole, PyXB looks to be a no-go. >> >> PyXB while interesting, and I respect it's complexity and depth, is >> lacking in design consistency in how it operates if you are trying to >> modify and work with the resulting structure intuitively. >> ... problem with simple types ... > > I use `PyXB` in `dm.saml2` and `dm.zope.saml2`, i.e. with > the SAML2 schema definitions (which include those > of XML signature and XML encryption). > I had no problems with simple types. I just assign them to attributes > of the Python objects representing the XML elements. > `PyXB` does the right thing when it serializes those objects into XML. It does do a lot of good things, and I am sad to see all the good work in it not get used, but for me it really boils down to what it can sum up itself in a couple comments from the author in it's first file (which I appreciate them and their honesty, because those are comments I could see myself writing in a similar situation)... ###### class cscRoot (object): """This little bundle of joy exists because in Python 2.6 it became an error to invoke C{object.__init__} with parameters (unless you also override C{__new__}, in which case it's only a warning. Whatever.). Since I'm bloody not going to check in every class whether C{super(Myclass,self)} refers to C{object} (even if I could figure out how to do that, 'cuz the obvious solutions don't work), we'll just make this thing the root of all U{cooperative super calling} hierarchies. ###### ###### def __init__ (self, *args, **kw): # Oh gross. If this class descends from list (and probably dict), we # get here when object is *not* our direct superclass. In that case, # we have to pass the arguments on up, or the strings don't get # created right. Below is the only way I've figured out to detect the # situation. # # Note that we might also get here if you mix-in a class that used # object as a parent instead of cscRoot. Don't do that. Printing the # mro() is a decent way of identifying the problem. ###### using that suggestion you can see that on simple types >>> pyxbxmlroot.SomeString._mro() [, , , , , , , , ] it has a python type that it sends all the way up right next to object, when that doesn't actually occur until after simpleType in class string (basis.simpleTypeDefinition, str): This makes the object dependent on it's parent, since it itself IS the value, I can't assign to or do anything to it by itself, or it and all the other stuff goes away. As designed it is very hard to change anything in it without breaking something. After working with xmlschema, it pretty much confirmed my assumptions that it doesn't need to be that way. I was able to follow what was going on and tweak xmlschema fairly easily. That and the fact that PyXB was abandoned 5-6 years ago make it a strong no-go to use in a project. It would need to be adopted with fresh development, stripped of the python2 stuff, and the object structure redesigned in a more uniform way with functionality properly containerized instead of all stuffed together... From dino at no.spam.ar Sun Jan 15 22:13:06 2023 From: dino at no.spam.ar (Dino) Date: Sun, 15 Jan 2023 22:13:06 -0500 Subject: Fast lookup of bulky "table" References: <68e30710-b177-2301-4228-6b3216c1145b@solute.de> Message-ID: On 1/15/2023 2:23 PM, Weatherby,Gerard wrote: > That?s about what I got using a Python dictionary on random data on a high memory machine. > > https://github.com/Gerardwx/database_testing.git > > It?s not obvious to me how to get it much faster than that. Gerard, you are a rockstar. This is going to be really useful if I do decide to adopt sqlite3 for my PoC, as I understand what's going on conceptually, but never really used sqlite (nor SQL in a long long time), so this may save me a bunch of time. I created a 300 Mb DB using your script. Then: $ ./readone.py testing 2654792 of 4655974 Found somedata0002654713 for 1ed9f9cd-0a9e-47e3-b0a7-3e1fcdabe166 in 0.239335000020219 seconds $ ./prefetch.py Index build 4.420937899994897 seconds testing 3058568 of 4655974 Found somedata0000202200 for 5dca1455-9cd6-4e4d-8e5a-7e6400de7ca7 in 4.4000043999403715e-06 seconds So, if I understand right: 1) once I built a dict out of the DB (in about 4 seconds), I was able to lookup an entry/record in 4 microseconds(!) 2) looking up a record/entry using a Sqlite query took 0.2 seconds (i.e. 500x slower) Interesting. Thank you for this. Very informative. I really appreciate that you took the time to write this. The conclusion seems to me that I probably don't want to go the Sqlite route, as I would be placing my data into a database just to extract it back into a dict when I need it if I want it fast. Ps: a few minor fixes to the README as this may be helpful to others. ./venv/... => ./env/.. i.e. ./env/bin/pip install -U pip ./env/bin/pip install -e . Also add part in [] Run create.py [size of DB in bytes] prior to running readone.py and/or prefetch.py BTW, can you tell me what is going on here? what's := ? while (increase := add_some(conn,adding)) == 0: https://github.com/Gerardwx/database_testing/blob/main/src/database_testing/create.py#L40 Dino From bouncingcats at gmail.com Mon Jan 16 02:53:38 2023 From: bouncingcats at gmail.com (David) Date: Mon, 16 Jan 2023 18:53:38 +1100 Subject: Fast lookup of bulky "table" In-Reply-To: References: <68e30710-b177-2301-4228-6b3216c1145b@solute.de> Message-ID: On Mon, 16 Jan 2023 at 16:15, Dino wrote: > BTW, can you tell me what is going on here? what's := ? > > while (increase := add_some(conn,adding)) == 0: See here: https://docs.python.org/3/reference/expressions.html#assignment-expressions https://realpython.com/python-walrus-operator/ From dino at no.spam.ar Mon Jan 16 09:43:01 2023 From: dino at no.spam.ar (Dino) Date: Mon, 16 Jan 2023 09:43:01 -0500 Subject: Fast lookup of bulky "table" References: <68e30710-b177-2301-4228-6b3216c1145b@solute.de> Message-ID: On 1/16/2023 2:53 AM, David wrote: > See here: > https://docs.python.org/3/reference/expressions.html#assignment-expressions > https://realpython.com/python-walrus-operator/ Thank you, brother. From dino at no.spam.ar Mon Jan 16 09:44:34 2023 From: dino at no.spam.ar (Dino) Date: Mon, 16 Jan 2023 09:44:34 -0500 Subject: Fast lookup of bulky "table" References: Message-ID: Just wanted to take a moment to express my gratitude to everyone who responded here. You have all been so incredibly helpful. Thank you Dino On 1/14/2023 11:26 PM, Dino wrote: > > Hello, I have built a PoC service in Python Flask for my work, and - now > that the point is made - I need to make it a little more performant (to > be honest, chances are that someone else will pick up from where I left > off, and implement the same service from scratch in a different language > (GoLang? .Net? Java?) but I am digressing). From bowman at montana.com Mon Jan 16 11:56:43 2023 From: bowman at montana.com (rbowman) Date: 16 Jan 2023 16:56:43 GMT Subject: Fast lookup of bulky "table" References: <20230115111424.xbd732bkmqumycno@hjp.at> <499510f2-c5e9-9ed7-d09e-4207488d87fc@tompassin.net> Message-ID: On 16 Jan 2023 15:14:06 GMT, Stefan Ram wrote: > When none of those reasons matter, one can use dictionaries in Python > as well. And then what Chandler Carruth showed us applies: I am missing something. Where is the data in your dictionary coming from? From list1 at tompassin.net Mon Jan 16 12:26:32 2023 From: list1 at tompassin.net (Thomas Passin) Date: Mon, 16 Jan 2023 12:26:32 -0500 Subject: Fast lookup of bulky "table" In-Reply-To: References: <20230115111424.xbd732bkmqumycno@hjp.at> <499510f2-c5e9-9ed7-d09e-4207488d87fc@tompassin.net> Message-ID: <8cb7ccd2-5989-384a-83e3-ed1fa907e7ef@tompassin.net> On 1/16/2023 10:14 AM, Stefan Ram wrote: > However, operating systems and databases also try to cache > information in main memory that is estimated to be accessed > often. Yes, and you can only know by testing, when that's possible. Also, if you know that you have the same queries repeated over and over, you can intern (basically, cache) their results in Python and get a performance boost that way. From edmondo.giovannozzi at gmail.com Mon Jan 16 13:18:57 2023 From: edmondo.giovannozzi at gmail.com (Edmondo Giovannozzi) Date: Mon, 16 Jan 2023 10:18:57 -0800 (PST) Subject: Fast lookup of bulky "table" In-Reply-To: References: Message-ID: <36eae363-3f32-41cd-8a53-a7fe7771ecf9n@googlegroups.com> Il giorno domenica 15 gennaio 2023 alle 05:26:50 UTC+1 Dino ha scritto: > Hello, I have built a PoC service in Python Flask for my work, and - now > that the point is made - I need to make it a little more performant (to > be honest, chances are that someone else will pick up from where I left > off, and implement the same service from scratch in a different language > (GoLang? .Net? Java?) but I am digressing). > > Anyway, my Flask service initializes by loading a big "table" of 100k > rows and 40 columns or so (memory footprint: order of 300 Mb) and then > accepts queries through a REST endpoint. Columns are strings, enums, and > numbers. Once initialized, the table is read only. The endpoint will > parse the query and match it against column values (equality, > inequality, greater than, etc.) Finally, it will return a (JSON) list of > all rows that satisfy all conditions in the query. > > As you can imagine, this is not very performant in its current form, but > performance was not the point of the PoC - at least initially. > > Before I deliver the PoC to a more experienced software architect who > will look at my code, though, I wouldn't mind to look a bit less lame > and do something about performance in my own code first, possibly by > bringing the average time for queries down from where it is now (order > of 1 to 4 seconds per query on my laptop) to 1 or 2 milliseconds on > average). > > To be honest, I was already able to bring the time down to a handful of > microseconds thanks to a rudimentary cache that will associate the > "signature" of a query to its result, and serve it the next time the > same query is received, but this may not be good enough: 1) queries > might be many and very different from one another each time, AND 2) I am > not sure the server will have a ton of RAM if/when this thing - or > whatever is derived from it - is placed into production. > > How can I make my queries generally more performant, ideally also in > case of a new query? > > Here's what I have been considering: > > 1. making my cache more "modular", i.e. cache the result of certain > (wide) queries. When a complex query comes in, I may be able to restrict > my search to a subset of the rows (as determined by a previously cached > partial query). This should keep the memory footprint under control. > > 2. Load my data into a numpy.array and use numpy.array operations to > slice and dice my data. > > 3. load my data into sqlite3 and use SELECT statement to query my table. > I have never used sqllite, plus there's some extra complexity as > comparing certain colum requires custom logic, but I wonder if this > architecture would work well also when dealing with a 300Mb database. > > 4. Other ideas? > > Hopefully I made sense. Thank you for your attention > > Dino As a comparison with numpy. Given the following lines: import numpy as np a = np.random.randn(400,100_000) ia = np.argsort(a[0,:]) a_elem = a[56, ia[0]] I have just taken an element randomly in a numeric table of 400x100000 elements To find it with numpy: %timeit isel = a == a_elem 35.5 ms ? 2.79 ms per loop (mean ? std. dev. of 7 runs, 10 loops each) And %timeit a[isel] 9.18 ms ? 371 ?s per loop (mean ? std. dev. of 7 runs, 100 loops each) As data are not ordered it is searching it one by one but at C level. Of course it depends on a lot of thing... From hjp-python at hjp.at Mon Jan 16 14:25:22 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Mon, 16 Jan 2023 20:25:22 +0100 Subject: Fast lookup of bulky "table" In-Reply-To: <911c6267-7e6c-f2c2-fa88-1073070bbbb5@DancesWithMice.info> References: <68e30710-b177-2301-4228-6b3216c1145b@solute.de> <911c6267-7e6c-f2c2-fa88-1073070bbbb5@DancesWithMice.info> Message-ID: <20230116192522.ky3umvlajszirarf@hjp.at> On 2023-01-16 09:12:30 +1300, dn via Python-list wrote: > On 16/01/2023 08.36, Weatherby,Gerard wrote: > > I think any peformance improvements would have to come from a language change or better indexing of the data. > Expanding on @Peter's post: databases (relational or not) are best organised > according to use. I probably wasn't very clear here. I think one of the main advantages of relational databases over earlier models (especially hierarchical databases but also network databases) is that you *don't* have to do that. In a hierarchical database you have to decide what's higher or lower in the hierarchy (e.g., does a department have employees, or do employees have a department?) and that has a dramatic effect on performance so you must structure the database on which queries are expected to be more common. In a relational database employees and departments are at the same level and you have a relationship between them. You don't need to know whether you'll have to look up all employees of a given department or the department of a given employee more often. Both will be similarly fast with appropriate indexes. (Of course you should still have an idea what the data is used for when designing your data model. But semantics are much more important than use cases at this stage and you don't have to redesign your entire database just because you need a new query.) This flexibility comes with a cost, though: A relational database is almost always good enough, but almost never optimal for any given use. > Postgres and MySQL (for example) enable the establishment of multiple and > sophisticated indices/indexes, and the aptly-named "views" of data. > > If the queries can be grouped according to the manner in which the data must > be accessed, a view could be built for each. At which time, even if every > row must be accessed, the retrieval will be made more efficient and/or the > response better-organised. Nitpick: A view will not be more efficient (unless it's a materialized view which is basically just a table). To retrieve that data, the RDBMS has to perform exactly the same operations as for the underlying query. Views make life simpler for the programmer (or analyst) by letting them *write* simpler queries, but under the surface nothing changes. In fact the performance may be worse, since the perceived simplicity of the view may cause the human to write queries which are hard to optimize. (There is another important use case for views: Access control and enforcement of business rules. But I fear we are straying far from the original topic now.) > Thus, if we have a DB of people. Many retrievals are likely to utilise an > index on 'name'. However, if at times interest is limited to place or > suburb, an index and view of such will speed things from O(n). Similarly, if > a query is only to return people with a dog license. I don't see where a view would help here - but maybe you are thinking of a more complex database than you describe. > Some programmers don't realise that SQL can also be used for calculations, > eg the eponymous COUNT(), which saves (CPU-time and coding-effort) over > post-processing in Python. Agreed. Aggregate and window functions can do a lot of work in the database. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From hjp-python at hjp.at Mon Jan 16 14:32:16 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Mon, 16 Jan 2023 20:32:16 +0100 Subject: Fast lookup of bulky "table" In-Reply-To: <92a3381c-5101-b946-8e2c-1decc5cd0817@tompassin.net> References: <68e30710-b177-2301-4228-6b3216c1145b@solute.de> <911c6267-7e6c-f2c2-fa88-1073070bbbb5@DancesWithMice.info> <92a3381c-5101-b946-8e2c-1decc5cd0817@tompassin.net> Message-ID: <20230116193216.w4pdcpzs4ehd7cq5@hjp.at> On 2023-01-15 18:06:36 -0500, Thomas Passin wrote: > You especially want to avoid letting the database engine do full-table > scans over and over. And you never want to send a lot of rows to > Python and do post-filtering on them if you can avoid it. Another thing to avoid: Lots of small queries. If you have the choice between a single query which returns 10000 rows and 1000 queries which return on average 5 rows each, the former is almost always much faster. Each query carries some overhead. This may not be very noticeable with SQLite (as everything is in the same process), but it becomes more and more pronounced the farther the database is from the client. For complicated queries the parsing and planning overhead can also become significant. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From list1 at tompassin.net Mon Jan 16 12:28:37 2023 From: list1 at tompassin.net (Thomas Passin) Date: Mon, 16 Jan 2023 12:28:37 -0500 Subject: Fast lookup of bulky "table" In-Reply-To: References: <20230115111424.xbd732bkmqumycno@hjp.at> <499510f2-c5e9-9ed7-d09e-4207488d87fc@tompassin.net> Message-ID: <5360d84d-ec03-f6f4-5d2e-11eb6848c639@tompassin.net> On 1/16/2023 11:56 AM, rbowman wrote: > On 16 Jan 2023 15:14:06 GMT, Stefan Ram wrote: > > >> When none of those reasons matter, one can use dictionaries in Python >> as well. And then what Chandler Carruth showed us applies: > > I am missing something. Where is the data in your dictionary coming from? It would get imported on startup. This is assuming that the data does not get changed during operation, which the OP said is the case. From sjeik_appie at hotmail.com Mon Jan 16 14:59:30 2023 From: sjeik_appie at hotmail.com (Albert-Jan Roskam) Date: Mon, 16 Jan 2023 20:59:30 +0100 Subject: Fast lookup of bulky "table" In-Reply-To: Message-ID: On Jan 15, 2023 05:26, Dino wrote: Hello, I have built a PoC service in Python Flask for my work, and - now that the point is made - I need to make it a little more performant (to be honest, chances are that someone else will pick up from where I left off, and implement the same service from scratch in a different language (GoLang? .Net? Java?) but I am digressing). ======= Hi, * I'd start by measuring where your program spends its time:?https://docs.python.org/3/library/profile.html * It might be useful try DuckDB instead of Sqlite.?https://duckdb.org/why_duckdb.html Best wishes, AJ From dino at no.spam.ar Mon Jan 16 18:17:42 2023 From: dino at no.spam.ar (Dino) Date: Mon, 16 Jan 2023 18:17:42 -0500 Subject: Fast lookup of bulky "table" References: <36eae363-3f32-41cd-8a53-a7fe7771ecf9n@googlegroups.com> Message-ID: On 1/16/2023 1:18 PM, Edmondo Giovannozzi wrote: > > As a comparison with numpy. Given the following lines: > > import numpy as np > a = np.random.randn(400,100_000) > ia = np.argsort(a[0,:]) > a_elem = a[56, ia[0]] > > I have just taken an element randomly in a numeric table of 400x100000 elements > To find it with numpy: > > %timeit isel = a == a_elem > 35.5 ms ? 2.79 ms per loop (mean ? std. dev. of 7 runs, 10 loops each) > > And > %timeit a[isel] > 9.18 ms ? 371 ?s per loop (mean ? std. dev. of 7 runs, 100 loops each) > > As data are not ordered it is searching it one by one but at C level. > Of course it depends on a lot of thing... thank you for this. It's probably my lack of experience with Numpy, but... can you explain what is going on here in more detail? Thank you Dino From bowman at montana.com Mon Jan 16 20:47:19 2023 From: bowman at montana.com (rbowman) Date: 17 Jan 2023 01:47:19 GMT Subject: Fast lookup of bulky "table" References: <20230115111424.xbd732bkmqumycno@hjp.at> <499510f2-c5e9-9ed7-d09e-4207488d87fc@tompassin.net> <5360d84d-ec03-f6f4-5d2e-11eb6848c639@tompassin.net> Message-ID: On Mon, 16 Jan 2023 12:28:37 -0500, Thomas Passin wrote: > On 1/16/2023 11:56 AM, rbowman wrote: >> On 16 Jan 2023 15:14:06 GMT, Stefan Ram wrote: >> >> >>> When none of those reasons matter, one can use dictionaries in >>> Python as well. And then what Chandler Carruth showed us applies: >> >> I am missing something. Where is the data in your dictionary coming >> from? > > It would get imported on startup. This is assuming that the data does > not get changed during operation, which the OP said is the case. Okay, so there is no data loss if the program or machine crashes. That makes sense. In the cases Where I'm using SQLite the input data consists of relatively expensive spatial queries that are processed and written out to persist the results for use by other applications in the suite. From stephen_tucker at sil.org Tue Jan 17 07:47:29 2023 From: stephen_tucker at sil.org (Stephen Tucker) Date: Tue, 17 Jan 2023 12:47:29 +0000 Subject: IDLE "Codepage" Switching? Message-ID: I have four questions. 1. Can anybody explain the behaviour in IDLE (Python version 2.7.10) reported below? (It seems that the way it renders a given sequence of bytes depends on the sequence.) 2. Does the IDLE in Python 3.x behave the same way? 3. If it does, is this as it should behave? 4. If it is, then why is it as it should behave? ============================== >>> mylongstr = "" >>> for thisCP in range (157, 169): mylongstr += chr (thisCP) + " " >>> print mylongstr ? ? ? ? ? ? ? ? ? ? ? ? >>> mylongstr = "" >>> for thisCP in range (158, 169): mylongstr += chr (thisCP) + " " >>> print mylongstr ? ? ? ? ? ? ? ? ? ? >>> mylongstr = "" >>> for thisCP in range (157, 169): mylongstr += chr (thisCP) + " " >>> print mylongstr ? ? ? ? ? ? ? ? ? ? ? ? ============================== Stephen Tucker. From edmondo.giovannozzi at gmail.com Tue Jan 17 05:42:02 2023 From: edmondo.giovannozzi at gmail.com (Edmondo Giovannozzi) Date: Tue, 17 Jan 2023 02:42:02 -0800 (PST) Subject: Fast lookup of bulky "table" In-Reply-To: References: <36eae363-3f32-41cd-8a53-a7fe7771ecf9n@googlegroups.com> Message-ID: Il giorno marted? 17 gennaio 2023 alle 00:18:04 UTC+1 Dino ha scritto: > On 1/16/2023 1:18 PM, Edmondo Giovannozzi wrote: > > > > As a comparison with numpy. Given the following lines: > > > > import numpy as np > > a = np.random.randn(400,100_000) > > ia = np.argsort(a[0,:]) > > a_elem = a[56, ia[0]] > > > > I have just taken an element randomly in a numeric table of 400x100000 elements > > To find it with numpy: > > > > %timeit isel = a == a_elem > > 35.5 ms ? 2.79 ms per loop (mean ? std. dev. of 7 runs, 10 loops each) > > > > And > > %timeit a[isel] > > 9.18 ms ? 371 ?s per loop (mean ? std. dev. of 7 runs, 100 loops each) > > > > As data are not ordered it is searching it one by one but at C level. > > Of course it depends on a lot of thing... > thank you for this. It's probably my lack of experience with Numpy, > but... can you explain what is going on here in more detail? > > Thank you > > Dino Sorry, I was just creating an array of 400x100000 elements that I fill with random numbers: a = np.random.randn(400,100_000) Then I pick one element randomly, it is just a stupid sort on a row and then I take an element in another row, but it doesn't matter, I'm just taking a random element. I may have used other ways to get that but was the first that came to my mind. ia = np.argsort(a[0,:]) a_elem = a[56, ia[0]] The I'm finding that element in the all the matrix a (of course I know where it is, but I want to test the speed of a linear search done on the C level): %timeit isel = a == a_elem Actually isel is a logic array that is True where a[i,j] == a_elem and False where a[i,j] != a_elem. It may find more then one element but, of course, in our case it will find only the element that we have selected at the beginning. So it will give the speed of a linear search plus the time needed to allocate the logic array. The search is on the all matrix of 40 million of elements not just on one of its row of 100k element. On the single row (that I should say I have chosen to be contiguous) is much faster. %timeit isel = a[56,:] == a_elem 26 ?s ? 588 ns per loop (mean ? std. dev. of 7 runs, 10000 loops each) the matrix is a double precision numbers that is 8 byte, I haven't tested it on string of characters. This wanted to be an estimate of the speed that one can get going to the C level. You loose of course the possibility to have a relational database, you need to have everything in memory, etc... A package that implements tables based on numpy is pandas: https://pandas.pydata.org/ I hope that it can be useful. From dino at no.spam.ar Tue Jan 17 10:39:14 2023 From: dino at no.spam.ar (Dino) Date: Tue, 17 Jan 2023 10:39:14 -0500 Subject: Fast lookup of bulky "table" References: <36eae363-3f32-41cd-8a53-a7fe7771ecf9n@googlegroups.com> Message-ID: Thanks a lot, Edmondo. Or better... Grazie mille. On 1/17/2023 5:42 AM, Edmondo Giovannozzi wrote: > > Sorry, > I was just creating an array of 400x100000 elements that I fill with random numbers: > > a = np.random.randn(400,100_000) > > Then I pick one element randomly, it is just a stupid sort on a row and then I take an element in another row, but it doesn't matter, I'm just taking a random element. I may have used other ways to get that but was the first that came to my mind. > > ia = np.argsort(a[0,:]) > a_elem = a[56, ia[0]] > > The I'm finding that element in the all the matrix a (of course I know where it is, but I want to test the speed of a linear search done on the C level): > > %timeit isel = a == a_elem > > Actually isel is a logic array that is True where a[i,j] == a_elem and False where a[i,j] != a_elem. It may find more then one element but, of course, in our case it will find only the element that we have selected at the beginning. So it will give the speed of a linear search plus the time needed to allocate the logic array. The search is on the all matrix of 40 million of elements not just on one of its row of 100k element. > > On the single row (that I should say I have chosen to be contiguous) is much faster. > > %timeit isel = a[56,:] == a_elem > 26 ?s ? 588 ns per loop (mean ? std. dev. of 7 runs, 10000 loops each) > > the matrix is a double precision numbers that is 8 byte, I haven't tested it on string of characters. > > This wanted to be an estimate of the speed that one can get going to the C level. > You loose of course the possibility to have a relational database, you need to have everything in memory, etc... > > A package that implements tables based on numpy is pandas: https://pandas.pydata.org/ > > I hope that it can be useful. > > From bowman at montana.com Tue Jan 17 20:46:49 2023 From: bowman at montana.com (rbowman) Date: 18 Jan 2023 01:46:49 GMT Subject: IDLE "Codepage" Switching? References: Message-ID: On Tue, 17 Jan 2023 12:47:29 +0000, Stephen Tucker wrote: > 2. Does the IDLE in Python 3.x behave the same way? fwiw Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux Type "help", "copyright", "credits" or "license()" for more information. str = "" for c in range(157, 169): str += chr(c) + "" print(str) ???????????? str = "" for c in range(140, 169): str += chr(c) + " " print(str) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? I don't know how this will appear since Pan is showing the icon for a character not in its set. However, even with more undefined characters the printable one do not change. I get the same output running Python3 from the terminal so it's not an IDLE thing. From list1 at tompassin.net Tue Jan 17 22:58:53 2023 From: list1 at tompassin.net (Thomas Passin) Date: Tue, 17 Jan 2023 22:58:53 -0500 Subject: IDLE "Codepage" Switching? In-Reply-To: References: Message-ID: <5fa51674-80b2-e64b-7353-1a663d436b29@tompassin.net> On 1/17/2023 8:46 PM, rbowman wrote: > On Tue, 17 Jan 2023 12:47:29 +0000, Stephen Tucker wrote: > >> 2. Does the IDLE in Python 3.x behave the same way? > > fwiw > > Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux > Type "help", "copyright", "credits" or "license()" for more information. > str = "" > for c in range(157, 169): > str += chr(c) + "" > > > print(str) > ???????????? > str = "" > for c in range(140, 169): > str += chr(c) + " " > > > print(str) > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? > > > I don't know how this will appear since Pan is showing the icon for a > character not in its set. However, even with more undefined characters > the printable one do not change. I get the same output running Python3 > from the terminal so it's not an IDLE thing. I'm not sure what explanation is being asked for here. Let's take Python3, so we can be sure that the strings are in unicode. The font being used by the console isn't mentioned, but there's no reason it should have glyphs for any random unicode character. In my case, I see the same missing and printable characters as in the previous post (above). The font is Source Code Pro Medium. Changing the console's code page won't magically provide the missing glyphs. I wrote these characters to a file using utf-8 encoding and opened it in an editor that recognized the content as utf-8 (EditPlus). It displayed the same characters but had fewer leading spaces (i.e., missing glyphs), and did not show any default "missing-character" glyphs. The editor is using the Cousine font. The second factor that could be in play is what the default character encoding is, which is set by Windows and could be different in different places (locales). I don't recall just now how Python3 handles this. Since Python2 strings are not unicode unless specified, and Python2 probably handles the locale/default encoding differently from Python3, it would not be a surprise if the two give different results. If you print such a Python2 string, you will get glyphs for (non-ascii) ord(chr) > 127 that come from the Windows code page table, which will be different from what Python3 will display. Python3 uses Windows Unicode API functions, and isn't subject to the same limitations as Python2 was - Python2 had to go though the Windows code page apparatus and didn't use the Unicode API. See PEP 528 - https://peps.python.org/pep-0528/) IDLE sets up its own window itself, and probably uses a different font from the default Windows console, so there could be some differences there too, especially as to whether missing glyphs show a visible symbol or not. Code Page 65001 was often claimed to be for utf-8. It's not really correct in general, but it's OK for many utf-8 characters. But in Python2, the codecs module does not know about code page 65001 - unless you apply a simple patch - so if you try to set the console to cp65001, you cannot get anything printed. You get an exception raised instead. Yes, it's all confusing, and especially with Python2. From hjp-python at hjp.at Wed Jan 18 04:41:05 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Wed, 18 Jan 2023 10:41:05 +0100 Subject: IDLE "Codepage" Switching? In-Reply-To: <5fa51674-80b2-e64b-7353-1a663d436b29@tompassin.net> References: <5fa51674-80b2-e64b-7353-1a663d436b29@tompassin.net> Message-ID: <20230118094105.c7rw44bjuvjnu4gu@hjp.at> On 2023-01-17 22:58:53 -0500, Thomas Passin wrote: > On 1/17/2023 8:46 PM, rbowman wrote: > > On Tue, 17 Jan 2023 12:47:29 +0000, Stephen Tucker wrote: > > > 2. Does the IDLE in Python 3.x behave the same way? > > > > fwiw > > > > Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux > > Type "help", "copyright", "credits" or "license()" for more information. > > str = "" > > for c in range(140, 169): > > str += chr(c) + " " > > > > print(str) > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > > ? ? ? > > > > > > I don't know how this will appear since Pan is showing the icon for a > > character not in its set. However, even with more undefined characters > > the printable one do not change. I get the same output running Python3 > > from the terminal so it's not an IDLE thing. > > I'm not sure what explanation is being asked for here. Let's take Python3, > so we can be sure that the strings are in unicode. The font being used by > the console isn't mentioned, but there's no reason it should have glyphs for > any random unicode character. Also note that the characters between 128 (U+0080) and 159 (U+009F) inclusive aren't printable characters. They are control characters. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From stephen_tucker at sil.org Wed Jan 18 05:43:01 2023 From: stephen_tucker at sil.org (Stephen Tucker) Date: Wed, 18 Jan 2023 10:43:01 +0000 Subject: IDLE "Codepage" Switching? In-Reply-To: <20230118094105.c7rw44bjuvjnu4gu@hjp.at> References: <5fa51674-80b2-e64b-7353-1a663d436b29@tompassin.net> <20230118094105.c7rw44bjuvjnu4gu@hjp.at> Message-ID: Thanks for these responses. I was encouraged to read that I'm not the only one to find this all confusing. I have investigated a little further. 1. I produced the following IDLE log: >>> mylongstr = "" >>> for thisCP in range (1, 256): mylongstr += chr (thisCP) + " " + str (ord (chr (thisCP))) + ", " >>> print mylongstr 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, ! 33, " 34, # 35, $ 36, % 37, & 38, ' 39, ( 40, ) 41, * 42, + 43, , 44, - 45, . 46, / 47, 0 48, 1 49, 2 50, 3 51, 4 52, 5 53, 6 54, 7 55, 8 56, 9 57, : 58, ; 59, < 60, = 61, > 62, ? 63, @ 64, A 65, B 66, C 67, D 68, E 69, F 70, G 71, H 72, I 73, J 74, K 75, L 76, M 77, N 78, O 79, P 80, Q 81, R 82, S 83, T 84, U 85, V 86, W 87, X 88, Y 89, Z 90, [ 91, \ 92, ] 93, ^ 94, _ 95, ` 96, a 97, b 98, c 99, d 100, e 101, f 102, g 103, h 104, i 105, j 106, k 107, l 108, m 109, n 110, o 111, p 112, q 113, r 114, s 115, t 116, u 117, v 118, w 119, x 120, y 121, z 122, { 123, | 124, } 125, ~ 126, 127, ? 128, ? 129, ? 130, ? 131, ? 132, ? 133, ? 134, ? 135, ? 136, ? 137, ? 138, ? 139, ? 140, ? 141, ? 142, ? 143, ? 144, ? 145, ? 146, ? 147, ? 148, ? 149, ? 150, ? 151, ? 152, ? 153, ? 154, ? 155, ? 156, ? 157, ? 158, ? 159, ? 160, ? 161, ? 162, ? 163, ? 164, ? 165, ? 166, ? 167, ? 168, ? 169, ? 170, ? 171, ? 172, ? 173, ? 174, ? 175, ? 176, ? 177, ? 178, ? 179, ? 180, ? 181, ? 182, ? 183, ? 184, ? 185, ? 186, ? 187, ? 188, ? 189, ? 190, ? 191, ? 192, ? 193, ? 194, ? 195, ? 196, ? 197, ? 198, ? 199, ? 200, ? 201, ? 202, ? 203, ? 204, ? 205, ? 206, ? 207, ? 208, ? 209, ? 210, ? 211, ? 212, ? 213, ? 214, ? 215, ? 216, ? 217, ? 218, ? 219, ? 220, ? 221, ? 222, ? 223, ? 224, ? 225, ? 226, ? 227, ? 228, ? 229, ? 230, ? 231, ? 232, ? 233, ? 234, ? 235, ? 236, ? 237, ? 238, ? 239, ? 240, ? 241, ? 242, ? 243, ? 244, ? 245, ? 246, ? 247, ? 248, ? 249, ? 250, ? 251, ? 252, ? 253, ? 254, ? 255, >>> 2. I copied and pasted the IDLE log into a text file and ran a program on it that told me about every byte in the log. 3. I discovered the following: Bytes 001 to 127 (01 to 7F hex) inclusive were printed as-is; Bytes 128 to 191 (80 to BF) inclusive were output as UTF-8-encoded characters whose codepoints were FF00 hex more than the byte values (hence the strange glyphs); Bytes 192 to 255 (C0 to FF) inclusive were output as UTF-8-encoded characters - without any offset being added to their codepoints in the meantime! I thought you might just be interested in this - there does seem to be some method in IDLE's mind, at least. Stephen Tucker. On Wed, Jan 18, 2023 at 9:41 AM Peter J. Holzer wrote: > On 2023-01-17 22:58:53 -0500, Thomas Passin wrote: > > On 1/17/2023 8:46 PM, rbowman wrote: > > > On Tue, 17 Jan 2023 12:47:29 +0000, Stephen Tucker wrote: > > > > 2. Does the IDLE in Python 3.x behave the same way? > > > > > > fwiw > > > > > > Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux > > > Type "help", "copyright", "credits" or "license()" for more > information. > > > str = "" > > > for c in range(140, 169): > > > str += chr(c) + " " > > > > > > print(str) > > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > > > ? ? ? > > > > > > > > > I don't know how this will appear since Pan is showing the icon for a > > > character not in its set. However, even with more undefined characters > > > the printable one do not change. I get the same output running Python3 > > > from the terminal so it's not an IDLE thing. > > > > I'm not sure what explanation is being asked for here. Let's take > Python3, > > so we can be sure that the strings are in unicode. The font being used > by > > the console isn't mentioned, but there's no reason it should have glyphs > for > > any random unicode character. > > Also note that the characters between 128 (U+0080) and 159 (U+009F) > inclusive aren't printable characters. They are control characters. > > hp > > -- > _ | Peter J. Holzer | Story must make more sense than reality. > |_|_) | | > | | | hjp at hjp.at | -- Charles Stross, "Creative writing > __/ | http://www.hjp.at/ | challenge!" > -- > https://mail.python.org/mailman/listinfo/python-list > From list1 at tompassin.net Wed Jan 18 11:05:24 2023 From: list1 at tompassin.net (Thomas Passin) Date: Wed, 18 Jan 2023 11:05:24 -0500 Subject: IDLE "Codepage" Switching? In-Reply-To: References: <5fa51674-80b2-e64b-7353-1a663d436b29@tompassin.net> <20230118094105.c7rw44bjuvjnu4gu@hjp.at> Message-ID: <64f97adb-e7a5-a912-13d6-f2aff3301f90@tompassin.net> On 1/18/2023 5:43 AM, Stephen Tucker wrote: > Thanks for these responses. > > I was encouraged to read that I'm not the only one to find this all > confusing. > > I have investigated a little further. > > 1. I produced the following IDLE log: > >>>> mylongstr = "" >>>> for thisCP in range (1, 256): > mylongstr += chr (thisCP) + " " + str (ord (chr (thisCP))) + ", " > > >>>> print mylongstr > 1, 2, 3, 4, 5, 6, 7, 8, 9, > 10, 11, 12, > 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, > 31, 32, ! 33, " 34, # 35, $ 36, % 37, & 38, ' 39, ( 40, ) 41, * 42, + 43, > , 44, - 45, . 46, / 47, 0 48, 1 49, 2 50, 3 51, 4 52, 5 53, 6 54, 7 55, 8 > 56, 9 57, : 58, ; 59, < 60, = 61, > 62, ? 63, @ 64, A 65, B 66, C 67, D 68, > E 69, F 70, G 71, H 72, I 73, J 74, K 75, L 76, M 77, N 78, O 79, P 80, Q > 81, R 82, S 83, T 84, U 85, V 86, W 87, X 88, Y 89, Z 90, [ 91, \ 92, ] 93, > ^ 94, _ 95, ` 96, a 97, b 98, c 99, d 100, e 101, f 102, g 103, h 104, i > 105, j 106, k 107, l 108, m 109, n 110, o 111, p 112, q 113, r 114, s 115, > t 116, u 117, v 118, w 119, x 120, y 121, z 122, { 123, | 124, } 125, ~ > 126, 127, ? 128, ? 129, ? 130, ? 131, ? 132, ? 133, ? 134, ? 135, ? 136, ? > 137, ? 138, ? 139, ? 140, ? 141, ? 142, ? 143, ? 144, ? 145, ? 146, ? 147, > ? 148, ? 149, ? 150, ? 151, ? 152, ? 153, ? 154, ? 155, ? 156, ? 157, ? > 158, ? 159, ? 160, ? 161, ? 162, ? 163, ? 164, ? 165, ? 166, ? 167, ? 168, > ? 169, ? 170, ? 171, ? 172, ? 173, ? 174, ? 175, ? 176, ? 177, ? 178, ? > 179, ? 180, ? 181, ? 182, ? 183, ? 184, ? 185, ? 186, ? 187, ? 188, ? 189, > ? 190, ? 191, ? 192, ? 193, ? 194, ? 195, ? 196, ? 197, ? 198, ? 199, ? > 200, ? 201, ? 202, ? 203, ? 204, ? 205, ? 206, ? 207, ? 208, ? 209, ? 210, > ? 211, ? 212, ? 213, ? 214, ? 215, ? 216, ? 217, ? 218, ? 219, ? 220, ? > 221, ? 222, ? 223, ? 224, ? 225, ? 226, ? 227, ? 228, ? 229, ? 230, ? 231, > ? 232, ? 233, ? 234, ? 235, ? 236, ? 237, ? 238, ? 239, ? 240, ? 241, ? > 242, ? 243, ? 244, ? 245, ? 246, ? 247, ? 248, ? 249, ? 250, ? 251, ? 252, > ? 253, ? 254, ? 255, >>>> > > 2. I copied and pasted the IDLE log into a text file and ran a program on > it that told me about every byte in the log. > > 3. I discovered the following: > > Bytes 001 to 127 (01 to 7F hex) inclusive were printed as-is; > > Bytes 128 to 191 (80 to BF) inclusive were output as UTF-8-encoded > characters whose codepoints were FF00 hex more than the byte values (hence > the strange glyphs); > > Bytes 192 to 255 (C0 to FF) inclusive were output as UTF-8-encoded > characters - without any offset being added to their codepoints in the > meantime! > > I thought you might just be interested in this - there does seem to be some > method in IDLE's mind, at least. This has nothing to do with IDLE. The UTF-8 encoding of those code points uses two bytes instead of one. See https://stackoverflow.com/questions/8732025/why-degree-symbol-differs-from-utf-8-from-unicode#:~:text=UTF-8%20encodes%20the%20value%200xB0%20as%20two%20consecutive,on%20endianness%20(I%20suppose%20other%20orderings%20are%20possible).coding-in-vs-code-on-ubuntu-leading-to-unicode-error/62652695#62652695 > > Stephen Tucker. > > > > > > > > > On Wed, Jan 18, 2023 at 9:41 AM Peter J. Holzer wrote: > >> On 2023-01-17 22:58:53 -0500, Thomas Passin wrote: >>> On 1/17/2023 8:46 PM, rbowman wrote: >>>> On Tue, 17 Jan 2023 12:47:29 +0000, Stephen Tucker wrote: >>>>> 2. Does the IDLE in Python 3.x behave the same way? >>>> >>>> fwiw >>>> >>>> Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux >>>> Type "help", "copyright", "credits" or "license()" for more >> information. >>>> str = "" >>>> for c in range(140, 169): >>>> str += chr(c) + " " >>>> >>>> print(str) >>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? >>>> ? ? ? >>>> >>>> >>>> I don't know how this will appear since Pan is showing the icon for a >>>> character not in its set. However, even with more undefined characters >>>> the printable one do not change. I get the same output running Python3 >>>> from the terminal so it's not an IDLE thing. >>> >>> I'm not sure what explanation is being asked for here. Let's take >> Python3, >>> so we can be sure that the strings are in unicode. The font being used >> by >>> the console isn't mentioned, but there's no reason it should have glyphs >> for >>> any random unicode character. >> >> Also note that the characters between 128 (U+0080) and 159 (U+009F) >> inclusive aren't printable characters. They are control characters. >> >> hp >> >> -- >> _ | Peter J. Holzer | Story must make more sense than reality. >> |_|_) | | >> | | | hjp at hjp.at | -- Charles Stross, "Creative writing >> __/ | http://www.hjp.at/ | challenge!" >> -- >> https://mail.python.org/mailman/listinfo/python-list >> From hjp-python at hjp.at Wed Jan 18 11:57:46 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Wed, 18 Jan 2023 17:57:46 +0100 Subject: IDLE "Codepage" Switching? In-Reply-To: <64f97adb-e7a5-a912-13d6-f2aff3301f90@tompassin.net> References: <5fa51674-80b2-e64b-7353-1a663d436b29@tompassin.net> <20230118094105.c7rw44bjuvjnu4gu@hjp.at> <64f97adb-e7a5-a912-13d6-f2aff3301f90@tompassin.net> Message-ID: <20230118165746.nfczpfx4tph2eusc@hjp.at> On 2023-01-18 11:05:24 -0500, Thomas Passin wrote: > On 1/18/2023 5:43 AM, Stephen Tucker wrote: > > Thanks for these responses. > > > > I was encouraged to read that I'm not the only one to find this all > > confusing. > > > > I have investigated a little further. > > > > 1. I produced the following IDLE log: > > > > > > > mylongstr = "" > > > > > for thisCP in range (1, 256): > > mylongstr += chr (thisCP) + " " + str (ord (chr (thisCP))) + ", " > > > > > > > > > print mylongstr > > 1, 2, 3, 4, 5, 6, 7, 8, 9, > > 10, 11, 12, > > 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, > > 31, 32, ! 33, " 34, # 35, $ 36, % 37, & 38, ' 39, ( 40, ) 41, * 42, + 43, > > , 44, - 45, . 46, / 47, 0 48, 1 49, 2 50, 3 51, 4 52, 5 53, 6 54, 7 55, 8 > > 56, 9 57, : 58, ; 59, < 60, = 61, > 62, ? 63, @ 64, A 65, B 66, C 67, D 68, > > E 69, F 70, G 71, H 72, I 73, J 74, K 75, L 76, M 77, N 78, O 79, P 80, Q > > 81, R 82, S 83, T 84, U 85, V 86, W 87, X 88, Y 89, Z 90, [ 91, \ 92, ] 93, > > ^ 94, _ 95, ` 96, a 97, b 98, c 99, d 100, e 101, f 102, g 103, h 104, i > > 105, j 106, k 107, l 108, m 109, n 110, o 111, p 112, q 113, r 114, s 115, > > t 116, u 117, v 118, w 119, x 120, y 121, z 122, { 123, | 124, } 125, ~ > > 126, 127, ? 128, ? 129, ? 130, ? 131, ? 132, ? 133, ? 134, ? 135, ? 136, ? > > 137, ? 138, ? 139, ? 140, ? 141, ? 142, ? 143, ? 144, ? 145, ? 146, ? 147, > > ? 148, ? 149, ? 150, ? 151, ? 152, ? 153, ? 154, ? 155, ? 156, ? 157, ? > > 158, ? 159, ? 160, ? 161, ? 162, ? 163, ? 164, ? 165, ? 166, ? 167, ? 168, > > ? 169, ? 170, ? 171, ? 172, ? 173, ? 174, ? 175, ? 176, ? 177, ? 178, ? > > 179, ? 180, ? 181, ? 182, ? 183, ? 184, ? 185, ? 186, ? 187, ? 188, ? 189, > > ? 190, ? 191, ? 192, ? 193, ? 194, ? 195, ? 196, ? 197, ? 198, ? 199, ? > > 200, ? 201, ? 202, ? 203, ? 204, ? 205, ? 206, ? 207, ? 208, ? 209, ? 210, > > ? 211, ? 212, ? 213, ? 214, ? 215, ? 216, ? 217, ? 218, ? 219, ? 220, ? > > 221, ? 222, ? 223, ? 224, ? 225, ? 226, ? 227, ? 228, ? 229, ? 230, ? 231, > > ? 232, ? 233, ? 234, ? 235, ? 236, ? 237, ? 238, ? 239, ? 240, ? 241, ? > > 242, ? 243, ? 244, ? 245, ? 246, ? 247, ? 248, ? 249, ? 250, ? 251, ? 252, > > ? 253, ? 254, ? 255, > > > > > > > > > 2. I copied and pasted the IDLE log into a text file and ran a program on > > it that told me about every byte in the log. > > > > 3. I discovered the following: > > > > Bytes 001 to 127 (01 to 7F hex) inclusive were printed as-is; Which might mean that they are also UTF-8-encoded (there is no difference between UTF-8-encoding and ASCII-encoding for these characters). > > Bytes 128 to 191 (80 to BF) inclusive were output as UTF-8-encoded > > characters whose codepoints were FF00 hex more than the byte values (hence > > the strange glyphs); > > > > Bytes 192 to 255 (C0 to FF) inclusive were output as UTF-8-encoded > > characters - without any offset being added to their codepoints in the > > meantime! > > > > I thought you might just be interested in this - there does seem to be some > > method in IDLE's mind, at least. > > This has nothing to do with IDLE. The UTF-8 encoding of those code points > uses two bytes instead of one. See That's not the peculiar thing. The peculiar thing is that characters U+0080 to U+00BF are recoded to U+FF80 to U+FFBF (but U+00C0 to U+00FF are printed normally). I have no idea what's happening here. I can only urge Stephen to use Python 3.x instead of Python 2.7. Python2 has been deprecated for years has has reached its official end of life 3 years ago. There really shouldn't be any reason to use Python 2.7 any more except reverse-engineering old applications in order to port them to Python 3. In particular, the type "str" is very different in Python2 and Python3. In Python2 it is a sequence of bytes (similar to the Python3 type "bytes") and in Python3 it is a sequence of (Unicode) characters (similar to the Python2 type "unicode"). hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From dankolis at gmail.com Wed Jan 18 13:43:54 2023 From: dankolis at gmail.com (Dan Kolis) Date: Wed, 18 Jan 2023 10:43:54 -0800 (PST) Subject: Improvement to imports, what is a better way ? Message-ID: <0f144325-df35-47fa-a0e6-107991260c14n@googlegroups.com> This program has lots of files and each is well segregated for a concept. the top of each as a heap of imports, all identical. Well the very top has some one of's import os as os import sys as sys import importlib as importlib import datetime as dt from tokenize import maybe import signal from tkinter import filedialog as fd from tkinter.messagebox import showinfo import tkinter as tk from tkinter import ttk from tkinter import font from PIL import Image as pl from PIL import ImageTk as ii # Imports part of this program import aboutTime as tt # Time dates timestamps and the like import avMedia as av # Audio and maybe video 'someday' well definitely lots of TTS text to speech import basicSwitchboard as sc # First switchboard lurking. Kickoff to sequence viewers import bestNotifications as bn # Alerts, modals and notifications of many kinds including TK windows import bitBlitting as bb # Help - # A setup window with basic facts written into files and the workspace import configSetting as cs # A setup window with basic facts written into files and the workspace import devPeeks as dp # Developments peek assets code for adding features and debugging import fileIo as io # File reads, writes even some of the the pickle and unpickling import forFun as ff # Staging area for uncertian features, also some dynamic code inclusion import fractionScreens as fs # Common window pieces for TK windows in sequence viewers import genomeDb as gd # IO to from databases external files gbk and others sure import globalIdeas as gi # Common code for stuff used everywhere and definition of share DOT OBJECTS import insertableEditors as ie # Text editor(s) and support for that, some language issues managed too import modalSimple as mo # Modal windows icp seq ssues, not used much currently import osDependant as od # Operating system dependant calls and adaptions import pickle as pk # Packaging to and from file system for objects especially DOT OBJECTS import screenBuilder1 as sb # Screen maker for TKinter and TKinter ++ windows import selectViewer as sv # Window for picking which sequence viewers are enabled import sequencesMaintain as sm # Touching up temporal version issues of DOT OBJECTS import sequencesOverview as so # Working with sequences as macro facts, lists, see turn off etc import serverHttp1 as sh1 # Servers for HTTP - User Support import serverHttp2 as sh2 # Servers for HTTP - Protein viewing import specialFour as sf # Classes and stuff for viewer four, a very complex viewer import stagingNow as sn # More stabler new ideas getting tried out and moved in import taskingFunction as tf # Tasker for real timeish event processor calls. Has two versions both work import toastMaker as tm # Notifications import transFormations as xf # Sequence transFormations from mother natures ideas like DNA and RNA import unlovedMenus as um # Pretty rarely used ond fashioned menus. A while interface is in here that works import wayOffline as wo # Code not discarded but just studies, etc import windowsX as wx # Windows Life ! X11 esp TK not MS Windoze ... So the file for instance fileIo.py has a duplicate of this. I want to be able to call any code by using the two char prefix form anywhere. ex: io.get_Sequence( 'abc.fa' ) this seems weirdly disorderly. when it starts it sort of iterates down into objects, backs up etc. What's the right way to do this ? Thank you Dan From pbryan at anode.ca Wed Jan 18 14:29:27 2023 From: pbryan at anode.ca (Paul Bryan) Date: Wed, 18 Jan 2023 11:29:27 -0800 Subject: Improvement to imports, what is a better way ? In-Reply-To: <0f144325-df35-47fa-a0e6-107991260c14n@googlegroups.com> References: <0f144325-df35-47fa-a0e6-107991260c14n@googlegroups.com> Message-ID: <4a9214965a630ae3c1175c564b850bd1c6e706b3.camel@anode.ca> I would suggest allowing each module to define its own imports, don't import what a module doesn't consume, keep them simple, avoid devising a common namespace for each, and let tools like isort/black work out how to order/express them in source files. On Wed, 2023-01-18 at 10:43 -0800, Dan Kolis wrote: > This program has lots of files and each is well segregated for a > concept. > > the top of each as a heap of imports, all identical. Well the very > top has some one of's > > import os as??????????????????????? os > import sys as?????????????????????? sys > import importlib as???????????????? importlib > import datetime as????????????????? dt > from?? tokenize import????????????? maybe > import signal > > from?? tkinter import filedialog as fd > from?? tkinter.messagebox import??? showinfo > import tkinter as?????????????????? tk? > from?? tkinter import?????????????? ttk > from?? tkinter import?????????????? font > ? > from?? PIL import Image???? as????? pl > from?? PIL import ImageTk?? as????? ii > > # Imports part of this program > import aboutTime as???????? tt????? # Time dates timestamps and the > like > import avMedia as?????????? av????? # Audio and maybe video 'someday' > well definitely lots of TTS text to speech > import basicSwitchboard as? sc????? # First switchboard lurking. > Kickoff to sequence viewers > import bestNotifications as bn????? # Alerts, modals and > notifications of many kinds including TK windows > import bitBlitting as?????? bb????? # Help - # A setup window with > basic facts written into files and the workspace > import configSetting as???? cs????? # A setup window with basic facts > written into files and the workspace > import devPeeks as????????? dp????? # Developments peek assets code > for adding features and debugging > import fileIo as??????????? io????? # File reads, writes even some of > the the pickle and unpickling > import forFun as??????????? ff????? # Staging area for uncertian > features, also some dynamic code inclusion > import fractionScreens as?? fs????? # Common window pieces for TK > windows in sequence viewers > import genomeDb as????????? gd????? # IO to from databases external > files gbk and others sure > import globalIdeas as?????? gi????? # Common code for stuff used > everywhere and definition of share DOT OBJECTS > import insertableEditors as ie????? # Text editor(s) and support for > that, some language issues managed too > import modalSimple as?????? mo????? # Modal windows icp seq ssues, > not used much currently > import osDependant as?????? od????? # Operating system dependant > calls and adaptions > import pickle as??????????? pk????? # Packaging to and from file > system for objects especially DOT OBJECTS > import screenBuilder1 as??? sb????? # Screen maker for TKinter and > TKinter ++ windows > import selectViewer as????? sv????? # Window for picking which > sequence viewers are enabled > import sequencesMaintain as sm????? # Touching up temporal version > issues of DOT OBJECTS > import sequencesOverview as so????? # Working with sequences as macro > facts, lists, see turn off etc > import serverHttp1 as?????? sh1???? # Servers for HTTP - User Support > import serverHttp2 as?????? sh2???? # Servers for HTTP - Protein > viewing > import specialFour as?????? sf????? # Classes and stuff for viewer > four, a very complex viewer > import stagingNow as??????? sn????? # More stabler new ideas getting > tried out and moved in > import taskingFunction as?? tf????? # Tasker for real timeish event > processor calls. Has two versions both work > import toastMaker as??????? tm????? # Notifications > import transFormations as?? xf????? # Sequence transFormations from > mother natures ideas like DNA and RNA > import unlovedMenus as????? um????? # Pretty rarely used ond > fashioned menus. A while interface is in here that works > import wayOffline as??????? wo????? # Code not discarded but just > studies, etc > import windowsX as????????? wx????? # Windows Life ! X11 esp TK not > MS Windoze ... > > So the file for instance fileIo.py has a duplicate of this. > > I want to be able to call any code by using the two char prefix form > anywhere. ex: io.get_Sequence(? 'abc.fa' ) > > this seems weirdly disorderly. when it starts it sort of iterates > down into objects, backs up etc. > > What's the right way to do this ? > > Thank you > Dan From mats at wichmann.us Wed Jan 18 14:56:39 2023 From: mats at wichmann.us (Mats Wichmann) Date: Wed, 18 Jan 2023 12:56:39 -0700 Subject: Improvement to imports, what is a better way ? In-Reply-To: <4a9214965a630ae3c1175c564b850bd1c6e706b3.camel@anode.ca> References: <0f144325-df35-47fa-a0e6-107991260c14n@googlegroups.com> <4a9214965a630ae3c1175c564b850bd1c6e706b3.camel@anode.ca> Message-ID: <3dc7778d-eb44-9b3f-cf75-77d4a039fb34@wichmann.us> On 1/18/23 12:29, Paul Bryan wrote: > I would suggest allowing each module to define its own imports, don't > import what a module doesn't consume, keep them simple, avoid devising > a common namespace for each, and let tools like isort/black work out > how to order/express them in source files. Indeed. And various checkers will report on unused imports. These add no value... why import something "as" and give it its own name? >> import os as??????????????????????? os >> import sys as?????????????????????? sys >> import importlib as???????????????? importlib A general comment: there are some very common "import ... as" idioms (for example, it seems like *everyone* does "import pandas as pd") and those are okay to follow, but in general I would stay away from trying to give everything short-names. Each module imported with a name other than their own is a memory burden for the reader (maybe even for you!). >> import aboutTime as???????? tt????? # Time dates timestamps and the >> like >> import avMedia as?????????? av????? # Audio and maybe video 'someday' >> well definitely lots of TTS text to speech >> import basicSwitchboard as? sc????? # First switchboard lurking. >> Kickoff to sequence viewers Any decent editor these days will autocomplete for you, so there's really not much if any typing burden in using the full names. From eryksun at gmail.com Wed Jan 18 15:41:01 2023 From: eryksun at gmail.com (Eryk Sun) Date: Wed, 18 Jan 2023 14:41:01 -0600 Subject: IDLE "Codepage" Switching? In-Reply-To: References: Message-ID: On 1/17/23, Stephen Tucker wrote: > > 1. Can anybody explain the behaviour in IDLE (Python version 2.7.10) > reported below? (It seems that the way it renders a given sequence of bytes > depends on the sequence.) In 2.x, IDLE tries to decode a byte string via unicode() before writing to the Tk text widget. However, if the locale encoding (e.g. the process ANSI code page) fails to decode one or more characters, IDLE lets Tk figure out how to decode the byte string. Python 2.7 has an older version of Tk that has peculiar behavior on Windows when bytes in the range 0x80-0xBF are written to a text box. Bytes in this range get translated to native wide characters (16-bit characters) in the halfwidth/fullwidth Unicode block, i.e. translated to Unicode U+FF80 - U+FFBF. If IDLE decodes using code page 1252, then the ordinals 0x81, 0x8d, 0x8f, 0x90 and 0x9d can't be decoded. IDLE thus passes the undecoded byte string to Tk. The example you provided that demonstrates the behavior contains ordinal 0x9d (157). I get similar behavior for the other undefined ordinal values in code page 1252. For example, using IDLE 2.7.18 on Windows: >>> print '\x81\xa1' ?? >>> print 'a\xa1' a? In the first case, ordinal 0x81 causes decoding to fail in IDLE, so the byte string is passed as is to Tk, which maps it to '\uff81\uffa1'. In the second case, OTOH, "\xa1" is decoded by IDLE as "?". > 2. Does the IDLE in Python 3.x behave the same way? No, in 3.x only Unicode str() objects are written to the Tk text widget. Moreover, the text widget doesn't have the same behavior in newer versions. It ignores bytes in the control-block range 0x80-0x9F, and it decodes bytes in the range 0xA0-0xBF normally. From PythonList at DancesWithMice.info Wed Jan 18 15:47:50 2023 From: PythonList at DancesWithMice.info (dn) Date: Thu, 19 Jan 2023 09:47:50 +1300 Subject: Improvement to imports, what is a better way ? In-Reply-To: <3dc7778d-eb44-9b3f-cf75-77d4a039fb34@wichmann.us> References: <0f144325-df35-47fa-a0e6-107991260c14n@googlegroups.com> <4a9214965a630ae3c1175c564b850bd1c6e706b3.camel@anode.ca> <3dc7778d-eb44-9b3f-cf75-77d4a039fb34@wichmann.us> Message-ID: <050354dd-f252-f40d-efbb-aa02fd4a8bba@DancesWithMice.info> On 19/01/2023 08.56, Mats Wichmann wrote: > On 1/18/23 12:29, Paul Bryan wrote: ... >>> import os as??????????????????????? os >>> import sys as?????????????????????? sys >>> import importlib as???????????????? importlib > > A general comment: there are some very common "import ... as" idioms > (for example, it seems like *everyone* does "import pandas as pd") and > those are okay to follow, but in general I would stay away from trying > to give everything short-names.? Each module imported with a name other > than their own is a memory burden for the reader (maybe even for you!). > >>> import aboutTime as???????? tt????? # Time dates timestamps and the >>> like >>> import avMedia as?????????? av????? # Audio and maybe video 'someday' >>> well definitely lots of TTS text to speech >>> import basicSwitchboard as? sc????? # First switchboard lurking. >>> Kickoff to sequence viewers > > Any decent editor these days will autocomplete for you, so there's > really not much if any typing burden in using the full names. Have vacillated on aspects of this topic, over the years. Currently I'm in favor of abbreviations and not only the commonly-accepted/-used ones, eg np for numpy. (but am keen to learn from your wisdom) I've argued the auto-complete cf typing point. So, that not at-issue. The longer an identifier, the more it 'pushes' code over to the right or to expand over multiple screen-lines. Some thoughts on this are behind PEP-008 philosophies, eg line-limit. In the body of the code, if every time an external identifier is used it must be prefixed by a full 'path', the cognitive "burden" shifts from the aspect highlighted (above), to a reading-burden. Thus, (in extreme cases) moving towards a 'wall of text' problem. The primary interest is to remove "burden" aka complexity. In using TDD, code is constructed module-by-module (not necessarily a Python Module). So, when it comes time to call avMedia.fetch_the_file() [sic] there is little thinking about the "avMedia" bit. The emphasis is on the function-name, and it's parameters. 'I need that file so that I can ...'. Using the IDE-as-a-tool argument (similar to above): if I miss-out, mistype, use the wrong abbreviation, or otherwise fail to identify where fetch_the_file() is located, the IDE will immediately tell me. (dn you're wrong - again!) Accordingly, the abbreviation/full-module-name is almost taken-on-trust. (you're going to prove/test the linkage, either way, right?) Personal Biases: - TDD - when starting to write the program[me]-code one of the first steps is to marshal resources, which includes the question: which functions will be called and thus which Python-modules should be imported? Thus, listing all imports 'at the top' is a separable task. - an early language learned (back-when) was COBOL, which has a formal structure and separation between elements/phases of a program[me]'s construction and execution. Some such thinking no doubt lingers... -- Regards, =dn From dankolis at gmail.com Wed Jan 18 15:57:51 2023 From: dankolis at gmail.com (Dan Kolis) Date: Wed, 18 Jan 2023 12:57:51 -0800 (PST) Subject: Improvement to imports, what is a better way ? In-Reply-To: References: <0f144325-df35-47fa-a0e6-107991260c14n@googlegroups.com> <4a9214965a630ae3c1175c564b850bd1c6e706b3.camel@anode.ca> <050354dd-f252-f40d-efbb-aa02fd4a8bba@DancesWithMice.info> <3dc7778d-eb44-9b3f-cf75-77d4a039fb34@wichmann.us> Message-ID: dn and Mats, thanks for your advice. I'm not sure what to do. Do either / both know if there is a way to make it parse each import list to bytecode in one shot ?? The hop around read keeps making me worry it migth leave some memory leak or something. I dont know. Thanks though, both your dialog's are very helpful. Dan From gweatherby at uchc.edu Wed Jan 18 16:11:28 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Wed, 18 Jan 2023 21:11:28 +0000 Subject: Improvement to imports, what is a better way ? In-Reply-To: <050354dd-f252-f40d-efbb-aa02fd4a8bba@DancesWithMice.info> References: <0f144325-df35-47fa-a0e6-107991260c14n@googlegroups.com> <4a9214965a630ae3c1175c564b850bd1c6e706b3.camel@anode.ca> <3dc7778d-eb44-9b3f-cf75-77d4a039fb34@wichmann.us> <050354dd-f252-f40d-efbb-aa02fd4a8bba@DancesWithMice.info> Message-ID: In the body of the code, if every time an external identifier is used it must be prefixed by a full 'path', the cognitive "burden" shifts from the aspect highlighted (above), to a reading-burden. As the fictional Jack McCoy ( https://en.wikipedia.org/wiki/Jack_McCoy) would say -- Objection: Assumes facts not in evidence! If I read A.py: import ? # many lines of code x = obscure_module.widget(4) I know immediately where the function widget is. If I read B.py from obscure_module import widget # many lines of code x = widget(4) Now I have to find out where the heck ?widget? came from And C.py from obscure_module import widget as gadget # many lines of code x = gadget(4) which is actually easier and harder to read. Personally, I?m much happier with A.py than B.py or C.py. From: Python-list on behalf of dn via Python-list Date: Wednesday, January 18, 2023 at 3:49 PM To: python-list at python.org Subject: Re: Improvement to imports, what is a better way ? *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** On 19/01/2023 08.56, Mats Wichmann wrote: > On 1/18/23 12:29, Paul Bryan wrote: ... >>> import os as os >>> import sys as sys >>> import importlib as importlib > > A general comment: there are some very common "import ... as" idioms > (for example, it seems like *everyone* does "import pandas as pd") and > those are okay to follow, but in general I would stay away from trying > to give everything short-names. Each module imported with a name other > than their own is a memory burden for the reader (maybe even for you!). > >>> import aboutTime as tt # Time dates timestamps and the >>> like >>> import avMedia as av # Audio and maybe video 'someday' >>> well definitely lots of TTS text to speech >>> import basicSwitchboard as sc # First switchboard lurking. >>> Kickoff to sequence viewers > > Any decent editor these days will autocomplete for you, so there's > really not much if any typing burden in using the full names. Have vacillated on aspects of this topic, over the years. Currently I'm in favor of abbreviations and not only the commonly-accepted/-used ones, eg np for numpy. (but am keen to learn from your wisdom) I've argued the auto-complete cf typing point. So, that not at-issue. The longer an identifier, the more it 'pushes' code over to the right or to expand over multiple screen-lines. Some thoughts on this are behind PEP-008 philosophies, eg line-limit. In the body of the code, if every time an external identifier is used it must be prefixed by a full 'path', the cognitive "burden" shifts from the aspect highlighted (above), to a reading-burden. Thus, (in extreme cases) moving towards a 'wall of text' problem. The primary interest is to remove "burden" aka complexity. In using TDD, code is constructed module-by-module (not necessarily a Python Module). So, when it comes time to call avMedia.fetch_the_file() [sic] there is little thinking about the "avMedia" bit. The emphasis is on the function-name, and it's parameters. 'I need that file so that I can ...'. Using the IDE-as-a-tool argument (similar to above): if I miss-out, mistype, use the wrong abbreviation, or otherwise fail to identify where fetch_the_file() is located, the IDE will immediately tell me. (dn you're wrong - again!) Accordingly, the abbreviation/full-module-name is almost taken-on-trust. (you're going to prove/test the linkage, either way, right?) Personal Biases: - TDD - when starting to write the program[me]-code one of the first steps is to marshal resources, which includes the question: which functions will be called and thus which Python-modules should be imported? Thus, listing all imports 'at the top' is a separable task. - an early language learned (back-when) was COBOL, which has a formal structure and separation between elements/phases of a program[me]'s construction and execution. Some such thinking no doubt lingers... -- Regards, =dn -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!nUeLMY3g5EV0oaXoGfknoeDOLU9qMoIxHOe7jhHbHfN1v7-wj17OJCgqL_IyHpAr0EN8gyzj5ZEQZB9_A-rB3A_Q5g$ From pbryan at anode.ca Wed Jan 18 16:32:31 2023 From: pbryan at anode.ca (Paul Bryan) Date: Wed, 18 Jan 2023 13:32:31 -0800 Subject: Improvement to imports, what is a better way ? In-Reply-To: <050354dd-f252-f40d-efbb-aa02fd4a8bba@DancesWithMice.info> References: <0f144325-df35-47fa-a0e6-107991260c14n@googlegroups.com> <4a9214965a630ae3c1175c564b850bd1c6e706b3.camel@anode.ca> <3dc7778d-eb44-9b3f-cf75-77d4a039fb34@wichmann.us> <050354dd-f252-f40d-efbb-aa02fd4a8bba@DancesWithMice.info> Message-ID: <16f1d5582f1048798a2c1425f4dc54bd5c5e9ce4.camel@anode.ca> On Thu, 2023-01-19 at 09:47 +1300, dn via Python-list wrote: > The longer an identifier, the more it 'pushes' code over to the right > or? > to expand over multiple screen-lines. Some thoughts on this are > behind > PEP-008 philosophies, eg line-limit. I sympathize with this issue. I've pushed the line limit to 96, which in my case removed this pain point. My argument?though subjective?is that we have wide screens now, and it's quite unlikely production code will be accessed through a vintage VT-100 terminal or printed on a 80- column dot-matrix printer. That said, even if you stick to PEP-8 proper, `black` does most of the formatting work, and it seems to do a reasonable job despite longer module identifiers. > In the body of the code, if every time an external identifier is used > it > must be prefixed by a full 'path', the cognitive "burden" shifts from > the aspect highlighted (above), to a reading-burden. Thus, (in > extreme > cases) moving towards a 'wall of text' problem. I tend to agree this can be a problem in some cases, but I think those can be made manageable. For brevity's sake, I import often-used data types directly into the module's name space. It's far more rare that I try to abbreviate a module name; common ones like importing pandas as pd is a good counterexample. > In using TDD, code is constructed module-by-module (not necessarily > a? > Python Module). So, when it comes time to call > avMedia.fetch_the_file() > [sic] there is little thinking about the "avMedia" bit. The emphasis > is > on the function-name, and it's parameters. 'I need that file so that > I > can ...'. I'm not entirely sure what you mean by not a Python module. Even in test modules, I would only import what's required. If it turns out `fetch_the_file` is called dozens of times or more, I would have no qualm about importing it directly into the test module's namespace so that it doesn't need to be prefixed at all. From cs at cskk.id.au Wed Jan 18 16:34:03 2023 From: cs at cskk.id.au (Cameron Simpson) Date: Thu, 19 Jan 2023 08:34:03 +1100 Subject: Improvement to imports, what is a better way ? In-Reply-To: References: Message-ID: On 18Jan2023 12:57, Dan Kolis wrote: >I'm not sure what to do. Do either / both know if there is a way to >make it parse each import list to bytecode in one shot ?? Python usually compiles files to bytecode and leaves those around as .pyc files to be used until the original source is modified. And anyway, an imported module is only loaded a ready once; imports are kept track of in the sys.modules dict, and just pulled directly from there when the same module is requests by another import. >The hop around read keeps making me worry it migth leave some memory >leak or something. I don't think you've described this. I don't know what you mean here. Cheers, Cameron Simpson From dankolis at gmail.com Wed Jan 18 16:40:13 2023 From: dankolis at gmail.com (Dan Kolis) Date: Wed, 18 Jan 2023 13:40:13 -0800 (PST) Subject: Improvement to imports, what is a better way ? In-Reply-To: References: <0f144325-df35-47fa-a0e6-107991260c14n@googlegroups.com> <4a9214965a630ae3c1175c564b850bd1c6e706b3.camel@anode.ca> <3dc7778d-eb44-9b3f-cf75-77d4a039fb34@wichmann.us> <16f1d5582f1048798a2c1425f4dc54bd5c5e9ce4.camel@anode.ca> <050354dd-f252-f40d-efbb-aa02fd4a8bba@DancesWithMice.info> Message-ID: Most of the time the newbie I visualize adding a function copies and pastes maybe a screen of python and changes it a little, maybe 5% of the lines have a xx. code so they rarely have to understand it, thats not the part of the program there changing. Also, I suffered long and hard for extreme uniformity in the names and so on, there utterly uniform, so there not so bad to consider, ( I hope ). I guess I don't full understand what bothers me about the repetition of the imports so much. The tracing of it seems so bizarre, it just seems like its wrong. Regs Dan From dankolis at gmail.com Wed Jan 18 16:42:23 2023 From: dankolis at gmail.com (Dan Kolis) Date: Wed, 18 Jan 2023 13:42:23 -0800 (PST) Subject: Improvement to imports, what is a better way ? In-Reply-To: References: Message-ID: > I don't think you've described this. I don't know what you mean here. When I trace it in VSCode the imports seem like they endlessly suspend scanning and go to other ones over and over. Like "Whats this doing ?" From greg.ewing at canterbury.ac.nz Wed Jan 18 17:29:54 2023 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 19 Jan 2023 11:29:54 +1300 Subject: Improvement to imports, what is a better way ? In-Reply-To: References: <0f144325-df35-47fa-a0e6-107991260c14n@googlegroups.com> <4a9214965a630ae3c1175c564b850bd1c6e706b3.camel@anode.ca> <3dc7778d-eb44-9b3f-cf75-77d4a039fb34@wichmann.us> <16f1d5582f1048798a2c1425f4dc54bd5c5e9ce4.camel@anode.ca> <050354dd-f252-f40d-efbb-aa02fd4a8bba@DancesWithMice.info> Message-ID: On 19/01/23 10:40 am, Dan Kolis wrote: > I guess I don't full understand what bothers me about the repetition of the imports so much. It's doubtful that every module uses every one of those imports. It looks like someone had a standard block of imports that they blindly pasted at the top of every file, to avoid having to think about which ones were actually needed. If you're looking for advice, I would suggest: * Only import what you use in a particular file (this doesn't have much effect on efficiency, but there's less clutter for the reader.) * Only use short names for modules that are *very* frequently referenced. Use full unabbreviated names for everything else. * Don't 'import foo as foo', just 'import foo'. * Don't try to line the code up in columns, it doesn't really help readability IMO. -- Greg From torriem at gmail.com Wed Jan 18 17:47:51 2023 From: torriem at gmail.com (Michael Torrie) Date: Wed, 18 Jan 2023 15:47:51 -0700 Subject: Improvement to imports, what is a better way ? In-Reply-To: References: Message-ID: <9fa0ad06-3c36-7c3c-63e2-fbc6ae7008a8@gmail.com> On 1/18/23 14:42, Dan Kolis wrote: > >> I don't think you've described this. I don't know what you mean here. > > When I trace it in VSCode the imports seem like they endlessly suspend scanning and go to other ones over and over. Like "Whats this doing ?" > Nothing to worry about there. Python knows what it's doing! :) Lots of modules import other modules, so there will be a spanning tree of sorts. Each module will only actually be formally imported once. The rest is just setting up name spaces. Yes it all adds to run time latency, but it certainly won't lead to any leaks. Definitely there's no need to import a module's dependencies; it will do that itself. Just import what your own module explicitly needs. From dankolis at gmail.com Wed Jan 18 17:41:12 2023 From: dankolis at gmail.com (Dan Kolis) Date: Wed, 18 Jan 2023 14:41:12 -0800 (PST) Subject: Improvement to imports, what is a better way ? In-Reply-To: References: <0f144325-df35-47fa-a0e6-107991260c14n@googlegroups.com> <4a9214965a630ae3c1175c564b850bd1c6e706b3.camel@anode.ca> <3dc7778d-eb44-9b3f-cf75-77d4a039fb34@wichmann.us> <16f1d5582f1048798a2c1425f4dc54bd5c5e9ce4.camel@anode.ca> <050354dd-f252-f40d-efbb-aa02fd4a8bba@DancesWithMice.info> Message-ID: <8a1e4301-0aa4-436f-9434-ca5ef80a96a3n@googlegroups.com> Thank you From dankolis at gmail.com Wed Jan 18 19:10:15 2023 From: dankolis at gmail.com (Dan Kolis) Date: Wed, 18 Jan 2023 16:10:15 -0800 (PST) Subject: Improvement to imports, what is a better way ? In-Reply-To: References: <9fa0ad06-3c36-7c3c-63e2-fbc6ae7008a8@gmail.com> Message-ID: I have written a very sizable and elaborate program that uses tKinter for X11 displays of genomics. Yet maybe 1 of 6 times it freezes, so I decided to extract the minimum that works perfectly and add back big pieces. It does it both running .pyc and in VSCode. so even the most minor of oddities is suspect. I cant really think of any better way to debug it. I see no evidence in WWW searches its a generic problem. So tracing the startup and watching it hop all around is about one of five lesser problem line items. Regs Dan From dankolis at gmail.com Wed Jan 18 20:01:01 2023 From: dankolis at gmail.com (Dan Kolis) Date: Wed, 18 Jan 2023 17:01:01 -0800 (PST) Subject: A natural magnet for the craziest TKinter lovers out there Message-ID: Hangs after maybe between 4 and 50 screen rewrites. sometimes CTRL C under Ubuntu starts it up again. Click go rewrites al the fonts the thing can find in a few windows.... Repeated. TKinter has a cult following. I don''t really expect others to fix my problems, but you know, es since its a cult thing to love TCL ( or something .... ) Here is is. Same on two different computers in Python 3.6 and 3.8 Plus I suppose you get to see for sure all your finest fonts as rendered in a real TK thing, not that this changes the worlds so much. Thanks for the advice so far... Regards, Dan Kolis # Code version 18Jan2023 19:48 EST Dan Kolis # Click 'go' on main page. when run over and over after a while it freezes import tkinter as tk from tkinter import ttk from tkinter import font as luFont # Empty object maker ( M T ) ... get it say it ! class MT(): pass # Module level variables mod = MT() mod.itsPlaces = MT() mod.itsPlaces = [] # Apply XY Size to things like windows, variable place 31Jul22 def apply_Width_Height_Position( toThis, aWidth, aHeight, *argsPlace ): # Width and height is expected # Argument overflow is quite variable size is it its usual hint for me aLen = len( argsPlace ) # Sometimes, just width and height are given ( not that + position ) wAndH = str( int( aWidth ) ) +'x' + str( int( aHeight ) ) # Size has a funny text line interface toThis.geometry( wAndH ) # Apply # Sometimes position too if aLen == 2: eP = "+" + str( argsPlace[ 0 ] ) + "+" + str( argsPlace[ 1 ] ) param4 = wAndH + eP # Combine size and position toThis.geometry( param4 ) # Apply jjjjjj = 560 # Create local toplevel to whoever calls me 11Dec22 bc-endive def make_Tk_Window( sReason, *attribKc ): # Top level window maker and rememberer # Kc = 'Kind code' aCode = "na" # Not available if attribKc: aCode = attribKc [ 0 ] # Better code then 'na' # Make window ansWin = tk.Toplevel() # Keep track of this creation step newEntry = {} newEntry[ 'tkaddress' ] = ansWin # Real TK window accessor newEntry[ 'ideanamed' ] = sReason # Casual description newEntry[ 'kindcode' ] = aCode # So can be worked with together mod.itsPlaces.append( newEntry ) return ansWin # Idle time rewrite def justRedisplayNow( wThings, *argsR ): # Redisplay a named tj thing # Dont use this frivolously, if it's needed its here you have to study it wThings.update() # Look at fonts possible 6Oct22 def showAllFonts(): # Show fonts possible mod.lines_In_Window = 30; mod.window_Seq_Number = 1 # What we want to see mod.seeFonts = list( luFont.families() ) mod.seeFonts.sort() # Alpha order is nice # Fill windows as a look at me report try: populateW() except: tttttt = 456 rrrrrr = 245 # Show fonts regarding avoiding BAD ones that crash, etc size limits in buffers too 19Dec22 def populateW(): # Put in the fonts aLen = len( mod.seeFonts ) # Fun to know, debug really last_considered_name = "gghfjhljfl" # Don't show doubles # Loop through all fonts the system named thisPassMakeWindow = True; a_Line_Count = 0; list_Number = 0 for item in mod.seeFonts: # Multiple windows ( Python has funny ideas on this, ok like this ) if thisPassMakeWindow: winTk_Now = winMakerFontsW(); thisPassMakeWindow = False; a_Line_Count = 0 # We gots one now if a_Line_Count < mod.lines_In_Window: # Will a window we made list_Number += 1 # Counts aN = str( list_Number ) + " "; name_In_lower = item.lower() # Various reasons to not render this one isOk = True if name_In_lower == last_considered_name: isOk = False a_Bad_Thing = name_In_lower.find( 'emoji' ) > -1 if a_Bad_Thing: isOk = False a_Bad_Thing = item.find( 'Noto Sans Mono CJK' ) > -1 if a_Bad_Thing: isOk = False # Good good if isOk: label = "listlabel" + str( list_Number ) label = tk.Label( winTk_Now, text=aN + item, font=( item, 18 ), anchor='w' ).pack() a_Line_Count += 1 last_considered_name = name_In_lower else: justRedisplayNow( winTk_Now ) thisPassMakeWindow = True; mod.window_Seq_Number += 1 # New window next time # Done a_Msg = "on this machine" # Look at fonts possible 6Oct22 def showAllFonts(): # Show fonts possible mod.lines_In_Window = 30; mod.window_Seq_Number = 1 # What we want to see mod.seeFonts = list( luFont.families() ) mod.seeFonts.sort() # Alpha order is nice # Fill windows as a look at me report try: populateW() except: tttttt = 456 rrrrrr = 245 # Just window shape aka size def applyWidthHeight( toThis, aWidth, aHeight ): # TK window size no change in code anyway, to placement # Op sys like thing likes this x that numbers wAndH = str( int( aWidth ) ) +'x' + str( int( aHeight ) ) # Size toThis.geometry( wAndH ) # Apply tttttt = 385 # Window making def winMakerFontsW(): # Window with scroll bars, etc canvassy, ready to fill up # The real window gets created mInf = ' ' + str( mod.window_Seq_Number ) wsFr = make_Tk_Window( 'Font peek' + mInf, 'fontpeek' ); wsFr.title( 'Fonts' + mInf ) start_Hor = 50; Kh = 110; start_Ver = 35; Kv = 40 moved_Some_H = start_Hor + ( mod.window_Seq_Number * ( Kh - 1 ) ) moved_Some_V = start_Ver + ( mod.window_Seq_Number * ( Kv - 1 ) ) apply_Width_Height_Position( wsFr, 480, 500, moved_Some_H, moved_Some_V ) # Canvas is the form pretty much my_Canvas = tk.Canvas( wsFr, borderwidth=0 ) myFrm = tk.Frame( my_Canvas ) # Now the 'do it on me' thing # Vertical scroll bars go onto form vsb = tk.Scrollbar( wsFr, orient="vertical", command=my_Canvas.yview ) my_Canvas.configure( yscrollcommand=vsb.set ) # Assemble via relations of pieces vsb.pack( side="right", fill="y" ) my_Canvas.pack( side="left", fill="both", expand=True ) my_Canvas.create_window( (4, 4 ), window=myFrm, anchor="nw" ) return myFrm def callIt(): # A click do it showAllFonts() # This is main root = tk.Tk() applyWidthHeight( root, 300, 300 ) b1 = tk.Button( root, text='go', command=callIt ) b1.grid( row=0, column=0 ) root.mainloop() From torriem at gmail.com Wed Jan 18 20:56:48 2023 From: torriem at gmail.com (Michael Torrie) Date: Wed, 18 Jan 2023 18:56:48 -0700 Subject: A natural magnet for the craziest TKinter lovers out there In-Reply-To: References: Message-ID: <48cf44a7-3c91-0354-f184-ebd7374d5178@gmail.com> On 1/18/23 18:01, Dan Kolis wrote: > Hangs after maybe between 4 and 50 screen rewrites. sometimes CTRL C under Ubuntu starts it up again. Click go rewrites al the fonts the thing can find in a few windows.... Repeated. > Not sure what you mean by "screen rewrites." I ran your test program here and it generates 25 windows on my machine, and I can click "run" at least half a dozen times. I tried closing the font windows before clicking run again, and also just leaving the windows up and generating many more windows. 300 windows. No hangs here at all. Fedora 35 with Mate Desktop on X11 with compositing enabled. From list1 at tompassin.net Wed Jan 18 23:46:38 2023 From: list1 at tompassin.net (Thomas Passin) Date: Wed, 18 Jan 2023 23:46:38 -0500 Subject: A natural magnet for the craziest TKinter lovers out there In-Reply-To: <48cf44a7-3c91-0354-f184-ebd7374d5178@gmail.com> References: <48cf44a7-3c91-0354-f184-ebd7374d5178@gmail.com> Message-ID: <11c2901f-a01d-f7f9-727a-5baef404a7ad@tompassin.net> On 1/18/2023 8:56 PM, Michael Torrie wrote: > On 1/18/23 18:01, Dan Kolis wrote: >> Hangs after maybe between 4 and 50 screen rewrites. sometimes CTRL C under Ubuntu starts it up again. Click go rewrites al the fonts the thing can find in a few windows.... Repeated. >> > > Not sure what you mean by "screen rewrites." > > I ran your test program here and it generates 25 windows on my machine, > and I can click "run" at least half a dozen times. I tried closing the > font windows before clicking run again, and also just leaving the > windows up and generating many more windows. 300 windows. No hangs here > at all. Fedora 35 with Mate Desktop on X11 with compositing enabled. It could be using up some system resource. tk could have some memory or resource leak, possibly specific to the OP's machine. Try looking at system memory with the system monitor as the program goes through its paces. Maybe you will notice something suspicious. From cs at cskk.id.au Thu Jan 19 00:08:32 2023 From: cs at cskk.id.au (Cameron Simpson) Date: Thu, 19 Jan 2023 16:08:32 +1100 Subject: Improvement to imports, what is a better way ? In-Reply-To: References: Message-ID: On 18Jan2023 16:10, Dan Kolis wrote: >I have written a very sizable and elaborate program that uses tKinter for X11 displays of genomics. >Yet maybe 1 of 6 times it freezes, so I decided to extract the minimum >that works perfectly and add back big pieces. It does it both running >.pyc and in VSCode. The other thing you can do is put in a heap of print() calls in strategic places. See if you can get a handle on what its doing when it freezes. This might be iterative, zooming in on where this happens, if it's a specific thing. I know this is vague. Once you find its stalling in a particular function (if it is) you may be able to run that function directly. Also, a print() at the top abd bottom/return of the stalling function. And so on. If the GUI freezes, maybe you're running something time consuming inline in the event loop, such that it blocks reaching the "idle" point where the tk gui updates widgets and polls for user actions. Cheers, Cameron Simpson From list1 at tompassin.net Thu Jan 19 00:41:42 2023 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 19 Jan 2023 00:41:42 -0500 Subject: A natural magnet for the craziest TKinter lovers out there In-Reply-To: <11c2901f-a01d-f7f9-727a-5baef404a7ad@tompassin.net> References: <48cf44a7-3c91-0354-f184-ebd7374d5178@gmail.com> <11c2901f-a01d-f7f9-727a-5baef404a7ad@tompassin.net> Message-ID: <12454c35-0487-975f-52e1-b98e34654540@tompassin.net> On 1/18/2023 11:46 PM, Thomas Passin wrote: > On 1/18/2023 8:56 PM, Michael Torrie wrote: >> On 1/18/23 18:01, Dan Kolis wrote: >>> Hangs after maybe between 4 and 50 screen rewrites. sometimes CTRL C >>> under Ubuntu starts it up again. Click go rewrites al the fonts the >>> thing can find in a few windows.... Repeated. >>> >> >> Not sure what you mean by "screen rewrites." >> >> I ran your test program here and it generates 25 windows on my machine, >> and I can click "run" at least half a dozen times. I tried closing the >> font windows before clicking run again, and also just leaving the >> windows up and generating many more windows.? 300 windows. No hangs here >> at all. Fedora 35 with Mate Desktop on X11 with compositing enabled. > > It could be using up some system resource.? tk could have some memory or > resource leak, possibly specific to the OP's machine.? Try looking at > system memory with the system monitor as the program goes through its > paces.? Maybe you will notice something suspicious. > Also reduce the number of windows that get opened to a small number. If there are no freezes, increase the number little by little. This may give you some bounds on what's going on, or at least let your ultimate program work by keeping the resource use under control. From gweatherby at uchc.edu Thu Jan 19 07:34:00 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Thu, 19 Jan 2023 12:34:00 +0000 Subject: A natural magnet for the craziest TKinter lovers out there In-Reply-To: <48cf44a7-3c91-0354-f184-ebd7374d5178@gmail.com> References: <48cf44a7-3c91-0354-f184-ebd7374d5178@gmail.com> Message-ID: Works fine on my work machine. (Ubuntu 20.04 / 32 G / 32 CPUS). Scalene (https://github.com/plasma-umass/scalene) shows it using 9 MB of memory. From: Python-list on behalf of Michael Torrie Date: Wednesday, January 18, 2023 at 8:58 PM To: python-list at python.org Subject: Re: A natural magnet for the craziest TKinter lovers out there *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** On 1/18/23 18:01, Dan Kolis wrote: > Hangs after maybe between 4 and 50 screen rewrites. sometimes CTRL C under Ubuntu starts it up again. Click go rewrites al the fonts the thing can find in a few windows.... Repeated. > Not sure what you mean by "screen rewrites." I ran your test program here and it generates 25 windows on my machine, and I can click "run" at least half a dozen times. I tried closing the font windows before clicking run again, and also just leaving the windows up and generating many more windows. 300 windows. No hangs here at all. Fedora 35 with Mate Desktop on X11 with compositing enabled. -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!iGQaGdd9t2QPVQJDhyDTQmXgCwx4YG7m8jFbRd11dF1BweJHCd0wC4UyZUfQGQjqxeulMYagnDqInfA81A$ From list1 at tompassin.net Thu Jan 19 10:06:43 2023 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 19 Jan 2023 10:06:43 -0500 Subject: A natural magnet for the craziest TKinter lovers out there In-Reply-To: References: <48cf44a7-3c91-0354-f184-ebd7374d5178@gmail.com> Message-ID: Works fine through 10 "go" button presses on my Windows 10 machine. You might want to run pylint and pyflakes on it On 1/19/2023 7:34 AM, Weatherby,Gerard wrote: > Works fine on my work machine. (Ubuntu 20.04 / 32 G / 32 CPUS). Scalene (https://github.com/plasma-umass/scalene) shows it using 9 MB of memory. > > From: Python-list on behalf of Michael Torrie > Date: Wednesday, January 18, 2023 at 8:58 PM > To: python-list at python.org > Subject: Re: A natural magnet for the craziest TKinter lovers out there > *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** > > On 1/18/23 18:01, Dan Kolis wrote: >> Hangs after maybe between 4 and 50 screen rewrites. sometimes CTRL C under Ubuntu starts it up again. Click go rewrites al the fonts the thing can find in a few windows.... Repeated. >> > > Not sure what you mean by "screen rewrites." > > I ran your test program here and it generates 25 windows on my machine, > and I can click "run" at least half a dozen times. I tried closing the > font windows before clicking run again, and also just leaving the > windows up and generating many more windows. 300 windows. No hangs here > at all. Fedora 35 with Mate Desktop on X11 with compositing enabled. > > -- > https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!iGQaGdd9t2QPVQJDhyDTQmXgCwx4YG7m8jFbRd11dF1BweJHCd0wC4UyZUfQGQjqxeulMYagnDqInfA81A$ From list1 at tompassin.net Thu Jan 19 11:24:38 2023 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 19 Jan 2023 11:24:38 -0500 Subject: A natural magnet for the craziest TKinter lovers out there In-Reply-To: References: <48cf44a7-3c91-0354-f184-ebd7374d5178@gmail.com> Message-ID: Works through 20 presses of the "go" button on a Linux Mint VM. Because of limited RAM allocated for the VM, after some iterations the program slowed down because the VM had to start using swap memory. But there did not seem to be any glitches or failures. Python 3.9.5 Linux Mint 20 Ulyana /Cinnamon 3.07 GB Linux kernel 5.4.0-137-generic tk: 0.1.0 On 1/19/2023 10:06 AM, Thomas Passin wrote: > Works fine through 10 "go" button presses on my Windows 10 machine.? You > might want to run pylint and pyflakes on it > > On 1/19/2023 7:34 AM, Weatherby,Gerard wrote: >> Works fine on my work machine. (Ubuntu 20.04 / 32 G / 32 CPUS). >> Scalene (https://github.com/plasma-umass/scalene) shows it using 9 MB >> of memory. >> >> From: Python-list >> on behalf of Michael Torrie >> Date: Wednesday, January 18, 2023 at 8:58 PM >> To: python-list at python.org >> Subject: Re: A natural magnet for the craziest TKinter lovers out there >> *** Attention: This is an external email. Use caution responding, >> opening attachments or clicking on links. *** >> >> On 1/18/23 18:01, Dan Kolis wrote: >>> Hangs after maybe between 4 and 50 screen rewrites. sometimes CTRL C >>> under Ubuntu starts it up again. Click go rewrites al the fonts the >>> thing can find in a few windows.... Repeated. >>> >> >> Not sure what you mean by "screen rewrites." >> >> I ran your test program here and it generates 25 windows on my machine, >> and I can click "run" at least half a dozen times. I tried closing the >> font windows before clicking run again, and also just leaving the >> windows up and generating many more windows.? 300 windows. No hangs here >> at all. Fedora 35 with Mate Desktop on X11 with compositing enabled. >> >> -- >> https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!iGQaGdd9t2QPVQJDhyDTQmXgCwx4YG7m8jFbRd11dF1BweJHCd0wC4UyZUfQGQjqxeulMYagnDqInfA81A$ > From dankolis at gmail.com Thu Jan 19 10:21:22 2023 From: dankolis at gmail.com (Dan Kolis) Date: Thu, 19 Jan 2023 07:21:22 -0800 (PST) Subject: A natural magnet for the craziest TKinter lovers out there In-Reply-To: References: <48cf44a7-3c91-0354-f184-ebd7374d5178@gmail.com> Message-ID: <5b7eb064-c002-4851-bd77-d77001e7d332n@googlegroups.com> Hello ! > Works fine on my work machine. (Ubuntu 20.04 / 32 G / 32 CPUS). Scalene (https://github.com/plasma-umass/scalene) shows it using 9 MB of memory. > I ran your test program here and it generates 25 windows on my machine, > and I can click "run" at least half a dozen times. I tried closing the > font windows before clicking run again, and also just leaving the > windows up and generating many more windows. 300 windows. No hangs here > at all. Fedora 35 with Mate Desktop on X11 with compositing enabled. Thanks a lot These reports are very helpful ! I seemed to have 'fixed it' by changing one line, really. I used: # Do each thing ..for aWs in workWsL: ....aWs.update() TO: # Do each thing ..for aWs in workWsL: ....aWs.update_idletasks() Dan says: Thanks a lot ! This helps me visualise this is managed as a problem in a technical sense. I mean, there is always a requirement for real integration testing of all sorts for an attempt to release a program on a larger scale. Now I know it works without stopping on 4 computers. Better then yesterday hugely. Thank you. Regs Daniel B. Kolis my ref: nafl, 19 Jan 2023, https://groups.google.com/g/comp.lang.python/c/FNlXg0Od39o/m/9stiUtLSAQAJ From dankolis at gmail.com Thu Jan 19 10:34:11 2023 From: dankolis at gmail.com (Dan Kolis) Date: Thu, 19 Jan 2023 07:34:11 -0800 (PST) Subject: Improvement to imports, what is a better way ? In-Reply-To: References: Message-ID: On Thursday, January 19, 2023 at 12:09:02 AM UTC-5, cameron wrote: > I know this is vague. Once you find its stalling in a particular > function (if it is) you may be able to run that function directly. Also, > a print() at the top abd bottom/return of the stalling function. And so > on. Dan says: After lots of iterations, I changed one line in a call used to refresh windows from .update() to .update_idle_tasks() Some other dudes were so nice they tested it before that, it worked perfectly on their computers anyway. Now it seems to work 'all the way' too after this change on my favroritish computer. Since a CTRL C often started it going again I had a hint there, reading other peoples various troubles all over the web made me think it was 'someplace in this part'. Regards, Daniel B. Kolis From dankolis at gmail.com Thu Jan 19 10:43:26 2023 From: dankolis at gmail.com (Dan Kolis) Date: Thu, 19 Jan 2023 07:43:26 -0800 (PST) Subject: Python - working with xml/lxml/objectify/schemas, datatypes, and assignments In-Reply-To: References: <25525.36948.543184.270051@ixdm.fritz.box> <25534.65042.617254.20506@ixdm.fritz.box> Message-ID: Editing text intended primarily for machine reading that involves metadata and lower level facts is a horror show. I sort of worked for a company years ago and a smart ass suggested I was making labor for myself by doing changes to a scripting language for db users, maybe a few hours a week. He suggested I leave them to do it themselves in XML. I tried it and the community went ape-shit angry. NONE of the people succeeded routinely at all. Just because machine readable nomenclatures as well known and some s/w edits them, doesn't mean there suddenly not mostly mini computer programs. I suspect there isn't an easy way out, and probably the thing your making has to be 100% usably done before a maintenance tools can be created to make it easy, anyway. Its the way it is, maybe, Daniel B. Kolis From roel at roelschroeven.net Thu Jan 19 11:55:46 2023 From: roel at roelschroeven.net (Roel Schroeven) Date: Thu, 19 Jan 2023 17:55:46 +0100 Subject: Improvement to imports, what is a better way ? In-Reply-To: References: <0f144325-df35-47fa-a0e6-107991260c14n@googlegroups.com> <4a9214965a630ae3c1175c564b850bd1c6e706b3.camel@anode.ca> <3dc7778d-eb44-9b3f-cf75-77d4a039fb34@wichmann.us> <050354dd-f252-f40d-efbb-aa02fd4a8bba@DancesWithMice.info> Message-ID: <5ebee5b8-83b6-31a7-54a7-79611ef3fc5a@roelschroeven.net> Op 19/01/2023 om 11:32 schreef Stefan Ram: > dn writes: > >The longer an identifier, the more it 'pushes' code over to the right or > >to expand over multiple screen-lines. Some thoughts on this are behind > >PEP-008 philosophies, eg line-limit. > > Raymond Hettinger (transcribed, shortened and partially > paraphrased by me [Stefan Ram]): > > |The line-width part of PEP 8 bugs me. > | > |You have to wrap your commits in seventy-two characters. You have > |to end your lines at seventy-nine characters. > | > |One time it bugs me is when I'm writing unit tests. > | > |When I write unit tests, I have to start with a class, and then > |inside the class there's a "def" for tests, and then the test > |starts with a "self.assertEqual", and by then most of my line > |is gone. So by the time I get to any business logic in my test, > |I'm near the end of the line. > | > |If I go over seventy-nine characters, somebody will come and > |PEP 8 me. > | > |They'll come in and say: "Oh, Raymond's line hit eighty-one > |characters, I'm going to PEP 8 it!". And so, while I'm not > |looking, they come in and reformat my code. > | > |They'll just throw a line break at a really awkward place. > | > |Does that make the code better? > | > |So, to escape that pressure, I think: Maybe I can just commit > |a little atrocity and that way no one will ever come and PEP 8 me. > |I'll just shorten my variable names. > | > |Does that make the code better? > | > freely adapted from Raymond Hettinger He then goes on to say he uses 90-ish, IIRC: he tries to stay under that, but doesn' t mine if a line goes a bit over. Or at least that's what I remember from that talk, I might be wrong. -- "If you don't read the newspaper, you're uninformed. If you read the newspaper, you're mis-informed." -? Onbekend (dikwijls toegeschreven aan Mark Twain, waarschijnlijk onterecht) From list1 at tompassin.net Thu Jan 19 12:59:21 2023 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 19 Jan 2023 12:59:21 -0500 Subject: Improvement to imports, what is a better way ? In-Reply-To: <5ebee5b8-83b6-31a7-54a7-79611ef3fc5a@roelschroeven.net> References: <0f144325-df35-47fa-a0e6-107991260c14n@googlegroups.com> <4a9214965a630ae3c1175c564b850bd1c6e706b3.camel@anode.ca> <3dc7778d-eb44-9b3f-cf75-77d4a039fb34@wichmann.us> <050354dd-f252-f40d-efbb-aa02fd4a8bba@DancesWithMice.info> <5ebee5b8-83b6-31a7-54a7-79611ef3fc5a@roelschroeven.net> Message-ID: <07805baa-325b-69c2-b691-b46b10550a01@tompassin.net> On 1/19/2023 11:55 AM, Roel Schroeven wrote: > Op 19/01/2023 om 11:32 schreef Stefan Ram: >> dn writes: >> >The longer an identifier, the more it 'pushes' code over to the right >> or >to expand over multiple screen-lines. Some thoughts on this are >> behind >PEP-008 philosophies, eg line-limit. >> >> ?? Raymond Hettinger (transcribed, shortened and partially >> ?? paraphrased by me [Stefan Ram]): >> >> |The line-width part of PEP 8 bugs me. >> | >> |You have to wrap your commits in seventy-two characters. You have >> |to end your lines at seventy-nine characters. >> | >> |One time it bugs me is when I'm writing unit tests. >> | >> |When I write unit tests, I have to start with a class, and then >> |inside the class there's a "def" for tests, and then the test >> |starts with a "self.assertEqual", and by then most of my line >> |is gone. So by the time I get to any business logic in my test, >> |I'm near the end of the line. >> | >> |If I go over seventy-nine characters, somebody will come and >> |PEP 8 me. [snip] Well, it's an art, not a science. Very long lines are hard to read. Multi-line "lines" can be hard because we are all used to taking in a line of code at a time. And some things you can't shorten, like a long URL. Personally I prefer shorter lines, even shorter than 72 characters. But splitting a long line of code while still making it easy to grasp is an art in itself. Some constructions do lend themselves nicely to splitting into several lines (I hope they don't get wrapped by your email reader!): # Create a plot g2 = ( ggplot(df2, aes('Days Since Jan 22', # Comments can clarify these params + geom_point(size=.1, color='blue') # size, color params optional + theme_bw() # Optional theme (background, grid color, ...) ) # Compose a long string: msg = ('A very long line .....\n' + 'Another long bit of text ....' + 'plus another ....' ) For the second example, I will sometimes tokenize it: M1A = 'A very long line .....\n' M1B = 'Another long bit of text ....' M1C = 'plus another ....' msg = M1A + M1B + M1C But I usually find the first form easier to grasp, so I tend to use it more - and I don't have to invent throwaway variable names. The PEP-8 rules are good, but they can't cover all cases perfectly. From list1 at tompassin.net Thu Jan 19 12:24:00 2023 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 19 Jan 2023 12:24:00 -0500 Subject: A natural magnet for the craziest TKinter lovers out there In-Reply-To: <5b7eb064-c002-4851-bd77-d77001e7d332n@googlegroups.com> References: <48cf44a7-3c91-0354-f184-ebd7374d5178@gmail.com> <5b7eb064-c002-4851-bd77-d77001e7d332n@googlegroups.com> Message-ID: <9733b3d6-68c9-0244-493d-446ee38e8ff9@tompassin.net> On 1/19/2023 10:21 AM, Dan Kolis wrote: > Hello ! > >> Works fine on my work machine. (Ubuntu 20.04 / 32 G / 32 CPUS). Scalene (https://github.com/plasma-umass/scalene) shows it using 9 MB of memory. > >> I ran your test program here and it generates 25 windows on my machine, >> and I can click "run" at least half a dozen times. I tried closing the >> font windows before clicking run again, and also just leaving the >> windows up and generating many more windows. 300 windows. No hangs here >> at all. Fedora 35 with Mate Desktop on X11 with compositing enabled. > > > Thanks a lot These reports are very helpful ! > > I seemed to have 'fixed it' by changing one line, really. I used: > > # Do each thing > ..for aWs in workWsL: > ....aWs.update() > > > TO: > # Do each thing > ..for aWs in workWsL: > ....aWs.update_idletasks() I was going to suggest update_idletasks(), but I forgot to scan your code to see it it was already being done. > Dan says: > Thanks a lot ! This helps me visualise this is managed as a problem in a technical sense. I mean, there is always a requirement for real integration testing of all sorts for an attempt to release a program on a larger scale. > > Now I know it works without stopping on 4 computers. Better then yesterday hugely. > > Thank you. > > Regs > Daniel B. Kolis > > my ref: nafl, 19 Jan 2023, https://groups.google.com/g/comp.lang.python/c/FNlXg0Od39o/m/9stiUtLSAQAJ > > > > > > > > From 2QdxY4RzWzUUiLuE at potatochowder.com Thu Jan 19 13:30:19 2023 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Thu, 19 Jan 2023 13:30:19 -0500 Subject: Improvement to imports, what is a better way ? In-Reply-To: <07805baa-325b-69c2-b691-b46b10550a01@tompassin.net> References: <0f144325-df35-47fa-a0e6-107991260c14n@googlegroups.com> <4a9214965a630ae3c1175c564b850bd1c6e706b3.camel@anode.ca> <3dc7778d-eb44-9b3f-cf75-77d4a039fb34@wichmann.us> <050354dd-f252-f40d-efbb-aa02fd4a8bba@DancesWithMice.info> <5ebee5b8-83b6-31a7-54a7-79611ef3fc5a@roelschroeven.net> <07805baa-325b-69c2-b691-b46b10550a01@tompassin.net> Message-ID: On 2023-01-19 at 12:59:21 -0500, Thomas Passin wrote: > Well, it's an art, not a science [...] +1 > # Create a plot > g2 = ( > ggplot(df2, > aes('Days Since Jan 22', # Comments can clarify these params > + geom_point(size=.1, color='blue') # size, color params optional > + theme_bw() # Optional theme (background, grid color, ...) > ) You've got a comma followed by a plus sign in there, so I'm not exactly sure where the parameters to which function begin and end. When it starts to look like this, I begin breaking out the parameters: label = 'Days Since Jan 22' size = geom_point(size=.1, color='blue') theme = theme_bw() g2 = ggplot(df2, aes(label, size, theme)) > # Compose a long string: > msg = ('A very long line .....\n' > + 'Another long bit of text ....' > + 'plus another ....' > ) If all the pieces are constants, then Python will concatenate them for you: msg = ('A very long line .....\n' 'Another long bit of text ....' 'plus another') You can even mix in "f" strings: msg = ('long line\n' f'left text {name} right text' 'more here') But watch out for missing spaces between the pieces! :-) > The PEP-8 rules are good, but they can't cover all cases perfectly. Some the PEP-8 rules are debatable. Regardless, they can't cover all cases perfectly. (IOW, we agree on the bit that's relevant to this thread.) From roel at roelschroeven.net Thu Jan 19 14:00:49 2023 From: roel at roelschroeven.net (Roel Schroeven) Date: Thu, 19 Jan 2023 20:00:49 +0100 Subject: Improvement to imports, what is a better way ? In-Reply-To: References: <0f144325-df35-47fa-a0e6-107991260c14n@googlegroups.com> <4a9214965a630ae3c1175c564b850bd1c6e706b3.camel@anode.ca> <3dc7778d-eb44-9b3f-cf75-77d4a039fb34@wichmann.us> <050354dd-f252-f40d-efbb-aa02fd4a8bba@DancesWithMice.info> <5ebee5b8-83b6-31a7-54a7-79611ef3fc5a@roelschroeven.net> <07805baa-325b-69c2-b691-b46b10550a01@tompassin.net> Message-ID: <9c1f5a5e-226c-1d11-41a2-aa79bdac2e4d@roelschroeven.net> 2QdxY4RzWzUUiLuE at potatochowder.com schreef op 19/01/2023 om 19:30: > > The PEP-8 rules are good, but they can't cover all cases perfectly. > > Some the PEP-8 rules are debatable. Regardless, they can't cover all > cases perfectly. (IOW, we agree on the bit that's relevant to this > thread.) PEP 8 even covers that: A Foolish Consistency is the Hobgoblin of Little Minds [...] However, know when to be inconsistent ? sometimes style guide recommendations just aren?t applicable. When in doubt, use your best judgment. [...] -- "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman From list1 at tompassin.net Thu Jan 19 14:09:54 2023 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 19 Jan 2023 14:09:54 -0500 Subject: Improvement to imports, what is a better way ? In-Reply-To: References: <0f144325-df35-47fa-a0e6-107991260c14n@googlegroups.com> <4a9214965a630ae3c1175c564b850bd1c6e706b3.camel@anode.ca> <3dc7778d-eb44-9b3f-cf75-77d4a039fb34@wichmann.us> <050354dd-f252-f40d-efbb-aa02fd4a8bba@DancesWithMice.info> <5ebee5b8-83b6-31a7-54a7-79611ef3fc5a@roelschroeven.net> <07805baa-325b-69c2-b691-b46b10550a01@tompassin.net> Message-ID: On 1/19/2023 1:30 PM, 2QdxY4RzWzUUiLuE at potatochowder.com wrote: > On 2023-01-19 at 12:59:21 -0500, > Thomas Passin wrote: > >> Well, it's an art, not a science [...] > > +1 > >> # Create a plot >> g2 = ( >> ggplot(df2, >> aes('Days Since Jan 22', # Comments can clarify these params >> + geom_point(size=.1, color='blue') # size, color params optional >> + theme_bw() # Optional theme (background, grid color, ...) >> ) > > You've got a comma followed by a plus sign in there, so I'm not exactly > sure where the parameters to which function begin and end. > > When it starts to look like this, I begin breaking out the parameters: > > label = 'Days Since Jan 22' > size = geom_point(size=.1, color='blue') > theme = theme_bw() > g2 = ggplot(df2, aes(label, size, theme)) > >> # Compose a long string: >> msg = ('A very long line .....\n' >> + 'Another long bit of text ....' >> + 'plus another ....' >> ) > > If all the pieces are constants, then Python will concatenate them for > you: > > msg = ('A very long line .....\n' > 'Another long bit of text ....' > 'plus another') Yes, the "+" sign is not actually needed in Python, but I find it's helpful anyway. Using one also mirrors the use of other operators, such as "or", where you cannot omit the operator. Another matter of taste ... > You can even mix in "f" strings: > > msg = ('long line\n' > f'left text {name} right text' > 'more here') > > But watch out for missing spaces between the pieces! :-) > >> The PEP-8 rules are good, but they can't cover all cases perfectly. > > Some the PEP-8 rules are debatable. Regardless, they can't cover all > cases perfectly. (IOW, we agree on the bit that's relevant to this > thread.) From cs at cskk.id.au Thu Jan 19 18:04:14 2023 From: cs at cskk.id.au (Cameron Simpson) Date: Fri, 20 Jan 2023 10:04:14 +1100 Subject: Improvement to imports, what is a better way ? In-Reply-To: References: Message-ID: On 19Jan2023 07:34, Dan Kolis wrote: >On Thursday, January 19, 2023 at 12:09:02 AM UTC-5, cameron wrote: >> I know this is vague. Once you find its stalling in a particular >> function (if it is) you may be able to run that function directly. Also, >> a print() at the top abd bottom/return of the stalling function. And so >> on. > >Dan says: >After lots of iterations, I changed one line in a call used to refresh >windows from .update() to .update_idle_tasks() Yeah. See the bottom of this page: https://tkdocs.com/tutorial/eventloop.html >Some other dudes were so nice they tested it before that, it worked perfectly on their computers anyway. This kind of thing can be timing and user interaction sensitive. >Now it seems to work 'all the way' too after this change on my favroritish computer. >Since a CTRL C often started it going again I had a hint there, reading >other peoples various troubles all over the web made me think it was >'someplace in this part'. The `update_idle_tasks()` call runs pending display updates (i.e. redraws) but does not process any events (mostly user actions). So it generally returns promptly. The `update()` call processes events as well. If those events make callbacks, the callbacks can block (depending what they do). Or they themselves might call update() if you've written your code that way. There's a note on the page mentioned above. Calling update() effectively runs the main event loop _here_, at the call. This is also how things like interactive dialogue boxes work: draw the dialogue, then internally call update() to process user interaction until the dialogue is done, then return. This description is lousy, and we'd need to see the code to pin point exactly how your interface is deadlocking (or, possibly, just pausing for a long time while something happens). But the key points are: - there's no threading in the stuff above, so if you need GUI updates _during_ some process you need to _call_ the GUI main loop for it to do some work - you've blocked until that call comes back, like any other function call - the update_idle_tasks just does redraws; because it doesn't process events those events' callbacks do not run, therefore those callbacks cannot block your current code - the update function does the full mainloop, including processing events, and processing events can make callbacks, whcih can themselves call update etc and ... Cheers, Cameron Simpson From avi.e.gross at gmail.com Thu Jan 19 19:33:40 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Thu, 19 Jan 2023 19:33:40 -0500 Subject: Improvement to imports, what is a better way ? In-Reply-To: References: <0f144325-df35-47fa-a0e6-107991260c14n@googlegroups.com> <4a9214965a630ae3c1175c564b850bd1c6e706b3.camel@anode.ca> <3dc7778d-eb44-9b3f-cf75-77d4a039fb34@wichmann.us> <050354dd-f252-f40d-efbb-aa02fd4a8bba@DancesWithMice.info> <5ebee5b8-83b6-31a7-54a7-79611ef3fc5a@roelschroeven.net> <07805baa-325b-69c2-b691-b46b10550a01@tompassin.net> Message-ID: <009401d92c66$d80828c0$88187a40$@gmail.com> Just FYI, the example Dave supplied was not using python code and using a rather strange re-definition in the R language package he was using. Or maybe not. Anyone not interested, skip the rest. First, R does not use indentation for almost anything. So you can break one long line up into many lines all over the place where it is clear you are continuing. Second, the ggplot package (another name for module) came along before pipes were used a lot and made a choice to overload the meaning of the plus sign selectively between "verbs" of sorts. The code Thomas shared says: - Set up the beginning of a plot using the arguments provided and create a DATA STRUCTURE. This structure is a rather complex list structure composed internally of many kinds of named parts, some optional. - Then call a verb of sorts, called geom_point() to update the same data structure appropriately and store info to be used LATER when you call the print driver that knows how to handle this data structure. In a sense, the first command sends the dta structure to the second command which changes it and passes it along to - the next verb is theme_bw() which changes all kinds of elements in the data structure and returns the update data structure. The result is stored as the value of g2 and NOTHING HAPPENS. If you even wrote print(g2) or implicitly printed it at the REPL level such as by typing "g2" or not assigning it at all to a variable at the REPL level, then the driver that handles such an object generates a graph to whatever output device is currently set in whatever format is set such as a JPEG image or a PDF file. There are errors in what Thomas wrote as an example but not what is mentioned. The overall flow looks like: g2 <- ggplot(ARGS) + geom_point(ARGS) + theme_bw(ARGS) The parentheses around the RHS are NOT needed and the above is very typically written as: g2 <- ggplot(ARGS) + geom_point(ARGS) + theme_bw(ARGS) The plus signs at the end of each line tell the REPL to keep reading. The code shown is not working code but was an illustration. The ggplot() function takes many potential arguments including one for the data.frame or tibble that holds rows and columns of data. You then have to map some aesthetics such as what X and Y are bound to directly to some column, or perhaps to a function involving columns. That is not shown inside the aes() inner function and what is shown is nonsense. As usual, function arguments are separated by commas and I assume a line or more are missing. This example does not tell us what columns I being graphed against what other column or other necessary things. That can sometimes be told or changed later but I see none in this example. This function call should have ended without a comma and with a close parentheses followed by a plus sign. The geom_point() function that follows can be empty and then would ask for a chart overlay showing just the points. This example added the size and color of those points. The last verb of theme_bw() adjusts all kinds of parameters to provide a certain look to be basically black and white for various background items. The example chose to include parens around everything so the plus signs could now be anywhere, including the start of future lines. I generally do not do things this way but it is a valid way. I do note there is a python version of the ggplot package and maybe Thomas is writing from that point of view in python. I have never used that. As noted, most other R constructs use PIPES not the plus sign and I wish this package would change to conform but it is a bit late! LOL! Avi -----Original Message----- From: Python-list On Behalf Of 2QdxY4RzWzUUiLuE at potatochowder.com Sent: Thursday, January 19, 2023 1:30 PM To: python-list at python.org Subject: Re: Improvement to imports, what is a better way ? On 2023-01-19 at 12:59:21 -0500, Thomas Passin wrote: > Well, it's an art, not a science [...] +1 > # Create a plot > g2 = ( > ggplot(df2, > aes('Days Since Jan 22', # Comments can clarify these params > + geom_point(size=.1, color='blue') # size, color params optional > + theme_bw() # Optional theme (background, grid color, ...) > ) You've got a comma followed by a plus sign in there, so I'm not exactly sure where the parameters to which function begin and end. When it starts to look like this, I begin breaking out the parameters: label = 'Days Since Jan 22' size = geom_point(size=.1, color='blue') theme = theme_bw() g2 = ggplot(df2, aes(label, size, theme)) > # Compose a long string: > msg = ('A very long line .....\n' > + 'Another long bit of text ....' > + 'plus another ....' > ) If all the pieces are constants, then Python will concatenate them for you: msg = ('A very long line .....\n' 'Another long bit of text ....' 'plus another') You can even mix in "f" strings: msg = ('long line\n' f'left text {name} right text' 'more here') But watch out for missing spaces between the pieces! :-) > The PEP-8 rules are good, but they can't cover all cases perfectly. Some the PEP-8 rules are debatable. Regardless, they can't cover all cases perfectly. (IOW, we agree on the bit that's relevant to this thread.) -- https://mail.python.org/mailman/listinfo/python-list From list1 at tompassin.net Thu Jan 19 21:36:59 2023 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 19 Jan 2023 21:36:59 -0500 Subject: Improvement to imports, what is a better way ? In-Reply-To: <009401d92c66$d80828c0$88187a40$@gmail.com> References: <0f144325-df35-47fa-a0e6-107991260c14n@googlegroups.com> <4a9214965a630ae3c1175c564b850bd1c6e706b3.camel@anode.ca> <3dc7778d-eb44-9b3f-cf75-77d4a039fb34@wichmann.us> <050354dd-f252-f40d-efbb-aa02fd4a8bba@DancesWithMice.info> <5ebee5b8-83b6-31a7-54a7-79611ef3fc5a@roelschroeven.net> <07805baa-325b-69c2-b691-b46b10550a01@tompassin.net> <009401d92c66$d80828c0$88187a40$@gmail.com> Message-ID: On 1/19/2023 7:33 PM, avi.e.gross at gmail.com wrote: > Just FYI, the example Dave supplied was not using python code and using a > rather strange re-definition in the R language package he was using. Or > maybe not. > > Anyone not interested, skip the rest. > > First, R does not use indentation for almost anything. So you can break one > long line up into many lines all over the place where it is clear you are > continuing. > > Second, the ggplot package (another name for module) came along before pipes > were used a lot and made a choice to overload the meaning of the plus sign > selectively between "verbs" of sorts. > > The code Thomas shared says: > > - Set up the beginning of a plot using the arguments provided and create a > DATA STRUCTURE. This structure is a rather complex list structure composed > internally of many kinds of named parts, some optional. > - Then call a verb of sorts, called geom_point() to update the same data > structure appropriately and store info to be used LATER when you call the > print driver that knows how to handle this data structure. In a sense, the > first command sends the dta structure to the second command which changes it > and passes it along to > - the next verb is theme_bw() which changes all kinds of elements in the > data structure and returns the update data structure. > > The result is stored as the value of g2 and NOTHING HAPPENS. > > If you even wrote print(g2) or implicitly printed it at the REPL level such > as by typing "g2" or not assigning it at all to a variable at the REPL > level, then the driver that handles such an object generates a graph to > whatever output device is currently set in whatever format is set such as a > JPEG image or a PDF file. > > There are errors in what Thomas wrote as an example but not what is > mentioned. Just to be clear, my example was copy-pasted from working Python code. It uses the plotnine package, available from PyPi, which is a partial implementation of R's ggplot2 API as adapted for Python; both are implementations of the ideas from The Language of Graphics of Wilkinson. I shortened a line or two to prevent wrapping by email clients and perhaps I chopped something off by mistake. > The overall flow looks like: > > g2 <- ggplot(ARGS) + geom_point(ARGS) + theme_bw(ARGS) > > The parentheses around the RHS are NOT needed and the above is very > typically written as: > > g2 <- ggplot(ARGS) + > geom_point(ARGS) + > theme_bw(ARGS) > > The plus signs at the end of each line tell the REPL to keep reading. > > The code shown is not working code but was an illustration. The ggplot() > function takes many potential arguments including one for the data.frame or > tibble that holds rows and columns of data. You then have to map some > aesthetics such as what X and Y are bound to directly to some column, or > perhaps to a function involving columns. That is not shown inside the aes() > inner function and what is shown is nonsense. As usual, function arguments > are separated by commas and I assume a line or more are missing. This > example does not tell us what columns I being graphed against what other > column or other necessary things. That can sometimes be told or changed > later but I see none in this example. This function call should have ended > without a comma and with a close parentheses followed by a plus sign. > > The geom_point() function that follows can be empty and then would ask for a > chart overlay showing just the points. This example added the size and color > of those points. > > The last verb of theme_bw() adjusts all kinds of parameters to provide a > certain look to be basically black and white for various background items. > > The example chose to include parens around everything so the plus signs > could now be anywhere, including the start of future lines. I generally do > not do things this way but it is a valid way. > > I do note there is a python version of the ggplot package and maybe Thomas > is writing from that point of view in python. I have never used that. > > As noted, most other R constructs use PIPES not the plus sign and I wish > this package would change to conform but it is a bit late! LOL! > > Avi > > > > -----Original Message----- > From: Python-list On > Behalf Of 2QdxY4RzWzUUiLuE at potatochowder.com > Sent: Thursday, January 19, 2023 1:30 PM > To: python-list at python.org > Subject: Re: Improvement to imports, what is a better way ? > > On 2023-01-19 at 12:59:21 -0500, > Thomas Passin wrote: > >> Well, it's an art, not a science [...] > > +1 > >> # Create a plot >> g2 = ( >> ggplot(df2, >> aes('Days Since Jan 22', # Comments can clarify these params >> + geom_point(size=.1, color='blue') # size, color params optional >> + theme_bw() # Optional theme (background, grid color, ...) >> ) > > You've got a comma followed by a plus sign in there, so I'm not exactly sure > where the parameters to which function begin and end. > > When it starts to look like this, I begin breaking out the parameters: > > label = 'Days Since Jan 22' > size = geom_point(size=.1, color='blue') theme = theme_bw() > g2 = ggplot(df2, aes(label, size, theme)) > >> # Compose a long string: >> msg = ('A very long line .....\n' >> + 'Another long bit of text ....' >> + 'plus another ....' >> ) > > If all the pieces are constants, then Python will concatenate them for > you: > > msg = ('A very long line .....\n' > 'Another long bit of text ....' > 'plus another') > > You can even mix in "f" strings: > > msg = ('long line\n' > f'left text {name} right text' > 'more here') > > But watch out for missing spaces between the pieces! :-) > >> The PEP-8 rules are good, but they can't cover all cases perfectly. > > Some the PEP-8 rules are debatable. Regardless, they can't cover all cases > perfectly. (IOW, we agree on the bit that's relevant to this > thread.) > -- > https://mail.python.org/mailman/listinfo/python-list > From avi.e.gross at gmail.com Fri Jan 20 00:21:38 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Fri, 20 Jan 2023 00:21:38 -0500 Subject: Improvement to imports, what is a better way ? In-Reply-To: References: <0f144325-df35-47fa-a0e6-107991260c14n@googlegroups.com> <4a9214965a630ae3c1175c564b850bd1c6e706b3.camel@anode.ca> <3dc7778d-eb44-9b3f-cf75-77d4a039fb34@wichmann.us> <050354dd-f252-f40d-efbb-aa02fd4a8bba@DancesWithMice.info> <5ebee5b8-83b6-31a7-54a7-79611ef3fc5a@roelschroeven.net> <07805baa-325b-69c2-b691-b46b10550a01@tompassin.net> <009401d92c66$d80828c0$88187a40$@gmail.com> Message-ID: <00a201d92c8f$127c9400$3775bc00$@gmail.com> I stand corrected. Thomas is (mostly) writing valid Python if you use the module that looks (deliberately) like the R implementation. The use of "+" in two contexts such as when not needed to concatenate strings, reminded me too much of R. Either way, this thread has moved on from any original question. Now it is about the long and short of it. -----Original Message----- From: Python-list On Behalf Of Thomas Passin Sent: Thursday, January 19, 2023 9:37 PM To: python-list at python.org Subject: Re: Improvement to imports, what is a better way ? On 1/19/2023 7:33 PM, avi.e.gross at gmail.com wrote: > Just FYI, the example Dave supplied was not using python code and > using a rather strange re-definition in the R language package he was > using. Or maybe not. > > Anyone not interested, skip the rest. > > First, R does not use indentation for almost anything. So you can > break one long line up into many lines all over the place where it is > clear you are continuing. > > Second, the ggplot package (another name for module) came along before > pipes were used a lot and made a choice to overload the meaning of the > plus sign selectively between "verbs" of sorts. > > The code Thomas shared says: > > - Set up the beginning of a plot using the arguments provided and > create a DATA STRUCTURE. This structure is a rather complex list > structure composed internally of many kinds of named parts, some optional. > - Then call a verb of sorts, called geom_point() to update the same > data structure appropriately and store info to be used LATER when you > call the print driver that knows how to handle this data structure. In > a sense, the first command sends the dta structure to the second > command which changes it and passes it along to > - the next verb is theme_bw() which changes all kinds of elements in > the data structure and returns the update data structure. > > The result is stored as the value of g2 and NOTHING HAPPENS. > > If you even wrote print(g2) or implicitly printed it at the REPL level > such as by typing "g2" or not assigning it at all to a variable at the > REPL level, then the driver that handles such an object generates a > graph to whatever output device is currently set in whatever format is > set such as a JPEG image or a PDF file. > > There are errors in what Thomas wrote as an example but not what is > mentioned. Just to be clear, my example was copy-pasted from working Python code. It uses the plotnine package, available from PyPi, which is a partial implementation of R's ggplot2 API as adapted for Python; both are implementations of the ideas from The Language of Graphics of Wilkinson. I shortened a line or two to prevent wrapping by email clients and perhaps I chopped something off by mistake. > The overall flow looks like: > > g2 <- ggplot(ARGS) + geom_point(ARGS) + theme_bw(ARGS) > > The parentheses around the RHS are NOT needed and the above is very > typically written as: > > g2 <- ggplot(ARGS) + > geom_point(ARGS) + > theme_bw(ARGS) > > The plus signs at the end of each line tell the REPL to keep reading. > > The code shown is not working code but was an illustration. The > ggplot() function takes many potential arguments including one for the > data.frame or tibble that holds rows and columns of data. You then > have to map some aesthetics such as what X and Y are bound to directly > to some column, or perhaps to a function involving columns. That is > not shown inside the aes() inner function and what is shown is > nonsense. As usual, function arguments are separated by commas and I > assume a line or more are missing. This example does not tell us what > columns I being graphed against what other column or other necessary > things. That can sometimes be told or changed later but I see none in > this example. This function call should have ended without a comma and with a close parentheses followed by a plus sign. > > The geom_point() function that follows can be empty and then would ask > for a chart overlay showing just the points. This example added the > size and color of those points. > > The last verb of theme_bw() adjusts all kinds of parameters to provide > a certain look to be basically black and white for various background items. > > The example chose to include parens around everything so the plus > signs could now be anywhere, including the start of future lines. I > generally do not do things this way but it is a valid way. > > I do note there is a python version of the ggplot package and maybe > Thomas is writing from that point of view in python. I have never used that. > > As noted, most other R constructs use PIPES not the plus sign and I > wish this package would change to conform but it is a bit late! LOL! > > Avi > > > > -----Original Message----- > From: Python-list > On Behalf Of > 2QdxY4RzWzUUiLuE at potatochowder.com > Sent: Thursday, January 19, 2023 1:30 PM > To: python-list at python.org > Subject: Re: Improvement to imports, what is a better way ? > > On 2023-01-19 at 12:59:21 -0500, > Thomas Passin wrote: > >> Well, it's an art, not a science [...] > > +1 > >> # Create a plot >> g2 = ( >> ggplot(df2, >> aes('Days Since Jan 22', # Comments can clarify these params >> + geom_point(size=.1, color='blue') # size, color params optional >> + theme_bw() # Optional theme (background, grid color, ...) >> ) > > You've got a comma followed by a plus sign in there, so I'm not > exactly sure where the parameters to which function begin and end. > > When it starts to look like this, I begin breaking out the parameters: > > label = 'Days Since Jan 22' > size = geom_point(size=.1, color='blue') theme = theme_bw() > g2 = ggplot(df2, aes(label, size, theme)) > >> # Compose a long string: >> msg = ('A very long line .....\n' >> + 'Another long bit of text ....' >> + 'plus another ....' >> ) > > If all the pieces are constants, then Python will concatenate them for > you: > > msg = ('A very long line .....\n' > 'Another long bit of text ....' > 'plus another') > > You can even mix in "f" strings: > > msg = ('long line\n' > f'left text {name} right text' > 'more here') > > But watch out for missing spaces between the pieces! :-) > >> The PEP-8 rules are good, but they can't cover all cases perfectly. > > Some the PEP-8 rules are debatable. Regardless, they can't cover all > cases perfectly. (IOW, we agree on the bit that's relevant to this > thread.) > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list From dino at no.spam.ar Fri Jan 20 10:29:26 2023 From: dino at no.spam.ar (Dino) Date: Fri, 20 Jan 2023 10:29:26 -0500 Subject: ok, I feel stupid, but there must be a better way than this! (finding name of unique key in dict) Message-ID: let's say I have this list of nested dicts: [ { "some_key": {'a':1, 'b':2}}, { "some_other_key": {'a':3, 'b':4}} ] I need to turn this into: [ { "value": "some_key", 'a':1, 'b':2}, { "value": "some_other_key", 'a':3, 'b':4} ] I actually did it with: listOfDescriptors = list() for cd in origListOfDescriptors: cn = list(cd.keys())[0] # There must be a better way than this! listOfDescriptors.append({ "value": cn, "type": cd[cn]["a"], "description": cd[cn]["b"] }) and it works, but I look at this and think that there must be a better way. Am I missing something obvious? PS: Screw OpenAPI! Dino From toby at tobiah.org Fri Jan 20 11:06:54 2023 From: toby at tobiah.org (Tobiah) Date: Fri, 20 Jan 2023 08:06:54 -0800 Subject: ok, I feel stupid, but there must be a better way than this! (finding name of unique key in dict) References: Message-ID: On 1/20/23 07:29, Dino wrote: > > let's say I have this list of nested dicts: > > [ > ? { "some_key": {'a':1, 'b':2}}, > ? { "some_other_key": {'a':3, 'b':4}} > ] > > I need to turn this into: > > [ > ? { "value": "some_key", 'a':1, 'b':2}, > ? { "value": "some_other_key", 'a':3, 'b':4} > ] This doesn't look like the program output you're getting. From dino at no.spam.ar Fri Jan 20 11:16:33 2023 From: dino at no.spam.ar (Dino) Date: Fri, 20 Jan 2023 11:16:33 -0500 Subject: ok, I feel stupid, but there must be a better way than this! (finding name of unique key in dict) References: Message-ID: On 1/20/2023 11:06 AM, Tobiah wrote: > On 1/20/23 07:29, Dino wrote: >> > > This doesn't look like the program output you're getting. you are right that I tweaked the name of fields and variables manually (forgot a couple of places, my bad) to illustrate the problem more generally, but hopefully you get the spirit. "value": cn, "a": cd[cn]["a"], "b": cd[cn]["b"] Anyway, the key point (ooops, a pun) is if there's a more elegant way to do this (i.e. get a reference to the unique key in a dict() when the key is unknown): cn = list(cd.keys())[0] # There must be a better way than this! Thanks From jon+usenet at unequivocal.eu Fri Jan 20 11:46:03 2023 From: jon+usenet at unequivocal.eu (Jon Ribbens) Date: Fri, 20 Jan 2023 16:46:03 -0000 (UTC) Subject: ok, I feel stupid, but there must be a better way than this! (finding name of unique key in dict) References: Message-ID: On 2023-01-20, Dino wrote: > > let's say I have this list of nested dicts: > > [ > { "some_key": {'a':1, 'b':2}}, > { "some_other_key": {'a':3, 'b':4}} > ] > > I need to turn this into: > > [ > { "value": "some_key", 'a':1, 'b':2}, > { "value": "some_other_key", 'a':3, 'b':4} > ] [{"value": key, **value} for d in input_data for key, value in d.items()] From oscar.j.benjamin at gmail.com Fri Jan 20 12:40:09 2023 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 20 Jan 2023 17:40:09 +0000 Subject: ok, I feel stupid, but there must be a better way than this! (finding name of unique key in dict) In-Reply-To: References: Message-ID: On Fri, 20 Jan 2023 at 17:30, Dino wrote: > > let's say I have this list of nested dicts: > > [ > { "some_key": {'a':1, 'b':2}}, > { "some_other_key": {'a':3, 'b':4}} > ] > > I need to turn this into: > > [ > { "value": "some_key", 'a':1, 'b':2}, > { "value": "some_other_key", 'a':3, 'b':4} > ] You want both the key and the value so you can use items(): In [39]: L = [ ...: { "some_key": {'a':1, 'b':2}}, ...: { "some_other_key": {'a':3, 'b':4}} ...: ] In [40]: [{"value": k, **m} for l in L for k, m in l.items()] Out[40]: [{'value': 'some_key', 'a': 1, 'b': 2}, {'value': 'some_other_key', 'a': 3, 'b': 4}] -- Oscar From rob.cliffe at btinternet.com Fri Jan 20 13:44:17 2023 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Fri, 20 Jan 2023 18:44:17 +0000 Subject: ok, I feel stupid, but there must be a better way than this! (finding name of unique key in dict) In-Reply-To: References: Message-ID: On 20/01/2023 15:29, Dino wrote: > > let's say I have this list of nested dicts: > > [ > ? { "some_key": {'a':1, 'b':2}}, > ? { "some_other_key": {'a':3, 'b':4}} > ] > > I need to turn this into: > > [ > ? { "value": "some_key", 'a':1, 'b':2}, > ? { "value": "some_other_key", 'a':3, 'b':4} > ] Assuming that I believe the above, rather than the code below, this works: listOfDescriptors = [ ??? { **? (L := list(D.items())[0])[1], **{'value' : L[0] } } ??? for D in origListOfDescriptors] I believe that from Python 3.9 onwards this can be written more concisely as: listOfDescriptors = [ ??? { (L := list(D.items())[0])[1] } | {'value' : L[0] } ??? for D in origListOfDescriptors] ? ? # untested Best wishes Rob Cliffe > > I actually did it with: > > listOfDescriptors = list() > for cd in origListOfDescriptors: > ??? cn = list(cd.keys())[0] # There must be a better way than this! > ??? listOfDescriptors.append({ > ??????? "value": cn, > ??????? "type": cd[cn]["a"], > ??????? "description": cd[cn]["b"] > ??? }) > > and it works, but I look at this and think that there must be a better > way. Am I missing something obvious? > > PS: Screw OpenAPI! > > Dino From list1 at tompassin.net Sat Jan 21 01:32:12 2023 From: list1 at tompassin.net (Thomas Passin) Date: Sat, 21 Jan 2023 01:32:12 -0500 Subject: PEP-8, Line Length, And All That Message-ID: <306ee72c-1b24-9fd7-58b3-5d26dd9dde56@tompassin.net> In another thread ("Improvement to imports, what is a better way ?") there was a lot of talk about line length, PEP-8, etc. I realized that one subject did not really come up, yet it can greatly affect the things we were talking about. I'm referring to the design of the functions, methods, and classes. When they are well designed, or more likely, refactored over and over again, they can lead to code that reads almost like pseudo-code. Here's an example from one of my little projects. You don't need to know anything about the details of the functions or exactly what a "root" is to see what I mean. fileinfo = language, path, ext = getExeKind(root) processor = getProcessor(*fileinfo) runfile(path, processor, ext) [Please don't try to guess exactly what this code needs to do or explain how it could be done differently. That's not the point.] In words, given a source of information (root), we can get some information about a file. Given that information, we can find a suitable processor for it. And given that processor, we can "run" it. When I first put this together, the functionality was not cleanly separated, the various functions did more than one thing, and I had some long lines. Each line did not necessarily convey cleanly what it was doing. It took many iterations while I learned how to make the details work before I was able to see how to structure this part of the functionality into these three nice, clear lines. In fact, the restructuring isn't quite finished, because the near-final version of runfile() does not actually use "ext" (the extension of a file) any more. Its presence is leftover from when runfile() tried to do too much. Why did I assign the (language, path, ext) tuple to fileinfo? Because it was easier and shorter when used as the argument for getProcessor(), and I thought it conveyed the intent more clearly than the tuple. Some people might think to write processor = getProcessor(*getExeKind(root)) Oops, that doesn't expose path and ext. Well, not if we are going to avoid using a walrus operator, anyway, and if we used it, well, readability would go out the window. In a different context, a "fluent" style can be very readable and pleasant to work with. Here are some lines from a Windows batch file that invokes a small Python library written in a fluent style. The first line defines a processing pipeline, and the second passes a data file to it ("xy1" is the feeble name for a batch file that calls the fluent processing library): set process=low(30).diff().norm_last_n(%N%).write() type "c:\data\%1" |xy1 %process% >> %temp% In words, we smooth the data with a LOWESS smooth using a window width of 30, differentiate it, normalize it in a specific way (norm_last_n), and write it out (to stdout). Not shown here, we eventually pipe it to a plotting program. I intended from the start that this library would work in a "fluent" manner. It took a lot of iterations before I worked out how to design it so the it could be invoked in a clean, simple way by a batch file. All this is to say that program design and refactoring can play a large role in writing code that can be understood relatively easily, follow the style guidelines as closely as possible, and be as easy as possible to maintain. From dino at no.spam.ar Sat Jan 21 09:50:54 2023 From: dino at no.spam.ar (Dino) Date: Sat, 21 Jan 2023 09:50:54 -0500 Subject: ok, I feel stupid, but there must be a better way than this! (finding name of unique key in dict) References: Message-ID: I learned new things today and I thank you all for your responses. Please consider yourself thanked individually. Dino On 1/20/2023 10:29 AM, Dino wrote: > > let's say I have this list of nested dicts: > From dino at no.spam.ar Sat Jan 21 10:03:28 2023 From: dino at no.spam.ar (Dino) Date: Sat, 21 Jan 2023 10:03:28 -0500 Subject: tree representation of Python data Message-ID: I have a question that is a bit of a shot in the dark. I have this nice bash utility installed: $ tree -d unit/ unit/ ??? mocks ??? plugins ??? ??? ast ??? ??? editor ??? ??? editor-autosuggest ??? ??? editor-metadata ??? ??? json-schema-validator ??? ??? ??? test-documents ??? ??? validate-semantic ??? ??? 2and3 ??? ??? bugs ??? ??? oas3 ??? standalone ??? topbar-insert I just thought that it would be great if there was a Python utility that visualized a similar graph for nested data structures. Of course I am aware of indent (json.dumps()) and pprint, and they are OK options for my need. It's just that the compact, improved visualization would be nice to have. Not so nice that I would go out of my way to build, but nice enough to use an exising package. Thanks Dino From gweatherby at uchc.edu Sat Jan 21 15:32:17 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Sat, 21 Jan 2023 20:32:17 +0000 Subject: tree representation of Python data In-Reply-To: References: Message-ID: https://docs.python.org/3/library/pprint.html From: Python-list on behalf of Dino Date: Saturday, January 21, 2023 at 11:42 AM To: python-list at python.org Subject: tree representation of Python data *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** I have a question that is a bit of a shot in the dark. I have this nice bash utility installed: $ tree -d unit/ unit/ ??? mocks ??? plugins ? ??? ast ? ??? editor ? ??? editor-autosuggest ? ??? editor-metadata ? ??? json-schema-validator ? ? ??? test-documents ? ??? validate-semantic ? ??? 2and3 ? ??? bugs ? ??? oas3 ??? standalone ??? topbar-insert I just thought that it would be great if there was a Python utility that visualized a similar graph for nested data structures. Of course I am aware of indent (json.dumps()) and pprint, and they are OK options for my need. It's just that the compact, improved visualization would be nice to have. Not so nice that I would go out of my way to build, but nice enough to use an exising package. Thanks Dino -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!iGkQhjN2DTLNPZj7JaAhVrakd6oiQrV3IUV08E2ayIK1hWH2AaJ4OQ_uEobpiLQuWde2974mF41mvsnO$ From robbiemezazem at hotmail.com Sat Jan 21 16:08:56 2023 From: robbiemezazem at hotmail.com (Robbie mezazem) Date: Sat, 21 Jan 2023 21:08:56 +0000 Subject: I could not install pygame no matter what Message-ID: 70916814/error-metadata-generation-failed-cant-install-artic-module Sent from Mail for Windows From dino at no.spam.ar Sat Jan 21 17:58:50 2023 From: dino at no.spam.ar (Dino) Date: Sat, 21 Jan 2023 17:58:50 -0500 Subject: tree representation of Python data References: Message-ID: you rock. Thank you, Stefan. Dino On 1/21/2023 2:41 PM, Stefan Ram wrote: > ram at zedat.fu-berlin.de (Stefan Ram) writes: >> def display_( object, last ): >> directory = object; result = ''; count = len( directory ) >> for entry in directory: >> count -= 1; name = entry; indent = '' >> for c in last[ 1: ]: indent += '??? ' if c else ' ' >> indent += '???' if count else '???' if last else '' >> result += '\n' + indent +( ' ' if indent else '' )+ name >> if directory[ entry ]: >> result += display_( directory[ entry ], last +[ count ]) >> return result > > This ultimate version has some variable names made more speaking: > > def display_( directory, container_counts ): > result = ''; count = len( directory ) > for name in directory: > count -= 1; indent = '' > for container_count in container_counts[ 1: ]: > indent += '??? ' if container_count else ' ' > indent += '???' if count else '???' if container_counts else '' > result += '\n' + indent +( ' ' if indent else '' )+ name > if directory[ name ]: > result += display_\ > ( directory[ name ], container_counts +[ count ]) > return result > > From cs at cskk.id.au Sat Jan 21 18:50:01 2023 From: cs at cskk.id.au (Cameron Simpson) Date: Sun, 22 Jan 2023 10:50:01 +1100 Subject: PEP-8, Line Length, And All That In-Reply-To: <306ee72c-1b24-9fd7-58b3-5d26dd9dde56@tompassin.net> References: <306ee72c-1b24-9fd7-58b3-5d26dd9dde56@tompassin.net> Message-ID: On 21Jan2023 01:32, Thomas Passin wrote: >In another thread ("Improvement to imports, what is a better way ?") >there was a lot of talk about line length, PEP-8, etc. I realized >that one subject did not really come up, yet it can greatly affect the >things we were talking about. > >I'm referring to the design of the functions, methods, and classes. >When they are well designed, or more likely, refactored over and over >again, they can lead to code that reads almost like pseudo-code. [...] Yeah. Personally, I summarise this as the maxim: "ergonomics matter". Code should be easy to use and easy to read. I've got a heap of code I've refactored after the initial implementation to make it easy to use, if only to make it easier for _me_ to use to do more things with it. I've also got decorators and context managers and special classes all aimed at making all the things easier to use and easier to _write_ (because some decorators (a) provide flexibility for the caller - easier to use and (b) implement standard boilerplate logic for common types of parameters, making the function/method implementation easier to write.) And increasingly I've got docstrings with _example_ code which doubles as a doctest. Making it easier to understand how to easily use this thing. Cheers, Cameron Simpson From jfong at ms4.hinet.net Sat Jan 21 22:11:11 2023 From: jfong at ms4.hinet.net (Jach Feng) Date: Sat, 21 Jan 2023 19:11:11 -0800 (PST) Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? Message-ID: Fail on command line, e:\Works\Python>py infix2postfix.py "-4^2+5.3*abs(-2-1)/2" usage: infix2postfix.py [-h] [infix] infix2postfix.py: error: unrecognized arguments: -4^2+5.3*abs(-2-1)/2 Also fail in REPL, e:\Works\Python>py Python 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:08:11) [MSC v.1928 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import argparse >>> parser = argparse.ArgumentParser(description='Convert infix notation to postfix') >>> parser.parse_args("-4^2+5.3*abs(-2-1)/2") usage: [-h] : error: unrecognized arguments: - 4 ^ 2 + 5 . 3 * a b s ( - 2 - 1 ) / 2 Just can't figure out where is wrong!? --Jach From list1 at tompassin.net Sun Jan 22 00:30:07 2023 From: list1 at tompassin.net (Thomas Passin) Date: Sun, 22 Jan 2023 00:30:07 -0500 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: On 1/21/2023 10:11 PM, Jach Feng wrote: > Fail on command line, > > e:\Works\Python>py infix2postfix.py "-4^2+5.3*abs(-2-1)/2" > usage: infix2postfix.py [-h] [infix] > infix2postfix.py: error: unrecognized arguments: -4^2+5.3*abs(-2-1)/2 > > Also fail in REPL, > > e:\Works\Python>py > Python 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:08:11) [MSC v.1928 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. >>>> import argparse >>>> parser = argparse.ArgumentParser(description='Convert infix notation to postfix') >>>> parser.parse_args("-4^2+5.3*abs(-2-1)/2") > usage: [-h] > : error: unrecognized arguments: - 4 ^ 2 + 5 . 3 * a b s ( - 2 - 1 ) / 2 > > Just can't figure out where is wrong!? It just doesn't work like that. If you download the package, there is only one python file, __init__.py. This file contains one class. It has a demo at the end, commented out. If you uncomment those lines and run the file, you get a result printed. These remarks are based on downloading the link for the source distribution from Pypi (https://pypi.org/project/infix2postfix/). When I installed it with pip, nothing seems to have gotten installed although pip went through the motions and claimed it was. So I just downloaded the source package. The test expression is "-(a*b)+(c+d)-(a+b+c+d)". The test output for this is "ab*-cd++ab+c+d+-". If you substitute your expression, the result is abs1-2-*2/3.5+2^4- This may or may not be correct. I'm not sure but I think it's as intended except for reversing "3.5". But maybe that's right, I'm not too sure. Notice that this file is in its first release, version 0.0.1 - the metadata that says it's 'Development Status :: 5 - Production/Stable' seems to be bogus. So it may very well be buggy. At any rate, if you want to use it in a program that can accept arguments, you will have to write that part yourself. And the expression you feed it would need to be a single string, meaning it has to be quoted on the command line as you have done (although on Windows you should be using double quotes instead of single quotes). As for argparse, it isn't doing what you want because you haven't told it what to do with the arguments. From hjp-python at hjp.at Sun Jan 22 10:45:56 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 22 Jan 2023 16:45:56 +0100 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: <20230122154556.lcfxcmbndr3cqc26@hjp.at> On 2023-01-22 00:30:07 -0500, Thomas Passin wrote: > On 1/21/2023 10:11 PM, Jach Feng wrote: > > e:\Works\Python>py infix2postfix.py "-4^2+5.3*abs(-2-1)/2" > > usage: infix2postfix.py [-h] [infix] > > infix2postfix.py: error: unrecognized arguments: -4^2+5.3*abs(-2-1)/2 [...] > > > > > import argparse > > > > > parser = argparse.ArgumentParser(description='Convert infix notation to postfix') > > > > > parser.parse_args("-4^2+5.3*abs(-2-1)/2") > > usage: [-h] > > : error: unrecognized arguments: - 4 ^ 2 + 5 . 3 * a b s ( - 2 - 1 ) / 2 > > > > Just can't figure out where is wrong!? > > It just doesn't work like that. [...] > If you substitute your expression, the result is > > abs1-2-*2/3.5+2^4- > > This may or may not be correct. I'm not sure but I think it's as intended > except for reversing "3.5". But maybe that's right, I'm not too sure. It depends on the intended meaning. But I'm pretty sure that "5.3" is supposed to be a floating point number and not (5 operatordot 3), so it's wrong to split that apart. (Also it seems that the code treats digits as operators, not operands?) Also "abs(...)" is almost certainly intended to be a function call, so it should be invoked after its parameters. I think the correct output (using newlines as delimiters) would be: -4 2 ^ 5.3 -2 1 - abs * 2 / + > Notice that this file is in its first release, version 0.0.1 - the metadata > that says it's 'Development Status :: 5 - Production/Stable' seems to be > bogus. So it may very well be buggy. It is at least too incomplete to be useful. It handles only single letters as operands and treats everything else (except parentheses) as an operator, handling only +, -, *, / and ^ correctly (actually, ^ is typically right associative, so that's arguably wrong, too). hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From jfong at ms4.hinet.net Sun Jan 22 01:34:13 2023 From: jfong at ms4.hinet.net (Jach Feng) Date: Sat, 21 Jan 2023 22:34:13 -0800 (PST) Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: <9a49f181-4eb4-452e-a382-f624097bc7c7n@googlegroups.com> Thomas Passin ? 2023?1?22? ?????1:30:39 [UTC+8] ?????? > On 1/21/2023 10:11 PM, Jach Feng wrote: > > Fail on command line, > > > > e:\Works\Python>py infix2postfix.py "-4^2+5.3*abs(-2-1)/2" > > usage: infix2postfix.py [-h] [infix] > > infix2postfix.py: error: unrecognized arguments: -4^2+5.3*abs(-2-1)/2 > > > > Also fail in REPL, > > > > e:\Works\Python>py > > Python 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:08:11) [MSC v.1928 32 bit (Intel)] on win32 > > Type "help", "copyright", "credits" or "license" for more information. > >>>> import argparse > >>>> parser = argparse.ArgumentParser(description='Convert infix notation to postfix') > >>>> parser.parse_args("-4^2+5.3*abs(-2-1)/2") > > usage: [-h] > > : error: unrecognized arguments: - 4 ^ 2 + 5 . 3 * a b s ( - 2 - 1 ) / 2 > > > > Just can't figure out where is wrong!? > It just doesn't work like that. If you download the package, there is > only one python file, __init__.py. This file contains one class. It > has a demo at the end, commented out. If you uncomment those lines and > run the file, you get a result printed. > > These remarks are based on downloading the link for the source > distribution from Pypi > (https://pypi.org/project/infix2postfix/). When I installed it with > pip, nothing seems to have gotten installed although pip went through > the motions and claimed it was. So I just downloaded the source package. > > The test expression is "-(a*b)+(c+d)-(a+b+c+d)". The test output for > this is "ab*-cd++ab+c+d+-". > > If you substitute your expression, the result is > > abs1-2-*2/3.5+2^4- > > This may or may not be correct. I'm not sure but I think it's as > intended except for reversing "3.5". But maybe that's right, I'm not > too sure. Notice that this file is in its first release, version 0.0.1 > - the metadata that says it's 'Development Status :: 5 - > Production/Stable' seems to be bogus. So it may very well be buggy. > > At any rate, if you want to use it in a program that can accept > arguments, you will have to write that part yourself. And the > expression you feed it would need to be a single string, meaning it has > to be quoted on the command line as you have done (although on Windows > you should be using double quotes instead of single quotes). > > As for argparse, it isn't doing what you want because you haven't told > it what to do with the arguments. Sorry to cause confusion here. I don't know there is a Pypi project with the same name infix2postfix.py:-( Nevertheless, Is there anyway to make parse_args works? >>> parser.parse_args("-4^2+5.3*abs(-2-1)/2") From nntp.mbourne at spamgourmet.com Sun Jan 22 09:01:37 2023 From: nntp.mbourne at spamgourmet.com (Mark Bourne) Date: Sun, 22 Jan 2023 14:01:37 +0000 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? References: Message-ID: Jach Feng wrote: > Fail on command line, > > e:\Works\Python>py infix2postfix.py "-4^2+5.3*abs(-2-1)/2" > usage: infix2postfix.py [-h] [infix] > infix2postfix.py: error: unrecognized arguments: -4^2+5.3*abs(-2-1)/2 > > Also fail in REPL, > > e:\Works\Python>py > Python 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:08:11) [MSC v.1928 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. >>>> import argparse >>>> parser = argparse.ArgumentParser(description='Convert infix notation to postfix') >>>> parser.parse_args("-4^2+5.3*abs(-2-1)/2") > usage: [-h] > : error: unrecognized arguments: - 4 ^ 2 + 5 . 3 * a b s ( - 2 - 1 ) / 2 > > Just can't figure out where is wrong!? First, you need to add an argument to the parser, so that it expects an argument: >>> parser.add_argument("expression") Secondly, `parse_args` expects a list of arguments. By passing a string, it interprets each character as a separate argument (since iterating over a string yields each character in turn). Here, I intentionally leave off the initial hyphen because that's the next problem: >>> parser.parse_args(["4^2+5.3*abs(-2-1)/2"]) Namespace(expression='4^2+5.3*abs(-2-1)/2') Thirdly, an initial hyphen indicates an optional argument so, for example if you pass "-l" it will expect a "-l" argument to be defined as one of the valid options, and also complain that you haven't specified the required expression: >>> parser.parse_args(["-4^2+5.3*abs(-2-1)/2"]) usage: [-h] expression : error: the following arguments are required: expression If you need to pass a value starting with a "-" there are a couple of options... Perhaps it would be acceptable to represent it as "0-...": >>> parser.parse_args(["0-4^2+5.3*abs(-2-1)/2"]) Namespace(expression='0-4^2+5.3*abs(-2-1)/2') While mathematically equivalent, that might have different meaning for whatever you're trying to do. Alternatively, a double hyphen indicates that there are no more options and that anything else is positional arguments even if they begin with a hyphen: >>> parser.parse_args(["--", "-4^2+5.3*abs(-2-1)/2"]) Namespace(expression='-4^2+5.3*abs(-2-1)/2') You wouldn't usually explicitly pass a list of arguments to `parse_args` like that, but it can be useful for testing and experimentation. Usually, you'd call `parse_args()` without any arguments, and it would parse the arguments passed on the command-line when calling your script. e.g. you'd call (from a Windows command prompt / Linux shell / etc.): > ./convert_infix.py -- '-4^2+5.3*abs(-2-1)/2' (it's probably a good idea to quote the expression, in case it includes any characters which would be interpreted specially by the shell - e.g. "*" without quotes usually expands to all matching files in the current directory) -- Mark. From list1 at tompassin.net Sun Jan 22 11:47:37 2023 From: list1 at tompassin.net (Thomas Passin) Date: Sun, 22 Jan 2023 11:47:37 -0500 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <20230122154556.lcfxcmbndr3cqc26@hjp.at> References: <20230122154556.lcfxcmbndr3cqc26@hjp.at> Message-ID: <2d00c299-8106-f0cd-eda9-bb81ad421930@tompassin.net> On 1/22/2023 10:45 AM, Peter J. Holzer wrote: >> Notice that this file is in its first release, version 0.0.1 - the metadata >> that says it's 'Development Status :: 5 - Production/Stable' seems to be >> bogus. So it may very well be buggy. > It is at least too incomplete to be useful. It handles only single > letters as operands and treats everything else (except parentheses) as an operator, handling > only +, -, *, / and ^ correctly (actually, ^ is typically right > associative, so that's arguably wrong, too). This script is basically useless without some documentation on the intended inputs and outputs, and how to invoke it. The documentation could (in the minimal case) have been included in docstrings and comments within the script. As it is, the script seems tantalizing but unusable. Please document your work, even if it's just for yourself! From list1 at tompassin.net Sun Jan 22 12:11:20 2023 From: list1 at tompassin.net (Thomas Passin) Date: Sun, 22 Jan 2023 12:11:20 -0500 Subject: tree representation of Python data In-Reply-To: References: Message-ID: <016498db-aca3-9f53-4118-448ce35bb92d@tompassin.net> On 1/21/2023 10:03 AM, Dino wrote: > > I have a question that is a bit of a shot in the dark. I have this nice > bash utility installed: > > $ tree -d unit/ > unit/ > ??? mocks > ??? plugins > ??? ??? ast > ??? ??? editor > ??? ??? editor-autosuggest > ??? ??? editor-metadata > ??? ??? json-schema-validator > ??? ??? ??? test-documents > ??? ??? validate-semantic > ??????? ??? 2and3 > ??????? ??? bugs > ??????? ??? oas3 > ??? standalone > ??? ??? topbar-insert > > I just thought that it would be great if there was a Python utility that > visualized a similar graph for nested data structures. > Of course I am aware of indent (json.dumps()) and pprint, and they are > OK options for my need. It's just that the compact, improved > visualization would be nice to have. Not so nice that I would go out of > my way to build, but nice enough to use an exising package. It's not clear to me whether you want to display JSON structures or Python objects. However, Python dictionaries are so similar to JSON structures that a dictionary example should be pretty close. This can actually be a tricky problem because for complicated nesting, it's not that easy to work out how to display the elements. The examples posted in this thread so far have been somewhat obscured by the use of leader lines, etc, so that it's a little hard to discern the simple core of the algorithm. Below I include code that displays each leaf element on its own indented line, can only display keys and simple leaf values, and does not construct leader lines so as to keep the code easy to read. In this form it can only display Python dictionaries but could be modified for JSON without too much work. """Display nested objects as an indented list.""" INDENT = ' ' * 3 DICT = {'a': {'aa':'AA', 'ab':'AB'}, 'b': {'ba': {'baa': 'BAA', 'bab': 'BAB'}, 'bb':'BB'}, 'c': ['CA', 'CB'], 'd': 'D' } SIMPLE_TYPES = (int, float, str, bytes) def unroll(obj, indent = ''): if isinstance(obj, dict): unroll_dict(obj, indent) else: """unroll other kinds of objects (not implemented).""" print(indent, 'Can only unroll nested dictionaries') def unroll_dict(dct, indent = ''): """Unroll a dictionary whose values can be either simple or nested. """ for k, v in dct.items(): g.es(indent, k) new_indent = indent + INDENT if type(v) in SIMPLE_TYPES: print(new_indent, v) else: unroll(v, new_indent) # print(f'{new_indent}{v}') unroll(DICT) Output: a aa AA ab AB b ba baa BAA bab BAB bb BB c Can only unroll nested dictionaries d D If you change the last else block, you can make use of the object's internal representation (not for JSON, of course). Change the block to read: # unroll(v, new_indent) print(f'{new_indent}{v}') This change gives the following output: a {'aa': 'AA', 'ab': 'AB'} b {'ba': {'baa': 'BAA', 'bab': 'BAB'}, 'bb': 'BB'} c ['CA', 'CB'] d D In practice, I would collect the text fragments into a list instead of printing them, then print the joined list at the end. From gweatherby at uchc.edu Sun Jan 22 16:04:48 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Sun, 22 Jan 2023 21:04:48 +0000 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <9a49f181-4eb4-452e-a382-f624097bc7c7n@googlegroups.com> References: <9a49f181-4eb4-452e-a382-f624097bc7c7n@googlegroups.com> Message-ID: Argparse is for parsing command line arguments and options. If you just want to evaluate an Python expression, use eval( ) Your string isn?t valid Python due to order of operations, but -(4^2)+5.3*abs(-2-1)/2 is. From: Python-list on behalf of Jach Feng Date: Sunday, January 22, 2023 at 11:24 AM To: python-list at python.org Subject: Re: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** Thomas Passin ? 2023?1?22? ?????1:30:39 [UTC+8] ?????? > On 1/21/2023 10:11 PM, Jach Feng wrote: > > Fail on command line, > > > > e:\Works\Python>py infix2postfix.py "-4^2+5.3*abs(-2-1)/2" > > usage: infix2postfix.py [-h] [infix] > > infix2postfix.py: error: unrecognized arguments: -4^2+5.3*abs(-2-1)/2 > > > > Also fail in REPL, > > > > e:\Works\Python>py > > Python 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:08:11) [MSC v.1928 32 bit (Intel)] on win32 > > Type "help", "copyright", "credits" or "license" for more information. > >>>> import argparse > >>>> parser = argparse.ArgumentParser(description='Convert infix notation to postfix') > >>>> parser.parse_args("-4^2+5.3*abs(-2-1)/2") > > usage: [-h] > > : error: unrecognized arguments: - 4 ^ 2 + 5 . 3 * a b s ( - 2 - 1 ) / 2 > > > > Just can't figure out where is wrong!? > It just doesn't work like that. If you download the package, there is > only one python file, __init__.py. This file contains one class. It > has a demo at the end, commented out. If you uncomment those lines and > run the file, you get a result printed. > > These remarks are based on downloading the link for the source > distribution from Pypi > (https://urldefense.com/v3/__https://pypi.org/project/infix2postfix/__;!!Cn_UX_p3!gqKmYLlyUndAzxmJsqCB429izQ-2-KMbpGP2eVzp_iDKtbgQXfrCu21UBvepq-F9EXb4SJwP516MHeUFMBtW$ ). When I installed it with > pip, nothing seems to have gotten installed although pip went through > the motions and claimed it was. So I just downloaded the source package. > > The test expression is "-(a*b)+(c+d)-(a+b+c+d)". The test output for > this is "ab*-cd++ab+c+d+-". > > If you substitute your expression, the result is > > abs1-2-*2/3.5+2^4- > > This may or may not be correct. I'm not sure but I think it's as > intended except for reversing "3.5". But maybe that's right, I'm not > too sure. Notice that this file is in its first release, version 0.0.1 > - the metadata that says it's 'Development Status :: 5 - > Production/Stable' seems to be bogus. So it may very well be buggy. > > At any rate, if you want to use it in a program that can accept > arguments, you will have to write that part yourself. And the > expression you feed it would need to be a single string, meaning it has > to be quoted on the command line as you have done (although on Windows > you should be using double quotes instead of single quotes). > > As for argparse, it isn't doing what you want because you haven't told > it what to do with the arguments. Sorry to cause confusion here. I don't know there is a Pypi project with the same name infix2postfix.py:-( Nevertheless, Is there anyway to make parse_args works? >>> parser.parse_args("-4^2+5.3*abs(-2-1)/2") -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!gqKmYLlyUndAzxmJsqCB429izQ-2-KMbpGP2eVzp_iDKtbgQXfrCu21UBvepq-F9EXb4SJwP516MHezHygV-$ From cs at cskk.id.au Sun Jan 22 16:13:20 2023 From: cs at cskk.id.au (Cameron Simpson) Date: Mon, 23 Jan 2023 08:13:20 +1100 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: On 21Jan2023 19:11, Jach Feng wrote: >Fail on command line, > >e:\Works\Python>py infix2postfix.py "-4^2+5.3*abs(-2-1)/2" >usage: infix2postfix.py [-h] [infix] >infix2postfix.py: error: unrecognized arguments: -4^2+5.3*abs(-2-1)/2 The usual convention for having "nonoption" arguments beginning with a dash/minus is to explicitly _end_ the option arguments, eg: py infix2postfix.py -- "-4^2+5.3*abs(-2-1)/2" That "--" indicates the end of the options, and that what follows should not be treated as an option. _However_, it requires support in the script parsing the options. I'm pretty sure argparse gives that support for free, as does getopt and probably any others implementing "normal UNIXish options". SO try adding a "--" argument and see how it behaves. >Also fail in REPL, > >e:\Works\Python>py >Python 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:08:11) [MSC v.1928 32 bit (Intel)] on win32 >Type "help", "copyright", "credits" or "license" for more information. >>>> import argparse >>>> parser = argparse.ArgumentParser(description='Convert infix notation to postfix') >>>> parser.parse_args("-4^2+5.3*abs(-2-1)/2") >usage: [-h] >: error: unrecognized arguments: - 4 ^ 2 + 5 . 3 * a b s ( - 2 - 1 ) / 2 This is a different error. `parse_args()` expects a list of arguments, not a single argument. So it has iterated over what you gave it, which produces a series of single character strings which it has taken to be individual arguments. Try this: parser.parse_args(["-4^2+5.3*abs(-2-1)/2"]) and of course: parser.parse_args(["--", "-4^2+5.3*abs(-2-1)/2"]) You can see this behaviour of strings as: print(list("abc")) or: for s in "abc": print("s =", repr(s)) Cheers, Cameron Simpson From torriem at gmail.com Sun Jan 22 18:38:59 2023 From: torriem at gmail.com (Michael Torrie) Date: Sun, 22 Jan 2023 16:38:59 -0700 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: On 1/22/23 11:44, Stefan Ram wrote: > Jach Feng writes: >> e:\Works\Python>py infix2postfix.py "-4^2+5.3*abs(-2-1)/2" > > Well, it's a nice exercise! But I only made it work for the > specific example given. I have not tested whether it always > works. Haha. Yes a nice exercise, but has nothing to do with the original question which is how to convince argparse to accept a string like that without thinking it's a switch. Many unix utilities treat "--" as a special argument that turns off argument parsing for the rest of the command line. Maybe argparse follows this convention too; I don't know. But if it did you'd do: infix2postfix.py -- "-4^2+5.3*abs(-2-1)/2" From grant.b.edwards at gmail.com Mon Jan 23 10:34:33 2023 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Mon, 23 Jan 2023 07:34:33 -0800 (PST) Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? References: <9a49f181-4eb4-452e-a382-f624097bc7c7n@googlegroups.com> =?utf-8?q?=3CSJ0PR?= =?utf-8?q?14MB58645215AFACFEBDC0685542B9CB9=40SJ0PR14MB5864=2Enamprd14=2Epr?= =?utf-8?q?od=2Eoutlook=2Ecom=3E?= Message-ID: <63cea909.020a0220.bebd7.c898@mx.google.com> On 2023-01-22, Weatherby,Gerard wrote: > Argparse is for parsing command line arguments and options. > > If you just want to evaluate an Python expression, use eval( ) Only use eval() if the expression is always under your control (and you don't make mistakes when typing). Don't use eval() on strings that come from the outside world. From jfong at ms4.hinet.net Sun Jan 22 21:19:13 2023 From: jfong at ms4.hinet.net (Jach Feng) Date: Sun, 22 Jan 2023 18:19:13 -0800 (PST) Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: <58b4e9ac-a31a-4751-8e45-292d519155ccn@googlegroups.com> Jach Feng ? 2023?1?22? ?????11:11:22 [UTC+8] ?????? > Fail on command line, > > e:\Works\Python>py infix2postfix.py "-4^2+5.3*abs(-2-1)/2" > usage: infix2postfix.py [-h] [infix] > infix2postfix.py: error: unrecognized arguments: -4^2+5.3*abs(-2-1)/2 > > Also fail in REPL, > > e:\Works\Python>py > Python 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:08:11) [MSC v.1928 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import argparse > >>> parser = argparse.ArgumentParser(description='Convert infix notation to postfix') > >>> parser.parse_args("-4^2+5.3*abs(-2-1)/2") > usage: [-h] > : error: unrecognized arguments: - 4 ^ 2 + 5 . 3 * a b s ( - 2 - 1 ) / 2 > > Just can't figure out where is wrong!? > > --Jach Thank you for all your suggestions. From it, I get two solutions for my problem. 1) Modify the sys.argv by inserting an item '--' before parsing it, ie. sys.argv.insert(1, '--') args = parser.parse_args() It works, and maybe more reliable. e:\Works\Python>py infix2postfix.py "-4^2+5.3*abs(-2-1)/2" -4 2 ^ 5.3 -2 1 - abs * 2 / + 2) By adding an extra space character before the leading '-' sign, ie. e:\Works\Python>py infix2postfix.py " -4^2+5.3*abs(-2-1)/2" -4 2 ^ 5.3 -2 1 - abs * 2 / + But no idea how it works? and if it can survive in a newer argparse version?:-) From dfnsonfsduifb at gmx.de Mon Jan 23 11:24:15 2023 From: dfnsonfsduifb at gmx.de (Johannes Bauer) Date: Mon, 23 Jan 2023 17:24:15 +0100 Subject: Evaluation of variable as f-string Message-ID: Hi there, is there an easy way to evaluate a string stored in a variable as if it were an f-string at runtime? I.e., what I want is to be able to do this: x = { "y": "z" } print(f"-> {x['y']}") This prints "-> z", as expected. But consider: x = { "y": "z" } s = "-> {x['y']}" print(s.format(x = x)) Traceback (most recent call last): File "", line 1, in KeyError: "'y'" Even though s = "-> {x}" print(s.format(x = x)) Prints the expected "-> {'y': 'z'}". This is supposedly for security reasons. However, when trying to emulate this behavior that I wanted (and know the security implications of), my solutions will tend to be less secure. Here is what I have been thinking about: 1. Somehow wrap "s" into an f-string, then eval. E.g.: eval("f'" + s + "'") This is a pain in the ass because you have to know what kind of quotation signs are used inside the expression. In the given case, this wouldn't work (but 'f"' prefix and '"' suffix would). 2. Parse the expression (regex?), then eval() the individual arguments, then run through format(). Pain in the ass to get the exact same behavior as f-strings. Probably by regex alone not even guaranteed to be parsable (especially corner cases with escaped '{' signs or ':' or '{' included inside the expression as a literal). 3. Somehow compile the bytecode representing an actual f-string expression, then execute it. Sounds like a royal pain in the butt, have not tried it. All solutions are extremely undesirable and come with heavy drawbacks. Is there any standard solution (Py3.10+) that does what I would? Anything I'm missing? Thanks, Johannes From rosuav at gmail.com Mon Jan 23 13:02:56 2023 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 Jan 2023 05:02:56 +1100 Subject: Evaluation of variable as f-string In-Reply-To: References: Message-ID: On Tue, 24 Jan 2023 at 04:56, Johannes Bauer wrote: > > Hi there, > > is there an easy way to evaluate a string stored in a variable as if it > were an f-string at runtime? > > ... > > This is supposedly for security reasons. However, when trying to emulate > this behavior that I wanted (and know the security implications of), my > solutions will tend to be less secure. Here is what I have been thinking > about: If you really want the full power of an f-string, then you're asking for the full power of eval(), and that means all the security implications thereof, not to mention the difficulties of namespacing. Have you considered using the vanilla format() method instead? But if you really REALLY know what you're doing, just use eval() directly. I don't really see what you'd gain from an f-string. At very least, work with a well-defined namespace and eval whatever you need in that context. Maybe, rather than asking for a way to treat a string as code, ask for what you ACTUALLY need, and we can help? ChrisA From 2QdxY4RzWzUUiLuE at potatochowder.com Mon Jan 23 13:46:42 2023 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Mon, 23 Jan 2023 13:46:42 -0500 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <58b4e9ac-a31a-4751-8e45-292d519155ccn@googlegroups.com> References: <58b4e9ac-a31a-4751-8e45-292d519155ccn@googlegroups.com> Message-ID: On 2023-01-22 at 18:19:13 -0800, Jach Feng wrote: > 1) Modify the sys.argv by inserting an item '--' before parsing it, ie. > sys.argv.insert(1, '--') > args = parser.parse_args() Please don't do that. :-) In my mind, sys.argv belongs to Python, not the application. Instead, pass a newly created argument list to parse_args: args = parser.parse_args(['--'] + sys.argv) This approach (adjusting the actual arguments) will work until your program actually has options. > 2) By adding an extra space character before the leading '-' sign, ie. > e:\Works\Python>py infix2postfix.py " -4^2+5.3*abs(-2-1)/2" > -4 2 ^ 5.3 -2 1 - abs * 2 / + > > But no idea how it works? and if it can survive in a newer argparse version?:-) It works because argparse checks the first character of each argument, and *doesn't* strip/trim whitespace. So "-x" looks like an option, and " -x" looks an argument. From cs at cskk.id.au Mon Jan 23 15:31:45 2023 From: cs at cskk.id.au (Cameron Simpson) Date: Tue, 24 Jan 2023 07:31:45 +1100 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: On 23Jan2023 13:46, 2QdxY4RzWzUUiLuE at potatochowder.com <2QdxY4RzWzUUiLuE at potatochowder.com> wrote: >On 2023-01-22 at 18:19:13 -0800, >Jach Feng wrote: >> 1) Modify the sys.argv by inserting an item '--' before parsing it, >> ie. >> sys.argv.insert(1, '--') >> args = parser.parse_args() > >Please don't do that. :-) Well... We routine mmodify argv when parsing a command line. It's just a list. It does assume I'm the only user of it. >In my mind, sys.argv belongs to Python, not the application. Instead, >pass a newly created argument list to parse_args: > >args = parser.parse_args(['--'] + sys.argv) I do sometimes copy it: def main(argv): argv = list(argv) for exactly the reasons you're alluding to. Then I'm free to modify the copy. But for Jach Feng: the "--" is really expected as something the user does when they invoke your programme, _explicitly_ saying that what follows from here is not an argument. So the user is expected to type: your_script -x -y -- "-4^2+5.3*abs(-2-1)/2" where there are -x and -y options, then end of options, then an argument, which would look like an option if there wasn't the "--" argument. Cheers, Cameron Simpson From rosuav at gmail.com Mon Jan 23 15:59:53 2023 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 Jan 2023 07:59:53 +1100 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: On Tue, 24 Jan 2023 at 07:47, Cameron Simpson wrote: > > But for Jach Feng: the "--" is really expected as something the user > does when they invoke your programme, _explicitly_ saying that what > follows from here is not an argument. So the user is expected to type: > > your_script -x -y -- "-4^2+5.3*abs(-2-1)/2" > > where there are -x and -y options, then end of options, then an > argument, which would look like an option if there wasn't the "--" > argument. And if you DON'T expect the user to enter the "--", then why use argparse? You can just check argv directly to get your arguments. This entire thread is a massive "how can I use X to do Y?" problem. ChrisA From greg.ewing at canterbury.ac.nz Mon Jan 23 18:33:18 2023 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 24 Jan 2023 12:33:18 +1300 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: >> On 2023-01-22 at 18:19:13 -0800, >> Jach Feng wrote: >>> 1) Modify the sys.argv by inserting an item '--' before parsing it, ie. >>> sys.argv.insert(1, '--') >>> args = parser.parse_args() If you do that, you'll never be able to have any actual options, so using argparse seems like overkill. Just pull the argument out of argv directly. -- Greg From jfong at ms4.hinet.net Mon Jan 23 20:58:42 2023 From: jfong at ms4.hinet.net (Jach Feng) Date: Mon, 23 Jan 2023 17:58:42 -0800 (PST) Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: <58b4e9ac-a31a-4751-8e45-292d519155ccn@googlegroups.com> Message-ID: <5ab3bc15-1628-4289-9d59-f981d50ac4ffn@googlegroups.com> 2QdxY4Rz... at potatochowder.com ? 2023?1?24? ?????2:47:12 [UTC+8] ?????? > On 2023-01-22 at 18:19:13 -0800, > Jach Feng wrote: > > > 1) Modify the sys.argv by inserting an item '--' before parsing it, ie. > > sys.argv.insert(1, '--') > > args = parser.parse_args() > Please don't do that. :-) > > In my mind, sys.argv belongs to Python, not the application. Instead, > pass a newly created argument list to parse_args: > > args = parser.parse_args(['--'] + sys.argv) > > This approach (adjusting the actual arguments) will work until your > program actually has options. > > 2) By adding an extra space character before the leading '-' sign, ie. > > e:\Works\Python>py infix2postfix.py " -4^2+5.3*abs(-2-1)/2" > > -4 2 ^ 5.3 -2 1 - abs * 2 / + > > > > But no idea how it works? and if it can survive in a newer argparse version?:-) > It works because argparse checks the first character of each argument, > and *doesn't* strip/trim whitespace. So "-x" looks like an option, and > " -x" looks an argument. More pathonic, but don't work. The '--' must be at index 1:-) >>> parser.parse_args(['--', 'infix2postfix.py', '-4.3+5']) usage: [-h] infix : error: unrecognized arguments: -4.3+5 From jfong at ms4.hinet.net Mon Jan 23 21:04:59 2023 From: jfong at ms4.hinet.net (Jach Feng) Date: Mon, 23 Jan 2023 18:04:59 -0800 (PST) Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: <34488860-2ea8-4c5f-9ee6-e513740b354dn@googlegroups.com> Chris Angelico ? 2023?1?24? ?????5:00:27 [UTC+8] ?????? > On Tue, 24 Jan 2023 at 07:47, Cameron Simpson wrote: > > > > But for Jach Feng: the "--" is really expected as something the user > > does when they invoke your programme, _explicitly_ saying that what > > follows from here is not an argument. So the user is expected to type: > > > > your_script -x -y -- "-4^2+5.3*abs(-2-1)/2" > > > > where there are -x and -y options, then end of options, then an > > argument, which would look like an option if there wasn't the "--" > > argument. > And if you DON'T expect the user to enter the "--", then why use > argparse? You can just check argv directly to get your arguments. > > This entire thread is a massive "how can I use X to do Y?" problem. > > ChrisA The '--' requirement makes its usage less instinctive, and handling argv directly makes me loss the benefit of using '-h':-) From rosuav at gmail.com Mon Jan 23 21:12:54 2023 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 Jan 2023 13:12:54 +1100 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <34488860-2ea8-4c5f-9ee6-e513740b354dn@googlegroups.com> References: <34488860-2ea8-4c5f-9ee6-e513740b354dn@googlegroups.com> Message-ID: On Tue, 24 Jan 2023 at 13:09, Jach Feng wrote: > > Chris Angelico ? 2023?1?24? ?????5:00:27 [UTC+8] ?????? > > On Tue, 24 Jan 2023 at 07:47, Cameron Simpson wrote: > > > > > > But for Jach Feng: the "--" is really expected as something the user > > > does when they invoke your programme, _explicitly_ saying that what > > > follows from here is not an argument. So the user is expected to type: > > > > > > your_script -x -y -- "-4^2+5.3*abs(-2-1)/2" > > > > > > where there are -x and -y options, then end of options, then an > > > argument, which would look like an option if there wasn't the "--" > > > argument. > > And if you DON'T expect the user to enter the "--", then why use > > argparse? You can just check argv directly to get your arguments. > > > > This entire thread is a massive "how can I use X to do Y?" problem. > > > > ChrisA > The '--' requirement makes its usage less instinctive, and handling argv directly makes me loss the benefit of using '-h':-) if "-h" in sys.argv: usage() else: do_stuff_with(sys.argv[1:]) What is argparse really doing for you? ChrisA From jfong at ms4.hinet.net Mon Jan 23 21:11:46 2023 From: jfong at ms4.hinet.net (Jach Feng) Date: Mon, 23 Jan 2023 18:11:46 -0800 (PST) Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: <9fd8e56c-b815-40f6-9a26-315f4d652ea5n@googlegroups.com> Greg Ewing ? 2023?1?24? ?????7:33:43 [UTC+8] ?????? > >> On 2023-01-22 at 18:19:13 -0800, > >> Jach Feng wrote: > >>> 1) Modify the sys.argv by inserting an item '--' before parsing it, ie. > >>> sys.argv.insert(1, '--') > >>> args = parser.parse_args() > If you do that, you'll never be able to have any actual options, so > using argparse seems like overkill. Just pull the argument out of > argv directly. > > -- > Greg Any easy way to "pull the argument out of argv directly" before parse_args()?:-) From cs at cskk.id.au Tue Jan 24 01:04:57 2023 From: cs at cskk.id.au (Cameron Simpson) Date: Tue, 24 Jan 2023 17:04:57 +1100 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <5ab3bc15-1628-4289-9d59-f981d50ac4ffn@googlegroups.com> References: <5ab3bc15-1628-4289-9d59-f981d50ac4ffn@googlegroups.com> Message-ID: On 23Jan2023 17:58, Jach Feng wrote: >>>> parser.parse_args(['--', 'infix2postfix.py', '-4.3+5']) >usage: [-h] infix >: error: unrecognized arguments: -4.3+5 This error doesn't look like "-4.3+5 looks like an option" but instead "we don't expect any arguments after "infix". Not personally a fan of argparse myself, but then I have my own elaborate command line framework which generally uses getopt for the option stuff. Cheers, Cameron Simpson From list1 at tompassin.net Tue Jan 24 10:12:01 2023 From: list1 at tompassin.net (Thomas Passin) Date: Tue, 24 Jan 2023 10:12:01 -0500 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: <34488860-2ea8-4c5f-9ee6-e513740b354dn@googlegroups.com> Message-ID: <527ff2cc-95d6-f8a0-7705-cacf0b2ab42e@tompassin.net> On 1/23/2023 9:12 PM, Chris Angelico wrote: > On Tue, 24 Jan 2023 at 13:09, Jach Feng wrote: >> >> Chris Angelico ? 2023?1?24? ?????5:00:27 [UTC+8] ?????? >>> On Tue, 24 Jan 2023 at 07:47, Cameron Simpson wrote: >>>> >>>> But for Jach Feng: the "--" is really expected as something the user >>>> does when they invoke your programme, _explicitly_ saying that what >>>> follows from here is not an argument. So the user is expected to type: >>>> >>>> your_script -x -y -- "-4^2+5.3*abs(-2-1)/2" >>>> >>>> where there are -x and -y options, then end of options, then an >>>> argument, which would look like an option if there wasn't the "--" >>>> argument. >>> And if you DON'T expect the user to enter the "--", then why use >>> argparse? You can just check argv directly to get your arguments. >>> >>> This entire thread is a massive "how can I use X to do Y?" problem. >>> >>> ChrisA >> The '--' requirement makes its usage less instinctive, and handling argv directly makes me loss the benefit of using '-h':-) > > if "-h" in sys.argv: usage() > else: do_stuff_with(sys.argv[1:]) > > What is argparse really doing for you? I second this. "if '-h' in sys.argv:" is usually what I do. Alternatively, you could use "--arg=" syntax and place your string "-4^2+5.3*abs(-2-1)/2" its right-hand side": infix2postfix [options] "--infix=-4^2+5.3*abs(-2-1)/2" This shouldn't be too hard for a user to work with. You could scan the argument list for the presence of "--infix=" and display the help message if it isn't there. From list1 at tompassin.net Tue Jan 24 10:21:51 2023 From: list1 at tompassin.net (Thomas Passin) Date: Tue, 24 Jan 2023 10:21:51 -0500 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <1982769640.893925.1674573213247@mail.yahoo.com> References: <34488860-2ea8-4c5f-9ee6-e513740b354dn@googlegroups.com> <527ff2cc-95d6-f8a0-7705-cacf0b2ab42e@tompassin.net> <1982769640.893925.1674573213247@mail.yahoo.com> Message-ID: <56159013-a2ca-1a16-33d7-3178785923e1@tompassin.net> On 1/24/2023 10:13 AM, Mike Baskin wrote: > Can you stop please It's way past time, isn't it! > Sent from Yahoo Mail for iPhone > > On Tuesday, January 24, 2023, 10:12 AM, Thomas Passin > wrote: > > On 1/23/2023 9:12 PM, Chris Angelico wrote: > > On Tue, 24 Jan 2023 at 13:09, Jach Feng > wrote: > >> > >> Chris Angelico ? 2023?1?24? ?????5:00:27 [UTC+8] ??? > ??? > >>> On Tue, 24 Jan 2023 at 07:47, Cameron Simpson > wrote: > >>>> > >>>> But for Jach Feng: the "--" is really expected as something > the user > >>>> does when they invoke your programme, _explicitly_ saying that > what > >>>> follows from here is not an argument. So the user is expected > to type: > >>>> > >>>> your_script -x -y -- "-4^2+5.3*abs(-2-1)/2" > >>>> > >>>> where there are -x and -y options, then end of options, then an > >>>> argument, which would look like an option if there wasn't the "--" > >>>> argument. > >>> And if you DON'T expect the user to enter the "--", then why use > >>> argparse? You can just check argv directly to get your arguments. > >>> > >>> This entire thread is a massive "how can I use X to do Y?" problem. > >>> > >>> ChrisA > >> The '--' requirement makes its usage less instinctive, and > handling argv directly makes me loss the benefit of using '-h':-) > > > > if "-h" in sys.argv: usage() > > else: do_stuff_with(sys.argv[1:]) > > > > What is argparse really doing for you? > > I second this.? "if '-h' in sys.argv:"? is usually what I do. > > Alternatively, you could use "--arg=" syntax and place your string > "-4^2+5.3*abs(-2-1)/2" its right-hand side": > > infix2postfix [options] "--infix=-4^2+5.3*abs(-2-1)/2" > > This shouldn't be too hard for a user to work with.? You could scan the > argument list for the presence of "--infix=" and display the help > message if it isn't there. > > > -- > https://mail.python.org/mailman/listinfo/python-list > > From gweatherby at uchc.edu Tue Jan 24 10:28:47 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Tue, 24 Jan 2023 15:28:47 +0000 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? WRONG TOOL In-Reply-To: <56159013-a2ca-1a16-33d7-3178785923e1@tompassin.net> References: <34488860-2ea8-4c5f-9ee6-e513740b354dn@googlegroups.com> <527ff2cc-95d6-f8a0-7705-cacf0b2ab42e@tompassin.net> <1982769640.893925.1674573213247@mail.yahoo.com> <56159013-a2ca-1a16-33d7-3178785923e1@tompassin.net> Message-ID: I understand we all want to be helpful and friendly, but it seems to me that helping someone use the wrong tool for the job isn?t really helpful in the long run. argparse is for parsing command line arguments. It?s the wrong tool for this job. As Chris Angelico already said: This entire thread is a massive "how can I use X to do Y?" problem. From: Python-list on behalf of Thomas Passin Date: Tuesday, January 24, 2023 at 10:23 AM To: Mike Baskin , python-list at python.org Subject: Re: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** On 1/24/2023 10:13 AM, Mike Baskin wrote: > Can you stop please It's way past time, isn't it! > Sent from Yahoo Mail for iPhone > > On Tuesday, January 24, 2023, 10:12 AM, Thomas Passin > wrote: > > On 1/23/2023 9:12 PM, Chris Angelico wrote: > > On Tue, 24 Jan 2023 at 13:09, Jach Feng > wrote: > >> > >> Chris Angelico ? 2023?1?24? ?????5:00:27 [UTC+8] ??? > ??? > >>> On Tue, 24 Jan 2023 at 07:47, Cameron Simpson > wrote: > >>>> > >>>> But for Jach Feng: the "--" is really expected as something > the user > >>>> does when they invoke your programme, _explicitly_ saying that > what > >>>> follows from here is not an argument. So the user is expected > to type: > >>>> > >>>> your_script -x -y -- "-4^2+5.3*abs(-2-1)/2" > >>>> > >>>> where there are -x and -y options, then end of options, then an > >>>> argument, which would look like an option if there wasn't the "--" > >>>> argument. > >>> And if you DON'T expect the user to enter the "--", then why use > >>> argparse? You can just check argv directly to get your arguments. > >>> > >>> This entire thread is a massive "how can I use X to do Y?" problem. > >>> > >>> ChrisA > >> The '--' requirement makes its usage less instinctive, and > handling argv directly makes me loss the benefit of using '-h':-) > > > > if "-h" in sys.argv: usage() > > else: do_stuff_with(sys.argv[1:]) > > > > What is argparse really doing for you? > > I second this. "if '-h' in sys.argv:" is usually what I do. > > Alternatively, you could use "--arg=" syntax and place your string > "-4^2+5.3*abs(-2-1)/2" its right-hand side": > > infix2postfix [options] "--infix=-4^2+5.3*abs(-2-1)/2" > > This shouldn't be too hard for a user to work with. You could scan the > argument list for the presence of "--infix=" and display the help > message if it isn't there. > > > -- > https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!g3YRQIW4wminyTVpjV9kATW-QRZ1YhvDi82oJvatyhNe_OABSPTU9c8KIOlztBNofs690OPwUdIykR5ILj4$ > > -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!g3YRQIW4wminyTVpjV9kATW-QRZ1YhvDi82oJvatyhNe_OABSPTU9c8KIOlztBNofs690OPwUdIykR5ILj4$ From torriem at gmail.com Tue Jan 24 14:05:15 2023 From: torriem at gmail.com (Michael Torrie) Date: Tue, 24 Jan 2023 12:05:15 -0700 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <5ab3bc15-1628-4289-9d59-f981d50ac4ffn@googlegroups.com> References: <58b4e9ac-a31a-4751-8e45-292d519155ccn@googlegroups.com> <5ab3bc15-1628-4289-9d59-f981d50ac4ffn@googlegroups.com> Message-ID: <3e728885-1e66-d665-2ad9-bdab12c11e0c@gmail.com> On 1/23/23 18:58, Jach Feng wrote: > More pathonic, but don't work. The '--' must be at index 1:-) I'm very confused. Why are you even using argparse, since if you put -- at index 1 then argparse wont't do any argument parsing at all. If all you want is the expression on the command line, just access it directly. If it's spread out with spaces, you can do something like this to put it back together: expression = " ".join(sys.argv[1:] Otherwise the standard unix way of doing this is to require the user to either provide the -- himself, or put the expression in quotes so it's one unit. From jfong at ms4.hinet.net Tue Jan 24 02:21:02 2023 From: jfong at ms4.hinet.net (Jach Feng) Date: Mon, 23 Jan 2023 23:21:02 -0800 (PST) Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: <5ab3bc15-1628-4289-9d59-f981d50ac4ffn@googlegroups.com> Message-ID: <09425115-57fa-4e89-9d11-3d87cc1ad5a7n@googlegroups.com> cameron... at gmail.com ? 2023?1?24? ?????2:05:33 [UTC+8] ?????? > On 23Jan2023 17:58, Jach Feng wrote: > >>>> parser.parse_args(['--', 'infix2postfix.py', '-4.3+5']) > >usage: [-h] infix > >: error: unrecognized arguments: -4.3+5 > This error doesn't look like "-4.3+5 looks like an option" but instead > "we don't expect any arguments after "infix". > > Not personally a fan of argparse myself, but then I have my own > elaborate command line framework which generally uses getopt for the > option stuff. > > Cheers, > Cameron Simpson Hmm, good to you. During experiments in these days, I had found argparse shows some strange/unstable behaviors in dealing with the leading '-' situation. From dino at no.spam.ar Mon Jan 23 23:22:00 2023 From: dino at no.spam.ar (Dino) Date: Mon, 23 Jan 2023 23:22:00 -0500 Subject: bool and int Message-ID: $ python Python 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> b = True >>> isinstance(b,bool) True >>> isinstance(b,int) True >>> WTF! From bowman at montana.com Tue Jan 24 00:17:50 2023 From: bowman at montana.com (rbowman) Date: 24 Jan 2023 05:17:50 GMT Subject: bool and int References: Message-ID: On Mon, 23 Jan 2023 23:22:00 -0500, Dino wrote: > $ python Python 3.8.10 (default, Mar 15 2022, 12:22:08) > [GCC 9.4.0] on linux Type "help", "copyright", "credits" or "license" > for more information. > >>> b = True isinstance(b,bool) > True > >>> isinstance(b,int) > True > >>> > >>> > WTF! >>> b = True >>> isinstance(b, bool) True >>> isinstance(b, int) True >>> c = b + 10 >>> print(c) 11 >>> b = False >>> c = b + 10 >>> print(c) 10 bool is a subtype of integer. I never dug that deep into Python's guts but I assume it goes back to boolean being an afterthought in C. Some people fancy it up with #defines but I always use int. 0 is false, anything else is true. C# is pickier, which I guess is a good thing. From mikebaskin6538 at yahoo.com Tue Jan 24 09:03:58 2023 From: mikebaskin6538 at yahoo.com (Mike Baskin) Date: Tue, 24 Jan 2023 14:03:58 +0000 (UTC) Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: <5ab3bc15-1628-4289-9d59-f981d50ac4ffn@googlegroups.com> Message-ID: <890186892.886538.1674569038414@mail.yahoo.com> Stop please Sent from Yahoo Mail for iPhone On Tuesday, January 24, 2023, 1:05 AM, Cameron Simpson wrote: On 23Jan2023 17:58, Jach Feng wrote: >>>> parser.parse_args(['--', 'infix2postfix.py', '-4.3+5']) >usage: [-h] infix >: error: unrecognized arguments: -4.3+5 This error doesn't look like "-4.3+5 looks like an option" but instead "we don't expect any arguments after "infix". Not personally a fan of argparse myself, but then I have my own elaborate command line framework which generally uses getopt for the option stuff. Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list From mikebaskin6538 at yahoo.com Tue Jan 24 10:13:33 2023 From: mikebaskin6538 at yahoo.com (Mike Baskin) Date: Tue, 24 Jan 2023 15:13:33 +0000 (UTC) Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <527ff2cc-95d6-f8a0-7705-cacf0b2ab42e@tompassin.net> References: <34488860-2ea8-4c5f-9ee6-e513740b354dn@googlegroups.com> <527ff2cc-95d6-f8a0-7705-cacf0b2ab42e@tompassin.net> Message-ID: <1982769640.893925.1674573213247@mail.yahoo.com> Can you stop please Sent from Yahoo Mail for iPhone On Tuesday, January 24, 2023, 10:12 AM, Thomas Passin wrote: On 1/23/2023 9:12 PM, Chris Angelico wrote: > On Tue, 24 Jan 2023 at 13:09, Jach Feng wrote: >> >> Chris Angelico ? 2023?1?24? ?????5:00:27 [UTC+8] ?????? >>> On Tue, 24 Jan 2023 at 07:47, Cameron Simpson wrote: >>>> >>>> But for Jach Feng: the "--" is really expected as something the user >>>> does when they invoke your programme, _explicitly_ saying that what >>>> follows from here is not an argument. So the user is expected to type: >>>> >>>> your_script -x -y -- "-4^2+5.3*abs(-2-1)/2" >>>> >>>> where there are -x and -y options, then end of options, then an >>>> argument, which would look like an option if there wasn't the "--" >>>> argument. >>> And if you DON'T expect the user to enter the "--", then why use >>> argparse? You can just check argv directly to get your arguments. >>> >>> This entire thread is a massive "how can I use X to do Y?" problem. >>> >>> ChrisA >> The '--' requirement makes its usage less instinctive, and handling argv directly makes me loss the benefit of using '-h':-) > > if "-h" in sys.argv: usage() > else: do_stuff_with(sys.argv[1:]) > > What is argparse really doing for you? I second this.? "if '-h' in sys.argv:"? is usually what I do. Alternatively, you could use "--arg=" syntax and place your string "-4^2+5.3*abs(-2-1)/2" its right-hand side": infix2postfix [options] "--infix=-4^2+5.3*abs(-2-1)/2" This shouldn't be too hard for a user to work with.? You could scan the argument list for the presence of "--infix=" and display the help message if it isn't there. -- https://mail.python.org/mailman/listinfo/python-list From gweatherby at uchc.edu Tue Jan 24 15:29:39 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Tue, 24 Jan 2023 20:29:39 +0000 Subject: bool and int In-Reply-To: References: Message-ID: Booleans work exactly the way the language documentation says they work: Booleans (bool) These represent the truth values False and True. The two objects representing the values False and True are the only Boolean objects. The Boolean type is a subtype of the integer type, and Boolean values behave like the values 0 and 1, respectively, in almost all contexts, the exception being that when converted to a string, the strings "False" or "True"are returned, respectively. https://docs.python.org/3/reference/datamodel.html#the-standard-type-hierarchy From: Python-list on behalf of Dino Date: Tuesday, January 24, 2023 at 3:04 PM To: python-list at python.org Subject: bool and int *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** $ python Python 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> b = True >>> isinstance(b,bool) True >>> isinstance(b,int) True >>> WTF! -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!jPbvUX9ZXFGYt-q850YI6aQ7ET7BwbF-LIT4XT7MKKwF9OSOgqnaHdM4MbQ7p6YaRCYJYXZ4-XiT3Sko$ From gweatherby at uchc.edu Tue Jan 24 15:32:49 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Tue, 24 Jan 2023 20:32:49 +0000 Subject: bool and int In-Reply-To: References: Message-ID: https://peps.python.org/pep-0285/ From: Python-list on behalf of rbowman Date: Tuesday, January 24, 2023 at 3:01 PM To: python-list at python.org Subject: Re: bool and int bool is a subtype of integer. I never dug that deep into Python's guts but I assume it goes back to boolean being an afterthought in C. Some people fancy it up with #defines but I always use int. 0 is false, anything else is true. C# is pickier, which I guess is a good thing. -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!hdxuMNsprOXvH5ouxGfbGLLq6wuXs-_gOESRVYUxDsHYCmlrpv9ru-WYMziYU4FRdum02bS6DfRnNDnCNQ$ From python at mrabarnett.plus.com Tue Jan 24 16:17:48 2023 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 24 Jan 2023 21:17:48 +0000 Subject: bool and int In-Reply-To: References: Message-ID: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> On 2023-01-24 20:32, Weatherby,Gerard wrote: > https://peps.python.org/pep-0285/ > > From: Python-list on behalf of rbowman > Date: Tuesday, January 24, 2023 at 3:01 PM > To: python-list at python.org > Subject: Re: bool and int > > > bool is a subtype of integer. I never dug that deep into Python's guts but > I assume it goes back to boolean being an afterthought in C. Some people > fancy it up with #defines but I always use int. 0 is false, anything else > is true. > bool was introduced early in Python 2. Before then 0 and 1 were used for false and true, like in C, which also gained 'false' and 'true'. For backwards compatibility, bool was made a subclass of int. > C# is pickier, which I guess is a good thing. From rosuav at gmail.com Tue Jan 24 16:58:06 2023 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 25 Jan 2023 08:58:06 +1100 Subject: bool and int In-Reply-To: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> Message-ID: On Wed, 25 Jan 2023 at 08:22, MRAB wrote: > For backwards compatibility, bool was made a subclass of int. Plus, it's really REALLY handy in quite a lot of situations. > > C# is pickier, which I guess is a good thing. > Nope, not a good thing. Actually a highly frustrating thing on those occasions when I have to write C# code. ChrisA From avi.e.gross at gmail.com Tue Jan 24 17:38:44 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Tue, 24 Jan 2023 17:38:44 -0500 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <09425115-57fa-4e89-9d11-3d87cc1ad5a7n@googlegroups.com> References: <5ab3bc15-1628-4289-9d59-f981d50ac4ffn@googlegroups.com> <09425115-57fa-4e89-9d11-3d87cc1ad5a7n@googlegroups.com> Message-ID: <002601d93044$9e1cb090$da5611b0$@gmail.com> If I understood the issue, the problem is the unary minus at the start of the expression. So if you know the expression is in that argument, would it make sense to pad it in one of many ways that are otherwise harmless? Could you put parens on both sides of "-4^2+5.3*abs(-2-1)/2": "(-4^2+5.3*abs(-2-1)/2)" Or perhaps place a zero in front as in the awkward case where it begins with a minus sign: "0 -4^2+5.3*abs(-2-1)/2" Some people suggest you ask the user to modify what they type in and such changes may work. Your program could also potentially diddle with argv and recognize it albeit as you allow calling a function like abs() I can easily imagine a potentially valid looking "-abs(...)..." that could look like -a followed by something. I see a deeper issue with interactions at the command shell level if parts of the arithmetic expression are evaluated or changed before python is even invoked. Then again, I may be misunderstanding the issue. -----Original Message----- From: Python-list On Behalf Of Jach Feng Sent: Tuesday, January 24, 2023 2:21 AM To: python-list at python.org Subject: Re: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? cameron... at gmail.com ? 2023?1?24? ?????2:05:33 [UTC+8] ?????? > On 23Jan2023 17:58, Jach Feng wrote: > >>>> parser.parse_args(['--', 'infix2postfix.py', '-4.3+5']) > >usage: [-h] infix > >: error: unrecognized arguments: -4.3+5 > This error doesn't look like "-4.3+5 looks like an option" but instead > "we don't expect any arguments after "infix". > > Not personally a fan of argparse myself, but then I have my own > elaborate command line framework which generally uses getopt for the > option stuff. > > Cheers, > Cameron Simpson Hmm, good to you. During experiments in these days, I had found argparse shows some strange/unstable behaviors in dealing with the leading '-' situation. -- https://mail.python.org/mailman/listinfo/python-list From wlfraed at ix.netcom.com Tue Jan 24 16:35:32 2023 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Tue, 24 Jan 2023 16:35:32 -0500 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? References: <34488860-2ea8-4c5f-9ee6-e513740b354dn@googlegroups.com> <527ff2cc-95d6-f8a0-7705-cacf0b2ab42e@tompassin.net> <1982769640.893925.1674573213247@mail.yahoo.com> Message-ID: On Tue, 24 Jan 2023 15:13:33 +0000 (UTC), Mike Baskin declaimed the following: >Can you stop please > Stop what? You appear to be subscribed to the Python mailing list -- as such you WILL receive all traffic that appears on that list. There is nothing we can do. It is not a personal "you post, and only direct replies to you show up". BTW: the mailing list is gatewayed to newsgroup comp.lang.python... -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From 2QdxY4RzWzUUiLuE at potatochowder.com Tue Jan 24 18:30:54 2023 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Tue, 24 Jan 2023 18:30:54 -0500 Subject: bool and int In-Reply-To: References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> Message-ID: On 2023-01-25 at 08:58:06 +1100, Chris Angelico wrote: > On Wed, 25 Jan 2023 at 08:22, MRAB wrote: > > For backwards compatibility, bool was made a subclass of int. > > Plus, it's really REALLY handy in quite a lot of situations. > > > > C# is pickier, which I guess is a good thing. > > > > Nope, not a good thing. Actually a highly frustrating thing on those > occasions when I have to write C# code. The usual complaint is that some people write FORTRAN no matter what language they're actually using. Are you writing Python in C#? ;-) There's a sweet spot somewhere that includes dynamic typing, high powered global type inference and optimization systems, a thriving community, and a metric [boatload] of rock solid libraries. And an alomost fanatical devotion to the Pope. :-/ From rosuav at gmail.com Tue Jan 24 20:14:50 2023 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 25 Jan 2023 12:14:50 +1100 Subject: bool and int In-Reply-To: References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> Message-ID: On Wed, 25 Jan 2023 at 10:32, <2QdxY4RzWzUUiLuE at potatochowder.com> wrote: > > On 2023-01-25 at 08:58:06 +1100, > Chris Angelico wrote: > > > On Wed, 25 Jan 2023 at 08:22, MRAB wrote: > > > For backwards compatibility, bool was made a subclass of int. > > > > Plus, it's really REALLY handy in quite a lot of situations. > > > > > > C# is pickier, which I guess is a good thing. > > > > > > > Nope, not a good thing. Actually a highly frustrating thing on those > > occasions when I have to write C# code. > > The usual complaint is that some people write FORTRAN no matter what > language they're actually using. Are you writing Python in C#? ;-) Well, let's see. If I were writing C code, I would write: if (state & PRELAUNCH) If I were writing Python, it would probably be very different, but it depends on the API. Could be: if "PreLaunch" in state: But the way I have to write it in C# is a messed-up version of C: if ((state & StartState.PreLaunch) > 0) { because bool and int are fundamentally different. Using the C style results in: VelocimeterModule.cs(37,8): error CS0029: Cannot implicitly convert type `PartModule.StartState' to `bool' Here's another example. If I were writing C, I would write: if (TimeWarp_CurrentRateIndex) Python? if TimeWarp.CurrentRateIndex: C#? if (TimeWarp.CurrentRateIndex > 0) And, again, if I do it C style, I get: VelocimeterModule.cs(261,17): error CS0029: Cannot implicitly convert type `int' to `bool' I'm pretty sure I've had a case where I wanted to use a boolean in an arithmetic context, too, but it's less obvious from the final code, so I can't give an example. So this is synthetic: autothrust_last_dv *= AT_mode == AT.Idle; VelocimeterModule.cs(252,4): error CS0019: Operator `*=' cannot be applied to operands of type `double' and `bool' So the problem isn't that I'm trying to write Python in C#, but that I'm trying to write code that would work on pretty much *any other C-family language*, but doesn't work on C#. I could use those techniques in plenty of C-derived and C-inspired languages, but nooooo not in C#, despite looking very much C-inspired. Unfortunately the truth is that C# is not *actually* C-inspired; it's really Microsoft Java, so it has all the stupidities of Java: int x = 3 + (args.length > 1); test.java:4: error: bad operand types for binary operator '+' if (args.length) System.out.println("There are args!"); test.java:6: error: incompatible types: int cannot be converted to boolean But this is hardly a Python-versus-C# thing; it's Java versus most of the rest of the world, and C# feigns to be part of the C style while retaining the limitations of Java. (My apologies if the Java entries look synthetic. It's because they are, and that's a consequence of me not having ANY reason to write Java code in, like, ever. In fact, I had to go and install a JDK just to confirm that Java really did have these limitations.) ChrisA From mikebaskin6538 at yahoo.com Tue Jan 24 20:16:31 2023 From: mikebaskin6538 at yahoo.com (Mike Baskin) Date: Tue, 24 Jan 2023 20:16:31 -0500 Subject: bool and int In-Reply-To: References: Message-ID: <310C22AA-D2C5-48E8-B1AB-933D8EACE52C@yahoo.com> Will all of you please stop sending me emails Sent from my iPhone > On Jan 24, 2023, at 2:59 PM, rbowman wrote: > > ?On Mon, 23 Jan 2023 23:22:00 -0500, Dino wrote: > >> $ python Python 3.8.10 (default, Mar 15 2022, 12:22:08) >> [GCC 9.4.0] on linux Type "help", "copyright", "credits" or "license" >> for more information. >>>>> b = True isinstance(b,bool) >> True >>>>> isinstance(b,int) >> True >>>>> >>>>> >> WTF! > > >>>> b = True >>>> isinstance(b, bool) > True >>>> isinstance(b, int) > True >>>> c = b + 10 >>>> print(c) > 11 >>>> b = False >>>> c = b + 10 >>>> print(c) > 10 > > > bool is a subtype of integer. I never dug that deep into Python's guts but > I assume it goes back to boolean being an afterthought in C. Some people > fancy it up with #defines but I always use int. 0 is false, anything else > is true. > > C# is pickier, which I guess is a good thing. > -- > https://mail.python.org/mailman/listinfo/python-list From rosuav at gmail.com Tue Jan 24 20:23:37 2023 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 25 Jan 2023 12:23:37 +1100 Subject: bool and int In-Reply-To: <310C22AA-D2C5-48E8-B1AB-933D8EACE52C@yahoo.com> References: <310C22AA-D2C5-48E8-B1AB-933D8EACE52C@yahoo.com> Message-ID: On Wed, 25 Jan 2023 at 12:20, Mike Baskin via Python-list wrote: > > Will all of you please stop sending me emails Learn to manage your own mailing list subscription. HINT: Look at the *ENTIRE* email, not just the bit up the top that makes you angry. > Sent from my iPhone Ahh, I see the problem here. You probably can't even read the entire email. Good luck. If you can find the link, you can unsubscribe from the mailing list, although you still probably won't be able to stop yourself from being a constant advertisement bragging "Oh I am the greatest, I use default iPhone apps, ain't I awesome". Forgive me for thinking that it's utterly lame. There's probably a good reason for bragging about iPhone usage on EVERY SINGLE EMAIL, but I've just never seen it. ChrisA From bouncingcats at gmail.com Tue Jan 24 20:36:06 2023 From: bouncingcats at gmail.com (David) Date: Wed, 25 Jan 2023 12:36:06 +1100 Subject: bool and int In-Reply-To: <310C22AA-D2C5-48E8-B1AB-933D8EACE52C@yahoo.com> References: <310C22AA-D2C5-48E8-B1AB-933D8EACE52C@yahoo.com> Message-ID: On Wed, 25 Jan 2023 at 12:19, Mike Baskin via Python-list wrote: > Will all of you please stop sending me emails Hi. We don't have the power to do that. Because this is a public list, which works by people adding and removing themselves. You, or perhaps someone messing with you, added your email address to it. So you have to remove yourself. The link to do that is at the bottom of every message. See here: https://mail.python.org/mailman/listinfo/python-list Go to the bottom, enter your email address, click the button next to it named "Unsubscribe or edit options" and follow the instructions. From avi.e.gross at gmail.com Tue Jan 24 20:42:02 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Tue, 24 Jan 2023 20:42:02 -0500 Subject: bool and int In-Reply-To: References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> Message-ID: <002901d9305e$395f8280$ac1e8780$@gmail.com> Python yet again is being asked why something is the way it is and not as someone insists it should be. It is a tool to be used the way it SAYS it works so the bug is perhaps in the user and their expectations. It is what it is and would break lots of things if changed without much thought. Every other language I know has similar detractors asking why it is the way it is. A useful example where the use of Booleans as 0/1 is commonly used is to find how many of something satisfy some criteria, such as how many items in a list are numbers greater than 65 representing test scores that "passed". It could also be a numpy type of array or a column of a data.frame and so on. The common method is to create a Boolean data structure (often implicitly) that is then viewed as a bunch of 0's and 1's and you can count them to see how many are True (meaning they qualify) or calculate a percentage simply by taking a mean of the 0/1 values. There are other ways, but some also use sums of Booleans to calculate things like any() or all() and I am sure many other things. Yes, all these can be done another way if Booleans were not allowed to be viewed as very small integers. So only use a Boolean in relatively pure situations where arithmetic as a zero or one is meaningful. In other places, maybe don't. The reality though is that if you have 10 apples and you get another one if you flip a coin and it lands on heads. Then 10 + Boolean would indeed be 11. If you have five losing flips (or tickets or whatever) then 5 * Boolean indeed is a zero. Python has a different philosophy than some other languages with strong typing. In some of those, you would not be allowed to add or multiply at random but would need to convert parts of your calculation to all be the same, such as a 32-bit integer. You could still do things like I mention above but only after consciously mapping your Boolean to an actual zero or one of the kind wanted. I worry a tad more about the other direction where something like an integer containing a number like 12 is used in a context where it gets downgraded to a True/False and later may inadvertently be used as a "1" as the conversion is not anticipated. There is data loss there more than in the case of a Boolean becoming a 1. -----Original Message----- From: Python-list On Behalf Of 2QdxY4RzWzUUiLuE at potatochowder.com Sent: Tuesday, January 24, 2023 6:31 PM To: python-list at python.org Subject: Re: bool and int On 2023-01-25 at 08:58:06 +1100, Chris Angelico wrote: > On Wed, 25 Jan 2023 at 08:22, MRAB wrote: > > For backwards compatibility, bool was made a subclass of int. > > Plus, it's really REALLY handy in quite a lot of situations. > > > > C# is pickier, which I guess is a good thing. > > > > Nope, not a good thing. Actually a highly frustrating thing on those > occasions when I have to write C# code. The usual complaint is that some people write FORTRAN no matter what language they're actually using. Are you writing Python in C#? ;-) There's a sweet spot somewhere that includes dynamic typing, high powered global type inference and optimization systems, a thriving community, and a metric [boatload] of rock solid libraries. And an alomost fanatical devotion to the Pope. :-/ -- https://mail.python.org/mailman/listinfo/python-list From rosuav at gmail.com Tue Jan 24 21:01:24 2023 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 25 Jan 2023 13:01:24 +1100 Subject: bool and int In-Reply-To: <002901d9305e$395f8280$ac1e8780$@gmail.com> References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> <002901d9305e$395f8280$ac1e8780$@gmail.com> Message-ID: On Wed, 25 Jan 2023 at 12:43, wrote: > Python has a different philosophy than some other languages with strong > typing. In some of those, you would not be allowed to add or multiply at > random but would need to convert parts of your calculation to all be the > same, such as a 32-bit integer. You could still do things like I mention > above but only after consciously mapping your Boolean to an actual zero or > one of the kind wanted. Python is strongly dynamically typed. You may be thinking of "static typing" rather than "strong typing" here, and there are plenty of strongly statically typed languages that allow you to do arithmetic on booleans (with them usually behaving as if False is 0 and True is 1, although not always). > I worry a tad more about the other direction where something like an integer > containing a number like 12 is used in a context where it gets downgraded to > a True/False and later may inadvertently be used as a "1" as the conversion > is not anticipated. There is data loss there more than in the case of a > Boolean becoming a 1. Well, yes, but that's no different from a float like 6.25 getting downgraded to an integer 6. There's a reason that most languages upcast silently but don't downcast without being asked to. ChrisA From grant.b.edwards at gmail.com Tue Jan 24 22:02:15 2023 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Tue, 24 Jan 2023 19:02:15 -0800 (PST) Subject: bool and int References: <310C22AA-D2C5-48E8-B1AB-933D8EACE52C@yahoo.com> Message-ID: <63d09bb7.050a0220.c9e76.45e9@mx.google.com> On 2023-01-25, Mike Baskin via Python-list wrote: > Will all of you please stop sending me emails Oh dear. You might want to try unsubscribing from the list. Telling everybody to stop using the mailing list and newsgroup is a bit silly. -- Grant From jfong at ms4.hinet.net Tue Jan 24 22:09:37 2023 From: jfong at ms4.hinet.net (Jach Feng) Date: Tue, 24 Jan 2023 19:09:37 -0800 (PST) Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: <58b4e9ac-a31a-4751-8e45-292d519155ccn@googlegroups.com> <3e728885-1e66-d665-2ad9-bdab12c11e0c@gmail.com> <5ab3bc15-1628-4289-9d59-f981d50ac4ffn@googlegroups.com> Message-ID: <20aaaffb-6846-4277-ae9a-7a4ec482b392n@googlegroups.com> Michael Torrie ? 2023?1?25? ?????3:05:44 [UTC+8] ?????? > On 1/23/23 18:58, Jach Feng wrote: > > More pathonic, but don't work. The '--' must be at index 1:-) > I'm very confused. Why are you even using argparse, since if you put -- > at index 1 then argparse wont't do any argument parsing at all. If all > you want is the expression on the command line, just access it directly. > If it's spread out with spaces, you can do something like this to put > it back together: > > expression = " ".join(sys.argv[1:] > > Otherwise the standard unix way of doing this is to require the user to > either provide the -- himself, or put the expression in quotes so it's > one unit. Maybe a condition is required before the modification, if len(sys.argv) > 1 and not '-h' in sys.argv: sys.argv.insert(1, '--') From jfong at ms4.hinet.net Tue Jan 24 22:11:18 2023 From: jfong at ms4.hinet.net (Jach Feng) Date: Tue, 24 Jan 2023 19:11:18 -0800 (PST) Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: <5ab3bc15-1628-4289-9d59-f981d50ac4ffn@googlegroups.com> <002601d93044$9e1cb090$da5611b0$@gmail.com> <09425115-57fa-4e89-9d11-3d87cc1ad5a7n@googlegroups.com> Message-ID: <6616025e-e7b8-4772-9ccd-cfa285612086n@googlegroups.com> avi.e... at gmail.com ? 2023?1?25? ?????6:39:13 [UTC+8] ?????? > If I understood the issue, the problem is the unary minus at the start of the expression. > > So if you know the expression is in that argument, would it make sense to pad it in one of many ways that are otherwise harmless? > > Could you put parens on both sides of "-4^2+5.3*abs(-2-1)/2": > > "(-4^2+5.3*abs(-2-1)/2)" > > Or perhaps place a zero in front as in the awkward case where it begins with a minus sign: > > "0 -4^2+5.3*abs(-2-1)/2" > > Some people suggest you ask the user to modify what they type in and such changes may work. Your program could also potentially diddle with argv and recognize it albeit as you allow calling a function like abs() I can easily imagine a potentially valid looking "-abs(...)..." that could look like -a followed by something. > > I see a deeper issue with interactions at the command shell level if parts of the arithmetic expression are evaluated or changed before python is even invoked. > > Then again, I may be misunderstanding the issue. > -----Original Message----- > From: Python-list On Behalf Of Jach Feng > Sent: Tuesday, January 24, 2023 2:21 AM > To: pytho... at python.org > Subject: Re: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? > cameron... at gmail.com ? 2023?1?24? ?????2:05:33 [UTC+8] ?????? > > On 23Jan2023 17:58, Jach Feng wrote: > > >>>> parser.parse_args(['--', 'infix2postfix.py', '-4.3+5']) > > >usage: [-h] infix > > >: error: unrecognized arguments: -4.3+5 > > This error doesn't look like "-4.3+5 looks like an option" but instead > > "we don't expect any arguments after "infix". > > > > Not personally a fan of argparse myself, but then I have my own > > elaborate command line framework which generally uses getopt for the > > option stuff. > > > > Cheers, > > Cameron Simpson > Hmm, good to you. During experiments in these days, I had found argparse shows some strange/unstable behaviors in dealing with the leading '-' situation. > -- > https://mail.python.org/mailman/listinfo/python-list The user may not know any unix idiom, or even don't know Python. Ask them to add extra characters in their equations to avoid the argparse conflict is strange to me. From jfong at ms4.hinet.net Tue Jan 24 22:18:52 2023 From: jfong at ms4.hinet.net (Jach Feng) Date: Tue, 24 Jan 2023 19:18:52 -0800 (PST) Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: <741f9e53-ecdd-4c49-ae1c-4697661e21e1n@googlegroups.com> Jach Feng ? 2023?1?22? ?????11:11:22 [UTC+8] ?????? > Fail on command line, > > e:\Works\Python>py infix2postfix.py "-4^2+5.3*abs(-2-1)/2" > usage: infix2postfix.py [-h] [infix] > infix2postfix.py: error: unrecognized arguments: -4^2+5.3*abs(-2-1)/2 > > Also fail in REPL, > > e:\Works\Python>py > Python 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:08:11) [MSC v.1928 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import argparse > >>> parser = argparse.ArgumentParser(description='Convert infix notation to postfix') > >>> parser.parse_args("-4^2+5.3*abs(-2-1)/2") > usage: [-h] > : error: unrecognized arguments: - 4 ^ 2 + 5 . 3 * a b s ( - 2 - 1 ) / 2 > > Just can't figure out where is wrong!? > > --Jach I was happy working with argparse during implement my script. To save the typing, I used a default equation for testing. sample = "-4^2+5.3*abs(-2-1)/2, abs(Abc)*(B+C)/D, (-3) * sqrt(1-(x1/7)*(y1/7)) * sqrt(abs((x0-4.5)/(y0-4)))" parser = argparse.ArgumentParser(description='Convert infix notation to postfix') parser.add_argument('infix', nargs='?', default=sample, help="....") The argparse has no complain at all, even I enter it in the command line. e:\Works\Python>py infix2postfix.py "-4^2+5.3*abs(-2-1)/2, abs(Abc)*(B+C)/D, (-3) * sqrt(1-(x1/7)*(y1/7)) * sqrt(abs((x0 -4.5)/(y0-4)))" -4 2 ^ 5.3 -2 1 - abs * 2 / + Abc abs B C + * D / -3 1 x1 7 / y1 7 / * - sqrt * x0 4.5 - y0 4 - / abs sqrt * But the happiness ends when this day comes, e:\Works\Python>py infix2postfix.py "-4^2+5.3*abs(-2-1)/2" usage: infix2postfix.py [-h] [infix] infix2postfix.py: error: unrecognized arguments: -4^2+5.3*abs(-2-1)/2 Then, I know I am in trouble, and You know the rest of the story:-) From bowman at montana.com Tue Jan 24 22:25:08 2023 From: bowman at montana.com (rbowman) Date: 25 Jan 2023 03:25:08 GMT Subject: bool and int References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> Message-ID: On Wed, 25 Jan 2023 12:14:50 +1100, Chris Angelico wrote: > So the problem isn't that I'm trying to write Python in C#, but that I'm > trying to write code that would work on pretty much *any other C-family > language*, but doesn't work on C#. I could use those techniques in > plenty of C-derived and C-inspired languages, but nooooo not in C#, > despite looking very much C-inspired. Unfortunately the truth is that C# > is not *actually* C-inspired; it's really Microsoft Java, so it has all > the stupidities of Java: I have the installation media for Visual J++ around here someplace. It lasted for a few years before Sun sued for non-compliance. There definitely is some of its DNA in C#. I prefer C# to C++ but it does have annoyances. It's only a warning that can be pragma'd but string foo; complaining that foo may be null unless you shut it up with string? foo; can be frustrating. It has a germ of a good idea for structs (classes) where some of the members are optional. The bool hoops definitely are another sore point after a few decades of C. I can switch gears for Python but the first day or two is painful. I was getting a bit frustrated until I remembered the range() thing. From rosuav at gmail.com Wed Jan 25 00:15:49 2023 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 25 Jan 2023 16:15:49 +1100 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <741f9e53-ecdd-4c49-ae1c-4697661e21e1n@googlegroups.com> References: <741f9e53-ecdd-4c49-ae1c-4697661e21e1n@googlegroups.com> Message-ID: On Wed, 25 Jan 2023 at 14:42, Jach Feng wrote: > I was happy working with argparse during implement my script. To save the typing, I used a default equation for testing. > > sample = "-4^2+5.3*abs(-2-1)/2, abs(Abc)*(B+C)/D, (-3) * sqrt(1-(x1/7)*(y1/7)) * sqrt(abs((x0-4.5)/(y0-4)))" > parser = argparse.ArgumentParser(description='Convert infix notation to postfix') > parser.add_argument('infix', nargs='?', default=sample, help="....") > You're still not really using argparse as an argument parser. Why not just do your own -h checking? Stop trying to use argparse for what it's not designed for, and then wondering why it isn't doing what you expect it to magically know. ChrisA From cs at cskk.id.au Wed Jan 25 01:45:28 2023 From: cs at cskk.id.au (Cameron Simpson) Date: Wed, 25 Jan 2023 17:45:28 +1100 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: On 25Jan2023 16:15, Chris Angelico wrote: >On Wed, 25 Jan 2023 at 14:42, Jach Feng wrote: >> I was happy working with argparse during implement my script. To save the typing, I used a default equation for testing. Sure, but what benefit was it bringing you? Just the usage (help) message? Did you have many options to handle? >> sample = "-4^2+5.3*abs(-2-1)/2, abs(Abc)*(B+C)/D, (-3) * sqrt(1-(x1/7)*(y1/7)) * sqrt(abs((x0-4.5)/(y0-4)))" >> parser = argparse.ArgumentParser(description='Convert infix notation to postfix') >> parser.add_argument('infix', nargs='?', default=sample, help="....") If this was the whole thing, I don't see what argparse was doing for you which was better than just handling the arguments yourself - there's only one after all. >You're still not really using argparse as an argument parser. Why not >just do your own -h checking? Stop trying to use argparse for what >it's not designed for, and then wondering why it isn't doing what you >expect it to magically know. I'm with Chris here. As someone who pretty well _never_ uses argparse, and occasionally uses getopt, I'd do something like this: usage = '''Usage: infix Parse the argument infix as an expression, print the result.''' badopts = False cmd = argv.pop(0) if not argv: print(f'{cmd}: missing infix argument', file=sys.stderr) badopts = True else: infix = argv.pop(0) if infix in ('-h', '--help'): print(usage) exit(0) if argv: print(f'{cmd}: extra arguments after infix: {argv!r}', file=sys.stderr) badopts = True if badopts: print(usage, file=sys.stderr) exit(2) ... work with infix as desired ... This: - avoids trying to shoehorn argparse into behaving in a way it was not designed for - avoids users needing to know the standard UNIX/POSIX "--" idiom for marking off the end of options - supports a -h or --help leading option Cheers, Cameron Simpson From 2QdxY4RzWzUUiLuE at potatochowder.com Wed Jan 25 06:53:44 2023 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Wed, 25 Jan 2023 06:53:44 -0500 Subject: bool and int In-Reply-To: References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> Message-ID: On 2023-01-25 at 12:14:50 +1100, Chris Angelico wrote: > On Wed, 25 Jan 2023 at 10:32, <2QdxY4RzWzUUiLuE at potatochowder.com> wrote: > > The usual complaint is that some people write FORTRAN no matter what > > language they're actually using. Are you writing Python in C#? ;-) > But the way I have to write it in C# is a messed-up version of C: There's your problem: C# isn't C, it's Java. Java looks like C, too, but it isn't C, either. > So the problem isn't that I'm trying to write Python in C#, but that > I'm trying to write code that would work on pretty much *any other > C-family language*, but doesn't work on C#. I could use those > techniques in plenty of C-derived and C-inspired languages, but nooooo > not in C#, despite looking very much C-inspired. Unfortunately the > truth is that C# is not *actually* C-inspired; it's really Microsoft > Java, so it has all the stupidities of Java: There. Even ChrisA agrees with me. ;-) So, I think what you're trying to say is that you prefer the razor sharp quality of truthiness to the zen of explicit being better than implicit. To bring this back to Python (sorry), blurring the line between booleans and integers is an old machine language trick, born of the days when we measured memory in bytes (and large sums of cash!) rather than gigs[0]. In Python3, there's no more reason to use a boolean value as integer (whether to accumulate values or to test a value against zero) as there is to use a string (e.g., from an HTML form) as an integer. [0] I remember meetings where the agenda was to allocate memory (yes, at design time) for a particular value, and the answer was along the lines of "you can have these five bits, and this would have taken a lot less time had you told us sooner that you needed that value to persist across input messages." > But this is hardly a Python-versus-C# thing; it's Java versus most of > the rest of the world, and C# feigns to be part of the C style while > retaining the limitations of Java. IMO, the problem started when Java tried to be too much like C to attract (or should I say "trap"?) C developers. > (My apologies if the Java entries look synthetic. It's because they > are, and that's a consequence of me not having ANY reason to write > Java code in, like, ever. In fact, I had to go and install a JDK just > to confirm that Java really did have these limitations.) They used Java at my last job (as in, the last job I had before I retired), and it was absolutely awful, for any number of reasons, the gymnastics (on many levels) required to support "primitive types" being one of them. From rosuav at gmail.com Wed Jan 25 07:11:37 2023 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 25 Jan 2023 23:11:37 +1100 Subject: bool and int In-Reply-To: References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> Message-ID: On Wed, 25 Jan 2023 at 22:55, <2QdxY4RzWzUUiLuE at potatochowder.com> wrote: > > On 2023-01-25 at 12:14:50 +1100, > Chris Angelico wrote: > > > On Wed, 25 Jan 2023 at 10:32, <2QdxY4RzWzUUiLuE at potatochowder.com> wrote: > > > > The usual complaint is that some people write FORTRAN no matter what > > > language they're actually using. Are you writing Python in C#? ;-) > > > But the way I have to write it in C# is a messed-up version of C: > > There's your problem: C# isn't C, it's Java. Java looks like C, too, > but it isn't C, either. > > > So the problem isn't that I'm trying to write Python in C#, but that > > I'm trying to write code that would work on pretty much *any other > > C-family language*, but doesn't work on C#. I could use those > > techniques in plenty of C-derived and C-inspired languages, but nooooo > > not in C#, despite looking very much C-inspired. Unfortunately the > > truth is that C# is not *actually* C-inspired; it's really Microsoft > > Java, so it has all the stupidities of Java: > > There. Even ChrisA agrees with me. ;-) > > So, I think what you're trying to say is that you prefer the razor sharp > quality of truthiness to the zen of explicit being better than implicit. Not sure what you mean here. If you want to bring this back to the Zen of Python, I would reference "practicality beats purity". We can do arithmetic on integers and floats without having to explicitly cast one to the other, because there's really no point in distinguishing them. We can do that with booleans and other types, too. > To bring this back to Python (sorry), blurring the line between booleans > and integers is an old machine language trick, born of the days when we > measured memory in bytes (and large sums of cash!) rather than gigs[0]. > In Python3, there's no more reason to use a boolean value as integer > (whether to accumulate values or to test a value against zero) as there > is to use a string (e.g., from an HTML form) as an integer. Strongly disagree. There is PLENTY of practical value in using booleans as numbers. This is nothing to do with counting bytes, and everything to do with how useful it is in practice. > > But this is hardly a Python-versus-C# thing; it's Java versus most of > > the rest of the world, and C# feigns to be part of the C style while > > retaining the limitations of Java. > > IMO, the problem started when Java tried to be too much like C to > attract (or should I say "trap"?) C developers. Maybe. No idea. In any case, Java's restrictiveness is VERY different from the way Python works (for instance, operator overloading, the way strings are handled, etc), and I don't think people here want the shackles of Java. > > (My apologies if the Java entries look synthetic. It's because they > > are, and that's a consequence of me not having ANY reason to write > > Java code in, like, ever. In fact, I had to go and install a JDK just > > to confirm that Java really did have these limitations.) > > They used Java at my last job (as in, the last job I had before I > retired), and it was absolutely awful, for any number of reasons, the > gymnastics (on many levels) required to support "primitive types" being > one of them. Yeah. "Primitive types" being different from "boxed types" is an unnecessary and arbitrary distinction, like between types and classes, or old-style classes and those that subclass object. Neither is needed in modern Python, neither is beneficial. Keeping bool as a subclass of int has never been in dispute. Nor has the treatment of other types in a boolean context. I can't find anything in the Python 3000 proposals that ever even suggested changing either; the best I can find is this reference in PEP 285 which rejected the notions: """ Should we strive to eliminate non-Boolean operations on bools in the future, through suitable warnings, so that for example True+1 would eventually (in Python 3000) be illegal? => No. Should we strive to require that Boolean operations (like ?if?, ?and?, ?not?) have a bool as an argument in the future, so that for example ?if []:? would become illegal and would have to be written as ?if bool([]):? ??? => No!!! """ See: https://peps.python.org/pep-0285/ C# is a pain to work with because it fails on these points of practicality. I'm morbidly curious as to how C# fanatics justify this sort of thing, given that so many languages just DTRT without needing to be told. ChrisA From list1 at tompassin.net Wed Jan 25 08:47:28 2023 From: list1 at tompassin.net (Thomas Passin) Date: Wed, 25 Jan 2023 08:47:28 -0500 Subject: bool and int In-Reply-To: References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> Message-ID: <22c09ec5-5f73-49e5-80b0-71b23214312d@tompassin.net> On 1/25/2023 6:53 AM, 2QdxY4RzWzUUiLuE at potatochowder.com wrote: > They used Java at my last job (as in, the last job I had before I > retired), and it was absolutely awful, for any number of reasons, the > gymnastics (on many levels) required to support "primitive types" being > one of them. In my one serious java program, a Tomcat application, I use a small amount of Java for the servlets and startup filters, and Jython for the heart of the computing. Thank goodness for Jython! From 2QdxY4RzWzUUiLuE at potatochowder.com Wed Jan 25 11:18:07 2023 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Wed, 25 Jan 2023 11:18:07 -0500 Subject: bool and int In-Reply-To: References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> Message-ID: On 2023-01-25 at 23:11:37 +1100, Chris Angelico wrote: > On Wed, 25 Jan 2023 at 22:55, <2QdxY4RzWzUUiLuE at potatochowder.com> wrote: > > So, I think what you're trying to say is that you prefer the razor sharp > > quality of truthiness to the zen of explicit being better than implicit. > > Not sure what you mean here. If you want to bring this back to the Zen > of Python, I would reference "practicality beats purity". We can do > arithmetic on integers and floats without having to explicitly cast > one to the other, because there's really no point in distinguishing > them. We can do that with booleans and other types, too. My point was that we all have our own preferences and biases, and in this case, I think you and I lean in opposite directions, and Python is big enough for both of us. > > To bring this back to Python (sorry), blurring the line between booleans > > and integers is an old machine language trick, born of the days when we > > measured memory in bytes (and large sums of cash!) rather than gigs[0]. > > In Python3, there's no more reason to use a boolean value as integer > > (whether to accumulate values or to test a value against zero) as there > > is to use a string (e.g., from an HTML form) as an integer. > > Strongly disagree. There is PLENTY of practical value in using > booleans as numbers. This is nothing to do with counting bytes, and > everything to do with how useful it is in practice. IMO, the difference in readability between autothrust_last_dv *= AT_mode == AT.Idle; and if(AT_mode != AT.Idle) autothrust_last_dv = 0; outweighs the practicality, whether C, C#, Java, or Python (ignoring the insignificant differences in syntax). Maintainability, too: as soon as there's something else to do when AT_mode isn't AT.Idle, I'm going to rewrite the first one as the second one anyway. (No, I won't mess up the braces.) I could argue that the only reason the first one is readable at all is that we've both been exposed to languages where fanatics assume that True is 1 and False is 0. I've also written low-level code with hardware fanatics who insist that True is 0 and False is -1 (or 255, depending on how much math they know). In a posix shell script (or a program that knows it might be run inside such a script), 0 is "true" and non-zero is "false." My point here is that you have to understand how to work within whatever environment you're using, and that future programmers (and future you!) will have to deal with your choices regardless of their background, biases, and preferences. > C# is a pain to work with because it fails on these points of > practicality. I'm morbidly curious as to how C# fanatics justify this > sort of thing, given that so many languages just DTRT without needing > to be told. They say that Perl is practical. ;-) From rosuav at gmail.com Wed Jan 25 12:10:30 2023 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 26 Jan 2023 04:10:30 +1100 Subject: bool and int In-Reply-To: References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> Message-ID: On Thu, 26 Jan 2023 at 03:19, <2QdxY4RzWzUUiLuE at potatochowder.com> wrote: > > Strongly disagree. There is PLENTY of practical value in using > > booleans as numbers. This is nothing to do with counting bytes, and > > everything to do with how useful it is in practice. > > IMO, the difference in readability between > > autothrust_last_dv *= AT_mode == AT.Idle; > > and > > if(AT_mode != AT.Idle) > autothrust_last_dv = 0; > > outweighs the practicality, whether C, C#, Java, or Python (ignoring the > insignificant differences in syntax). Yeah, that example was completely synthetic and not really a good showcase. But what often comes up is either multiplying a string by a boolean (equivalent to "have this string if this is true") or subscripting something with a boolean. > I could argue that the only reason the first one is readable at all is > that we've both been exposed to languages where fanatics assume that > True is 1 and False is 0. I'm fine with any definition as long as it's dependable. > I've also written low-level code with > hardware fanatics who insist that True is 0 and False is -1 (or 255, > depending on how much math they know). BASIC was like that too, although it (at least, the versions I used in my childhood) didn't have "True" and "False", you just got the actual values -1 and 0. They were the other way around compared to what you're saying here though. > In a posix shell script (or a > program that knows it might be run inside such a script), 0 is "true" > and non-zero is "false." And that's a consequence of a system wherein there is only one concept of "success", but many concepts of "failure". Whoever devised that system was clearly a pessimist :) > My point here is that you have to understand > how to work within whatever environment you're using, and that future > programmers (and future you!) will have to deal with your choices > regardless of their background, biases, and preferences. That's always the case, though. ChrisA From jfong at ms4.hinet.net Wed Jan 25 01:28:58 2023 From: jfong at ms4.hinet.net (Jach Feng) Date: Tue, 24 Jan 2023 22:28:58 -0800 (PST) Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: <741f9e53-ecdd-4c49-ae1c-4697661e21e1n@googlegroups.com> Message-ID: <31e34753-97a2-4ccb-bbd4-03922357ce78n@googlegroups.com> Chris Angelico ? 2023?1?25? ?????1:16:25 [UTC+8] ?????? > On Wed, 25 Jan 2023 at 14:42, Jach Feng wrote: > > I was happy working with argparse during implement my script. To save the typing, I used a default equation for testing. > > > > sample = "-4^2+5.3*abs(-2-1)/2, abs(Abc)*(B+C)/D, (-3) * sqrt(1-(x1/7)*(y1/7)) * sqrt(abs((x0-4.5)/(y0-4)))" > > parser = argparse.ArgumentParser(description='Convert infix notation to postfix') > > parser.add_argument('infix', nargs='?', default=sample, help="....") > > > You're still not really using argparse as an argument parser. Why not > just do your own -h checking? Stop trying to use argparse for what > it's not designed for, and then wondering why it isn't doing what you > expect it to magically know. > > ChrisA I just don't get what you mean? > You're still not really using argparse as an argument parser. Why not just do your own -h checking? Is a math equation not qualified as a command line "argument"? What criteria do you use when judging the quality of an "argument"? > Stop trying to use argparse for what it's not designed for, Even the author considers a positional argument begin with '-' is a legal argument. Below is a quote from its manual. "If you have positional arguments that must begin with - and don?t look like negative numbers, you can insert the pseudo-argument '--' which tells parse_args() that everything after that is a positional argument" > and then wondering why it isn't doing what you expect it to magically know." I don't expect magic, I expect the consistency of a parser. From dino at no.spam.ar Wed Jan 25 10:53:34 2023 From: dino at no.spam.ar (Dino) Date: Wed, 25 Jan 2023 10:53:34 -0500 Subject: HTTP server benchmarking/load testing in Python Message-ID: <4gcAL.2393259$JNZ4.383220@fx12.ams4> Hello, I could use something like Apache ab in Python ( https://httpd.apache.org/docs/2.4/programs/ab.html ). The reason why ab doesn't quite cut it for me is that I need to define a pool of HTTP requests and I want the tool to run those (as opposed to running the same request over and over again) Does such a marvel exist? Thinking about it, it doesn't necessarily need to be Python, but I guess I would have a chance to tweak things if it was. Thanks Dino From bowman at montana.com Wed Jan 25 11:24:14 2023 From: bowman at montana.com (rbowman) Date: 25 Jan 2023 16:24:14 GMT Subject: bool and int References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> Message-ID: On Wed, 25 Jan 2023 06:53:44 -0500, 2QdxY4RzWzUUiLuE wrote: > They used Java at my last job (as in, the last job I had before I > retired), and it was absolutely awful, for any number of reasons, the > gymnastics (on many levels) required to support "primitive types" being > one of them. My first brush with Java was around '98 when it was first becoming popular. To familiarize myself with the AWT I decided to write a simple IDE for the AVR microcontrollers. What a disaster. The UI wasn't bad but the instructions for 8-bit processors require a lot of bit fiddling that was extraordinarily difficult in Java. Then they came out with Swing and the assumption if the app ran with glacial slowness you should get a faster machine. The company I work for has one Java app created around 2000 as a cross platform solution as people moved to Windows. Originally it ran as an applet but when that window was slammed shut it became increasingly unwieldy. For what I'm developing today I used either .NET C# or Python3. The .NET UI's on Linux aren't quite there yet but back end applications are fine. PyQt (PySide actually. If there is a way to screw up commercial licensing Qt will find it) is fine. From rosuav at gmail.com Wed Jan 25 12:28:33 2023 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 26 Jan 2023 04:28:33 +1100 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <31e34753-97a2-4ccb-bbd4-03922357ce78n@googlegroups.com> References: <741f9e53-ecdd-4c49-ae1c-4697661e21e1n@googlegroups.com> <31e34753-97a2-4ccb-bbd4-03922357ce78n@googlegroups.com> Message-ID: On Thu, 26 Jan 2023 at 04:25, Jach Feng wrote: > > Chris Angelico ? 2023?1?25? ?????1:16:25 [UTC+8] ?????? > > On Wed, 25 Jan 2023 at 14:42, Jach Feng wrote: > > > I was happy working with argparse during implement my script. To save the typing, I used a default equation for testing. > > > > > > sample = "-4^2+5.3*abs(-2-1)/2, abs(Abc)*(B+C)/D, (-3) * sqrt(1-(x1/7)*(y1/7)) * sqrt(abs((x0-4.5)/(y0-4)))" > > > parser = argparse.ArgumentParser(description='Convert infix notation to postfix') > > > parser.add_argument('infix', nargs='?', default=sample, help="....") > > > > > You're still not really using argparse as an argument parser. Why not > > just do your own -h checking? Stop trying to use argparse for what > > it's not designed for, and then wondering why it isn't doing what you > > expect it to magically know. > > > > ChrisA > I just don't get what you mean? > > > You're still not really using argparse as an argument parser. Why not just do your own -h checking? > > Is a math equation not qualified as a command line "argument"? What criteria do you use when judging the quality of an "argument"? > Print out sys.argv and then figure out whether you need an argument *parser* to *parse* your arguments. From what I'm seeing, you don't. You just need the arguments. ChrisA From gweatherby at uchc.edu Wed Jan 25 13:03:46 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Wed, 25 Jan 2023 18:03:46 +0000 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <31e34753-97a2-4ccb-bbd4-03922357ce78n@googlegroups.com> References: <741f9e53-ecdd-4c49-ae1c-4697661e21e1n@googlegroups.com> <31e34753-97a2-4ccb-bbd4-03922357ce78n@googlegroups.com> Message-ID: Use a different prefix character parser = argparse.ArgumentParser(prefix_chars='%') parser.add_argument('expression') args = parser.parse_args() print(args.expression) argparser is for allowing multiple command line options to be passed, providing default, controlling the number of arguments and the like. From: Python-list on behalf of Jach Feng Date: Wednesday, January 25, 2023 at 12:25 PM To: python-list at python.org Subject: Re: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** Chris Angelico ? 2023?1?25? ?????1:16:25 [UTC+8] ?????? > On Wed, 25 Jan 2023 at 14:42, Jach Feng wrote: > > I was happy working with argparse during implement my script. To save the typing, I used a default equation for testing. > > > > sample = "-4^2+5.3*abs(-2-1)/2, abs(Abc)*(B+C)/D, (-3) * sqrt(1-(x1/7)*(y1/7)) * sqrt(abs((x0-4.5)/(y0-4)))" > > parser = argparse.ArgumentParser(description='Convert infix notation to postfix') > > parser.add_argument('infix', nargs='?', default=sample, help="....") > > > You're still not really using argparse as an argument parser. Why not > just do your own -h checking? Stop trying to use argparse for what > it's not designed for, and then wondering why it isn't doing what you > expect it to magically know. > > ChrisA I just don't get what you mean? > You're still not really using argparse as an argument parser. Why not just do your own -h checking? Is a math equation not qualified as a command line "argument"? What criteria do you use when judging the quality of an "argument"? > Stop trying to use argparse for what it's not designed for, Even the author considers a positional argument begin with '-' is a legal argument. Below is a quote from its manual. "If you have positional arguments that must begin with - and don?t look like negative numbers, you can insert the pseudo-argument '--' which tells parse_args() that everything after that is a positional argument" > and then wondering why it isn't doing what you expect it to magically know." I don't expect magic, I expect the consistency of a parser. -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!jfetVCL_U0v2RwXvkQWkim9VOYbZSJsWxRjWVlhpCDNHLoBWGyPJKGC_vh1Hwkrm3AzcX2KNEOUFKNNBx0tG$ From list1 at tompassin.net Wed Jan 25 13:21:23 2023 From: list1 at tompassin.net (Thomas Passin) Date: Wed, 25 Jan 2023 13:21:23 -0500 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: <4gcAL.2393259$JNZ4.383220@fx12.ams4> References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> Message-ID: On 1/25/2023 10:53 AM, Dino wrote: > > Hello, I could use something like Apache ab in Python ( > https://httpd.apache.org/docs/2.4/programs/ab.html ). > > The reason why ab doesn't quite cut it for me is that I need to define a > pool of HTTP requests and I want the tool to run those (as opposed to > running the same request over and over again) > > Does such a marvel exist? > > Thinking about it, it doesn't necessarily need to be Python, but I guess > I would have a chance to tweak things if it was. I actually have a Python program that does exactly this. The intention was to simulate a large number of independent users hitting a particular web site as rapidly as possible, so see what the typical throughput is. The program is somewhat specialized in the nature of the requests, but the method is easy enough to implement. The requests are composed from a pool of 300 pieces, and for each request, four pieces are selected randomly with replacement and combined to form the entire request. The idea here is to try to minimize caching, so as to better assess the throughput for random queries. The program runs a configurable number of threads. Each thread tries to maintain an average query rate, but you have to throttle them to prevent an exponential buildup of the request queue. If you run the program, the server machine (usually the same as the querying machine) is likely to get very hot - it's can be quite a stress test - and you want to monitor the CPU temperatures just in case. I can't share the actual code for copyright reasons, but the above description should be helpful. The actual code is not very complicated nor hard to develop. I also have a version that uses async techniques instead of threads. To give a feel for using a program like this, I think I can show the '__main__' bit: if __name__ == '__main__': handleCmdLine() # Warm up [redacted] in case it is not ready to get flooded with queries for n in range(WARMUP_REPS): HTTPClient(HOST, setpath(), True) asyncore.loop() # Warmup done, reset hit counter reps = 0 # And away we go ... for n in range(NUMCLIENTS): HTTPClient(HOST, setpath()) start = clock() asyncore.loop(timeout=50) now = clock() sys.stderr.write('\n') reps_per_sec = reps / (now - start) print ('%0.1f hits/sec' % reps_per_sec) There are also some polling-based systems available that do a similar job. I don't remember the name of the one I tried a few years ago. It ran a server to run the queries and reported the results via your browser. From antoon.pardon at vub.be Wed Jan 25 13:26:47 2023 From: antoon.pardon at vub.be (Antoon Pardon) Date: Wed, 25 Jan 2023 19:26:47 +0100 Subject: Evaluation of variable as f-string In-Reply-To: References: Message-ID: Op 23/01/2023 om 17:24 schreef Johannes Bauer: > Hi there, > > is there an easy way to evaluate a string stored in a variable as if > it were an f-string at runtime? > > I.e., what I want is to be able to do this: > > x = { "y": "z" } > print(f"-> {x['y']}") > > This prints "-> z", as expected. But consider: > > x = { "y": "z" } > s = "-> {x['y']}" > print(s.format(x = x)) > Traceback (most recent call last): > ? File "", line 1, in > KeyError: "'y'" > > Even though > > s = "-> {x}" > print(s.format(x = x)) > > Prints the expected "-> {'y': 'z'}". > I am probably missing something but is there a reason why the following wouldn't do what you want: x = { "y": "z" } s = "-> {target}" print(s.format(target = x['y'])) From orzodk at fastmail.com Wed Jan 25 13:33:38 2023 From: orzodk at fastmail.com (orzodk) Date: Wed, 25 Jan 2023 11:33:38 -0700 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: <4gcAL.2393259$JNZ4.383220@fx12.ams4> (dino@no.spam.ar's message of "Wed, 25 Jan 2023 10:53:34 -0500") References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> Message-ID: Dino writes: > Hello, I could use something like Apache ab in Python ( > https://httpd.apache.org/docs/2.4/programs/ab.html ). > > The reason why ab doesn't quite cut it for me is that I need to define > a pool of HTTP requests and I want the tool to run those (as opposed > to running the same request over and over again) > > Does such a marvel exist? > > Thinking about it, it doesn't necessarily need to be Python, but I > guess I would have a chance to tweak things if it was. > > Thanks > > Dino I have used locust with success in the past. https://locust.io From wlfraed at ix.netcom.com Wed Jan 25 13:42:03 2023 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Wed, 25 Jan 2023 13:42:03 -0500 Subject: bool and int References: <310C22AA-D2C5-48E8-B1AB-933D8EACE52C@yahoo.com> Message-ID: <3qt2th9ii4n0kketfqv6l0chan834galib@4ax.com> On Tue, 24 Jan 2023 20:16:31 -0500, Mike Baskin declaimed the following: >Will all of you please stop sending me emails > Nobody is sending "you" emails deliberately (unless they have a poorly [to me] set up client that sends to the list AND the person who wrote the message they replied to) -- they are sending them to a Python mailing list. The mailing list only sends them to /subscribed/ users. Read the headers in the message for instructions... List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From avi.e.gross at gmail.com Wed Jan 25 14:21:42 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Wed, 25 Jan 2023 14:21:42 -0500 Subject: Languages working together In-Reply-To: <22c09ec5-5f73-49e5-80b0-71b23214312d@tompassin.net> References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> <22c09ec5-5f73-49e5-80b0-71b23214312d@tompassin.net> Message-ID: <002b01d930f2$42036d40$c60a47c0$@gmail.com> Thomas, I changed the subject line as we are not talking about bool and int anymore. For me, there are several sides to JAVA that go beyond the "language" to the JVM, or Java Virtual Machine. What you are describing is an example of interoperability you can get if your language also is built on using the JVM. Jython is an example as you mention and thus you can access functionality written in another language as they sort of meet at the JVM level. I have studied several others like Kotlin and Scala. Every language has to have some reason to exist, sometimes not a great reason, and it is nice when you can combine the strengths of some language you like to use for some purposes, with another language that already has lots of code and resources you may find useful. They need not necessarily share an underlying sub-language like the JVM, though. For example, I have written combination programs that in a sense operate back and forth on data using Python and R and let me use things I handle well in each, but of course only up to some point. But as you pointed out a while ago, it may not always be necessary as you showed a ggplot() implementation within python, so if all I wanted was to finish off a program using a graphics system I am fluent in, I could have possibly remained just within Python. Sharing can come at many levels. I am fairly certain many very different languages may still share libraries written ages ago and written in C or FORTRAN and thus external to other languages and just need some way to interface to them. Avi -----Original Message----- From: Python-list On Behalf Of Thomas Passin Sent: Wednesday, January 25, 2023 8:47 AM To: python-list at python.org Subject: Re: bool and int On 1/25/2023 6:53 AM, 2QdxY4RzWzUUiLuE at potatochowder.com wrote: > They used Java at my last job (as in, the last job I had before I > retired), and it was absolutely awful, for any number of reasons, the > gymnastics (on many levels) required to support "primitive types" > being one of them. In my one serious java program, a Tomcat application, I use a small amount of Java for the servlets and startup filters, and Jython for the heart of the computing. Thank goodness for Jython! -- https://mail.python.org/mailman/listinfo/python-list From list1 at tompassin.net Wed Jan 25 14:38:38 2023 From: list1 at tompassin.net (Thomas Passin) Date: Wed, 25 Jan 2023 14:38:38 -0500 Subject: Evaluation of variable as f-string In-Reply-To: References: Message-ID: <5bb2546d-8c16-262d-9336-5b41cbf992b5@tompassin.net> On 1/25/2023 1:26 PM, Antoon Pardon wrote: > Op 23/01/2023 om 17:24 schreef Johannes Bauer: >> Hi there, >> >> is there an easy way to evaluate a string stored in a variable as if >> it were an f-string at runtime? >> >> I.e., what I want is to be able to do this: >> >> x = { "y": "z" } >> print(f"-> {x['y']}") >> >> This prints "-> z", as expected. But consider: >> >> x = { "y": "z" } >> s = "-> {x['y']}" >> print(s.format(x = x)) >> Traceback (most recent call last): >> ? File "", line 1, in >> KeyError: "'y'" >> >> Even though >> >> s = "-> {x}" >> print(s.format(x = x)) >> >> Prints the expected "-> {'y': 'z'}". >> > I am probably missing something but is there a reason why the following > wouldn't do what you want: > > x = { "y": "z" } > s = "-> {target}" > print(s.format(target = x['y'])) Stack overflow to the rescue: Search phrase: "python evaluate string as fstring" https://stackoverflow.com/questions/47339121/how-do-i-convert-a-string-into-an-f-string def effify(non_f_str: str): return eval(f'f"""{non_f_str}"""') print(effify(s)) # prints as expected: "-> z" From list1 at tompassin.net Wed Jan 25 14:42:42 2023 From: list1 at tompassin.net (Thomas Passin) Date: Wed, 25 Jan 2023 14:42:42 -0500 Subject: Languages working together In-Reply-To: <002b01d930f2$42036d40$c60a47c0$@gmail.com> References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> <22c09ec5-5f73-49e5-80b0-71b23214312d@tompassin.net> <002b01d930f2$42036d40$c60a47c0$@gmail.com> Message-ID: <8310dbc2-b940-4f03-d17b-d05a932a18a9@tompassin.net> On 1/25/2023 2:21 PM, avi.e.gross at gmail.com wrote: > [...] Sharing can come at many levels. I am fairly certain many > very different languages may still share libraries written ages ago and > written in C or FORTRAN and thus external to other languages and just need > some way to interface to them. For example, good old FORTRAN LINPACK has been wrapped for many languages and computing frameworks. From dino at no.spam.ar Wed Jan 25 15:27:51 2023 From: dino at no.spam.ar (Dino) Date: Wed, 25 Jan 2023 15:27:51 -0500 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> Message-ID: <4hgAL.2067293$nwq3.1326233@fx13.ams4> On 1/25/2023 1:33 PM, orzodk wrote: > > > I have used locust with success in the past. > > https://locust.io First impression, exactly what I need. Thank you Orzo! From dino at no.spam.ar Wed Jan 25 15:29:41 2023 From: dino at no.spam.ar (Dino) Date: Wed, 25 Jan 2023 15:29:41 -0500 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> Message-ID: On 1/25/2023 1:21 PM, Thomas Passin wrote: > > > I actually have a Python program that does exactly this. Thank you, Thomas. I'll check out Locust, mentioned by Orzodk, as it looks like a mature library that appears to do exactly what I was hoping. From dino at no.spam.ar Wed Jan 25 15:32:50 2023 From: dino at no.spam.ar (Dino) Date: Wed, 25 Jan 2023 15:32:50 -0500 Subject: bool and int In-Reply-To: References: Message-ID: On 1/23/2023 11:22 PM, Dino wrote: > >>> b = True > >>> isinstance(b,bool) > True > >>> isinstance(b,int) > True > >>> ok, I read everything you guys wrote. Everyone's got their reasons obviously, but allow me to observe that there's also something called "principle of least surprise". In my case, it took me some time to figure out where a nasty bug was hidden. Letting a bool be a int is quite a gotcha, no matter how hard the benevolent dictator tries to convince me otherwise! From jules.tillinghast at stu.northampton-k12.us Wed Jan 25 15:55:31 2023 From: jules.tillinghast at stu.northampton-k12.us (Jules Tillinghast) Date: Wed, 25 Jan 2023 15:55:31 -0500 Subject: Android APK Message-ID: Is there a good python library for converting python3 to android APK -- Please consider the environment before printing this email.Remember when writing or responding to email, the Massachusetts Secretary of State has determined that e-mail is a public record.?All electronic messages sent from the Northampton Public Schools are?archived in conformance with Massachusetts and Federal Public Records?law. From list1 at tompassin.net Wed Jan 25 16:30:56 2023 From: list1 at tompassin.net (Thomas Passin) Date: Wed, 25 Jan 2023 16:30:56 -0500 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> Message-ID: <33eaf8a0-02e2-ae19-30ba-c60815142930@tompassin.net> On 1/25/2023 3:29 PM, Dino wrote: > On 1/25/2023 1:21 PM, Thomas Passin wrote: >> >> >> I actually have a Python program that does exactly this. > > Thank you, Thomas. I'll check out Locust, mentioned by Orzodk, as it > looks like a mature library that appears to do exactly what I was hoping. Great! Don't forget what I said about potential overheating if you hit the server with as many requests as it can handle. From rosuav at gmail.com Wed Jan 25 17:42:48 2023 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 26 Jan 2023 09:42:48 +1100 Subject: bool and int In-Reply-To: References: Message-ID: On Thu, 26 Jan 2023 at 08:19, Dino wrote: > > On 1/23/2023 11:22 PM, Dino wrote: > > >>> b = True > > >>> isinstance(b,bool) > > True > > >>> isinstance(b,int) > > True > > >>> > > ok, I read everything you guys wrote. Everyone's got their reasons > obviously, but allow me to observe that there's also something called > "principle of least surprise". > > In my case, it took me some time to figure out where a nasty bug was > hidden. Letting a bool be a int is quite a gotcha, no matter how hard > the benevolent dictator tries to convince me otherwise! > Try this (or its equivalent) in as many languages as possible: x = (1 > 2) x == 0 You'll find that x (which has effectively been set to False, or its equivalent in any language) will be equal to zero in a very large number of languages. Thus, to an experienced programmer, it would actually be quite the opposite: having it NOT be a number would be the surprising thing! ChrisA From bouncingcats at gmail.com Wed Jan 25 18:10:23 2023 From: bouncingcats at gmail.com (David) Date: Thu, 26 Jan 2023 10:10:23 +1100 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <31e34753-97a2-4ccb-bbd4-03922357ce78n@googlegroups.com> References: <741f9e53-ecdd-4c49-ae1c-4697661e21e1n@googlegroups.com> <31e34753-97a2-4ccb-bbd4-03922357ce78n@googlegroups.com> Message-ID: On Thu, 26 Jan 2023 at 04:24, Jach Feng wrote: > Chris Angelico ? 2023?1?25? ?????1:16:25 [UTC+8] ?????? > > On Wed, 25 Jan 2023 at 14:42, Jach Feng wrote: > > You're still not really using argparse as an argument parser. Why not > > just do your own -h checking? Stop trying to use argparse for what > > it's not designed for, and then wondering why it isn't doing what you > > expect it to magically know. > I just don't get what you mean? Hi, I am writing because there seems to be a communication problem here, not a programming problem. The thread is long and seems to be making no progress. You seem like a traveller who is lost and keeps looking at the map, but they have the wrong map for where they are. You seem to be not understanding the purpose of argparse. In this thread, you have not mentioned any reason that requires the use of argparse. argparse is intended to help when your program accepts >>> more than one kind of argument <<< and then uses those different kinds of argument >>> for different purposes <<<. The programmer can handle that situation without argparse. But argparse makes it easier. In this thread, you have not described your program as accepting different kinds of arguments. Therefore the people replying to you are telling you that you do not need it. You seem to be not listening to that, and you have not explained why, you seem to just keep ignoring what people are telling you. If you disagree with this, then a way forward is for you to think harder about what benefits you are getting in your program by using argparse, and >>> explain those benefits clearly to your readers here <<<. An alternative approach would be for you to remove argparse from your program, and try to write your program without it. And then ask here about any difficulties that you have with doing that, and I imagine people will be glad to help you. All of the replies you have received so far are trying to help you, even if you don't feel that. Good luck. From hjp-python at hjp.at Wed Jan 25 19:38:27 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Thu, 26 Jan 2023 01:38:27 +0100 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: <33eaf8a0-02e2-ae19-30ba-c60815142930@tompassin.net> References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> <33eaf8a0-02e2-ae19-30ba-c60815142930@tompassin.net> Message-ID: <20230126003827.bepnprceb26a4jb4@hjp.at> On 2023-01-25 16:30:56 -0500, Thomas Passin wrote: > Great! Don't forget what I said about potential overheating if you > hit the server with as many requests as it can handle. Frankly, if you can overheat a server by hitting it with HTTP requests, get better hardware and/or put it into a place with better airflow. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From skip.montanaro at gmail.com Wed Jan 25 19:48:54 2023 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Wed, 25 Jan 2023 18:48:54 -0600 Subject: Android APK In-Reply-To: References: Message-ID: > > Is there a good python library for converting python3 to android APK > I'm not an app type of person, so don't watch that space. I've heard of Kivy: https://kivy.org/ I also something called python-for-android on PyPI with a recent release: https://pypi.org/project/python-for-android/ I'd be interested to see what else turns up. Skip > From list1 at tompassin.net Wed Jan 25 20:04:32 2023 From: list1 at tompassin.net (Thomas Passin) Date: Wed, 25 Jan 2023 20:04:32 -0500 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: <20230126003827.bepnprceb26a4jb4@hjp.at> References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> <33eaf8a0-02e2-ae19-30ba-c60815142930@tompassin.net> <20230126003827.bepnprceb26a4jb4@hjp.at> Message-ID: <44ce3f73-45eb-7148-3f1f-91ddaf624f73@tompassin.net> On 1/25/2023 7:38 PM, Peter J. Holzer wrote: > On 2023-01-25 16:30:56 -0500, Thomas Passin wrote: >> Great! Don't forget what I said about potential overheating if you >> hit the server with as many requests as it can handle. > > Frankly, if you can overheat a server by hitting it with HTTP requests, > get better hardware and/or put it into a place with better airflow. > Frankly, if you have a server-grade machine then well and good but if you are running a nice quiet consumer grade laptop - my development machine - you need to be careful. We don't know what hardware the OP is using. And it's not servicing the requests per se that's the issue, it's the heavy computing load that has to be done for each request. The CPU is generally pegged at 100% for most or all of the test. From rosuav at gmail.com Wed Jan 25 20:36:16 2023 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 26 Jan 2023 12:36:16 +1100 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: <44ce3f73-45eb-7148-3f1f-91ddaf624f73@tompassin.net> References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> <33eaf8a0-02e2-ae19-30ba-c60815142930@tompassin.net> <20230126003827.bepnprceb26a4jb4@hjp.at> <44ce3f73-45eb-7148-3f1f-91ddaf624f73@tompassin.net> Message-ID: On Thu, 26 Jan 2023 at 12:06, Thomas Passin wrote: > > On 1/25/2023 7:38 PM, Peter J. Holzer wrote: > > On 2023-01-25 16:30:56 -0500, Thomas Passin wrote: > >> Great! Don't forget what I said about potential overheating if you > >> hit the server with as many requests as it can handle. > > > > Frankly, if you can overheat a server by hitting it with HTTP requests, > > get better hardware and/or put it into a place with better airflow. > > > > Frankly, if you have a server-grade machine then well and good but if > you are running a nice quiet consumer grade laptop - my development > machine - you need to be careful. We don't know what hardware the OP is > using. And it's not servicing the requests per se that's the issue, > it's the heavy computing load that has to be done for each request. The > CPU is generally pegged at 100% for most or all of the test. If you have to worry about thermals because of CPU load, then worry about thermals because of CPU load. The HTTP request testing is completely separate. Load testing means putting a system under load. I'm not sure why you'd be concerned about one specific possible consequence, rather than, I dunno, just put the system under load and see how it performs? ChrisA From avi.e.gross at gmail.com Wed Jan 25 21:14:33 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Wed, 25 Jan 2023 21:14:33 -0500 Subject: bool and int In-Reply-To: References: Message-ID: <001901d9312b$ef077b60$cd167220$@gmail.com> Dino, There is no such things as a "principle of least surprise" or if you insist there is, I can nominate many more such "rules" such as "the principle of get out of my way and let me do what I want!" Computer languages with too many rules are sometimes next to unusable in practical situations. I am neither defending or attacking choices Python or other languages have made. I merely observe and agree to use languages carefully and as documented. But consider that the history of computer science when it came to counting numbers depends on bits. Pretty much everything is in base two. When space was a major concern, or efficiency, people often used smaller units when possible. If you look at something like how one stored files on UNIX, they set aside a group of bits of some size as part of some larger entity and each bit represented a BOOLEAN concept of 1 and 0 to mean is that bit set to some sort of ON or off. Three of those bits controlled whether the owner of a file had permission to read, write or execute the file. Another three governed the same idea for everyone listed as being in the same group and another three listed permissions for "other". The whole thing took up more room in something larger where perhaps 14 bytes (of 8 bits each) were set aside to hold a filename and other parts held things like perhaps a link count. At some point they noticed the chink had a few unused bits and invented a purpose for them and patented the setuid bit concept and the setgid bit. So back to integers. You can use 16 bits in a signed or unsigned manner and cover numbers between a low and a high. You can use 64 bits or 128 bits or whatever makes you happy and it remains an integer albeit trying to do arithmetic between them can result in an overflow situation and the hardware often does not trivially support it. It turned out that often there was a use for 8 bits even though they hold only 256 possible integers and before the alphabets grew to need more room, that was enough space for an ASCII or EBCDIC character. Sometimes you did things at the nibble level of 4 bits or a hex digit. Booleans were even simpler at 1 bit. But all SIZES can be used to hold some range of integers. Modern python has made some extensions that let you store integers of unlimited size but underneath it all, the operations are still dealing with the same ideas as shorter versions including ones that hold more limited numbers down to binary/Boolean. So if you want to save space in memory or on a hard disk or sending across the internet, you well may want the ability to set aside a "char" to hold smallish numbers or treat a Boolean value as a "1" and so on. If you know what you are doing, it is not much of a surprise. On many systems, there may well be some registers that load 32 or 64 bits and then you use them for something like addition. If I supply a 16-bit or 8-bit or even 1-bit quantity, the software generally ends up putting it into the standard size register and pads unused areas with zeroes. No big deal. If the number is too large to fit, as with any larger integer in python, then the software does whatever it needs to such as calculating how many chunks need adding and doing the addition in the same registers in smaller chunks while managing any carryover and finally assembling the possibly longer result. Floating point is a slightly different animal than the others that arguably are all of a similar if not identical type. But even there, your storage has two components which are actually both seen as integers of sorts. 6.02 times ten to the 23rd might also be written as 602 times ten to the 21st and now you have two integers you can use in calculations that registers do on integers and recombine into a new floating point format. Or, of course, you might have hardware that does the work. If you move on to complex variables, they tend to be a sort of named tuple containing a real and imaginary part with special rules on how to add and do other mathematical operations. Underneath it all, they are often implemented as a pair of linked floating point structures. And in one sense, all real numbers represented in a floating point variable are a strict subset of complex numbers with imaginary part being zero. All integers are in a sense floating point numbers with fractional part (meaning beyond the decimal point) being all zeroes. Similarly, all smaller integral types such as bytes are Booleans are a limited subset. So some of us who know a bit about CS and not just this language, are less often surprised by a choice of implementations. We often do not mix things but when we do, we know what we are doing much of the time and appreciate how easily it can be done. Consider what happens if you want to circularly permute alphabetic characters of the English Alphabet by 13, sometimes called rot13. There are quite a few ways to do it but one of them depends on the fact that the numerical value in ASCII of the letter 'A' is one less than of the next letter, 'B' and so on to 'Z'. They are contiguous but note upper and lower cases letters are not back to back. So a piece of code that says that if a character is between the numerical representation of 'A' and halfway to the end, then add 13, and if it is higher, then add 13 and subtract 26 (or obviously just subtract 13) to slide it circularly back, is a valid way to do the permutation. But it involves adding what may seem to be a 32-bit integer containing a number like 13 or 26 to an 8-bit character resulting in an 8-bit character. Should that be illegal? Well, in a sense it is illegal and you should be forced to create something of type char that holds whatever 13 is, which probably is a control-character and then either use operations that add or subtract 8-bit items, OR you may widen the character to 32 bits and do the math with a 32-bit representation of 13/26 then narrow the result back into eight bits. But lots of systems know how to automate this for you or even do it more efficiently than you might have done on your own and do it differently on different computer architectures. One goal of programming is to make life as easy as possible for a programmer to be productive or make fewer errors and so on. Sometimes not being surprised is a valid goal. But a language like Python may not lean the way you think. I once bashed my head at languages like PASCAL and even C as it often took some work to do things like process a list of heterogenous contents including perhaps some I may have no idea about at compile time. You sometimes needed to convince the compiler to loosen up on rules such as by declaring a union of several types. Python often does not care what anything is and just passes it on. You, as the writer of a function may have to check things like whether what you were passed is hashable or is of a numeric type. You may simply take any argument(s) and coerce them into a new list object to guarantee that whatever it is can now handle what lists supply. This means you are often free to use the function with arguments that are tuples or iterators and not just lists. Yes, someone using the function may yet surprise you and you may want to write defensive code that checks up-front and stops if called in an unexpected way. But you can write the main function fairly quickly while assuming no surprises and bullet-proof it later once it seems to work for what you want. The alternative is to keep fighting with a compiler (or a static checker like mpy) before anything gets done. The surprise there is when they fire you from your job for not seeming to do much for many months. Having said that, I have had many surprises in many languages such as some that help you so much that they handle 3 + "two" for you and return 5 or maybe "five" rather than suggest you may have name an error and meant 3 + int(english_number_to_string("two")) or perhaps "3" + "two" ... I am surprised how this message got this long! Aaargh! -----Original Message----- From: Python-list On Behalf Of Dino Sent: Wednesday, January 25, 2023 3:33 PM To: python-list at python.org Subject: Re: bool and int On 1/23/2023 11:22 PM, Dino wrote: > >>> b = True > >>> isinstance(b,bool) > True > >>> isinstance(b,int) > True > >>> ok, I read everything you guys wrote. Everyone's got their reasons obviously, but allow me to observe that there's also something called "principle of least surprise". In my case, it took me some time to figure out where a nasty bug was hidden. Letting a bool be a int is quite a gotcha, no matter how hard the benevolent dictator tries to convince me otherwise! -- https://mail.python.org/mailman/listinfo/python-list From avi.e.gross at gmail.com Wed Jan 25 21:20:55 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Wed, 25 Jan 2023 21:20:55 -0500 Subject: bool and int In-Reply-To: References: Message-ID: <001e01d9312c$d232ffe0$7698ffa0$@gmail.com> Chris, We generally agree albeit I have a question in python with the concept of being truthy that results in either a Boolean value that boils down to 0 and 1 but in some cases may boil down to the last evaluated argument which remains in a form that may not be either a Boolean or an integer. I think this can happen with something like a short-circuit OR. Assuming I vaguely remember something real and actual, you can end up with a variable holding either a Boolean or almost anything and should not use it in an arithmetical capacity directly but perhaps only use bool(thing) ... Avi -----Original Message----- From: Python-list On Behalf Of Chris Angelico Sent: Wednesday, January 25, 2023 5:43 PM To: python-list at python.org Subject: Re: bool and int On Thu, 26 Jan 2023 at 08:19, Dino wrote: > > On 1/23/2023 11:22 PM, Dino wrote: > > >>> b = True > > >>> isinstance(b,bool) > > True > > >>> isinstance(b,int) > > True > > >>> > > ok, I read everything you guys wrote. Everyone's got their reasons > obviously, but allow me to observe that there's also something called > "principle of least surprise". > > In my case, it took me some time to figure out where a nasty bug was > hidden. Letting a bool be a int is quite a gotcha, no matter how hard > the benevolent dictator tries to convince me otherwise! > Try this (or its equivalent) in as many languages as possible: x = (1 > 2) x == 0 You'll find that x (which has effectively been set to False, or its equivalent in any language) will be equal to zero in a very large number of languages. Thus, to an experienced programmer, it would actually be quite the opposite: having it NOT be a number would be the surprising thing! ChrisA -- https://mail.python.org/mailman/listinfo/python-list From jfong at ms4.hinet.net Wed Jan 25 21:50:26 2023 From: jfong at ms4.hinet.net (Jach Feng) Date: Wed, 25 Jan 2023 18:50:26 -0800 (PST) Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: Jach Feng ? 2023?1?22? ?????11:11:22 [UTC+8] ?????? > Fail on command line, > > e:\Works\Python>py infix2postfix.py "-4^2+5.3*abs(-2-1)/2" > usage: infix2postfix.py [-h] [infix] > infix2postfix.py: error: unrecognized arguments: -4^2+5.3*abs(-2-1)/2 > > Also fail in REPL, > > e:\Works\Python>py > Python 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:08:11) [MSC v.1928 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import argparse > >>> parser = argparse.ArgumentParser(description='Convert infix notation to postfix') > >>> parser.parse_args("-4^2+5.3*abs(-2-1)/2") > usage: [-h] > : error: unrecognized arguments: - 4 ^ 2 + 5 . 3 * a b s ( - 2 - 1 ) / 2 > > Just can't figure out where is wrong!? > > --Jach I appreciate every comment/suggestion in this thread even sometimes I may not fully understand what they mean. To me, argparse has been just a tool which I can use in a CLI app. I use it as long as the situation allows, even it's as simple as only one positional argument is needed. Now I understand some oppose this idea and saying that you shouldn't use a kitchen knife to cut a cake:-) From python at bladeshadow.org Wed Jan 25 22:06:12 2023 From: python at bladeshadow.org (Python) Date: Wed, 25 Jan 2023 21:06:12 -0600 Subject: bool and int In-Reply-To: References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> <002901d9305e$395f8280$ac1e8780$@gmail.com> Message-ID: <20230126030612.GA4979@bladeshadow.org> On Wed, Jan 25, 2023 at 01:01:24PM +1100, Chris Angelico wrote: > On Wed, 25 Jan 2023 at 12:43, wrote: > > Python has a different philosophy than some other languages with strong > > typing. In some of those, you would not be allowed to add or multiply at > > random but would need to convert parts of your calculation to all be the > > same, such as a 32-bit integer. You could still do things like I mention > > above but only after consciously mapping your Boolean to an actual zero or > > one of the kind wanted. > > Python is strongly dynamically typed. You may be thinking of "static > typing" rather than "strong typing" here, You often insist on this but frankly it does not jibe with the definitions of "strongly typed language" that I was taught or that I still see used commonly, including in literature and on sites that aim to teach people about computer science, which basically amount to: 1. A language whose variables are defined by type, and can only hold that type, typically but not necessarily compiled. 2. A language which strongly enforces restrictions on mixing or operating on, and/or implicitly converting different data types, with the implication that the structure of types is well-defined and rigid. Python conforms to neither--its VARIABLES are normally untyped (though the object data they hold obviously is). Object instances can have new fields added to them on the fly, willy-nilly, and Python allows for any object which has an interface--or rather only the portion of interface you care about in the moment--like the one it expects, to be used in a given context, i.e. duck typing. Useful properties (when used carefully!) but not intuitively consistent with the idea of "strong typing" and potentially dangerous if care is not taken. When YOU say that Python is strongly typed, you're using some other definition--one that is perhaps technically correct, but seemingly quite a lot of people in the field--including active students, college professors, and seasoned professionsals--are unaware of... The above usages are common and widespread, widely accepted--which is inherently what makes word usages correct--and you very obviously know full well what people mean when they use them; so "correcting" people who use them seems rather unhelpful (certainly without detailing what you think it means), inappropriate, and arguably just simply wrong. It seems to serve no purpose other than to make communication harder, and possibly to irritate people. I would encourage you to consider ceasing the practice, so as to not needlessly detract from the otherwise usually good info you routinely provide... And FWIW if you want some references, a google search will return voluminous examples of people using the term as I described--from acadamia to business--but here are just a few: https://www.cs.cornell.edu/courses/cs1130/2012sp/1130selfpaced/module1/module1part4/strongtyping.html https://courses.yarrahills.vic.edu.au/moodle/mod/book/view.php?id=18778&chapterid=28 https://www.oreilly.com/library/view/mastering-c-and/9781785884375/ch02s02.html https://www.postgresql.org/docs/current/typeconv-overview.html https://www.sciencedirect.com/topics/computer-science/strongly-typed-language https://www.techtarget.com/whatis/definition/strongly-typed From rosuav at gmail.com Wed Jan 25 22:52:17 2023 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 26 Jan 2023 14:52:17 +1100 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: On Thu, 26 Jan 2023 at 14:13, Jach Feng wrote: > Now I understand some oppose this idea and saying that you shouldn't use a kitchen knife to cut a cake:-) You shouldn't use a chainsaw to cut a cake, and then ask us why cake-cutting is so noisy. ChrisA From mccardle.john at gmail.com Wed Jan 25 22:31:29 2023 From: mccardle.john at gmail.com (John McCardle) Date: Wed, 25 Jan 2023 22:31:29 -0500 Subject: [Help Request] Embedding Python in a CPP Application Responsibly & Functionally Message-ID: <943ae183-6b26-4430-50d4-71fcbdb5c13a@gmail.com> Greetings, I'm working on embedding a Python interpreter into a C++ application. My embedding example program is here, largely taken from Python docs: https://gist.github.com/jmccardle/f3f19d3753ae023aa52b927f0d181c43 I'm simply not interested in writing in Lua, so regardless of any particular downsides like `sys` in the standard library or performance issues, I'm committed to Python itself as what I want to hack in. This is for fun. I started by compiling Python: `./configure --enable-shared --enable-optimizations` Then I can compile my example embedded program: `g++ -I Python-3.11.1/Include -I Python-3.11.1 -L Python-3.11.1 -pthread scripting_engine.cpp libpython3.11.a -o scripteng -lm -ldl -lutil` This is working not so bad! I can control what C++ functionality is exposed and I seemingly don't need anything but the Python shared object to execute. But the finer details of making this work truly correctly are eluding me. 1) To get the compiled Python to run independently, I have to hack LD_LIBRARY_PATH to get it to execute. `LD_LIBRARY_PATH=./Python-3.11.1 ./Python-3.11.1/python` . Even when trying to execute from the same directory as the binary & executable, I get an error, `/python: error while loading shared libraries: libpython3.11.so.1.0: cannot open shared object file: No such file or directory`. 2) When running the C++ program that embeds Python, I see these messages after initializing: `Could not find platform independent libraries Could not find platform dependent libraries ` This is seemingly connected to some issues regarding libraries: When I run the Python interpreter directly, I can get some of the way through the process of creating a virtual environment, but it doesn't seem to leave me with a working pip: `$ LD_LIBRARY_PATH=./Python-3.11.1 ./Python-3.11.1/python >>> import venv >>> venv.create("./venv", with_pip=True) subprocess.CalledProcessError: Command '['/home/john/Development/7DRL/cpp_embedded_python/venv/bin/python', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 127.` Meanwhile, if I try to run a script from the C++ program that includes `import venv`, I get a traceback about a platform library: `Traceback (most recent call last): ? File "engine_user.py", line 7, in ??? import venv ? File "/home/john/Development/7DRL/cpp_embedded_python/Python-3.11.1/Lib/venv/__init__.py", line 10, in ??? import subprocess ? File "/home/john/Development/7DRL/cpp_embedded_python/Python-3.11.1/Lib/subprocess.py", line 104, in ??? from _posixsubprocess import fork_exec as _fork_exec ModuleNotFoundError: No module named '_posixsubprocess' ` 3) I'm not sure I even need to be statically linking the interpreter. My desired end state is this: * Deploy a C++ program that doesn't rely on a system Python. I'm not sure if I need just the shared object / DLLs, or a Python executable in a subdirectory - I'd like to "do it the right way". * C++ program can run a script to create a virtual environment, which the embedded Python environment will use. Users can activate the venv like any other Python environment and install packages with pip. * ideally, some sort of "inside-out" runnable mode, where the API exposed by the C++ executable is available in that venv, so that I can test a script in Thonny or other IDE. I think I'd do this by providing a separate test-mode library in the venv, and when C++ executes `PyImport_AppendInittab("scriptable", &PyInit_scriptable);` then the module of the same name should be overwritten with the C++ program's functionality. I've been through the embedded programming docs a bit, and they seem quite good as a reference, but I don't know what I'm doing well enough to solve my problems using them. Thanks for reading. My ultimate goal is to expose features written in C++ for a game engine using SFML, and run .py files in a subdirectory to generate maps, control NPC dialogue and actions, etc. I'm hoping to have something usable for this year's 7DRL, which starts March 4th. I'd like to spend the time until then getting this engine working smoothly and porting it to Windows, so I can focus exclusively on "game content" for the timeboxed 7-day portion. Kind Regards, -John McCardle From avi.e.gross at gmail.com Wed Jan 25 22:59:20 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Wed, 25 Jan 2023 22:59:20 -0500 Subject: bool and int In-Reply-To: <20230126030612.GA4979@bladeshadow.org> References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> <002901d9305e$395f8280$ac1e8780$@gmail.com> <20230126030612.GA4979@bladeshadow.org> Message-ID: <00ab01d9313a$92234730$b669d590$@gmail.com> Like Chris, I appreciate precision when it matters but since I am not writing a textbook here, I often talk more informally. There are many variations on now variables or objects are treated differently in certain languages and when I said "STRONG" I simply meant a sort of opposite to "WEAK". I could have used any number of silly words like "mighty typing" and most would have been meaningless. As python at bladeshadow.org points out, from some perspectives, Python plays quite fast and lose about what a variable can refer to. It is not a bug but a proud feature of the language that you can easily do all kinds of things with a kind of polymorphism (and I do NOT want to hear how I misused that term in some technical way). I have been reading about the ways Python keeps amending the ways you can tell a program about the types a variable can hold and it gets quite amusing in some ways. First, almost anything you write has to be IGNORED at run-time. You are sort of just slowing down things and making your program text longer, albeit the byte-compiled files may toss much of that. The main purpose if of static type checkers like mpy to badger you until you come up with just the right incantation and even then, your program can have plenty of bugs and fail at run-time. There is a rich and seemingly ever-growing new set of quasi-language features for specifying more closely what a function expects as arguments and what it might return and some are potentially very useful except for the minor detail that at runtime, your suggestion that your function takes only objects that implement some protocol such as defining __lt__ is absolutely ignored. All that mpy can do is static testing of what you CLAIM you want. I am not making fun, of course, but in code I develop, I have negligible interest in using the optional typing system and checking at first. It can truly slow things down and some of the constraints can turn out to be too tight if you later find the method you used excludes lots of things people would use it on. I prefer having working code before I mark it up to be less legible but in a way that finds some possible errors, or maybe mainly forces me to change the markup to something else. And, if I really want my code to CATCH errors when running, the beginning of many functions will have to contain active tests such as checking if it is one of the types I want it to support or implements some protocol, and if not, handle the error gracefully. Such code can be way more detailed or correct than the type system or may be way worse but it can also in some sense be a self-documentation that some can read easier than trying to enforce strong typing. If I was working on a large project with many people and the organization had all kinds of protocols and rules, then sure, you follow them or leave. But at some point you may ask the dumb question of why they are using Python at all, rather than a much "safer" language designed to minimize any ability to make many errors as the program refuses to run until highly polished? -----Original Message----- From: Python-list On Behalf Of Python Sent: Wednesday, January 25, 2023 10:06 PM To: python-list at python.org Subject: Re: bool and int On Wed, Jan 25, 2023 at 01:01:24PM +1100, Chris Angelico wrote: > On Wed, 25 Jan 2023 at 12:43, wrote: > > Python has a different philosophy than some other languages with > > strong typing. In some of those, you would not be allowed to add or > > multiply at random but would need to convert parts of your > > calculation to all be the same, such as a 32-bit integer. You could > > still do things like I mention above but only after consciously > > mapping your Boolean to an actual zero or one of the kind wanted. > > Python is strongly dynamically typed. You may be thinking of "static > typing" rather than "strong typing" here, You often insist on this but frankly it does not jibe with the definitions of "strongly typed language" that I was taught or that I still see used commonly, including in literature and on sites that aim to teach people about computer science, which basically amount to: 1. A language whose variables are defined by type, and can only hold that type, typically but not necessarily compiled. 2. A language which strongly enforces restrictions on mixing or operating on, and/or implicitly converting different data types, with the implication that the structure of types is well-defined and rigid. Python conforms to neither--its VARIABLES are normally untyped (though the object data they hold obviously is). Object instances can have new fields added to them on the fly, willy-nilly, and Python allows for any object which has an interface--or rather only the portion of interface you care about in the moment--like the one it expects, to be used in a given context, i.e. duck typing. Useful properties (when used carefully!) but not intuitively consistent with the idea of "strong typing" and potentially dangerous if care is not taken. When YOU say that Python is strongly typed, you're using some other definition--one that is perhaps technically correct, but seemingly quite a lot of people in the field--including active students, college professors, and seasoned professionsals--are unaware of... The above usages are common and widespread, widely accepted--which is inherently what makes word usages correct--and you very obviously know full well what people mean when they use them; so "correcting" people who use them seems rather unhelpful (certainly without detailing what you think it means), inappropriate, and arguably just simply wrong. It seems to serve no purpose other than to make communication harder, and possibly to irritate people. I would encourage you to consider ceasing the practice, so as to not needlessly detract from the otherwise usually good info you routinely provide... And FWIW if you want some references, a google search will return voluminous examples of people using the term as I described--from acadamia to business--but here are just a few: https://www.cs.cornell.edu/courses/cs1130/2012sp/1130selfpaced/module1/modul e1part4/strongtyping.html https://courses.yarrahills.vic.edu.au/moodle/mod/book/view.php?id=18778&chap terid=28 https://www.oreilly.com/library/view/mastering-c-and/9781785884375/ch02s02.h tml https://www.postgresql.org/docs/current/typeconv-overview.html https://www.sciencedirect.com/topics/computer-science/strongly-typed-languag e https://www.techtarget.com/whatis/definition/strongly-typed -- https://mail.python.org/mailman/listinfo/python-list From list1 at tompassin.net Wed Jan 25 23:02:33 2023 From: list1 at tompassin.net (Thomas Passin) Date: Wed, 25 Jan 2023 23:02:33 -0500 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> <33eaf8a0-02e2-ae19-30ba-c60815142930@tompassin.net> <20230126003827.bepnprceb26a4jb4@hjp.at> <44ce3f73-45eb-7148-3f1f-91ddaf624f73@tompassin.net> Message-ID: <7a71a40a-19d5-3f33-a50c-4919d4c7e111@tompassin.net> On 1/25/2023 8:36 PM, Chris Angelico wrote: > On Thu, 26 Jan 2023 at 12:06, Thomas Passin wrote: >> >> On 1/25/2023 7:38 PM, Peter J. Holzer wrote: >>> On 2023-01-25 16:30:56 -0500, Thomas Passin wrote: >>>> Great! Don't forget what I said about potential overheating if you >>>> hit the server with as many requests as it can handle. >>> >>> Frankly, if you can overheat a server by hitting it with HTTP requests, >>> get better hardware and/or put it into a place with better airflow. >>> >> >> Frankly, if you have a server-grade machine then well and good but if >> you are running a nice quiet consumer grade laptop - my development >> machine - you need to be careful. We don't know what hardware the OP is >> using. And it's not servicing the requests per se that's the issue, >> it's the heavy computing load that has to be done for each request. The >> CPU is generally pegged at 100% for most or all of the test. > > If you have to worry about thermals because of CPU load, then worry > about thermals because of CPU load. The HTTP request testing is > completely separate. > > Load testing means putting a system under load. I'm not sure why you'd > be concerned about one specific possible consequence, rather than, I > dunno, just put the system under load and see how it performs? This is not that hard, folks! I needed to know the throughput of this system if it were hit with a great many queries at once, as if it were a busy help desk, for instance. The intent is not to bring the server to its knees as a load test for the server, it's to measure the maximum throughput for independent queries. It happens that each query takes a lot of processing, so the system is not IO bound, it's CPU bound. The result is a large CPU load and a large amount of heat generated. "just put the system under load and see how it performs" This is exactly what was happening, but not in the service of stressing the computer, but in finding the throughput for this particular Tomcat app with representative queries. As a byproduct, I noticed very high CPU temperatures, which probably wouldn't have occurred in a datacenter server with much better cooling. I didn't care about that except for protecting my own laptop. And this is way OT for the OP's question. From dino at no.spam.ar Wed Jan 25 23:23:17 2023 From: dino at no.spam.ar (Dino) Date: Wed, 25 Jan 2023 23:23:17 -0500 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: <4hgAL.2067293$nwq3.1326233@fx13.ams4> References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> <4hgAL.2067293$nwq3.1326233@fx13.ams4> Message-ID: On 1/25/2023 3:27 PM, Dino wrote: > On 1/25/2023 1:33 PM, orzodk wrote: >> >> I have used locust with success in the past. >> >> https://locust.io > > First impression, exactly what I need. Thank you Orzo! the more I learn about Locust and I tinker with it, the more I love it. Thanks again. From jacob.kruger.work at gmail.com Thu Jan 26 02:09:27 2023 From: jacob.kruger.work at gmail.com (jacob kruger) Date: Thu, 26 Jan 2023 09:09:27 +0200 Subject: Android APK In-Reply-To: References: Message-ID: Beeware is the only one I have tried out: https://beeware.org/ Jacob Kruger +2782 413 4791 "Resistance is futile...but, acceptance is versatile..." On 2023/01/25 22:55, Jules Tillinghast wrote: > Is there a good python library for converting python3 to android APK > From dieter at handshake.de Thu Jan 26 02:50:51 2023 From: dieter at handshake.de (Dieter Maurer) Date: Thu, 26 Jan 2023 08:50:51 +0100 Subject: [Help Request] Embedding Python in a CPP Application Responsibly & Functionally In-Reply-To: <943ae183-6b26-4430-50d4-71fcbdb5c13a@gmail.com> References: <943ae183-6b26-4430-50d4-71fcbdb5c13a@gmail.com> Message-ID: <25554.12507.902369.225551@ixdm.fritz.box> John McCardle wrote at 2023-1-25 22:31 -0500: > ... >1) To get the compiled Python to run independently, I have to hack >LD_LIBRARY_PATH to get it to execute. `LD_LIBRARY_PATH=./Python-3.11.1 >./Python-3.11.1/python` . The need to set `LD_LIBRARY_PATH` usually can be avoided via a link time option: it tells the linker to add library path information into the created shared object. Read the docs to find out which option this is (I think it was `-r` but I am not sure). >Even when trying to execute from the same >directory as the binary & executable, I get an error, `/python: error >while loading shared libraries: libpython3.11.so.1.0: cannot open shared >object file: No such file or directory`. It might be necessary, to provide the option mentioned above for all shared libraries involved in your final application. Alternatively, you could try to put the shared objects into a stadard place (searched by default). >2) When running the C++ program that embeds Python, I see these messages >after initializing: >`Could not find platform independent libraries >Could not find platform dependent libraries ` Again: either put your installation in a standard place or tell the Python generation process about your non-standard place. >This is seemingly connected to some issues regarding libraries: When I >run the Python interpreter directly, I can get some of the way through >the process of creating a virtual environment, but it doesn't seem to >leave me with a working pip: > >`$ LD_LIBRARY_PATH=./Python-3.11.1 ./Python-3.11.1/python > >>> import venv > >>> venv.create("./venv", with_pip=True) >subprocess.CalledProcessError: Command >'['/home/john/Development/7DRL/cpp_embedded_python/venv/bin/python', >'-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit >status 127.` Run the command manually and see what errors this gives. > ... >3) I'm not sure I even need to be statically linking the interpreter. There should be no need (if all you want in the embedding). From frank at chagford.com Thu Jan 26 05:12:29 2023 From: frank at chagford.com (Frank Millman) Date: Thu, 26 Jan 2023 12:12:29 +0200 Subject: asyncio questions Message-ID: <2dea614d-b55a-fadc-7358-85f82e3f8afa@chagford.com> Hi all I have written a simple HTTP server using asyncio. It works, but I don't always understand how it works, so I was pleased that Python 3.11 introduced some new high-level concepts that hide the gory details. I want to refactor my code to use these concepts, but I am not finding it easy. In simple terms my main loop looked like this - ??? loop = asyncio.get_event_loop() ??? server = loop.run_until_complete( ??????? asyncio.start_server(handle_client, host, port)) ??? loop.run_until_complete(setup_companies()) ??? session_check = asyncio.ensure_future( ??????? check_sessions())? # start background task ??? print('Press Ctrl+C to stop') ??? try: ??????? loop.run_forever() ??? except KeyboardInterrupt: ??????? print() ??? finally: ??????? session_check.cancel()? # tell session_check to stop running ??????? loop.run_until_complete(asyncio.wait([session_check])) ??????? server.close() ??????? loop.stop() Using 3.11 it now looks like this - ??? with asyncio.Runner() as runner: ??????? server = runner.run(asyncio.start_server( ??????????? handle_client, host, port) ??????? runner.run(setup_companies()) ??????? session_check = asyncio.ensure_future( ??????????? check_sessions())? # start background task ??????? print('Press Ctrl+C to stop') ??????? try: ??????????? runner.run(server.serve_forever()) ??????? except KeyboardInterrupt: ??????????? print() ??????? finally: ??????????? session_check.cancel()? # tell session_check to stop running ??????????? runner.run(asyncio.wait([session_check])) ??????????? server.close() It works, and I guess it looks a bit neater. Problem 1. The docs to 'asyncio.ensure_future' state 'See also the create_task() function which is the preferred way for creating new Tasks' If I change 'ensure_future' to 'create_task', I get ??? RuntimeError: no running event loop I don't know how to fix this. Problem 2. The docs have a section on 'Handling Keyboard Interruption' https://docs.python.org/3.11/library/asyncio-runner.html#asyncio.Runner I have not figured out how to adapt my code to use this new approach. Any suggestions appreciated. Frank Millman P.S. Might it be better to ask these questions on the Async_SIG Discussion Forum? From gweatherby at uchc.edu Thu Jan 26 05:52:06 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Thu, 26 Jan 2023 10:52:06 +0000 Subject: bool and int In-Reply-To: References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> Message-ID: I can?t help but wonder if there exists some Java forum /mailing list going on about how horrible Python is. From: Python-list on behalf of rbowman Date: Wednesday, January 25, 2023 at 12:25 PM To: python-list at python.org Subject: Re: bool and int *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** On Wed, 25 Jan 2023 06:53:44 -0500, 2QdxY4RzWzUUiLuE wrote: > They used Java at my last job (as in, the last job I had before I > retired), and it was absolutely awful, for any number of reasons, the > gymnastics (on many levels) required to support "primitive types" being > one of them. My first brush with Java was around '98 when it was first becoming popular. To familiarize myself with the AWT I decided to write a simple IDE for the AVR microcontrollers. What a disaster. The UI wasn't bad but the instructions for 8-bit processors require a lot of bit fiddling that was extraordinarily difficult in Java. Then they came out with Swing and the assumption if the app ran with glacial slowness you should get a faster machine. The company I work for has one Java app created around 2000 as a cross platform solution as people moved to Windows. Originally it ran as an applet but when that window was slammed shut it became increasingly unwieldy. For what I'm developing today I used either .NET C# or Python3. The .NET UI's on Linux aren't quite there yet but back end applications are fine. PyQt (PySide actually. If there is a way to screw up commercial licensing Qt will find it) is fine. -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!iVrdROvxcoNV-GhzezJe8fJSLSAUoPkZHaXF58tWtyogy37PB6b9DH-gINgbVLuU64V4RovArDpnC5jjiQ$ From 2QdxY4RzWzUUiLuE at potatochowder.com Thu Jan 26 06:48:09 2023 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Thu, 26 Jan 2023 06:48:09 -0500 Subject: Flamebait (was: Re: bool and int) In-Reply-To: References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> Message-ID: On 2023-01-26 at 10:52:06 +0000, "Weatherby,Gerard" wrote: > I can?t help but wonder if there exists some Java forum /mailing list > going on about how horrible Python is. Not some of them. *All* of them. Here's the summary: - Dynamic Typing causes defects and makes non-toy software projects impossible. - Python is slow. - Significant Whitespace [insert pejorative here]. - Python allows code to exist outside of methods, and methods to exist outside of classes. What's a function? - There is no backwards compatibility. Many, if not most, languages are created as improvements over others, because those others have serious flaws (whether those flaws are real or perceived). And even at that, the earliest languages were created because cores, wires, and machine languages were extremely painful. From rosuav at gmail.com Thu Jan 26 06:55:12 2023 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 26 Jan 2023 22:55:12 +1100 Subject: bool and int In-Reply-To: References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> Message-ID: On Thu, 26 Jan 2023 at 21:53, Weatherby,Gerard wrote: > > I can?t help but wonder if there exists some Java forum /mailing list going on about how horrible Python is. Try https://www.reddit.com/r/ProgrammerHumor/ for plenty of people whining about how horrible Python is. But along the way, you'll also find people whining about ChatGPT, reposting memes with minor updates, and sharing the very best (worst?) of dad jokes. ChrisA From list1 at tompassin.net Thu Jan 26 09:06:51 2023 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 26 Jan 2023 09:06:51 -0500 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> <4hgAL.2067293$nwq3.1326233@fx13.ams4> Message-ID: <7df12b26-aa2d-5c42-0183-e0f44d734763@tompassin.net> On 1/25/2023 11:23 PM, Dino wrote: > On 1/25/2023 3:27 PM, Dino wrote: >> On 1/25/2023 1:33 PM, orzodk wrote: >>> >>> I have used locust with success in the past. >>> >>> https://locust.io >> >> First impression, exactly what I need. Thank you Orzo! > > the more I learn about Locust and I tinker with it, the more I love it. > Thanks again. That's the one I was trying to remember! I think it was in in its early days when I tried it out. From avi.e.gross at gmail.com Thu Jan 26 09:40:48 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Thu, 26 Jan 2023 09:40:48 -0500 Subject: bool and int In-Reply-To: References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> Message-ID: <004101d93194$2e350dd0$8a9f2970$@gmail.com> Gerard, I am sure there is. I have been on many forums that discuss programming languages and since nothing is perfect and people differ in many ways, there is always grumbling and comparison. If we all agreed and there was only one of something, I suspect we still would complain and that is precisely why there are generally many variations as others like their own ideas better. Python remains a quite decent and useful language, warts and especially imagined warts and all. Avi -----Original Message----- From: Python-list On Behalf Of Weatherby,Gerard Sent: Thursday, January 26, 2023 5:52 AM To: python-list at python.org Subject: Re: bool and int I can't help but wonder if there exists some Java forum /mailing list going on about how horrible Python is. From: Python-list on behalf of rbowman Date: Wednesday, January 25, 2023 at 12:25 PM To: python-list at python.org Subject: Re: bool and int *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** On Wed, 25 Jan 2023 06:53:44 -0500, 2QdxY4RzWzUUiLuE wrote: > They used Java at my last job (as in, the last job I had before I > retired), and it was absolutely awful, for any number of reasons, the > gymnastics (on many levels) required to support "primitive types" > being one of them. My first brush with Java was around '98 when it was first becoming popular. To familiarize myself with the AWT I decided to write a simple IDE for the AVR microcontrollers. What a disaster. The UI wasn't bad but the instructions for 8-bit processors require a lot of bit fiddling that was extraordinarily difficult in Java. Then they came out with Swing and the assumption if the app ran with glacial slowness you should get a faster machine. The company I work for has one Java app created around 2000 as a cross platform solution as people moved to Windows. Originally it ran as an applet but when that window was slammed shut it became increasingly unwieldy. For what I'm developing today I used either .NET C# or Python3. The .NET UI's on Linux aren't quite there yet but back end applications are fine. PyQt (PySide actually. If there is a way to screw up commercial licensing Qt will find it) is fine. -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python- list__;!!Cn_UX_p3!iVrdROvxcoNV-GhzezJe8fJSLSAUoPkZHaXF58tWtyogy37PB6b9DH-gIN gbVLuU64V4RovArDpnC5jjiQ$ -- https://mail.python.org/mailman/listinfo/python-list From mats at wichmann.us Thu Jan 26 10:12:33 2023 From: mats at wichmann.us (Mats Wichmann) Date: Thu, 26 Jan 2023 08:12:33 -0700 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <31e34753-97a2-4ccb-bbd4-03922357ce78n@googlegroups.com> References: <741f9e53-ecdd-4c49-ae1c-4697661e21e1n@googlegroups.com> <31e34753-97a2-4ccb-bbd4-03922357ce78n@googlegroups.com> Message-ID: On 1/24/23 23:28, Jach Feng wrote: > Chris Angelico ? 2023?1?25? ?????1:16:25 [UTC+8] ?????? >> On Wed, 25 Jan 2023 at 14:42, Jach Feng wrote: >>> I was happy working with argparse during implement my script. To save the typing, I used a default equation for testing. >>> >>> sample = "-4^2+5.3*abs(-2-1)/2, abs(Abc)*(B+C)/D, (-3) * sqrt(1-(x1/7)*(y1/7)) * sqrt(abs((x0-4.5)/(y0-4)))" >>> parser = argparse.ArgumentParser(description='Convert infix notation to postfix') >>> parser.add_argument('infix', nargs='?', default=sample, help="....") >>> >> You're still not really using argparse as an argument parser. Why not >> just do your own -h checking? Stop trying to use argparse for what >> it's not designed for, and then wondering why it isn't doing what you >> expect it to magically know. >> >> ChrisA > I just don't get what you mean? > >> You're still not really using argparse as an argument parser. Why not just do your own -h checking? > > Is a math equation not qualified as a command line "argument"? What criteria do you use when judging the quality of an "argument"? The ancient UNIX concept, later codified in the POSIX international standard, is that if an argument begins with a '-' it specifies an option. The three standard library CLI argument parsers (getopt, optparse and argparse) all follow this convention. You're feeding it something that they will conclude is an option, but you have given argparse no information how to deal with such an option. The advice you've been given is: this is built in behavior. You don't want that behavior. Either "fool" the module somehow (quoting with leading spaces, use the special option identifier of a bare double dash indicating "end of options", etc.) - or just don't use a module that behaves in a way you don't like. From grant.b.edwards at gmail.com Thu Jan 26 11:02:11 2023 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Thu, 26 Jan 2023 08:02:11 -0800 (PST) Subject: HTTP server benchmarking/load testing in Python References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> <33eaf8a0-02e2-ae19-30ba-c60815142930@tompassin.net> <20230126003827.bepnprceb26a4jb4@hjp.at> <44ce3f73-45eb-7148-3f1f-91ddaf624f73@tompassin.net> Message-ID: <63d2a403.050a0220.21432.10c7@mx.google.com> On 2023-01-26, Thomas Passin wrote: > On 1/25/2023 7:38 PM, Peter J. Holzer wrote: >> On 2023-01-25 16:30:56 -0500, Thomas Passin wrote: >>> Great! Don't forget what I said about potential overheating if you >>> hit the server with as many requests as it can handle. >> >> Frankly, if you can overheat a server by hitting it with HTTP requests, >> get better hardware and/or put it into a place with better airflow. >> > > Frankly, if you have a server-grade machine then well and good but if > you are running a nice quiet consumer grade laptop - my development > machine - you need to be careful. A properly designed laptop with a non-broken OS will not overheat regardless of the computing load you throw at it. The fan might get annoying loud, but if it overheats either your hardware or OS needs to be fixed. -- Grant From grant.b.edwards at gmail.com Thu Jan 26 11:06:57 2023 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Thu, 26 Jan 2023 08:06:57 -0800 (PST) Subject: asyncio questions References: <2dea614d-b55a-fadc-7358-85f82e3f8afa@chagford.com> Message-ID: <63d2a521.920a0220.1d6fa.0ebb@mx.google.com> On 2023-01-26, Frank Millman wrote: > I have written a simple HTTP server using asyncio. It works, but I don't > always understand how it works, I thought that was the rule with asyncio. ;) From bschollnick at schollnick.net Thu Jan 26 11:13:32 2023 From: bschollnick at schollnick.net (Benjamin Schollnick) Date: Thu, 26 Jan 2023 11:13:32 -0500 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: <63d2a403.050a0220.21432.10c7@mx.google.com> References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> <33eaf8a0-02e2-ae19-30ba-c60815142930@tompassin.net> <20230126003827.bepnprceb26a4jb4@hjp.at> <44ce3f73-45eb-7148-3f1f-91ddaf624f73@tompassin.net> <63d2a403.050a0220.21432.10c7@mx.google.com> Message-ID: <8A5E292A-56AC-4203-AA97-09748059866E@schollnick.net> > On Jan 26, 2023, at 11:02 AM, Grant Edwards wrote: > > On 2023-01-26, Thomas Passin wrote: >> On 1/25/2023 7:38 PM, Peter J. Holzer wrote: >>> On 2023-01-25 16:30:56 -0500, Thomas Passin wrote: >>>> Great! Don't forget what I said about potential overheating if you >>>> hit the server with as many requests as it can handle. >>> >>> Frankly, if you can overheat a server by hitting it with HTTP requests, >>> get better hardware and/or put it into a place with better airflow. >>> >> >> Frankly, if you have a server-grade machine then well and good but if >> you are running a nice quiet consumer grade laptop - my development >> machine - you need to be careful. > > A properly designed laptop with a non-broken OS will not overheat > regardless of the computing load you throw at it. The fan might get > annoying loud, but if it overheats either your hardware or OS needs > to be fixed. Exactly. But what he might be thinking about is Thermal Throttling, which I keep seeing people attribute to overheating?. Overheating is not thermal throttling, it?s the OS and CPU protecting themselves from overheating. Usually because the manufacturer didn?t add enough cooling to keep the system cool enough with a continuous load. (Which to be honest, almost no laptop designers do, because they assuming you are going to be having a spiky load instead? - Benjamin From perso.olivier.barthelemy at gmail.com Thu Jan 26 08:11:12 2023 From: perso.olivier.barthelemy at gmail.com (Olivier B.) Date: Thu, 26 Jan 2023 14:11:12 +0100 Subject: Module use of python3_d.dll conflicts with this version of Python Message-ID: Hi,I am in the process of trying to make my code (an c++ executable and swig modules using the Python C API) lose the dependency to python 3.7, to be compatible with all Python 3.2+ I tried linking to python.lib instead of python37.lib. As i am still using a few things that are not in the limited API, my binaries now have a dependency to python3.dll for the base stuff, and pyhton37.dll for the few symbols that are not backwards compatible. In release, that does not seem to bring issues. In debug (debug build of my program that uses python debug build) however, when the process is importing the swig python/c++ module, i get a "Module use of python3_d.dll conflicts with this version of Python". I checked that i am properly loading the python3_d.dll of my python37, and python python37_d.dll too Looking in dynload_win.c of python sources where the message is triggered, indeed it is comparing in py case python37_d.dll to python3_d.dll. And i guess, in release, it works because in GetPythonModule(), there' is #ifndef _DEBUG /* In a release version, don't claim that python3.dll is a Python DLL. */ if (strcmp(import_name, "python3.dll") == 0) { import_data += 20; continue; } #endif Does someone know why it would have been chosen to be different for debug builds? From bowman at montana.com Thu Jan 26 00:13:07 2023 From: bowman at montana.com (rbowman) Date: 26 Jan 2023 05:13:07 GMT Subject: bool and int References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> Message-ID: On Thu, 26 Jan 2023 04:10:30 +1100, Chris Angelico wrote: > BASIC was like that too, although it (at least, the versions I used in > my childhood) didn't have "True" and "False", you just got the actual > values -1 and 0. They were the other way around compared to what you're > saying here though. I've see header files from people with boolean envy that are something like #define FALSE 0 #define TRUE ~FALSE From dino at no.spam.ar Thu Jan 26 08:33:45 2023 From: dino at no.spam.ar (Dino) Date: Thu, 26 Jan 2023 08:33:45 -0500 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> <33eaf8a0-02e2-ae19-30ba-c60815142930@tompassin.net> Message-ID: <0jvAL.2089656$odm4.145135@fx14.ams4> On 1/25/2023 4:30 PM, Thomas Passin wrote: > On 1/25/2023 3:29 PM, Dino wrote: > Great!? Don't forget what I said about potential overheating if you hit > the server with as many requests as it can handle. Noted. Thank you. From dino at no.spam.ar Thu Jan 26 09:25:36 2023 From: dino at no.spam.ar (Dino) Date: Thu, 26 Jan 2023 09:25:36 -0500 Subject: bool and int In-Reply-To: References: <001901d9312b$ef077b60$cd167220$@gmail.com> Message-ID: Wow. That was quite a message and an interesting read. Tempted to go deep and say what I agree and what I disagree with, but there are two issues: 1) time 2) I will soon be at a disadvantage discussing with people (you or others) who know more than me (which doesn't make them right necessarily, but certainly they'll have the upper-hand in a discussion). Personally, in the first part of my career I got into the habit of learning things fast, sometimes superficially I confess, and then get stuff done hopefully within time and budget. Not the recommended approach if you need to build software for a nuclear plant. An OK approach (within reason) if you build websites or custom solutions for this or that organization and the budget is what it is. After all, technology moves sooo fast, and what we learn in detail today is bound to be old and possibly useless 5 years down the road. Also, I argue that there is value in having familiarity with lots of different technologies (front-end and back-end) and knowing (or at lease, having a sense) of how they can all be made play together with an appreciation of the different challenges and benefits that each domain offers. Anyway, everything is equivalent to a Turing machine and IA will screw everyone, including programmers, eventually. Thanks again and have a great day Dino On 1/25/2023 9:14 PM, avi.e.gross at gmail.com wrote: > Dino, > > There is no such things as a "principle of least surprise" or if you insist > there is, I can nominate many more such "rules" such as "the principle of > get out of my way and let me do what I want!" > > Computer languages with too many rules are sometimes next to unusable in > practical situations. > > I am neither defending or attacking choices Python or other languages have > made. I merely observe and agree to use languages carefully and as > documented. > From list1 at tompassin.net Thu Jan 26 11:32:02 2023 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 26 Jan 2023 11:32:02 -0500 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: <63d2a403.050a0220.21432.10c7@mx.google.com> References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> <33eaf8a0-02e2-ae19-30ba-c60815142930@tompassin.net> <20230126003827.bepnprceb26a4jb4@hjp.at> <44ce3f73-45eb-7148-3f1f-91ddaf624f73@tompassin.net> <63d2a403.050a0220.21432.10c7@mx.google.com> Message-ID: <9de8830a-cc0e-6150-0127-839326783829@tompassin.net> On 1/26/2023 11:02 AM, Grant Edwards wrote: > On 2023-01-26, Thomas Passin wrote: >> On 1/25/2023 7:38 PM, Peter J. Holzer wrote: >>> On 2023-01-25 16:30:56 -0500, Thomas Passin wrote: >>>> Great! Don't forget what I said about potential overheating if you >>>> hit the server with as many requests as it can handle. >>> >>> Frankly, if you can overheat a server by hitting it with HTTP requests, >>> get better hardware and/or put it into a place with better airflow. >>> >> >> Frankly, if you have a server-grade machine then well and good but if >> you are running a nice quiet consumer grade laptop - my development >> machine - you need to be careful. > > A properly designed laptop with a non-broken OS will not overheat > regardless of the computing load you throw at it. The fan might get > annoying loud, but if it overheats either your hardware or OS needs > to be fixed. A nice theory but nothing to do with the real world. I've had a number of laptops that overheat (or would, if I let test program continue) running this test program. They have been different brands, different CPUs, different levels of noisy fans. I don't know how I would find one of your "properly designed laptops with a non-broken OS", or what could be done to fix it. Maybe a high-end gaming machine... which I don't wish to invest in or hear the fan noise from. Anyway, the point was to warn other people - who probably also wouldn't have a "properly designed laptop with a non-broken OS" - that they should keep an eye on their CPU core temperatures. In my experience, that's a real concern, whether or not it "should not" be an issue. From rosuav at gmail.com Thu Jan 26 11:32:04 2023 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 27 Jan 2023 03:32:04 +1100 Subject: bool and int In-Reply-To: References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> Message-ID: On Fri, 27 Jan 2023 at 03:31, rbowman wrote: > > On Thu, 26 Jan 2023 04:10:30 +1100, Chris Angelico wrote: > > > > BASIC was like that too, although it (at least, the versions I used in > > my childhood) didn't have "True" and "False", you just got the actual > > values -1 and 0. They were the other way around compared to what you're > > saying here though. > > I've see header files from people with boolean envy that are something > like > > #define FALSE 0 > #define TRUE ~FALSE > Yeah, that's the same logic that BASIC uses: false is zero, and true is the all-ones integer (which, interpreted as a two's complement signed number, is -1). ChrisA From rosuav at gmail.com Thu Jan 26 11:41:18 2023 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 27 Jan 2023 03:41:18 +1100 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: <9de8830a-cc0e-6150-0127-839326783829@tompassin.net> References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> <33eaf8a0-02e2-ae19-30ba-c60815142930@tompassin.net> <20230126003827.bepnprceb26a4jb4@hjp.at> <44ce3f73-45eb-7148-3f1f-91ddaf624f73@tompassin.net> <63d2a403.050a0220.21432.10c7@mx.google.com> <9de8830a-cc0e-6150-0127-839326783829@tompassin.net> Message-ID: On Fri, 27 Jan 2023 at 03:34, Thomas Passin wrote: > A nice theory but nothing to do with the real world. I've had a number > of laptops that overheat (or would, if I let test program continue) > running this test program. Define "overheat". If all you're saying is "the fan began to whine and I got annoyed so I shut off the program", that is absolutely NOT overheating. I would accept "the CPU thermally throttled to the point where the test was non-indicative" as a form of overheating, though then the warning should be "be aware that, if your web server is CPU-limited, this test may result in hard-to-interpret numbers due to requests per second varying with the change in CPU temperature", which isn't nearly as punchy. But unless you have a system where the heat sink isn't attached to the CPU properly, I'd be very surprised if you were able to actually damage your CPU this way. Maybe you could reduce the lifetime that way (the same way that crypto mining can shorten the lifespan of a GPU), but it shouldn't cause any sort of immediate damage. Even on a laptop. Feel free to prove me wrong, though. ChrisA From grant.b.edwards at gmail.com Thu Jan 26 12:14:24 2023 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Thu, 26 Jan 2023 09:14:24 -0800 (PST) Subject: HTTP server benchmarking/load testing in Python References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> <33eaf8a0-02e2-ae19-30ba-c60815142930@tompassin.net> <20230126003827.bepnprceb26a4jb4@hjp.at> <44ce3f73-45eb-7148-3f1f-91ddaf624f73@tompassin.net> <63d2a403.050a0220.21432.10c7@mx.google.com> <9de8830a-cc0e-6150-0127-839326783829@tompassin.net> Message-ID: <63d2b4f0.050a0220.c9e0b.11a2@mx.google.com> On 2023-01-26, Thomas Passin wrote: > On 1/26/2023 11:02 AM, Grant Edwards wrote: > >[...] > >> A properly designed laptop with a non-broken OS will not overheat >> regardless of the computing load you throw at it. The fan might get >> annoying loud, but if it overheats either your hardware or OS needs >> to be fixed. > > A nice theory but nothing to do with the real world. I've had a number > of laptops that overheat (or would, if I let test program continue) > running this test program. You mean they actually fail/crash? Or they just throttle the fans up and the CPU down to keep the core temperature within limits? -- Grant From dieter at handshake.de Thu Jan 26 12:16:01 2023 From: dieter at handshake.de (Dieter Maurer) Date: Thu, 26 Jan 2023 18:16:01 +0100 Subject: asyncio questions In-Reply-To: <2dea614d-b55a-fadc-7358-85f82e3f8afa@chagford.com> References: <2dea614d-b55a-fadc-7358-85f82e3f8afa@chagford.com> Message-ID: <25554.46417.1028.73937@ixdm.fritz.box> Frank Millman wrote at 2023-1-26 12:12 +0200: >I have written a simple HTTP server using asyncio. It works, but I don't >always understand how it works, so I was pleased that Python 3.11 >introduced some new high-level concepts that hide the gory details. I >want to refactor my code to use these concepts, but I am not finding it >easy. > >In simple terms my main loop looked like this - > > ??? loop = asyncio.get_event_loop() > ??? server = loop.run_until_complete( > ??????? asyncio.start_server(handle_client, host, port)) > ??? loop.run_until_complete(setup_companies()) > ??? session_check = asyncio.ensure_future( > ??????? check_sessions())? # start background task > ??? print('Press Ctrl+C to stop') > ??? try: > ??????? loop.run_forever() > ??? except KeyboardInterrupt: > ??????? print() > ??? finally: > ??????? session_check.cancel()? # tell session_check to stop running > ??????? loop.run_until_complete(asyncio.wait([session_check])) > ??????? server.close() > ??????? loop.stop() Why does your code uses several `loop.run*` calls? In fact, I would define a single coroutine and run that with `asyncio.run`. This way, the coroutine can use all `asyncio` features, including `loop.create_task`. From list1 at tompassin.net Thu Jan 26 12:29:50 2023 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 26 Jan 2023 12:29:50 -0500 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> <33eaf8a0-02e2-ae19-30ba-c60815142930@tompassin.net> <20230126003827.bepnprceb26a4jb4@hjp.at> <44ce3f73-45eb-7148-3f1f-91ddaf624f73@tompassin.net> <63d2a403.050a0220.21432.10c7@mx.google.com> <9de8830a-cc0e-6150-0127-839326783829@tompassin.net> Message-ID: On 1/26/2023 11:41 AM, Chris Angelico wrote: > On Fri, 27 Jan 2023 at 03:34, Thomas Passin wrote: >> A nice theory but nothing to do with the real world. I've had a number >> of laptops that overheat (or would, if I let test program continue) >> running this test program. > > Define "overheat". If all you're saying is "the fan began to whine and > I got annoyed so I shut off the program", that is absolutely NOT > overheating. CPU core temperatures up to 95 deg C and rising rapidly, as reported by a number of utilities including NZXT and CoreTemp. Max junction temperature is given as 100 deg C, and I don't want to risk reducing the lifetime of my CPU. Maybe five or ten minutes at or above 100 deg C every few months might not make a noticeable lifetime difference, who knows? I don't want to make a habit of it. I wouldn't drive my car very long with a low oil pressure warning active, either. From dino at no.spam.ar Thu Jan 26 12:12:30 2023 From: dino at no.spam.ar (Dino) Date: Thu, 26 Jan 2023 12:12:30 -0500 Subject: bool and int In-Reply-To: References: Message-ID: On 1/25/2023 5:42 PM, Chris Angelico wrote: > > Try this (or its equivalent) in as many languages as possible: > > x = (1 > 2) > x == 0 > > You'll find that x (which has effectively been set to False, or its > equivalent in any language) will be equal to zero in a very large > number of languages. Thus, to an experienced programmer, it would > actually be quite the opposite: having it NOT be a number would be the > surprising thing! I thought I had already responded to this, but I can't see it. Weird. Anyway, straight out of the Chrome DevTools console: ? x = (1>2) false x == 0 true typeof(x) 'boolean' typeof(0) 'number' typeof(x) == 'number' false So, you are technically correct, but you can see that JavaScript - which comes with many gotchas - does not offer this particular one. From rosuav at gmail.com Thu Jan 26 12:57:58 2023 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 27 Jan 2023 04:57:58 +1100 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> <33eaf8a0-02e2-ae19-30ba-c60815142930@tompassin.net> <20230126003827.bepnprceb26a4jb4@hjp.at> <44ce3f73-45eb-7148-3f1f-91ddaf624f73@tompassin.net> <63d2a403.050a0220.21432.10c7@mx.google.com> <9de8830a-cc0e-6150-0127-839326783829@tompassin.net> Message-ID: On Fri, 27 Jan 2023 at 04:31, Thomas Passin wrote: > > On 1/26/2023 11:41 AM, Chris Angelico wrote: > > On Fri, 27 Jan 2023 at 03:34, Thomas Passin wrote: > >> A nice theory but nothing to do with the real world. I've had a number > >> of laptops that overheat (or would, if I let test program continue) > >> running this test program. > > > > Define "overheat". If all you're saying is "the fan began to whine and > > I got annoyed so I shut off the program", that is absolutely NOT > > overheating. > > CPU core temperatures up to 95 deg C and rising rapidly, as reported by > a number of utilities including NZXT and CoreTemp. Max junction > temperature is given as 100 deg C, and I don't want to risk reducing the > lifetime of my CPU. > > Maybe five or ten minutes at or above 100 deg C every few months might > not make a noticeable lifetime difference, who knows? I don't want to > make a habit of it. I wouldn't drive my car very long with a low oil > pressure warning active, either. Did you get a warning, or did you just decide to stop the test? Did you continue the test and see what would happen? Did you, when the temperature got up to 95?, check what the CPU's clock frequency was? The easiest way to recognize thermal throttling is a reduction in frequency while at 100% utilization. Or did you just assume that, with a mere five degree buffer and your own personal analysis, that the CPU was just seconds away from total destruction? ChrisA From list1 at tompassin.net Thu Jan 26 14:53:07 2023 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 26 Jan 2023 14:53:07 -0500 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> <33eaf8a0-02e2-ae19-30ba-c60815142930@tompassin.net> <20230126003827.bepnprceb26a4jb4@hjp.at> <44ce3f73-45eb-7148-3f1f-91ddaf624f73@tompassin.net> <63d2a403.050a0220.21432.10c7@mx.google.com> <9de8830a-cc0e-6150-0127-839326783829@tompassin.net> Message-ID: On 1/26/2023 12:57 PM, Chris Angelico wrote: > On Fri, 27 Jan 2023 at 04:31, Thomas Passin wrote: >> >> On 1/26/2023 11:41 AM, Chris Angelico wrote: >>> On Fri, 27 Jan 2023 at 03:34, Thomas Passin wrote: >>>> A nice theory but nothing to do with the real world. I've had a number >>>> of laptops that overheat (or would, if I let test program continue) >>>> running this test program. >>> >>> Define "overheat". If all you're saying is "the fan began to whine and >>> I got annoyed so I shut off the program", that is absolutely NOT >>> overheating. >> >> CPU core temperatures up to 95 deg C and rising rapidly, as reported by >> a number of utilities including NZXT and CoreTemp. Max junction >> temperature is given as 100 deg C, and I don't want to risk reducing the >> lifetime of my CPU. >> >> Maybe five or ten minutes at or above 100 deg C every few months might >> not make a noticeable lifetime difference, who knows? I don't want to >> make a habit of it. I wouldn't drive my car very long with a low oil >> pressure warning active, either. > > Did you get a warning, or did you just decide to stop the test? (At least) one of the utilities, I forget which one, did show the temperature in a danger zone. > Did you continue the test and see what would happen? No, why would I? Would you go up to the edge of a cliff, past the warning signs, and when the ground started to crumble take another step to see if it would really collapse? > Did you, when the temperature got up to 95?, check what the CPU's > clock frequency was? The easiest way to recognize thermal throttling > is a reduction in frequency while at 100% utilization. No, there was no point. Maybe it would have throttled, maybe no damage would have occurred. But doing so would not have accomplished anything, since I already had the throughput numbers I needed and the purpose of the test was not to see how hard I could drive the system before hardware failure. I'll leave that to Tom's Hardware or some gamers' site. > Or did you just assume that, with a mere five degree buffer and your > own personal analysis, that the CPU was just seconds away from total > destruction? To quote myself from my last message: "Maybe five or ten minutes at or above 100 deg C every few months might not make a noticeable lifetime difference, who knows? I don't want to make a habit of it. I wouldn't drive my car very long with a low oil pressure warning active, either." From 2QdxY4RzWzUUiLuE at potatochowder.com Thu Jan 26 15:31:37 2023 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Thu, 26 Jan 2023 15:31:37 -0500 Subject: bool and int In-Reply-To: References: Message-ID: On 2023-01-26 at 12:12:30 -0500, Dino wrote: > On 1/25/2023 5:42 PM, Chris Angelico wrote: > > > > > Try this (or its equivalent) in as many languages as possible: > > > > x = (1 > 2) > > x == 0 > > > > You'll find that x (which has effectively been set to False, or its > > equivalent in any language) will be equal to zero in a very large > > number of languages. Thus, to an experienced programmer, it would > > actually be quite the opposite: having it NOT be a number would be the > > surprising thing! > > I thought I had already responded to this, but I can't see it. Weird. > > Anyway, straight out of the Chrome DevTools console: > ? > x = (1>2) > false > > x == 0 > true > > typeof(x) > 'boolean' > > typeof(0) > 'number' > > typeof(x) == 'number' > false > > So, you are technically correct, but you can see that JavaScript - which > comes with many gotchas - does not offer this particular one. When you start a new language, try to start from the beginning. Yes, you know other languages, to varying degrees, and the language you are learning is very likely similar, at least superficially, in some way or another, to one of those other languages. Use that existing knowledge to the extent that it is useful; be prepared to forget everything you know. Python's choice of type hierarchy is not at all uncommon, and it's only a gotcha if you come in with a predetermined idea of how certain things "should" work. I've already noted that writing FORTRAN in any language is a meme. For a completely different take on booleans, take a look at Python's logical "and" and "or" operators (*not* the arithmetic "|" and "&" operators), which only sometimes return an actual boolean value. Then compare them to the "any" and "all" builtins, which came along much later. Idiomatic Python uses the right tool for the job *in Python*, whether or not that same tool is or isn;'t the right tool for the same job in some other language. Python is like Lisp in that it (Python) has strong dynamic typing. From my Common Lisp REPL: CL-USER> (let ((x (< 1 2))) (cons x (type-of x))) (T . BOOLEAN) CL-USER> (let ((x (> 1 2))) (cons x (type-of x))) (NIL . NULL) In English, the canonical "true" value in Lisp is T, and its type is BOOLEAN. Likewise, the canonical "false" value in Lisp is NIL, and its type is NULL. Out of the box, Lisp will not convert T or NIL to a number. From ben.usenet at bsb.me.uk Thu Jan 26 15:29:33 2023 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Thu, 26 Jan 2023 20:29:33 +0000 Subject: bool and int References: Message-ID: <87o7qlyr82.fsf@bsb.me.uk> Chris Angelico writes: > On Thu, 26 Jan 2023 at 08:19, Dino wrote: >> >> On 1/23/2023 11:22 PM, Dino wrote: >> > >>> b = True >> > >>> isinstance(b,bool) >> > True >> > >>> isinstance(b,int) >> > True >> > >>> >> >> ok, I read everything you guys wrote. Everyone's got their reasons >> obviously, but allow me to observe that there's also something called >> "principle of least surprise". >> >> In my case, it took me some time to figure out where a nasty bug was >> hidden. Letting a bool be a int is quite a gotcha, no matter how hard >> the benevolent dictator tries to convince me otherwise! >> > > Try this (or its equivalent) in as many languages as possible: > > x = (1 > 2) > x == 0 > > You'll find that x (which has effectively been set to False, or its > equivalent in any language) will be equal to zero in a very large > number of languages. Thus, to an experienced programmer, it would > actually be quite the opposite: having it NOT be a number would be the > surprising thing! I think the programmer's experience would have to have been rather narrow to be surprised by x not being treated as a number. I started with Fortran, Lisp, Pascal and two variants of Algol so I started out un-surprised by Boolean being a non-arithmetic type. To be surprised by x (above) /not/ being treated as a number, an experienced programmer would have had to have avoided a lot of strictly typed languages. I've just tried Scheme, Haskell, Common Lisp, Ada, Algol-68, go, ML, Rust, Ruby, Java, Lua, Prolog and Fortran and they all either say "no way!" or give false for x == 0. Of course these are not random choices, but it shows that Python's design is very far from universal. But then this is not a numbers game, anyway. A programmer need only to have used one or two languages that are rather more strict about types to find such a thing unsurprising in future. -- Ben. From rosuav at gmail.com Thu Jan 26 17:00:32 2023 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 27 Jan 2023 09:00:32 +1100 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> <33eaf8a0-02e2-ae19-30ba-c60815142930@tompassin.net> <20230126003827.bepnprceb26a4jb4@hjp.at> <44ce3f73-45eb-7148-3f1f-91ddaf624f73@tompassin.net> <63d2a403.050a0220.21432.10c7@mx.google.com> <9de8830a-cc0e-6150-0127-839326783829@tompassin.net> Message-ID: On Fri, 27 Jan 2023 at 06:54, Thomas Passin wrote: > > Did you get a warning, or did you just decide to stop the test? > > (At least) one of the utilities, I forget which one, did show the > temperature in a danger zone. I'm very curious as to which utility, and on what basis it called it "danger". Notably, whether there's any sort of actual manufacturer threshold that that was based on. Personally? Very dubious. Your entire premise is "five degrees MUST be a problem", without any visible basis. ChrisA From rosuav at gmail.com Thu Jan 26 17:03:26 2023 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 27 Jan 2023 09:03:26 +1100 Subject: bool and int In-Reply-To: References: Message-ID: On Fri, 27 Jan 2023 at 06:32, Dino wrote: > > On 1/25/2023 5:42 PM, Chris Angelico wrote: > > > > > Try this (or its equivalent) in as many languages as possible: > > > > x = (1 > 2) > > x == 0 > > > > You'll find that x (which has effectively been set to False, or its > > equivalent in any language) will be equal to zero in a very large > > number of languages. Thus, to an experienced programmer, it would > > actually be quite the opposite: having it NOT be a number would be the > > surprising thing! > > I thought I had already responded to this, but I can't see it. Weird. > > Anyway, straight out of the Chrome DevTools console: > ? > x = (1>2) > false > > x == 0 > true > > typeof(x) > 'boolean' > > typeof(0) > 'number' > > typeof(x) == 'number' > false > > So, you are technically correct, but you can see that JavaScript - which > comes with many gotchas - does not offer this particular one. > That's not what I said, though! What you did in JS was the equivalent of this: >>> type(True) == type(1) False which isn't the same as an isinstance check. Instead, try *exactly what I said*. You'll find that false is equal to zero and true is equal to one, just like it is in Python. ChrisA From bgesztesi at msn.com Thu Jan 26 16:40:55 2023 From: bgesztesi at msn.com (Bela Gesztesi) Date: Thu, 26 Jan 2023 21:40:55 +0000 Subject: Python not found Message-ID: I have downloaded python, checking path installation and I receive the following: C:\DJI>py comm_og_service_tool.py WM231 --port COM3 GimbalCalib JointCoarse Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases. Please offer any suggestions, Thank you, BG From eryksun at gmail.com Thu Jan 26 17:47:36 2023 From: eryksun at gmail.com (Eryk Sun) Date: Thu, 26 Jan 2023 16:47:36 -0600 Subject: Python not found In-Reply-To: References: Message-ID: On 1/26/23, Bela Gesztesi wrote: > > C:\DJI>py comm_og_service_tool.py WM231 --port COM3 GimbalCalib JointCoarse > > Python was not found; run without arguments to install from the Microsoft > Store, or disable this shortcut from Settings > Manage App Execution > Aliases. Do what it's telling you to do. Either install the app version of Python, or disable the "python.exe" and "python3.exe" app aliases. The "python.exe" and "python3.exe" app aliases that are distributed with Windows run a "PythonRedirector" app that just opens the Microsoft Store to download the latest version of the app version of Python. When run with command-line arguments, they print the message that's quoted above. What's happening here is that the py launcher reads a shebang line from "comm_og_service_tool.py" of the form "#!/usr/bin/env python3". It thus searches PATH for "python3.exe" and finds the app alias. We should probably update the launcher to ignore an alias to the "PythonRedirector" app. From eryksun at gmail.com Thu Jan 26 17:55:05 2023 From: eryksun at gmail.com (Eryk Sun) Date: Thu, 26 Jan 2023 16:55:05 -0600 Subject: Module use of python3_d.dll conflicts with this version of Python In-Reply-To: References: Message-ID: On 1/26/23, Olivier B. wrote: > > Does someone know why it would have been chosen to be different for > debug builds? It's assumed that a debug build would normally link with "pythonXY_d.dll". Maybe it should be more defensive. Refer to the following setup in PC/pyconfig.h: /* For an MSVC DLL, we can nominate the .lib files used by extensions */ #ifdef MS_COREDLL # if !defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_BUILTIN) /* not building the core - must be an ext */ # if defined(_MSC_VER) /* So MSVC users need not specify the .lib file in their Makefile (other compilers are generally taken care of by distutils.) */ # if defined(_DEBUG) # pragma comment(lib,"python312_d.lib") # elif defined(Py_LIMITED_API) # pragma comment(lib,"python3.lib") # else # pragma comment(lib,"python312.lib") # endif /* _DEBUG */ # endif /* _MSC_VER */ # endif /* Py_BUILD_CORE */ #endif /* MS_COREDLL */ From barry at barrys-emacs.org Thu Jan 26 18:39:24 2023 From: barry at barrys-emacs.org (Barry) Date: Thu, 26 Jan 2023 23:39:24 +0000 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: References: Message-ID: <93DF43DD-1798-45A4-AA0A-665F9C64D1DC@barrys-emacs.org> > On 26 Jan 2023, at 17:32, Thomas Passin wrote: > > ?On 1/26/2023 11:41 AM, Chris Angelico wrote: >>> On Fri, 27 Jan 2023 at 03:34, Thomas Passin wrote: >>> A nice theory but nothing to do with the real world. I've had a number >>> of laptops that overheat (or would, if I let test program continue) >>> running this test program. >> Define "overheat". If all you're saying is "the fan began to whine and >> I got annoyed so I shut off the program", that is absolutely NOT >> overheating. > > CPU core temperatures up to 95 deg C and rising rapidly, as reported by a number of utilities including NZXT and CoreTemp. Max junction temperature is given as 100 deg C, and I don't want to risk reducing the lifetime of my CPU. Silicon junctions melt something like 400C ish not 100C. The max you see is the operating temp of the CPU. For intel CPU if you go beyond what the slow clocking can deal with the CPU turns itself off to prevent damage. Intel did this to stop people asking for replacement parts when there cooling was at fault. Barry > > Maybe five or ten minutes at or above 100 deg C every few months might not make a noticeable lifetime difference, who knows? I don't want to make a habit of it. I wouldn't drive my car very long with a low oil pressure warning active, either. > -- > https://mail.python.org/mailman/listinfo/python-list > From avi.e.gross at gmail.com Thu Jan 26 22:09:34 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Thu, 26 Jan 2023 22:09:34 -0500 Subject: bool and int In-Reply-To: References: <001901d9312b$ef077b60$cd167220$@gmail.com> Message-ID: <007801d931fc$c8a97a50$59fc6ef0$@gmail.com> [Dino has a deliberately invalid email address so sending him anything privately is not an option.] Dino, I would agree with you that for some purposes, you do NOT need to dig deep into a language to get fairly routine things done. You can often borrow ideas and code from an online search and hopefully cobble "a" solution together that works well enough. Of course it may suddenly fall apart. An example is code written that assumes 4 specific files exist in the current directory and unconditionally reads them and does stuff and eventually overwrites some other file with new data. OOPS, what does the program do it one or more files do not exist in the current directory under those exact names? Does it check if they exist and exit gracefully, or start throwing error messages as the code blindly proceeds on trying to change variables that were never created and so on, and then end with perhaps an emptied file and ruin lots of things? Too many examples you get from the internet are a bit like that. They often just tell you how to do something specific but leave out lots of details that are a good idea for many kinds of code. And some adjustments you make may break things and produce a result you trust but that is actually wrong. So if you find something you don't like about the language, guess what. You have very little standing if you never learned much more than what it took to copy some code. You are likely to be told things by others ranging from suggesting you need to do it another way or that the language is doing exactly what was designed and suggestions you go back to school and get a proper education and then the answer may be obvious. It truly does NOT matter what YOU think should happen unless you point to documentation that says something else should have happened. I will skip a lengthier reply but am thinking how some languages use a boxing/unboxing approach to wrap native "objects" like an integer. In many cases, your program is allowed to treat this as either a wrapped object or unboxed as needed and the compiler or interpreter keeps making changes behind the scenes so you see it as needed. In a sense, Booleans are a restricted form of integer and are viewed one of several ways. Python goes many steps further and has a concept of truthy that maps practically anything to True or False. The bottom line is not that Python or any language is wrong or right or great or horrible. It is that based on your own admission, you have not taken steps to understand more than you have to and thus are in a weak position to argue anything. Not because any of us are smarter in some sense, but because some of us do study more intensively and come ready to re-evaluate our ideas when we encounter others. What Python does in the situation discussed is pretty much what an assortment of languages include many C-based ones do. If it happens that your background is limited, fine. Many of us here have been exposed to the ideas in 5 or a dozen or literally hundreds of languages and variants and are well aware that some languages treat Booleans differently. But note we are posting on this forum so we generally don't find it that objectionable the way Python has chosen. We welcome well-intentioned discussions and the question is not at all stupid. But our answers may not be being seen as reasonable and that can be of some concern. The answer remains that the language was designed this way and many are happy with the design. Interestingly, I wonder if anyone has designed an alternate object type that can be used mostly in place of Booleans but which imposes changes and restrictions so trying to add a Boolean to an integer, or vice versa, results in an error. Python is flexible enough to do that and perhaps there already is a module out there ... -----Original Message----- From: Python-list On Behalf Of Dino Sent: Thursday, January 26, 2023 9:26 AM To: python-list at python.org Subject: Re: RE: bool and int Wow. That was quite a message and an interesting read. Tempted to go deep and say what I agree and what I disagree with, but there are two issues: 1) time 2) I will soon be at a disadvantage discussing with people (you or others) who know more than me (which doesn't make them right necessarily, but certainly they'll have the upper-hand in a discussion). Personally, in the first part of my career I got into the habit of learning things fast, sometimes superficially I confess, and then get stuff done hopefully within time and budget. Not the recommended approach if you need to build software for a nuclear plant. An OK approach (within reason) if you build websites or custom solutions for this or that organization and the budget is what it is. After all, technology moves sooo fast, and what we learn in detail today is bound to be old and possibly useless 5 years down the road. Also, I argue that there is value in having familiarity with lots of different technologies (front-end and back-end) and knowing (or at lease, having a sense) of how they can all be made play together with an appreciation of the different challenges and benefits that each domain offers. Anyway, everything is equivalent to a Turing machine and IA will screw everyone, including programmers, eventually. Thanks again and have a great day Dino On 1/25/2023 9:14 PM, avi.e.gross at gmail.com wrote: > Dino, > > There is no such things as a "principle of least surprise" or if you > insist there is, I can nominate many more such "rules" such as "the > principle of get out of my way and let me do what I want!" > > Computer languages with too many rules are sometimes next to unusable > in practical situations. > > I am neither defending or attacking choices Python or other languages > have made. I merely observe and agree to use languages carefully and > as documented. > -- https://mail.python.org/mailman/listinfo/python-list From list1 at tompassin.net Thu Jan 26 22:10:34 2023 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 26 Jan 2023 22:10:34 -0500 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> <33eaf8a0-02e2-ae19-30ba-c60815142930@tompassin.net> <20230126003827.bepnprceb26a4jb4@hjp.at> <44ce3f73-45eb-7148-3f1f-91ddaf624f73@tompassin.net> <63d2a403.050a0220.21432.10c7@mx.google.com> <9de8830a-cc0e-6150-0127-839326783829@tompassin.net> Message-ID: On 1/26/2023 5:00 PM, Chris Angelico wrote: > On Fri, 27 Jan 2023 at 06:54, Thomas Passin wrote: >>> Did you get a warning, or did you just decide to stop the test? >> >> (At least) one of the utilities, I forget which one, did show the >> temperature in a danger zone. > > I'm very curious as to which utility, and on what basis it called it > "danger". Notably, whether there's any sort of actual manufacturer > threshold that that was based on. 1. we're talking maybe a dozen years ago, I don't remember every detail about wordings. Coretemp e.g., gives clear warnings (though at what I think are lower temperatures than necessary). 2. "What is Tjunction max temperature?" Tjunction max is the maximum thermal junction temperature that a processor will allow prior to using internal thermal control mechanisms to reduce power and limit temperature. Activation of the processor's thermal control system may cause performance loss as the processor typically reduces frequency and power to prevent overheating. The maximum junction temperature limit varies per product and usually is between 100?C-110?C." https://www.intel.com/content/www/us/en/support/articles/000005597/processors.html The utilities I used always stated a 100 deg limit for Tj. 3. "Is it bad if my processor frequently approaches or reaches its maximum temperature? Not necessarily. Many Intel? processors make use of Intel? Turbo Boost Technology, which allows them to operate at very high frequency for a short amount of time. When the processor is operating at or near its maximum frequency it's possible for the temperature to climb very rapidly and quickly reach its maximum temperature. In sustained workloads, it's possible the processor will operate at or near its maximum temperature limit. Being at maximum temperature while running a workload isn't necessarily cause for concern. Intel processors constantly monitor their temperature and can very rapidly adjust their frequency and power consumption to prevent overheating and damage." (same source) But automatic throttling wasn't common back when I first noticed the heating issue. > Personally? Very dubious. Your entire premise is "five degrees MUST be > a problem", without any visible basis. Bridges are built with 150 - 200 % strength margin. This doesn't mean you should deliberately overload one. Heat is the enemy of electronics - a very old lesson. Tj =~ 100 deg C for CPUs, a familiar figure. My premise, to use your word, is not what you say. It is to avoid excessive heat if at all possible, and if the manufacturer says the max junction temperature is 100 deg, I'm going to avoid approaching 100 deg if possible - or to minimize the stay there. Most chemical effects are exponentially sensitive to temperature and problems with semiconductors are likely to be chemical - remember, e.g., the purple plague? A chemical problem. So yes, checking with HWiNFO, my current system is throttling and power limiting during this particular test. That's good. And I'm still going to stay away from the highest temperatures when possible. Nuff said! From rosuav at gmail.com Thu Jan 26 22:32:00 2023 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 27 Jan 2023 14:32:00 +1100 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> <33eaf8a0-02e2-ae19-30ba-c60815142930@tompassin.net> <20230126003827.bepnprceb26a4jb4@hjp.at> <44ce3f73-45eb-7148-3f1f-91ddaf624f73@tompassin.net> <63d2a403.050a0220.21432.10c7@mx.google.com> <9de8830a-cc0e-6150-0127-839326783829@tompassin.net> Message-ID: On Fri, 27 Jan 2023 at 14:21, Thomas Passin wrote: > 2. "What is Tjunction max temperature?" > Tjunction max is the maximum thermal junction temperature that a > processor will allow prior to using internal thermal control mechanisms > to reduce power and limit temperature. Activation of the processor's > thermal control system may cause performance loss as the processor > typically reduces frequency and power to prevent overheating. The > maximum junction temperature limit varies per product and usually is > between 100?C-110?C." > > https://www.intel.com/content/www/us/en/support/articles/000005597/processors.html > > The utilities I used always stated a 100 deg limit for Tj. > Yeah, so "maximum" is "before performance loss", not "before damage". ChrisA From list1 at tompassin.net Thu Jan 26 23:21:31 2023 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 26 Jan 2023 23:21:31 -0500 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: References: <4gcAL.2393259$JNZ4.383220@fx12.ams4> <33eaf8a0-02e2-ae19-30ba-c60815142930@tompassin.net> <20230126003827.bepnprceb26a4jb4@hjp.at> <44ce3f73-45eb-7148-3f1f-91ddaf624f73@tompassin.net> <63d2a403.050a0220.21432.10c7@mx.google.com> <9de8830a-cc0e-6150-0127-839326783829@tompassin.net> Message-ID: On 1/26/2023 10:32 PM, Chris Angelico wrote: > On Fri, 27 Jan 2023 at 14:21, Thomas Passin wrote: >> 2. "What is Tjunction max temperature?" >> Tjunction max is the maximum thermal junction temperature that a >> processor will allow prior to using internal thermal control mechanisms >> to reduce power and limit temperature. Activation of the processor's >> thermal control system may cause performance loss as the processor >> typically reduces frequency and power to prevent overheating. The >> maximum junction temperature limit varies per product and usually is >> between 100?C-110?C." >> >> https://www.intel.com/content/www/us/en/support/articles/000005597/processors.html >> >> The utilities I used always stated a 100 deg limit for Tj. >> > > Yeah, so "maximum" is "before performance loss", not "before damage". Yeah, so a dozen years ago, when I first noticed the matter, most computers didn't have throttling and power reduction, so I got sensitized to it. Better safe than sorry. Airliners have safeguards against stalling, but it's a lot better not to try to stall them anyway (speaking as a (non-airline) pilot). Let's give this a rest, shall we? From list1 at tompassin.net Thu Jan 26 20:32:53 2023 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 26 Jan 2023 20:32:53 -0500 Subject: HTTP server benchmarking/load testing in Python In-Reply-To: <93DF43DD-1798-45A4-AA0A-665F9C64D1DC@barrys-emacs.org> References: <93DF43DD-1798-45A4-AA0A-665F9C64D1DC@barrys-emacs.org> Message-ID: <40b19496-048f-3ab4-042e-0481d0b3009d@tompassin.net> On 1/26/2023 6:39 PM, Barry wrote: > > >> On 26 Jan 2023, at 17:32, Thomas Passin wrote: >> >> ?On 1/26/2023 11:41 AM, Chris Angelico wrote: >>>> On Fri, 27 Jan 2023 at 03:34, Thomas Passin wrote: >>>> A nice theory but nothing to do with the real world. I've had a number >>>> of laptops that overheat (or would, if I let test program continue) >>>> running this test program. >>> Define "overheat". If all you're saying is "the fan began to whine and >>> I got annoyed so I shut off the program", that is absolutely NOT >>> overheating. >> >> CPU core temperatures up to 95 deg C and rising rapidly, as reported by a number of utilities including NZXT and CoreTemp. Max junction temperature is given as 100 deg C, and I don't want to risk reducing the lifetime of my CPU. > > Silicon junctions melt something like 400C ish not 100C. > The max you see is the operating temp of the CPU. Of course I know the junction isn't going to melt at 100 deg C. We're not talking low temperature solder here! > For intel CPU if you go beyond what the slow clocking can deal with the CPU turns itself off to prevent damage. > > Intel did this to stop people asking for replacement parts when there cooling was at fault. > > Barry > >> >> Maybe five or ten minutes at or above 100 deg C every few months might not make a noticeable lifetime difference, who knows? I don't want to make a habit of it. I wouldn't drive my car very long with a low oil pressure warning active, either. >> -- >> https://mail.python.org/mailman/listinfo/python-list >> > From frank at chagford.com Fri Jan 27 07:14:35 2023 From: frank at chagford.com (Frank Millman) Date: Fri, 27 Jan 2023 14:14:35 +0200 Subject: asyncio questions In-Reply-To: <25554.46417.1028.73937@ixdm.fritz.box> References: <2dea614d-b55a-fadc-7358-85f82e3f8afa@chagford.com> <25554.46417.1028.73937@ixdm.fritz.box> Message-ID: <3e260200-8541-82bf-dfc3-906e751462f1@chagford.com> On 2023-01-26 7:16 PM, Dieter Maurer wrote: > Frank Millman wrote at 2023-1-26 12:12 +0200: >> I have written a simple HTTP server using asyncio. It works, but I don't >> always understand how it works, so I was pleased that Python 3.11 >> introduced some new high-level concepts that hide the gory details. I >> want to refactor my code to use these concepts, but I am not finding it >> easy. >> >> In simple terms my main loop looked like this - >> >> ??? loop = asyncio.get_event_loop() >> ??? server = loop.run_until_complete( >> ??????? asyncio.start_server(handle_client, host, port)) >> ??? loop.run_until_complete(setup_companies()) >> ??? session_check = asyncio.ensure_future( >> ??????? check_sessions())? # start background task >> ??? print('Press Ctrl+C to stop') >> ??? try: >> ??????? loop.run_forever() >> ??? except KeyboardInterrupt: >> ??????? print() >> ??? finally: >> ??????? session_check.cancel()? # tell session_check to stop running >> ??????? loop.run_until_complete(asyncio.wait([session_check])) >> ??????? server.close() >> ??????? loop.stop() > > Why does your code uses several `loop.run*` calls? > > In fact, I would define a single coroutine and run that > with `asyncio.run`. > This way, the coroutine can use all `asyncio` features, > including `loop.create_task`. You are right, Dieter. The function that I showed above is a normal function, not an async one. There was no particular reason for this - I must have got it working like that at some point in the past, and 'if it ain't broke ...' I have changed it to async, which I call with 'asyncio.run'. It now looks like this - server = await asyncio.start_server(handle_client, host, port) await setup_companies() session_check = asyncio.create_task( check_sessions()) # start background task print('Press Ctrl+C to stop') try: await server.serve_forever() except asyncio.CancelledError: pass finally: session_check.cancel() # tell session_check to stop running await asyncio.wait([session_check]) server.close() It works exactly the same as before, and it is now much neater. Thanks for the input. Frank From frank at chagford.com Fri Jan 27 07:24:18 2023 From: frank at chagford.com (Frank Millman) Date: Fri, 27 Jan 2023 14:24:18 +0200 Subject: asyncio questions In-Reply-To: <3e260200-8541-82bf-dfc3-906e751462f1@chagford.com> References: <2dea614d-b55a-fadc-7358-85f82e3f8afa@chagford.com> <25554.46417.1028.73937@ixdm.fritz.box> <3e260200-8541-82bf-dfc3-906e751462f1@chagford.com> Message-ID: On 2023-01-27 2:14 PM, Frank Millman wrote: > > I have changed it to async, which I call with 'asyncio.run'. It now > looks like this - > > ??? server = await asyncio.start_server(handle_client, host, port) > ??? await setup_companies() > ??? session_check = asyncio.create_task( > ??????? check_sessions())? # start background task > > ??? print('Press Ctrl+C to stop') > > ??? try: > ??????? await server.serve_forever() > ??? except asyncio.CancelledError: > ??????? pass > ??? finally: > ??????? session_check.cancel()? # tell session_check to stop running > ??????? await asyncio.wait([session_check]) > ??????? server.close() > I don't think I need the 'finally' clause - the cleanup can all happen in the 'except' block. Frank From eryksun at gmail.com Fri Jan 27 11:23:10 2023 From: eryksun at gmail.com (Eryk Sun) Date: Fri, 27 Jan 2023 10:23:10 -0600 Subject: Python not found In-Reply-To: References: Message-ID: On 1/27/23, Bela Gesztesi wrote: > > I'm not that familiar with the steps to be taken. > > How do I find the app version of Python for my desktop? > or > I don't know how to disable the "python.exe" and "python3.exe" app aliases To install the app version, run "python3" from the command line. This will open the Microsoft Store to install the Python app. (If it doesn't, simply open the store from the start menu.) It may not open to the latest version of Python, which currently is 3.11.1. Search for "python" to find the latest version. Make sure the package is from "Python Software Foundation". Click the "Get" button. OR To disable the aliases for the default installer app, begin by right-clicking the start button and select "settings". Click on "Apps" in the sidebar. Then click on "Apps & Features" in the window. Click the drop-down arrow on "More settings". Click on "App execution aliases". Disable the aliases for "App Installer / python.exe" and "App Installer / python3.exe". Some of the above names will be localized to your preferred user interface language. Hopefully the translations are obvious. From rob.cliffe at btinternet.com Wed Jan 25 19:58:12 2023 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Thu, 26 Jan 2023 00:58:12 +0000 Subject: Evaluation of variable as f-string In-Reply-To: <5bb2546d-8c16-262d-9336-5b41cbf992b5@tompassin.net> References: <5bb2546d-8c16-262d-9336-5b41cbf992b5@tompassin.net> Message-ID: <5221ab5f-90fd-b7c9-4b84-e7a99e8320ff@btinternet.com> On 25/01/2023 19:38, Thomas Passin wrote: > > Stack overflow to the rescue: > > Search phrase:? "python evaluate string as fstring" > > https://stackoverflow.com/questions/47339121/how-do-i-convert-a-string-into-an-f-string > > > def effify(non_f_str: str): > ??? return eval(f'f"""{non_f_str}"""') > > print(effify(s))? # prints as expected: "-> z" Duh!? Am I the only one who feels stupid not thinking of this? Although of course it won't work if the string already contains triple double quotes. I believe this could be got round with some ingenuity (having the effify function parse the string and replace genuine (not embedded in single-quotes) triple double quotes with triple single quotes, though there are some complications). And the effify function can't be put in its own module unless it can be passed the globals and/or locals dictionaries as needed for eval to use.? Something like this: def effify(non_f_str, glob=None, loc=None): ??? return eval(f'f"""{non_f_str}"""', ??????? glob if glob is not None else globals(), ??????? loc if loc is not None else locals()) Best wishes Rob Cliffe From rob.cliffe at btinternet.com Tue Jan 24 16:40:48 2023 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Tue, 24 Jan 2023 21:40:48 +0000 Subject: bool and int In-Reply-To: References: Message-ID: On 24/01/2023 04:22, Dino wrote: > > $ python > Python 3.8.10 (default, Mar 15 2022, 12:22:08) > [GCC 9.4.0] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> b = True > >>> isinstance(b,bool) > True > >>> isinstance(b,int) > True > >>> > That immediately tells you that either ??? bool is a subclass of int ??? int is a subclass of bool ??? bool and int are both subclasses of some other class In fact the first one is true. This is not a logical necessity, but the way Python happens to be designed. Best wishes Rob Cliffe From rob.cliffe at btinternet.com Wed Jan 25 15:04:56 2023 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Wed, 25 Jan 2023 20:04:56 +0000 Subject: Evaluation of variable as f-string In-Reply-To: References: Message-ID: On 23/01/2023 18:02, Chris Angelico wrote: > On Tue, 24 Jan 2023 at 04:56, Johannes Bauer wrote: >> Hi there, >> >> is there an easy way to evaluate a string stored in a variable as if it >> were an f-string at runtime? >> >> ... >> >> This is supposedly for security reasons. However, when trying to emulate >> this behavior that I wanted (and know the security implications of), my >> solutions will tend to be less secure. Here is what I have been thinking >> about: > If you really want the full power of an f-string, then you're asking > for the full power of eval(), and that means all the security > implications thereof, not to mention the difficulties of namespacing. > Have you considered using the vanilla format() method instead? > > But if you really REALLY know what you're doing, just use eval() > directly. I don't really see what you'd gain from an f-string. At very > least, work with a well-defined namespace and eval whatever you need > in that context. > > Maybe, rather than asking for a way to treat a string as code, ask for > what you ACTUALLY need, and we can help? > > ChrisA Fair enough, Chris, but still ISTM that it is reasonable to ask (perhaps for a different use-case) whether there is a way of evaluating a string at runtime as if it were an f-string.? We encourage people to ask questions on this list, even though the answer will not always be what they're hoping for. I appreciate that the answer may be "No, because it would be a lot of work - and increase the maintenance burden - to support a relatively rare requirement". Perhaps someone will be inspired to write a function to do it. ? Best wishes Rob Cliffe From rosuav at gmail.com Fri Jan 27 14:18:16 2023 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 28 Jan 2023 06:18:16 +1100 Subject: Evaluation of variable as f-string In-Reply-To: References: Message-ID: On Sat, 28 Jan 2023 at 05:31, Rob Cliffe via Python-list wrote: > On 23/01/2023 18:02, Chris Angelico wrote: > > Maybe, rather than asking for a way to treat a string as code, ask for > > what you ACTUALLY need, and we can help? > > > > ChrisA > Fair enough, Chris, but still ISTM that it is reasonable to ask (perhaps > for a different use-case) whether there is a way of evaluating a string > at runtime as if it were an f-string. We encourage people to ask > questions on this list, even though the answer will not always be what > they're hoping for. No, it's not, because that's the "how do I use X to do Y" problem. Instead, just ask how to do *what you actually need*. If the best way to do that is to eval an f-string, then someone will suggest that. But, much much much more likely, the best way to do it would be something completely different. What, exactly? That's hard to say, because *we don't know what you actually need*. All you tell us is what you're attempting to do, which there is *no good way to achieve*. > I appreciate that the answer may be "No, because it would be a lot of > work - and increase the maintenance burden - to support a relatively > rare requirement". What about: "No, because it's a terrible TERRIBLE idea, requires that you do things horribly backwards, and we still don't even know what you're trying to do"? > Perhaps someone will be inspired to write a function to do it. ? See, we don't know what "it" is, so it's hard to write a function that's any better than the ones we've seen. Using eval() to construct an f-string and then parse it is TERRIBLE because: 1) It still doesn't work in general, and thus has caveats like "you can't use this type of quote character" 2) You would have to pass it a dictionary of variables, which also can't be done with full generality 3) These are the exact same problems, but backwards, that led to f-strings in the first place 4) eval is extremely slow and horrifically inefficient. For some reason, str.format() isn't suitable, but *you haven't said why*, so we have to avoid that in our solutions. So, to come back to your concern: > We encourage people to ask > questions on this list, even though the answer will not always be what > they're hoping for. Well, yes. If you asked "how can I do X", hoping the answer would be "with a runtime-evaluated f-string", then you're quite right - the answer might not be what you were hoping for. But since you asked "how can I evaluate a variable as if it were an f-string", the only possible answer is "you can't, and that's a horrible idea". Don't ask how to use X to do Y. Ask how to do Y. ChrisA From rambiusparkisanius at gmail.com Fri Jan 27 15:31:22 2023 From: rambiusparkisanius at gmail.com (Ivan "Rambius" Ivanov) Date: Fri, 27 Jan 2023 15:31:22 -0500 Subject: Custom help format for a choice argparse argument Message-ID: Hello, I am developing a script that accepts a time zone as an option. The time zone can be any from pytz.all_timezones. I have def main(): parser = argparse.ArgumentParser() parser.add_argument("-z", "--zone", choices=pytz.all_timezones) args = parser.parse_args() print(args) print(f"Specified timezone: {args.zone}") It works, but when I run it with the -h option it dumps all entries in pytz.all_timezones. I would like to modify the help format for just -z|--zone option. I read the docs about HelpFormatter and argparse.py and I ended up with class CustomHelpFormatter(argparse.HelpFormatter): def _metavar_formatter(self, action, default_metavar): if action.dest == 'zone': result = 'zone from pytz.all_timezones' def format(tuple_size): if isinstance(result, tuple): return result else: return (result, ) * tuple_size return format else: return super(CustomHelpFormatter, self)._metavar_formatter(action, default_metavar) def main(): parser = argparse.ArgumentParser(formatter_class=CustomHelpFormatter) parser.add_argument("-z", "--zone", choices=pytz.all_timezones) args = parser.parse_args() print(args) print(f"Specified timezone: {args.zone}") This works, but is there a more elegant way to achieve it? Regards rambius -- Tangra Mega Rock: http://www.radiotangra.com From gweatherby at uchc.edu Fri Jan 27 16:22:47 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Fri, 27 Jan 2023 21:22:47 +0000 Subject: Custom help format for a choice argparse argument In-Reply-To: References: Message-ID: Why not something like: parser.add_argument("-z", "--zone") args = parser.parse_args() if args.zone and args.zone not in ptyz.all_timezones: print(?Invalid timezone?,file=sys.stderr) ? From: Python-list on behalf of Ivan "Rambius" Ivanov Date: Friday, January 27, 2023 at 3:33 PM To: Python Subject: Custom help format for a choice argparse argument *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** Hello, I am developing a script that accepts a time zone as an option. The time zone can be any from pytz.all_timezones. I have def main(): parser = argparse.ArgumentParser() parser.add_argument("-z", "--zone", choices=pytz.all_timezones) args = parser.parse_args() print(args) print(f"Specified timezone: {args.zone}") It works, but when I run it with the -h option it dumps all entries in pytz.all_timezones. I would like to modify the help format for just -z|--zone option. I read the docs about HelpFormatter and argparse.py and I ended up with class CustomHelpFormatter(argparse.HelpFormatter): def _metavar_formatter(self, action, default_metavar): if action.dest == 'zone': result = 'zone from pytz.all_timezones' def format(tuple_size): if isinstance(result, tuple): return result else: return (result, ) * tuple_size return format else: return super(CustomHelpFormatter, self)._metavar_formatter(action, default_metavar) def main(): parser = argparse.ArgumentParser(formatter_class=CustomHelpFormatter) parser.add_argument("-z", "--zone", choices=pytz.all_timezones) args = parser.parse_args() print(args) print(f"Specified timezone: {args.zone}") This works, but is there a more elegant way to achieve it? Regards rambius -- Tangra Mega Rock: https://urldefense.com/v3/__http://www.radiotangra.com__;!!Cn_UX_p3!kiJusdm5pCptP3sOBX85KXqUJkqr2jSa4C_-WAqND7WkL-aw3BYbW50td_AcuzJ1XUPYIVO3JiLMc4gRWS885vTKFsFvaQ$ -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!kiJusdm5pCptP3sOBX85KXqUJkqr2jSa4C_-WAqND7WkL-aw3BYbW50td_AcuzJ1XUPYIVO3JiLMc4gRWS885vRXq-JKLg$ From rambiusparkisanius at gmail.com Fri Jan 27 16:27:17 2023 From: rambiusparkisanius at gmail.com (Ivan "Rambius" Ivanov) Date: Fri, 27 Jan 2023 16:27:17 -0500 Subject: Custom help format for a choice argparse argument In-Reply-To: References: Message-ID: Hello, On Fri, Jan 27, 2023 at 4:22 PM Weatherby,Gerard wrote: > > Why not something like: > > > parser.add_argument("-z", "--zone") > > args = parser.parse_args() > if args.zone and args.zone not in ptyz.all_timezones: > > print(?Invalid timezone?,file=sys.stderr) > This is what I use now. I still wonder if I can mold HelpFormatter to do what I want it to do. > ? > > > > > From: Python-list on behalf of Ivan "Rambius" Ivanov > Date: Friday, January 27, 2023 at 3:33 PM > To: Python > Subject: Custom help format for a choice argparse argument > > *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** > > Hello, > > I am developing a script that accepts a time zone as an option. The > time zone can be any from pytz.all_timezones. I have > > def main(): > parser = argparse.ArgumentParser() > parser.add_argument("-z", "--zone", choices=pytz.all_timezones) > args = parser.parse_args() > print(args) > print(f"Specified timezone: {args.zone}") > > It works, but when I run it with the -h option it dumps all entries in > pytz.all_timezones. I would like to modify the help format for just > -z|--zone option. I read the docs about HelpFormatter and argparse.py > and I ended up with > > class CustomHelpFormatter(argparse.HelpFormatter): > def _metavar_formatter(self, action, default_metavar): > if action.dest == 'zone': > result = 'zone from pytz.all_timezones' > def format(tuple_size): > if isinstance(result, tuple): > return result > else: > return (result, ) * tuple_size > return format > else: > return super(CustomHelpFormatter, > self)._metavar_formatter(action, default_metavar) > > > def main(): > parser = argparse.ArgumentParser(formatter_class=CustomHelpFormatter) > parser.add_argument("-z", "--zone", choices=pytz.all_timezones) > args = parser.parse_args() > print(args) > print(f"Specified timezone: {args.zone}") > > This works, but is there a more elegant way to achieve it? > > Regards > rambius > > -- > Tangra Mega Rock: https://urldefense.com/v3/__http://www.radiotangra.com__;!!Cn_UX_p3!kiJusdm5pCptP3sOBX85KXqUJkqr2jSa4C_-WAqND7WkL-aw3BYbW50td_AcuzJ1XUPYIVO3JiLMc4gRWS885vTKFsFvaQ$ > -- > https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!kiJusdm5pCptP3sOBX85KXqUJkqr2jSa4C_-WAqND7WkL-aw3BYbW50td_AcuzJ1XUPYIVO3JiLMc4gRWS885vRXq-JKLg$ -- Tangra Mega Rock: http://www.radiotangra.com From cs at cskk.id.au Fri Jan 27 16:06:39 2023 From: cs at cskk.id.au (Cameron Simpson) Date: Sat, 28 Jan 2023 08:06:39 +1100 Subject: Custom help format for a choice argparse argument In-Reply-To: References: Message-ID: On 27Jan2023 15:31, Ivan "Rambius" Ivanov wrote: >I am developing a script that accepts a time zone as an option. The >time zone can be any from pytz.all_timezones. I have > >def main(): > parser = argparse.ArgumentParser() > parser.add_argument("-z", "--zone", choices=pytz.all_timezones) [...] > >It works, but when I run it with the -h option it dumps all entries in >pytz.all_timezones. What happens if you just presupply a `help=` parameter in `add_argument`? Cheers, Cameron Simpson From rambiusparkisanius at gmail.com Fri Jan 27 16:53:02 2023 From: rambiusparkisanius at gmail.com (Ivan "Rambius" Ivanov) Date: Fri, 27 Jan 2023 16:53:02 -0500 Subject: Custom help format for a choice argparse argument In-Reply-To: References: Message-ID: Hello Cameron, On Fri, Jan 27, 2023 at 4:45 PM Cameron Simpson wrote: > > On 27Jan2023 15:31, Ivan "Rambius" Ivanov wrote: > >I am developing a script that accepts a time zone as an option. The > >time zone can be any from pytz.all_timezones. I have > > > >def main(): > > parser = argparse.ArgumentParser() > > parser.add_argument("-z", "--zone", choices=pytz.all_timezones) > [...] > > > >It works, but when I run it with the -h option it dumps all entries in > >pytz.all_timezones. > > What happens if you just presupply a `help=` parameter in > `add_argument`? I tried with def main(): parser = argparse.ArgumentParser() parser.add_argument("-z", "--zone", choices=pytz.all_timezones, help="a timezone from pytz.all_timezones") args = parser.parse_args() print(args) -h still shows all the contents of pytz.all_timezones. Regards rambius > > Cheers, > Cameron Simpson > -- > https://mail.python.org/mailman/listinfo/python-list -- Tangra Mega Rock: http://www.radiotangra.com From rob.cliffe at btinternet.com Fri Jan 27 17:54:48 2023 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Fri, 27 Jan 2023 22:54:48 +0000 Subject: Evaluation of variable as f-string In-Reply-To: References: Message-ID: <8ebc73d2-e14a-b9ad-fa35-5078e5f9fd6c@btinternet.com> Whoa! Whoa! Whoa! I appreciate the points you are making, Chris, but I am a bit taken aback by such forceful language. On 27/01/2023 19:18, Chris Angelico wrote: > On Sat, 28 Jan 2023 at 05:31, Rob Cliffe via Python-list > wrote: >> On 23/01/2023 18:02, Chris Angelico wrote: >>> Maybe, rather than asking for a way to treat a string as code, ask for >>> what you ACTUALLY need, and we can help? >>> >>> ChrisA >> Fair enough, Chris, but still ISTM that it is reasonable to ask (perhaps >> for a different use-case) whether there is a way of evaluating a string >> at runtime as if it were an f-string. We encourage people to ask >> questions on this list, even though the answer will not always be what >> they're hoping for. > No, it's not, because that's the "how do I use X to do Y" problem. > Instead, just ask how to do *what you actually need*. If the best way > to do that is to eval an f-string, then someone will suggest that. > But, much much much more likely, the best way to do it would be > something completely different. What, exactly? That's hard to say, > because *we don't know what you actually need*. All you tell us is > what you're attempting to do, which there is *no good way to achieve*. If the above is addressed to the OP, I can't answer for him. If it's addressed to me:? How about if I wanted a program (a learning tool) to allow the user to play with f-strings? I.e. to type in a string, and then see what the result would be if it had been an f-string? I suspect there are other use cases, but I confess I can't think of one right now. > >> I appreciate that the answer may be "No, because it would be a lot of >> work - and increase the maintenance burden - to support a relatively >> rare requirement". > What about: "No, because it's a terrible TERRIBLE idea, requires that > you do things horribly backwards, and we still don't even know what > you're trying to do"? > >> Perhaps someone will be inspired to write a function to do it. ? > See, we don't know what "it" is, so it's hard to write a function > that's any better than the ones we've seen. Again, if this is addressed to the OP: I'm not his keeper. ? If it's addressed to me: "it" means a function that will take a string and evaluate it at runtime as if it were an f-string.? Sure, with caveats and limitations.? And indeed Thomas Passim found this partial solution on Stack Overflow: def effify(non_f_str: str): ??? return eval(f'f"""{non_f_str}"""') > Using eval() to construct > an f-string and then parse it is TERRIBLE because: > > 1) It still doesn't work in general, and thus has caveats like "you > can't use this type of quote character" > 2) You would have to pass it a dictionary of variables, which also > can't be done with full generality > 3) These are the exact same problems, but backwards, that led to > f-strings in the first place > 4) eval is extremely slow and horrifically inefficient. I understand these limitations.? Nonetheless I can conceive that there may be scenarios where it is an acceptable solution (perhaps the learning tool program I suggested above). Addressing your points specifically: ??? 1) I believe the quote character limitation could be overcome. It would need a fair amount of work, for which I haven't (yet) the time or inclination. ??? 2) Yes in general you would have to pass it one dictionary, maybe two.? I don't see this as an insuperable obstacle.? I am not sure what you mean by "can't be done with full generality" and perhaps that's not important. ??? 3) Not sure I understand this. ??? 4) On the fairly rare occasions that I have used eval(), I can't remember speed ever being a consideration. > > For some reason, str.format() isn't suitable, but *you haven't said > why*, so we have to avoid that in our solutions. So, to come back to > your concern: > >> We encourage people to ask >> questions on this list, even though the answer will not always be what >> they're hoping for. > Well, yes. If you asked "how can I do X", hoping the answer would be > "with a runtime-evaluated f-string", then you're quite right - the > answer might not be what you were hoping for. But since you asked "how > can I evaluate a variable as if it were an f-string", the only > possible answer is "you can't, and that's a horrible idea". I hope that I have shown that this is a somewhat dogmatic response. > > Don't ask how to use X to do Y. Ask how to do Y. Good advice. Best wishes Rob Cliffe > > ChrisA From list1 at tompassin.net Fri Jan 27 18:21:47 2023 From: list1 at tompassin.net (Thomas Passin) Date: Fri, 27 Jan 2023 18:21:47 -0500 Subject: Custom help format for a choice argparse argument In-Reply-To: References: Message-ID: On 1/27/2023 4:53 PM, Ivan "Rambius" Ivanov wrote: > Hello Cameron, > > On Fri, Jan 27, 2023 at 4:45 PM Cameron Simpson wrote: >> >> On 27Jan2023 15:31, Ivan "Rambius" Ivanov wrote: >>> I am developing a script that accepts a time zone as an option. The >>> time zone can be any from pytz.all_timezones. I have >>> >>> def main(): >>> parser = argparse.ArgumentParser() >>> parser.add_argument("-z", "--zone", choices=pytz.all_timezones) >> [...] >>> >>> It works, but when I run it with the -h option it dumps all entries in >>> pytz.all_timezones. >> >> What happens if you just presupply a `help=` parameter in >> `add_argument`? > > I tried with def main(): > parser = argparse.ArgumentParser() > parser.add_argument("-z", "--zone", choices=pytz.all_timezones, > help="a timezone from pytz.all_timezones") > args = parser.parse_args() > print(args) > > -h still shows all the contents of pytz.all_timezones. > Adding a few arguments makes it work (with the help of the argparse doc page and Stack Overflow: https://stackoverflow.com/questions/14950964/overriding-default-argparse-h-behaviour): import argparse import pytz HELP ="""\nThis is the help message. This is the second line of the help message.""" def main(): parser = argparse.ArgumentParser(add_help=False, usage = HELP) parser.add_argument("-z", "--zone", choices=pytz.all_timezones, help = argparse.SUPPRESS) parser.add_argument('-h', '--h', action = 'help', help = argparse.SUPPRESS) args = parser.parse_args() print(args) print(f"Specified timezone: {args.zone}") main() Your help message will display after "usage", like this - usage: This is the help message. This is the second line of the help message. You would give up have argparse automatically list all the options, but you can add add them manually to the HELP string. From rosuav at gmail.com Fri Jan 27 18:41:40 2023 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 28 Jan 2023 10:41:40 +1100 Subject: Evaluation of variable as f-string In-Reply-To: <8ebc73d2-e14a-b9ad-fa35-5078e5f9fd6c@btinternet.com> References: <8ebc73d2-e14a-b9ad-fa35-5078e5f9fd6c@btinternet.com> Message-ID: On Sat, 28 Jan 2023 at 10:08, Rob Cliffe via Python-list wrote: > > Whoa! Whoa! Whoa! > I appreciate the points you are making, Chris, but I am a bit taken > aback by such forceful language. The exact same points have already been made, but not listened to. Sometimes, forceful language is required in order to get people to listen. > If it's addressed to me: How about if I wanted a program (a learning > tool) to allow the user to play with f-strings? > I.e. to type in a string, and then see what the result would be if it > had been an f-string? > I suspect there are other use cases, but I confess I can't think of one > right now. Use the REPL, which will happily evaluate f-strings in their original context, just like any other code would. You're already eval'ing, so it's exactly what you'd expect. This is not the same thing as "typing in a string", though - it's typing in code and seeing what the result would be. (Except to the extent that source code can be considered a string.) > If it's addressed to me: "it" means a function that will take a string > and evaluate it at runtime as if it were an f-string. Sure, with > caveats and limitations. And that's what I am saying is a terrible terrible idea. It will evaluate things in the wrong context, it has all the normal problems of eval, and then it introduces its own unique problems with quote characters. > And indeed Thomas Passim found this partial > solution on Stack Overflow: > def effify(non_f_str: str): > return eval(f'f"""{non_f_str}"""') You can find anything on Stack Overflow. Just because you found it there doesn't mean it's any good - even if it's been massively upvoted. > Addressing your points specifically: > 1) I believe the quote character limitation could be overcome. It > would need a fair amount of work, for which I haven't (yet) the time or > inclination. No problem. Here, solve it for this string: eval_me = ' f"""{f\'\'\'{f"{f\'{1+2}\'}"}\'\'\'}""" ' F-strings can be nested, remember. > 2) Yes in general you would have to pass it one dictionary, maybe > two. I don't see this as an insuperable obstacle. I am not sure what > you mean by "can't be done with full generality" and perhaps that's not > important. >>> def func(): ... x = 1 ... class cls: ... y = 2 ... print(f"{x=} {y=}") ... print(locals()) ... >>> func() x=1 y=2 {'__module__': '__main__', '__qualname__': 'func..cls', 'y': 2} Maybe you don't care. Maybe you do. But locals() is not the same as "all names currently available in this scope". And, this example is definitely not something I would recommend, but good luck making this work with eval: >>> def func(): ... x = 1 ... print(f"{(x:=2)}") ... print(x) ... >>> func() 2 2 ... x = 1 ... print(eval("(x:=2)", globals(), locals())) ... print(x) ... >>> func() 2 1 > 3) Not sure I understand this. Before f-strings existed, one of the big problems with "just use str.format_map" was that you can't just pass it locals() to get all the available names. You also can't eval arbitrary code and expect to get the same results, even if you pass it globals and locals. And various other considerations here - the exact issues seen here, but flipped on their heads. So the obvious question is: why not just use str.format_map? > > Well, yes. If you asked "how can I do X", hoping the answer would be > > "with a runtime-evaluated f-string", then you're quite right - the > > answer might not be what you were hoping for. But since you asked "how > > can I evaluate a variable as if it were an f-string", the only > > possible answer is "you can't, and that's a horrible idea". > I hope that I have shown that this is a somewhat dogmatic response. And I hope that I have shown that it is fully justified. > > Don't ask how to use X to do Y. Ask how to do Y. > Good advice. Exactly. As I have shown, asking how to use f-strings to achieve this is simply not suitable, and there's no useful way to discuss other than to argue semantics. If we had a GOAL to discuss, we could find much better options. ChrisA From list1 at tompassin.net Fri Jan 27 18:38:58 2023 From: list1 at tompassin.net (Thomas Passin) Date: Fri, 27 Jan 2023 18:38:58 -0500 Subject: Evaluation of variable as f-string In-Reply-To: <8ebc73d2-e14a-b9ad-fa35-5078e5f9fd6c@btinternet.com> References: <8ebc73d2-e14a-b9ad-fa35-5078e5f9fd6c@btinternet.com> Message-ID: <42aedd2c-f4c6-1fd3-e40c-79fb547dc179@tompassin.net> On 1/27/2023 5:54 PM, Rob Cliffe via Python-list wrote: > Whoa! Whoa! Whoa! > I appreciate the points you are making, Chris, but I am a bit taken > aback by such forceful language. I generally agree with asking for what the intent is. In this case it seems pretty clear that the OP wants to use these string fragments as templates, and he needs to be able to insert variables into them at runtime, not compile time. So I think a good response would have been roughly "It looks like you want to use these strings as templates, is that right? If not, please tell us what you are trying to do, because it's hard to help without knowing that. If it's right, here's a way you could go about it." Short and amiable. > On 27/01/2023 19:18, Chris Angelico wrote: >> On Sat, 28 Jan 2023 at 05:31, Rob Cliffe via Python-list >> wrote: >>> On 23/01/2023 18:02, Chris Angelico wrote: >>>> Maybe, rather than asking for a way to treat a string as code, ask for >>>> what you ACTUALLY need, and we can help? >>>> >>>> ChrisA >>> Fair enough, Chris, but still ISTM that it is reasonable to ask (perhaps >>> for a different use-case) whether there is a way of evaluating a string >>> at runtime as if it were an f-string.? We encourage people to ask >>> questions on this list, even though the answer will not always be what >>> they're hoping for. >> No, it's not, because that's the "how do I use X to do Y" problem. >> Instead, just ask how to do *what you actually need*. If the best way >> to do that is to eval an f-string, then someone will suggest that. >> But, much much much more likely, the best way to do it would be >> something completely different. What, exactly? That's hard to say, >> because *we don't know what you actually need*. All you tell us is >> what you're attempting to do, which there is *no good way to achieve*. > If the above is addressed to the OP, I can't answer for him. > If it's addressed to me:? How about if I wanted a program (a learning > tool) to allow the user to play with f-strings? > I.e. to type in a string, and then see what the result would be if it > had been an f-string? > I suspect there are other use cases, but I confess I can't think of one > right now. >> >>> I appreciate that the answer may be "No, because it would be a lot of >>> work - and increase the maintenance burden - to support a relatively >>> rare requirement". >> What about: "No, because it's a terrible TERRIBLE idea, requires that >> you do things horribly backwards, and we still don't even know what >> you're trying to do"? >> >>> Perhaps someone will be inspired to write a function to do it. ? >> See, we don't know what "it" is, so it's hard to write a function >> that's any better than the ones we've seen. > Again, if this is addressed to the OP: I'm not his keeper. ? > If it's addressed to me: "it" means a function that will take a string > and evaluate it at runtime as if it were an f-string.? Sure, with > caveats and limitations.? And indeed Thomas Passim found this partial > solution on Stack Overflow: > def effify(non_f_str: str): > ??? return eval(f'f"""{non_f_str}"""') >> ? Using eval() to construct >> an f-string and then parse it is TERRIBLE because: >> >> 1) It still doesn't work in general, and thus has caveats like "you >> can't use this type of quote character" >> 2) You would have to pass it a dictionary of variables, which also >> can't be done with full generality >> 3) These are the exact same problems, but backwards, that led to >> f-strings in the first place >> 4) eval is extremely slow and horrifically inefficient. > I understand these limitations.? Nonetheless I can conceive that there > may be scenarios where it is an acceptable solution (perhaps the > learning tool program I suggested above). > Addressing your points specifically: > ??? 1) I believe the quote character limitation could be overcome. It > would need a fair amount of work, for which I haven't (yet) the time or > inclination. > ??? 2) Yes in general you would have to pass it one dictionary, maybe > two.? I don't see this as an insuperable obstacle.? I am not sure what > you mean by "can't be done with full generality" and perhaps that's not > important. > ??? 3) Not sure I understand this. > ??? 4) On the fairly rare occasions that I have used eval(), I can't > remember speed ever being a consideration. >> >> For some reason, str.format() isn't suitable, but *you haven't said >> why*, so we have to avoid that in our solutions. So, to come back to >> your concern: >> >>> We encourage people to ask >>> questions on this list, even though the answer will not always be what >>> they're hoping for. >> Well, yes. If you asked "how can I do X", hoping the answer would be >> "with a runtime-evaluated f-string", then you're quite right - the >> answer might not be what you were hoping for. But since you asked "how >> can I evaluate a variable as if it were an f-string", the only >> possible answer is "you can't, and that's a horrible idea". > I hope that I have shown that this is a somewhat dogmatic response. >> >> Don't ask how to use X to do Y. Ask how to do Y. > Good advice. > Best wishes > Rob Cliffe >> >> ChrisA > From torriem at gmail.com Fri Jan 27 19:34:53 2023 From: torriem at gmail.com (Michael Torrie) Date: Fri, 27 Jan 2023 17:34:53 -0700 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: On 1/25/23 19:50, Jach Feng wrote: > To me, argparse has been just a tool which I can use in a CLI app. argparse is just a tool for dealing with command-line *flags*, which are common in command-line tools. argparse interprets the command line as a bunch of flags because that's what it's designed to do. I find it baffling that you expect some other behavior from argparse. You don't need or want argparse in this situation. sys.argv has everything you need in it. Use it directly! From greg.ewing at canterbury.ac.nz Fri Jan 27 00:56:04 2023 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 27 Jan 2023 18:56:04 +1300 Subject: bool and int In-Reply-To: References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> Message-ID: On 26/01/23 6:10 am, Chris Angelico wrote: > And that's a consequence of a system wherein there is only one concept > of "success", but many concepts of "failure". Whoever devised that > system was clearly a pessimist :) Murphy's Law for Unix: If something can go wrong, it will go wrong 255 times out of 256. -- Greg From Muttley at dastardlyhq.com Fri Jan 27 11:37:38 2023 From: Muttley at dastardlyhq.com (Muttley at dastardlyhq.com) Date: Fri, 27 Jan 2023 16:37:38 -0000 (UTC) Subject: evaluation question Message-ID: Hi This is probably a dumb newbie question but I've just started to learn python3 and eval() isn't behaving as I'd expect in that it works for some things and not others. eg: >>> eval("1+1") 2 >>> eval("print(123)") 123 >>> eval("for i in range(1,10): i") Traceback (most recent call last): File "", line 1, in File "", line 1 for i in range(1,10): i ^ SyntaxError: invalid syntax Why did the 3rd one fail? Does it not handle complex expressions? Thanks for any help From dfnsonfsduifb at gmx.de Fri Jan 27 15:31:05 2023 From: dfnsonfsduifb at gmx.de (Johannes Bauer) Date: Fri, 27 Jan 2023 21:31:05 +0100 Subject: Evaluation of variable as f-string In-Reply-To: References: Message-ID: Am 23.01.23 um 19:02 schrieb Chris Angelico: >> This is supposedly for security reasons. However, when trying to emulate >> this behavior that I wanted (and know the security implications of), my >> solutions will tend to be less secure. Here is what I have been thinking >> about: > > If you really want the full power of an f-string, then you're asking > for the full power of eval(), Exactly. > and that means all the security > implications thereof, Precisely, as I had stated myself. > not to mention the difficulties of namespacing. Not an issue in my case. > Have you considered using the vanilla format() method instead? Yes. It does not provide the functionality I want. Not even the utterly trivial example that I gave. To quote myself again, let's say I have an arbitrary dictionary x (with many nested data structures), I want an expression to be evaluated that can access any members in there. x = { "y": "z" } s = "-> {x['y']}" print(s.format(x = x)) Traceback (most recent call last): File "", line 1, in KeyError: "'y'" I also want to be able to say things like {'x' * 100}, which .format() also does not do. In other words: I want the evaluation of a variable as an f-string. > But if you really REALLY know what you're doing, just use eval() > directly. I do, actually, but I hate it. Not because of the security issue, not because of namespaces, but because it does not reliably work: >>> s = "{\"x\" * 4}" >>> eval("f'" + s + "'") 'xxxx' As I mentioned, it depends on the exact quoting. Triple quotes only shift the problem. Actually replacing/escaping the relevant quotation marks is also not trivial. > I don't really see what you'd gain from an f-string. The full power of eval. > At very > least, work with a well-defined namespace and eval whatever you need > in that context. That's what I'm doing. > Maybe, rather than asking for a way to treat a string as code, ask for > what you ACTUALLY need, and we can help? I want to render data from a template using an easily understandable syntax (like an f-string), ideally using native Python. I want the template to make use of Python code constructs AND formatting (e.g. {x['time']['runtime']['seconds'] // 60:02d}). Cheers, Johannes From dfnsonfsduifb at gmx.de Fri Jan 27 15:32:00 2023 From: dfnsonfsduifb at gmx.de (Johannes Bauer) Date: Fri, 27 Jan 2023 21:32:00 +0100 Subject: Evaluation of variable as f-string In-Reply-To: References: Message-ID: Am 23.01.23 um 17:43 schrieb Stefan Ram: > Johannes Bauer writes: >> x = { "y": "z" } >> s = "-> {x['y']}" >> print(s.format(x = x)) > > x = { "y": "z" } > def s( x ): return '-> ' + x[ 'y' ] > print( s( x = x )) Except this is not at all what I asked for. The string "s" in my example is just that, an example. I want to render *arbitrary* strings "s" together with arbitrary dictionaries "x". Cheers, Johannes From dfnsonfsduifb at gmx.de Fri Jan 27 15:33:32 2023 From: dfnsonfsduifb at gmx.de (Johannes Bauer) Date: Fri, 27 Jan 2023 21:33:32 +0100 Subject: Evaluation of variable as f-string In-Reply-To: References: <5bb2546d-8c16-262d-9336-5b41cbf992b5@tompassin.net> Message-ID: Am 25.01.23 um 20:38 schrieb Thomas Passin: >> x = { "y": "z" } >> s = "-> {target}" >> print(s.format(target = x['y'])) > > Stack overflow to the rescue: No. > Search phrase:? "python evaluate string as fstring" > > https://stackoverflow.com/questions/47339121/how-do-i-convert-a-string-into-an-f-string > > def effify(non_f_str: str): > ??? return eval(f'f"""{non_f_str}"""') > > print(effify(s))? # prints as expected: "-> z" Great. s = '"""' > def effify(non_f_str: str): > return eval(f'f"""{non_f_str}"""') > > print(effify(s)) # prints as expected: "-> z" >>> print(effify(s)) Traceback (most recent call last): File "", line 1, in File "", line 2, in effify File "", line 1 f""""""""" ^ SyntaxError: unterminated triple-quoted string literal (detected at line 1) This is literally the version I described myself, except using triple quotes. It only modifies the underlying problem, but doesn't solve it. Cheers, Johannes From dfnsonfsduifb at gmx.de Fri Jan 27 15:43:09 2023 From: dfnsonfsduifb at gmx.de (Johannes Bauer) Date: Fri, 27 Jan 2023 21:43:09 +0100 Subject: Evaluation of variable as f-string In-Reply-To: References: Message-ID: Am 27.01.23 um 20:18 schrieb Chris Angelico: > All you tell us is > what you're attempting to do, which there is *no good way to achieve*. Fair enough, that is the answer. It's not possible. >> Perhaps someone will be inspired to write a function to do it. ? > > See, we don't know what "it" is, so it's hard to write a function > that's any better than the ones we've seen. Using eval() to construct > an f-string and then parse it is TERRIBLE because: > > 1) It still doesn't work in general, and thus has caveats like "you > can't use this type of quote character" Exactly my observation as well, which is why I was thinking there's something else I missed. > 2) You would have to pass it a dictionary of variables, which also > can't be done with full generality Nonsense. I only am passing a SINGLE variable to eval, called "x". That is fully general. > 3) These are the exact same problems, but backwards, that led to > f-strings in the first place I don't know what you mean by that. > 4) eval is extremely slow and horrifically inefficient. Let me worry about it. > For some reason, str.format() isn't suitable, I don't understand why you fully ignore literally the FIRST example I gave in my original post and angrily claim that you solution works when it does not: x = { "y": "z" } s = "-> {x['y']}" print(s.format(x = x)) Traceback (most recent call last): File "", line 1, in KeyError: "'y'" This. Does. Not. Work. I want to pass a single variable as a dictionary and access its members inside the expression. > but *you haven't said > why*, Yes I have, see above. > Well, yes. If you asked "how can I do X", hoping the answer would be > "with a runtime-evaluated f-string", then you're quite right - the > answer might not be what you were hoping for. But since you asked "how > can I evaluate a variable as if it were an f-string", the only > possible answer is "you can't, and that's a horrible idea". "You can't" would have been sufficient. Pity. Your judgement is unnecessary and, frankly, uncalled for as well. Multiple instances you claim that you have no idea what I am doing so how would you even begin to judge a solution as fit or unfit? > Don't ask how to use X to do Y. Ask how to do Y. You don't have to be angry that my question does not have a solution. I will manage and so might you. Cheers, Johannes From ben.usenet at bsb.me.uk Fri Jan 27 16:04:58 2023 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Fri, 27 Jan 2023 21:04:58 +0000 Subject: evaluation question References: Message-ID: <87v8krvgcl.fsf@bsb.me.uk> Muttley at dastardlyhq.com writes: > Hi It looks like you posted this question via Usenet. comp.lang.python is essentially dead as a Usenet group. It exists, and gets NNTP versions of mail sent to the mailing list, but nothing posted to the group via NNTP get send on the mailing list. I prefer Usenet and dislike mailing lists but that just means I can't really contribute to this "group" The "python-list" an an excellent resource (if you like the email interface) and you can subscribe here: https://mail.python.org/mailman/listinfo/python-list>, > This is probably a dumb newbie question but I've just started to learn > python3 and eval() isn't behaving as I'd expect in that it works for > some things and not others. eg: > >>>> eval("1+1") > 2 >>>> eval("print(123)") > 123 >>>> eval("for i in range(1,10): i") > Traceback (most recent call last): > File "", line 1, in > File "", line 1 > for i in range(1,10): i > ^ > SyntaxError: invalid syntax > > Why did the 3rd one fail? Does it not handle complex expressions? It handles only expressions, and "for i in range(1,10): i" is not an expression. You can use >>> exec("for i in range(1,10): i") or, to confirm that something is happening: >>> exec("for i in range(1,10): print(i)") 1 2 3 4 5 6 7 8 9 See: https://docs.python.org/3/library/functions.html?highlight=eval#eval and the immediately following entry. -- Ben. From auriocus at gmx.de Fri Jan 27 17:10:52 2023 From: auriocus at gmx.de (Christian Gollwitzer) Date: Fri, 27 Jan 2023 23:10:52 +0100 Subject: Evaluation of variable as f-string In-Reply-To: References: Message-ID: Am 27.01.23 um 21:43 schrieb Johannes Bauer: > I don't understand why you fully ignore literally the FIRST example I > gave in my original post and angrily claim that you solution works when > it does not: > > x = { "y": "z" } > s = "-> {x['y']}" > print(s.format(x = x)) > Traceback (most recent call last): > ? File "", line 1, in > KeyError: "'y'" > > This. Does. Not. Work. It's because "you're holding it wrong!". Notice the error message; it says that the key 'y' does not exist. (base) Apfelkiste:Abschlussmeeting chris$ ipython Python 3.8.8 (default, Apr 13 2021, 12:59:45) Type 'copyright', 'credits' or 'license' for more information IPython 7.22.0 -- An enhanced Interactive Python. Type '?' for help. In [1]: x = { "y": "z" } In [2]: s = "-> {x[y]}" In [3]: print(s.format(x = x)) -> z In [4]: Christian From nntp.mbourne at spamgourmet.com Fri Jan 27 17:14:57 2023 From: nntp.mbourne at spamgourmet.com (Mark Bourne) Date: Fri, 27 Jan 2023 22:14:57 +0000 Subject: bool and int In-Reply-To: References: <001901d9312b$ef077b60$cd167220$@gmail.com> <007801d931fc$c8a97a50$59fc6ef0$@gmail.com> Message-ID: avi.e.gross at gmail.com wrote: ... > Interestingly, I wonder if anyone has > designed an alternate object type that can be used mostly in place of > Booleans but which imposes changes and restrictions so trying to add a > Boolean to an integer, or vice versa, results in an error. Python is > flexible enough to do that and perhaps there already is a module out there Not exactly what you describe, but I did once write a subclass of int for use in an SNMP interface, where true is represented as 1 and false as 2. I don't recall the exact details offhand, but it involved overriding __bool__ so that a value of 1 returned True and anything else False. The way it was used should have ensured that it only ever had the value 1 or 2, but doing it this way round (rather than e.g. 2 being False and anything else True) meant that if it somehow got the value 0, that would also be treated as False. I think it also overrode __init__ (or perhaps __new__) to covert a bool True or False to 1 or 2 (rather than 1 or 0) for its own value, so it could be initialised from either an int or a bool and correctly converted in either direction via int() or bool(). So Python is even flexible enough to be made to deal with insane situations where False is 2! -- Mark. From rosuav at gmail.com Fri Jan 27 19:46:59 2023 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 28 Jan 2023 11:46:59 +1100 Subject: evaluation question In-Reply-To: References: Message-ID: On Sat, 28 Jan 2023 at 11:45, wrote: > > Hi > > This is probably a dumb newbie question but I've just started to learn > python3 and eval() isn't behaving as I'd expect in that it works for > some things and not others. eg: > > >>> eval("1+1") > 2 > >>> eval("print(123)") > 123 > >>> eval("for i in range(1,10): i") > Traceback (most recent call last): > File "", line 1, in > File "", line 1 > for i in range(1,10): i > ^ > SyntaxError: invalid syntax > > Why did the 3rd one fail? Does it not handle complex expressions? > There's a difference between *expressions* (which have values) and *statements* (which do stuff, including control flow like loops). You may want the exec function instead. ChrisA From rosuav at gmail.com Fri Jan 27 19:49:06 2023 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 28 Jan 2023 11:49:06 +1100 Subject: bool and int In-Reply-To: References: <6db0368a-4d29-4a47-5657-05a875ee2df7@mrabarnett.plus.com> Message-ID: On Sat, 28 Jan 2023 at 11:45, Greg Ewing wrote: > > On 26/01/23 6:10 am, Chris Angelico wrote: > > And that's a consequence of a system wherein there is only one concept > > of "success", but many concepts of "failure". Whoever devised that > > system was clearly a pessimist :) > > Murphy's Law for Unix: If something can go wrong, it will go > wrong 255 times out of 256. > Murphy's Law for people working with small integers: "255 out of 256" will, just when you least expect it, change into "255 out of 0". ChrisA From PythonList at DancesWithMice.info Fri Jan 27 20:22:01 2023 From: PythonList at DancesWithMice.info (dn) Date: Sat, 28 Jan 2023 14:22:01 +1300 Subject: evaluation question In-Reply-To: References: Message-ID: On 28/01/2023 05.37, Muttley at dastardlyhq.com wrote: > This is probably a dumb newbie question but I've just started to learn > python3 and eval() isn't behaving as I'd expect in that it works for > some things and not others. eg: > >>>> eval("1+1") > 2 >>>> eval("print(123)") > 123 >>>> eval("for i in range(1,10): i") > Traceback (most recent call last): > File "", line 1, in > File "", line 1 > for i in range(1,10): i > ^ > SyntaxError: invalid syntax > > Why did the 3rd one fail? Does it not handle complex expressions? eval() is very powerful, and therefore rather dangerous in the risks it presents. Thus, seems a strange/advanced question for a "newbie" to be asking. YMMV! Do you know about the Python REPL? If you open python within a terminal, each of the three expressions/compound-statements listed will work, as desired, without eval(). dn $ ... python Python 3.11.1 (main, Jan 6 2023, 00:00:00) [GCC 12.2.1 20221121 (Red Hat 12.2.1-4)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> 1+1 2 >>> print( 123 ) 123 >>> for i in range( 1, 10 ): i ... 1 2 3 4 5 6 7 8 9 >>> exit() -- Regards, =dn From list1 at tompassin.net Fri Jan 27 20:51:29 2023 From: list1 at tompassin.net (Thomas Passin) Date: Fri, 27 Jan 2023 20:51:29 -0500 Subject: Evaluation of variable as f-string In-Reply-To: References: <5bb2546d-8c16-262d-9336-5b41cbf992b5@tompassin.net> Message-ID: <9db7a5e1-076c-a9aa-aee4-e044e45decb7@tompassin.net> On 1/27/2023 3:33 PM, Johannes Bauer wrote: > Am 25.01.23 um 20:38 schrieb Thomas Passin: > >>> x = { "y": "z" } >>> s = "-> {target}" >>> print(s.format(target = x['y'])) >> >> Stack overflow to the rescue: > > No. > >> Search phrase:? "python evaluate string as fstring" >> >> https://stackoverflow.com/questions/47339121/how-do-i-convert-a-string-into-an-f-string >> >> def effify(non_f_str: str): >> ???? return eval(f'f"""{non_f_str}"""') >> >> print(effify(s))? # prints as expected: "-> z" > > Great. > > s = '"""' > > > def effify(non_f_str: str): > >????? return eval(f'f"""{non_f_str}"""') > > > > print(effify(s))? # prints as expected: "-> z" > > >>> print(effify(s)) > Traceback (most recent call last): > ? File "", line 1, in > ? File "", line 2, in effify > ? File "", line 1 > ??? f""""""""" > ?????????? ^ > SyntaxError: unterminated triple-quoted string literal (detected at line 1) > > This is literally the version I described myself, except using triple > quotes. It only modifies the underlying problem, but doesn't solve it. Ok, so now we are in the territory of "Tell us what you are trying to accomplish". And part of that is why you cannot put some constraints on what your string fragments are. The example I gave, copied out of your earlier message, worked and now you are springing triple quotes on us. Stop with the rock management already and explain (briefly if possible) what you are up to. From list1 at tompassin.net Fri Jan 27 20:56:49 2023 From: list1 at tompassin.net (Thomas Passin) Date: Fri, 27 Jan 2023 20:56:49 -0500 Subject: Evaluation of variable as f-string In-Reply-To: References: Message-ID: On 1/27/2023 5:10 PM, Christian Gollwitzer wrote: > Am 27.01.23 um 21:43 schrieb Johannes Bauer: >> I don't understand why you fully ignore literally the FIRST example I >> gave in my original post and angrily claim that you solution works >> when it does not: >> >> x = { "y": "z" } >> s = "-> {x['y']}" >> print(s.format(x = x)) >> Traceback (most recent call last): >> ?? File "", line 1, in >> KeyError: "'y'" >> >> This. Does. Not. Work. > > It's because "you're holding it wrong!". Notice the error message; it > says that the key 'y' does not exist. > > > (base) Apfelkiste:Abschlussmeeting chris$ ipython > Python 3.8.8 (default, Apr 13 2021, 12:59:45) > Type 'copyright', 'credits' or 'license' for more information > IPython 7.22.0 -- An enhanced Interactive Python. Type '?' for help. > > In [1]: x = { "y": "z" } > > In [2]: s = "-> {x[y]}" > > In [3]: print(s.format(x = x)) > -> z > > In [4]: > > ????Christian Oops, that's not quite what he wrote. You: s = "-> {x[y]}" # Works Him: s = "-> {x['y']}" # Fails From avi.e.gross at gmail.com Sat Jan 28 00:19:21 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Sat, 28 Jan 2023 00:19:21 -0500 Subject: Evaluation of variable as f-string In-Reply-To: References: Message-ID: <00c701d932d8$145d8be0$3d18a3a0$@gmail.com> May I point out that some dynamic situations can in a sense be normalized? The example below posits a dynamically allocated dictionary during run time. But why can't you have a placeholder variable name in place and make your placeholder a link to the dictionary (or other item) before invoking the existing f-string with the placeholder built-in, rather than trying to evaluate an F? ??? Of course many situations may not have as much of a possible work-around. But as so many have noted, we never got a really good explanation of what the OP really wants to do. There have been replies that may be suitable solutions and some clearly have potential to be security holes if you let the users dynamically create strings to be evaluated. In some languages, many of the facets of the language can be passed along as a function with some name to be used in functional programming techniques and this can be very useful. The "operator" module can be used for quite a few things like operator.add or operator.__add__ or operator.concat and many more. If the logic used to evaluate an f-string (and for that matter the other string variants like b'..' and r'...') could be encapsulated in a function like that, it would be potentially usable as in passing something like dangerous_operator.f_string and a list of strings and having that return a list of evaluated strings. The fact that something like this is not known to the people here may hint that it is not something considered safe to use by amateurs. But then again, anyone who wants to can use eval() as Chris points out. Of course, there may be other reasons too. An f-string is evaluated in a context that may be different if a string is passed along and then looked at in another context. -----Original Message----- From: Python-list On Behalf Of Stefan Ram Sent: Friday, January 27, 2023 4:31 PM To: python-list at python.org Subject: Re: Evaluation of variable as f-string Johannes Bauer writes: >>Johannes Bauer writes: >>>x = { "y": "z" } >>>s = "-> {x['y']}" >>>print(s.format(x = x)) >>x = { "y": "z" } >>def s( x ): return '-> ' + x[ 'y' ] >>print( s( x = x )) >Except this is not at all what I asked for. The string "s" in my >example is just that, an example. I want to render *arbitrary* strings "s" >together with arbitrary dictionaries "x". I take this to mean that you want to process a dictionary name, a string and a dictionary that is only specified as late as at run time. import string name = input( 'name of the dictionary? ' ) string_ = input( 'string? ' ) # "-> {x['y']}" dictionary = eval( input( 'dictionary? ' )) print( eval( 'f"""' + string_ + '"""', {name:dictionary} )) name of the dictionary? x string? -> {x['y']} dictionary? { 'y': 'z' } -> z -- https://mail.python.org/mailman/listinfo/python-list From grant.b.edwards at gmail.com Sat Jan 28 00:35:11 2023 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Fri, 27 Jan 2023 21:35:11 -0800 (PST) Subject: bool and int References: <001901d9312b$ef077b60$cd167220$@gmail.com> <007801d931fc$c8a97a50$59fc6ef0$@gmail.com> Message-ID: <63d4b40f.020a0220.ef1d.4c85@mx.google.com> On 2023-01-27, Mark Bourne wrote: > So Python is even flexible enough to be made to deal with insane > situations where False is 2! IIRC, in VMS DCL even numbers were false and odd numbers were true. In Unix shells, a return code of 0 is true and non-0 is false. Though that's not really the same environment that Python lives in... -- Grant From hjp-python at hjp.at Sat Jan 28 08:28:47 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sat, 28 Jan 2023 14:28:47 +0100 Subject: Evaluation of variable as f-string In-Reply-To: References: Message-ID: <20230128132847.fdc4d3hx7m7yedi4@hjp.at> On 2023-01-27 20:56:49 -0500, Thomas Passin wrote: > On 1/27/2023 5:10 PM, Christian Gollwitzer wrote: > > Am 27.01.23 um 21:43 schrieb Johannes Bauer: > > > x = { "y": "z" } > > > s = "-> {x['y']}" > > > print(s.format(x = x)) > > > Traceback (most recent call last): > > > ?? File "", line 1, in > > > KeyError: "'y'" > > > > > > This. Does. Not. Work. > > > > It's because "you're holding it wrong!". Notice the error message; it ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > says that the key 'y' does not exist. [...] > > In [1]: x = { "y": "z" } > > In [2]: s = "-> {x[y]}" > > In [3]: print(s.format(x = x)) > > -> z > > In [4]: > > Oops, that's not quite what he wrote. > > You: s = "-> {x[y]}" # Works > Him: s = "-> {x['y']}" # Fails That was the point. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From hjp-python at hjp.at Sat Jan 28 08:41:13 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sat, 28 Jan 2023 14:41:13 +0100 Subject: Evaluation of variable as f-string In-Reply-To: References: Message-ID: <20230128134113.l3awsy2dcflzeets@hjp.at> On 2023-01-27 21:43:09 +0100, Johannes Bauer wrote: > x = { "y": "z" } > s = "-> {x['y']}" > print(s.format(x = x)) > Traceback (most recent call last): > File "", line 1, in > KeyError: "'y'" > > This. Does. Not. Work. > > I want to pass a single variable as a dictionary and access its members > inside the expression. You can do that (see other responses), but you can't have arbitrary Python expressions inside a format string. It works with f-strings because f-strings are a compiler construct. The f-string never exists at run-time. Instead the compiler transforms f"-> {x['y']}" into the equivalent of "-> " + format_value(x["y"]) + "" So either you need to pass it to the Python compiler (via eval), or you need to implement enough of a Python parser/interpreter to cover the cases you are interested in. The latter might be an interesting exercise, but I would suggest looking at existing template engines like Jinja2 for production purposes. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From hjp-python at hjp.at Sat Jan 28 09:22:18 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sat, 28 Jan 2023 15:22:18 +0100 Subject: Evaluation of variable as f-string In-Reply-To: References: Message-ID: <20230128142218.atkybkdf7elzqrky@hjp.at> On 2023-01-27 21:31:05 +0100, Johannes Bauer wrote: > > But if you really REALLY know what you're doing, just use eval() > > directly. > > I do, actually, but I hate it. Not because of the security issue, not > because of namespaces, but because it does not reliably work: > > >>> s = "{\"x\" * 4}" > >>> eval("f'" + s + "'") > 'xxxx' That's exactly the result I expected. What did you expect? hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From hjp-python at hjp.at Sat Jan 28 09:30:59 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sat, 28 Jan 2023 15:30:59 +0100 Subject: Usenet vs. Mailing-list (was: evaluation question) In-Reply-To: <87v8krvgcl.fsf@bsb.me.uk> References: <87v8krvgcl.fsf@bsb.me.uk> Message-ID: <20230128143059.cbigz4ad4uklqtbk@hjp.at> On 2023-01-27 21:04:58 +0000, Ben Bacarisse wrote: > Muttley at dastardlyhq.com writes: > > > Hi > > It looks like you posted this question via Usenet. comp.lang.python is > essentially dead as a Usenet group. It exists, and gets NNTP versions > of mail sent to the mailing list, but nothing posted to the group via > NNTP get send on the mailing list. This is wrong. I did get Muttley's any your postings via the mailing-list. (I also get Stefan Ram's postings despite his "no distribution outside of Usenet" header). hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From avi.e.gross at gmail.com Sat Jan 28 12:31:48 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Sat, 28 Jan 2023 12:31:48 -0500 Subject: logically Boolean Message-ID: <007a01d9333e$6695e9f0$33c1bdd0$@gmail.com> The topic has somewhat modified to asking what is a BOOLEAN. The right answer is that the question is too simple. It is what YOU want it to be in your situation. And it is also what the LANGUAGE designers and implementers have chosen within their domain. Mathematically, as part of classical logic, there is a concept of having two sets with the union of the two being a null set. The sets are disjoint and in some cases, cover all possibilities and in other cases the possibilities are narrowed so that just two distinct groupings remain. Using 0 and 1 makes sense for a binary bit, albeit in either order for True versus False. No other numbers can be stored in a single binary digit so this is a trivial but useful use of Booleans in some languages. In hardware or storage mediums, there are not literal zeroes or ones. You can have a high or low value or a magnetic field or spin or whatever you want to implement it, and sometimes it fails anyway and you get indeterminate values or a bit seems to flip. That is an implementation detail but Boolean logic remains a mathematical concept. UNIX programs did not return a Boolean value on exit so that discussion is a tad off the point. They returned a STATUS of sorts. Mostly the status was zero for all forms of success and you could signal many forms of failure using any other number. But I can imagine designing a program like grep to return 0 if nothing was found but the program did not fail, and a positive number up to some limit that specified how many lines matched and perhaps -1 or other numbers on failure. A return status need not be seen as Boolean, albeit in my made-up example, any non-zero exit status up to the limit would be in the set of SUCCESS and -1 or whatever would be in the set for FAILURE. Consider how a Python function can return OR throw an error. Many different errors can be thrown and some propagate up from deeper levels if not caught and handled. Generally success simply means no errors BUT you can also design a function that throws various success "errors" and indirectly returns one of several kinds of values in the payload, as well as errors meant to be treated as bad. Heck, you can have some errors be warnings and the caller of the function may have to call it from a "try" and specify what to catch and how to deal with it. I repeat, not everything must be Boolean when it comes to status types of things. As noted, many other schemes have been used to represent a Boolean variable. And given some implementations, something emerges that can confuse some people. Grant mentioned even/odd. On most machines this simply means you can safely ignore a longer clustering of bits such as with a 32-bit integer and simply examine the least significant bit which determines if it is even or odd. Heck, you can even store other useful info in the remaining bits, if you wish. As has been said repeatedly, it does not matter what the OP expects, as many implementations are possible including some that do not follow the above suggestions. Much of my work has included other possibilities often using data that is not always able to be analyzed in a truly Boolean fashion. Modern Programming languages have to grapple with items that look like they can have three or more values. I mean you can have a variable that can be TRUE or FALSE but also one of many forms of Not Available or more. In a language like R, their version of Boolean can have a value of NA but actually in some contexts you have to specify which kind of NA such as NA_integer_ or NA_character_ so obviously a vector of Booleans cannot be implemented using a single bit for each. I have seen packages that can import data with many kinds of missing values as there are other programs that allow you to tag missing values to mean things like "missing: child did not show up to take test" or "missing: test not completed, will re-test" or "missing: caught cheating" or "missing: identity of test taker not verified" and so on. I helped with one such package to create a type within R that does not discard the other info so it supports multiple NA variants while at the same time it can be processed as if every NA was the same. Python has concepts along the same lines such as np.nan so there has to be a way of storing data with perhaps a wrapper around things that can specify if the contents are what is expected, or maybe something else. And I won't get into fuzzy logic. But just plain classical Boolean logic is a mathematical concept that can be expressed many ways. Some of the ways in python look compatible with integers but that was a design choice. If they had chosen to store a character containing either "T" or "F" then perhaps they would allow Booleans to be treated as characters and let them be concatenated to strings and so on. -----Original Message----- From: Python-list > On Behalf Of Grant Edwards Sent: Saturday, January 28, 2023 12:35 AM To: python-list at python.org Subject: Re: bool and int On 2023-01-27, Mark Bourne > wrote: > So Python is even flexible enough to be made to deal with insane > situations where False is 2! IIRC, in VMS DCL even numbers were false and odd numbers were true. In Unix shells, a return code of 0 is true and non-0 is false. Though that's not really the same environment that Python lives in... -- Grant -- https://mail.python.org/mailman/listinfo/python-list From gweatherby at uchc.edu Sat Jan 28 14:40:04 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Sat, 28 Jan 2023 19:40:04 +0000 Subject: Unix / linux programs In-Reply-To: <007a01d9333e$6695e9f0$33c1bdd0$@gmail.com> References: <007a01d9333e$6695e9f0$33c1bdd0$@gmail.com> Message-ID: The Unix convention is 0 means everything went well, and non-zero means something else happened. Here?s a contrived example of a bash wrapper around GNU tar. By contrived I mean it works but I would not use it in practice ? I?d just use tar directly or use the Python tarfile module if I wanted finer grain control of the process. #!/bin/bash if [ $# -lt 2 ]; then echo "Usage: $0 [tar] [directory to compare]" fi tar --diff -f $1 $2 2>/dev/null r=$? if [ $r -eq 0 ]; then echo $1 has $2 in it, unmodified elif [ $r -eq 1 ]; then echo $1 does not have $2 in it unmodified elif [ $r -eq 2 ]; then # maybe do more tests here? echo There is a problem with $1 or with $2 else echo "Other error $r" fi Incidentally, I found the man page installed on Ubuntu20.04 for tar is out of date. tar returns 64 if doesn?t like the command line arguments. Nothing works until it is tested. From gweatherby at uchc.edu Sat Jan 28 14:44:03 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Sat, 28 Jan 2023 19:44:03 +0000 Subject: Non int Boolean In-Reply-To: <007a01d9333e$6695e9f0$33c1bdd0$@gmail.com> References: <007a01d9333e$6695e9f0$33c1bdd0$@gmail.com> Message-ID: If someone really really wants a non-int Boolean, it is easy to implement. 5 or 6 lines, depending on whether you count the import statement: from enum import Enum class MyBool(Enum): TRUE = 42 FALSE = 54 def __bool__(self): return self == MyBool.TRUE # # testing # mytrue = MyBool.TRUE try: print(int(mytrue)) #this fails except TypeError as te: print(te) asbool = bool(mytrue) if mytrue: print("yep") myfalse = MyBool.FALSE if myfalse: print("nope") --- I would never use such a thing, and I would be annoyed if I can across code that did. As has been said (beaten to death?) Python has an existing well-understood boolean type. From rosuav at gmail.com Sat Jan 28 15:28:20 2023 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 29 Jan 2023 07:28:20 +1100 Subject: logically Boolean In-Reply-To: <007a01d9333e$6695e9f0$33c1bdd0$@gmail.com> References: <007a01d9333e$6695e9f0$33c1bdd0$@gmail.com> Message-ID: On Sun, 29 Jan 2023 at 04:33, wrote: > The right answer is that the question is too simple. It is what YOU want it > to be in your situation. And it is also what the LANGUAGE designers and > implementers have chosen within their domain. The REAL boolean is the friends we made along the way? ChrisA From PythonList at DancesWithMice.info Sat Jan 28 16:07:26 2023 From: PythonList at DancesWithMice.info (dn) Date: Sun, 29 Jan 2023 10:07:26 +1300 Subject: logically Boolean In-Reply-To: References: <007a01d9333e$6695e9f0$33c1bdd0$@gmail.com> Message-ID: On 29/01/2023 09.28, Chris Angelico wrote: > The REAL boolean is the friends we made along the way? By REAL did you mean float - True or False? (for the FORTRAN-free: https://fortranwiki.org/fortran/show/real) -- -- Regards, =dn From dino at no.spam.ar Fri Jan 27 22:38:25 2023 From: dino at no.spam.ar (Dino) Date: Fri, 27 Jan 2023 22:38:25 -0500 Subject: bool and int In-Reply-To: References: <001901d9312b$ef077b60$cd167220$@gmail.com> <007801d931fc$c8a97a50$59fc6ef0$@gmail.com> Message-ID: you have your reasons, and I was tempted to stop there, but... I have to pick this... On 1/26/2023 10:09 PM, avi.e.gross at gmail.com wrote: > You can often borrow > ideas and code from an online search and hopefully cobble "a" solution > together that works well enough. Of course it may suddenly fall apart. also carefully designed systems that are the work of experts may suddenly fall apart. Thank you for all the time you have used to address the points I raised. It was interesting reading. Dino From jfong at ms4.hinet.net Sat Jan 28 00:03:39 2023 From: jfong at ms4.hinet.net (Jach Feng) Date: Fri, 27 Jan 2023 21:03:39 -0800 (PST) Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: <6bdc1904-4f0b-4c05-a67d-b0965b30624dn@googlegroups.com> Jach Feng ? 2023?1?22? ?????11:11:22 [UTC+8] ?????? > Fail on command line, > > e:\Works\Python>py infix2postfix.py "-4^2+5.3*abs(-2-1)/2" > usage: infix2postfix.py [-h] [infix] > infix2postfix.py: error: unrecognized arguments: -4^2+5.3*abs(-2-1)/2 > > Also fail in REPL, > > e:\Works\Python>py > Python 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:08:11) [MSC v.1928 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import argparse > >>> parser = argparse.ArgumentParser(description='Convert infix notation to postfix') > >>> parser.parse_args("-4^2+5.3*abs(-2-1)/2") > usage: [-h] > : error: unrecognized arguments: - 4 ^ 2 + 5 . 3 * a b s ( - 2 - 1 ) / 2 > > Just can't figure out where is wrong!? > > --Jach I have to admit that I don't know the background upon which the argparse was built. The good side is that I don't carry its historical knowledge ( or burden?), that's why I can use it in a new way which may make someone feel uneasy. The reason I am still keep on using argparse on this "one positional argument only" CLI app is that I can't see what advantage I can get when writing code to handling sys.argv directly for the following two situations, ----- e:\Works\Python>py infix2postfix.py usage: infix2postfix.py [-h] infix infix2postfix.py: error: the following arguments are required: infix e:\Works\Python>py infix2postfix.py -h usage: infix2postfix.py [-h] infix Convert infix notation to postfix positional arguments: infix Put equations in quote if there is space in it and separate each one with a comma, ie. "-4^2+5.3*abs(-2-1)/2, abs(Abc)*(B+C)/D, (-3) * sqrt(1-(x1/7)*(y1/7)) * sqrt(abs((x0-4.5)/(y0-4)))" optional arguments: -h, --help show this help message and exit ----- comparing with code using the argparse below, import argparse sample = "-4^2+5.3*abs(-2-1)/2, abs(Abc)*(B+C)/D, (-3) * sqrt(1-(x1/7)*(y1/7)) * sqrt(abs((x0-4.5)/(y0-4)))" parser = argparse.ArgumentParser(description='Convert infix notation to postfix') parser.add_argument('infix', help='Put equations in quote if there is space in it and separate each one with a comma, ie. "{}"'.format(sample)) import sys if len(sys.argv) > 1 and not '-h' in sys.argv: ....sys.argv.insert(1, '--') args = parser.parse_args() From bowman at montana.com Sat Jan 28 01:35:04 2023 From: bowman at montana.com (rbowman) Date: 28 Jan 2023 06:35:04 GMT Subject: bool and int References: <001901d9312b$ef077b60$cd167220$@gmail.com> <007801d931fc$c8a97a50$59fc6ef0$@gmail.com> <63d4b40f.020a0220.ef1d.4c85@mx.google.com> Message-ID: On Fri, 27 Jan 2023 21:35:11 -0800 (PST), Grant Edwards wrote: > In Unix shells, a return code of 0 is true and non-0 is false. That carries over to some C functions like strcmp() although it's more complex. strcmp() returns the value of subtracting the nth character of string b from string a if the value is not 0. For matching strings, the result is 0 for all character positions. This plays nicely with sorting functions but naive programmers assume it's a boolean and strcmp("foo", "foo") should return 1 or true when the strings match. Returning 0 for success gives you much more latitude in return values. From Muttley at dastardlyhq.com Sat Jan 28 05:11:38 2023 From: Muttley at dastardlyhq.com (Muttley at dastardlyhq.com) Date: Sat, 28 Jan 2023 10:11:38 -0000 (UTC) Subject: evaluation question References: <87v8krvgcl.fsf@bsb.me.uk> Message-ID: On Fri, 27 Jan 2023 21:04:58 +0000 Ben Bacarisse wrote: >Muttley at dastardlyhq.com writes: > >> Hi > >It looks like you posted this question via Usenet. comp.lang.python is >essentially dead as a Usenet group. It exists, and gets NNTP versions >of mail sent to the mailing list, but nothing posted to the group via >NNTP get send on the mailing list. I prefer Usenet and dislike mailing >lists but that just means I can't really contribute to this "group" > >The "python-list" an an excellent resource (if you like the email >interface) and you can subscribe here: > >https://mail.python.org/mailman/listinfo/python-list>, > >> This is probably a dumb newbie question but I've just started to learn >> python3 and eval() isn't behaving as I'd expect in that it works for >> some things and not others. eg: >> >>>>> eval("1+1") >> 2 >>>>> eval("print(123)") >> 123 >>>>> eval("for i in range(1,10): i") >> Traceback (most recent call last): >> File "", line 1, in >> File "", line 1 >> for i in range(1,10): i >> ^ >> SyntaxError: invalid syntax >> >> Why did the 3rd one fail? Does it not handle complex expressions? > >It handles only expressions, and "for i in range(1,10): i" is not an >expression. You can use > >>>> exec("for i in range(1,10): i") Ok, thanks. From Muttley at dastardlyhq.com Sat Jan 28 05:13:00 2023 From: Muttley at dastardlyhq.com (Muttley at dastardlyhq.com) Date: Sat, 28 Jan 2023 10:13:00 -0000 (UTC) Subject: evaluation question References: Message-ID: On Sat, 28 Jan 2023 14:22:01 +1300 dn wrote: >On 28/01/2023 05.37, Muttley at dastardlyhq.com wrote: >> This is probably a dumb newbie question but I've just started to learn >> python3 and eval() isn't behaving as I'd expect in that it works for >> some things and not others. eg: >> >>>>> eval("1+1") >> 2 >>>>> eval("print(123)") >> 123 >>>>> eval("for i in range(1,10): i") >> Traceback (most recent call last): >> File "", line 1, in >> File "", line 1 >> for i in range(1,10): i >> ^ >> SyntaxError: invalid syntax >> >> Why did the 3rd one fail? Does it not handle complex expressions? > >eval() is very powerful, and therefore rather dangerous in the risks it >presents. > >Thus, seems a strange/advanced question for a "newbie" to be asking. YMMV! Well ok, new-ish :) >Do you know about the Python REPL? Haven't learnt the acronyms yet. >If you open python within a terminal, each of the three >expressions/compound-statements listed will work, as desired, without >eval(). Umm, yeah, thats kind of obvious isn't it? From lkrupp at invalid.pssw.com.invalid Sat Jan 28 07:18:50 2023 From: lkrupp at invalid.pssw.com.invalid (Louis Krupp) Date: Sat, 28 Jan 2023 05:18:50 -0700 Subject: evaluation question In-Reply-To: References: Message-ID: On 1/27/2023 9:37 AM, Muttley at dastardlyhq.com wrote: > Hi > > This is probably a dumb newbie question but I've just started to learn > python3 and eval() isn't behaving as I'd expect in that it works for > some things and not others. eg: > >>>> eval("1+1") > 2 >>>> eval("print(123)") > 123 >>>> eval("for i in range(1,10): i") > Traceback (most recent call last): > File "", line 1, in > File "", line 1 > for i in range(1,10): i > ^ > SyntaxError: invalid syntax > > Why did the 3rd one fail? Does it not handle complex expressions? > > Thanks for any help > > > This might -- or might not -- be useful: eval( "print( [i for i in range(1, 10)] )" ) It prints a list, but you probably knew that: [1, 2, 3, 4, 5, 6, 7, 8, 9] Louis From nntp.mbourne at spamgourmet.com Sat Jan 28 08:59:41 2023 From: nntp.mbourne at spamgourmet.com (Mark Bourne) Date: Sat, 28 Jan 2023 13:59:41 +0000 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <6bdc1904-4f0b-4c05-a67d-b0965b30624dn@googlegroups.com> References: <6bdc1904-4f0b-4c05-a67d-b0965b30624dn@googlegroups.com> Message-ID: Jach Feng wrote: > Jach Feng ? 2023?1?22? ?????11:11:22 [UTC+8] ?????? >> Fail on command line, >> >> e:\Works\Python>py infix2postfix.py "-4^2+5.3*abs(-2-1)/2" >> usage: infix2postfix.py [-h] [infix] >> infix2postfix.py: error: unrecognized arguments: -4^2+5.3*abs(-2-1)/2 >> >> Also fail in REPL, >> >> e:\Works\Python>py >> Python 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:08:11) [MSC v.1928 32 bit (Intel)] on win32 >> Type "help", "copyright", "credits" or "license" for more information. >>>>> import argparse >>>>> parser = argparse.ArgumentParser(description='Convert infix notation to postfix') >>>>> parser.parse_args("-4^2+5.3*abs(-2-1)/2") >> usage: [-h] >> : error: unrecognized arguments: - 4 ^ 2 + 5 . 3 * a b s ( - 2 - 1 ) / 2 >> >> Just can't figure out where is wrong!? >> >> --Jach > I have to admit that I don't know the background upon which the argparse was built. The good side is that I don't carry its historical knowledge ( or burden?), that's why I can use it in a new way which may make someone feel uneasy. If you can use it in the way you want, that's great. I thought you were asking here because you *couldn't* use it the way you want. You're writing a command-line application. Your users are either already familiar with the conventions of command-line applications, or they'll soon need to be. If your application supports options beginning with a "-" (which is what argparse gives you, even if only the default "-h" and "--help" options are actually valid), and you also need to be able to pass positional arguments which begin with a "-", you need some way for the user to indicate whether any particular argument is an option or positional. Using "--" to indicate that all subsequent arguments are positional, not options, is a common convention. Some of your users might already be familiar with that convention from other command-line tools. By using the same convention, you'd be making it easier for them to use yours. Others might not already be familiar with the convention, but by being consistent with other tools you'd still be helping them when they eventually do come across it elsewhere. You can always include an explanation of using "--" in your usage output or other documentation. Apart from dealing with how to pass an argument beginning with "-", your users will also likely have to deal with the features of whichever shell they're using, which you have no control over. For example, it's quite common to need to enclose an argument in quotes if it contains spaces. It may also be necessary to use quotes if certain special characters are used, such as "*", "?" or "$" (perhaps "%" on Windows). > The reason I am still keep on using argparse on this "one positional argument only" CLI app is that I can't see what advantage I can get when writing code to handling sys.argv directly for the following two situations, > ----- > e:\Works\Python>py infix2postfix.py > usage: infix2postfix.py [-h] infix > infix2postfix.py: error: the following arguments are required: infix > > e:\Works\Python>py infix2postfix.py -h > usage: infix2postfix.py [-h] infix > > Convert infix notation to postfix > > positional arguments: > infix Put equations in quote if there is space in it and separate each one with a comma, ie. > "-4^2+5.3*abs(-2-1)/2, abs(Abc)*(B+C)/D, (-3) * sqrt(1-(x1/7)*(y1/7)) * sqrt(abs((x0-4.5)/(y0-4)))" > > optional arguments: > -h, --help show this help message and exit > ----- > > comparing with code using the argparse below, > > import argparse > sample = "-4^2+5.3*abs(-2-1)/2, abs(Abc)*(B+C)/D, (-3) * sqrt(1-(x1/7)*(y1/7)) * sqrt(abs((x0-4.5)/(y0-4)))" > parser = argparse.ArgumentParser(description='Convert infix notation to postfix') > parser.add_argument('infix', > help='Put equations in quote if there is space in it and separate each one with a comma, ie. "{}"'.format(sample)) > > import sys > if len(sys.argv) > 1 and not '-h' in sys.argv: > ....sys.argv.insert(1, '--') > args = parser.parse_args() Personally, I do quite often use argparse even for simple cases, partly because it produces nice usage details, and deals with wrapping the output to whatever width the console is. But I work with it rather than against it, and accept that a "--" is needed if a positional argument starts with a "-". I notice you explain the need to enclose the equation in quotes if it contains spaces. That's not even a feature of your application, but of the shell used to call it. So why so much objection to explaining the need for "--"? Depending on the shell, there are other cases where quoting might be needed, e.g. if the equation includes a "*" or "?" and happens to look like a pattern matching files in the current directory (most shells I've used pass the pattern unchanged if it doesn't match any files). In bash, if a "$" is used I'd need to enclose that in 'single quotes' (can't even use "double quotes" for that one). You can't really expect to document all that sort of thing, because it depends on which shell the user happens to run your application from - you just have to trust the user to know or learn how to use their shell. -- Mark. From nntp.mbourne at spamgourmet.com Sat Jan 28 09:39:27 2023 From: nntp.mbourne at spamgourmet.com (Mark Bourne) Date: Sat, 28 Jan 2023 14:39:27 +0000 Subject: evaluation question In-Reply-To: References: Message-ID: Muttley at dastardlyhq.com wrote: > On Sat, 28 Jan 2023 14:22:01 +1300 > dn wrote: >> Do you know about the Python REPL? > > Haven't learnt the acronyms yet. REPL stands for "Read Evaluate Print Loop". It basically refers to the interactive interpreter, which reads input you type, evaluates it, prints the result, and loops (repeatedly does that). An interesting point from your examples is that the output from the first two comes from different steps in that loop. >>> eval("1+1") 2 Here, the E (evaluation) step runs eval("1+1"), which returns 2. The P (print) step then prints that result. If this was in a script, you wouldn't see any output, and the statement is pretty much useless - you'd need to assign the result to a variable or explicitly print it. >>> eval("print(123)") 123 Here, the E step runs eval("print(123)"), which prints 123 and returns None. The P step doesn't print anything if the result is None. You'd still see that output if this was in a script. Using eval in those examples is pretty pointless, since: >>> 1+1 >>> print(123) would produce the same results - but of course they were just simple examples. -- Mark. From jon+usenet at unequivocal.eu Sat Jan 28 13:10:00 2023 From: jon+usenet at unequivocal.eu (Jon Ribbens) Date: Sat, 28 Jan 2023 18:10:00 -0000 (UTC) Subject: Usenet vs. Mailing-list (was: evaluation question) References: <87v8krvgcl.fsf@bsb.me.uk> <20230128143059.cbigz4ad4uklqtbk@hjp.at> Message-ID: On 2023-01-28, Peter J. Holzer wrote: > On 2023-01-27 21:04:58 +0000, Ben Bacarisse wrote: >> It looks like you posted this question via Usenet. comp.lang.python is >> essentially dead as a Usenet group. It exists, and gets NNTP versions >> of mail sent to the mailing list, but nothing posted to the group via >> NNTP get send on the mailing list. > > This is wrong. I did get Muttley's any your postings via the > mailing-list. Yes, it's certainly false. I only ever post via the newsgroup, and I can see my postings reach the list because they appear in the list archive on the web. https://mail.python.org/pipermail/python-list/ From dfnsonfsduifb at gmx.de Sat Jan 28 14:50:52 2023 From: dfnsonfsduifb at gmx.de (Johannes Bauer) Date: Sat, 28 Jan 2023 20:50:52 +0100 Subject: Evaluation of variable as f-string In-Reply-To: References: <5bb2546d-8c16-262d-9336-5b41cbf992b5@tompassin.net> <9db7a5e1-076c-a9aa-aee4-e044e45decb7@tompassin.net> Message-ID: Am 28.01.23 um 02:51 schrieb Thomas Passin: >> This is literally the version I described myself, except using triple >> quotes. It only modifies the underlying problem, but doesn't solve it. > > Ok, so now we are in the territory of "Tell us what you are trying to > accomplish". And part of that is why you cannot put some constraints on > what your string fragments are.? The example I gave, copied out of your > earlier message, worked and now you are springing triple quotes on us. It works in this particular case, yes. Just like the example I gave in my original case: eval("f'" + s + "'") "works" if there are no apostrophes used. And just like eval("f\"" + s + "\"") "works" if there are no quotation marks used. I don't want to have to care about what quotation is used inside the string, as long as it could successfully evaluate using the f-string grammar. > Stop with the rock management already and explain (briefly if possible) > what you are up to. I have a string. I want to evaluate it as if it were an f-string. I.e., there *are* obviously restrictions that apply (namely, the syntax and semantics of f-strings), but that's it. Best, Johannes From dfnsonfsduifb at gmx.de Sat Jan 28 14:57:23 2023 From: dfnsonfsduifb at gmx.de (Johannes Bauer) Date: Sat, 28 Jan 2023 20:57:23 +0100 Subject: Evaluation of variable as f-string In-Reply-To: References: <8ebc73d2-e14a-b9ad-fa35-5078e5f9fd6c@btinternet.com> Message-ID: Am 28.01.23 um 00:41 schrieb Chris Angelico: > On Sat, 28 Jan 2023 at 10:08, Rob Cliffe via Python-list > wrote: >> >> Whoa! Whoa! Whoa! >> I appreciate the points you are making, Chris, but I am a bit taken >> aback by such forceful language. > > The exact same points have already been made, but not listened to. > Sometimes, forceful language is required in order to get people to > listen. An arrogant bully's rationale. Personally, I'm fine with it. I've been to Usenet for a long time, in which this way of "educating" people was considered normal. But I do think it creates a deterring, toxic environment and reflects back to you as a person negatively. >> Addressing your points specifically: >> 1) I believe the quote character limitation could be overcome. It >> would need a fair amount of work, for which I haven't (yet) the time or >> inclination. > > No problem. Here, solve it for this string: > > eval_me = ' f"""{f\'\'\'{f"{f\'{1+2}\'}"}\'\'\'}""" ' > > F-strings can be nested, remember. Exactly. This is precisely what I want to avoid. Essentially, proper quotation of such a string requires to write a fully fledged f-string parser, in which case the whole problem solves itself. >>> Don't ask how to use X to do Y. Ask how to do Y. >> Good advice. > > Exactly. As I have shown, asking how to use f-strings to achieve this > is simply not suitable, and there's no useful way to discuss other > than to argue semantics. If we had a GOAL to discuss, we could find > much better options. I was not asking how to use f-strings. I was asking to evaluate a string *as if it were* an f-string. Those are two completely different things which you entirely ignored. In other words, if there were a magic function: evalfstring(s, x = x) That would have been the ideal answer. There does not seem to be one, however. So I'm back to silly workarounds. Cheers, Johannes From dfnsonfsduifb at gmx.de Sat Jan 28 15:02:03 2023 From: dfnsonfsduifb at gmx.de (Johannes Bauer) Date: Sat, 28 Jan 2023 21:02:03 +0100 Subject: Evaluation of variable as f-string In-Reply-To: References: Message-ID: Am 27.01.23 um 23:10 schrieb Christian Gollwitzer: > Am 27.01.23 um 21:43 schrieb Johannes Bauer: >> I don't understand why you fully ignore literally the FIRST example I >> gave in my original post and angrily claim that you solution works >> when it does not: >> >> x = { "y": "z" } >> s = "-> {x['y']}" >> print(s.format(x = x)) >> Traceback (most recent call last): >> ?? File "", line 1, in >> KeyError: "'y'" >> >> This. Does. Not. Work. > > It's because "you're holding it wrong!". Notice the error message; it > says that the key 'y' does not exist. Ah, that is neat! I didn't know that. Thanks for the info. In my case, I do also however want to have some functionality that actually does math or even calls functions. That would be possible with templates or f-strings, but not format: x = { "t": 12345 } s = "{x['t'] // 60:02d}:{x['t'] % 60:02d}" print(s.format(x = x)) Traceback (most recent call last): File "", line 1, in KeyError: "'t'" and s = "{x[t] // 60:02d}:{x[t] % 60:02d}" print(s.format(x = x)) Traceback (most recent call last): File "", line 1, in ValueError: Only '.' or '[' may follow ']' in format field specifier but of course: print(f"{x['t'] // 60:02d}:{x['t'] % 60:02d}") 205:45 Best, Johannes From cl at isbd.net Sat Jan 28 15:07:44 2023 From: cl at isbd.net (Chris Green) Date: Sat, 28 Jan 2023 20:07:44 +0000 Subject: Usenet vs. Mailing-list References: <87v8krvgcl.fsf@bsb.me.uk> <20230128143059.cbigz4ad4uklqtbk@hjp.at> Message-ID: Jon Ribbens wrote: > On 2023-01-28, Peter J. Holzer wrote: > > On 2023-01-27 21:04:58 +0000, Ben Bacarisse wrote: > >> It looks like you posted this question via Usenet. comp.lang.python is > >> essentially dead as a Usenet group. It exists, and gets NNTP versions > >> of mail sent to the mailing list, but nothing posted to the group via > >> NNTP get send on the mailing list. > > > > This is wrong. I did get Muttley's any your postings via the > > mailing-list. > > Yes, it's certainly false. I only ever post via the newsgroup, > and I can see my postings reach the list because they appear > in the list archive on the web. > As far as I am aware the mirroring of the Python mailing list on comp.lan.python works perfectly. I love gmane! :-) -- Chris Green ? From cl at isbd.net Sat Jan 28 15:26:06 2023 From: cl at isbd.net (Chris Green) Date: Sat, 28 Jan 2023 20:26:06 +0000 Subject: Usenet vs. Mailing-list References: <87v8krvgcl.fsf@bsb.me.uk> <20230128143059.cbigz4ad4uklqtbk@hjp.at> Message-ID: Chris Green wrote: > Jon Ribbens wrote: > > On 2023-01-28, Peter J. Holzer wrote: > > > On 2023-01-27 21:04:58 +0000, Ben Bacarisse wrote: > > >> It looks like you posted this question via Usenet. comp.lang.python is > > >> essentially dead as a Usenet group. It exists, and gets NNTP versions > > >> of mail sent to the mailing list, but nothing posted to the group via > > >> NNTP get send on the mailing list. > > > > > > This is wrong. I did get Muttley's any your postings via the > > > mailing-list. > > > > Yes, it's certainly false. I only ever post via the newsgroup, > > and I can see my postings reach the list because they appear > > in the list archive on the web. > > > As far as I am aware the mirroring of the Python mailing list on > comp.lan.python works perfectly. I love gmane! :-) > That is, of course, comp.lang.python. Too much wine! :-) -- Chris Green ? From wlfraed at ix.netcom.com Sat Jan 28 16:36:26 2023 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Sat, 28 Jan 2023 16:36:26 -0500 Subject: Usenet vs. Mailing-list (was: evaluation question) References: <87v8krvgcl.fsf@bsb.me.uk> <20230128143059.cbigz4ad4uklqtbk@hjp.at> Message-ID: <255bth98ettkgisai1fgb1lo47eeiqg4qt@4ax.com> On Sat, 28 Jan 2023 15:30:59 +0100, "Peter J. Holzer" declaimed the following: >On 2023-01-27 21:04:58 +0000, Ben Bacarisse wrote: >> Muttley at dastardlyhq.com writes: >> >> > Hi >> >> It looks like you posted this question via Usenet. comp.lang.python is >> essentially dead as a Usenet group. It exists, and gets NNTP versions >> of mail sent to the mailing list, but nothing posted to the group via >> NNTP get send on the mailing list. > >This is wrong. I did get Muttley's any your postings via the >mailing-list. > I had been tempted to ask when the list<>newsgroup link had changed, since it hadn't been announced... Now -- last time I checked the gmane server says posting is prohibited. I used to use gmane as it retrieved directly from the mailing list (which has been filtered from much spam) and not Usenet (and its open spam problem). -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From wlfraed at ix.netcom.com Sat Jan 28 16:39:19 2023 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Sat, 28 Jan 2023 16:39:19 -0500 Subject: Usenet vs. Mailing-list References: <87v8krvgcl.fsf@bsb.me.uk> <20230128143059.cbigz4ad4uklqtbk@hjp.at> Message-ID: <5d5bth9bkjs3bt0ulo7djuvp7m0f6t33c5@4ax.com> On Sat, 28 Jan 2023 20:07:44 +0000, Chris Green declaimed the following: >As far as I am aware the mirroring of the Python mailing list on >comp.lan.python works perfectly. I love gmane! :-) Is gmane's gmane.comp.python.general allowing posts to go through again? I had to revert to comp.lang.python some time back when gmane kept rejecting outgoing posts. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From rosuav at gmail.com Sat Jan 28 19:32:47 2023 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 29 Jan 2023 11:32:47 +1100 Subject: bool and int In-Reply-To: References: <001901d9312b$ef077b60$cd167220$@gmail.com> <007801d931fc$c8a97a50$59fc6ef0$@gmail.com> <63d4b40f.020a0220.ef1d.4c85@mx.google.com> Message-ID: On Sun, 29 Jan 2023 at 11:27, rbowman wrote: > > On Fri, 27 Jan 2023 21:35:11 -0800 (PST), Grant Edwards wrote: > > > In Unix shells, a return code of 0 is true and non-0 is false. > > That carries over to some C functions like strcmp() although it's more > complex. strcmp() returns the value of subtracting the nth character of > string b from string a if the value is not 0. For matching strings, the > result is 0 for all character positions. > > This plays nicely with sorting functions but naive programmers assume it's > a boolean and strcmp("foo", "foo") should return 1 or true when the > strings match. > > Returning 0 for success gives you much more latitude in return values. That's not really about booleans, it's more like "imagine subtracting one string from another". You can compare two integers by subtracting one from another; you'll get a number that's less than zero, zero, or greater than zero, just like with strcmp. And yes, that plays perfectly with sorting functions. So when you want to compare strings, what do you do? You make something that subtracts one string from another. It's "String Compare", not "Are Strings Equal", so it doesn't return a boolean. Zero for equal makes very good sense, even though a "strings equal" function would return zero for non-equal. ChrisA From rosuav at gmail.com Sat Jan 28 20:05:51 2023 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 29 Jan 2023 12:05:51 +1100 Subject: Evaluation of variable as f-string In-Reply-To: References: <5bb2546d-8c16-262d-9336-5b41cbf992b5@tompassin.net> <9db7a5e1-076c-a9aa-aee4-e044e45decb7@tompassin.net> Message-ID: On Sun, 29 Jan 2023 at 11:53, Johannes Bauer wrote: > I don't want to have to care about what quotation is used inside the > string, as long as it could successfully evaluate using the f-string > grammar. > Not possible. An f-string can contain other f-strings, and it is entirely possible to use EVERY quote type. So you can never add quotes around the outside of a string and then evaluate it as an f-string, without making sure that it doesn't already contain that string. (That MAY be changing in a future version of Python, but it's currently true.) ChrisA From rosuav at gmail.com Sat Jan 28 20:09:16 2023 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 29 Jan 2023 12:09:16 +1100 Subject: Evaluation of variable as f-string In-Reply-To: References: <8ebc73d2-e14a-b9ad-fa35-5078e5f9fd6c@btinternet.com> Message-ID: On Sun, 29 Jan 2023 at 11:56, Johannes Bauer wrote: > > Am 28.01.23 um 00:41 schrieb Chris Angelico: > > On Sat, 28 Jan 2023 at 10:08, Rob Cliffe via Python-list > > wrote: > >> > >> Whoa! Whoa! Whoa! > >> I appreciate the points you are making, Chris, but I am a bit taken > >> aback by such forceful language. > > > > The exact same points have already been made, but not listened to. > > Sometimes, forceful language is required in order to get people to > > listen. > > An arrogant bully's rationale. Personally, I'm fine with it. I've been > to Usenet for a long time, in which this way of "educating" people was > considered normal. But I do think it creates a deterring, toxic > environment and reflects back to you as a person negatively. Arrogant bully? Or someone who has tried *multiple times* to explain to you that what you're asking for is IMPOSSIBLE, and you need to ask a better question if you want a better answer? If that's "bullying", then fine, ban me for bullying, and go find somewhere else where you'll be coddled and told that your question is fine, it's awesome, and yes, wouldn't it be nice if magic were a thing. > Exactly. This is precisely what I want to avoid. Essentially, proper > quotation of such a string requires to write a fully fledged f-string > parser, in which case the whole problem solves itself. > > >>> Don't ask how to use X to do Y. Ask how to do Y. > >> Good advice. > > > > Exactly. As I have shown, asking how to use f-strings to achieve this > > is simply not suitable, and there's no useful way to discuss other > > than to argue semantics. If we had a GOAL to discuss, we could find > > much better options. > > I was not asking how to use f-strings. I was asking to evaluate a string > *as if it were* an f-string. Those are two completely different things > which you entirely ignored. They're not different things, because what you asked for is NOT POSSIBLE without the caveats that I gave. It is *fundamentally not possible* to "evaluate a string as if it were an f-string", other than by wrapping it in an f-string and evaluating it - with the consequences of that. > In other words, if there were a magic function: > > evalfstring(s, x = x) > > That would have been the ideal answer. There does not seem to be one, > however. So I'm back to silly workarounds. > Right. Exactly. Now if you'd asked for what you REALLY need, maybe there'd be a solution involving format_map, but no, you're so utterly intransigent that you cannot adjust your question to fit reality. If that makes me a bad guy, then fine. I'll be the bad guy. But you're not going to change the laws of physics. ChrisA From rosuav at gmail.com Sat Jan 28 20:09:52 2023 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 29 Jan 2023 12:09:52 +1100 Subject: Usenet vs. Mailing-list In-Reply-To: References: <87v8krvgcl.fsf@bsb.me.uk> <20230128143059.cbigz4ad4uklqtbk@hjp.at> Message-ID: On Sun, 29 Jan 2023 at 12:07, Chris Green wrote: > > Chris Green wrote: > > Jon Ribbens wrote: > > > On 2023-01-28, Peter J. Holzer wrote: > > > > On 2023-01-27 21:04:58 +0000, Ben Bacarisse wrote: > > > >> It looks like you posted this question via Usenet. comp.lang.python is > > > >> essentially dead as a Usenet group. It exists, and gets NNTP versions > > > >> of mail sent to the mailing list, but nothing posted to the group via > > > >> NNTP get send on the mailing list. > > > > > > > > This is wrong. I did get Muttley's any your postings via the > > > > mailing-list. > > > > > > Yes, it's certainly false. I only ever post via the newsgroup, > > > and I can see my postings reach the list because they appear > > > in the list archive on the web. > > > > > As far as I am aware the mirroring of the Python mailing list on > > comp.lan.python works perfectly. I love gmane! :-) > > > That is, of course, comp.lang.python. Too much wine! :-) > You can run Python with wine, but it runs natively and that's usually better :) ChrisA From ben.usenet at bsb.me.uk Sat Jan 28 20:51:52 2023 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Sun, 29 Jan 2023 01:51:52 +0000 Subject: Usenet vs. Mailing-list References: <87v8krvgcl.fsf@bsb.me.uk> <20230128143059.cbigz4ad4uklqtbk@hjp.at> Message-ID: <87fsbuumyv.fsf@bsb.me.uk> "Peter J. Holzer" writes: > On 2023-01-27 21:04:58 +0000, Ben Bacarisse wrote: >> Muttley at dastardlyhq.com writes: >> >> > Hi >> >> It looks like you posted this question via Usenet. comp.lang.python is >> essentially dead as a Usenet group. It exists, and gets NNTP versions >> of mail sent to the mailing list, but nothing posted to the group via >> NNTP get send on the mailing list. > > This is wrong. I did get Muttley's any your postings via the > mailing-list. Ah, OK. I thought that was the case but I am obviously wrong. Has there been a change, or have I been wrong for a long time!? There may be a timing issue because I tried to find a reply a Usenet injected post from a mailing-list user and, at the time I looked, I could not find one. -- Ben. From jon+usenet at unequivocal.eu Sat Jan 28 21:09:28 2023 From: jon+usenet at unequivocal.eu (Jon Ribbens) Date: Sun, 29 Jan 2023 02:09:28 -0000 (UTC) Subject: Usenet vs. Mailing-list References: <87v8krvgcl.fsf@bsb.me.uk> <20230128143059.cbigz4ad4uklqtbk@hjp.at> <87fsbuumyv.fsf@bsb.me.uk> Message-ID: On 2023-01-29, Ben Bacarisse wrote: > "Peter J. Holzer" writes: > >> On 2023-01-27 21:04:58 +0000, Ben Bacarisse wrote: >>> Muttley at dastardlyhq.com writes: >>> >>> > Hi >>> >>> It looks like you posted this question via Usenet. comp.lang.python is >>> essentially dead as a Usenet group. It exists, and gets NNTP versions >>> of mail sent to the mailing list, but nothing posted to the group via >>> NNTP get send on the mailing list. >> >> This is wrong. I did get Muttley's any your postings via the >> mailing-list. > > Ah, OK. I thought that was the case but I am obviously wrong. Has > there been a change, or have I been wrong for a long time!? I'm not aware of any significant period in the last twenty-one years that it hasn't been working. Although sometimes it does feel like it isn't, in that I reply to a post with an answer and then several other people reply significantly later with the same answer, as if my one had never existed... but whenever I check into it, my message has actually always made it to the list. From jfong at ms4.hinet.net Sat Jan 28 21:55:34 2023 From: jfong at ms4.hinet.net (Jach Feng) Date: Sat, 28 Jan 2023 18:55:34 -0800 (PST) Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: <6bdc1904-4f0b-4c05-a67d-b0965b30624dn@googlegroups.com> Message-ID: <7118e963-7594-4d95-97bf-e15cea2c2bfen@googlegroups.com> Mark Bourne ? 2023?1?28? ?????10:00:01 [UTC+8] ?????? > Jach Feng wrote: > > Jach Feng ? 2023?1?22? ?????11:11:22 [UTC+8] ?????? > >> Fail on command line, > >> > >> e:\Works\Python>py infix2postfix.py "-4^2+5.3*abs(-2-1)/2" > >> usage: infix2postfix.py [-h] [infix] > >> infix2postfix.py: error: unrecognized arguments: -4^2+5.3*abs(-2-1)/2 > >> > >> Also fail in REPL, > >> > >> e:\Works\Python>py > >> Python 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:08:11) [MSC v.1928 32 bit (Intel)] on win32 > >> Type "help", "copyright", "credits" or "license" for more information. > >>>>> import argparse > >>>>> parser = argparse.ArgumentParser(description='Convert infix notation to postfix') > >>>>> parser.parse_args("-4^2+5.3*abs(-2-1)/2") > >> usage: [-h] > >> : error: unrecognized arguments: - 4 ^ 2 + 5 . 3 * a b s ( - 2 - 1 ) / 2 > >> > >> Just can't figure out where is wrong!? > >> > >> --Jach > > I have to admit that I don't know the background upon which the argparse was built. The good side is that I don't carry its historical knowledge ( or burden?), that's why I can use it in a new way which may make someone feel uneasy. > If you can use it in the way you want, that's great. I thought you were > asking here because you *couldn't* use it the way you want. > > You're writing a command-line application. Your users are either > already familiar with the conventions of command-line applications, or > they'll soon need to be. > > If your application supports options beginning with a "-" (which is what > argparse gives you, even if only the default "-h" and "--help" options > are actually valid), and you also need to be able to pass positional > arguments which begin with a "-", you need some way for the user to > indicate whether any particular argument is an option or positional. > Using "--" to indicate that all subsequent arguments are positional, not > options, is a common convention. > > Some of your users might already be familiar with that convention from > other command-line tools. By using the same convention, you'd be making > it easier for them to use yours. Others might not already be familiar > with the convention, but by being consistent with other tools you'd > still be helping them when they eventually do come across it elsewhere. > You can always include an explanation of using "--" in your usage output > or other documentation. > > Apart from dealing with how to pass an argument beginning with "-", your > users will also likely have to deal with the features of whichever shell > they're using, which you have no control over. For example, it's quite > common to need to enclose an argument in quotes if it contains spaces. > It may also be necessary to use quotes if certain special characters are > used, such as "*", "?" or "$" (perhaps "%" on Windows). > > The reason I am still keep on using argparse on this "one positional argument only" CLI app is that I can't see what advantage I can get when writing code to handling sys.argv directly for the following two situations, > > ----- > > e:\Works\Python>py infix2postfix.py > > usage: infix2postfix.py [-h] infix > > infix2postfix.py: error: the following arguments are required: infix > > > > e:\Works\Python>py infix2postfix.py -h > > usage: infix2postfix.py [-h] infix > > > > Convert infix notation to postfix > > > > positional arguments: > > infix Put equations in quote if there is space in it and separate each one with a comma, ie. > > "-4^2+5.3*abs(-2-1)/2, abs(Abc)*(B+C)/D, (-3) * sqrt(1-(x1/7)*(y1/7)) * sqrt(abs((x0-4.5)/(y0-4)))" > > > > optional arguments: > > -h, --help show this help message and exit > > ----- > > > > comparing with code using the argparse below, > > > > import argparse > > sample = "-4^2+5.3*abs(-2-1)/2, abs(Abc)*(B+C)/D, (-3) * sqrt(1-(x1/7)*(y1/7)) * sqrt(abs((x0-4.5)/(y0-4)))" > > parser = argparse.ArgumentParser(description='Convert infix notation to postfix') > > parser.add_argument('infix', > > help='Put equations in quote if there is space in it and separate each one with a comma, ie. "{}"'.format(sample)) > > > > import sys > > if len(sys.argv) > 1 and not '-h' in sys.argv: > > ....sys.argv.insert(1, '--') > > args = parser.parse_args() > Personally, I do quite often use argparse even for simple cases, partly > because it produces nice usage details, and deals with wrapping the > output to whatever width the console is. But I work with it rather than > against it, and accept that a "--" is needed if a positional argument > starts with a "-". > > I notice you explain the need to enclose the equation in quotes if it > contains spaces. That's not even a feature of your application, but of > the shell used to call it. So why so much objection to explaining the > need for "--"? > > Depending on the shell, there are other cases where quoting might be > needed, e.g. if the equation includes a "*" or "?" and happens to look > like a pattern matching files in the current directory (most shells I've > used pass the pattern unchanged if it doesn't match any files). In > bash, if a "$" is used I'd need to enclose that in 'single quotes' > (can't even use "double quotes" for that one). You can't really expect > to document all that sort of thing, because it depends on which shell > the user happens to run your application from - you just have to trust > the user to know or learn how to use their shell. > > -- > Mark. Thank you for detail explanation of the role the shell is involved in this problem. I'm very appreciated! It seems that a CLI app may become very complex when dealing with different kind of shell, and may not be possible to solve its problem. But the good thing in my app is that I need only to handle math equation:-) > So why so much objection to explaining the need for "--"? Because of using " to enclose a space separated string is a common convention, and adding a "--" is not:-) --Jach From ben.usenet at bsb.me.uk Sat Jan 28 22:02:36 2023 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Sun, 29 Jan 2023 03:02:36 +0000 Subject: Usenet vs. Mailing-list References: <87v8krvgcl.fsf@bsb.me.uk> <20230128143059.cbigz4ad4uklqtbk@hjp.at> <87fsbuumyv.fsf@bsb.me.uk> Message-ID: <87357uujoz.fsf@bsb.me.uk> Jon Ribbens writes: > On 2023-01-29, Ben Bacarisse wrote: >> "Peter J. Holzer" writes: >> >>> On 2023-01-27 21:04:58 +0000, Ben Bacarisse wrote: >>>> Muttley at dastardlyhq.com writes: >>>> >>>> > Hi >>>> >>>> It looks like you posted this question via Usenet. comp.lang.python is >>>> essentially dead as a Usenet group. It exists, and gets NNTP versions >>>> of mail sent to the mailing list, but nothing posted to the group via >>>> NNTP get send on the mailing list. >>> >>> This is wrong. I did get Muttley's any your postings via the >>> mailing-list. >> >> Ah, OK. I thought that was the case but I am obviously wrong. Has >> there been a change, or have I been wrong for a long time!? > > I'm not aware of any significant period in the last twenty-one years > that it hasn't been working. Although sometimes it does feel like it > isn't, in that I reply to a post with an answer and then several > other people reply significantly later with the same answer, as if > my one had never existed... but whenever I check into it, my message > has actually always made it to the list. I have had the same experience and, as a result, I rarely post. Maybe what I have to say is simply not interesting! -- Ben. From avi.e.gross at gmail.com Sat Jan 28 23:15:22 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Sat, 28 Jan 2023 23:15:22 -0500 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <6bdc1904-4f0b-4c05-a67d-b0965b30624dn@googlegroups.com> References: <6bdc1904-4f0b-4c05-a67d-b0965b30624dn@googlegroups.com> Message-ID: <00bf01d93398$4ee9c8d0$ecbd5a70$@gmail.com> Jack, I get uneasy when someone thinks a jackhammer is a handy dandy tool for pushing in a thumbtack that is sitting on my expensive table. I agree it is quite easy to grab some code that does lot of things and also does something truly minor, and use it for that purpose. Sometimes the cost is that it is far slower. I mean if I want to print the smaller of variables alpha and beta containing integers, I could combine them in something like a list or array and call a nifty sort() function someone wrote that takes a function argument that lets you compare whatever is needed and it likely will work. You get back the result and print the first item in the sorted output or maybe the last. You could have used a simpler function like min(alpha, beta) and skipped making a list of them. Heck, you could have used a fairly simple if statement. But if you searched the internet and found a function that takes any number of data items and performs many statistical tests on it tat start with things like mean, median and mode and continues to calculate standard deviation, and skew and standard error and also the min/max/range. Sounds good. It returns some structure/object and you dig into it and find the minimum and you are done! It sounds like the jackhammer approach to me. In your case, there is indeed nothing wrong with using the function to help parse the command line arguments and you are being told that it generally works EXCEPT when it is documented NOT TO WORK. It was designed to deal with UNIX conventions so you could call a program with a mix of optional flags and then optional arguments such as filenames or anything else or ALMOST anything else. The early design allowed something like to list files using: ls - la *.c or ls - l -a *.c Or many more variations that can include newer ways that use longer options. People ran into the problem of having some options that included a word following them that is to be parsed as part of that option and other enhancements. The minus sign or hyphen was chosen and is not trivial to change. So the idea was to add a flag with two minus signs that has no meaning other than to say that anything after that is to be somewhat ignored as not being a flag. Thus it is safe for any further arguments to start with a minus sign. How does that line up with some of the advice you got? One idea was to have your users told they should add a " -- " (without quotes) before the formula you want to evaluate. Another was asking if you had a more complex command line with many options that needed a swiss army knife approach. If not, since you have formulas starting with a minus sign that may be used, consider NOT using a tool you admit you did not understand. And since python like many languages provides you with a data structure that already holds the info you need, use the thumbtack approach or find or write a different library function that extracts what you need in an environment where it is NOT looking for things with a minus sign. You know when you go to a doctor and ask why a part of your skin is bleeding and so on, and you admit you keep scratching yourself, rather than suggesting complex surgeries or putting your hands in a cast to keep you from scratching, they may first try telling you to NOT DO THAT and let it heal. Do note a final point. There is nothing magical about using a minus sign in UNIX and they could have insisted say that command line arguments be placed in square brackets or some other method to identify them. It seemed harmless to use a minus sign at the time. But the programs they includes starting using so many allowed tweaks of many kinds, that it got really hard to parse what was being asked and thus several libraries of functions have been built that manage all that. Your problem is far from unique as any program trying to deal with an argument for a filename that begins with a minus sign will have a similar problem. Programs like grep often take an argument that is a regular expression or other utilities take a "program" that also may for whatever reason start with a minus sign. Some people ask a question and you have asked many here, and got answers. Usually you accept what people tell you better. I am trying to say that the issue is not whether you are using the wrong tool in general. It is that NOW that you know there is sometimes a problem, you resist understanding it was never designed to do what you want. -----Original Message----- From: Python-list On Behalf Of Jach Feng Sent: Saturday, January 28, 2023 12:04 AM To: python-list at python.org Subject: Re: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? Jach Feng ? 2023?1?22? ?????11:11:22 [UTC+8] ?????? > Fail on command line, > > e:\Works\Python>py infix2postfix.py "-4^2+5.3*abs(-2-1)/2" > usage: infix2postfix.py [-h] [infix] > infix2postfix.py: error: unrecognized arguments: -4^2+5.3*abs(-2-1)/2 > > Also fail in REPL, > > e:\Works\Python>py > Python 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:08:11) [MSC v.1928 > 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. > >>> import argparse > >>> parser = argparse.ArgumentParser(description='Convert infix > >>> notation to postfix') > >>> parser.parse_args("-4^2+5.3*abs(-2-1)/2") > usage: [-h] > : error: unrecognized arguments: - 4 ^ 2 + 5 . 3 * a b s ( - 2 - 1 ) / > 2 > > Just can't figure out where is wrong!? > > --Jach I have to admit that I don't know the background upon which the argparse was built. The good side is that I don't carry its historical knowledge ( or burden?), that's why I can use it in a new way which may make someone feel uneasy. The reason I am still keep on using argparse on this "one positional argument only" CLI app is that I can't see what advantage I can get when writing code to handling sys.argv directly for the following two situations, ----- e:\Works\Python>py infix2postfix.py usage: infix2postfix.py [-h] infix infix2postfix.py: error: the following arguments are required: infix e:\Works\Python>py infix2postfix.py -h usage: infix2postfix.py [-h] infix Convert infix notation to postfix positional arguments: infix Put equations in quote if there is space in it and separate each one with a comma, ie. "-4^2+5.3*abs(-2-1)/2, abs(Abc)*(B+C)/D, (-3) * sqrt(1-(x1/7)*(y1/7)) * sqrt(abs((x0-4.5)/(y0-4)))" optional arguments: -h, --help show this help message and exit ----- comparing with code using the argparse below, import argparse sample = "-4^2+5.3*abs(-2-1)/2, abs(Abc)*(B+C)/D, (-3) * sqrt(1-(x1/7)*(y1/7)) * sqrt(abs((x0-4.5)/(y0-4)))" parser = argparse.ArgumentParser(description='Convert infix notation to postfix') parser.add_argument('infix', help='Put equations in quote if there is space in it and separate each one with a comma, ie. "{}"'.format(sample)) import sys if len(sys.argv) > 1 and not '-h' in sys.argv: ....sys.argv.insert(1, '--') args = parser.parse_args() -- https://mail.python.org/mailman/listinfo/python-list From rosuav at gmail.com Sat Jan 28 23:47:47 2023 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 29 Jan 2023 15:47:47 +1100 Subject: Evaluation of variable as f-string In-Reply-To: References: <5bb2546d-8c16-262d-9336-5b41cbf992b5@tompassin.net> <9db7a5e1-076c-a9aa-aee4-e044e45decb7@tompassin.net> Message-ID: On Sun, 29 Jan 2023 at 14:36, Stefan Ram wrote: > > Johannes Bauer writes: > >I have a string. I want to evaluate it as if it were an f-string. I.e., > >there *are* obviously restrictions that apply (namely, the syntax and > >semantics of f-strings), but that's it. > > (This message was written for Usenet. If you read it in a > mailing list or the Web, it has been stolen from Usenet.) > I'm curious as to the copyright protections available to you, but if you're going to threaten python-list's owners with legal action for daring to rebroadcast a public post, I would have to recommend that you get promptly banned from the list in order to reduce liability. What's so bad about mailing lists that you don't want your messages to be seen on them? ChrisA From rosuav at gmail.com Sat Jan 28 23:55:34 2023 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 29 Jan 2023 15:55:34 +1100 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <7118e963-7594-4d95-97bf-e15cea2c2bfen@googlegroups.com> References: <6bdc1904-4f0b-4c05-a67d-b0965b30624dn@googlegroups.com> <7118e963-7594-4d95-97bf-e15cea2c2bfen@googlegroups.com> Message-ID: On Sun, 29 Jan 2023 at 14:29, Jach Feng wrote: > Thank you for detail explanation of the role the shell is involved in this problem. I'm very appreciated! > > It seems that a CLI app may become very complex when dealing with different kind of shell, and may not be possible to solve its problem. But the good thing in my app is that I need only to handle math equation:-) > > > So why so much objection to explaining the need for "--"? > Because of using " to enclose a space separated string is a common convention, and adding a "--" is not:-) Double hyphen is incredibly common. On most Unix shells, double quotes are just one way of marking that a space should be included in an argument rather than separating (others include single quotes, escaping the space with a backslash, and changing the field separator). Of course, everything is conventions, not requirements, but it's generally better to let the person's shell define the argument splitting; that way, if your script is called from another process, it's dead easy to control the splitting yourself (just pass an array/list of arguments to the subprocess spawner). Maybe you come from Windows, where there are fewer conventions and less shell parsing, but argparse follows Unix conventions, so if you don't want Unix conventions, don't use argparse. Just read sys.argv directly. ChrisA From list1 at tompassin.net Sat Jan 28 23:27:58 2023 From: list1 at tompassin.net (Thomas Passin) Date: Sat, 28 Jan 2023 23:27:58 -0500 Subject: Evaluation of variable as f-string In-Reply-To: References: <5bb2546d-8c16-262d-9336-5b41cbf992b5@tompassin.net> <9db7a5e1-076c-a9aa-aee4-e044e45decb7@tompassin.net> Message-ID: <8ca25af2-8a5b-b915-4706-da1398d56d24@tompassin.net> On 1/28/2023 2:50 PM, Johannes Bauer wrote: > Am 28.01.23 um 02:51 schrieb Thomas Passin: > >>> This is literally the version I described myself, except using triple >>> quotes. It only modifies the underlying problem, but doesn't solve it. >> >> Ok, so now we are in the territory of "Tell us what you are trying to >> accomplish". And part of that is why you cannot put some constraints >> on what your string fragments are.? The example I gave, copied out of >> your earlier message, worked and now you are springing triple quotes >> on us. > > It works in this particular case, yes. Just like the example I gave in > my original case: > > eval("f'" + s + "'") > > "works" if there are no apostrophes used. And just like > > eval("f\"" + s + "\"") > > "works" if there are no quotation marks used. > > I don't want to have to care about what quotation is used inside the > string, as long as it could successfully evaluate using the f-string > grammar. > >> Stop with the rock management already and explain (briefly if >> possible) what you are up to. > > I have a string. I want to evaluate it as if it were an f-string. I.e., > there *are* obviously restrictions that apply (namely, the syntax and > semantics of f-strings), but that's it. Well, yes, we do see that. What we don't see is what you want to accomplish by doing it, and why you don't seem willing to accept some restrictions on the string fragments so that they will evaluate correctly. IOW, perhaps there is a more practical way to accomplish what you want. Except that we don't know what that is. From cs at cskk.id.au Sun Jan 29 00:51:20 2023 From: cs at cskk.id.au (Cameron Simpson) Date: Sun, 29 Jan 2023 16:51:20 +1100 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <7118e963-7594-4d95-97bf-e15cea2c2bfen@googlegroups.com> References: <7118e963-7594-4d95-97bf-e15cea2c2bfen@googlegroups.com> Message-ID: On 28Jan2023 18:55, Jach Feng wrote: >Mark Bourne ? 2023?1?28? ?????10:00:01 [UTC+8] ?????? >> I notice you explain the need to enclose the equation in quotes if it >> contains spaces. That's not even a feature of your application, but of >> the shell used to call it. So why so much objection to explaining the >> need for "--"? >> >> Depending on the shell, there are other cases where quoting might be >> needed, e.g. if the equation includes a "*" or "?" and happens to look >> like a pattern matching files in the current directory (most shells I've >> used pass the pattern unchanged if it doesn't match any files). In >> bash, if a "$" is used I'd need to enclose that in 'single quotes' >> (can't even use "double quotes" for that one). You can't really expect >> to document all that sort of thing, because it depends on which shell >> the user happens to run your application from - you just have to trust >> the user to know or learn how to use their shell. > >Thank you for detail explanation of the role the shell is involved in this problem. I'm very appreciated! The shell has basicly _nothing_ to do with your problems. By the time you've got sys.argv in your Python programme you will have no idea whether quotes were used with an argument. (UNIX/POSIX, not Windows, where things are ... more complex.) This means you don't know if the use typed: -4.5 or "-4.5" You'll just get a string '4.5' in your Python programme both ways. All the quotes in the shell do is delimit what things should be kept together as a single argument versus several, or where variables should be interpolated when computing arguments etc. It's just _shell_ punctuation and the invoked programme doesn't see it. >It seems that a CLI app may become very complex when dealing with different kind of shell, and may not be possible to solve its problem. It doesn't matter what shell is used. The just controls what punctuation the end user may need to use to invoke your programme. You programme doesn't need to care (and can't because it doesn't get the quotes etc, only their result). >> So why so much objection to explaining the need for "--"? >Because of using " to enclose a space separated string is a common convention, and adding a "--" is not:-) They're unrelated. As others have mentioned, "--" is _extremely_ common; almost _all_ UNIX command like programmes which handle -* style options honour the "--" convention. _argparse_ itself honours that convention, as does getopt etc. The "--" convention has nothing to do with the shell. Cheers, Cameron Simpson From 2QdxY4RzWzUUiLuE at potatochowder.com Sun Jan 29 07:12:29 2023 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Sun, 29 Jan 2023 07:12:29 -0500 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: <7118e963-7594-4d95-97bf-e15cea2c2bfen@googlegroups.com> Message-ID: On 2023-01-29 at 16:51:20 +1100, Cameron Simpson wrote: > They're unrelated. As others have mentioned, "--" is _extremely_ common; > almost _all_ UNIX command like programmes which handle -* style options > honour the "--" convention. _argparse_ itself honours that convention, as > does getopt etc. And why do UNIX programs behave this way? Because POSIX says they should: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html From avi.e.gross at gmail.com Sun Jan 29 07:59:21 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Sun, 29 Jan 2023 07:59:21 -0500 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: <7118e963-7594-4d95-97bf-e15cea2c2bfen@googlegroups.com> Message-ID: <000a01d933e1$819a94f0$84cfbed0$@gmail.com> Although today you could say POSIX is the reason for many things including the use of "--" I hesitate to mention I and many others used that convention long before as a standard part of many UNIX utilities. Like many other such things, you build things first and by the time you standardize, ... -----Original Message----- From: Python-list On Behalf Of 2QdxY4RzWzUUiLuE at potatochowder.com Sent: Sunday, January 29, 2023 7:12 AM To: python-list at python.org Subject: Re: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? On 2023-01-29 at 16:51:20 +1100, Cameron Simpson wrote: > They're unrelated. As others have mentioned, "--" is _extremely_ > common; almost _all_ UNIX command like programmes which handle -* > style options honour the "--" convention. _argparse_ itself honours > that convention, as does getopt etc. And why do UNIX programs behave this way? Because POSIX says they should: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html -- https://mail.python.org/mailman/listinfo/python-list From avi.e.gross at gmail.com Sun Jan 29 08:10:52 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Sun, 29 Jan 2023 08:10:52 -0500 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: <7118e963-7594-4d95-97bf-e15cea2c2bfen@googlegroups.com> Message-ID: <002301d933e3$1d44c820$57ce5860$@gmail.com> Cameron, You are technically correct but perhaps off the mark. Yes, a python program only sees what is handed to it by some shell if invoked a certain way. The issue here is what you tell people using your program about what they need to type to get it to work. That means if their shell is going to make changes in what they typed, they need to know how to avoid unintended changes. As one example not mentioned, whitespace disappears if not somehow protected as in quotes. What the OP is being told is that their Python program only controls what is fed to it. A user needs to know enough to avoid doing silly things like provide an unquoted string containing reserved symbols like a pipe symbol or odd things may happen and their program may not even be called. So the documentation of how to use the program may need to spell some things out alongside suggesting use of "--" ... -----Original Message----- From: Python-list On Behalf Of Cameron Simpson Sent: Sunday, January 29, 2023 12:51 AM To: python-list at python.org Subject: Re: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? On 28Jan2023 18:55, Jach Feng wrote: >Mark Bourne ? 2023?1?28? ?????10:00:01 [UTC+8] ?????? >> I notice you explain the need to enclose the equation in quotes if it >> contains spaces. That's not even a feature of your application, but >> of the shell used to call it. So why so much objection to explaining >> the need for "--"? >> >> Depending on the shell, there are other cases where quoting might be >> needed, e.g. if the equation includes a "*" or "?" and happens to >> look like a pattern matching files in the current directory (most >> shells I've used pass the pattern unchanged if it doesn't match any >> files). In bash, if a "$" is used I'd need to enclose that in 'single quotes' >> (can't even use "double quotes" for that one). You can't really >> expect to document all that sort of thing, because it depends on >> which shell the user happens to run your application from - you just >> have to trust the user to know or learn how to use their shell. > >Thank you for detail explanation of the role the shell is involved in this problem. I'm very appreciated! The shell has basicly _nothing_ to do with your problems. By the time you've got sys.argv in your Python programme you will have no idea whether quotes were used with an argument. (UNIX/POSIX, not Windows, where things are ... more complex.) This means you don't know if the use typed: -4.5 or "-4.5" You'll just get a string '4.5' in your Python programme both ways. All the quotes in the shell do is delimit what things should be kept together as a single argument versus several, or where variables should be interpolated when computing arguments etc. It's just _shell_ punctuation and the invoked programme doesn't see it. >It seems that a CLI app may become very complex when dealing with different kind of shell, and may not be possible to solve its problem. It doesn't matter what shell is used. The just controls what punctuation the end user may need to use to invoke your programme. You programme doesn't need to care (and can't because it doesn't get the quotes etc, only their result). >> So why so much objection to explaining the need for "--"? >Because of using " to enclose a space separated string is a common >convention, and adding a "--" is not:-) They're unrelated. As others have mentioned, "--" is _extremely_ common; almost _all_ UNIX command like programmes which handle -* style options honour the "--" convention. _argparse_ itself honours that convention, as does getopt etc. The "--" convention has nothing to do with the shell. Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list From hjp-python at hjp.at Sun Jan 29 08:40:09 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 29 Jan 2023 14:40:09 +0100 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <000a01d933e1$819a94f0$84cfbed0$@gmail.com> References: <7118e963-7594-4d95-97bf-e15cea2c2bfen@googlegroups.com> <000a01d933e1$819a94f0$84cfbed0$@gmail.com> Message-ID: <20230129134009.lc3taek7xw7yakqh@hjp.at> On 2023-01-29 07:59:21 -0500, avi.e.gross at gmail.com wrote: > Although today you could say POSIX is the reason for many things including > the use of "--" I hesitate to mention I and many others used that convention > long before as a standard part of many UNIX utilities. I somewhat doubt that you are remembering correctly. The POSIX standard was published in 1988. I was starting to use Unix shortly before that and I don't think there was really a common way to indicate the end of the options. Some utilities used "-", some "--", some just stopped at the first argument which didn't look like an option ... The convergence to "--" happened later. My guess is that GNU getopt and the popularity of the GNU utilities had a lot to do with it. > Like many other such things, you build things first and by the time > you standardize, ... Right. But in this case lots of people were building different things and then *one* of them was standardized. Truth to be told, I don't know when that happened. It might have been in the original 1988 edition of POSIX, but I don't think so. I think it was in a later (possibly much later) edition, wennn "--" was already really common. But POSIX did lead the way in some cases and it might have been the case here, too. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From hjp-python at hjp.at Sun Jan 29 08:54:32 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 29 Jan 2023 14:54:32 +0100 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <6bdc1904-4f0b-4c05-a67d-b0965b30624dn@googlegroups.com> References: <6bdc1904-4f0b-4c05-a67d-b0965b30624dn@googlegroups.com> Message-ID: <20230129135432.p5dmd6irkgfmeqfj@hjp.at> On 2023-01-27 21:03:39 -0800, Jach Feng wrote: > I have to admit that I don't know the background upon which the > argparse was built. The good side is that I don't carry its historical > knowledge ( or burden?), that's why I can use it in a new way I don't see what's new about that. > which may make someone feel uneasy. It seems that the person who feels most uneasy about it is yourself because it doesn't work like you want it to. > The reason I am still keep on using argparse on this "one positional > argument only" CLI app is that I can't see what advantage I can get > when writing code to handling sys.argv directly for the following two > situations, It would almost certainly be shorter and easier to understand, especially since you are *also* handling sys.argv directly in your code to prevent argparse to work as designed. But since you are probably the only person who will have to read your code in the future that's not our problem. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From hjp-python at hjp.at Sun Jan 29 09:15:13 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 29 Jan 2023 15:15:13 +0100 Subject: To gateway or not to gateway a specific person (was: Evaluation of variable as f-string) In-Reply-To: References: <5bb2546d-8c16-262d-9336-5b41cbf992b5@tompassin.net> <9db7a5e1-076c-a9aa-aee4-e044e45decb7@tompassin.net> Message-ID: <20230129141513.sxmsi363jebxyg2x@hjp.at> On 2023-01-29 15:47:47 +1100, Chris Angelico wrote: > On Sun, 29 Jan 2023 at 14:36, Stefan Ram wrote: > > (This message was written for Usenet. If you read it in a > > mailing list or the Web, it has been stolen from Usenet.) > > I'm curious as to the copyright protections available to you, but if > you're going to threaten python-list's owners with legal action for > daring to rebroadcast a public post, I would have to recommend that > you get promptly banned from the list in order to reduce liability. Stefan had been banned for years because of this issue. In mid-2021 (wow, that long ago? I thought that was a lot more recently) his posts started to appear again. I don't know if this was a concious decision of the moderators, a technical error or whether Stefan changed something which caused him to escape the filter. I don't think it matters. If someone posts to a Usenet group with the full knowledge that it is gatewayed to a mailing list it is unreasonable to expect the moderators of the list to jump through hoops to comply with wishes hidden away in a non-standard header. > What's so bad about mailing lists that you don't want your messages to > be seen on them? He's Stefan Ram :-). hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From hjp-python at hjp.at Sun Jan 29 09:33:01 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 29 Jan 2023 15:33:01 +0100 Subject: Usenet vs. Mailing-list In-Reply-To: References: <87v8krvgcl.fsf@bsb.me.uk> <20230128143059.cbigz4ad4uklqtbk@hjp.at> <87fsbuumyv.fsf@bsb.me.uk> Message-ID: <20230129143301.2syjtp3fsdljdhyl@hjp.at> On 2023-01-29 02:09:28 -0000, Jon Ribbens via Python-list wrote: > I'm not aware of any significant period in the last twenty-one years > that [the gateway] > hasn't been working. Although sometimes it does feel like it isn't, in > that I reply to a post with an answer and then several other people > reply significantly later with the same answer, as if my one had never > existed... That's just because people don't read before they post. Happens in any usenet group or mailing list (and probably in web forums, too; but I don't really use those). I have to admit that I'm sometimes guilty of this behaviour, too. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From codewizard at gmail.com Sat Jan 28 23:47:05 2023 From: codewizard at gmail.com (Igor Berger) Date: Sat, 28 Jan 2023 20:47:05 -0800 (PST) Subject: Usenet vs. Mailing-list In-Reply-To: <87357uujoz.fsf@bsb.me.uk> References: <87v8krvgcl.fsf@bsb.me.uk> <20230128143059.cbigz4ad4uklqtbk@hjp.at> <87fsbuumyv.fsf@bsb.me.uk> <87357uujoz.fsf@bsb.me.uk> Message-ID: <770d27de-222c-40a4-b624-0b7a84561b0bn@googlegroups.com> On Saturday, January 28, 2023 at 10:02:57 PM UTC-5, Ben Bacarisse wrote: > Jon Ribbens writes: > > > On 2023-01-29, Ben Bacarisse wrote: > >> "Peter J. Holzer" writes: > >> > >>> On 2023-01-27 21:04:58 +0000, Ben Bacarisse wrote: > >>>> Mut... at dastardlyhq.com writes: > >>>> > >>>> > Hi > >>>> > >>>> It looks like you posted this question via Usenet. comp.lang.python is > >>>> essentially dead as a Usenet group. It exists, and gets NNTP versions > >>>> of mail sent to the mailing list, but nothing posted to the group via > >>>> NNTP get send on the mailing list. > >>> > >>> This is wrong. I did get Muttley's any your postings via the > >>> mailing-list. > >> > >> Ah, OK. I thought that was the case but I am obviously wrong. Has > >> there been a change, or have I been wrong for a long time!? > > > > I'm not aware of any significant period in the last twenty-one years > > that it hasn't been working. Although sometimes it does feel like it > > isn't, in that I reply to a post with an answer and then several > > other people reply significantly later with the same answer, as if > > my one had never existed... but whenever I check into it, my message > > has actually always made it to the list. > I have had the same experience and, as a result, I rarely post. Maybe > what I have to say is simply not interesting! > > -- > Ben. If I remember correctly, multiple regulars that use the mailing list mentioned that they "killfiled" posts originating from Google groups. This may contribute to such situations. From jfong at ms4.hinet.net Sun Jan 29 01:42:03 2023 From: jfong at ms4.hinet.net (Jach Feng) Date: Sat, 28 Jan 2023 22:42:03 -0800 (PST) Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: Jach Feng ? 2023?1?22? ?????11:11:22 [UTC+8] ?????? > Fail on command line, > > e:\Works\Python>py infix2postfix.py "-4^2+5.3*abs(-2-1)/2" > usage: infix2postfix.py [-h] [infix] > infix2postfix.py: error: unrecognized arguments: -4^2+5.3*abs(-2-1)/2 > > Also fail in REPL, > > e:\Works\Python>py > Python 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:08:11) [MSC v.1928 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import argparse > >>> parser = argparse.ArgumentParser(description='Convert infix notation to postfix') > >>> parser.parse_args("-4^2+5.3*abs(-2-1)/2") > usage: [-h] > : error: unrecognized arguments: - 4 ^ 2 + 5 . 3 * a b s ( - 2 - 1 ) / 2 > > Just can't figure out where is wrong!? > > --Jach OK, I take a quick try to use argv directly. I write the code in a way even get rid of the -h option. import sys if len(sys.argv) == 1: ....infix = " " ....print("Usage: a math equation must follow!") else: ....infix = "".join(sys.argv[1:]) Simple enough, right? But unfortunately it didn't survive. The ^ symbol was lost! e:\Works\Python>py infix2postfix.py Usage: a math equation must follow! e:\Works\Python>py infix2postfix.py -4^2 +5.3 * abs(-2-1)/2 -42 5.3 -2 1 - abs * 2 / + Hmm... From dfnsonfsduifb at gmx.de Sun Jan 29 04:05:31 2023 From: dfnsonfsduifb at gmx.de (Johannes Bauer) Date: Sun, 29 Jan 2023 10:05:31 +0100 Subject: Evaluation of variable as f-string In-Reply-To: References: <8ebc73d2-e14a-b9ad-fa35-5078e5f9fd6c@btinternet.com> Message-ID: Am 29.01.23 um 02:09 schrieb Chris Angelico: >>> The exact same points have already been made, but not listened to. >>> Sometimes, forceful language is required in order to get people to >>> listen. >> >> An arrogant bully's rationale. Personally, I'm fine with it. I've been >> to Usenet for a long time, in which this way of "educating" people was >> considered normal. But I do think it creates a deterring, toxic >> environment and reflects back to you as a person negatively. > > Arrogant bully? Or someone who has tried *multiple times* to explain > to you that what you're asking for is IMPOSSIBLE, and you need to ask > a better question if you want a better answer? In literally your first answer you resorted to aggressive language and implied that what I asked wasn't what I actually wanted. It was. Also note that in your first answer you did not answer "sorry, this is not possible", which would have been completely sufficient as an answer. Instead you tried your best at guesswork, implying I didn't know what I was doing. So, yes, absolutely toxic behavior. I fully stand by that judgement of mine. I'll go a step further and again repeat that THIS sort of behavior is what gives open source forums a bad rep. There's always a Lennart Poettering, an Ulrich Drepper or maybe a Chris Angelico around who may have great technical skill but think they can treat people like shit. > If that's "bullying", then fine, ban me for bullying, and go find > somewhere else where you'll be coddled and told that your question is > fine, it's awesome, and yes, wouldn't it be nice if magic were a > thing. LOL, "ban you"? What the heck are you talking about, my friend? I don't need to be coddled by you. I'm trying to give you the favor of honest feedback, which is that you sound like an utter bully. If you don't care, that is totally fine by me. > They're not different things, because what you asked for is NOT > POSSIBLE without the caveats that I gave. It is *fundamentally not > possible* to "evaluate a string as if it were an f-string", other than > by wrapping it in an f-string and evaluating it - with the > consequences of that. Yeah that sucks, unfortunately. But I'll live. >> In other words, if there were a magic function: >> >> evalfstring(s, x = x) >> >> That would have been the ideal answer. There does not seem to be one, >> however. So I'm back to silly workarounds. > > Right. Exactly. Now if you'd asked for what you REALLY need, maybe > there'd be a solution involving format_map, but no, you're so utterly > intransigent that you cannot adjust your question to fit reality. Does format_map do exactly what f-strings can do? Can I pass arbitrary functions and Python expressions insode a format_map? No? Of course not. Then it does not answer the question. > If that makes me a bad guy, then fine. I'll be the bad guy. Awww, it's adorable how you're trying to frame yourself as the victim. I'll be here if you need a hug, buddy. > But you're not going to change the laws of physics. Yeah we'll have to disagree about the fact that it's the "laws of physics" preventing a specific implementation of a Python function. Cheers, Johannes From dfnsonfsduifb at gmx.de Sun Jan 29 04:18:00 2023 From: dfnsonfsduifb at gmx.de (Johannes Bauer) Date: Sun, 29 Jan 2023 10:18:00 +0100 Subject: Evaluation of variable as f-string In-Reply-To: References: <5bb2546d-8c16-262d-9336-5b41cbf992b5@tompassin.net> <9db7a5e1-076c-a9aa-aee4-e044e45decb7@tompassin.net> <8ca25af2-8a5b-b915-4706-da1398d56d24@tompassin.net> Message-ID: Am 29.01.23 um 05:27 schrieb Thomas Passin: > Well, yes, we do see that.? What we don't see is what you want to > accomplish by doing it, and why you don't seem willing to accept some > restrictions on the string fragments so that they will evaluate correctly. I'll have to accept the restrictions. That's a good enough answer for me, actually. I was just thinking that possibly there's something like (made-up code): x = { "foo": "bar" } fstr = string.fstring_compile(s) fstr.eval(x = x) Which I didn't know about. It would make sense to me, but possibly not enough of a usecase to make it into Python. The format() flavors do not > IOW, perhaps there is a more practical way to accomplish what you want. > Except that we don't know what that is. Well, I don't know. I pretty much want a generic Python mechanism that allows for exactly what f-strings do: execute arbitrary Python snippets of code and format them in one go. In other words, I want to be able to do things like that, given an *arbitrary* dictionary x and a string s (which has the only restriction that its content needs to be vald f-string grammar): x = { "d": 12, "t": 12345, "dt": datetime.datetime, "td": datetime.timedelta } s = "{x['d']:09b} {'->' * (x['d'] // 3)} {(x['dt'](2000, 1, x['d']) + x['td'](120)).strftime('%y.%m.%d')} {'<-' * (x['d'] // 4)}" q = magic_function(s, x = x) and have "q" then be '000001100 ->->->-> 00.05.11 <-<-<-' I believe the closest solution would be using a templating mechanism (like Mako), but that has slightly different syntax and doesn't do string formatting as nice as f-strings do. f-strings really are the perfect syntax for what I want to do. Cheers, Johannes From auriocus at gmx.de Sun Jan 29 06:09:59 2023 From: auriocus at gmx.de (Christian Gollwitzer) Date: Sun, 29 Jan 2023 12:09:59 +0100 Subject: Evaluation of variable as f-string In-Reply-To: References: Message-ID: Am 28.01.23 um 02:56 schrieb Thomas Passin: > On 1/27/2023 5:10 PM, Christian Gollwitzer wrote: >> Am 27.01.23 um 21:43 schrieb Johannes Bauer: >>> I don't understand why you fully ignore literally the FIRST example I >>> gave in my original post and angrily claim that you solution works >>> when it does not: >>> >>> x = { "y": "z" } >>> s = "-> {x['y']}" >>> print(s.format(x = x)) >>> Traceback (most recent call last): >>> ?? File "", line 1, in >>> KeyError: "'y'" >>> >>> This. Does. Not. Work. >> >> It's because "you're holding it wrong!". Notice the error message; it >> says that the key 'y' does not exist. >> >> >> (base) Apfelkiste:Abschlussmeeting chris$ ipython >> Python 3.8.8 (default, Apr 13 2021, 12:59:45) >> Type 'copyright', 'credits' or 'license' for more information >> IPython 7.22.0 -- An enhanced Interactive Python. Type '?' for help. >> >> In [1]: x = { "y": "z" } >> >> In [2]: s = "-> {x[y]}" >> >> In [3]: print(s.format(x = x)) >> -> z >> >> In [4]: >> >> ?????Christian > > Oops, that's not quite what he wrote. > > You: s = "-> {x[y]}"??? # Works > Him: s = "-> {x['y']}"? # Fails > You might want to reconsider why I could have possibly written this message.... Christian From nntp.mbourne at spamgourmet.com Sun Jan 29 08:58:20 2023 From: nntp.mbourne at spamgourmet.com (Mark Bourne) Date: Sun, 29 Jan 2023 13:58:20 +0000 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: Jach Feng wrote: > Jach Feng ? 2023?1?22? ?????11:11:22 [UTC+8] ?????? >> Fail on command line, >> >> e:\Works\Python>py infix2postfix.py "-4^2+5.3*abs(-2-1)/2" >> usage: infix2postfix.py [-h] [infix] >> infix2postfix.py: error: unrecognized arguments: -4^2+5.3*abs(-2-1)/2 >> >> Also fail in REPL, >> >> e:\Works\Python>py >> Python 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:08:11) [MSC v.1928 32 bit (Intel)] on win32 >> Type "help", "copyright", "credits" or "license" for more information. >>>>> import argparse >>>>> parser = argparse.ArgumentParser(description='Convert infix notation to postfix') >>>>> parser.parse_args("-4^2+5.3*abs(-2-1)/2") >> usage: [-h] >> : error: unrecognized arguments: - 4 ^ 2 + 5 . 3 * a b s ( - 2 - 1 ) / 2 >> >> Just can't figure out where is wrong!? >> >> --Jach > OK, I take a quick try to use argv directly. I write the code in a way even get rid of the -h option. > > import sys > if len(sys.argv) == 1: > ....infix = " " > ....print("Usage: a math equation must follow!") > else: > ....infix = "".join(sys.argv[1:]) > > Simple enough, right? But unfortunately it didn't survive. The ^ symbol was lost! > > e:\Works\Python>py infix2postfix.py > Usage: a math equation must follow! > > e:\Works\Python>py infix2postfix.py -4^2 +5.3 * abs(-2-1)/2 > -42 5.3 -2 1 - abs * 2 / + > > Hmm... I'm not certain, but I think that's a shell feature again. In Windows' command prompt, "^" can be used at the end of a line to indicate that the command continues on the next line. I'm not sure what happens if it's in the middle of a line (and not on Windows to be able to check), but it's quite possible that it just gets ignored. Enclosing your argument in quotes might help. Again, this is a feature of the shell, not your application, and other shells might behave differently. You probably don't want to go down the line of trying to document this kind of thing in your applications usage information, because it won't work like that for someone using e.g. the bash shell (which can be run on Windows). -- Mark. From ben.usenet at bsb.me.uk Sun Jan 29 09:29:06 2023 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Sun, 29 Jan 2023 14:29:06 +0000 Subject: Usenet vs. Mailing-list References: <87v8krvgcl.fsf@bsb.me.uk> <20230128143059.cbigz4ad4uklqtbk@hjp.at> <87fsbuumyv.fsf@bsb.me.uk> <87357uujoz.fsf@bsb.me.uk> <770d27de-222c-40a4-b624-0b7a84561b0bn@googlegroups.com> Message-ID: <87ilgptnwt.fsf@bsb.me.uk> Igor Berger writes: > On Saturday, January 28, 2023 at 10:02:57 PM UTC-5, Ben Bacarisse wrote: >> Jon Ribbens writes: >> >> > On 2023-01-29, Ben Bacarisse wrote: >> >> "Peter J. Holzer" writes: >> >> >> >>> On 2023-01-27 21:04:58 +0000, Ben Bacarisse wrote: >> >>>> Mut... at dastardlyhq.com writes: >> >>>> >> >>>> > Hi >> >>>> >> >>>> It looks like you posted this question via Usenet. comp.lang.python is >> >>>> essentially dead as a Usenet group. It exists, and gets NNTP versions >> >>>> of mail sent to the mailing list, but nothing posted to the group via >> >>>> NNTP get send on the mailing list. >> >>> >> >>> This is wrong. I did get Muttley's any your postings via the >> >>> mailing-list. >> >> >> >> Ah, OK. I thought that was the case but I am obviously wrong. Has >> >> there been a change, or have I been wrong for a long time!? >> > >> > I'm not aware of any significant period in the last twenty-one years >> > that it hasn't been working. Although sometimes it does feel like it >> > isn't, in that I reply to a post with an answer and then several >> > other people reply significantly later with the same answer, as if >> > my one had never existed... but whenever I check into it, my message >> > has actually always made it to the list. >> >> I have had the same experience and, as a result, I rarely post. Maybe >> what I have to say is simply not interesting! >> > > If I remember correctly, multiple regulars that use the mailing list > mentioned that they "killfiled" posts originating from Google groups. > This may contribute to such situations. Ah, that may explain most of my confusion. I know I stopped posting because there didn't seem to be much engagement, but maybe it was just many list members choosing not to see my posts rather than all list members not being able to see my posts. As I said, I did check to see if I could find a reply to a Usenet-originated post from a list member but my not being able to find one could just be down to many list members filtering Usenet posts. It's clearly not universal. -- Ben. From nntp.mbourne at spamgourmet.com Sun Jan 29 09:39:00 2023 From: nntp.mbourne at spamgourmet.com (Mark Bourne) Date: Sun, 29 Jan 2023 14:39:00 +0000 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <7118e963-7594-4d95-97bf-e15cea2c2bfen@googlegroups.com> References: <6bdc1904-4f0b-4c05-a67d-b0965b30624dn@googlegroups.com> <7118e963-7594-4d95-97bf-e15cea2c2bfen@googlegroups.com> Message-ID: Jach Feng wrote: > Thank you for detail explanation of the role the shell is involved in this problem. I'm very appreciated! > > It seems that a CLI app may become very complex when dealing with different kind of shell, and may not be possible to solve its problem. But the good thing in my app is that I need only to handle math equation:-) If you want to try to tell the user how to deal with their shell's requirements for quoting arguments, regardless of which shell they might be using, yes, that explanation would become very complicated. It doesn't affect the rest of the implementation of the application though - the user just needs to know how to use their shell to pass the arguments they want into the application. That's really something they should look up in their shell's documentation, rather than something your application should attempt to document. Since your application requires equations to be passed in, and they're quite likely to include characters handled specially by the shell (space, "*" and "^" have already come up in this thread, but there may be others), it may be worth highlighting that, but directing the user to consult the documentation for their shell rather than assuming a particular shell and attempting to cover all its features and limitations. >> So why so much objection to explaining the need for "--"? > Because of using " to enclose a space separated string is a common convention, and adding a "--" is not:-) If you don't consider use of "--" to be a common convention, that seems even more reason to mention it in your application's documentation. Telling the user to use quotes around an argument containing spaces is nothing to do with your application, and might not even be applicable if they use a different shell to call your application. In most shells I've come across, there are also various other characters that need special handling (either quoting or escaping) - but exactly which characters again depends on the shell. Yet you seem quite happy to document that one particular case in your usage information. Using "--" is also a common convention (as I and others have mentioned), although perhaps not as common in Windows (where it's more common to use "/" rather than "-" for options anyway). But more to the point, it is a feature that is implemented under your application's control (if you don't want this feature, don't use argparse). Use of "--" is applicable regardless of which shell your user calls it from, and other applications might not use that convention even if called from the same shell, so it seems *more* in scope for your application to document than using quotes around spaces. -- Mark. From jon+usenet at unequivocal.eu Sun Jan 29 10:26:43 2023 From: jon+usenet at unequivocal.eu (Jon Ribbens) Date: Sun, 29 Jan 2023 15:26:43 -0000 (UTC) Subject: Usenet vs. Mailing-list References: <87v8krvgcl.fsf@bsb.me.uk> <20230128143059.cbigz4ad4uklqtbk@hjp.at> <87fsbuumyv.fsf@bsb.me.uk> <20230129143301.2syjtp3fsdljdhyl@hjp.at> Message-ID: On 2023-01-29, Peter J. Holzer wrote: > On 2023-01-29 02:09:28 -0000, Jon Ribbens via Python-list wrote: >> I'm not aware of any significant period in the last twenty-one years >> that > [the gateway] >> hasn't been working. Although sometimes it does feel like it isn't, in >> that I reply to a post with an answer and then several other people >> reply significantly later with the same answer, as if my one had never >> existed... > > That's just because people don't read before they post. > > Happens in any usenet group or mailing list (and probably in web forums, > too; but I don't really use those). I have to admit that I'm sometimes > guilty of this behaviour, too. Well, let's just assume for a moment that I'm familiar with Usenet and with mailing lists ;-) This sort of replying-without-reading seems to happen on comp.lang.python/python-list more than usual. From ethan at stoneleaf.us Sun Jan 29 13:05:03 2023 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 29 Jan 2023 10:05:03 -0800 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: <9981dd10-e384-2882-c69a-55b48f17d3c8@stoneleaf.us> This thread has run its course and seems to now be generating more heat than light. It is now closed (at least on the Python List side). Thank you everyone for your participation and understanding. -- ~Ethan~ Moderator From list1 at tompassin.net Sun Jan 29 13:52:02 2023 From: list1 at tompassin.net (Thomas Passin) Date: Sun, 29 Jan 2023 13:52:02 -0500 Subject: Evaluation of variable as f-string In-Reply-To: References: Message-ID: On 1/29/2023 6:09 AM, Christian Gollwitzer wrote: > Am 28.01.23 um 02:56 schrieb Thomas Passin: >> On 1/27/2023 5:10 PM, Christian Gollwitzer wrote: >>> Am 27.01.23 um 21:43 schrieb Johannes Bauer: >>>> I don't understand why you fully ignore literally the FIRST example >>>> I gave in my original post and angrily claim that you solution works >>>> when it does not: >>>> >>>> x = { "y": "z" } >>>> s = "-> {x['y']}" >>>> print(s.format(x = x)) >>>> Traceback (most recent call last): >>>> ?? File "", line 1, in >>>> KeyError: "'y'" >>>> >>>> This. Does. Not. Work. >>> >>> It's because "you're holding it wrong!". Notice the error message; it >>> says that the key 'y' does not exist. >>> >>> >>> (base) Apfelkiste:Abschlussmeeting chris$ ipython >>> Python 3.8.8 (default, Apr 13 2021, 12:59:45) >>> Type 'copyright', 'credits' or 'license' for more information >>> IPython 7.22.0 -- An enhanced Interactive Python. Type '?' for help. >>> >>> In [1]: x = { "y": "z" } >>> >>> In [2]: s = "-> {x[y]}" >>> >>> In [3]: print(s.format(x = x)) >>> -> z >>> >>> In [4]: >>> >>> ?????Christian >> >> Oops, that's not quite what he wrote. >> >> You: s = "-> {x[y]}"??? # Works >> Him: s = "-> {x['y']}"? # Fails >> > You might want to reconsider why I could have possibly written this > message.... I might .. or I might wish you had actually said what you wanted to convey ... From hjp-python at hjp.at Sun Jan 29 19:10:50 2023 From: hjp-python at hjp.at (Peter J. Holzer) Date: Mon, 30 Jan 2023 01:10:50 +0100 Subject: Evaluation of variable as f-string In-Reply-To: References: <5bb2546d-8c16-262d-9336-5b41cbf992b5@tompassin.net> <9db7a5e1-076c-a9aa-aee4-e044e45decb7@tompassin.net> <8ca25af2-8a5b-b915-4706-da1398d56d24@tompassin.net> Message-ID: <20230130001050.qpb22dxa72c22pvj@hjp.at> On 2023-01-29 10:18:00 +0100, Johannes Bauer wrote: > Am 29.01.23 um 05:27 schrieb Thomas Passin: > > IOW, perhaps there is a more practical way to accomplish what you want. > > Except that we don't know what that is. > > Well, I don't know. I pretty much want a generic Python mechanism that > allows for exactly what f-strings do: execute arbitrary Python snippets of > code That exists. Use eval (or exec). > and format them in one go. Include an f-string in the code you eval. > In other words, I want to be able to do things like that, given an > *arbitrary* dictionary x and a string s As I wrote before: An f-string isn't a string. It's a grammatical construct. So you want to execute Python code which is what eval and exec do. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From cs at cskk.id.au Sun Jan 29 16:19:03 2023 From: cs at cskk.id.au (Cameron Simpson) Date: Mon, 30 Jan 2023 08:19:03 +1100 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: References: Message-ID: On 29Jan2023 07:12, 2QdxY4RzWzUUiLuE at potatochowder.com <2QdxY4RzWzUUiLuE at potatochowder.com> wrote: >On 2023-01-29 at 16:51:20 +1100, >Cameron Simpson wrote: >> They're unrelated. As others have mentioned, "--" is _extremely_ >> common; >> almost _all_ UNIX command like programmes which handle -* style options >> honour the "--" convention. _argparse_ itself honours that convention, as >> does getopt etc. > >And why do UNIX programs behave this way? > >Because POSIX says they should: > > https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html Aye, but POSIX says they should because POSIX formalises a lot of existing practices so that people have a reference to let them _also_ use that practice in a consistent way. I used to lurk on the POSIX discussion list, and the length of the discussions around corner cases was quite wearing, even just to read. Cheers, Cameron Simpson From cs at cskk.id.au Sun Jan 29 16:20:21 2023 From: cs at cskk.id.au (Cameron Simpson) Date: Mon, 30 Jan 2023 08:20:21 +1100 Subject: How to make argparse accept "-4^2+5.3*abs(-2-1)/2" string argument? In-Reply-To: <002301d933e3$1d44c820$57ce5860$@gmail.com> References: <002301d933e3$1d44c820$57ce5860$@gmail.com> Message-ID: On 29Jan2023 08:10, avi.e.gross at gmail.com wrote: >You are technically correct but perhaps off the mark. > >Yes, a python program only sees what is handed to it by some shell if invoked a certain way. > >The issue here is what you tell people using your program about what they need to type to get it to work. That means if their shell is going to make changes in what they typed, they need to know how to avoid unintended changes. As one example not mentioned, whitespace disappears if not somehow protected as in quotes. Hmm, there is that. But the OP needs clarity on what happens in a shell and what happens in a programme once the shell has invoked it for a user. Cheers, Cameron Simpson From list1 at tompassin.net Sun Jan 29 23:57:51 2023 From: list1 at tompassin.net (Thomas Passin) Date: Sun, 29 Jan 2023 23:57:51 -0500 Subject: evaluation question In-Reply-To: References: Message-ID: On 1/29/2023 4:15 PM, elvis-85792 at notatla.org.uk wrote: > On 2023-01-28, Louis Krupp wrote: >> On 1/27/2023 9:37 AM, Muttley at dastardlyhq.com wrote: > > >>>>>> eval("print(123)") >>> 123 > > > Does OP expect the text to come from the eval or from the print? > >>>> x = print( [i for i in range(1, 10)] ) > [1, 2, 3, 4, 5, 6, 7, 8, 9] > >>>> x > (nothing printed) Because print() returns nothing (i.e., the statement x is None is True). Other common constructs that return nothing are append(), sort(), and add(). It can be easy to forget this and write l2 = l1.sort() # l2 == None OTOH, you can (by slightly abusing the lambda) use this behavior to make a lambda expression print what it's receiving: >>> y = lambda x: print(f'Got {x}') or x**2 >>> z = y(3) Got 3 >>> z 9 >>> From grant.b.edwards at gmail.com Mon Jan 30 11:24:58 2023 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Mon, 30 Jan 2023 08:24:58 -0800 (PST) Subject: Usenet vs. Mailing-list References: <87v8krvgcl.fsf@bsb.me.uk> <20230128143059.cbigz4ad4uklqtbk@hjp.at> Message-ID: <63d7ef5a.050a0220.d1c5.6a7b@mx.google.com> On 2023-01-28, Chris Green wrote: > As far as I am aware the mirroring of the Python mailing list on > comp.lan.python works perfectly. I love gmane! :-) If gmane stopped working, I'd have to retire and give up on computers. I supposed I might be able to hammer procmail and mutt into something tolerable, but slrn pointed at gmane works wonderfully without any futzing. NNTP and newsreaders are designed specifically for the task to which people have coopted e-mail into via mailing lists. -- Grant From grant.b.edwards at gmail.com Mon Jan 30 11:29:27 2023 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Mon, 30 Jan 2023 08:29:27 -0800 (PST) Subject: Usenet vs. Mailing-list References: <87v8krvgcl.fsf@bsb.me.uk> <20230128143059.cbigz4ad4uklqtbk@hjp.at> <5d5bth9bkjs3bt0ulo7djuvp7m0f6t33c5@4ax.com> Message-ID: <63d7f067.050a0220.21432.6a70@mx.google.com> On 2023-01-28, Dennis Lee Bieber wrote: > On Sat, 28 Jan 2023 20:07:44 +0000, Chris Green declaimed the > following: > > >>As far as I am aware the mirroring of the Python mailing list on >>comp.lan.python works perfectly. I love gmane! :-) > > Is gmane's gmane.comp.python.general allowing posts to go through > again? I had to revert to comp.lang.python some time back when gmane kept > rejecting outgoing posts. No. FWIW, it's the mailing list that's blocking them, not Gmane. That's why I wrote this: https://github.com/GrantEdwards/hybrid-inews It's an inews work-alike that submits most posts via gmanes NNTP server, but will deal with particular groups (e.g. gmane.comp.python.general) that want posts submitted via email. It allows me to continue to read (and post to) the Python mailling list via slrn pointed at gmane. It is, of course, written in Python. From Muttley at dastardlyhq.com Mon Jan 30 04:41:03 2023 From: Muttley at dastardlyhq.com (Muttley at dastardlyhq.com) Date: Mon, 30 Jan 2023 09:41:03 -0000 (UTC) Subject: evaluation question References: Message-ID: On Sun, 29 Jan 2023 23:57:51 -0500 Thomas Passin wrote: >On 1/29/2023 4:15 PM, elvis-85792 at notatla.org.uk wrote: >> On 2023-01-28, Louis Krupp wrote: >>> On 1/27/2023 9:37 AM, Muttley at dastardlyhq.com wrote: >> >> >>>>>>> eval("print(123)") >>>> 123 >> >> >> Does OP expect the text to come from the eval or from the print? >> >>>>> x = print( [i for i in range(1, 10)] ) >> [1, 2, 3, 4, 5, 6, 7, 8, 9] >> >>>>> x >> (nothing printed) > >Because print() returns nothing (i.e., the statement x is None is True). I don't understand this. What was the point of the upheaval of converting the print command in python 2 into a function in python 3 if as a function print() doesn't return anything useful? Surely even the length of the formatted string as per C's sprintf() function would be helpful? From rob.cliffe at btinternet.com Mon Jan 30 14:00:04 2023 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 30 Jan 2023 19:00:04 +0000 Subject: evaluation question In-Reply-To: References: Message-ID: <322a101c-53c0-a267-c910-991f3db43841@btinternet.com> On 30/01/2023 09:41, Muttley at dastardlyhq.com wrote: > On Sun, 29 Jan 2023 23:57:51 -0500 > Thomas Passin wrote: >> On 1/29/2023 4:15 PM, elvis-85792 at notatla.org.uk wrote: >>> On 2023-01-28, Louis Krupp wrote: >>>> On 1/27/2023 9:37 AM, Muttley at dastardlyhq.com wrote: >>> >>>>>>>> eval("print(123)") >>>>> 123 >>> >>> Does OP expect the text to come from the eval or from the print? >>> >>>>>> x = print( [i for i in range(1, 10)] ) >>> [1, 2, 3, 4, 5, 6, 7, 8, 9] >>> >>>>>> x >>> (nothing printed) >> Because print() returns nothing (i.e., the statement x is None is True). > I don't understand this. What was the point of the upheaval of converting > the print command in python 2 into a function in python 3 if as a function > print() doesn't return anything useful? Surely even the length of the > formatted string as per C's sprintf() function would be helpful? > That's a fair question, or rather 2 fair questions. There is an explanation of why the change was made at ??? https://snarky.ca/why-print-became-a-function-in-python-3/ In brief: (a) the print() function is more flexible and can be used in expressions ?? ??? ??? ? ? (b) Python's syntax was simplified by dropping the special syntax used by the print statement. sys.stdout.write() does return the number of characters output (you could use this instead of print() if you need this; remember to add a '\n' character at the end of? a line).? I guess the option of making print() do the same either was not considered, or was rejected, when print was made a function. Best wishes Rob Cliffe From __peter__ at web.de Mon Jan 30 17:20:26 2023 From: __peter__ at web.de (Peter Otten) Date: Mon, 30 Jan 2023 23:20:26 +0100 Subject: Custom help format for a choice argparse argument In-Reply-To: References: Message-ID: <380526f2-2dd0-8de4-114a-5be4acc91fb3@web.de> On 27/01/2023 21:31, Ivan "Rambius" Ivanov wrote: > Hello, > > I am developing a script that accepts a time zone as an option. The > time zone can be any from pytz.all_timezones. I have > > def main(): > parser = argparse.ArgumentParser() > parser.add_argument("-z", "--zone", choices=pytz.all_timezones) > args = parser.parse_args() > print(args) > print(f"Specified timezone: {args.zone}") > > It works, but when I run it with the -h option it dumps all entries in > pytz.all_timezones. I would like to modify the help format for just > -z|--zone option. I read the docs about HelpFormatter and argparse.py > and I ended up with > > class CustomHelpFormatter(argparse.HelpFormatter): > def _metavar_formatter(self, action, default_metavar): > if action.dest == 'zone': > result = 'zone from pytz.all_timezones' > def format(tuple_size): > if isinstance(result, tuple): > return result > else: > return (result, ) * tuple_size > return format > else: > return super(CustomHelpFormatter, > self)._metavar_formatter(action, default_metavar) > > > def main(): > parser = argparse.ArgumentParser(formatter_class=CustomHelpFormatter) > parser.add_argument("-z", "--zone", choices=pytz.all_timezones) > args = parser.parse_args() > print(args) > print(f"Specified timezone: {args.zone}") > > This works, but is there a more elegant way to achieve it? It may be sufficient to specify a metavar: >>> import argparse >>> p = argparse.ArgumentParser() >>> p.add_argument("--foo", choices="alpha beta gamma".split(), metavar="") [...] >>> p.parse_args(["-h"]) usage: [-h] [--foo ] optional arguments: -h, --help show this help message and exit --foo While that helps with --help it doesn't always prevent the choices list from being spelt out: >>> p.parse_args(["--foo", "whatever"]) usage: [-h] [--foo ] : error: argument --foo: invalid choice: 'whatever' (choose from 'alpha', 'beta', 'gamma') From greg.ewing at canterbury.ac.nz Mon Jan 30 18:57:33 2023 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 31 Jan 2023 12:57:33 +1300 Subject: evaluation question In-Reply-To: References: Message-ID: On 30/01/23 10:41 pm, Muttley at dastardlyhq.com wrote: > What was the point of the upheaval of converting > the print command in python 2 into a function in python 3 if as a function > print() doesn't return anything useful? It was made a function because there's no good reason for it to have special syntax in the language. Functions don't need to return things to justify their existence, and in fact the usual convention is that functions whose purpose is to have an effect just return None. -- Greg From as at sci.fi Tue Jan 31 05:08:18 2023 From: as at sci.fi (Anssi Saari) Date: Tue, 31 Jan 2023 12:08:18 +0200 Subject: Usenet vs. Mailing-list In-Reply-To: <63d7f067.050a0220.21432.6a70@mx.google.com> (Grant Edwards's message of "Mon, 30 Jan 2023 08:29:27 -0800 (PST)") References: <87v8krvgcl.fsf@bsb.me.uk> <20230128143059.cbigz4ad4uklqtbk@hjp.at> <5d5bth9bkjs3bt0ulo7djuvp7m0f6t33c5@4ax.com> <63d7f067.050a0220.21432.6a70@mx.google.com> Message-ID: Grant Edwards writes: > No. FWIW, it's the mailing list that's blocking them, not Gmane. > > That's why I wrote this: > > https://github.com/GrantEdwards/hybrid-inews > > It's an inews work-alike that submits most posts via gmanes NNTP > server, but will deal with particular groups > (e.g. gmane.comp.python.general) that want posts submitted via email. > > It allows me to continue to read (and post to) the Python mailling > list via slrn pointed at gmane. Interesting. In Gnus it was just a couple of settings to make it understand that in this group (i.e. gmane group gmane.comp.python.general) posts and follow-ups should be sent via mail to the mailing list address. From grant.b.edwards at gmail.com Tue Jan 31 09:11:05 2023 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Tue, 31 Jan 2023 06:11:05 -0800 (PST) Subject: Usenet vs. Mailing-list References: <87v8krvgcl.fsf@bsb.me.uk> <20230128143059.cbigz4ad4uklqtbk@hjp.at> <5d5bth9bkjs3bt0ulo7djuvp7m0f6t33c5@4ax.com> <63d7f067.050a0220.21432.6a70@mx.google.com> Message-ID: <63d92179.920a0220.39bbe.0a0b@mx.google.com> On 2023-01-31, Anssi Saari wrote: > Grant Edwards writes: > >> No. FWIW, it's the mailing list that's blocking them, not Gmane. >> >> That's why I wrote this: >> >> https://github.com/GrantEdwards/hybrid-inews >> >> It's an inews work-alike that submits most posts via gmanes NNTP >> server, but will deal with particular groups >> (e.g. gmane.comp.python.general) that want posts submitted via email. >> >> It allows me to continue to read (and post to) the Python mailling >> list via slrn pointed at gmane. > > Interesting. In Gnus it was just a couple of settings to make it > understand that in this group (i.e. gmane group > gmane.comp.python.general) posts and follow-ups should be sent via mail > to the mailing list address. Ah, slrn is missing that feature. It will let you e-mail a response instead of posting a followup (IIRC, you hit 'r' instead of 'f'), but when you do that it sends the e-mail to the poster rather than to the list. You can edit the To: header to fix that, but that's too much to remember. :) I looked into adding such a feature to slrn, but it was far easier to switch the slrn setting so that it posted via an external inews utility instead of the built-in NNTP client. Then all I needed was a smarter 'inews'. Thanks the the NNTP and SMTP libraries in Python, it took only 50 lines of Python to implement an external inews utility that routed postings to either gname via NNTP or list-servers via SMTP depending on the group. -- Grant From Muttley at dastardlyhq.com Tue Jan 31 04:24:10 2023 From: Muttley at dastardlyhq.com (Muttley at dastardlyhq.com) Date: Tue, 31 Jan 2023 09:24:10 -0000 (UTC) Subject: evaluation question References: Message-ID: On Tue, 31 Jan 2023 12:57:33 +1300 Greg Ewing wrote: >On 30/01/23 10:41 pm, Muttley at dastardlyhq.com wrote: >> What was the point of the upheaval of converting >> the print command in python 2 into a function in python 3 if as a function >> print() doesn't return anything useful? > >It was made a function because there's no good reason for it >to have special syntax in the language. All languages have their ugly corners due to initial design mistakes and/or constraints. Eg: java with the special behaviour of its string class, C++ with "=0" pure virtual declaration. But they don't dump them and make all old code suddenly cease to execute. Pragmatism should always come before language purity. From learn2program at gmail.com Tue Jan 31 13:07:30 2023 From: learn2program at gmail.com (Alan Gauld) Date: Tue, 31 Jan 2023 18:07:30 +0000 Subject: Usenet vs. Mailing-list (was: evaluation question) In-Reply-To: <255bth98ettkgisai1fgb1lo47eeiqg4qt@4ax.com> References: <87v8krvgcl.fsf@bsb.me.uk> <20230128143059.cbigz4ad4uklqtbk@hjp.at> <255bth98ettkgisai1fgb1lo47eeiqg4qt@4ax.com> Message-ID: On 28/01/2023 21:36, Dennis Lee Bieber wrote: > Now -- last time I checked the gmane server says posting is prohibited. > I used to use gmane as it retrieved directly from the mailing list I still use gmane but its no posting thing is a pain because responding (or posting new stuff) is a now more complicated than before. So I have to be very highly motivated to jump through the hoops. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From list1 at tompassin.net Tue Jan 31 13:15:40 2023 From: list1 at tompassin.net (Thomas Passin) Date: Tue, 31 Jan 2023 13:15:40 -0500 Subject: evaluation question In-Reply-To: References: Message-ID: <674a1779-ea08-ef69-2a2d-b8f2dd924d07@tompassin.net> On 1/31/2023 4:24 AM, Muttley at dastardlyhq.com wrote: > On Tue, 31 Jan 2023 12:57:33 +1300 > Greg Ewing wrote: >> On 30/01/23 10:41 pm, Muttley at dastardlyhq.com wrote: >>> What was the point of the upheaval of converting >>> the print command in python 2 into a function in python 3 if as a function >>> print() doesn't return anything useful? >> >> It was made a function because there's no good reason for it >> to have special syntax in the language. > > All languages have their ugly corners due to initial design mistakes and/or > constraints. Eg: java with the special behaviour of its string class, C++ > with "=0" pure virtual declaration. But they don't dump them and make all old > code suddenly cease to execute. > > Pragmatism should always come before language purity. > It was more fundamental than that, and not mainly about print(): https://snarky.ca/why-python-3-exists/ From grant.b.edwards at gmail.com Tue Jan 31 13:24:11 2023 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Tue, 31 Jan 2023 10:24:11 -0800 (PST) Subject: Usenet vs. Mailing-list (was: evaluation question) References: <87v8krvgcl.fsf@bsb.me.uk> <20230128143059.cbigz4ad4uklqtbk@hjp.at> <255bth98ettkgisai1fgb1lo47eeiqg4qt@4ax.com> Message-ID: <63d95ccb.050a0220.f6a8b.4dd0@mx.google.com> On 2023-01-31, Alan Gauld wrote: > On 28/01/2023 21:36, Dennis Lee Bieber wrote: > >> Now -- last time I checked the gmane server says posting is prohibited. >> I used to use gmane as it retrieved directly from the mailing list > > I still use gmane but its no posting thing is a pain because responding > (or posting new stuff) is a now more complicated than before. So > I have to be very highly motivated to jump through the hoops. If your newsreader can post via an external inews utility, you can use this Python implementation of inews to post to specified groups via e-mail instead of NNTP: https://github.com/GrantEdwards/hybrid-inews From avi.e.gross at gmail.com Tue Jan 31 14:59:49 2023 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Tue, 31 Jan 2023 14:59:49 -0500 Subject: evaluation question In-Reply-To: <674a1779-ea08-ef69-2a2d-b8f2dd924d07@tompassin.net> References: <674a1779-ea08-ef69-2a2d-b8f2dd924d07@tompassin.net> Message-ID: <00a601d935ae$93e48930$bbad9b90$@gmail.com> I think its has been discussed here that many functions are DELIBERATELY designed to return without returning anything. Earlier languages like Pascal had explicit ideas that a function that did not return a value was declared as a "procedure" but many other languages like python make no real differentiation. Some functions are designed for a sort of side-effect and often there is nothing much that needs to be returned or even can be. If a function prints a dozen items one at a time, should it return nothing, or a copy of the last item or somehow of all items? Generally nothing looks right. If you want to return something, fine. Do it explicitly. Similar arguments have been made about methods that do things like sort the contents of an object internally and then return nothing. Some would like the return to be the (now altered) object itself. You can emulate that by not sorting internally but instead sorted(object) returns a new object that has been sorted from the old one. So should or could print return anything? Other languages exist, like R, that do return (and often ignore) whatever print displayed elsewhere. This can be of use in many ways such as making it easier to print or store additional copies without recalculating. My preference might be to simply allow a local option at the end of a print statement such as print(..., return=True) or even a way to set a global option so all print statements can be turned on when you want. But is this pythonic? In particular, people who want to give type hints now can safely claim it returns None and would have to modify that so it can optionally return something like str or None. And, of course, once you change print() this way, someone else will want the number of characters (or perhaps bytes) returned instead. Much of this can be worked around by simply making your own customized print function which evaluates the arguments to make a string and then calls print, perhaps with the results pre-calculated, and returns what you wanted. That is not as easy as it sounds, though as print supports various arguments like sep= and end= and file= and flush= so a weird but doable idea is simply to substitute a temporary file for any file= argument and write the results to a temporary file or something in memory that emulates a file. You can then read that back in and return what you want after handling the original print statement with the original arguments, or perhaps just use your result to any actually specified file or the default. You can thus create something like what you want and leave the original print() command alone to do what it was designed to do. And, in general, people who want a copy of what they print, often use other python functionality to craft some or all parts of the text they want printed and only then call print() and thus already may have the ability to use the text afterwards. For many purposes, including efficiency, returning nothing makes good sense. But it is not really the only choice or the right choice and yet, if you want to use THIS language, it has to be accepted as the documented choice. -----Original Message----- From: Python-list On Behalf Of Thomas Passin Sent: Tuesday, January 31, 2023 1:16 PM To: python-list at python.org Subject: Re: evaluation question On 1/31/2023 4:24 AM, Muttley at dastardlyhq.com wrote: > On Tue, 31 Jan 2023 12:57:33 +1300 > Greg Ewing wrote: >> On 30/01/23 10:41 pm, Muttley at dastardlyhq.com wrote: >>> What was the point of the upheaval of converting the print command >>> in python 2 into a function in python 3 if as a function >>> print() doesn't return anything useful? >> >> It was made a function because there's no good reason for it to have >> special syntax in the language. > > All languages have their ugly corners due to initial design mistakes > and/or constraints. Eg: java with the special behaviour of its string > class, C++ with "=0" pure virtual declaration. But they don't dump > them and make all old code suddenly cease to execute. > > Pragmatism should always come before language purity. > It was more fundamental than that, and not mainly about print(): https://snarky.ca/why-python-3-exists/ -- https://mail.python.org/mailman/listinfo/python-list From nntp.mbourne at spamgourmet.com Tue Jan 31 16:00:53 2023 From: nntp.mbourne at spamgourmet.com (Mark Bourne) Date: Tue, 31 Jan 2023 21:00:53 +0000 Subject: evaluation question In-Reply-To: References: Message-ID: Greg Ewing wrote: > On 30/01/23 10:41 pm, Muttley at dastardlyhq.com wrote: >> What was the point of the upheaval of converting >> the print command in python 2 into a function in python 3 if as a >> function >> print() doesn't return anything useful? > > It was made a function because there's no good reason for it > to have special syntax in the language. I think I saw somewhere that making print a function also had something to do with being able to add extra keyword arguments like sep and end. The syntax for printing to a specific file already seemed a bit odd with the print statement, and adding extra arguments would have made it even more clunky (yeah, I know ">>" is similar to C++ streams, but it looks out of place in Python). They couldn't fully make the change from print statement to print function without breaking backward compatibility for existing code. But there were other breaking changes being made in Python 3 anyway, so may as well sort print out while at it and have all the breaking changes at once. > Functions don't need to return things to justify their existence, > and in fact the usual convention is that functions whose purpose > is to have an effect just return None. -- Mark. From gweatherby at uchc.edu Tue Jan 31 16:16:50 2023 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Tue, 31 Jan 2023 21:16:50 +0000 Subject: evaluation question In-Reply-To: <00a601d935ae$93e48930$bbad9b90$@gmail.com> References: <674a1779-ea08-ef69-2a2d-b8f2dd924d07@tompassin.net> <00a601d935ae$93e48930$bbad9b90$@gmail.com> Message-ID: import io def countprint(*args, **kwargs): capturekw = {k:v for k,v in kwargs.items() if k != 'file'} buffer = io.StringIO() capturekw['file'] = buffer print(*args,**kwargs) print(*args,**capturekw) return len(buffer.getvalue()) def boolprint(*args,active:bool, **kwargs): if active: print(*args,**kwargs) with open("text.txt",'w') as f: y = countprint(1, 3, 3, sep=',', end='\n\n',file=f) print(y) boolprint(3,4,5,sep='/',active=True) boolprint(7,11,active=False) From: Python-list on behalf of avi.e.gross at gmail.com Date: Tuesday, January 31, 2023 at 3:01 PM To: 'Thomas Passin' , python-list at python.org Subject: RE: evaluation question *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** I think its has been discussed here that many functions are DELIBERATELY designed to return without returning anything. Earlier languages like Pascal had explicit ideas that a function that did not return a value was declared as a "procedure" but many other languages like python make no real differentiation. Some functions are designed for a sort of side-effect and often there is nothing much that needs to be returned or even can be. If a function prints a dozen items one at a time, should it return nothing, or a copy of the last item or somehow of all items? Generally nothing looks right. If you want to return something, fine. Do it explicitly. Similar arguments have been made about methods that do things like sort the contents of an object internally and then return nothing. Some would like the return to be the (now altered) object itself. You can emulate that by not sorting internally but instead sorted(object) returns a new object that has been sorted from the old one. So should or could print return anything? Other languages exist, like R, that do return (and often ignore) whatever print displayed elsewhere. This can be of use in many ways such as making it easier to print or store additional copies without recalculating. My preference might be to simply allow a local option at the end of a print statement such as print(..., return=True) or even a way to set a global option so all print statements can be turned on when you want. But is this pythonic? In particular, people who want to give type hints now can safely claim it returns None and would have to modify that so it can optionally return something like str or None. And, of course, once you change print() this way, someone else will want the number of characters (or perhaps bytes) returned instead. Much of this can be worked around by simply making your own customized print function which evaluates the arguments to make a string and then calls print, perhaps with the results pre-calculated, and returns what you wanted. That is not as easy as it sounds, though as print supports various arguments like sep= and end= and file= and flush= so a weird but doable idea is simply to substitute a temporary file for any file= argument and write the results to a temporary file or something in memory that emulates a file. You can then read that back in and return what you want after handling the original print statement with the original arguments, or perhaps just use your result to any actually specified file or the default. You can thus create something like what you want and leave the original print() command alone to do what it was designed to do. And, in general, people who want a copy of what they print, often use other python functionality to craft some or all parts of the text they want printed and only then call print() and thus already may have the ability to use the text afterwards. For many purposes, including efficiency, returning nothing makes good sense. But it is not really the only choice or the right choice and yet, if you want to use THIS language, it has to be accepted as the documented choice. -----Original Message----- From: Python-list On Behalf Of Thomas Passin Sent: Tuesday, January 31, 2023 1:16 PM To: python-list at python.org Subject: Re: evaluation question On 1/31/2023 4:24 AM, Muttley at dastardlyhq.com wrote: > On Tue, 31 Jan 2023 12:57:33 +1300 > Greg Ewing wrote: >> On 30/01/23 10:41 pm, Muttley at dastardlyhq.com wrote: >>> What was the point of the upheaval of converting the print command >>> in python 2 into a function in python 3 if as a function >>> print() doesn't return anything useful? >> >> It was made a function because there's no good reason for it to have >> special syntax in the language. > > All languages have their ugly corners due to initial design mistakes > and/or constraints. Eg: java with the special behaviour of its string > class, C++ with "=0" pure virtual declaration. But they don't dump > them and make all old code suddenly cease to execute. > > Pragmatism should always come before language purity. > It was more fundamental than that, and not mainly about print(): https://urldefense.com/v3/__https://snarky.ca/why-python-3-exists/__;!!Cn_UX_p3!ldtLZeCkbC0gawIJRrbrunA0AqA_btcJ9M1TOlajHrezsXh5AqpuLFCg1mITKq8qvR8f3u-d8K2WF9yGhlD2rfk$ -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!ldtLZeCkbC0gawIJRrbrunA0AqA_btcJ9M1TOlajHrezsXh5AqpuLFCg1mITKq8qvR8f3u-d8K2WF9yG5f0h2No$ -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!ldtLZeCkbC0gawIJRrbrunA0AqA_btcJ9M1TOlajHrezsXh5AqpuLFCg1mITKq8qvR8f3u-d8K2WF9yG5f0h2No$ From rob.cliffe at btinternet.com Sat Jan 28 11:39:49 2023 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sat, 28 Jan 2023 16:39:49 +0000 Subject: Evaluation of variable as f-string In-Reply-To: References: <8ebc73d2-e14a-b9ad-fa35-5078e5f9fd6c@btinternet.com> Message-ID: <33f48974-2ddc-67e4-7a79-ef9a758980a9@btinternet.com> On 27/01/2023 23:41, Chris Angelico wrote: > On Sat, 28 Jan 2023 at 10:08, Rob Cliffe via Python-list > wrote: >> Whoa! Whoa! Whoa! >> I appreciate the points you are making, Chris, but I am a bit taken >> aback by such forceful language. > The exact same points have already been made, but not listened to. > Sometimes, forceful language is required in order to get people to > listen. > >> If it's addressed to me: How about if I wanted a program (a learning >> tool) to allow the user to play with f-strings? >> I.e. to type in a string, and then see what the result would be if it >> had been an f-string? >> I suspect there are other use cases, but I confess I can't think of one >> right now. > Use the REPL, which will happily evaluate f-strings in their original > context, just like any other code would. You're already eval'ing, so > it's exactly what you'd expect. This is not the same thing as "typing > in a string", though - it's typing in code and seeing what the result > would be. (Except to the extent that source code can be considered a > string.) This is hypothetical, but I might want to work on a platform where the REPL was not available. > >> If it's addressed to me: "it" means a function that will take a string >> and evaluate it at runtime as if it were an f-string. Sure, with >> caveats and limitations. > And that's what I am saying is a terrible terrible idea. It will > evaluate things in the wrong context, it has all the normal problems > of eval, and then it introduces its own unique problems with quote > characters. With great respect, Chris, isn't it for the OP (or anyone else) to decide - having been warned of the various drawbacks and limitations - to decide if it's a terrible idea *for him*?? He's entitled to decide that it's just what *he* needs, and that the drawbacks don't matter *for him".? Just as you're entitled to disagree. > >> And indeed Thomas Passim found this partial >> solution on Stack Overflow: >> def effify(non_f_str: str): >> return eval(f'f"""{non_f_str}"""') > You can find anything on Stack Overflow. Just because you found it > there doesn't mean it's any good - even if it's been massively > upvoted. > >> Addressing your points specifically: >> 1) I believe the quote character limitation could be overcome. It >> would need a fair amount of work, for which I haven't (yet) the time or >> inclination. > No problem. Here, solve it for this string: > > eval_me = ' f"""{f\'\'\'{f"{f\'{1+2}\'}"}\'\'\'}""" ' > > F-strings can be nested, remember. I remember it well. As far as I can see (and I may well be wrong; thinking about this example made my head hurt ?) this could be solved if PEP 701 were implemented (so that f-string expressions can contain backslashes) but not otherwise. > >> 2) Yes in general you would have to pass it one dictionary, maybe >> two. I don't see this as an insuperable obstacle. I am not sure what >> you mean by "can't be done with full generality" and perhaps that's not >> important. >>>> def func(): > ... x = 1 > ... class cls: > ... y = 2 > ... print(f"{x=} {y=}") > ... print(locals()) > ... >>>> func() > x=1 y=2 > {'__module__': '__main__', '__qualname__': 'func..cls', 'y': 2} Thanks for clarifying. Hm.? So 'x' is neither in locals() nor in globals().? Which starts me wondering (to go off on a tangent): Should there be a nonlocals() dictionary? > > Maybe you don't care. Maybe you do. But locals() is not the same as > "all names currently available in this scope". And, this example is > definitely not something I would recommend, but good luck making this > work with eval: > >>>> def func(): > ... x = 1 > ... print(f"{(x:=2)}") > ... print(x) > ... >>>> func() > 2 > 2 > ... x = 1 > ... print(eval("(x:=2)", globals(), locals())) > ... print(x) > ... >>>> func() > 2 > 1 Now that, I have to admit, IS a challenge! > >> 3) Not sure I understand this. > Before f-strings existed, one of the big problems with "just use > str.format_map" was that you can't just pass it locals() to get all > the available names. You also can't eval arbitrary code and expect to > get the same results, even if you pass it globals and locals. And > various other considerations here - the exact issues seen here, but > flipped on their heads. So the obvious question is: why not just use > str.format_map? > What this underlines to me is what a good thing f-strings are.? And with PEP 701 they will be IMO even better. Just as when you were working on PEP 463 (Exception-catching expressions) - which I still think would be a Good Thing - some research I did made me realise how good the existing try/except/else/finally mechanism actually is.? There's lots of Good Stuff in Python. ? Best wishes Rob From rosuav at gmail.com Tue Jan 31 17:33:45 2023 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 1 Feb 2023 09:33:45 +1100 Subject: Evaluation of variable as f-string In-Reply-To: <33f48974-2ddc-67e4-7a79-ef9a758980a9@btinternet.com> References: <8ebc73d2-e14a-b9ad-fa35-5078e5f9fd6c@btinternet.com> <33f48974-2ddc-67e4-7a79-ef9a758980a9@btinternet.com> Message-ID: On Wed, 1 Feb 2023 at 09:14, Rob Cliffe via Python-list wrote: > With great respect, Chris, isn't it for the OP (or anyone else) to > decide - having been warned of the various drawbacks and limitations - > to decide if it's a terrible idea *for him*? He's entitled to decide > that it's just what *he* needs, and that the drawbacks don't matter *for > him". Just as you're entitled to disagree. It's an objectively bad idea. If the OP wants to do it, well, it's a free world, but that doesn't mean I'm going to sugarcoat it and say "oh yes, yes, you are totally right to do that". > Thanks for clarifying. > Hm. So 'x' is neither in locals() nor in globals(). Which starts me > wondering (to go off on a tangent): Should there be a nonlocals() > dictionary? I don't think so, but there might be some value in a dictionary containing all available variables. It would have the same "don't depend on writing" caveats that locals() has (or would be specifically defined as a copy and thus disconnected), so its value would be limited. And it would probably STILL be imperfect, because perfection would require that it be a compiler construct, due to the way that nonlocals are implemented. >>> class Destructible: ... def __init__(self, name): self.name = name ... def __del__(self): print("Deleting", self.name) ... >>> def func(): ... x = Destructible("x") ... y = Destructible("y") ... return lambda: x ... >>> func() Deleting y . at 0x7ff8c9897ce0> The compiler is free to dispose of y as soon as func ends, but x has to be retained for the inner function. So if there were any function that could return every readable variable, it would have to force both x and y to be retained; as such, it would have to be a compiler construct. And given what happened with star imports in functions as of Python 3, I am highly dubious that such a pessimisation would ever be implemented. > > Maybe you don't care. Maybe you do. But locals() is not the same as > > "all names currently available in this scope". And, this example is > > definitely not something I would recommend, but good luck making this > > work with eval: > > > >>>> def func(): > > ... x = 1 > > ... print(f"{(x:=2)}") > > ... print(x) > > ... > >>>> func() > > 2 > > 2 > > ... x = 1 > > ... print(eval("(x:=2)", globals(), locals())) > > ... print(x) > > ... > >>>> func() > > 2 > > 1 > Now that, I have to admit, IS a challenge! Exactly. This sort of thing is why the OP's idea as written is so bad: it will cause many unnecessary corner cases, where the much simpler idea of working it around format_map will actually behave sanely. So I do not apologize for calling it a bad idea. It is a bad idea. Lying about it won't change anything and won't help anyone. ChrisA From greg.ewing at canterbury.ac.nz Tue Jan 31 17:59:25 2023 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 1 Feb 2023 11:59:25 +1300 Subject: evaluation question In-Reply-To: References: Message-ID: On 31/01/23 10:24 pm, Muttley at dastardlyhq.com wrote: > All languages have their ugly corners due to initial design mistakes and/or > constraints. Eg: java with the special behaviour of its string class, C++ > with "=0" pure virtual declaration. But they don't dump them and make all old > code suddenly cease to execute. No, but it was decided that Python 3 would have to be backwards incompatible, mainly to sort out the Unicode mess. Given that, the opportunity was taken to clean up some other mistakes as well. -- Greg From greg.ewing at canterbury.ac.nz Tue Jan 31 18:18:58 2023 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 1 Feb 2023 12:18:58 +1300 Subject: evaluation question In-Reply-To: References: <674a1779-ea08-ef69-2a2d-b8f2dd924d07@tompassin.net> Message-ID: On 1/02/23 7:33 am, Stefan Ram wrote: > Thomas Passin writes: > Some people say it is a function now so that you can redefine it. Well, that's one benefit, but I wouldn't say it's the main one. The point is really that you can do *anything* with it now that you can do with a regular function -- pass it as an argument, wrap it with another function, define your own function with a similar signature for duck-typing purposes, etc. > It would still be possible to have a special syntax for the outermost > expression of an expression statement that would allow one to omit > the parentheses, That's only one of the syntactic oddities of the old print statement, thogh. There was also the >> thing, special treatment of trailing commas, etc. Also, introducing a paren-less call syntax would be a very big and controversial change that would be way out of proportion to the problem. -- Greg From rosuav at gmail.com Tue Jan 31 19:06:07 2023 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 1 Feb 2023 11:06:07 +1100 Subject: evaluation question In-Reply-To: References: <674a1779-ea08-ef69-2a2d-b8f2dd924d07@tompassin.net> Message-ID: On Wed, 1 Feb 2023 at 10:47, Greg Ewing wrote: > That's only one of the syntactic oddities of the old print > statement, thogh. There was also the >> thing, special treatment > of trailing commas, etc. "Soft space" (the trailing comma behaviour) was an incredibly complex wart. Glad it's gone. > Also, introducing a paren-less call syntax would be a very big > and controversial change that would be way out of proportion to > the problem. Oddly enough, that WAS actually proposed recently - by Guido himself - as a demonstration of the power of the new PEG parser: https://mail.python.org/archives/list/python-ideas at python.org/thread/NCQX6ZIBREUTLS52VVG3DSZ43OEXJFTT/ (The mailing list archive messes up formatting a bit with REPL transcripts, thinking they're quoted text.) The general consensus was "allowing function calls without parens causes more issues than it solves", with plenty of examples from other programming languages to prove this - Ruby, while generally a decent language, shows a rather nasty wart with this particular feature (see "Ruby allows parens-less function calls" from Steven D'Aprano in that thread). I don't think it'll ever happen in Python, but it's nice to know that the parser is flexible enough. It means that other weird cases, where the intuitive expectation is different, can be better handled (see eg "with (x as y, a as b):" syntax). Having print as a function is WAY better than having it as a special case with lots of warts. And it's so much easier to add extra features to it; for instance, how would you add a "flush after printing" flag to Py2's print statement? With a function, it's easy - just print(..., flush=True). ChrisA From list1 at tompassin.net Tue Jan 31 19:11:55 2023 From: list1 at tompassin.net (Thomas Passin) Date: Tue, 31 Jan 2023 19:11:55 -0500 Subject: evaluation question In-Reply-To: References: <674a1779-ea08-ef69-2a2d-b8f2dd924d07@tompassin.net> Message-ID: <6c261ff8-be2a-4c48-a514-bf36fd4ff6c9@tompassin.net> On 1/31/2023 6:18 PM, Greg Ewing wrote: > On 1/02/23 7:33 am, Stefan Ram wrote: >> Thomas Passin writes: > >> ?? Some people say it is a function now so that you can redefine it. Hmm, I didn't write these quotes. Maybe someone got confused by the depth of the nested replies in this thread. Easy enough to do. > Well, that's one benefit, but I wouldn't say it's the main one. > > The point is really that you can do *anything* with it now that > you can do with a regular function -- pass it as an argument, > wrap it with another function, define your own function with a > similar signature for duck-typing purposes, etc. > >> ?? It would still be possible to have a special syntax for the outermost >> ?? expression of an expression statement that would allow one to omit >> ?? the parentheses, > > That's only one of the syntactic oddities of the old print > statement, thogh. There was also the >> thing, special treatment > of trailing commas, etc. > > Also, introducing a paren-less call syntax would be a very big > and controversial change that would be way out of proportion to > the problem. > From PythonList at DancesWithMice.info Tue Jan 31 19:17:33 2023 From: PythonList at DancesWithMice.info (dn) Date: Wed, 1 Feb 2023 13:17:33 +1300 Subject: evaluation question In-Reply-To: References: Message-ID: <062f691c-d4b7-09f9-c949-a14ef18119f7@DancesWithMice.info> On 01/02/2023 11.59, Greg Ewing wrote: > On 31/01/23 10:24 pm, Muttley at dastardlyhq.com wrote: >> All languages have their ugly corners due to initial design mistakes >> and/or >> constraints. Eg: java with the special behaviour of its string class, C++ >> with "=0" pure virtual declaration. But they don't dump them and make >> all old >> code suddenly cease to execute. > > No, but it was decided that Python 3 would have to be backwards > incompatible, mainly to sort out the Unicode mess. Given that, > the opportunity was taken to clean up some other mistakes as well. +1 and the move to Unicode has opened-up the Python community beyond the US, to embrace 'the world' - a proposition (still) not well-recognised by (only) English-speakers/writers/readers. Even though the proposition has a troll-bait smell to it:- 1 nothing "ceased to execute" and Python 2 was maintained and developed for quite some time and in-parallel to many Python 3 releases. 2 the only constant in this business is 'change'. I'd rather cope with an evolution in this language (which we know and love), than one day realise that it has become dated or inflexible, and have to learn a new, replacement, language! -- Regards, =dn From greg.ewing at canterbury.ac.nz Tue Jan 31 20:38:39 2023 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 1 Feb 2023 14:38:39 +1300 Subject: evaluation question In-Reply-To: References: <062f691c-d4b7-09f9-c949-a14ef18119f7@DancesWithMice.info> Message-ID: On 1/02/23 1:17 pm, dn wrote: > 1 nothing "ceased to execute" and Python 2 was maintained and developed > for quite some time and in-parallel to many Python 3 releases. And a lot of effort was put into making the transition as easy as possible, e.g. 2to3, and the features added to 2.7 to make it easier to write code that would work in both versions. -- Greg From grant.b.edwards at gmail.com Tue Jan 31 22:17:58 2023 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Tue, 31 Jan 2023 19:17:58 -0800 (PST) Subject: evaluation question References: <674a1779-ea08-ef69-2a2d-b8f2dd924d07@tompassin.net> Message-ID: <63d9d9e6.050a0220.45e1e.0057@mx.google.com> On 2023-01-31, Greg Ewing wrote: > That's only one of the syntactic oddities of the old print > statement, thogh. There was also the >> thing, special treatment > of trailing commas, etc. In "old" Python I used to use the trailing comma extensively, but I could never get myself to use the >> thing. I don't know why, but it just felt wrong. -- Grant