From jacob.kruger.work at gmail.com Tue Mar 5 13:13:21 2024 From: jacob.kruger.work at gmail.com (Jacob Kruger) Date: Tue, 5 Mar 2024 20:13:21 +0200 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference Message-ID: Hi there Working with python 3.11, and, issue that confused me for a little while, trying to figure out what was occurring - unless am completely confused, or missing something - was that, for example, when having pre-defined a variable, and then included it in the global statement inside a function, that function was still referring to a completely local instance, without manipulating outside variable object at all unless I first executed a form of referral to it, before then possibly assigning a new value to it. Now, this does not seem to occur consistently if, for example, I just run bare-bones test code inside the python interpreter, but consistently occurs inside my actual testing script. Basically, in a file with python code in that am using for a form of testing at the moment, at the top of the file, under all the import statements, I initiate the existence of a list variable to make use of later: # code snippet l_servers = [] # end of first code snippet Then, lower down, inside a couple of different functions, the first line inside the functions includes the following: # code snippet ??? global l_servers # end code snippet That should, in theory, mean that if I assign a value to that variable inside one of the functions, it should reflect globally? However, it seems like that, while inside those functions, it can be assigned a new list of values, but if I then return to the scope outside the functions, it has reverted back to being an empty list = []? The issue seems to specifically (or not) occur when I make a call to one function, and, in the steps it's executing in one context, while it's not doing anything to the list directly, it's then making a call to the second function, which is then meant to repopulate the list with a brand new set of values. Now, what almost seems to be occurring, is that while just manipulating the contents of a referenced variable is fine in this context, the moment I try to reassign it, that's where the issue is occurring . Here are relevant excerpts from the file:- # start code # original assignation in main part of file l_servers = [] # function wich is initially being executed def interact(): ??? global l_servers ??? # extra code inbetween choosing what to carry out ??? # ... ??? # end of other code ??? bl_response, o_out = list_servers() ??? if bl_response: # just make sure other function call was successful ??????? l_servers.clear() # first make reference to global variable ??????? for srv in o_out: l_servers.append(srv) # now re-populate items ??? # end code snippet from inside interact function # end of interact function # end of code snippet That other function being called from within, list_servers() was initially just trying to populate the values inside the global list variable itself, but was ending up in a similar fashion - reverting to initial empty value, but, the above now seems to work, as long as I first make reference to/manipulate/work with global variable instead of just trying to reassign it a brand new value/set of items? So, am I missing something obvious, have I forgotten about something else - yes, know that if was working from within an embedded function, I might need/want to then use the nonlocal statement against that variable name, but, honestly, just not sure how this can be occurring, and, it's not just with this one list variable, etc.? If I try simple test code from within the python interpreter, using different types of variables, this does also not seem to be the same all the time, but, don't think it can relate to an iterable like a list, or else, just in case, here is the code snippet with all the import statements from the top of that file, in case something could be overriding standard behaviour - not likely in this context, but, really not sure what's occurring: # import code snippet import requests, time from requests.auth import HTTPBasicAuth import psutil as psu import pytz import bcrypt from copy import copy from datetime import datetime, timedelta, timezone from dateutil.parser import parse # end of import snippet Thanks if you have any ideas/thoughts on the matter Jacob Kruger +2782 413 4791 "Resistance is futile!...Acceptance is versatile..." From PythonList at DancesWithMice.info Tue Mar 5 13:23:41 2024 From: PythonList at DancesWithMice.info (dn) Date: Wed, 6 Mar 2024 07:23:41 +1300 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: References: Message-ID: <0ccad7a9-eaba-48e6-b972-d89e5a930c11@DancesWithMice.info> Jacob, Please reduce the problem to a small code-set which reproduces the problem. If we can reproduce same, then that tells us something. At the very least, we can experiment without having to expend amounts of time in a (likely faulty) bid to reproduce the same environment. Also, code is the ultimate description! Perhaps start with a small experiment: - after l_servers is created, print its id() - after the global statement, print its id() - after the clear/reassignment, print its id() Is Python always working with the same list? Please advise... On 6/03/24 07:13, Jacob Kruger via Python-list wrote: > Hi there > > > Working with python 3.11, and, issue that confused me for a little > while, trying to figure out what was occurring - unless am completely > confused, or missing something - was that, for example, when having > pre-defined a variable, and then included it in the global statement > inside a function, that function was still referring to a completely > local instance, without manipulating outside variable object at all > unless I first executed a form of referral to it, before then possibly > assigning a new value to it. > > > Now, this does not seem to occur consistently if, for example, I just > run bare-bones test code inside the python interpreter, but consistently > occurs inside my actual testing script. > > > Basically, in a file with python code in that am using for a form of > testing at the moment, at the top of the file, under all the import > statements, I initiate the existence of a list variable to make use of > > later: > > > # code snippet > > l_servers = [] > > # end of first code snippet > > > Then, lower down, inside a couple of different functions, the first line > inside the functions includes the following: > # code snippet > ??? global l_servers > # end code snippet > > That should, in theory, mean that if I assign a value to that variable > inside one of the functions, it should reflect globally? > > However, it seems like that, while inside those functions, it can be > assigned a new list of values, but if I then return to the scope outside > > the functions, it has reverted back to being an empty list = []? > > > The issue seems to specifically (or not) occur when I make a call to one > function, and, in the steps it's executing in one context, while it's > not doing anything to the list directly, it's then making a call to the > second function, which is then meant to repopulate the list with a brand > new set of values. > > > Now, what almost seems to be occurring, is that while just manipulating > the contents of a referenced variable is fine in this context, the > moment I try to reassign it, that's where the issue is occurring . > > > Here are relevant excerpts from the file:- > > > # start code > > # original assignation in main part of file > > l_servers = [] > > > # function wich is initially being executed > > def interact(): > ??? global l_servers > ??? # extra code inbetween choosing what to carry out > > ??? # ... > > ??? # end of other code > > ??? bl_response, o_out = list_servers() > > ??? if bl_response: # just make sure other function call was successful > > ??????? l_servers.clear() # first make reference to global variable > > ??????? for srv in o_out: l_servers.append(srv) # now re-populate items > > ??? # end code snippet from inside interact function > > # end of interact function > > # end of code snippet > > > That other function being called from within, list_servers() was > initially just trying to populate the values inside the global list > variable itself, but was ending up in a similar fashion - reverting to > initial empty value, but, the above now seems to work, as long as I > first make reference to/manipulate/work with global variable instead of > just trying to reassign it a brand new value/set of items? > > > So, am I missing something obvious, have I forgotten about something > else - yes, know that if was working from within an embedded function, I > might need/want to then use the nonlocal statement against that variable > name, but, honestly, just not sure how this can be occurring, and, it's > not just with this one list variable, etc.? > > > If I try simple test code from within the python interpreter, using > different types of variables, this does also not seem to be the same all > the time, but, don't think it can relate to an iterable like a list, or > else, just in case, here is the code snippet with all the import > statements from the top of that file, in case something could be > overriding standard behaviour - not likely in this context, but, really > not sure what's occurring: > > # import code snippet > > import requests, time > from requests.auth import HTTPBasicAuth > import psutil as psu > import pytz > import bcrypt > from copy import copy > from datetime import datetime, timedelta, timezone > from dateutil.parser import parse > > # end of import snippet > > > Thanks if you have any ideas/thoughts on the matter > > > Jacob Kruger > +2782 413 4791 > "Resistance is futile!...Acceptance is versatile..." > > -- Regards, =dn From cs at cskk.id.au Tue Mar 5 16:20:00 2024 From: cs at cskk.id.au (Cameron Simpson) Date: Wed, 6 Mar 2024 08:20:00 +1100 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: References: Message-ID: On 05Mar2024 20:13, Jacob Kruger wrote: >Now, what almost seems to be occurring, is that while just manipulating >the contents of a referenced variable is fine in this context, the >moment I try to reassign it, that's where the issue is occurring . Because there are no variable definitions in Python, when you write a function Python does a static analysis of it to decide which variables are local and which are not. If there's an assignment to a variable, it is a local variable. _Regardless_ of whether that assignment has been executed, or gets executed at all (eg in an if-statement branch which doesn't fire). You can use `global` or `nonlocal` to change where Python looks for a particular name. In the code below, `f1` has no local variables and `f2` has an `x` and `l1` local variable. x = 1 l1 = [1, 2, 3] def f1(): print("f1 ...") l1[1] = 5 # _not_ an assignment to "l1" print("in f1, x =", x, "l1 =", l1) def f2(): print("f2 ...") x = 3 l1 = [6, 7, 9] # assignment to "l1" print("in f2, x =", x, "l1 =", l1) print("outside, x =", x, "l1 =", l1) f1() print("outside after f1, x =", x, "l1 =", l1) f2() print("outside after f2, x =", x, "l1 =", l1) Cheers, Cameron Simpson From grant.b.edwards at gmail.com Tue Mar 5 17:46:32 2024 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Tue, 5 Mar 2024 17:46:32 -0500 (EST) Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference References: Message-ID: <4Tq9fJ0qN7znVFY@mail.python.org> On 2024-03-05, Cameron Simpson via Python-list wrote: > Because there are no variable definitions in Python, when you write > a function Python does a static analysis of it to decide which > variables are local and which are not. If there's an assignment to a > variable, it is a local variable. _Regardless_ of whether that > assignment has been executed, or gets executed at all (eg in an > if-statement branch which doesn't fire). Unfortunately, crap "information" sites like geeksforgeeks often describe this either incorrectly or so vaguely as to be worthless. >From the page https://www.geeksforgeeks.org/global-local-variables-python/ Python Global variables are those which are not defined inside any function and have a global scope whereas Python local variables are those which are defined inside a function and their scope is limited to that function only. Since "define" (in this context) isn't a term of art in Python, and it's never defined on the page in question, the quoted paragraph is not meaningful: it simply says that "global variables are global and local variables are local". That page goes on to say: In other words, we can say that local variables are accessible only inside the function in which it was initialized This is equally crap. It doesn't matter whether the variable is initialized or not. As Cameron correctly stated, if a function contains an assignment to a variable, and that variable is not declared global, then that variable is local. For example: def foo(): print(s) if 0: s = "there" print(s) In the function above s _is_not_ initialized in the function foo(). However, foo() does contain an assignment to s, therefore s is local unless declared global/nonlocal. [And the first print() will throw an exception even if there is a value bound to the global name 's'.] Unfortunately (presumably thanks to SEO) the enshittification of Google has reached the point where searching for info on things like Python name scope, the first page of links are to worthless sites like geeksforgeeks. From chanofux at gmail.com Tue Mar 5 19:06:16 2024 From: chanofux at gmail.com (Chano Fucks) Date: Tue, 5 Mar 2024 21:06:16 -0300 Subject: Can u help me? Message-ID: [image: image.png] From ethan at stoneleaf.us Tue Mar 5 19:24:56 2024 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 5 Mar 2024 16:24:56 -0800 Subject: Can u help me? In-Reply-To: References: Message-ID: <6d6fcd5a-9e18-39ce-a300-dc6a1ed7a2dc@stoneleaf.us> On 3/5/24 16:06, Chano Fucks via Python-list wrote: > [image: image.png] The image is of MS-Windows with the python installation window of "Repair Successful". Hopefully somebody better at explaining that problem can take it from here... -- ~Ethan~ From python at mrabarnett.plus.com Tue Mar 5 19:34:11 2024 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 6 Mar 2024 00:34:11 +0000 Subject: Can u help me? In-Reply-To: References: Message-ID: <641868e8-d1c9-4c2b-9681-95585e26803a@mrabarnett.plus.com> On 2024-03-06 00:06, Chano Fucks via Python-list wrote: > [image: image.png] This list removes all images. From python at mrabarnett.plus.com Tue Mar 5 19:49:14 2024 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 6 Mar 2024 00:49:14 +0000 Subject: Can u help me? In-Reply-To: <6d6fcd5a-9e18-39ce-a300-dc6a1ed7a2dc@stoneleaf.us> References: <6d6fcd5a-9e18-39ce-a300-dc6a1ed7a2dc@stoneleaf.us> Message-ID: <5d7caebb-2929-417e-8a53-4b804b9aa5fc@mrabarnett.plus.com> On 2024-03-06 00:24, Ethan Furman via Python-list wrote: > On 3/5/24 16:06, Chano Fucks via Python-list wrote: > >> [image: image.png] > > The image is of MS-Windows with the python installation window of "Repair Successful". Hopefully somebody better at > explaining that problem can take it from here... > If the repair was successful, what's the problem? From ethan at stoneleaf.us Tue Mar 5 20:44:38 2024 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 5 Mar 2024 17:44:38 -0800 Subject: Can u help me? In-Reply-To: <5d7caebb-2929-417e-8a53-4b804b9aa5fc@mrabarnett.plus.com> References: <6d6fcd5a-9e18-39ce-a300-dc6a1ed7a2dc@stoneleaf.us> <5d7caebb-2929-417e-8a53-4b804b9aa5fc@mrabarnett.plus.com> Message-ID: On 3/5/24 16:49, MRAB via Python-list wrote: > On 2024-03-06 00:24, Ethan Furman via Python-list wrote: >> On 3/5/24 16:06, Chano Fucks via Python-list wrote: >> >>> [image: image.png] >> >> The image is of MS-Windows with the python installation window of "Repair Successful". Hopefully somebody better at >> explaining that problem can take it from here... >> > If the repair was successful, what's the problem? I imagine the issue is trying get Python to run (as I recall, the python icon on the MS-Windows desktop is the installer, not Python itself). From mats at wichmann.us Tue Mar 5 21:06:02 2024 From: mats at wichmann.us (Mats Wichmann) Date: Tue, 5 Mar 2024 19:06:02 -0700 Subject: Can u help me? In-Reply-To: References: <6d6fcd5a-9e18-39ce-a300-dc6a1ed7a2dc@stoneleaf.us> <5d7caebb-2929-417e-8a53-4b804b9aa5fc@mrabarnett.plus.com> Message-ID: On 3/5/24 18:44, Ethan Furman via Python-list wrote: > On 3/5/24 16:49, MRAB via Python-list wrote: > > On 2024-03-06 00:24, Ethan Furman via Python-list wrote: > >> On 3/5/24 16:06, Chano Fucks via Python-list wrote: > >> > >>> [image: image.png] > >> > >> The image is of MS-Windows with the python installation window of > "Repair Successful".? Hopefully somebody better at > >> explaining that problem can take it from here... > >> > > If the repair was successful, what's the problem? > > I imagine the issue is trying get Python to run (as I recall, the python > icon on the MS-Windows desktop is the installer, not Python itself). that's often it, yes - you keep getting the installer when you think you should get a snazzy Python window. Of course, you don't - Python is a command-line/terminal program, not a windows app. Now if you tried to launch IDLE instead, you'd get at least a window. But we're just guessing here. Perhaps Chano will come back with an updated question with some details. From python at mrabarnett.plus.com Tue Mar 5 21:25:10 2024 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 6 Mar 2024 02:25:10 +0000 Subject: Can u help me? In-Reply-To: References: <6d6fcd5a-9e18-39ce-a300-dc6a1ed7a2dc@stoneleaf.us> <5d7caebb-2929-417e-8a53-4b804b9aa5fc@mrabarnett.plus.com> Message-ID: <4401d264-fb23-4d92-ae79-911f5a6f1194@mrabarnett.plus.com> On 2024-03-06 01:44, Ethan Furman via Python-list wrote: > On 3/5/24 16:49, MRAB via Python-list wrote: > > On 2024-03-06 00:24, Ethan Furman via Python-list wrote: > >> On 3/5/24 16:06, Chano Fucks via Python-list wrote: > >> > >>> [image: image.png] > >> > >> The image is of MS-Windows with the python installation window of "Repair Successful". Hopefully somebody better at > >> explaining that problem can take it from here... > >> > > If the repair was successful, what's the problem? > > I imagine the issue is trying get Python to run (as I recall, the python icon on the MS-Windows desktop is the > installer, not Python itself). There was an issue 3 years ago about renaming the installer for clarity: https://github.com/python/cpython/issues/87322 From learn2program at gmail.com Wed Mar 6 05:59:38 2024 From: learn2program at gmail.com (Alan Gauld) Date: Wed, 6 Mar 2024 10:59:38 +0000 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: <4Tq9fJ0qN7znVFY@mail.python.org> References: <4Tq9fJ0qN7znVFY@mail.python.org> Message-ID: On 05/03/2024 22:46, Grant Edwards via Python-list wrote: > Unfortunately (presumably thanks to SEO) the enshittification of > Google has reached the point where searching for info on things like > Python name scope, the first page of links are to worthless sites like > geeksforgeeks. And not just Google, I just tried bing, yahoo and duckduckgo and they are all the same. Not a one listed anything from python.org on the first page... In fact it didn't even appear in the first 100 listings, although wikipedia did manage an entry, eventually. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From jacob.kruger.work at gmail.com Wed Mar 6 07:55:46 2024 From: jacob.kruger.work at gmail.com (Jacob Kruger) Date: Wed, 6 Mar 2024 14:55:46 +0200 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: <0ccad7a9-eaba-48e6-b972-d89e5a930c11@DancesWithMice.info> References: <0ccad7a9-eaba-48e6-b972-d89e5a930c11@DancesWithMice.info> Message-ID: Ok, simpler version - all the code in a simpler test file, and working with two separate variables to explain exactly what am talking about: # start code from datetime import datetime, timezone, timedelta from copy import copy # initialise original values dt_expiry = datetime.strptime("1970-01-01 00:00", "%Y-%m-%d %H:%M").replace(tzinfo=timezone.utc) l_test = [1, 2, 3] def do_it(): ??? global dt_expiry, l_test # asked python to refer to global variables for both ??? # assign new value immediately ??? dt_expiry = datetime.now()+timedelta(minutes=5) ??? print(dt_expiry.strftime("%Y-%m-%d %H:%M")) # just to show new value has been assigned ??? # grab copy of list for re-use of items ??? l_temp = copy(l_test) ??? # following line means l_test will later on retain value in global scope because it was manipulated inside function instead of just assigned new value ??? l_test.clear() ??? # replace original set of values ??? for i in l_temp: l_test.append(i) ??? # add new item ??? l_test.append(99) # end of do_it function # end code If you import the contents of that file into the python interpreter, dt_expiry will start off as "1970-01-01 00:00", and, if you execute do_it function, it will print out the new value assigned to the dt_expiry variable inside that function, but if you then again check the value of the dt_expiry variable afterwards, it's reverted to the 1970... value? If I take out the line that removes values from l_test # l_test.clear() # before appending new value to it, then it will also not retain it's new/additional child items after the function exits, and will just revert back to [1, 2, 3] each and every time. In other words, with some of the variable/object types, if you use a function that manipulates the contents of a variable, before then re-assigning it a new value, it seems like it might then actually update/manipulate the global variable, but, either just calling purely content retrieval functions against said objects, or assigning them new values from scratch seems to then ignore the global scope specified in the first line inside the function? Hope this makes more sense Jacob Kruger +2782 413 4791 "Resistance is futile!...Acceptance is versatile..." On 2024/03/05 20:23, dn via Python-list wrote: > Jacob, > > Please reduce the problem to a small code-set which reproduces the > problem. If we can reproduce same, then that tells us something. At > the very least, we can experiment without having to expend amounts of > time in a (likely faulty) bid to reproduce the same environment. > > Also, code is the ultimate description! > > > Perhaps start with a small experiment: > > - after l_servers is created, print its id() > - after the global statement, print its id() > - after the clear/reassignment, print its id() > > Is Python always working with the same list? > Please advise... > > > On 6/03/24 07:13, Jacob Kruger via Python-list wrote: >> Hi there >> >> >> Working with python 3.11, and, issue that confused me for a little >> while, trying to figure out what was occurring - unless am completely >> confused, or missing something - was that, for example, when having >> pre-defined a variable, and then included it in the global statement >> inside a function, that function was still referring to a completely >> local instance, without manipulating outside variable object at all >> unless I first executed a form of referral to it, before then >> possibly assigning a new value to it. >> >> >> Now, this does not seem to occur consistently if, for example, I just >> run bare-bones test code inside the python interpreter, but >> consistently occurs inside my actual testing script. >> >> >> Basically, in a file with python code in that am using for a form of >> testing at the moment, at the top of the file, under all the import >> statements, I initiate the existence of a list variable to make use of >> >> later: >> >> >> # code snippet >> >> l_servers = [] >> >> # end of first code snippet >> >> >> Then, lower down, inside a couple of different functions, the first line >> inside the functions includes the following: >> # code snippet >> ???? global l_servers >> # end code snippet >> >> That should, in theory, mean that if I assign a value to that variable >> inside one of the functions, it should reflect globally? >> >> However, it seems like that, while inside those functions, it can be >> assigned a new list of values, but if I then return to the scope outside >> >> the functions, it has reverted back to being an empty list = []? >> >> >> The issue seems to specifically (or not) occur when I make a call to >> one function, and, in the steps it's executing in one context, while >> it's not doing anything to the list directly, it's then making a call >> to the second function, which is then meant to repopulate the list >> with a brand new set of values. >> >> >> Now, what almost seems to be occurring, is that while just >> manipulating the contents of a referenced variable is fine in this >> context, the moment I try to reassign it, that's where the issue is >> occurring . >> >> >> Here are relevant excerpts from the file:- >> >> >> # start code >> >> # original assignation in main part of file >> >> l_servers = [] >> >> >> # function wich is initially being executed >> >> def interact(): >> ???? global l_servers >> ???? # extra code inbetween choosing what to carry out >> >> ???? # ... >> >> ???? # end of other code >> >> ???? bl_response, o_out = list_servers() >> >> ???? if bl_response: # just make sure other function call was successful >> >> ???????? l_servers.clear() # first make reference to global variable >> >> ???????? for srv in o_out: l_servers.append(srv) # now re-populate items >> >> ???? # end code snippet from inside interact function >> >> # end of interact function >> >> # end of code snippet >> >> >> That other function being called from within, list_servers() was >> initially just trying to populate the values inside the global list >> variable itself, but was ending up in a similar fashion - reverting >> to initial empty value, but, the above now seems to work, as long as >> I first make reference to/manipulate/work with global variable >> instead of just trying to reassign it a brand new value/set of items? >> >> >> So, am I missing something obvious, have I forgotten about something >> else - yes, know that if was working from within an embedded >> function, I might need/want to then use the nonlocal statement >> against that variable name, but, honestly, just not sure how this can >> be occurring, and, it's not just with this one list variable, etc.? >> >> >> If I try simple test code from within the python interpreter, using >> different types of variables, this does also not seem to be the same >> all the time, but, don't think it can relate to an iterable like a >> list, or else, just in case, here is the code snippet with all the >> import statements from the top of that file, in case something could >> be overriding standard behaviour - not likely in this context, but, >> really not sure what's occurring: >> >> # import code snippet >> >> import requests, time >> from requests.auth import HTTPBasicAuth >> import psutil as psu >> import pytz >> import bcrypt >> from copy import copy >> from datetime import datetime, timedelta, timezone >> from dateutil.parser import parse >> >> # end of import snippet >> >> >> Thanks if you have any ideas/thoughts on the matter >> >> >> Jacob Kruger >> +2782 413 4791 >> "Resistance is futile!...Acceptance is versatile..." >> >> > From jacob.kruger.work at gmail.com Wed Mar 6 08:12:46 2024 From: jacob.kruger.work at gmail.com (Jacob Kruger) Date: Wed, 6 Mar 2024 15:12:46 +0200 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: References: <0ccad7a9-eaba-48e6-b972-d89e5a930c11@DancesWithMice.info> Message-ID: So, this does not make sense to me in terms of the following snippet from the official python docs page: https://docs.python.org/3/faq/programming.html "In Python, variables that are only referenced inside a function are implicitly global. If a variable is assigned a value anywhere within the function?s body, it?s assumed to be a local unless explicitly declared as global." So, I would then assume that if I explicitly include a variable name inside the global statement, then even just assigning it a new value should update the variable in the global context, outside the function? Unless this is something that changed from 3.11 to 3.12 - since that snippet is more or less referring to 3.12, but, don't think it was changed in any way? Jacob Kruger +2782 413 4791 "Resistance is futile!...Acceptance is versatile..." On 2024/03/06 14:55, Jacob Kruger wrote: > Ok, simpler version - all the code in a simpler test file, and working > with two separate variables to explain exactly what am talking about: > > # start code > > from datetime import datetime, timezone, timedelta > > from copy import copy > > > # initialise original values > > dt_expiry = datetime.strptime("1970-01-01 00:00", "%Y-%m-%d > %H:%M").replace(tzinfo=timezone.utc) > > l_test = [1, 2, 3] > > > def do_it(): > ??? global dt_expiry, l_test # asked python to refer to global > variables for both > > ??? # assign new value immediately > > ??? dt_expiry = datetime.now()+timedelta(minutes=5) > ??? print(dt_expiry.strftime("%Y-%m-%d %H:%M")) # just to show new > value has been assigned > ??? # grab copy of list for re-use of items > ??? l_temp = copy(l_test) > ??? # following line means l_test will later on retain value in global > scope because it was manipulated inside function instead of just > assigned new value > ??? l_test.clear() > ??? # replace original set of values > ??? for i in l_temp: l_test.append(i) > ??? # add new item > ??? l_test.append(99) > # end of do_it function > > # end code > > > If you import the contents of that file into the python interpreter, > dt_expiry will start off as "1970-01-01 00:00", and, if you execute > do_it function, it will print out the new value assigned to the > dt_expiry variable inside that function, but if you then again check > the value of the dt_expiry variable afterwards, it's reverted to the > 1970... value? > > > If I take out the line that removes values from l_test # > l_test.clear() # before appending new value to it, then it will also > not retain it's new/additional child items after the function exits, > and will just revert back to [1, 2, 3] each and every time. > > > In other words, with some of the variable/object types, if you use a > function that manipulates the contents of a variable, before then > re-assigning it a new value, it seems like it might then actually > update/manipulate the global variable, but, either just calling purely > content retrieval functions against said objects, or assigning them > new values from scratch seems to then ignore the global scope > specified in the first line inside the function? > > > Hope this makes more sense > > > Jacob Kruger > +2782 413 4791 > "Resistance is futile!...Acceptance is versatile..." > > > On 2024/03/05 20:23, dn via Python-list wrote: >> Jacob, >> >> Please reduce the problem to a small code-set which reproduces the >> problem. If we can reproduce same, then that tells us something. At >> the very least, we can experiment without having to expend amounts of >> time in a (likely faulty) bid to reproduce the same environment. >> >> Also, code is the ultimate description! >> >> >> Perhaps start with a small experiment: >> >> - after l_servers is created, print its id() >> - after the global statement, print its id() >> - after the clear/reassignment, print its id() >> >> Is Python always working with the same list? >> Please advise... >> >> >> On 6/03/24 07:13, Jacob Kruger via Python-list wrote: >>> Hi there >>> >>> >>> Working with python 3.11, and, issue that confused me for a little >>> while, trying to figure out what was occurring - unless am >>> completely confused, or missing something - was that, for example, >>> when having pre-defined a variable, and then included it in the >>> global statement inside a function, that function was still >>> referring to a completely local instance, without manipulating >>> outside variable object at all unless I first executed a form of >>> referral to it, before then possibly assigning a new value to it. >>> >>> >>> Now, this does not seem to occur consistently if, for example, I >>> just run bare-bones test code inside the python interpreter, but >>> consistently occurs inside my actual testing script. >>> >>> >>> Basically, in a file with python code in that am using for a form of >>> testing at the moment, at the top of the file, under all the import >>> statements, I initiate the existence of a list variable to make use of >>> >>> later: >>> >>> >>> # code snippet >>> >>> l_servers = [] >>> >>> # end of first code snippet >>> >>> >>> Then, lower down, inside a couple of different functions, the first >>> line >>> inside the functions includes the following: >>> # code snippet >>> ???? global l_servers >>> # end code snippet >>> >>> That should, in theory, mean that if I assign a value to that variable >>> inside one of the functions, it should reflect globally? >>> >>> However, it seems like that, while inside those functions, it can be >>> assigned a new list of values, but if I then return to the scope >>> outside >>> >>> the functions, it has reverted back to being an empty list = []? >>> >>> >>> The issue seems to specifically (or not) occur when I make a call to >>> one function, and, in the steps it's executing in one context, while >>> it's not doing anything to the list directly, it's then making a >>> call to the second function, which is then meant to repopulate the >>> list with a brand new set of values. >>> >>> >>> Now, what almost seems to be occurring, is that while just >>> manipulating the contents of a referenced variable is fine in this >>> context, the moment I try to reassign it, that's where the issue is >>> occurring . >>> >>> >>> Here are relevant excerpts from the file:- >>> >>> >>> # start code >>> >>> # original assignation in main part of file >>> >>> l_servers = [] >>> >>> >>> # function wich is initially being executed >>> >>> def interact(): >>> ???? global l_servers >>> ???? # extra code inbetween choosing what to carry out >>> >>> ???? # ... >>> >>> ???? # end of other code >>> >>> ???? bl_response, o_out = list_servers() >>> >>> ???? if bl_response: # just make sure other function call was >>> successful >>> >>> ???????? l_servers.clear() # first make reference to global variable >>> >>> ???????? for srv in o_out: l_servers.append(srv) # now re-populate >>> items >>> >>> ???? # end code snippet from inside interact function >>> >>> # end of interact function >>> >>> # end of code snippet >>> >>> >>> That other function being called from within, list_servers() was >>> initially just trying to populate the values inside the global list >>> variable itself, but was ending up in a similar fashion - reverting >>> to initial empty value, but, the above now seems to work, as long as >>> I first make reference to/manipulate/work with global variable >>> instead of just trying to reassign it a brand new value/set of items? >>> >>> >>> So, am I missing something obvious, have I forgotten about something >>> else - yes, know that if was working from within an embedded >>> function, I might need/want to then use the nonlocal statement >>> against that variable name, but, honestly, just not sure how this >>> can be occurring, and, it's not just with this one list variable, etc.? >>> >>> >>> If I try simple test code from within the python interpreter, using >>> different types of variables, this does also not seem to be the same >>> all the time, but, don't think it can relate to an iterable like a >>> list, or else, just in case, here is the code snippet with all the >>> import statements from the top of that file, in case something could >>> be overriding standard behaviour - not likely in this context, but, >>> really not sure what's occurring: >>> >>> # import code snippet >>> >>> import requests, time >>> from requests.auth import HTTPBasicAuth >>> import psutil as psu >>> import pytz >>> import bcrypt >>> from copy import copy >>> from datetime import datetime, timedelta, timezone >>> from dateutil.parser import parse >>> >>> # end of import snippet >>> >>> >>> Thanks if you have any ideas/thoughts on the matter >>> >>> >>> Jacob Kruger >>> +2782 413 4791 >>> "Resistance is futile!...Acceptance is versatile..." >>> >>> >> From mats at wichmann.us Wed Mar 6 08:57:01 2024 From: mats at wichmann.us (Mats Wichmann) Date: Wed, 6 Mar 2024 06:57:01 -0700 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: References: <0ccad7a9-eaba-48e6-b972-d89e5a930c11@DancesWithMice.info> Message-ID: On 3/6/24 05:55, Jacob Kruger via Python-list wrote: > Ok, simpler version - all the code in a simpler test file, and working > with two separate variables to explain exactly what am talking about: > If you import the contents of that file into the python interpreter, > dt_expiry will start off as "1970-01-01 00:00", and, if you execute > do_it function, it will print out the new value assigned to the > dt_expiry variable inside that function, but if you then again check the > value of the dt_expiry variable afterwards, it's reverted to the 1970... > value? > > > If I take out the line that removes values from l_test # l_test.clear() > # before appending new value to it, then it will also not retain it's > new/additional child items after the function exits, and will just > revert back to [1, 2, 3] each and every time. > > > In other words, with some of the variable/object types, if you use a > function that manipulates the contents of a variable, before then > re-assigning it a new value, it seems like it might then actually > update/manipulate the global variable, but, either just calling purely > content retrieval functions against said objects, or assigning them new > values from scratch seems to then ignore the global scope specified in > the first line inside the function? > > > Hope this makes more sense No, it doesn't. Your code is working as one would expect. For example, adding prints for the l_test variable, and removing the .clear() which you claim makes it not work, shows me: before: l_test=[1, 2, 3], id(l_test)=140153285385856 leaving do_it: l_test=[1, 2, 3, 1, 2, 3, 99], id(l_test)=140153285385856 after: l_test=[1, 2, 3, 1, 2, 3, 99], id(l_test)=140153285385856 It's the same list object, as you can see by the id values. And the list is updating as expected. And... you don't need the global statement for l_test. As it's mutable, you can mutate it in the function; the global only acts on assignment. Using "global" for that may make your intent more clear to readers though, although static checkers will grumble at you. You must be doing something additional that you're not telling us about. From list1 at tompassin.net Wed Mar 6 08:16:01 2024 From: list1 at tompassin.net (Thomas Passin) Date: Wed, 6 Mar 2024 08:16:01 -0500 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: References: <4Tq9fJ0qN7znVFY@mail.python.org> Message-ID: On 3/6/2024 5:59 AM, Alan Gauld via Python-list wrote: > On 05/03/2024 22:46, Grant Edwards via Python-list wrote: >> Unfortunately (presumably thanks to SEO) the enshittification of >> Google has reached the point where searching for info on things like >> Python name scope, the first page of links are to worthless sites like >> geeksforgeeks. > And not just Google, I just tried bing, yahoo and duckduckgo > and they are all the same. Not a one listed anything from > python.org on the first page... In fact it didn't even appear > in the first 100 listings, although wikipedia did manage an > entry, eventually. I don't know ... I just searched for "python local vs global variables" and a python.org page on it was the second hit. I usually use StartPage - who knows where they aggregate from - but the same search on Google and Bing also popped up the python.org link as the second hit. As usual Bing was a nasty experience, though. Still, if your search phrase isn't as well focused as that or you are less lucky, for sure you'll get all sorts of junk. From list1 at tompassin.net Wed Mar 6 09:01:44 2024 From: list1 at tompassin.net (Thomas Passin) Date: Wed, 6 Mar 2024 09:01:44 -0500 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: References: <0ccad7a9-eaba-48e6-b972-d89e5a930c11@DancesWithMice.info> Message-ID: <2248adf8-551f-4e86-8de8-be892d5978ed@tompassin.net> On 3/6/2024 7:55 AM, Jacob Kruger via Python-list wrote: > Ok, simpler version - all the code in a simpler test file, and working > with two separate variables to explain exactly what am talking about: > > # start code > > from datetime import datetime, timezone, timedelta > > from copy import copy > > > # initialise original values > > dt_expiry = datetime.strptime("1970-01-01 00:00", "%Y-%m-%d > %H:%M").replace(tzinfo=timezone.utc) > > l_test = [1, 2, 3] > > > def do_it(): > ??? global dt_expiry, l_test # asked python to refer to global > variables for both > > ??? # assign new value immediately > > ??? dt_expiry = datetime.now()+timedelta(minutes=5) > ??? print(dt_expiry.strftime("%Y-%m-%d %H:%M")) # just to show new > value has been assigned > ??? # grab copy of list for re-use of items > ??? l_temp = copy(l_test) > ??? # following line means l_test will later on retain value in global > scope because it was manipulated inside function instead of just > assigned new value > ??? l_test.clear() > ??? # replace original set of values > ??? for i in l_temp: l_test.append(i) > ??? # add new item > ??? l_test.append(99) > # end of do_it function > > # end code > > > If you import the contents of that file into the python interpreter, > dt_expiry will start off as "1970-01-01 00:00", and, if you execute > do_it function, it will print out the new value assigned to the > dt_expiry variable inside that function, but if you then again check the > value of the dt_expiry variable afterwards, it's reverted to the 1970... > value? Not when I run your code. With a little annotation added to the print statements I get (I added the import statements to make it run, and I used the same date-time formatting for all three print statements): List before: [1, 2, 3] start: 1970-01-01 00:00 inside after reassignment: 2024-03-06 08:57 outside after: 2024-03-06 08:57 List after: [1, 2, 3, 99] As an aside, you have gone to some trouble to copy, clear, and reconstruct l_test. It would be simpler like this (and you wouldn't have to import the "copy" library): l_temp = l_test[:] l_test = [] Instead of those lines and then this: for i in l_temp: l_test.append(i) you could achieve the same thing with this single statement: l_test = l_test[:] > > If I take out the line that removes values from l_test # l_test.clear() > # before appending new value to it, then it will also not retain it's > new/additional child items after the function exits, and will just > revert back to [1, 2, 3] each and every time. > > > In other words, with some of the variable/object types, if you use a > function that manipulates the contents of a variable, before then > re-assigning it a new value, it seems like it might then actually > update/manipulate the global variable, but, either just calling purely > content retrieval functions against said objects, or assigning them new > values from scratch seems to then ignore the global scope specified in > the first line inside the function? > > > Hope this makes more sense > > > Jacob Kruger > +2782 413 4791 > "Resistance is futile!...Acceptance is versatile..." > > > On 2024/03/05 20:23, dn via Python-list wrote: >> Jacob, >> >> Please reduce the problem to a small code-set which reproduces the >> problem. If we can reproduce same, then that tells us something. At >> the very least, we can experiment without having to expend amounts of >> time in a (likely faulty) bid to reproduce the same environment. >> >> Also, code is the ultimate description! >> >> >> Perhaps start with a small experiment: >> >> - after l_servers is created, print its id() >> - after the global statement, print its id() >> - after the clear/reassignment, print its id() >> >> Is Python always working with the same list? >> Please advise... >> >> >> On 6/03/24 07:13, Jacob Kruger via Python-list wrote: >>> Hi there >>> >>> >>> Working with python 3.11, and, issue that confused me for a little >>> while, trying to figure out what was occurring - unless am completely >>> confused, or missing something - was that, for example, when having >>> pre-defined a variable, and then included it in the global statement >>> inside a function, that function was still referring to a completely >>> local instance, without manipulating outside variable object at all >>> unless I first executed a form of referral to it, before then >>> possibly assigning a new value to it. >>> >>> >>> Now, this does not seem to occur consistently if, for example, I just >>> run bare-bones test code inside the python interpreter, but >>> consistently occurs inside my actual testing script. >>> >>> >>> Basically, in a file with python code in that am using for a form of >>> testing at the moment, at the top of the file, under all the import >>> statements, I initiate the existence of a list variable to make use of >>> >>> later: >>> >>> >>> # code snippet >>> >>> l_servers = [] >>> >>> # end of first code snippet >>> >>> >>> Then, lower down, inside a couple of different functions, the first line >>> inside the functions includes the following: >>> # code snippet >>> ???? global l_servers >>> # end code snippet >>> >>> That should, in theory, mean that if I assign a value to that variable >>> inside one of the functions, it should reflect globally? >>> >>> However, it seems like that, while inside those functions, it can be >>> assigned a new list of values, but if I then return to the scope outside >>> >>> the functions, it has reverted back to being an empty list = []? >>> >>> >>> The issue seems to specifically (or not) occur when I make a call to >>> one function, and, in the steps it's executing in one context, while >>> it's not doing anything to the list directly, it's then making a call >>> to the second function, which is then meant to repopulate the list >>> with a brand new set of values. >>> >>> >>> Now, what almost seems to be occurring, is that while just >>> manipulating the contents of a referenced variable is fine in this >>> context, the moment I try to reassign it, that's where the issue is >>> occurring . >>> >>> >>> Here are relevant excerpts from the file:- >>> >>> >>> # start code >>> >>> # original assignation in main part of file >>> >>> l_servers = [] >>> >>> >>> # function wich is initially being executed >>> >>> def interact(): >>> ???? global l_servers >>> ???? # extra code inbetween choosing what to carry out >>> >>> ???? # ... >>> >>> ???? # end of other code >>> >>> ???? bl_response, o_out = list_servers() >>> >>> ???? if bl_response: # just make sure other function call was successful >>> >>> ???????? l_servers.clear() # first make reference to global variable >>> >>> ???????? for srv in o_out: l_servers.append(srv) # now re-populate items >>> >>> ???? # end code snippet from inside interact function >>> >>> # end of interact function >>> >>> # end of code snippet >>> >>> >>> That other function being called from within, list_servers() was >>> initially just trying to populate the values inside the global list >>> variable itself, but was ending up in a similar fashion - reverting >>> to initial empty value, but, the above now seems to work, as long as >>> I first make reference to/manipulate/work with global variable >>> instead of just trying to reassign it a brand new value/set of items? >>> >>> >>> So, am I missing something obvious, have I forgotten about something >>> else - yes, know that if was working from within an embedded >>> function, I might need/want to then use the nonlocal statement >>> against that variable name, but, honestly, just not sure how this can >>> be occurring, and, it's not just with this one list variable, etc.? >>> >>> >>> If I try simple test code from within the python interpreter, using >>> different types of variables, this does also not seem to be the same >>> all the time, but, don't think it can relate to an iterable like a >>> list, or else, just in case, here is the code snippet with all the >>> import statements from the top of that file, in case something could >>> be overriding standard behaviour - not likely in this context, but, >>> really not sure what's occurring: >>> >>> # import code snippet >>> >>> import requests, time >>> from requests.auth import HTTPBasicAuth >>> import psutil as psu >>> import pytz >>> import bcrypt >>> from copy import copy >>> from datetime import datetime, timedelta, timezone >>> from dateutil.parser import parse >>> >>> # end of import snippet >>> >>> >>> Thanks if you have any ideas/thoughts on the matter >>> >>> >>> Jacob Kruger >>> +2782 413 4791 >>> "Resistance is futile!...Acceptance is versatile..." >>> >>> >> From grant.b.edwards at gmail.com Wed Mar 6 09:42:04 2024 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Wed, 6 Mar 2024 09:42:04 -0500 (EST) Subject: Can u help me? References: <6d6fcd5a-9e18-39ce-a300-dc6a1ed7a2dc@stoneleaf.us> <5d7caebb-2929-417e-8a53-4b804b9aa5fc@mrabarnett.plus.com> <4401d264-fb23-4d92-ae79-911f5a6f1194@mrabarnett.plus.com> Message-ID: <4TqZrr0zpgznV37@mail.python.org> On 2024-03-06, MRAB via Python-list wrote: > On 2024-03-06 01:44, Ethan Furman via Python-list wrote: >> On 3/5/24 16:49, MRAB via Python-list wrote: >> > On 2024-03-06 00:24, Ethan Furman via Python-list wrote: >> >> On 3/5/24 16:06, Chano Fucks via Python-list wrote: >> >> >> >>> [image: image.png] >> >> >> >> The image is of MS-Windows with the python installation window of "Repair Successful". Hopefully somebody better at >> >> explaining that problem can take it from here... >> >> >> > If the repair was successful, what's the problem? >> >> I imagine the issue is trying get Python to run (as I recall, the python icon on the MS-Windows desktop is the >> installer, not Python itself). > > There was an issue 3 years ago about renaming the installer for clarity: > > https://github.com/python/cpython/issues/87322 Yea, this problem comes up constantly (and has for many years). People have suggested renaming the installer so it has "setup" or "install" in the name. People have suggested adding text to the installer splash screen to explain that it's the installer and not python itself, that you already have python installed, and if you want to _run_ python instead of _install_ python, here's how. People have suggested having the installer remove itself by default when it's done installing. People have suggested lots of solutions. AFAICT, nobody has actually done anything. From roel at roelschroeven.net Wed Mar 6 10:39:09 2024 From: roel at roelschroeven.net (Roel Schroeven) Date: Wed, 6 Mar 2024 16:39:09 +0100 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: References: <0ccad7a9-eaba-48e6-b972-d89e5a930c11@DancesWithMice.info> Message-ID: Op 6/03/2024 om 13:55 schreef Jacob Kruger via Python-list: > If you import the contents of that file into the python interpreter, [...] What exactly to you mean by "import the contents of that file into the python interpreter"? Other people have put your code in a script, executed it, and saw it working as expected. I pasted in IPython, and likewise saw it working as expected, and the same with IDLE. It seems to me you must be doing something different from us; maybe the way you execute that code might be the key to this whole confusion. -- "Il semble que la perfection soit atteinte non quand il n'y a plus rien ? ajouter, mais quand il n'y a plus rien ? retrancher." "Perfectie is niet bereikt als er niets meer toe te voegen is, maar als er niets meer weg te nemen is." -- Antoine de Saint-Exup?ry From roel at roelschroeven.net Wed Mar 6 10:55:39 2024 From: roel at roelschroeven.net (Roel Schroeven) Date: Wed, 6 Mar 2024 16:55:39 +0100 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: References: <0ccad7a9-eaba-48e6-b972-d89e5a930c11@DancesWithMice.info> Message-ID: <23ddcc43-7ab5-49ea-a188-8fc94e0a78c1@roelschroeven.net> Op 6/03/2024 om 16:39 schreef Roel Schroeven via Python-list: > Op 6/03/2024 om 13:55 schreef Jacob Kruger via Python-list: >> If you import the contents of that file into the python interpreter, >> [...] > > What exactly to you mean by "import the contents of that file into the > python interpreter"? Other people have put your code in a script, > executed it, and saw it working as expected. I pasted in IPython, and > likewise saw it working as expected, and the same with IDLE. It seems > to me you must be doing something different from us; maybe the way you > execute that code might be the key to this whole confusion. (As an aside, I made a type; "What exactly _do_ you mean" is what I meant to type) If you want, you can play with the code online here: https://tio.run/##pVPbitswEH3XVwwJITaNQ9KllATy0EKhL/2AUsqiWKNkWlkykrzZ7NJvT0e at JWT7VmPsEZozc87RqD7Ho7MPl8sUQpQ@QukUCqG9q0DJiJEqBKpqx1vDegHp@@JsHyk0UfaY0tXnIT/FQogpkKVI0lBAcJ4OZKWBJ2kaDEKo at IjPNfkz7MYGyxB9nYJsst58XBWrNb@wWm1Xq8kCJrPvxawqZgpmX7ezb5N86bE2ssQsvpDVbjewWzaxzIUwjxFD5PI/1gt4v4CHn0xKoQblHilm at VYAPwfj9kxrpLOAHjcFGX6jgrp1CqIDjxp9CnrMk/Qk9wYDaOdh7 at JRtCUTMtDBgsVTp5edYbuIZZpzl/NP at dadsvzdaG1WkW2Yy@5D3mJqTzZmIzK5pTu37p3JmcOvhkUw2XB0pxsmRxlgj2h7jqh6ygcv990pOg18ZLEV5bFo0ulpoIhVaHOTP1XNvFN21rmV91W0M8adyB64hEWoUNowGHoiY8CwVg/sp8coyQ722MHTwEWRCZYy9SVGMd9KWqqbBFWcGkgh6MaWkbgOryNKlSi2igdZV8kj6RCXpUHps4FtPz/X4QwYU6F at RlNSMoESv071digk6xqtymgoJVXXMdl027DP22xyvg8cpfLt/I0KRL at RLiDwhHanPP@M3Brn at bAu2mdc6/k4B7vXMfxzs98R2L12/@tOPrb48owlz1fH575TMTbsr8sb at CfNR3kH@x9 at l8tf (sorry for the long URL; that's where tio.run puts the code) Again it works as expected, but with or without the l_test.clear() line. -- "Il semble que la perfection soit atteinte non quand il n'y a plus rien ? ajouter, mais quand il n'y a plus rien ? retrancher." "Perfectie is niet bereikt als er niets meer toe te voegen is, maar als er niets meer weg te nemen is." -- Antoine de Saint-Exup?ry From jacob.kruger.work at gmail.com Wed Mar 6 11:28:59 2024 From: jacob.kruger.work at gmail.com (Jacob Kruger) Date: Wed, 6 Mar 2024 18:28:59 +0200 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: <2248adf8-551f-4e86-8de8-be892d5978ed@tompassin.net> References: <0ccad7a9-eaba-48e6-b972-d89e5a930c11@DancesWithMice.info> <2248adf8-551f-4e86-8de8-be892d5978ed@tompassin.net> Message-ID: <9c71abb0-b28d-4646-b130-9a4fdd529428@gmail.com> Thanks for all your input people, and, yes, I know that besides the scope oddities the rest of the code is not my normal style either - was partly due to forms of experimentation to try figure out what could be causing issues. For example, instead of [:] syntax, was specifically using copy() to make sure was not interacting with original variable values, etc. This will be a bit longer - copying-pasting command line output here to show you what I truly mean - first session, where I am importing code into interpreter and second session where I retype exact same code behave differently: #---first session--- C:\temp\py_try>type scoping2.py from datetime import datetime, timezone, timedelta dt_expiry = datetime.strptime("1970-01-01 00:00", "%Y-%m-%d %H:%M").replace(tzinfo=timezone.utc) def do_it(): ??? global dt_expiry ??? dt_expiry = datetime.now()+timedelta(minutes=5) ??? print(dt_expiry.strftime("%Y-%m-%d %H:%M")) # end of do_it function C:\temp\py_try>python Python 3.11.7 (tags/v3.11.7:fa7a6f2, Dec? 4 2023, 19:24:49) [MSC v.1937 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from scoping2 import * >>> print(dt_expiry) 1970-01-01 00:00:00+00:00 >>> do_it() 2024-03-06 18:12 >>> print(dt_expiry) 1970-01-01 00:00:00+00:00 >>> #---end first session--- And, if I now retype the contents of the file into the python interpreter instead: #---start second session--- C:\temp\py_try>python Python 3.11.7 (tags/v3.11.7:fa7a6f2, Dec? 4 2023, 19:24:49) [MSC v.1937 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from datetime import datetime, timezone, timedelta >>> dt_expiry = datetime.strptime("1970-01-01 00:00", "%Y-%m-%d %H:%M").replace(tzinfo=timezone.utc) >>> def do_it(): ...???? global dt_expiry ...???? dt_expiry = datetime.now()+timedelta(minutes=5) ...???? print(dt_expiry.strftime("%Y-%m-%d %H:%M")) ... >>> print(dt_expiry) 1970-01-01 00:00:00+00:00 >>> do_it() 2024-03-06 18:20 >>> print(dt_expiry) 2024-03-06 18:20:03.909825 >>> #---end second session--- So, in the second session, where retyped everything, it behaves as I would expect it to, but, during first session, the variable is being treated as a local variable inside the function - no code differences since literally copied-pasted each and every line into console, but, a different behaviour nonetheless? So, yes, know this comes across like some form of a scam/joke, or list-garbage, since it doesn't make any sense to me at all, but still just wondering if missing something, or should I shift over to 3.12 to see if if works differently, or just try reinstalling 3.11 from scratch, or should I retry the above in something like the VS code console, or a different python console, etc.? Sorry Jacob Kruger Jacob Kruger +2782 413 4791 "Resistance is futile!...Acceptance is versatile..." On 2024/03/06 16:01, Thomas Passin via Python-list wrote: > On 3/6/2024 7:55 AM, Jacob Kruger via Python-list wrote: >> Ok, simpler version - all the code in a simpler test file, and >> working with two separate variables to explain exactly what am >> talking about: >> >> # start code >> >> from datetime import datetime, timezone, timedelta >> >> from copy import copy >> >> >> # initialise original values >> >> dt_expiry = datetime.strptime("1970-01-01 00:00", "%Y-%m-%d >> %H:%M").replace(tzinfo=timezone.utc) >> >> l_test = [1, 2, 3] >> >> >> def do_it(): >> ???? global dt_expiry, l_test # asked python to refer to global >> variables for both >> >> ???? # assign new value immediately >> >> ???? dt_expiry = datetime.now()+timedelta(minutes=5) >> ???? print(dt_expiry.strftime("%Y-%m-%d %H:%M")) # just to show new >> value has been assigned >> ???? # grab copy of list for re-use of items >> ???? l_temp = copy(l_test) >> ???? # following line means l_test will later on retain value in >> global scope because it was manipulated inside function instead of >> just assigned new value >> ???? l_test.clear() >> ???? # replace original set of values >> ???? for i in l_temp: l_test.append(i) >> ???? # add new item >> ???? l_test.append(99) >> # end of do_it function >> >> # end code >> >> >> If you import the contents of that file into the python interpreter, >> dt_expiry will start off as "1970-01-01 00:00", and, if you execute >> do_it function, it will print out the new value assigned to the >> dt_expiry variable inside that function, but if you then again check >> the value of the dt_expiry variable afterwards, it's reverted to the >> 1970... value? > > Not when I run your code. With a little annotation added to the print > statements I get (I added the import statements to make it run, and I > used the same date-time formatting for all three print statements): > > List before: [1, 2, 3] > start: 1970-01-01 00:00 > inside after reassignment: 2024-03-06 08:57 > outside after: 2024-03-06 08:57 > List after: [1, 2, 3, 99] > > As an aside, you have gone to some trouble to copy, clear, and > reconstruct l_test.? It would be simpler like this (and you wouldn't > have to import the "copy" library): > > ??? l_temp = l_test[:] > ??? l_test = [] > > Instead of those lines and then this: > > ??? for i in l_temp: l_test.append(i) > > you could achieve the same thing with this single statement: > > ??? l_test = l_test[:] >> >> If I take out the line that removes values from l_test # >> l_test.clear() # before appending new value to it, then it will also >> not retain it's new/additional child items after the function exits, >> and will just revert back to [1, 2, 3] each and every time. >> >> >> In other words, with some of the variable/object types, if you use a >> function that manipulates the contents of a variable, before then >> re-assigning it a new value, it seems like it might then actually >> update/manipulate the global variable, but, either just calling >> purely content retrieval functions against said objects, or assigning >> them new values from scratch seems to then ignore the global scope >> specified in the first line inside the function? >> >> >> Hope this makes more sense >> >> >> Jacob Kruger >> +2782 413 4791 >> "Resistance is futile!...Acceptance is versatile..." >> >> >> On 2024/03/05 20:23, dn via Python-list wrote: >>> Jacob, >>> >>> Please reduce the problem to a small code-set which reproduces the >>> problem. If we can reproduce same, then that tells us something. At >>> the very least, we can experiment without having to expend amounts >>> of time in a (likely faulty) bid to reproduce the same environment. >>> >>> Also, code is the ultimate description! >>> >>> >>> Perhaps start with a small experiment: >>> >>> - after l_servers is created, print its id() >>> - after the global statement, print its id() >>> - after the clear/reassignment, print its id() >>> >>> Is Python always working with the same list? >>> Please advise... >>> >>> >>> On 6/03/24 07:13, Jacob Kruger via Python-list wrote: >>>> Hi there >>>> >>>> >>>> Working with python 3.11, and, issue that confused me for a little >>>> while, trying to figure out what was occurring - unless am >>>> completely confused, or missing something - was that, for example, >>>> when having pre-defined a variable, and then included it in the >>>> global statement inside a function, that function was still >>>> referring to a completely local instance, without manipulating >>>> outside variable object at all unless I first executed a form of >>>> referral to it, before then possibly assigning a new value to it. >>>> >>>> >>>> Now, this does not seem to occur consistently if, for example, I >>>> just run bare-bones test code inside the python interpreter, but >>>> consistently occurs inside my actual testing script. >>>> >>>> >>>> Basically, in a file with python code in that am using for a form of >>>> testing at the moment, at the top of the file, under all the import >>>> statements, I initiate the existence of a list variable to make use of >>>> >>>> later: >>>> >>>> >>>> # code snippet >>>> >>>> l_servers = [] >>>> >>>> # end of first code snippet >>>> >>>> >>>> Then, lower down, inside a couple of different functions, the first >>>> line >>>> inside the functions includes the following: >>>> # code snippet >>>> ???? global l_servers >>>> # end code snippet >>>> >>>> That should, in theory, mean that if I assign a value to that variable >>>> inside one of the functions, it should reflect globally? >>>> >>>> However, it seems like that, while inside those functions, it can be >>>> assigned a new list of values, but if I then return to the scope >>>> outside >>>> >>>> the functions, it has reverted back to being an empty list = []? >>>> >>>> >>>> The issue seems to specifically (or not) occur when I make a call >>>> to one function, and, in the steps it's executing in one context, >>>> while it's not doing anything to the list directly, it's then >>>> making a call to the second function, which is then meant to >>>> repopulate the list with a brand new set of values. >>>> >>>> >>>> Now, what almost seems to be occurring, is that while just >>>> manipulating the contents of a referenced variable is fine in this >>>> context, the moment I try to reassign it, that's where the issue is >>>> occurring . >>>> >>>> >>>> Here are relevant excerpts from the file:- >>>> >>>> >>>> # start code >>>> >>>> # original assignation in main part of file >>>> >>>> l_servers = [] >>>> >>>> >>>> # function wich is initially being executed >>>> >>>> def interact(): >>>> ???? global l_servers >>>> ???? # extra code inbetween choosing what to carry out >>>> >>>> ???? # ... >>>> >>>> ???? # end of other code >>>> >>>> ???? bl_response, o_out = list_servers() >>>> >>>> ???? if bl_response: # just make sure other function call was >>>> successful >>>> >>>> ???????? l_servers.clear() # first make reference to global variable >>>> >>>> ???????? for srv in o_out: l_servers.append(srv) # now re-populate >>>> items >>>> >>>> ???? # end code snippet from inside interact function >>>> >>>> # end of interact function >>>> >>>> # end of code snippet >>>> >>>> >>>> That other function being called from within, list_servers() was >>>> initially just trying to populate the values inside the global list >>>> variable itself, but was ending up in a similar fashion - reverting >>>> to initial empty value, but, the above now seems to work, as long >>>> as I first make reference to/manipulate/work with global variable >>>> instead of just trying to reassign it a brand new value/set of items? >>>> >>>> >>>> So, am I missing something obvious, have I forgotten about >>>> something else - yes, know that if was working from within an >>>> embedded function, I might need/want to then use the nonlocal >>>> statement against that variable name, but, honestly, just not sure >>>> how this can be occurring, and, it's not just with this one list >>>> variable, etc.? >>>> >>>> >>>> If I try simple test code from within the python interpreter, using >>>> different types of variables, this does also not seem to be the >>>> same all the time, but, don't think it can relate to an iterable >>>> like a list, or else, just in case, here is the code snippet with >>>> all the import statements from the top of that file, in case >>>> something could be overriding standard behaviour - not likely in >>>> this context, but, really not sure what's occurring: >>>> >>>> # import code snippet >>>> >>>> import requests, time >>>> from requests.auth import HTTPBasicAuth >>>> import psutil as psu >>>> import pytz >>>> import bcrypt >>>> from copy import copy >>>> from datetime import datetime, timedelta, timezone >>>> from dateutil.parser import parse >>>> >>>> # end of import snippet >>>> >>>> >>>> Thanks if you have any ideas/thoughts on the matter >>>> >>>> >>>> Jacob Kruger >>>> +2782 413 4791 >>>> "Resistance is futile!...Acceptance is versatile..." >>>> >>>> >>> > From jacob.kruger.work at gmail.com Wed Mar 6 11:40:40 2024 From: jacob.kruger.work at gmail.com (Jacob Kruger) Date: Wed, 6 Mar 2024 18:40:40 +0200 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: References: <0ccad7a9-eaba-48e6-b972-d89e5a930c11@DancesWithMice.info> Message-ID: Matt, other mail is more relevant - seems to maybe have more to do with different behavour if import code, or not - no, does not make sense to me - but, here's the command line contents including printing out id() results, but, only working via importing code: #---start session--- C:\temp\py_try>type scoping2.py from datetime import datetime, timezone, timedelta dt_expiry = datetime.strptime("1970-01-01 00:00", "%Y-%m-%d %H:%M").replace(tzinfo=timezone.utc) def do_it(): ??? global dt_expiry ??? dt_expiry = datetime.now()+timedelta(minutes=5) ??? print("date value", dt_expiry.strftime("%Y-%m-%d %H:%M")) ??? print("ID", id(dt_expiry)) # end of do_it function C:\temp\py_try>python Python 3.11.7 (tags/v3.11.7:fa7a6f2, Dec? 4 2023, 19:24:49) [MSC v.1937 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from scoping2 import * >>> print(dt_expiry) 1970-01-01 00:00:00+00:00 >>> print(id(dt_expiry)) 1808577867152 >>> do_it() date value 2024-03-06 18:39 ID 1808572660736 >>> print(dt_expiry) 1970-01-01 00:00:00+00:00 >>> print(id(dt_expiry)) 1808577867152 >>> ---end session--- As in, the two different ID values are being returned outside and inside the function, whereas, if I included that bit inside the interpreter while typing code manually, chances are the same ID would be retained both inside and outside function. Jacob Kruger +2782 413 4791 "Resistance is futile!...Acceptance is versatile..." On 2024/03/06 15:57, Mats Wichmann via Python-list wrote: > On 3/6/24 05:55, Jacob Kruger via Python-list wrote: >> Ok, simpler version - all the code in a simpler test file, and >> working with two separate variables to explain exactly what am >> talking about: > >> If you import the contents of that file into the python interpreter, >> dt_expiry will start off as "1970-01-01 00:00", and, if you execute >> do_it function, it will print out the new value assigned to the >> dt_expiry variable inside that function, but if you then again check >> the value of the dt_expiry variable afterwards, it's reverted to the >> 1970... value? >> >> >> If I take out the line that removes values from l_test # >> l_test.clear() # before appending new value to it, then it will also >> not retain it's new/additional child items after the function exits, >> and will just revert back to [1, 2, 3] each and every time. >> >> >> In other words, with some of the variable/object types, if you use a >> function that manipulates the contents of a variable, before then >> re-assigning it a new value, it seems like it might then actually >> update/manipulate the global variable, but, either just calling >> purely content retrieval functions against said objects, or assigning >> them new values from scratch seems to then ignore the global scope >> specified in the first line inside the function? >> >> >> Hope this makes more sense > > No, it doesn't. Your code is working as one would expect. For example, > adding prints for the l_test variable, and removing the .clear() which > you claim makes it not work, shows me: > > before: l_test=[1, 2, 3], id(l_test)=140153285385856 > leaving do_it: l_test=[1, 2, 3, 1, 2, 3, 99], id(l_test)=140153285385856 > after: l_test=[1, 2, 3, 1, 2, 3, 99], id(l_test)=140153285385856 > > It's the same list object, as you can see by the id values. And the > list is updating as expected. > > And... you don't need the global statement for l_test. As it's > mutable, you can mutate it in the function; the global only acts on > assignment. Using "global" for that may make your intent more clear to > readers though, although static checkers will grumble at you. > > You must be doing something additional that you're not telling us about. > > From jacob.kruger.work at gmail.com Wed Mar 6 11:50:31 2024 From: jacob.kruger.work at gmail.com (Jacob Kruger) Date: Wed, 6 Mar 2024 18:50:31 +0200 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: References: <0ccad7a9-eaba-48e6-b972-d89e5a930c11@DancesWithMice.info> Message-ID: <1479f660-61e8-45f1-9490-930140c6b1b4@gmail.com> You'll see more details in other mail, but, here I am firing up standard python interpreter from within windows terminal, and then executing following line: from scoping2 import * And, this is under windows 11 windows terminal, which is where I generally interact with my python code, via command line - generally working with flask, and/or other forms of command line interaction, most of the time. Jacob Kruger +2782 413 4791 "Resistance is futile!...Acceptance is versatile..." On 2024/03/06 17:39, Roel Schroeven via Python-list wrote: > Op 6/03/2024 om 13:55 schreef Jacob Kruger via Python-list: >> If you import the contents of that file into the python interpreter, >> [...] > > What exactly to you mean by "import the contents of that file into the > python interpreter"? Other people have put your code in a script, > executed it, and saw it working as expected. I pasted in IPython, and > likewise saw it working as expected, and the same with IDLE. It seems > to me you must be doing something different from us; maybe the way you > execute that code might be the key to this whole confusion. > From ethan at stoneleaf.us Wed Mar 6 11:57:27 2024 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 6 Mar 2024 08:57:27 -0800 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: <9c71abb0-b28d-4646-b130-9a4fdd529428@gmail.com> References: <0ccad7a9-eaba-48e6-b972-d89e5a930c11@DancesWithMice.info> <2248adf8-551f-4e86-8de8-be892d5978ed@tompassin.net> <9c71abb0-b28d-4646-b130-9a4fdd529428@gmail.com> Message-ID: <3d941cc0-2379-4771-e60c-5e14f31001fa@stoneleaf.us> On 3/6/24 08:28, Jacob Kruger via Python-list wrote: > C:\temp\py_try>python > Python 3.11.7 (tags/v3.11.7:fa7a6f2, Dec 4 2023, 19:24:49) [MSC v.1937 64 bit (AMD64)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> from scoping2 import * And it becomes clear: only do `from ... import *` when the module has been specifically designed to support that. If you were to also do `import scoping2` and, after calling `do_it()`, `print(scoping2.dt_expiry)`, you would see that it had changed. I know there are good explanations for how variables and names work in Python, but I couldn't find any at the moment. Sorry. -- ~Ethan~ From jacob.kruger.work at gmail.com Wed Mar 6 12:29:12 2024 From: jacob.kruger.work at gmail.com (Jacob Kruger) Date: Wed, 6 Mar 2024 19:29:12 +0200 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: <3d941cc0-2379-4771-e60c-5e14f31001fa@stoneleaf.us> References: <0ccad7a9-eaba-48e6-b972-d89e5a930c11@DancesWithMice.info> <2248adf8-551f-4e86-8de8-be892d5978ed@tompassin.net> <9c71abb0-b28d-4646-b130-9a4fdd529428@gmail.com> <3d941cc0-2379-4771-e60c-5e14f31001fa@stoneleaf.us> Message-ID: Ok, Ethan, that makes sense - I generally work with modules in folders, etc., but, this was just test code, but, 'see' if I instead import scoping2 as sc2, and then refer to sc2.dt_expiry and sc2.do_it, then it does operate as it should - thanks, again. Jacob Kruger +2782 413 4791 "Resistance is futile!...Acceptance is versatile..." On 2024/03/06 18:57, Ethan Furman via Python-list wrote: > On 3/6/24 08:28, Jacob Kruger via Python-list wrote: > > > C:\temp\py_try>python > > Python 3.11.7 (tags/v3.11.7:fa7a6f2, Dec? 4 2023, 19:24:49) [MSC > v.1937 64 bit (AMD64)] on win32 > > Type "help", "copyright", "credits" or "license" for more information. > >? >>> from scoping2 import * > > And it becomes clear:? only do `from ... import *` when the module has > been specifically designed to support that. > > If you were to also do `import scoping2` and, after calling `do_it()`, > `print(scoping2.dt_expiry)`, you would see that it had changed. > > I know there are good explanations for how variables and names work in > Python, but I couldn't find any at the moment. Sorry. > > -- > ~Ethan~ From roel at roelschroeven.net Wed Mar 6 12:33:17 2024 From: roel at roelschroeven.net (Roel Schroeven) Date: Wed, 6 Mar 2024 18:33:17 +0100 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: References: <0ccad7a9-eaba-48e6-b972-d89e5a930c11@DancesWithMice.info> Message-ID: <2007e9a9-01e9-48fd-b060-56b9ac4db4c9@roelschroeven.net> Op 6/03/2024 om 17:40 schreef Jacob Kruger via Python-list: > >>> from scoping2 import * Ah yes, that explains what's happening. After that statement, the name dt_expiry in the current namespace is bound to the same object that the name dt_expiry in the namespace of module scoping2 is bound to. Function do_it re-binds that last one to a new one, with the new value; name dt_expiry in the current namespace is still bound to the old object. (If all of that sounds like gibberish, have a look at "Facts and myths about Python names and values" (text: https://nedbatchelder.com/text/names.html; slides and video: https://nedbatchelder.com/text/names1.html) I would advice not to use 'import *', if at all possible, for multiple reasons, one of which is to prevent problems like this. I would also advice not to use global variables from other modules directly, and in fact would advice to minimize the use of globals in general as much as possible. If you need to keep state between methods, it might be better to use a class. -- "There is a theory which states that if ever anyone discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable. There is another theory which states that this has already happened." -- Douglas Adams, The Restaurant at the End of the Universe From grant.b.edwards at gmail.com Wed Mar 6 12:59:38 2024 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Wed, 6 Mar 2024 12:59:38 -0500 (EST) Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference References: <0ccad7a9-eaba-48e6-b972-d89e5a930c11@DancesWithMice.info> <2007e9a9-01e9-48fd-b060-56b9ac4db4c9@roelschroeven.net> Message-ID: <4TqgDp5mGRznVGY@mail.python.org> On 2024-03-06, Roel Schroeven via Python-list wrote: > Op 6/03/2024 om 17:40 schreef Jacob Kruger via Python-list: >> >>> from scoping2 import * > > [...] > > I would advice not to use 'import *', if at all possible, for multiple > reasons, one of which is to prevent problems like this. Unfortunately, many (most?) tutorials for particular modules (and even example code in the Python documentation itself) are all written assuming that you do "from import *". It saves the tutorial write a few keystrokes, but causes untold trouble for people who learn incorrectly that "from import *" is the proper way to do things. > I would also advice not to use global variables from other modules > directly, and in fact would advice to minimize the use of globals in > general as much as possible. If you need to keep state between > methods, it might be better to use a class. From roel at roelschroeven.net Wed Mar 6 13:28:29 2024 From: roel at roelschroeven.net (Roel Schroeven) Date: Wed, 6 Mar 2024 19:28:29 +0100 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: <4TqgDp5mGRznVGY@mail.python.org> References: <0ccad7a9-eaba-48e6-b972-d89e5a930c11@DancesWithMice.info> <2007e9a9-01e9-48fd-b060-56b9ac4db4c9@roelschroeven.net> <4TqgDp5mGRznVGY@mail.python.org> Message-ID: <78d2710e-ed8e-45fd-8268-5081f130cf8a@roelschroeven.net> Grant Edwards via Python-list schreef op 6/03/2024 om 18:59: > On 2024-03-06, Roel Schroeven via Python-list > wrote: > > Op 6/03/2024 om 17:40 schreef Jacob Kruger via Python-list: > >> >>> from scoping2 import * > > > > [...] > > > > I would advice not to use 'import *', if at all possible, for > multiple > reasons, one of which is to prevent problems like this. > > Unfortunately, many (most?) tutorials for particular modules (and even > example code in the Python documentation itself) are all written > assuming that you do "from import *". It saves the tutorial > write a few keystrokes, but causes untold trouble for people who learn > incorrectly that "from import *" is the proper way to do > things. I know ... it's really irritating. -- "There is a theory which states that if ever anyone discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable. There is another theory which states that this has already happened." -- Douglas Adams, The Restaurant at the End of the Universe From PythonList at DancesWithMice.info Wed Mar 6 20:10:11 2024 From: PythonList at DancesWithMice.info (dn) Date: Thu, 7 Mar 2024 14:10:11 +1300 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: <9c71abb0-b28d-4646-b130-9a4fdd529428@gmail.com> References: <0ccad7a9-eaba-48e6-b972-d89e5a930c11@DancesWithMice.info> <2248adf8-551f-4e86-8de8-be892d5978ed@tompassin.net> <9c71abb0-b28d-4646-b130-9a4fdd529428@gmail.com> Message-ID: On 7/03/24 05:28, Jacob Kruger via Python-list wrote: ... > So, yes, know this comes across like some form of a scam/joke, or > list-garbage, since it doesn't make any sense to me at all, but still > just wondering if missing something, or should I shift over to 3.12 to > see if if works differently, or just try reinstalling 3.11 from scratch, > or should I retry the above in something like the VS code console, or a > different python console, etc.? Some of the facts, such as HOW the code was being executed were missing (see earlier request for a cut-down scenario, AND reports from others saying 'but it works for me'). The idea of importing a module into the REPL and then (repeatedly) manually entering the code to set-up and execute is unusual (surely type such into a script (once), and run that (repeatedly). As you say, most of us would be working from an IDE and hitting 'Run'. Am wondering why you weren't - but it's not important. That said, the REPL is the perfect place to learn, experiment, and prototype - particularly when compared with the facilities of other language's eco-systems. The entirety of the on-line Tutorial cannot be wrong! (although, after a quick review, I've failed to see where the Tutorial mentions the usual development mode, apart from two very brief asides (the most useful is almost at the very end(?)) - but then (as they say) the objective is to show the language! The lesson-learned is that there are different 'environments' and different ways of building the environment in which the code will run. That's a valuable lesson, and full of subtlety! Glad to see that comparing id()s was useful - for diagnosis but not solution. Other tools might include the locals() and globals() functions. You may also have detected that many of us try to avoid globals and the implicit assumptions about the behavior of mutable collections (eg lists) when treated as 'global'. Then there are "closures", the "LEGB" rule, namespaces, scope, and ... -- -- Regards, =dn From grant.b.edwards at gmail.com Wed Mar 6 20:55:40 2024 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Wed, 6 Mar 2024 20:55:40 -0500 (EST) Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference References: <0ccad7a9-eaba-48e6-b972-d89e5a930c11@DancesWithMice.info> <2248adf8-551f-4e86-8de8-be892d5978ed@tompassin.net> <9c71abb0-b28d-4646-b130-9a4fdd529428@gmail.com> Message-ID: <4Tqsp41DHBznTq9@mail.python.org> On 2024-03-07, dn via Python-list wrote: > The idea of importing a module into the REPL and then (repeatedly) > manually entering the code to set-up and execute is unusual (surely type > such into a script (once), and run that (repeatedly). As you say, most > of us would be working from an IDE and hitting 'Run'. Am wondering why > you weren't - but it's not important. Unless the code is intended to be used as a module, 'import'ing it into the REPL doesn't make sense. A simple example: ---------------------------testit.py------------------------------ x = 'x' y = 'y' def foo(): global y print("hi") x = 'X' y = 'Y' print(x) print(y) ------------------------------------------------------------------ The usual method to play with that interactively is $ python -i testit.py >>> x 'x' >>> y 'y' >>> foo() hi X Y >>> x 'x' >>> y 'Y' >>> As we've seen, doing a 'from testit.py import *' doesn't let you test what the OP was trying to test. Doing 'import testit.py' gets you closer, but it's a hassle to test code that way. The right thing to do is 'python -i ' (or the equivalent button/option in an IDE). https://docs.python.org/3/tutorial/interpreter.html If you intended to use testit.py as a module, and wanted to experiment with its behavior as a module, then go ahead and import it. But, don't do 'from testit.py import *' until 1. you know how that differs from 'import testit.py' and 2. you want to use that difference From jacob.kruger.work at gmail.com Thu Mar 7 03:23:27 2024 From: jacob.kruger.work at gmail.com (Jacob Kruger) Date: Thu, 7 Mar 2024 10:23:27 +0200 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: <4Tqsp41DHBznTq9@mail.python.org> References: <0ccad7a9-eaba-48e6-b972-d89e5a930c11@DancesWithMice.info> <2248adf8-551f-4e86-8de8-be892d5978ed@tompassin.net> <9c71abb0-b28d-4646-b130-9a4fdd529428@gmail.com> <4Tqsp41DHBznTq9@mail.python.org> Message-ID: Thanks again, all. I think the python -i scoping2.py would have given me a good beginning as well - will archive that one for use. And, to maybe explain how I work - not an excuse at all - but, I am actually 100% blind, so a lot of the IDE's, or their common means/methods of interaction don't suit me all the time, which is why I generally work via programmer's text editor interfaces, or treat something like VS code as such, but then still prefer to run my code via command line, using pdb to then play around with forms of debugging, etc. And, yes, also generally prefer to work via classes, modules, etc. at runtime, but this was more or less mostly testing, which then caused confusion/interference on my side...LOL! Jacob Kruger +2782 413 4791 "Resistance is futile!...Acceptance is versatile..." On 2024/03/07 03:55, Grant Edwards via Python-list wrote: > On 2024-03-07, dn via Python-list wrote: > >> The idea of importing a module into the REPL and then (repeatedly) >> manually entering the code to set-up and execute is unusual (surely type >> such into a script (once), and run that (repeatedly). As you say, most >> of us would be working from an IDE and hitting 'Run'. Am wondering why >> you weren't - but it's not important. > Unless the code is intended to be used as a module, 'import'ing it into > the REPL doesn't make sense. > > A simple example: > > ---------------------------testit.py------------------------------ > x = 'x' > y = 'y' > def foo(): > global y > print("hi") > x = 'X' > y = 'Y' > print(x) > print(y) > ------------------------------------------------------------------ > > The usual method to play with that interactively is > > $ python -i testit.py > >>> x > 'x' > >>> y > 'y' > >>> foo() > hi > X > Y > >>> x > 'x' > >>> y > 'Y' > >>> > > As we've seen, doing a 'from testit.py import *' doesn't let you test > what the OP was trying to test. Doing 'import testit.py' gets you > closer, but it's a hassle to test code that way. The right thing to do > is 'python -i ' (or the equivalent button/option in an IDE). > > https://docs.python.org/3/tutorial/interpreter.html > > If you intended to use testit.py as a module, and wanted to experiment > with its behavior as a module, then go ahead and import it. But, don't > do 'from testit.py import *' until > > 1. you know how that differs from 'import testit.py' > > and > > 2. you want to use that difference > > From varunaseneviratna at gmail.com Thu Mar 7 09:11:11 2024 From: varunaseneviratna at gmail.com (Varuna Seneviratna) Date: Thu, 7 Mar 2024 19:41:11 +0530 Subject: If a dictionary key has a Python list as its value! Message-ID: If a dictionary key has a Python list as its value, you can read the values one by one in the list using a for-loop like in the following. d = {k: [1,2,3]} > for v in d[k]: > print(v) No tutorial describes this, why? What is the Python explanation for this behaviour? Varuna From python at mrabarnett.plus.com Thu Mar 7 13:23:46 2024 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 7 Mar 2024 18:23:46 +0000 Subject: If a dictionary key has a Python list as its value! In-Reply-To: References: Message-ID: On 2024-03-07 14:11, Varuna Seneviratna via Python-list wrote: > If a dictionary key has a Python list as its value, you can read the values > one by one in the list using a for-loop like in the following. > > d = {k: [1,2,3]} > > >> for v in d[k]: >> print(v) > > > No tutorial describes this, why? > What is the Python explanation for this behaviour? > If the value is a list, you can do list things to it. If the value is a number, you can do number things to it. If the value is a string, you can do string things to it. And so on. It's not mentioned in tutorials because it's not special. It just behaves how you'd expect it to behave. From mats at wichmann.us Thu Mar 7 13:29:07 2024 From: mats at wichmann.us (Mats Wichmann) Date: Thu, 7 Mar 2024 11:29:07 -0700 Subject: If a dictionary key has a Python list as its value! In-Reply-To: References: Message-ID: <7c299dae-6ed7-4ffd-8843-0f19f1de831a@wichmann.us> On 3/7/24 07:11, Varuna Seneviratna via Python-list wrote: > If a dictionary key has a Python list as its value, you can read the values > one by one in the list using a for-loop like in the following. > > d = {k: [1,2,3]} > > >> for v in d[k]: >> print(v) > > > No tutorial describes this, why? > What is the Python explanation for this behaviour? Sorry... why is this a surprise? If an object is iterable, you can iterate over it. >>> d = {'key': [1, 2, 3]} >>> type(d['key']) >>> val = d['key'] >>> type(val) >>> for v in val: ... print(v) ... ... 1 2 3 >>> From cs at cskk.id.au Thu Mar 7 17:48:08 2024 From: cs at cskk.id.au (Cameron Simpson) Date: Fri, 8 Mar 2024 09:48:08 +1100 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: References: Message-ID: On 06Mar2024 15:12, Jacob Kruger wrote: >So, this does not make sense to me in terms of the following snippet >from the official python docs page: >https://docs.python.org/3/faq/programming.html > >"In Python, variables that are only referenced inside a function are >implicitly global. If a variable is assigned a value anywhere within >the function?s body, it?s assumed to be a local unless explicitly >declared as global." > >So, I would then assume that if I explicitly include a variable name >inside the global statement, then even just assigning it a new value >should update the variable in the global context, outside the >function? Yes. Note that the "global" namespace is the module in which the function is defined. x = 1 def f(n): global x x += n This updates the `x` global variable in the module where `f` was defined. If you import `f` and use it in another module it will _still_ update `x` in the original module namespace. From grant.b.edwards at gmail.com Fri Mar 8 08:49:24 2024 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Fri, 8 Mar 2024 08:49:24 -0500 (EST) Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference References: Message-ID: <4Trnb856LRznVF4@mail.python.org> On 2024-03-07, Cameron Simpson via Python-list wrote: > Yes. Note that the "global" namespace is the module in which the > function is defined. One might argue that "global" isn't a good choice for what to call the scope in question, since it's not global. It's limited to that source file. It doesn't make sense to me to call a binding "global", when there can be multile different "global" bindings of the same name. -- Grant From rosuav at gmail.com Fri Mar 8 08:57:40 2024 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 9 Mar 2024 00:57:40 +1100 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: <4Trnb856LRznVF4@mail.python.org> References: <4Trnb856LRznVF4@mail.python.org> Message-ID: On Sat, 9 Mar 2024 at 00:51, Grant Edwards via Python-list wrote: > One might argue that "global" isn't a good choice for what to call the > scope in question, since it's not global. It's limited to that source > file. It doesn't make sense to me to call a binding "global", when > there can be multile different "global" bindings of the same name. > Most "globals" aren't global either, since you can have different globals in different running applications. ChrisA From grant.b.edwards at gmail.com Fri Mar 8 11:41:10 2024 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Fri, 8 Mar 2024 11:41:10 -0500 (EST) Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference References: <4Trnb856LRznVF4@mail.python.org> Message-ID: <4TrsPL1jK7znV1q@mail.python.org> On 2024-03-08, Chris Angelico via Python-list wrote: > On Sat, 9 Mar 2024 at 00:51, Grant Edwards via Python-list > wrote: > >> One might argue that "global" isn't a good choice for what to call the >> scope in question, since it's not global. It's limited to that source >> file. It doesn't make sense to me to call a binding "global", when >> there can be multile different "global" bindings of the same name. > > Most "globals" aren't global either, since you can have different > globals in different running applications. To me, "global" has always been limited to within a single process/address space, but that's probably just bias left over from C/Pascal/FORTRAN/assembly/etc. It never occurred to me that a global called "X" in one program on one computer would be the same as a global called "X" in a different program on a different computer somewhere else on the "globe". From sjeik_appie at hotmail.com Fri Mar 8 13:03:40 2024 From: sjeik_appie at hotmail.com (Albert-Jan Roskam) Date: Fri, 08 Mar 2024 19:03:40 +0100 Subject: pathlib.Path.is_file vs os.path.isfile difference Message-ID: Hi, I was replacing some os.path stuff with Pathlib and I discovered this: Path(256 * "x").is_file() # OSError os.path.isfile(256 * "x") # bool Is this intended? Does pathlib try to resemble os.path as closely as possible? Best wishes, Albert-Jan From list1 at tompassin.net Fri Mar 8 13:35:47 2024 From: list1 at tompassin.net (Thomas Passin) Date: Fri, 8 Mar 2024 13:35:47 -0500 Subject: pathlib.Path.is_file vs os.path.isfile difference In-Reply-To: References: Message-ID: <33cad416-6f2d-4e1e-9c86-227e30e220fd@tompassin.net> On 3/8/2024 1:03 PM, Albert-Jan Roskam via Python-list wrote: > Hi, > I was replacing some os.path stuff with Pathlib and I discovered this: > Path(256 * "x").is_file() # OSError > os.path.isfile(256 * "x") # bool > Is this intended? Does pathlib try to resemble os.path as closely as > possible? You must have an very old version of Python. I'm running 3.12.2 and it returns False. Either that or that path name exists and throws some kind of unexpected exception. The Python docs say "Return True if the path points to a regular file (or a symbolic link pointing to a regular file), False if it points to another kind of file. False is also returned if the path doesn?t exist or is a broken symlink; other errors (such as permission errors) are propagated" From grant.b.edwards at gmail.com Fri Mar 8 14:21:39 2024 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Fri, 8 Mar 2024 14:21:39 -0500 (EST) Subject: pathlib.Path.is_file vs os.path.isfile difference References: =?utf-8?q?=3CDB9PR10MB6689EA455D5C8580B90EA8F383272=40DB9PR10MB6?= =?utf-8?q?689=2EEURPRD10=2EPROD=2EOUTLOOK=2ECOM=3E?= <33cad416-6f2d-4e1e-9c86-227e30e220fd@tompassin.net> Message-ID: <4TrwyW50knznVDr@mail.python.org> On 2024-03-08, Thomas Passin via Python-list wrote: > On 3/8/2024 1:03 PM, Albert-Jan Roskam via Python-list wrote: >> Hi, >> I was replacing some os.path stuff with Pathlib and I discovered this: >> Path(256 * "x").is_file() # OSError >> os.path.isfile(256 * "x") # bool >> Is this intended? Does pathlib try to resemble os.path as closely as >> possible? > > You must have an very old version of Python. I'm running 3.12.2 and it > returns False. It throws OSError with Python 3.11.8 on Linux. $ python Python 3.11.8 (main, Feb 23 2024, 16:11:29) [GCC 13.2.1 20240113] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pathlib >>> pathlib.Path(256 * "x").is_file() Traceback (most recent call last): File "", line 1, in File "/usr/lib/python3.11/pathlib.py", line 1267, in is_file return S_ISREG(self.stat().st_mode) ^^^^^^^^^^^ File "/usr/lib/python3.11/pathlib.py", line 1013, in stat return os.stat(self, follow_symlinks=follow_symlinks) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ OSError: [Errno 36] File name too long: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' >>> >>> import os >>> os.path.isfile(256 * "x") False From grant.b.edwards at gmail.com Fri Mar 8 14:30:51 2024 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Fri, 8 Mar 2024 14:30:51 -0500 (EST) Subject: pathlib.Path.is_file vs os.path.isfile difference References: <33cad416-6f2d-4e1e-9c86-227e30e220fd@tompassin.net> <4TrwyW50knznVDr@mail.python.org> Message-ID: <4Trx9751XKznVF4@mail.python.org> On 2024-03-08, Grant Edwards via Python-list wrote: > On 2024-03-08, Thomas Passin via Python-list wrote: >> On 3/8/2024 1:03 PM, Albert-Jan Roskam via Python-list wrote: >>> Hi, >>> I was replacing some os.path stuff with Pathlib and I discovered this: >>> Path(256 * "x").is_file() # OSError >>> os.path.isfile(256 * "x") # bool >>> Is this intended? Does pathlib try to resemble os.path as closely as >>> possible? >> >> You must have an very old version of Python. I'm running 3.12.2 and it >> returns False. > > It throws OSError with Python 3.11.8 on Linux. > OSError: [Errno 36] File name too long: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' On all of the Linux filesystems I know about, the max length for a filename is 255 bytes, so the OSError is too surprising, and it does seem to follow the documentation. >>>> import os >>>> os.path.isfile(256 * "x") > False However, os.path.isfile() apprently masks that error somehow and returns False instead. I notice that the os.path.isfile() documentation does not specify what happens if the path is not a file or is illegal. It only specifies that True is returned if the path is a regular file. Presumably something other than "return True" is supposed to happen, but exactly what is not specified. From grant.b.edwards at gmail.com Fri Mar 8 14:37:24 2024 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Fri, 8 Mar 2024 14:37:24 -0500 (EST) Subject: pathlib.Path.is_file vs os.path.isfile difference References: <33cad416-6f2d-4e1e-9c86-227e30e220fd@tompassin.net> <4TrwyW50knznVDr@mail.python.org> <4Trx9751XKznVF4@mail.python.org> Message-ID: <4TrxJh3L4qznVF4@mail.python.org> On 2024-03-08, Grant Edwards via Python-list wrote: >> OSError: [Errno 36] File name too long: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' > > On all of the Linux filesystems I know about, the max length for a > filename is 255 bytes, so the OSError is too surprising, and it does > seem to follow the documentation. Doh. I meant "is not too surprising". From list1 at tompassin.net Fri Mar 8 15:06:24 2024 From: list1 at tompassin.net (Thomas Passin) Date: Fri, 8 Mar 2024 15:06:24 -0500 Subject: pathlib.Path.is_file vs os.path.isfile difference In-Reply-To: <4TrwyW50knznVDr@mail.python.org> References: <33cad416-6f2d-4e1e-9c86-227e30e220fd@tompassin.net> <4TrwyW50knznVDr@mail.python.org> Message-ID: <4155b23f-f7a0-484f-85f4-5c611908ae0f@tompassin.net> On 3/8/2024 2:21 PM, Grant Edwards via Python-list wrote: > On 2024-03-08, Thomas Passin via Python-list wrote: >> On 3/8/2024 1:03 PM, Albert-Jan Roskam via Python-list wrote: >>> Hi, >>> I was replacing some os.path stuff with Pathlib and I discovered this: >>> Path(256 * "x").is_file() # OSError >>> os.path.isfile(256 * "x") # bool >>> Is this intended? Does pathlib try to resemble os.path as closely as >>> possible? >> >> You must have an very old version of Python. I'm running 3.12.2 and it >> returns False. > > It throws OSError with Python 3.11.8 on Linux. Sorry, I should have said on Windows. > > $ python > Python 3.11.8 (main, Feb 23 2024, 16:11:29) [GCC 13.2.1 20240113] on linux > Type "help", "copyright", "credits" or "license" for more information. >>>> import pathlib >>>> pathlib.Path(256 * "x").is_file() > Traceback (most recent call last): > File "", line 1, in > File "/usr/lib/python3.11/pathlib.py", line 1267, in is_file > return S_ISREG(self.stat().st_mode) > ^^^^^^^^^^^ > File "/usr/lib/python3.11/pathlib.py", line 1013, in stat > return os.stat(self, follow_symlinks=follow_symlinks) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > OSError: [Errno 36] File name too long: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' >>>> >>>> import os >>>> os.path.isfile(256 * "x") > False > From rosuav at gmail.com Fri Mar 8 16:30:53 2024 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 9 Mar 2024 08:30:53 +1100 Subject: Variable scope inside and outside functions - global statement being overridden by assignation unless preceded by reference In-Reply-To: <4TrsPL1jK7znV1q@mail.python.org> References: <4Trnb856LRznVF4@mail.python.org> <4TrsPL1jK7znV1q@mail.python.org> Message-ID: On Sat, 9 Mar 2024 at 03:42, Grant Edwards via Python-list wrote: > > On 2024-03-08, Chris Angelico via Python-list wrote: > > On Sat, 9 Mar 2024 at 00:51, Grant Edwards via Python-list > > wrote: > > > >> One might argue that "global" isn't a good choice for what to call the > >> scope in question, since it's not global. It's limited to that source > >> file. It doesn't make sense to me to call a binding "global", when > >> there can be multile different "global" bindings of the same name. > > > > Most "globals" aren't global either, since you can have different > > globals in different running applications. > > To me, "global" has always been limited to within a single > process/address space, but that's probably just bias left over from > C/Pascal/FORTRAN/assembly/etc. It never occurred to me that a global > called "X" in one program on one computer would be the same as a > global called "X" in a different program on a different computer > somewhere else on the "globe". > Yeah. My point is, though, the name "global" is a bit of a hack anyway, so it's not THAT big a deal if it has other caveats too. For example, let's say you always "import globals" at the top of every script, and then assign "globals.x = 123" etc. Now you have a concept of globals that spans the entire application, right? Well, no, not if you use multiprocessing. So, go ahead and call them globals, but people will always have to learn about exactly what that means. ChrisA From sjeik_appie at hotmail.com Fri Mar 8 17:14:16 2024 From: sjeik_appie at hotmail.com (Albert-Jan Roskam) Date: Fri, 08 Mar 2024 23:14:16 +0100 Subject: pathlib.Path.is_file vs os.path.isfile difference In-Reply-To: <33cad416-6f2d-4e1e-9c86-227e30e220fd@tompassin.net> Message-ID: On Mar 8, 2024 19:35, Thomas Passin via Python-list wrote: On 3/8/2024 1:03 PM, Albert-Jan Roskam via Python-list wrote: > Hi, > I was replacing some os.path stuff with Pathlib and I discovered this: > Path(256 * "x").is_file() # OSError > os.path.isfile(256 * "x") # bool > Is this intended? Does pathlib try to resemble os.path as closely as > possible? You must have an very old version of Python. I'm running 3.12.2 and it returns False. Either that or that path name exists and throws some kind of unexpected exception. ==== Hi, I tested this with Python 3.8. Good to know that this was fixed! From list1 at tompassin.net Fri Mar 8 17:57:57 2024 From: list1 at tompassin.net (Thomas Passin) Date: Fri, 8 Mar 2024 17:57:57 -0500 Subject: pathlib.Path.is_file vs os.path.isfile difference In-Reply-To: References: Message-ID: On 3/8/2024 5:14 PM, Albert-Jan Roskam wrote: > > > On Mar 8, 2024 19:35, Thomas Passin via Python-list > wrote: > > On 3/8/2024 1:03 PM, Albert-Jan Roskam via Python-list wrote: > >???? Hi, > >???? I was replacing some os.path stuff with Pathlib and I > discovered this: > >???? Path(256 * "x").is_file()? # OSError > >???? os.path.isfile(256 * "x")? # bool > >???? Is this intended? Does pathlib try to resemble os.path as > closely as > >???? possible? > > You must have an very old version of Python.? I'm running 3.12.2 and it > returns False.? Either that or that path name exists and throws some > kind of unexpected exception. > > > > ==== > > Hi, I tested this with Python 3.8. Good to know that this was fixed! We just learned a few posts back that it might be specific to Linux; I ran it on Windows. From grant.b.edwards at gmail.com Fri Mar 8 18:39:51 2024 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Fri, 8 Mar 2024 18:39:51 -0500 (EST) Subject: pathlib.Path.is_file vs os.path.isfile difference References: =?utf-8?q?=3CDB9PR10MB6689728A4E7F95E99F80B9F283272=40DB9PR10MB6?= =?utf-8?q?689=2EEURPRD10=2EPROD=2EOUTLOOK=2ECOM=3E?= Message-ID: <4Ts2hR1hWnznTq9@mail.python.org> On 2024-03-08, Thomas Passin via Python-list wrote: > >> Hi, I tested this with Python 3.8. Good to know that this was fixed! > > We just learned a few posts back that it might be specific to Linux; I > ran it on Windows. On Linux, the limit is imposed by the filesystem. Most of the "real" filesystems on Linux have a 255 character limit, a few support 256, and some of the legacy filesystems have lower limits. Reiser4 is the only one that's even remotely common which supports more than 256 -- according to https://en.wikipedia.org/wiki/Comparison_of_file_systems#Limits it supports filenames up to 3976 bytes long. NB: The behavior when the limit is exceeded might also vary from one filesystem to another. In any case, the pathlib docs for is_file() are explicit: any errors from the underlying OS and libraries will be propogated. There is nothing to fix. https://docs.python.org/3/library/pathlib.html#pathlib.Path.is_file Path.is_file() Return True if the path points to a regular file (or a symbolic link pointing to a regular file), False if it points to another kind of file. False is also returned if the path doesn?t exist or is a broken symlink; other errors (such as permission errors) are propagated. From barry at barrys-emacs.org Sun Mar 10 06:17:45 2024 From: barry at barrys-emacs.org (Barry) Date: Sun, 10 Mar 2024 10:17:45 +0000 Subject: pathlib.Path.is_file vs os.path.isfile difference In-Reply-To: References: Message-ID: <89D8EE29-3520-416D-A5E6-ED5C2861FE09@barrys-emacs.org> > On 8 Mar 2024, at 23:19, Thomas Passin via Python-list wrote: > > We just learned a few posts back that it might be specific to Linux; I ran it on Windows. Depending on the exact win32 api used there is a 257 limit on windows. The 257 includes 2 for the device, C:, and 255 for the path part that will use 1 for the leading \. Getting an error for a name that is 255 is not surprising. Other api allow for 65535 limit, not sure on its additional limits. Barry From list1 at tompassin.net Sun Mar 10 07:59:44 2024 From: list1 at tompassin.net (Thomas Passin) Date: Sun, 10 Mar 2024 07:59:44 -0400 Subject: pathlib.Path.is_file vs os.path.isfile difference In-Reply-To: <89D8EE29-3520-416D-A5E6-ED5C2861FE09@barrys-emacs.org> References: <89D8EE29-3520-416D-A5E6-ED5C2861FE09@barrys-emacs.org> Message-ID: On 3/10/2024 6:17 AM, Barry wrote: > > >> On 8 Mar 2024, at 23:19, Thomas Passin via Python-list wrote: >> >> We just learned a few posts back that it might be specific to Linux; I ran it on Windows. > > Depending on the exact win32 api used there is a 257 limit on windows. > The 257 includes 2 for the device, C:, and 255 for the path part that will use 1 for the leading \. Getting an error for a name that is 255 is not surprising. > > Other api allow for 65535 limit, not sure on its additional limits. I seem to remember there is a setting to allow longer paths, but I forget any details. From sjeik_appie at hotmail.com Sun Mar 10 09:33:02 2024 From: sjeik_appie at hotmail.com (Albert-Jan Roskam) Date: Sun, 10 Mar 2024 14:33:02 +0100 Subject: pathlib.Path.is_file vs os.path.isfile difference In-Reply-To: Message-ID: On Mar 10, 2024 12:59, Thomas Passin via Python-list wrote: On 3/10/2024 6:17 AM, Barry wrote: > > >> On 8 Mar 2024, at 23:19, Thomas Passin via Python-list wrote: >> >> We just learned a few posts back that it might be specific to Linux; I ran it on Windows. > > Depending on the exact win32 api used there is a 257 limit on windows. > The 257 includes 2 for the device, C:, and 255 for the path part that will use 1 for the leading \. Getting an error for a name that is 255 is not surprising. > > Other api allow for 65535 limit, not sure on its additional limits. I seem to remember there is a setting to allow longer paths, but I forget any details. ===== You mean the "\\?\" prefix? https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry From list1 at tompassin.net Sun Mar 10 10:09:53 2024 From: list1 at tompassin.net (Thomas Passin) Date: Sun, 10 Mar 2024 10:09:53 -0400 Subject: pathlib.Path.is_file vs os.path.isfile difference In-Reply-To: References: Message-ID: On 3/10/2024 9:33 AM, Albert-Jan Roskam wrote: > > > On Mar 10, 2024 12:59, Thomas Passin via Python-list > wrote: > > On 3/10/2024 6:17 AM, Barry wrote: > > > > > >> On 8 Mar 2024, at 23:19, Thomas Passin via Python-list > wrote: > >> > >> We just learned a few posts back that it might be specific to > Linux; I ran it on Windows. > > > > Depending on the exact win32 api used there is a 257 limit on > windows. > > The 257 includes 2 for the device, C:, and 255 for the path part > that will use 1 for the leading \. Getting an error for a name that > is 255 is not surprising. > > > > Other api allow for 65535 limit, not sure on its additional limits. > > I seem to remember there is a setting to allow longer paths, but I > forget any details. > > > > ===== > > You mean the "\\?\" prefix? > > https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry That and there's a registry setting: https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation From barry at barrys-emacs.org Sun Mar 10 18:09:05 2024 From: barry at barrys-emacs.org (Barry) Date: Sun, 10 Mar 2024 22:09:05 +0000 Subject: pathlib.Path.is_file vs os.path.isfile difference In-Reply-To: References: Message-ID: <6B884EE3-A25F-44EB-90B0-CE7BBAA7350C@barrys-emacs.org> > On 10 Mar 2024, at 14:49, Thomas Passin via Python-list wrote: > > That and there's a registry setting: > > https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation > Yep that and rules about size of parts of the path. Barry From sanskarjoshiambaji at gmail.com Sun Mar 10 14:08:27 2024 From: sanskarjoshiambaji at gmail.com (Sanskar Mukeshbhai Joshi) Date: Sun, 10 Mar 2024 18:08:27 +0000 Subject: Error in Module Message-ID: Respected Sir/Ma'am I had made my project in BCA in Python. When I had complete my project and run the program, at that time I got the error in runnig my project. The error was ModuleNotFoundError: No module named 'flask'. I request you to check this problem and resolve it or guide me to solve this Error. From learn2program at gmail.com Mon Mar 11 12:28:40 2024 From: learn2program at gmail.com (Alan Gauld) Date: Mon, 11 Mar 2024 16:28:40 +0000 Subject: Error in Module In-Reply-To: References: Message-ID: On 10/03/2024 18:08, Sanskar Mukeshbhai Joshi via Python-list wrote: > I had made my project in BCA in Python. When I had complete my > project and run the program, at that time I got the error in > runnig my project. The error was ModuleNotFoundError: No module named 'flask'. Flask is a third party package that you need to install separately from Python. It does not come as standard. Have you installed Flask on the computer where you are running your project? If so, how did you download/install it? -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From dieter.maurer at online.de Mon Mar 11 13:06:35 2024 From: dieter.maurer at online.de (dieter.maurer at online.de) Date: Mon, 11 Mar 2024 18:06:35 +0100 Subject: Error in Module In-Reply-To: References: Message-ID: <26095.14875.4411.566276@ixdm.fritz.box> Sanskar Mukeshbhai Joshi wrote at 2024-3-10 18:08 +0000: >I had made my project in BCA in Python. When I had complete my project and run the program, at that time I got the error in runnig my project. The error was ModuleNotFoundError: No module named 'flask'. `flask` is not part of the Python library; it has to be installed separately. Apparently, you project is a `flask` project (a Web application platform). In this case, you must run your program in a special `flask` environment. I suggest you contact your project mentor to get help. From sanskarjoshiambaji at gmail.com Mon Mar 11 14:04:10 2024 From: sanskarjoshiambaji at gmail.com (Sanskar Mukeshbhai Joshi) Date: Mon, 11 Mar 2024 23:34:10 +0530 Subject: Error in Module In-Reply-To: <26095.14875.4411.566276@ixdm.fritz.box> References: <26095.14875.4411.566276@ixdm.fritz.box> Message-ID: Thank you for the information. On Mon, Mar 11, 2024, 22:36 wrote: > Sanskar Mukeshbhai Joshi wrote at 2024-3-10 18:08 +0000: > >I had made my project in BCA in Python. When I had complete my project > and run the program, at that time I got the error in runnig my project. The > error was ModuleNotFoundError: No module named 'flask'. > > `flask` is not part of the Python library; it has to be installed > separately. > > Apparently, you project is a `flask` project (a Web application platform). > In this case, you must run your program in a special `flask` > environment. > > I suggest you contact your project mentor to get help. > From rambiusparkisanius at gmail.com Mon Mar 11 16:53:00 2024 From: rambiusparkisanius at gmail.com (Ivan "Rambius" Ivanov) Date: Mon, 11 Mar 2024 16:53:00 -0400 Subject: A Single Instance of an Object? Message-ID: Hello, I am refactoring some code and I would like to get rid of a global variable. Here is the outline: import subprocess CACHE = {} def lookup(key): """"Runs the command cmd, parses its output, extract's the key's value, caches it and returns it. If the key has already been in the cache, returns its cached value. If the command cmd returns an error, the value is set to None and cached as None.""" if key in CACHE: return CACHE[key] value = None cmd = f"mycmd {key}" proc = subprocess(cmd, capture_output=True, text=True, check=False) if proc.returncode == 0: value = proc.stdout.strip() else: logger.error("cmd returned error") CACHE[key] = value return value ... def main(): while True: keys = load_keys() for key in keys: call_function_that_call_function_that_calls_lookup(key) The global cache variable made unit testing of the lookup(key) method clumsy, because I have to clean it after each unit test. I refactored it as: class Lookup: def __init__(self): self.cache = {} def lookup(key): if key in self.cache: return self.cache[key] value = None cmd = f"mycmd {key}" proc = subprocess(cmd, capture_output=True, text=True, check=False) if proc.returncode == 0: value = proc.stdout.strip() else: logger.error("cmd returned error") self.cache[key] = value return value Now it is easier to unit test, and the cache is not global. However, I cannot instantiate Lookup inside the while- or for- loops in main(), because the cache should be only one. I need to ensure there is only one instance of Lookup - this is why I made it a global variable, so that it is accessible to all functions in that script and the one that actually needs it is 4 levels down in the call stack. I have never done that in Python because I deliberately avoided such complicated situations up to now. I know about the Singleton pattern, but I have never implemented it in Python and I don't know if it is Pythonish. I am looking for the same behaviour as logging.getLogger(name). logging.getLogger("myname") will always return the same object no matter where it is called as long as the name argument is the same. How would you advise me to implement that? Regards rambius -- Tangra Mega Rock: http://www.radiotangra.com From rosuav at gmail.com Mon Mar 11 16:57:51 2024 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 12 Mar 2024 07:57:51 +1100 Subject: A Single Instance of an Object? In-Reply-To: References: Message-ID: On Tue, 12 Mar 2024 at 07:54, Ivan "Rambius" Ivanov via Python-list wrote: > I am refactoring some code and I would like to get rid of a global > variable. Here is the outline: > > ... > > I have never done that in Python because I deliberately avoided such > complicated situations up to now. I know about the Singleton pattern, > but I have never implemented it in Python and I don't know if it is > Pythonish. > A Singleton is just a global variable. Why do this? Did someone tell you "global variables are bad, don't use them"? ChrisA From PythonList at DancesWithMice.info Mon Mar 11 17:03:01 2024 From: PythonList at DancesWithMice.info (dn) Date: Tue, 12 Mar 2024 10:03:01 +1300 Subject: A Single Instance of an Object? In-Reply-To: References: Message-ID: <1a49ecb0-2a5f-4758-92d1-a70d3d3dc651@DancesWithMice.info> Good question Rambius! On 12/03/24 09:53, Ivan "Rambius" Ivanov via Python-list wrote: > Hello, > > I am refactoring some code and I would like to get rid of a global > variable. Here is the outline: > > import subprocess > > CACHE = {} First thought: don't reinvent-the-wheel, use lru_cache (https://docs.python.org/3/library/functools.html) > The global cache variable made unit testing of the lookup(key) method > clumsy, because I have to clean it after each unit test. I refactored > it as: > > class Lookup: > def __init__(self): > self.cache = {} > Change "cache" to be a class-attribute (it is currently an instance. Then, code AFTER the definition of Lookup can refer to Lookup.cache, regardless of instantiation, and code within Lookup can refer to self.cache as-is... -- Regards, =dn From rambiusparkisanius at gmail.com Mon Mar 11 17:04:32 2024 From: rambiusparkisanius at gmail.com (Ivan "Rambius" Ivanov) Date: Mon, 11 Mar 2024 17:04:32 -0400 Subject: A Single Instance of an Object? In-Reply-To: References: Message-ID: On Mon, Mar 11, 2024 at 5:01?PM Chris Angelico via Python-list wrote: > > On Tue, 12 Mar 2024 at 07:54, Ivan "Rambius" Ivanov via Python-list > wrote: > > I am refactoring some code and I would like to get rid of a global > > variable. Here is the outline: > > > > ... > > > > I have never done that in Python because I deliberately avoided such > > complicated situations up to now. I know about the Singleton pattern, > > but I have never implemented it in Python and I don't know if it is > > Pythonish. > > > > A Singleton is just a global variable. Why do this? Did someone tell > you "global variables are bad, don't use them"? I have bad experience with global variables because it is hard to track what and when modifies them. I don't consider them bad, but if I can I avoid them. Regards rambius -- Tangra Mega Rock: http://www.radiotangra.com From rambiusparkisanius at gmail.com Mon Mar 11 17:37:34 2024 From: rambiusparkisanius at gmail.com (Ivan "Rambius" Ivanov) Date: Mon, 11 Mar 2024 17:37:34 -0400 Subject: A Single Instance of an Object? In-Reply-To: <1a49ecb0-2a5f-4758-92d1-a70d3d3dc651@DancesWithMice.info> References: <1a49ecb0-2a5f-4758-92d1-a70d3d3dc651@DancesWithMice.info> Message-ID: On Mon, Mar 11, 2024 at 5:06?PM dn via Python-list wrote: > > Good question Rambius! > > On 12/03/24 09:53, Ivan "Rambius" Ivanov via Python-list wrote: > > Hello, > > > > I am refactoring some code and I would like to get rid of a global > > variable. Here is the outline: > > > > import subprocess > > > > CACHE = {} > > First thought: don't reinvent-the-wheel, use lru_cache > (https://docs.python.org/3/library/functools.html) > > > > The global cache variable made unit testing of the lookup(key) method > > clumsy, because I have to clean it after each unit test. I refactored > > it as: > > > > class Lookup: > > def __init__(self): > > self.cache = {} > > > > Change "cache" to be a class-attribute (it is currently an instance. > > Then, code AFTER the definition of Lookup can refer to Lookup.cache, > regardless of instantiation, and code within Lookup can refer to > self.cache as-is... > Thank you for your suggestions. I will research them! Regards rambius -- Tangra Mega Rock: http://www.radiotangra.com From rosuav at gmail.com Mon Mar 11 17:45:10 2024 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 12 Mar 2024 08:45:10 +1100 Subject: A Single Instance of an Object? In-Reply-To: References: Message-ID: On Tue, 12 Mar 2024 at 08:04, Ivan "Rambius" Ivanov wrote: > > A Singleton is just a global variable. Why do this? Did someone tell > > you "global variables are bad, don't use them"? > > I have bad experience with global variables because it is hard to > track what and when modifies them. I don't consider them bad, but if I > can I avoid them. > If you have a singleton, how will you track "what and when modifies" it? How is it any different from a global? ChrisA From hjp-python at hjp.at Mon Mar 11 19:37:14 2024 From: hjp-python at hjp.at (Peter J. Holzer) Date: Tue, 12 Mar 2024 00:37:14 +0100 Subject: A Single Instance of an Object? In-Reply-To: References: Message-ID: <20240311233714.zvnbmgzerr5cpxjs@hjp.at> On 2024-03-11 16:53:00 -0400, Ivan "Rambius" Ivanov via Python-list wrote: > I am refactoring some code and I would like to get rid of a global > variable. Here is the outline: ... > The global cache variable made unit testing of the lookup(key) method > clumsy, because I have to clean it after each unit test. I refactored > it as: > > class Lookup: > def __init__(self): > self.cache = {} > > def lookup(key): > if key in self.cache: > return self.cache[key] > > value = None > > cmd = f"mycmd {key}" > proc = subprocess(cmd, capture_output=True, text=True, check=False) > if proc.returncode == 0: > value = proc.stdout.strip() > else: > logger.error("cmd returned error") > > self.cache[key] = value > return value > > Now it is easier to unit test, and the cache is not global. However, I > cannot instantiate Lookup inside the while- or for- loops in main(), > because the cache should be only one. I need to ensure there is only > one instance of Lookup - this is why I made it a global variable, so > that it is accessible to all functions in that script and the one that > actually needs it is 4 levels down in the call stack. [...] > I am looking for the same behaviour as logging.getLogger(name). > logging.getLogger("myname") will always return the same object no > matter where it is called as long as the name argument is the same. > > How would you advise me to implement that? Just add a dict of Lookup objects to your module: lookups = {} def get_lookup(name): if name not in lookups: lookups[name] = Lookup() return lookups[name] Then (assuming your module is also called "lookup", in all other modules do import lookup lo = lookup.get_lookup("whatever") ... v = lo.lookup("a key") In your test cases where you need many different lookup tables use lo = lookup.get_lookup("test1") ... lo = lookup.get_lookup("test2") ... lo = lookup.get_lookup("test3") hp PS: You don't have to put that in a separate module but I think it's a lot cleaner that way. -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From thomas at python.org Tue Mar 12 20:03:24 2024 From: thomas at python.org (Thomas Wouters) Date: Wed, 13 Mar 2024 01:03:24 +0100 Subject: [RELEASE] Python 3.13.0a5 is now available Message-ID: We?re getting closer and closer? Alpha 5 is here. https://www.python.org/downloads/release/python-3130a5/ *This is an early developer preview of Python 3.13* Major new features of the 3.13 series, compared to 3.12 Python 3.13 is still in development. This release, 3.13.0a5, is the fifth of six planned alpha releases. Alpha releases are intended to make it easier to test the current state of new features and bug fixes and to test the release process. During the alpha phase, features may be added up until the start of the beta phase (2024-05-07) and, if necessary, may be modified or deleted up until the release candidate phase (2024-07-30). Please keep in mind that this is a preview release and its use is *not* recommended for production environments. Many new features for Python 3.13 are still being planned and written. Work continues apace on both the work to remove the Global Interpeter Lock , and to improve Python performance. The most notable changes so far: - In the interactive interpreter, exception tracebacks are now colorized by default . - A preliminary, *experimental* JIT was added , providing the ground work for significant performance improvements. - Docstrings now have their leading indentation stripped , reducing memory use and the size of .pyc files. (Most tools handling docstrings already strip leading indentation.) - The dbm module has a new dbm.sqlite3 backend that is used by default when creating new files. - PEP 594 (Removing dead batteries from the standard library) scheduled removals of many deprecated modules: aifc, audioop, chunk, cgi, cgitb, crypt, imghdr, mailcap, msilib, nis, nntplib, ossaudiodev, pipes, sndhdr, spwd, sunau, telnetlib, uu, xdrlib, lib2to3. - Many other removals of deprecated classes, functions and methods in various standard library modules. - New deprecations , most of which are scheduled for removal from Python 3.15 or 3.16. - C API removals and deprecations . (Some removals present in alpha 1 have been reverted in alpha 2, as the removals were deemed too disruptive at this time.) (Hey, *fellow core developer,* if a feature you find important is missing from this list, let Thomas know .) The next pre-release of Python 3.13 will be 3.13.0a6, currently scheduled for 2024-04-09. More resources - Online Documentation - PEP 719 , 3.13 Release Schedule - Report bugs at Issues ? python/cpython ? GitHub . - Help fund Python directly (or via GitHub Sponsors ), and support the Python community . Enjoy the new releases Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. Regards from wet and chilly Amsterdam, Your release team, Thomas Wouters Ned Deily Steve Dower ?ukasz Langa -- Thomas Wouters From loris.bennett at fu-berlin.de Fri Mar 15 05:30:03 2024 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Fri, 15 Mar 2024 10:30:03 +0100 Subject: Configuring an object via a dictionary Message-ID: <87y1ajdeqs.fsf@zedat.fu-berlin.de> Hi, I am initialising an object via the following: def __init__(self, config): self.connection = None self.source_name = config['source_name'] self.server_host = config['server_host'] self.server_port = config['server_port'] self.user_base = config['user_base'] self.user_identifier = config['user_identifier'] self.group_base = config['group_base'] self.group_identifier = config['group_identifier'] self.owner_base = config['owner_base'] However, some entries in the configuration might be missing. What is the best way of dealing with this? I could of course simply test each element of the dictionary before trying to use. I could also just write self.config = config but then addressing the elements will add more clutter to the code. However, with a view to asking forgiveness rather than permission, is there some simple way just to assign the dictionary elements which do in fact exist to self-variables? Or should I be doing this completely differently? Cheers, Loris -- This signature is currently under constuction. From mats at wichmann.us Fri Mar 15 14:35:26 2024 From: mats at wichmann.us (Mats Wichmann) Date: Fri, 15 Mar 2024 12:35:26 -0600 Subject: Configuring an object via a dictionary In-Reply-To: <87y1ajdeqs.fsf@zedat.fu-berlin.de> References: <87y1ajdeqs.fsf@zedat.fu-berlin.de> Message-ID: On 3/15/24 03:30, Loris Bennett via Python-list wrote: > Hi, > > I am initialising an object via the following: > self.source_name = config['source_name'] config.get('source_name', default_if_not_defined) is a common technique... From list1 at tompassin.net Fri Mar 15 14:45:21 2024 From: list1 at tompassin.net (Thomas Passin) Date: Fri, 15 Mar 2024 14:45:21 -0400 Subject: Configuring an object via a dictionary In-Reply-To: <87y1ajdeqs.fsf@zedat.fu-berlin.de> References: <87y1ajdeqs.fsf@zedat.fu-berlin.de> Message-ID: <3cbc94a7-bf8e-4ee7-ae70-ba8b73ec61bd@tompassin.net> On 3/15/2024 5:30 AM, Loris Bennett via Python-list wrote: > Hi, > > I am initialising an object via the following: > > def __init__(self, config): > > self.connection = None > > self.source_name = config['source_name'] > self.server_host = config['server_host'] > self.server_port = config['server_port'] > self.user_base = config['user_base'] > self.user_identifier = config['user_identifier'] > self.group_base = config['group_base'] > self.group_identifier = config['group_identifier'] > self.owner_base = config['owner_base'] > > However, some entries in the configuration might be missing. What is > the best way of dealing with this? > > I could of course simply test each element of the dictionary before > trying to use. I could also just write > > self.config = config > > but then addressing the elements will add more clutter to the code. > > However, with a view to asking forgiveness rather than > permission, is there some simple way just to assign the dictionary > elements which do in fact exist to self-variables? > > Or should I be doing this completely differently? self.source_name = config.get('source_name', default_value) Or, if you like this kind of expression better, self.source_name = config.get('source_name') or default_value .get() will return None if the key doesn't exist, or the default value if you specify one. From grant.b.edwards at gmail.com Fri Mar 15 15:09:43 2024 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Fri, 15 Mar 2024 15:09:43 -0400 (EDT) Subject: Configuring an object via a dictionary References: <87y1ajdeqs.fsf@zedat.fu-berlin.de> <3cbc94a7-bf8e-4ee7-ae70-ba8b73ec61bd@tompassin.net> Message-ID: <4TxDMW3Dn1znVFX@mail.python.org> On 2024-03-15, Thomas Passin via Python-list wrote: > On 3/15/2024 5:30 AM, Loris Bennett via Python-list wrote: >> Hi, >> >> I am initialising an object via the following: >> >> def __init__(self, config): >> >> self.connection = None >> >> self.source_name = config['source_name'] >> self.server_host = config['server_host'] >> self.server_port = config['server_port'] >> self.user_base = config['user_base'] >> self.user_identifier = config['user_identifier'] >> self.group_base = config['group_base'] >> self.group_identifier = config['group_identifier'] >> self.owner_base = config['owner_base'] >> >> However, some entries in the configuration might be missing. What is >> the best way of dealing with this? >> >> I could of course simply test each element of the dictionary before >> trying to use. I could also just write >> >> self.config = config >> >> but then addressing the elements will add more clutter to the code. >> >> However, with a view to asking forgiveness rather than >> permission, is there some simple way just to assign the dictionary >> elements which do in fact exist to self-variables? >> >> Or should I be doing this completely differently? > > self.source_name = config.get('source_name', default_value) > > Or, if you like this kind of expression better, > > self.source_name = config.get('source_name') or default_value Won't the latter version misbehave if the value of config['source_name'] has a "false" boolean value (e.g. "", 0, 0.0, None, [], (), {}, ...) >>> config = {} >>> config['source_name'] = "" >>> config.get('source_name') or 'default' 'default' From list1 at tompassin.net Fri Mar 15 15:48:17 2024 From: list1 at tompassin.net (Thomas Passin) Date: Fri, 15 Mar 2024 15:48:17 -0400 Subject: Configuring an object via a dictionary In-Reply-To: <4TxDMW3Dn1znVFX@mail.python.org> References: <87y1ajdeqs.fsf@zedat.fu-berlin.de> <3cbc94a7-bf8e-4ee7-ae70-ba8b73ec61bd@tompassin.net> <4TxDMW3Dn1znVFX@mail.python.org> Message-ID: On 3/15/2024 3:09 PM, Grant Edwards via Python-list wrote: > On 2024-03-15, Thomas Passin via Python-list wrote: >> On 3/15/2024 5:30 AM, Loris Bennett via Python-list wrote: >>> Hi, >>> >>> I am initialising an object via the following: >>> >>> def __init__(self, config): >>> >>> self.connection = None >>> >>> self.source_name = config['source_name'] >>> self.server_host = config['server_host'] >>> self.server_port = config['server_port'] >>> self.user_base = config['user_base'] >>> self.user_identifier = config['user_identifier'] >>> self.group_base = config['group_base'] >>> self.group_identifier = config['group_identifier'] >>> self.owner_base = config['owner_base'] >>> >>> However, some entries in the configuration might be missing. What is >>> the best way of dealing with this? >>> >>> I could of course simply test each element of the dictionary before >>> trying to use. I could also just write >>> >>> self.config = config >>> >>> but then addressing the elements will add more clutter to the code. >>> >>> However, with a view to asking forgiveness rather than >>> permission, is there some simple way just to assign the dictionary >>> elements which do in fact exist to self-variables? >>> >>> Or should I be doing this completely differently? >> >> self.source_name = config.get('source_name', default_value) >> >> Or, if you like this kind of expression better, >> >> self.source_name = config.get('source_name') or default_value > > Won't the latter version misbehave if the value of config['source_name'] has a > "false" boolean value (e.g. "", 0, 0.0, None, [], (), {}, ...) > >>>> config = {} >>>> config['source_name'] = "" >>>> config.get('source_name') or 'default' > 'default' Oh, well, picky, picky! I've always like writing using the "or" form and have never gotten bit - especially for configuration-type values where you really do expect a non-falsey value, it's probably low risk - but of course, you're right. In newer code I have been putting a default into get(). And I suppose there is always the possibility that sometime in the future an "or" clause like that will be changed to return a Boolean, which one would expect anyway. From 2QdxY4RzWzUUiLuE at potatochowder.com Fri Mar 15 17:33:09 2024 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Fri, 15 Mar 2024 17:33:09 -0400 Subject: Configuring an object via a dictionary In-Reply-To: References: <87y1ajdeqs.fsf@zedat.fu-berlin.de> <3cbc94a7-bf8e-4ee7-ae70-ba8b73ec61bd@tompassin.net> <4TxDMW3Dn1znVFX@mail.python.org> Message-ID: On 2024-03-15 at 15:48:17 -0400, Thomas Passin via Python-list wrote: > [...] And I suppose there is always the possibility that sometime in > the future an "or" clause like that will be changed to return a > Boolean, which one would expect anyway. Not only is the current value is way more useful, but changing it would be a compatibility and maintenance nightmare. If I want Java, I know where to find it. :-) From PythonList at DancesWithMice.info Fri Mar 15 18:23:38 2024 From: PythonList at DancesWithMice.info (dn) Date: Sat, 16 Mar 2024 11:23:38 +1300 Subject: Configuring an object via a dictionary In-Reply-To: <87y1ajdeqs.fsf@zedat.fu-berlin.de> References: <87y1ajdeqs.fsf@zedat.fu-berlin.de> Message-ID: <2e09744d-7f46-4c47-a439-b8a2f69d750e@DancesWithMice.info> On 15/03/24 22:30, Loris Bennett via Python-list wrote: > Hi, > > I am initialising an object via the following: > > def __init__(self, config): > > self.connection = None > > self.source_name = config['source_name'] > self.server_host = config['server_host'] > self.server_port = config['server_port'] > self.user_base = config['user_base'] > self.user_identifier = config['user_identifier'] > self.group_base = config['group_base'] > self.group_identifier = config['group_identifier'] > self.owner_base = config['owner_base'] > > However, some entries in the configuration might be missing. What is > the best way of dealing with this? How do you define "missing"? Thus, @Thomas' suggestion may/not apply. It is neat and easy. I usually plump for: self.source_name = config[ "source_name" ] or default_value but @Grant's warning applies! (which is why the pythonic way is to use (and test for) None as a definition of "missing" (see also impacts of using default values and mutable data-structures) LBYL cf EAFP: When setting-up an environment like this, elements are often set to a default-value, and then user-settings, command-line arguments, and the like, applied in a priority-order sequence, amend as-appropriate. In this way, such problems will never arise. This is the better course 90% of the time. Which raises the next question: What is the impact if some attribute is "missing"? ie if the 'whatever' must be identified by source_name (for example), then there is little point in assigning any values to the new class. Considerations which apply *after* this question, the __init__(), are at least equally-important considerations (see below)! > I could of course simply test each element of the dictionary before > trying to use. I could also just write > > self.config = config > > but then addressing the elements will add more clutter to the code. By which you mean that such "clutter" should only appear in the __init__() - which is not such a bad idea (but see below). OTOH it is often helpful to one's comprehension to be given a prompt as to the source of the data. Thus, in later processing *config[ "source_name" ] may add a small amount of extra information to the reader, over self.source_name. * maybe prepended "self.", or not... Am assuming that passing all eight elements as individual arguments is off-the-table, embodying far too many 'negatives' (see below). > However, with a view to asking forgiveness rather than > permission, is there some simple way just to assign the dictionary > elements which do in fact exist to self-variables? Assuming config is a dict: self.__dict__.update( config ) will work, but attracts similar criticism - if not "clutter" then an unnecessary view (and understanding) of the workings of classes under-the-hood. Another question: When these values are used, are they all used at the same time, and never again? It may only be worth 'breaking-out' and making attributes from those which are used in multiple situations within the class's methods. If, the other extreme, they are only (effectively) passed from the __init__() to some sort of open() method, then pass the data-structure as an whole and delegate/remove the "clutter" to there. In that scenario, such detail would *only* has meaning *and* purpose in the open() method and thus no point in cluttering-up the __init__() with detail that is only required elsewhere! > Or should I be doing this completely differently? YMMV, but I prefer the idea of transferring the environment/config as a whole (as above). If it were a class (cf the supposed dict) then "clutter" is reduced by eschewing brackets and quotation-marks, eg "config.source_name". If those eight-elements are not the entirety of that data-structure, then consider creating an interface-class, which is extracted (or built that way) from some wider 'environment', and used to set-up access to data-source(s) or whatever. Again, this aids understanding in knowing where data has been created/gathered, and improves confidence when utilising it down-the-line. The other principle in only providing the required (eight) items of data, is that the 'receiving-class' needs no understanding of the structure or workings of the 'sending-class' = separation of concerns, and each class has single purpose/reason to change. A variation on that might be to use a method/function as the interface: access = Access( config.access_data ) Thus, the config class (instance) will need an access_data method to collate the data-items. The complimentary code in this Access.__init__( self, config, ) might be something like: ( self.source_name, self.server_host, self.server_port, self.user_base, self.user_identifier, self.group_base, self.group_identifier, self.owner_base = config_access() ) If you know my style/preferences, notice that I'm breaking my own 'rule' of using named-parameters in preference to positional-parameters when there are three or more. However, this *may* be one of those exceptions (cf hobgoblins). That said, this is the third and least-preferred idea! -- Regards, =dn From list1 at tompassin.net Fri Mar 15 19:08:23 2024 From: list1 at tompassin.net (Thomas Passin) Date: Fri, 15 Mar 2024 19:08:23 -0400 Subject: Configuring an object via a dictionary In-Reply-To: References: <87y1ajdeqs.fsf@zedat.fu-berlin.de> <3cbc94a7-bf8e-4ee7-ae70-ba8b73ec61bd@tompassin.net> <4TxDMW3Dn1znVFX@mail.python.org> Message-ID: <092b12f3-b131-44d5-af19-1e3346d0f7df@tompassin.net> On 3/15/2024 5:33 PM, Dan Sommers via Python-list wrote: > On 2024-03-15 at 15:48:17 -0400, > Thomas Passin via Python-list wrote: > >> [...] And I suppose there is always the possibility that sometime in >> the future an "or" clause like that will be changed to return a >> Boolean, which one would expect anyway. > > Not only is the current value is way more useful, but changing it would > be a compatibility and maintenance nightmare. I'm with you here! > If I want Java, I know where to find it. :-) From avi.e.gross at gmail.com Fri Mar 15 21:06:31 2024 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Fri, 15 Mar 2024 21:06:31 -0400 Subject: Configuring an object via a dictionary In-Reply-To: References: <87y1ajdeqs.fsf@zedat.fu-berlin.de> <3cbc94a7-bf8e-4ee7-ae70-ba8b73ec61bd@tompassin.net> <4TxDMW3Dn1znVFX@mail.python.org> Message-ID: <003301da773e$2f54b850$8dfe28f0$@gmail.com> A part of the Python view of the world is about a concept of whether something is "truthy" or not and thus many corners of the language do not care what kind of object an expression returns. If the object is returned in a context looking for not a Boolean value but a truth value, it is evaluated and in other scenarios, left alone to propagate in the code. Changing such behavior would be a very serious undertaking, and frankly, silly. But if anyone really wants an actual Boolean, then the non-not operator should do the trick as !(whatever) takes what follows as a truthy value and negates it and a second ! brings it back as a True/False as in !!(whatever) And for many data types, perhaps all, you can use the bool() function that I believe follows the same rules about being truthy. Both of the above should be fairly easy to use in any rare contexts that demand a more standard Boolean result as in some other languages. It is one of many strengths of python that supports varieties of polymorphism. And it allows a value to be passed or returned that can both be viewed as some kind of object of many kinds and seen as a Boolean for considerations like flow of control. -----Original Message----- From: Python-list On Behalf Of Dan Sommers via Python-list Sent: Friday, March 15, 2024 5:33 PM To: python-list at python.org Subject: Re: Configuring an object via a dictionary On 2024-03-15 at 15:48:17 -0400, Thomas Passin via Python-list wrote: > [...] And I suppose there is always the possibility that sometime in > the future an "or" clause like that will be changed to return a > Boolean, which one would expect anyway. Not only is the current value is way more useful, but changing it would be a compatibility and maintenance nightmare. If I want Java, I know where to find it. :-) -- https://mail.python.org/mailman/listinfo/python-list From barry at barrys-emacs.org Sat Mar 16 04:15:19 2024 From: barry at barrys-emacs.org (Barry) Date: Sat, 16 Mar 2024 08:15:19 +0000 Subject: Configuring an object via a dictionary In-Reply-To: References: Message-ID: <60E92E67-1412-4DE6-B330-495343107474@barrys-emacs.org> > On 15 Mar 2024, at 19:51, Thomas Passin via Python-list wrote: > > I've always like writing using the "or" form and have never gotten bit I, on the other hand, had to fix a production problem that using ?or? introducted. I avoid this idiom because it fails on falsy values. Barry From roel at roelschroeven.net Sat Mar 16 08:12:58 2024 From: roel at roelschroeven.net (Roel Schroeven) Date: Sat, 16 Mar 2024 13:12:58 +0100 Subject: Configuring an object via a dictionary In-Reply-To: <60E92E67-1412-4DE6-B330-495343107474@barrys-emacs.org> References: <60E92E67-1412-4DE6-B330-495343107474@barrys-emacs.org> Message-ID: <146a44f2-eead-49dc-a57c-f2abe9d1516d@roelschroeven.net> Barry via Python-list schreef op 16/03/2024 om 9:15: > > > On 15 Mar 2024, at 19:51, Thomas Passin via Python-list wrote: > > > > I've always like writing using the "or" form and have never gotten bit > > I, on the other hand, had to fix a production problem that using ?or? introducted. > I avoid this idiom because it fails on falsy values. > Me too. It's just too fragile. When writing code you're going to need an alternative for cases where "config.get('source_name') or default_value" doesn't work correctly; much better to use that alternative for all cases. -- "This planet has - or rather had - a problem, which was this: most of the people living on it were unhappy for pretty much of the time. Many solutions were suggested for this problem, but most of these were largely concerned with the movement of small green pieces of paper, which was odd because on the whole it wasn't the small green pieces of paper that were unhappy." -- Douglas Adams From list1 at tompassin.net Sat Mar 16 09:02:56 2024 From: list1 at tompassin.net (Thomas Passin) Date: Sat, 16 Mar 2024 09:02:56 -0400 Subject: Configuring an object via a dictionary In-Reply-To: <146a44f2-eead-49dc-a57c-f2abe9d1516d@roelschroeven.net> References: <60E92E67-1412-4DE6-B330-495343107474@barrys-emacs.org> <146a44f2-eead-49dc-a57c-f2abe9d1516d@roelschroeven.net> Message-ID: <8f78784c-a108-4eef-b8cc-075acadcd078@tompassin.net> On 3/16/2024 8:12 AM, Roel Schroeven via Python-list wrote: > Barry via Python-list schreef op 16/03/2024 om 9:15: >> >> > On 15 Mar 2024, at 19:51, Thomas Passin via Python-list >> ? wrote: >> > > I've always like writing using the "or" form and have never gotten >> bit >> >> I, on the other hand, had to fix a production problem that using ?or? >> introducted. >> I avoid this idiom because it fails on falsy values. >> > Me too. It's just too fragile. When writing code you're going to need an > alternative for cases where "config.get('source_name') or default_value" > doesn't work correctly; much better to use that alternative for all cases. Trying to remember when I've used it, that was probably on personal code where I had a good idea what the values could be. Otherwise, I'm in agreement. From PythonList at DancesWithMice.info Sat Mar 16 17:19:10 2024 From: PythonList at DancesWithMice.info (dn) Date: Sun, 17 Mar 2024 10:19:10 +1300 Subject: Configuring an object via a dictionary In-Reply-To: <60E92E67-1412-4DE6-B330-495343107474@barrys-emacs.org> References: <60E92E67-1412-4DE6-B330-495343107474@barrys-emacs.org> Message-ID: On 16/03/24 21:15, Barry via Python-list wrote: > > >> On 15 Mar 2024, at 19:51, Thomas Passin via Python-list wrote: >> >> I've always like writing using the "or" form and have never gotten bit > > I, on the other hand, had to fix a production problem that using ?or? introducted. > I avoid this idiom because it fails on falsy values. As with any other facility, one has to understand ALL implications! It must be one of those intensely-frustrating errors to track-down, which is then oh-so-simple to fix! Are you able to list (real, if suitably anonymised) examples of where the truthy/falsy was inappropriate, please? -- Regards, =dn From hjp-python at hjp.at Sat Mar 16 19:06:06 2024 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 17 Mar 2024 00:06:06 +0100 Subject: Configuring an object via a dictionary In-Reply-To: <60E92E67-1412-4DE6-B330-495343107474@barrys-emacs.org> References: <60E92E67-1412-4DE6-B330-495343107474@barrys-emacs.org> Message-ID: <20240316230606.qjn5zepek645tsqq@hjp.at> On 2024-03-16 08:15:19 +0000, Barry via Python-list wrote: > > On 15 Mar 2024, at 19:51, Thomas Passin via Python-list wrote: > > I've always like writing using the "or" form and have never gotten bit > > I, on the other hand, had to fix a production problem that using ?or? introducted. > I avoid this idiom because it fails on falsy values. Perl has a // operator (pronounced "err"), which works like || (or), except that it tests whether the left side is defined (not None in Python terms) instead of truthy. This still isn't bulletproof but I've found it very handy. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From PythonList at DancesWithMice.info Sun Mar 17 00:15:32 2024 From: PythonList at DancesWithMice.info (dn) Date: Sun, 17 Mar 2024 17:15:32 +1300 Subject: Configuring an object via a dictionary In-Reply-To: <20240316230606.qjn5zepek645tsqq@hjp.at> References: <60E92E67-1412-4DE6-B330-495343107474@barrys-emacs.org> <20240316230606.qjn5zepek645tsqq@hjp.at> Message-ID: <98171cf7-c2ea-4496-8b68-fa9a8f247b3b@DancesWithMice.info> On 17/03/24 12:06, Peter J. Holzer via Python-list wrote: > On 2024-03-16 08:15:19 +0000, Barry via Python-list wrote: >>> On 15 Mar 2024, at 19:51, Thomas Passin via Python-list wrote: >>> I've always like writing using the "or" form and have never gotten bit >> >> I, on the other hand, had to fix a production problem that using ?or? introducted. >> I avoid this idiom because it fails on falsy values. > > Perl has a // operator (pronounced "err"), which works like || (or), > except that it tests whether the left side is defined (not None in > Python terms) instead of truthy. This still isn't bulletproof but I've > found it very handy. So, if starting from: def method( self, name=None, ): rather than: self.name = name if name else default_value ie self.name = name if name is True else default_value the more precise: self.name = name if name is not None or default_value or: self.name = default_value if name is None or name because "is" checks for identity, whereas "==" and True-thy encompass a range of possible alternate values? -- Regards, =dn From PythonList at DancesWithMice.info Sun Mar 17 02:46:01 2024 From: PythonList at DancesWithMice.info (dn) Date: Sun, 17 Mar 2024 19:46:01 +1300 Subject: MTG: Introductions to PyQt and DataClasses Message-ID: <8f393a18-9b22-484b-b9cb-5652d87ff842@DancesWithMice.info> The Auckland Branch of NZPUG meets this Wednesday, 20 March at 1830 NZDT (0530 UTC, midnight-ish Tue/Wed in American time-zones), for a virtual meeting. Part 1: Learn the basics of PyQt with code examples. Hannan Khan is currently consulting as a Data Scientist for the (US) National Oceanic and Atmospheric Administration. He holds a Bachelor's degree in Neuroscience as well as a Masters in Computer Science. As a keen member of the PySprings Users' Group (Colorado), his contribution is part of a collaboration between our two PUGs. Part 2: Why use Dataclasses? - will be the question asked, and answered, by yours truly. After surveying a number of groups, it seems most of us know that Dataclasses are available, but we don't use them - mostly because we haven't ascertained their place in our tool-box. By the end of this session you will, and will have good reason to use (or not) Dataclasses! Everyone is welcome from every location and any time-zone. The NZPUG Code of Conduct applies. JetBrains have kindly donated a door-prize. Our BigBlueButton web-conferencing instance is best accessed using Chromium, Brave, Vivaldi, Safari, etc, (rather than Firefox - for now). A head-set will facilitate asking questions but text-chat will be available. Please RSVP at https://www.meetup.com/nzpug-auckland/events/299764049/ See you there! =dn, Branch Leader From jschwar at sbcglobal.net Sun Mar 17 06:40:44 2024 From: jschwar at sbcglobal.net (Jim Schwartz) Date: Sun, 17 Mar 2024 05:40:44 -0500 Subject: MTG: Introductions to PyQt and DataClasses In-Reply-To: <8f393a18-9b22-484b-b9cb-5652d87ff842@DancesWithMice.info> References: <8f393a18-9b22-484b-b9cb-5652d87ff842@DancesWithMice.info> Message-ID: <1C9A4783-B8A0-436E-ADBC-B22D1F510C8E@sbcglobal.net> Will it be recorded? Sent from my iPhone > On Mar 17, 2024, at 1:47?AM, dn via Python-list wrote: > > ?The Auckland Branch of NZPUG meets this Wednesday, 20 March at 1830 NZDT (0530 UTC, midnight-ish Tue/Wed in American time-zones), for a virtual meeting. > > Part 1: Learn the basics of PyQt with code examples. > Hannan Khan is currently consulting as a Data Scientist for the (US) National Oceanic and Atmospheric Administration. He holds a Bachelor's degree in Neuroscience as well as a Masters in Computer Science. As a keen member of the PySprings Users' Group (Colorado), his contribution is part of a collaboration between our two PUGs. > > Part 2: Why use Dataclasses? > - will be the question asked, and answered, by yours truly. After surveying a number of groups, it seems most of us know that Dataclasses are available, but we don't use them - mostly because we haven't ascertained their place in our tool-box. By the end of this session you will, and will have good reason to use (or not) Dataclasses! > > Everyone is welcome from every location and any time-zone. The NZPUG Code of Conduct applies. JetBrains have kindly donated a door-prize. Our BigBlueButton web-conferencing instance is best accessed using Chromium, Brave, Vivaldi, Safari, etc, (rather than Firefox - for now). A head-set will facilitate asking questions but text-chat will be available. > > Please RSVP at https://www.meetup.com/nzpug-auckland/events/299764049/ > See you there! > =dn, Branch Leader > -- > https://mail.python.org/mailman/listinfo/python-list From hjp-python at hjp.at Sun Mar 17 11:11:53 2024 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 17 Mar 2024 16:11:53 +0100 Subject: Configuring an object via a dictionary In-Reply-To: <98171cf7-c2ea-4496-8b68-fa9a8f247b3b@DancesWithMice.info> References: <60E92E67-1412-4DE6-B330-495343107474@barrys-emacs.org> <20240316230606.qjn5zepek645tsqq@hjp.at> <98171cf7-c2ea-4496-8b68-fa9a8f247b3b@DancesWithMice.info> Message-ID: <20240317151153.cgnsvd6hd7pb767r@hjp.at> On 2024-03-17 17:15:32 +1300, dn via Python-list wrote: > On 17/03/24 12:06, Peter J. Holzer via Python-list wrote: > > On 2024-03-16 08:15:19 +0000, Barry via Python-list wrote: > > > > On 15 Mar 2024, at 19:51, Thomas Passin via Python-list wrote: > > > > I've always like writing using the "or" form and have never gotten bit > > > > > > I, on the other hand, had to fix a production problem that using ?or? introducted. > > > I avoid this idiom because it fails on falsy values. > > > > Perl has a // operator (pronounced "err"), which works like || (or), > > except that it tests whether the left side is defined (not None in > > Python terms) instead of truthy. This still isn't bulletproof but I've > > found it very handy. > > > So, if starting from: > > def method( self, name=None, ): > > rather than: > > self.name = name if name else default_value > > ie > > self.name = name if name is True else default_value These two lines don't have the same meaning (for the reason you outlined below). The second line is also not very useful. > the more precise: > > self.name = name if name is not None or default_value > > or: > > self.name = default_value if name is None or name Those are syntax errors. I think you meant to write "else" instead of "or". Yes, exactly. That's the semantic of Perl's // operator. JavaScript has a ?? operator with similar semantics (slightly complicated by the fact that JavaScript has two "nullish" values). hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From jschwar at sbcglobal.net Sun Mar 17 06:40:44 2024 From: jschwar at sbcglobal.net (Jim Schwartz) Date: Sun, 17 Mar 2024 05:40:44 -0500 Subject: MTG: Introductions to PyQt and DataClasses In-Reply-To: <8f393a18-9b22-484b-b9cb-5652d87ff842@DancesWithMice.info> References: <8f393a18-9b22-484b-b9cb-5652d87ff842@DancesWithMice.info> Message-ID: <1C9A4783-B8A0-436E-ADBC-B22D1F510C8E@sbcglobal.net> Will it be recorded? Sent from my iPhone > On Mar 17, 2024, at 1:47?AM, dn via Python-list wrote: > > ?The Auckland Branch of NZPUG meets this Wednesday, 20 March at 1830 NZDT (0530 UTC, midnight-ish Tue/Wed in American time-zones), for a virtual meeting. > > Part 1: Learn the basics of PyQt with code examples. > Hannan Khan is currently consulting as a Data Scientist for the (US) National Oceanic and Atmospheric Administration. He holds a Bachelor's degree in Neuroscience as well as a Masters in Computer Science. As a keen member of the PySprings Users' Group (Colorado), his contribution is part of a collaboration between our two PUGs. > > Part 2: Why use Dataclasses? > - will be the question asked, and answered, by yours truly. After surveying a number of groups, it seems most of us know that Dataclasses are available, but we don't use them - mostly because we haven't ascertained their place in our tool-box. By the end of this session you will, and will have good reason to use (or not) Dataclasses! > > Everyone is welcome from every location and any time-zone. The NZPUG Code of Conduct applies. JetBrains have kindly donated a door-prize. Our BigBlueButton web-conferencing instance is best accessed using Chromium, Brave, Vivaldi, Safari, etc, (rather than Firefox - for now). A head-set will facilitate asking questions but text-chat will be available. > > Please RSVP at https://www.meetup.com/nzpug-auckland/events/299764049/ > See you there! > =dn, Branch Leader > -- > https://mail.python.org/mailman/listinfo/python-list From PythonList at DancesWithMice.info Sun Mar 17 16:34:36 2024 From: PythonList at DancesWithMice.info (dn) Date: Mon, 18 Mar 2024 09:34:36 +1300 Subject: MTG: Introductions to PyQt and DataClasses In-Reply-To: <1C9A4783-B8A0-436E-ADBC-B22D1F510C8E@sbcglobal.net> References: <8f393a18-9b22-484b-b9cb-5652d87ff842@DancesWithMice.info> <1C9A4783-B8A0-436E-ADBC-B22D1F510C8E@sbcglobal.net> Message-ID: <8260dfe2-c027-4302-b623-1712eebc93b7@DancesWithMice.info> On 17/03/24 23:40, Jim Schwartz wrote: > Will it be recorded? Better than that (assumption) "coming soon" - please join-up or keep an eye on PySprings' Meetup ANNs: https://www.meetup.com/pysprings/ >> On Mar 17, 2024, at 1:47?AM, dn via Python-list wrote: >> >> ?The Auckland Branch of NZPUG meets this Wednesday, 20 March at 1830 NZDT (0530 UTC, midnight-ish Tue/Wed in American time-zones), for a virtual meeting. >> >> Part 1: Learn the basics of PyQt with code examples. >> Hannan Khan is currently consulting as a Data Scientist for the (US) National Oceanic and Atmospheric Administration. He holds a Bachelor's degree in Neuroscience as well as a Masters in Computer Science. As a keen member of the PySprings Users' Group (Colorado), his contribution is part of a collaboration between our two PUGs. >> >> Part 2: Why use Dataclasses? >> - will be the question asked, and answered, by yours truly. After surveying a number of groups, it seems most of us know that Dataclasses are available, but we don't use them - mostly because we haven't ascertained their place in our tool-box. By the end of this session you will, and will have good reason to use (or not) Dataclasses! >> >> Everyone is welcome from every location and any time-zone. The NZPUG Code of Conduct applies. JetBrains have kindly donated a door-prize. Our BigBlueButton web-conferencing instance is best accessed using Chromium, Brave, Vivaldi, Safari, etc, (rather than Firefox - for now). A head-set will facilitate asking questions but text-chat will be available. >> >> Please RSVP at https://www.meetup.com/nzpug-auckland/events/299764049/ >> See you there! >> =dn, Branch Leader >> -- >> https://mail.python.org/mailman/listinfo/python-list > -- Regards, =dn From jschwar at sbcglobal.net Sun Mar 17 17:02:12 2024 From: jschwar at sbcglobal.net (Jim Schwartz) Date: Sun, 17 Mar 2024 16:02:12 -0500 Subject: MTG: Introductions to PyQt and DataClasses In-Reply-To: <8260dfe2-c027-4302-b623-1712eebc93b7@DancesWithMice.info> References: <8260dfe2-c027-4302-b623-1712eebc93b7@DancesWithMice.info> Message-ID: <8588900C-3514-4B70-BD78-ABD3073BCC7F@sbcglobal.net> Actually, I have a sleep disorder that requires me to keep a constant sleep schedule. Thats why I asked. Sent from my iPhone > On Mar 17, 2024, at 3:36?PM, dn via Python-list wrote: > > ?On 17/03/24 23:40, Jim Schwartz wrote: >> Will it be recorded? > > Better than that (assumption) "coming soon" - please join-up or keep an eye on PySprings' Meetup ANNs: https://www.meetup.com/pysprings/ > > >>>> On Mar 17, 2024, at 1:47?AM, dn via Python-list wrote: >>> >>> ?The Auckland Branch of NZPUG meets this Wednesday, 20 March at 1830 NZDT (0530 UTC, midnight-ish Tue/Wed in American time-zones), for a virtual meeting. >>> >>> Part 1: Learn the basics of PyQt with code examples. >>> Hannan Khan is currently consulting as a Data Scientist for the (US) National Oceanic and Atmospheric Administration. He holds a Bachelor's degree in Neuroscience as well as a Masters in Computer Science. As a keen member of the PySprings Users' Group (Colorado), his contribution is part of a collaboration between our two PUGs. >>> >>> Part 2: Why use Dataclasses? >>> - will be the question asked, and answered, by yours truly. After surveying a number of groups, it seems most of us know that Dataclasses are available, but we don't use them - mostly because we haven't ascertained their place in our tool-box. By the end of this session you will, and will have good reason to use (or not) Dataclasses! >>> >>> Everyone is welcome from every location and any time-zone. The NZPUG Code of Conduct applies. JetBrains have kindly donated a door-prize. Our BigBlueButton web-conferencing instance is best accessed using Chromium, Brave, Vivaldi, Safari, etc, (rather than Firefox - for now). A head-set will facilitate asking questions but text-chat will be available. >>> >>> Please RSVP at https://www.meetup.com/nzpug-auckland/events/299764049/ >>> See you there! >>> =dn, Branch Leader >>> -- >>> https://mail.python.org/mailman/listinfo/python-list > > -- > Regards, > =dn > > -- > https://mail.python.org/mailman/listinfo/python-list From PythonList at DancesWithMice.info Sun Mar 17 17:09:27 2024 From: PythonList at DancesWithMice.info (dn) Date: Mon, 18 Mar 2024 10:09:27 +1300 Subject: Configuring an object via a dictionary In-Reply-To: <20240317151153.cgnsvd6hd7pb767r@hjp.at> References: <60E92E67-1412-4DE6-B330-495343107474@barrys-emacs.org> <20240316230606.qjn5zepek645tsqq@hjp.at> <98171cf7-c2ea-4496-8b68-fa9a8f247b3b@DancesWithMice.info> <20240317151153.cgnsvd6hd7pb767r@hjp.at> Message-ID: <9206c6aa-0701-4d06-a5b9-29c79af215b0@DancesWithMice.info> On 18/03/24 04:11, Peter J. Holzer via Python-list wrote: > On 2024-03-17 17:15:32 +1300, dn via Python-list wrote: >> On 17/03/24 12:06, Peter J. Holzer via Python-list wrote: >>> On 2024-03-16 08:15:19 +0000, Barry via Python-list wrote: >>>>> On 15 Mar 2024, at 19:51, Thomas Passin via Python-list wrote: >>>>> I've always like writing using the "or" form and have never gotten bit >>>> >>>> I, on the other hand, had to fix a production problem that using ?or? introducted. >>>> I avoid this idiom because it fails on falsy values. >>> >>> Perl has a // operator (pronounced "err"), which works like || (or), >>> except that it tests whether the left side is defined (not None in >>> Python terms) instead of truthy. This still isn't bulletproof but I've >>> found it very handy. >> >> >> So, if starting from: >> >> def method( self, name=None, ): >> >> rather than: >> >> self.name = name if name else default_value >> >> ie >> >> self.name = name if name is True else default_value > > These two lines don't have the same meaning (for the reason you outlined > below). The second line is also not very useful. > > > >> the more precise: >> >> self.name = name if name is not None or default_value >> >> or: >> >> self.name = default_value if name is None or name > > Those are syntax errors. I think you meant to write "else" instead of > "or". > > Yes, exactly. That's the semantic of Perl's // operator. > > JavaScript has a ?? operator with similar semantics (slightly > complicated by the fact that JavaScript has two "nullish" values). Thanks Peter! (yes, sad consequences of suffering a neighbor's party-til-midnight followed by an 0530 meeting-start - children: don't code exhausted!) Herewith, an illustration of how the corrected version of the above works - and how (in what seem unusual cases) it avoids any truthy/falsy confusion, as raised earlier in this thread: >>> default_value = "default" >>> name = "Fred Flintstone" >>> name if name is not None else default_value 'Fred Flintstone' >>> name = None >>> name if name is not None else default_value 'default' >>> name = False >>> name if name is not None else default_value False >>> name = 1 >>> name if name is not None else default_value 1 >>> name = 0 >>> name if name is not None else default_value 0 Personally: I find the above coding more logical, because our primary interest is on 'the happy case', ie where the value has been assigned (to "name"); and the default_value is only applied as a "guard". On the other hand, I dislike the not-condition because it forces me to think (and maybe dust-off DeMorgan). Accordingly: >>> name = "Fred Flintstone" >>> default_value if name is None else name 'Fred Flintstone' >>> name = None >>> default_value if name is None else name 'default' >>> name = False >>> default_value if name is None else name False ... YMMV! NB your corporate Style Guide may prefer 'the happy path'... -- Regards, =dn From PythonList at DancesWithMice.info Sun Mar 17 17:18:08 2024 From: PythonList at DancesWithMice.info (dn) Date: Mon, 18 Mar 2024 10:18:08 +1300 Subject: MTG: Introductions to PyQt and DataClasses In-Reply-To: <8588900C-3514-4B70-BD78-ABD3073BCC7F@sbcglobal.net> References: <8260dfe2-c027-4302-b623-1712eebc93b7@DancesWithMice.info> <8588900C-3514-4B70-BD78-ABD3073BCC7F@sbcglobal.net> Message-ID: <62cace3b-bcfa-481b-ab12-5b457caac18f@DancesWithMice.info> On 18/03/24 10:02, Jim Schwartz wrote: > Actually, I have a sleep disorder that requires me to keep a constant sleep schedule. Thats why I asked. At a weekend meeting, discussion swirled around topics such as the best way to learn/work, how much work we should attempt in one sitting, could/should I 'do more', and similar. One of the valuable observations is that most of us would benefit by improving our sleep-schedule and ensuring we do sleep for sufficient time (probably longer than current habit). -- Regards, =dn From avi.e.gross at gmail.com Sun Mar 17 17:26:51 2024 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Sun, 17 Mar 2024 17:26:51 -0400 Subject: Configuring an object via a dictionary In-Reply-To: <20240317151153.cgnsvd6hd7pb767r@hjp.at> References: <60E92E67-1412-4DE6-B330-495343107474@barrys-emacs.org> <20240316230606.qjn5zepek645tsqq@hjp.at> <98171cf7-c2ea-4496-8b68-fa9a8f247b3b@DancesWithMice.info> <20240317151153.cgnsvd6hd7pb767r@hjp.at> Message-ID: <002801da78b1$d3f982e0$7bec88a0$@gmail.com> If we are bringing up other languages, let's return to what was part of the original question. How van a dictionary be used in python if your goal is to sort of use it to instantiate it into a set of variables and values inside the local or global or other namespaces? Can we learn anything from other programming languages with other paradigms? I was thinking that in one sense, python has other kinds of collections such as a class or class member with internal variables that also has some features similar enough. I mean if I were to call a function with one argument being a dictionary like {"a":1, "c": 3} where in theory any number of similar key value pairs might exist or not be present, then the question sounded like a request to create variables, in this case a, and c, that hold those values. I will not debate the wisdom of doing that, of course. Nor will I debate what should happen if some of those variable names are already in use. OK? There are ways to do this albeit some are a tad obscure such as taking a printable representation of the dictionary and editing it and feeding that to an eval. But could you serve a similar purpose by passing an object containing varying internal fields (or methods) and values including some of the dataclasses Dave Neal is highlighting? Is there some overlap with using dictionaries? In Javascript, I would say they have a very different view in which all kinds of things overlap and their objects can even implement what others might call arrays albeit a tad weirder like allowing missing indices and ignoring non-numeric indices for some purposes. Someone may wish to chime in if people can and do take such objects passed in and split them into individual variables as requested. What I wanted to mention is some philosophical issues in the R language in which the default language used other data structures to loosely support what dictionaries often do in python. They have named lists where components optionally can have a name as in list(a=5, "b"=6, 7, "hello world") and they have data structures called environments. Loosely, an environment is a set of name=value pairs and is pretty much like a dictionary and is mostly used behind the scenes as the interpreter searches for variable names in a sequence of environments such as the parent environment. But you can use environments all over the place on purpose and as noted, a named list simulates an environment. So a fairly common usage when using base R is to take a data.frame (which is at first approximation a list of named vectors all the same length) and want to work with the column names without extra typing. If I had a data.frame that looked like mydf <- data.frame(a=1:3, b=3:1) then if I wanted to add corresponding entries, I might type: result <- mydf$a + mydf$b inside a with statement, an environment is put on the stack consisting of the contents of mydf and you can now use things like: result <- with(mydf, a+b) There is more but the point is for those who hate the extra typing of long names, this can be useful and a tad dangerous if the variable names are not unique. But as R delays evaluation in various ways, a similar style has evolved in an assortment of packages to the point where I often program in a style that looks like: result <- mydf |> mutate(newcol = a+b, doubled = 2*newcol, ...) The point is that all kinds of things that seem like local variables can be used as if they had been declared withing some environment but that are not available once you leave that region. So perhaps there is some validity in a request to be able to just pass an argument as a dictionary in python and have it unpacked. In actuality, I wonder if the OP is aware of the unpacking functionality you can get using **dict in a function invocation. Say you have a function that expects any combination of three variables called the unoriginal names of alpha/beta/gamma and you want to call it with a dictionary that contains any subset of those same keys and nothing else: mydict = {"alpha":5, "beta":6} def unpacked(alpha=None, beta=None, gamma=None): print(alpha if alpha != None else "No alpha") print(beta if beta != None else "No beta") print(gamma if gamma != None else "No gamma") If I now call unpacked with ** mydict: >>> unpacked(**mydict) 5 6 No gamma Within the body of that function, arguably, I can tell if something was passed or not, assuming None or any other sentinel I choose is not used except in setting the default. And if you add other unknown names, like delta, they seem to be ignored without harmful effects or can be caught in other ways. >>> unpacked({"gamma":7, "delta":8}) {'gamma': 7, 'delta': 8} No beta No gamma So given a function similar to this, and you wanting LOCAL variables set, how would this do if you wanted these three or any number, and in a row: mydict = {"beta":6, "alpha":5} def dict_to_vars(alpha=None, beta=None, gamma=None): return(alpha, beta, gamma) alpha, beta, gamma = dict_to_vars(**mydict) The result works no matter what order you have added to the dictionary as long as you know exactly which N you want. Obviously, you may not be thrilled with None as a value and can add code to remove empty variables after they have been instantiated. >>> alpha 5 >>> beta 6 >>> gamma >>> If you want to ignore some entries, use _ in your list of places for items you want to toss. alpha, _, _ = dict_to_vars(**mydict) The above is really just keeping alpha. Of course if the possible keys are not known in advance, this does not work but other languages that allow this may be better for your purpose. -----Original Message----- From: Python-list On Behalf Of Peter J. Holzer via Python-list Sent: Sunday, March 17, 2024 11:12 AM To: python-list at python.org Subject: Re: Configuring an object via a dictionary On 2024-03-17 17:15:32 +1300, dn via Python-list wrote: > On 17/03/24 12:06, Peter J. Holzer via Python-list wrote: > > On 2024-03-16 08:15:19 +0000, Barry via Python-list wrote: > > > > On 15 Mar 2024, at 19:51, Thomas Passin via Python-list wrote: > > > > I've always like writing using the "or" form and have never gotten bit > > > > > > I, on the other hand, had to fix a production problem that using ?or? introducted. > > > I avoid this idiom because it fails on falsy values. > > > > Perl has a // operator (pronounced "err"), which works like || (or), > > except that it tests whether the left side is defined (not None in > > Python terms) instead of truthy. This still isn't bulletproof but I've > > found it very handy. > > > So, if starting from: > > def method( self, name=None, ): > > rather than: > > self.name = name if name else default_value > > ie > > self.name = name if name is True else default_value These two lines don't have the same meaning (for the reason you outlined below). The second line is also not very useful. > the more precise: > > self.name = name if name is not None or default_value > > or: > > self.name = default_value if name is None or name Those are syntax errors. I think you meant to write "else" instead of "or". Yes, exactly. That's the semantic of Perl's // operator. JavaScript has a ?? operator with similar semantics (slightly complicated by the fact that JavaScript has two "nullish" values). hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" From h.goebel at crazy-compilers.com Mon Mar 18 08:50:57 2024 From: h.goebel at crazy-compilers.com (Hartmut Goebel) Date: Mon, 18 Mar 2024 13:50:57 +0100 Subject: [Ann] managesieve v0.8 released Message-ID: Announcing: ???????????? managesieve 0.8 ?? ?RFC-5804 Manage Sieve client library for remotely managing Sieve scripts, ?? ?including an user application (the interactive 'sieveshell'). :Homepage: https://managesieve.readthedocs.io/ :Author:?? Hartmut Goebel :License: ? - for the managesieve module: Python-Software-Foundation-like License ? - for sieveshell and test suite: GNU Public Licence v3 (GPLv3) :Quick Installation: ??? pip install -U managesieve :Tarballs:? https://pypi.org/project/managesieve/#files What is managesieve? --------------------- A ManageSieve client library for remotely managing Sieve scripts, including an user application (the interactive 'sieveshell'). Sieve scripts allow users to filter incoming email on the mail server and are supported by many IMAP servers. The ManageSieve protocol allows managing Sieve scripts on a remote mail server. These servers are commonly sealed so users cannot log into them, yet users must be able to update their scripts on them. This is what for the "ManageSieve" protocol is. For more information about the ManageSieve protocol see `RFC 5804 `_. This module allows accessing a Sieve-Server for managing Sieve scripts there. It is accompanied by a simple yet functional user application 'sieveshell'. Changes since last release --------------------------------------- * Now supports Python 3.6 to 3.12. :managesieve: ?? - Add support for the UNAUTHENTICATE command. ?? - Add a socket timeout parameter. ?? - Add support for IPv6. ?? - Allow disabling certificate verification. ?? - Follow the 'Logging for a Library' guideline. ?? - BREAKING: Rearrange DEBUG logging levels to be more reasonable. ???? See `docs/Logging.rst` for details. :sieveshell: ?? - Add option '--no-tls-verify'. ?? - Improve error message if TLS certificate verification fails. ?? - Keep line-endings on file IO. ?? - Remove temporary file on successful edit, too. ?? - Fix: Pass to sieve.login() the Authorization ID :general: ?? - Add support for Python 3.12. ?? - Improve testing, add a tox.ini file and add CI/CD. ?? - Fix SPDX license identifier. ?? - Fix several typos. ?? - Lint all the code. ?? - Remove unused code. -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel at crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible | From ajm at flonidan.dk Mon Mar 18 09:57:57 2024 From: ajm at flonidan.dk (Anders Munch) Date: Mon, 18 Mar 2024 13:57:57 +0000 Subject: Configuring an object via a dictionary In-Reply-To: <2e09744d-7f46-4c47-a439-b8a2f69d750e@DancesWithMice.info> References: <87y1ajdeqs.fsf@zedat.fu-berlin.de> <2e09744d-7f46-4c47-a439-b8a2f69d750e@DancesWithMice.info> Message-ID: dn wrote: >Loris Bennett wrote: >> However, with a view to asking forgiveness rather than >> permission, is there some simple way just to assign the dictionary >> elements which do in fact exist to self-variables? > >Assuming config is a dict: > > self.__dict__.update( config ) Here's another approach: config_defaults = dict( server_host='localhost', server_port=443, # etc. ) ... def __init__(self, config): self.conf = types.SimpleNamespace(**{**config_defaults, **config}) This gives you defaults, simple attribute access, and avoids the risk of name collisions that you get when updating __dict__. Using a dataclass may be better: @dataclasses.dataclass class Settings: group_base : str server_host : str = 'localhost' server_port : int = 443 ... def __init__(self, config): self.conf = Settings(**config) regards, Anders From toby at tobiah.org Fri Mar 15 14:59:20 2024 From: toby at tobiah.org (Tobiah) Date: Fri, 15 Mar 2024 11:59:20 -0700 Subject: Configuring an object via a dictionary In-Reply-To: <87y1ajdeqs.fsf@zedat.fu-berlin.de> References: <87y1ajdeqs.fsf@zedat.fu-berlin.de> Message-ID: On 3/15/24 02:30, Loris Bennett wrote: > Hi, > > I am initialising an object via the following: > > def __init__(self, config): > > self.connection = None > > self.source_name = config['source_name'] > self.server_host = config['server_host'] > However, with a view to asking forgiveness rather than > permission, is there some simple way just to assign the dictionary > elements which do in fact exist to self-variables? class Foo(): def __init__(self, config): for key, val in config.iteritems(): setattr(self, key, val) f = Foo({'cat': 'dog'}) print(f.cat) (outputs 'dog') From toby at tobiah.org Fri Mar 15 15:22:30 2024 From: toby at tobiah.org (Tobiah) Date: Fri, 15 Mar 2024 12:22:30 -0700 Subject: Configuring an object via a dictionary In-Reply-To: References: <87y1ajdeqs.fsf@zedat.fu-berlin.de> Message-ID: I should mention that I wanted to answer your question, but I wouldn't actually do this. I'd rather opt for your self.config = config solution. The config options should have their own namespace. I don't mind at all referencing foo.config['option'], or you could make foo.config an object by itself so you can do foo.config.option. You'd fill it's attributes in the same way I suggested for your main object. From loris.bennett at fu-berlin.de Mon Mar 18 02:42:32 2024 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Mon, 18 Mar 2024 07:42:32 +0100 Subject: Configuring an object via a dictionary References: <87y1ajdeqs.fsf@zedat.fu-berlin.de> Message-ID: <875xxk2g87.fsf@zedat.fu-berlin.de> Tobiah writes: > I should mention that I wanted to answer your question, > but I wouldn't actually do this. I'd rather opt for > your self.config = config solution. The config options > should have their own namespace. > > I don't mind at all referencing foo.config['option'], > or you could make foo.config an object by itself so > you can do foo.config.option. You'd fill it's attributes > in the same way I suggested for your main object. Thanks for the thoughts. I'll go for self.config = config after all, since, as you say, the clutter caused by the referencing is not that significant. Cheers, Loris -- This signature is currently under constuction. From lukasz at langa.pl Tue Mar 19 20:34:46 2024 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Wed, 20 Mar 2024 01:34:46 +0100 Subject: [RELEASE] Python 3.10.14, 3.9.19, and 3.8.19 is now available Message-ID: <6CA8C58A-3ABD-4F75-9357-D48FFCBFFDFC@langa.pl> Howdy! Those are the boring security releases that aren?t supposed to bring anything new. But not this time! We do have a bit of news, actually. But first things first: go update your systems! Python 3.10.14 Get it here: Python Release Python 3.10.14 26 commits since last release. Python 3.9.19 Get it here: Python Release Python 3.9.19 26 commits since last release. Python 3.8.19 Get it here: Python Release Python 3.8.19 28 commits since last release. Security content in this release gh-115399 & gh-115398 : bundled libexpat was updated to 2.6.0 to address CVE-2023-52425 , and control of the new reparse deferral functionality was exposed with new APIs. Thanks to Sebastian Pipping, the maintainer of libexpat, who worked with us directly on incorporating those fixes! gh-109858 : zipfile is now protected from the ?quoted-overlap? zipbomb to address CVE-2024-0450 . It now raises BadZipFile when attempting to read an entry that overlaps with another entry or central directory gh-91133 : tempfile.TemporaryDirectory cleanup no longer dereferences symlinks when working around file system permission errors to address CVE-2023-6597 gh-115197 : urllib.request no longer resolves the hostname before checking it against the system?s proxy bypass list on macOS and Windows gh-81194 : a crash in socket.if_indextoname() with a specific value (UINT_MAX) was fixed. Relatedly, an integer overflow in socket.if_indextoname() on 64-bit non-Windows platforms was fixed gh-113659 : .pth files with names starting with a dot or containing the hidden file attribute are now skipped gh-102388 : iso2022_jp_3 and iso2022_jp_2004 codecs no longer read out of bounds gh-114572 : ssl.SSLContext.cert_store_stats() and ssl.SSLContext.get_ca_certs() now correctly lock access to the certificate store, when the ssl.SSLContext is shared across multiple threads Stay safe and upgrade! Upgrading is highly recommended to all users of affected versions. Source builds are moving to GitHub Actions It?s not something you will notice when downloading, but 3.10.14 here is the first release we?ve done were the source artifacts were built on GHA and not on a local computer of one of the release managers. We have the Security Developer in Residence @sethmlarson to thank for that! It?s a big deal since public builds allow for easier auditing and repeatability. It also helps with the so-called bus factor. In fact, to test this out, this build of 3.10.14 was triggered by me and not Pablo, who would usually release Python 3.10. The artifacts are later still signed by the respective release manager, ensuring integrity when put on the downloads server. Python now manages its own CVEs The security releases you?re looking at are the first after the PSF became a CVE Numbering Authority . That?s also thanks to @sethmlarson . What being our own CNA allows us to ensure the quality of the vulnerability reports is high, and the severity estimate is accurate. Seth summarized it best in his announcement here . What this also allows us to do is to combine announcement of CVEs with the release of patched versions of Python. This is in fact the case with two of the CVEs listed above (CVE-2023-6597 and CVE-2024-0450 ). And since Seth is now traveling, this announcement duty was fulfilled by the PSF?s Director of Infrastructure @EWDurbin . Thanks! I?m happy to see us successfully testing bus factor resilience on multiple fronts with this round of releases. Thank you for your support Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. Python.org - the official home of the Python Programming Language. ? ?ukasz Langa @ambv on behalf of your friendly release team, Ned Deily @nad Steve Dower @steve.dower Pablo Galindo Salgado @pablogsal ?ukasz Langa @ambv Thomas Wouters @thomas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From pokemonchw at gmail.com Mon Mar 18 22:48:02 2024 From: pokemonchw at gmail.com (Pokemon Chw) Date: Tue, 19 Mar 2024 10:48:02 +0800 Subject: Configuring an object via a dictionary In-Reply-To: <875xxk2g87.fsf@zedat.fu-berlin.de> References: <87y1ajdeqs.fsf@zedat.fu-berlin.de> <875xxk2g87.fsf@zedat.fu-berlin.de> Message-ID: It's too complicated, there's no need for this def __init__(self, config): self.__dict__ = config self.connection = None """ other code ..... """ Note that you need to keep the fields in the config dict named the same as the fields you want to be assigned to in your class Loris Bennett via Python-list ?2024?3?19??? 01:39??? > Tobiah writes: > > > I should mention that I wanted to answer your question, > > but I wouldn't actually do this. I'd rather opt for > > your self.config = config solution. The config options > > should have their own namespace. > > > > I don't mind at all referencing foo.config['option'], > > or you could make foo.config an object by itself so > > you can do foo.config.option. You'd fill it's attributes > > in the same way I suggested for your main object. > > Thanks for the thoughts. I'll go for self.config = config after > all, since, as you say, the clutter caused by the referencing is not > that significant. > > Cheers, > > Loris > > -- > This signature is currently under constuction. > -- > https://mail.python.org/mailman/listinfo/python-list > From greg.ewing at canterbury.ac.nz Wed Mar 20 03:29:30 2024 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 20 Mar 2024 20:29:30 +1300 Subject: GIL-Removal Project Takes Another Step (Posting On Python-List Prohibited) In-Reply-To: References: <87v85jxepj.fsf@nightsong.com> <87r0g5ybbp.fsf@nightsong.com> Message-ID: On 20/03/24 4:14 pm, Lawrence D'Oliveiro wrote: > not to > mention the latency when there isn?t quite enough memory for an allocation > and you have to wait until the next GC run to proceed. Run the GC a > thousand times a second, and the latency is still 1 millisecond. That's not the way it usually works. If you run out of memory, you run a GC there and then. You don't have to wait for GCs to occur on a time schedule. Also, as a previous poster pointed out, GCs are typically scheduled by number of allocations, not by time. -- Greg From rosuav at gmail.com Wed Mar 20 03:42:21 2024 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 20 Mar 2024 18:42:21 +1100 Subject: GIL-Removal Project Takes Another Step (Posting On Python-List Prohibited) In-Reply-To: References: <87v85jxepj.fsf@nightsong.com> <87r0g5ybbp.fsf@nightsong.com> Message-ID: On Wed, 20 Mar 2024 at 18:31, Greg Ewing via Python-list wrote: > > On 20/03/24 4:14 pm, Lawrence D'Oliveiro wrote: > > not to > > mention the latency when there isn?t quite enough memory for an allocation > > and you have to wait until the next GC run to proceed. Run the GC a > > thousand times a second, and the latency is still 1 millisecond. > > That's not the way it usually works. If you run out of memory, you > run a GC there and then. You don't have to wait for GCs to occur on > a time schedule. > > Also, as a previous poster pointed out, GCs are typically scheduled > by number of allocations, not by time. > FYI you're violating someone's request by responding to them in a way that results in it getting onto python-list, so it's probably safest to just ignore cranks and trolls and let them stew in their own juices. But normally the GC doesn't need to be scheduled at all. In CPython, the only reason to "run garbage collection" is to detect cycles, so you would have to be generating inordinate amounts of cyclic garbage for this to matter at all. ChrisA From roel at roelschroeven.net Wed Mar 20 04:49:54 2024 From: roel at roelschroeven.net (Roel Schroeven) Date: Wed, 20 Mar 2024 09:49:54 +0100 Subject: Configuring an object via a dictionary In-Reply-To: References: <60E92E67-1412-4DE6-B330-495343107474@barrys-emacs.org> <20240316230606.qjn5zepek645tsqq@hjp.at> <98171cf7-c2ea-4496-8b68-fa9a8f247b3b@DancesWithMice.info> <20240317151153.cgnsvd6hd7pb767r@hjp.at> <9206c6aa-0701-4d06-a5b9-29c79af215b0@DancesWithMice.info> Message-ID: <6a691f90-3198-47c8-bb35-62a789280ae7@roelschroeven.net> Op 19/03/2024 om 0:44 schreef Gilmeh Serda via Python-list: > On Mon, 18 Mar 2024 10:09:27 +1300, dn wrote: > > > YMMV! > > NB your corporate Style Guide may prefer 'the happy path'... > > If you only want to check for None, this works too: > > >>> name = None > >>> dafault_value = "default" > >>> name or default_value > 'default' > >>> name = 'Fred Flintstone' > >>> name or default_value > 'Fred Flintstone' >>> name = '' >>> name or default_value 'default' >>> name = False >>> name or default_value 'default' >>> name = [] >>> name or default_value 'default' >>> name = 0 >>> name or default_value 'default' You haven't only checked for None! You have rejected *every* falsish value, even though they may very well be acceptable values. -- "Most of us, when all is said and done, like what we like and make up reasons for it afterwards." -- Soren F. Petersen From 2QdxY4RzWzUUiLuE at potatochowder.com Wed Mar 20 05:28:24 2024 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Wed, 20 Mar 2024 05:28:24 -0400 Subject: Configuring an object via a dictionary In-Reply-To: <6a691f90-3198-47c8-bb35-62a789280ae7@roelschroeven.net> References: <60E92E67-1412-4DE6-B330-495343107474@barrys-emacs.org> <20240316230606.qjn5zepek645tsqq@hjp.at> <98171cf7-c2ea-4496-8b68-fa9a8f247b3b@DancesWithMice.info> <20240317151153.cgnsvd6hd7pb767r@hjp.at> <9206c6aa-0701-4d06-a5b9-29c79af215b0@DancesWithMice.info> <6a691f90-3198-47c8-bb35-62a789280ae7@roelschroeven.net> Message-ID: On 2024-03-20 at 09:49:54 +0100, Roel Schroeven via Python-list wrote: > You haven't only checked for None! You have rejected *every* falsish value, > even though they may very well be acceptable values. OTOH, only you can answer these questions about your situations. Every application, every item of configuration data, is going to be a little bit different. What, exactly, does "missing" mean? That there's no entry in a config file? That there's some sort of degenerate entry with "missing" semantics (e.g. a line in a text file that contains the name of the value and an equals sign, but no value)? An empty string or list? Are you making your program easier for users to use, easier for testers to test, easier for authors to write and to maintain, or something else? What is your program allowed and not allowed to do in the face of "missing" configuration data? Once you've nailed down the semantics of the configuration data, then the code usually falls out pretty quickly. But arguing about corner cases and failure modes without specifications is a losing battle. Every piece of code is suspect unless you know what the inputs mean, and what the application "should" do if the don't look like that. Python's flexibiliry and expressiveness are double edge swords. Use them wisely. :-) Sorry for the rant. Carry on. From twn at thomasnyberg.com Wed Mar 20 04:22:51 2024 From: twn at thomasnyberg.com (Thomas Nyberg) Date: Wed, 20 Mar 2024 09:22:51 +0100 Subject: Using a background thread with asyncio/futures with flask Message-ID: <8a7231df-c30c-4fa6-9daa-546b3453b547@thomasnyberg.com> Hello, I have a simple (and not working) example of what I'm trying to do. This is a simplified version of what I'm trying to achieve (obviously the background workers and finalizer functions will do more later): `app.py` ``` import asyncio import threading import time from queue import Queue from flask import Flask in_queue = Queue() out_queue = Queue() def worker(): print("worker started running") while True: future = in_queue.get() print(f"worker got future: {future}") time.sleep(5) print("worker sleeped") out_queue.put(future) def finalizer(): print("finalizer started running") while True: future = out_queue.get() print(f"finalizer got future: {future}") future.set_result("completed") print("finalizer set result") threading.Thread(target=worker, daemon=True).start() threading.Thread(target=finalizer, daemon=True).start() app = Flask(__name__) @app.route("/") async def root(): future = asyncio.get_event_loop().create_future() in_queue.put(future) print(f"root put future: {future}") result = await future return result if __name__ == "__main__": app.run() ``` If I start up that server, and execute `curl http://localhost:5000`, it prints out the following in the server before hanging: ``` $ python3 app.py worker started running finalizer started running * Serving Flask app 'app' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on http://127.0.0.1:5000 Press CTRL+C to quit root put future: worker got future: worker sleeped finalizer got future: finalizer set result ``` Judging by what's printing out, the `final result = await future` doesn't seem to be happy here. Maybe someone sees something obvious I'm doing wrong here? I presume I'm mixing threads and asyncio in a way I shouldn't be. Here's some system information (just freshly installed with pip3 install flask[async] in a virtual environment for python version 3.11.2): ``` $ uname -a Linux x1carbon 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux $ python3 -V Python 3.11.2 $ pip3 freeze asgiref==3.7.2 blinker==1.7.0 click==8.1.7 Flask==3.0.2 itsdangerous==2.1.2 Jinja2==3.1.3 MarkupSafe==2.1.5 Werkzeug==3.0.1 ``` Thanks for any help! Cheers, Thomas From loris.bennett at fu-berlin.de Thu Mar 21 05:56:42 2024 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Thu, 21 Mar 2024 10:56:42 +0100 Subject: Popping key causes dict derived from object to revert to object Message-ID: <87zfurgb6t.fsf@zedat.fu-berlin.de> Hi, I am using SQLAlchemy to extract some rows from a table of 'events'. >From the call to the DB I get a list of objects of the type sqlalchemy.orm.state.InstanceState I would like to print these rows to the terminal using the 'tabulate' package, the documentation for which says The module provides just one function, tabulate, which takes a list of lists or another tabular data type as the first argument, and outputs a nicely formatted plain-text table So as I understand it, I need to convert the InstanceState-objects to, say, dicts, in order to print them. However I also want to remove one of the keys from the output and assumed I could just pop it off each event dict, thus: event_dicts = [vars(e) for e in events] print(type(event_dicts[0])) event_dicts = [e.pop('_sa_instance_state', None) for e in event_dicts] print(type(event_dicts[0])) However, this prints If I comment out the third line, which pops the unwanted key, I get Why does popping one of the keys cause the elements of the list to revert back to their original class? Cheers, Loris -- This signature is currently under constuction. From jfairchild at tudado.org Thu Mar 21 07:36:21 2024 From: jfairchild at tudado.org (Johanne Fairchild) Date: Thu, 21 Mar 2024 08:36:21 -0300 Subject: the name ``wheel'' Message-ID: <87plvnakay.fsf@tudado.org> Why is a whl-package called a ``wheel''? Is it just a pronunciation for the extension WHL or is it really a name? Also, it seems that when I install Python on Windows, it doesn't come with pip ready to run. I had to say python -m ensurepip and then I saw that a pip on a whl-package was installed. Why doesn't the official distribution make pip ready to run by default? Thank you! From dieter.maurer at online.de Thu Mar 21 12:21:52 2024 From: dieter.maurer at online.de (dieter.maurer at online.de) Date: Thu, 21 Mar 2024 17:21:52 +0100 Subject: Popping key causes dict derived from object to revert to object In-Reply-To: <87zfurgb6t.fsf@zedat.fu-berlin.de> References: <87zfurgb6t.fsf@zedat.fu-berlin.de> Message-ID: <26108.24224.409498.22809@ixdm.fritz.box> Loris Bennett wrote at 2024-3-21 10:56 +0100: > ... >So as I understand it, I need to convert the InstanceState-objects to, >say, dicts, in order to print them. However I also want to remove one >of the keys from the output and assumed I could just pop it off each >event dict, thus: > > event_dicts = [vars(e) for e in events] > print(type(event_dicts[0])) > event_dicts = [e.pop('_sa_instance_state', None) for e in event_dicts] > print(type(event_dicts[0])) > >However, this prints > > `vars` typically returns a `dict`. > This is what you have popped. > >If I comment out the third line, which pops the unwanted key, I get Then you do not change `event_dicts`. You problem likely is: `pop` does not return the `dict` after the removal of a key but the removed value. From olegsivokon at gmail.com Thu Mar 21 12:58:02 2024 From: olegsivokon at gmail.com (Left Right) Date: Thu, 21 Mar 2024 17:58:02 +0100 Subject: the name ``wheel'' In-Reply-To: <87plvnakay.fsf@tudado.org> References: <87plvnakay.fsf@tudado.org> Message-ID: I believe that the name "Wheel" was a reference to "reinventing the wheel". But I cannot find a quote to support this claim. I think the general sentiment was that it was the second attempt by the Python community to come up with a packaging format (first being Egg), and so they were reinventing the wheel, in a way. I cannot speak to the other question though: I don't know. This is however also a common practice on Linux, where Python is often installed in order to enable system tools, which, in turn, don't need a Python package manager to function. Not sure why this would be the case in MS Windows. On Thu, Mar 21, 2024 at 4:51?PM Johanne Fairchild via Python-list wrote: > > Why is a whl-package called a ``wheel''? Is it just a pronunciation for > the extension WHL or is it really a name? > > Also, it seems that when I install Python on Windows, it doesn't come > with pip ready to run. I had to say > > python -m ensurepip > > and then I saw that a pip on a whl-package was installed. Why doesn't > the official distribution make pip ready to run by default? Thank you! > -- > https://mail.python.org/mailman/listinfo/python-list From jfairchild at tudado.org Thu Mar 21 13:58:26 2024 From: jfairchild at tudado.org (Johanne Fairchild) Date: Thu, 21 Mar 2024 14:58:26 -0300 Subject: the name ``wheel'' References: <87plvnakay.fsf@tudado.org> Message-ID: <87frwj8o1p.fsf@tudado.org> ram at zedat.fu-berlin.de (Stefan Ram) writes: > Johanne Fairchild wrote or quoted: >>Why is a whl-package called a ``wheel''? Is it just a pronunciation for >>the extension WHL or is it really a name? > > PyPi in its initial state was named "cheese shop", as the famous > part in the show "Monty Python Cheese Shop". Because initially it > only hosted links to the packages, so it was empty like that shop. > And within a cheese shop what do you store? Wheels of cheese. Lol! Loved it. (Thanks very much.) >>Also, it seems that when I install Python on Windows, it doesn't come >>with pip ready to run. I had to say > > Some Python distributions do not come with pip pre-installed > because they have their own package management systems. But this was a Windows install. I don't think Windows has its own package management for Python packages. I'd be totally surprised. From python at mrabarnett.plus.com Thu Mar 21 14:14:53 2024 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 21 Mar 2024 18:14:53 +0000 Subject: the name ``wheel'' In-Reply-To: <87plvnakay.fsf@tudado.org> References: <87plvnakay.fsf@tudado.org> Message-ID: <6bf8d0ef-a29d-4fbf-a3da-76077aa4d988@mrabarnett.plus.com> On 2024-03-21 11:36, Johanne Fairchild via Python-list wrote: > Why is a whl-package called a ``wheel''? Is it just a pronunciation for > the extension WHL or is it really a name? > > Also, it seems that when I install Python on Windows, it doesn't come > with pip ready to run. I had to say > > python -m ensurepip > > and then I saw that a pip on a whl-package was installed. Why doesn't > the official distribution make pip ready to run by default? Thank you! When I install Python on Windows, I always get pip by default, although it might not be on the system search path. As it's recommended to use the Python Launcher py on Windows, I use that instead: py -m pip install something because it gives better support if you have multiple versions of Python installed. From grant.b.edwards at gmail.com Thu Mar 21 16:19:38 2024 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Thu, 21 Mar 2024 16:19:38 -0400 (EDT) Subject: the name ``wheel'' References: <87plvnakay.fsf@tudado.org> <6bf8d0ef-a29d-4fbf-a3da-76077aa4d988@mrabarnett.plus.com> Message-ID: <4V0xdQ6KvtznVHk@mail.python.org> On 2024-03-21, MRAB via Python-list wrote: > As it's recommended to use the Python Launcher py on Windows, I use > that instead: > > py -m pip install something > > because it gives better support if you have multiple versions of > Python installed. I adopted that practice years ago on Linux as well after wasting what seemed like most of a day trying to figure out problems which turned out to be caused by the fact that "pip" and "python" invoked different versions of Python. From list1 at tompassin.net Thu Mar 21 17:05:29 2024 From: list1 at tompassin.net (Thomas Passin) Date: Thu, 21 Mar 2024 17:05:29 -0400 Subject: the name ``wheel'' In-Reply-To: <4V0xdQ6KvtznVHk@mail.python.org> References: <87plvnakay.fsf@tudado.org> <6bf8d0ef-a29d-4fbf-a3da-76077aa4d988@mrabarnett.plus.com> <4V0xdQ6KvtznVHk@mail.python.org> Message-ID: On 3/21/2024 4:19 PM, Grant Edwards via Python-list wrote: > On 2024-03-21, MRAB via Python-list wrote: > >> As it's recommended to use the Python Launcher py on Windows, I use >> that instead: >> >> py -m pip install something >> >> because it gives better support if you have multiple versions of >> Python installed. > > I adopted that practice years ago on Linux as well after wasting what > seemed like most of a day trying to figure out problems which turned > out to be caused by the fact that "pip" and "python" invoked different > versions of Python. Although you still need to be aware that there might be a different Python installation between e.g. "python3 -m pip" and "python3.11 -m pip", etc. depending on what's been installed. From lal at solute.de Fri Mar 22 03:27:28 2024 From: lal at solute.de (Lars Liedtke) Date: Fri, 22 Mar 2024 08:27:28 +0100 Subject: Using a background thread with asyncio/futures with flask In-Reply-To: <8a7231df-c30c-4fa6-9daa-546b3453b547@thomasnyberg.com> References: <8a7231df-c30c-4fa6-9daa-546b3453b547@thomasnyberg.com> Message-ID: <09efe843-195e-48c0-a9cb-6fe05e3a6921@solute.de> Hey, As far as I know (might be old news) flask does not support asyncio. You would have to use a different framework, like e.g. FastAPI or similar. Maybe someone has already written "flask with asyncio" but I don't know about that. Cheers Lars Lars Liedtke Lead Developer [Tel.] +49 721 98993- [Fax] +49 721 98993- [E-Mail] lal at solute.de solute GmbH Zeppelinstra?e 15 76185 Karlsruhe Germany [Marken] Gesch?ftsf?hrer | Managing Director: Dr. Thilo Gans, Bernd Vermaaten Webseite | www.solute.de Sitz | Registered Office: Karlsruhe Registergericht | Register Court: Amtsgericht Mannheim Registernummer | Register No.: HRB 748044 USt-ID | VAT ID: DE234663798 Informationen zum Datenschutz | Information about privacy policy https://www.solute.de/ger/datenschutz/grundsaetze-der-datenverarbeitung.php Am 20.03.24 um 09:22 schrieb Thomas Nyberg via Python-list: Hello, I have a simple (and not working) example of what I'm trying to do. This is a simplified version of what I'm trying to achieve (obviously the background workers and finalizer functions will do more later): `app.py` ``` import asyncio import threading import time from queue import Queue from flask import Flask in_queue = Queue() out_queue = Queue() def worker(): print("worker started running") while True: future = in_queue.get() print(f"worker got future: {future}") time.sleep(5) print("worker sleeped") out_queue.put(future) def finalizer(): print("finalizer started running") while True: future = out_queue.get() print(f"finalizer got future: {future}") future.set_result("completed") print("finalizer set result") threading.Thread(target=worker, daemon=True).start() threading.Thread(target=finalizer, daemon=True).start() app = Flask(__name__) @app.route("/") async def root(): future = asyncio.get_event_loop().create_future() in_queue.put(future) print(f"root put future: {future}") result = await future return result if __name__ == "__main__": app.run() ``` If I start up that server, and execute `curl http://localhost:5000`, it prints out the following in the server before hanging: ``` $ python3 app.py worker started running finalizer started running * Serving Flask app 'app' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on http://127.0.0.1:5000 Press CTRL+C to quit root put future: worker got future: worker sleeped finalizer got future: finalizer set result ``` Judging by what's printing out, the `final result = await future` doesn't seem to be happy here. Maybe someone sees something obvious I'm doing wrong here? I presume I'm mixing threads and asyncio in a way I shouldn't be. Here's some system information (just freshly installed with pip3 install flask[async] in a virtual environment for python version 3.11.2): ``` $ uname -a Linux x1carbon 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux $ python3 -V Python 3.11.2 $ pip3 freeze asgiref==3.7.2 blinker==1.7.0 click==8.1.7 Flask==3.0.2 itsdangerous==2.1.2 Jinja2==3.1.3 MarkupSafe==2.1.5 Werkzeug==3.0.1 ``` Thanks for any help! Cheers, Thomas From rosuav at gmail.com Fri Mar 22 03:58:20 2024 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 22 Mar 2024 18:58:20 +1100 Subject: Using a background thread with asyncio/futures with flask In-Reply-To: <09efe843-195e-48c0-a9cb-6fe05e3a6921@solute.de> References: <8a7231df-c30c-4fa6-9daa-546b3453b547@thomasnyberg.com> <09efe843-195e-48c0-a9cb-6fe05e3a6921@solute.de> Message-ID: On Fri, 22 Mar 2024 at 18:35, Lars Liedtke via Python-list wrote: > > Hey, > > As far as I know (might be old news) flask does not support asyncio. > > You would have to use a different framework, like e.g. FastAPI or similar. Maybe someone has already written "flask with asyncio" but I don't know about that. > Did you try searching their documentation? https://flask.palletsprojects.com/en/3.0.x/async-await/ ChrisA From twn at thomasnyberg.com Fri Mar 22 06:08:58 2024 From: twn at thomasnyberg.com (Thomas Nyberg) Date: Fri, 22 Mar 2024 11:08:58 +0100 Subject: Using a background thread with asyncio/futures with flask In-Reply-To: <09efe843-195e-48c0-a9cb-6fe05e3a6921@solute.de> References: <8a7231df-c30c-4fa6-9daa-546b3453b547@thomasnyberg.com> <09efe843-195e-48c0-a9cb-6fe05e3a6921@solute.de> Message-ID: Hi, Yeah so flask does support async (when installed with `pip3 install flask[async]), but you are making a good point that flask in this case is a distraction. Here's an example using just the standard library that exhibits the same issue: `app.py` ``` import asyncio import threading import time from queue import Queue in_queue = Queue() out_queue = Queue() def worker(): print("worker started running") while True: future = in_queue.get() print(f"worker got future: {future}") time.sleep(5) print("worker sleeped") out_queue.put(future) def finalizer(): print("finalizer started running") while True: future = out_queue.get() print(f"finalizer got future: {future}") future.set_result("completed") print("finalizer set result") threading.Thread(target=worker).start() threading.Thread(target=finalizer).start() async def main(): future = asyncio.get_event_loop().create_future() in_queue.put(future) print(f"main put future: {future}") result = await future print(result) if __name__ == "__main__": loop = asyncio.get_event_loop() loop.run_until_complete(main()) ``` If I run that I see the following printed out (after which is just hangs): ``` $ python3 app.py worker started running finalizer started running main put future: worker got future: worker sleeped finalizer got future: finalizer set result ``` I believe async uses a cooperative multitasking setup under the hood, so I presume the way I'm doing this threading just isn't playing well with that (and presumably some csp yield isn't happening somewhere). Anyway at this point I feel like the easiest approach is to just throw away threads entirely and learn how to do all I want fully in the brave new async world, but I'm still curious why this is failing and how to make this sort of setup work since it points to my not understanding the basic implementation/semantics of async in python. Thanks for any help! /Thomas On 3/22/24 08:27, Lars Liedtke via Python-list wrote: > Hey, > > As far as I know (might be old news) flask does not support asyncio. > > You would have to use a different framework, like e.g. FastAPI or > similar. Maybe someone has already written "flask with asyncio" but I > don't know about that. > > Cheers > > Lars > > > Lars Liedtke > Lead Developer > > [Tel.]? +49 721 98993- > [Fax]?? +49 721 98993- > [E-Mail]??????? lal at solute.de > > > solute GmbH > Zeppelinstra?e 15 > 76185 Karlsruhe > Germany > > [Marken] > > Gesch?ftsf?hrer | Managing Director: Dr. Thilo Gans, Bernd Vermaaten > Webseite | www.solute.de > Sitz | Registered Office: Karlsruhe > Registergericht | Register Court: Amtsgericht Mannheim > Registernummer | Register No.: HRB 748044 > USt-ID | VAT ID: DE234663798 > > > > Informationen zum Datenschutz | Information about privacy policy > https://www.solute.de/ger/datenschutz/grundsaetze-der-datenverarbeitung.php > > > > > Am 20.03.24 um 09:22 schrieb Thomas Nyberg via Python-list: > > Hello, > > I have a simple (and not working) example of what I'm trying to do. This > is a simplified version of what I'm trying to achieve (obviously the > background workers and finalizer functions will do more later): > > `app.py` > > ``` > import asyncio > import threading > import time > from queue import Queue > > from flask import Flask > > in_queue = Queue() > out_queue = Queue() > > > def worker(): > ?? print("worker started running") > ?? while True: > ?????? future = in_queue.get() > ?????? print(f"worker got future: {future}") > ?????? time.sleep(5) > ?????? print("worker sleeped") > ?????? out_queue.put(future) > > > def finalizer(): > ?? print("finalizer started running") > ?? while True: > ?????? future = out_queue.get() > ?????? print(f"finalizer got future: {future}") > ?????? future.set_result("completed") > ?????? print("finalizer set result") > > > threading.Thread(target=worker, daemon=True).start() > threading.Thread(target=finalizer, daemon=True).start() > > app = Flask(__name__) > > > @app.route("/") > async def root(): > ?? future = asyncio.get_event_loop().create_future() > ?? in_queue.put(future) > ?? print(f"root put future: {future}") > ?? result = await future > ?? return result > > > if __name__ == "__main__": > ?? app.run() > ``` > > If I start up that server, and execute `curl http://localhost:5000`, it > prints out the following in the server before hanging: > > ``` > $ python3 app.py > worker started running > finalizer started running > * Serving Flask app 'app' > * Debug mode: off > WARNING: This is a development server. Do not use it in a production > deployment. Use a production WSGI server instead. > * Running on http://127.0.0.1:5000 > Press CTRL+C to quit > root put future: > worker got future: > worker sleeped > finalizer got future: > finalizer set result > ``` > > Judging by what's printing out, the `final result = await future` > doesn't seem to be happy here. > > Maybe someone sees something obvious I'm doing wrong here? I presume I'm > mixing threads and asyncio in a way I shouldn't be. > > Here's some system information (just freshly installed with pip3 install > flask[async] in a virtual environment for python version 3.11.2): > > ``` > $ uname -a > Linux x1carbon 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 > (2024-02-01) x86_64 GNU/Linux > > $ python3 -V > Python 3.11.2 > > $ pip3 freeze > asgiref==3.7.2 > blinker==1.7.0 > click==8.1.7 > Flask==3.0.2 > itsdangerous==2.1.2 > Jinja2==3.1.3 > MarkupSafe==2.1.5 > Werkzeug==3.0.1 > ``` > > Thanks for any help! > > Cheers, > Thomas From frank at chagford.com Fri Mar 22 06:09:10 2024 From: frank at chagford.com (Frank Millman) Date: Fri, 22 Mar 2024 12:09:10 +0200 Subject: Using a background thread with asyncio/futures with flask In-Reply-To: <8a7231df-c30c-4fa6-9daa-546b3453b547@thomasnyberg.com> References: <8a7231df-c30c-4fa6-9daa-546b3453b547@thomasnyberg.com> Message-ID: <7af73918-8209-4393-8aa2-140940567456@chagford.com> On 2024-03-20 10:22 AM, Thomas Nyberg via Python-list wrote: > > Hello, > > I have a simple (and not working) example of what I'm trying to do. This > is a simplified version of what I'm trying to achieve (obviously the > background workers and finalizer functions will do more later): > > `app.py` > > ``` > import asyncio > import threading > import time > from queue import Queue > > from flask import Flask > > in_queue = Queue() > out_queue = Queue() > > > def worker(): > ??? print("worker started running") > ??? while True: > ??????? future = in_queue.get() > ??????? print(f"worker got future: {future}") > ??????? time.sleep(5) > ??????? print("worker sleeped") > ??????? out_queue.put(future) > > > def finalizer(): > ??? print("finalizer started running") > ??? while True: > ??????? future = out_queue.get() > ??????? print(f"finalizer got future: {future}") > ??????? future.set_result("completed") > ??????? print("finalizer set result") > > > threading.Thread(target=worker, daemon=True).start() > threading.Thread(target=finalizer, daemon=True).start() > > app = Flask(__name__) > > > @app.route("/") > async def root(): > ??? future = asyncio.get_event_loop().create_future() > ??? in_queue.put(future) > ??? print(f"root put future: {future}") > ??? result = await future > ??? return result > > > if __name__ == "__main__": > ??? app.run() > ``` > > If I start up that server, and execute `curl http://localhost:5000`, it > prints out the following in the server before hanging: > > ``` > $ python3 app.py > worker started running > finalizer started running > ?* Serving Flask app 'app' > ?* Debug mode: off > WARNING: This is a development server. Do not use it in a production > deployment. Use a production WSGI server instead. > ?* Running on http://127.0.0.1:5000 > Press CTRL+C to quit > root put future: > worker got future: > worker sleeped > finalizer got future: > finalizer set result > ``` > > Judging by what's printing out, the `final result = await future` > doesn't seem to be happy here. > > Maybe someone sees something obvious I'm doing wrong here? I presume I'm > mixing threads and asyncio in a way I shouldn't be. > > Here's some system information (just freshly installed with pip3 install > flask[async] in a virtual environment for python version 3.11.2): > > ``` > $ uname -a > Linux x1carbon 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 > (2024-02-01) x86_64 GNU/Linux > > $ python3 -V > Python 3.11.2 > > $ pip3 freeze > asgiref==3.7.2 > blinker==1.7.0 > click==8.1.7 > Flask==3.0.2 > itsdangerous==2.1.2 > Jinja2==3.1.3 > MarkupSafe==2.1.5 > Werkzeug==3.0.1 > ``` > > Thanks for any help! > > Cheers, > Thomas Hi Thomas I am no expert. However, I do have something similar in my app, and it works. I do not use 'await future', I use 'asyncio.wait_for(future)'. HTH Frank Millman From lal at solute.de Fri Mar 22 06:16:29 2024 From: lal at solute.de (Lars Liedtke) Date: Fri, 22 Mar 2024 11:16:29 +0100 Subject: Using a background thread with asyncio/futures with flask In-Reply-To: References: <8a7231df-c30c-4fa6-9daa-546b3453b547@thomasnyberg.com> <09efe843-195e-48c0-a9cb-6fe05e3a6921@solute.de> Message-ID: <8e310c2b-29b7-40e9-9e5f-d7b202e77e92@solute.de> Sorry, must have missed that :-/ Lars Liedtke Lead Developer [Tel.] +49 721 98993- [Fax] +49 721 98993- [E-Mail] lal at solute.de solute GmbH Zeppelinstra?e 15 76185 Karlsruhe Germany [Marken] Gesch?ftsf?hrer | Managing Director: Dr. Thilo Gans, Bernd Vermaaten Webseite | www.solute.de Sitz | Registered Office: Karlsruhe Registergericht | Register Court: Amtsgericht Mannheim Registernummer | Register No.: HRB 748044 USt-ID | VAT ID: DE234663798 Informationen zum Datenschutz | Information about privacy policy https://www.solute.de/ger/datenschutz/grundsaetze-der-datenverarbeitung.php Am 22.03.24 um 08:58 schrieb Chris Angelico via Python-list: On Fri, 22 Mar 2024 at 18:35, Lars Liedtke via Python-list wrote: Hey, As far as I know (might be old news) flask does not support asyncio. You would have to use a different framework, like e.g. FastAPI or similar. Maybe someone has already written "flask with asyncio" but I don't know about that. Did you try searching their documentation? https://flask.palletsprojects.com/en/3.0.x/async-await/ ChrisA From frank at chagford.com Fri Mar 22 07:23:43 2024 From: frank at chagford.com (Frank Millman) Date: Fri, 22 Mar 2024 13:23:43 +0200 Subject: Using a background thread with asyncio/futures with flask In-Reply-To: <7af73918-8209-4393-8aa2-140940567456@chagford.com> References: <8a7231df-c30c-4fa6-9daa-546b3453b547@thomasnyberg.com> <7af73918-8209-4393-8aa2-140940567456@chagford.com> Message-ID: On 2024-03-22 12:09 PM, Frank Millman via Python-list wrote: > > I am no expert. However, I do have something similar in my app, and it > works. > > I do not use 'await future', I use 'asyncio.wait_for(future)'. > I tested it and it did not work. I am not sure, but I think the problem is that you have a mixture of blocking and non-blocking functions. Here is a version that works. However, it is a bit different, so I don't know if it fits your use case. I have replaced the threads with background asyncio tasks. I have replaced instances of queue.Queue with asyncio.Queue. Frank =============================================== import asyncio in_queue = asyncio.Queue() out_queue = asyncio.Queue() async def worker(): print("worker started running") while True: future = await in_queue.get() print(f"worker got future: {future}") await asyncio.sleep(5) print("worker sleeped") await out_queue.put(future) async def finalizer(): print("finalizer started running") while True: future = await out_queue.get() print(f"finalizer got future: {future}") future.set_result("completed") print("finalizer set result") async def main(): asyncio.create_task(worker()) # start a background task asyncio.create_task(finalizer()) # ditto future = asyncio.get_event_loop().create_future() await in_queue.put(future) print(f"main put future: {future}") result = await asyncio.wait_for(future, timeout=None) print(result) if __name__ == "__main__": # loop = asyncio.get_event_loop() # loop.run_until_complete(main()) # this is the preferred way to start an asyncio app asyncio.run(main()) From frank at chagford.com Fri Mar 22 07:28:56 2024 From: frank at chagford.com (Frank Millman) Date: Fri, 22 Mar 2024 13:28:56 +0200 Subject: Using a background thread with asyncio/futures with flask In-Reply-To: References: <8a7231df-c30c-4fa6-9daa-546b3453b547@thomasnyberg.com> <7af73918-8209-4393-8aa2-140940567456@chagford.com> Message-ID: On 2024-03-22 1:23 PM, Frank Millman via Python-list wrote: > On 2024-03-22 12:09 PM, Frank Millman via Python-list wrote: >> >> I am no expert. However, I do have something similar in my app, and it >> works. >> >> I do not use 'await future', I use 'asyncio.wait_for(future)'. >> > > I tested it and it did not work. > > I am not sure, but I think the problem is that you have a mixture of > blocking and non-blocking functions. > > Here is a version that works. However, it is a bit different, so I don't > know if it fits your use case. > > I have replaced the threads with background asyncio tasks. > > I have replaced instances of queue.Queue with asyncio.Queue. > > Frank > > =============================================== > > import asyncio > > in_queue = asyncio.Queue() > out_queue = asyncio.Queue() > > async def worker(): > ??? print("worker started running") > ??? while True: > ??????? future = await in_queue.get() > ??????? print(f"worker got future: {future}") > ??????? await asyncio.sleep(5) > ??????? print("worker sleeped") > ??????? await out_queue.put(future) > > async def finalizer(): > ??? print("finalizer started running") > ??? while True: > ??????? future = await out_queue.get() > ??????? print(f"finalizer got future: {future}") > ??????? future.set_result("completed") > ??????? print("finalizer set result") > > async def main(): > ??? asyncio.create_task(worker())? # start a background task > ??? asyncio.create_task(finalizer())? # ditto > ??? future = asyncio.get_event_loop().create_future() > ??? await in_queue.put(future) > ??? print(f"main put future: {future}") > ??? result = await asyncio.wait_for(future, timeout=None) > ??? print(result) > > if __name__ == "__main__": > ??? # loop = asyncio.get_event_loop() > ??? # loop.run_until_complete(main()) > > ??? # this is the preferred way to start an asyncio app > ??? asyncio.run(main()) > > One more point. If I change 'await asyncio.wait_for(future, timeout=None)' back to your original 'await future', it still works. From nntp.mbourne at spamgourmet.com Thu Mar 21 17:09:53 2024 From: nntp.mbourne at spamgourmet.com (Mark Bourne) Date: Thu, 21 Mar 2024 21:09:53 +0000 Subject: Popping key causes dict derived from object to revert to object In-Reply-To: <87zfurgb6t.fsf@zedat.fu-berlin.de> References: <87zfurgb6t.fsf@zedat.fu-berlin.de> Message-ID: Loris Bennett wrote: > Hi, > > I am using SQLAlchemy to extract some rows from a table of 'events'. > From the call to the DB I get a list of objects of the type > > sqlalchemy.orm.state.InstanceState > > I would like to print these rows to the terminal using the 'tabulate' > package, the documentation for which says > > The module provides just one function, tabulate, which takes a list of > lists or another tabular data type as the first argument, and outputs > a nicely formatted plain-text table > > So as I understand it, I need to convert the InstanceState-objects to, > say, dicts, in order to print them. However I also want to remove one > of the keys from the output and assumed I could just pop it off each > event dict, thus: > > event_dicts = [vars(e) for e in events] > print(type(event_dicts[0])) > event_dicts = [e.pop('_sa_instance_state', None) for e in event_dicts] > print(type(event_dicts[0])) vars() returns the __dict__ attribute of the object. It may not be a good idea to modify that dictionary directly (it will also affect the object), although it might be OK if you're not going to do anything else with the original objects. To be safer, you could copy the event objects: event_dicts = [dict(vars(e)) for e in events] or: event_dicts = [vars(e).copy()] > However, this prints > > > > > If I comment out the third line, which pops the unwanted key, I get > > > > > Why does popping one of the keys cause the elements of the list to > revert back to their original class? As Dieter pointed out, the main problem here is that pop() returns the value removed, not the dictionary with the rest of the values. You probably want something more like: for e in event_dicts: del e['_sa_instance_state'] (There's not really any point popping the value if you're not going to do anything with it - just delete the key from the dictionary) -- Mark. From loris.bennett at fu-berlin.de Fri Mar 22 03:58:28 2024 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Fri, 22 Mar 2024 08:58:28 +0100 Subject: Popping key causes dict derived from object to revert to object References: <87zfurgb6t.fsf@zedat.fu-berlin.de> Message-ID: <87plvmr93v.fsf@zedat.fu-berlin.de> Mark Bourne writes: > Loris Bennett wrote: >> Hi, >> I am using SQLAlchemy to extract some rows from a table of 'events'. >> From the call to the DB I get a list of objects of the type >> sqlalchemy.orm.state.InstanceState >> I would like to print these rows to the terminal using the >> 'tabulate' >> package, the documentation for which says >> The module provides just one function, tabulate, which takes a >> list of >> lists or another tabular data type as the first argument, and outputs >> a nicely formatted plain-text table >> So as I understand it, I need to convert the InstanceState-objects >> to, >> say, dicts, in order to print them. However I also want to remove one >> of the keys from the output and assumed I could just pop it off each >> event dict, thus: >> event_dicts = [vars(e) for e in events] >> print(type(event_dicts[0])) >> event_dicts = [e.pop('_sa_instance_state', None) for e in event_dicts] >> print(type(event_dicts[0])) > > vars() returns the __dict__ attribute of the object. It may not be a > good idea to modify that dictionary directly (it will also affect the > object), although it might be OK if you're not going to do anything > else with the original objects. To be safer, you could copy the event > objects: > event_dicts = [dict(vars(e)) for e in events] > or: > event_dicts = [vars(e).copy()] Thanks for making this clear to me. However, in the end I actually decided to use the list comprehension without either 'dict()' or 'vars(). Instead I just select the keys I want and so don't need to pop the unwanted key later and can simultaneously tweak the names of the key for better printing to the terminal. >> However, this prints >> >> >> If I comment out the third line, which pops the unwanted key, I get >> >> >> Why does popping one of the keys cause the elements of the list to >> revert back to their original class? > > As Dieter pointed out, the main problem here is that pop() returns the > value removed, not the dictionary with the rest of the values. You > probably want something more like: > for e in event_dicts: > del e['_sa_instance_state'] > (There's not really any point popping the value if you're not going to > do anything with it - just delete the key from the dictionary) Yes, I was mistakenly thinking that the popping the element would leave me with the dict minus the popped key-value pair. Seem like there is no such function. Cheers, Loris -- This signature is currently under constuction. From tschweikle at bfs.de Fri Mar 22 11:38:31 2024 From: tschweikle at bfs.de (Thomas Schweikle) Date: Fri, 22 Mar 2024 15:38:31 +0000 Subject: the name ``wheel'' In-Reply-To: <87frwj8o1p.fsf@tudado.org> References: <87plvnakay.fsf@tudado.org> <87frwj8o1p.fsf@tudado.org> Message-ID: <3a12cefe-2a92-4444-a800-7446164e993d@bfs.de> Am Do., 21.M?rz.2024 um 18:58:26 schrieb Johanne Fairchild via Python-list: > ram at zedat.fu-berlin.de (Stefan Ram) writes: > >> Johanne Fairchild wrote or quoted: >>> Why is a whl-package called a ``wheel''? Is it just a pronunciation for >>> the extension WHL or is it really a name? >> >> PyPi in its initial state was named "cheese shop", as the famous >> part in the show "Monty Python Cheese Shop". Because initially it >> only hosted links to the packages, so it was empty like that shop. >> And within a cheese shop what do you store? Wheels of cheese. > > Lol! Loved it. (Thanks very much.) > >>> Also, it seems that when I install Python on Windows, it doesn't come >>> with pip ready to run. I had to say >> >> Some Python distributions do not come with pip pre-installed >> because they have their own package management systems. > > But this was a Windows install. I don't think Windows has its own > package management for Python packages. I'd be totally surprised. The python packages found at https://www.python.org/downloads/ come with pip and pip installed out of the box. But there are others like active python having other packet managers they do not install pip out of the box. Refer to their instructions how to install packages. -- Thomas -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature.asc Type: application/pgp-signature Size: 321 bytes Desc: OpenPGP digital signature URL: From grant.b.edwards at gmail.com Fri Mar 22 12:01:15 2024 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Fri, 22 Mar 2024 12:01:15 -0400 (EDT) Subject: Popping key causes dict derived from object to revert to object References: <87zfurgb6t.fsf@zedat.fu-berlin.de> <87plvmr93v.fsf@zedat.fu-berlin.de> Message-ID: <4V1Rrq72BnznWJ6@mail.python.org> On 2024-03-22, Loris Bennett via Python-list wrote: > Yes, I was mistakenly thinking that the popping the element would > leave me with the dict minus the popped key-value pair. It does. > Seem like there is no such function. Yes, there is. You can do that with either pop or del: >>> d = {'a':1, 'b':2, 'c':3} >>> d {'a': 1, 'b': 2, 'c': 3} >>> d.pop('b') 2 >>> d {'a': 1, 'c': 3} >>> d = {'a':1, 'b':2, 'c':3} >>> del d['b'] >>> d {'a': 1, 'c': 3} In both cases, you're left with the dict minus the key/value pair. In the first case, the deleted value printed by the REPL because it was returned by the expression "d.pop('b')" (a method call). In the second case is no value shown by the REPL because "del d['b']" is a statement not an expression. From dieter.maurer at online.de Fri Mar 22 13:28:25 2024 From: dieter.maurer at online.de (dieter.maurer at online.de) Date: Fri, 22 Mar 2024 18:28:25 +0100 Subject: Using a background thread with asyncio/futures with flask In-Reply-To: References: <8a7231df-c30c-4fa6-9daa-546b3453b547@thomasnyberg.com> <09efe843-195e-48c0-a9cb-6fe05e3a6921@solute.de> Message-ID: <26109.49081.441135.764314@ixdm.fritz.box> Thomas Nyberg wrote at 2024-3-22 11:08 +0100: > ... `future` use across thread boundaries ... > Here's an example using just the standard library that > exhibits the same issue: I think all `asyncio` objects (futures, tasks, ...) are meant to be used in a single thread. If you use them across different threads, you must do special things. Note that an `await(future)` registers a callback at *future*. When the future gets its result or exception, the registered callback calls are scheduled via `self._loop.call_soon`. `call_soon` must be called from the `asyncio` thread (to which `self._loop` belongs). A different thread may schedule activities for a loop but it must use `call_soon_threadsafe` (not `call_soon`). I would expect that the forbidden `call_soon` call raises an exception which for reasons I do not know appears to be hidden. For use across thread boundaries, you likely will use `concurrent.Future` (not `asyncio.Future`). You can use `asyncio.futures._chain_futures` to associate an `asyncio.Future` with a `concurrent.Future`. Then the fate (result or exception set) of one will be reflected in the other. From barry at barrys-emacs.org Fri Mar 22 13:45:28 2024 From: barry at barrys-emacs.org (Barry) Date: Fri, 22 Mar 2024 17:45:28 +0000 Subject: the name ``wheel'' Message-ID: <02D89CBD-93CC-4614-AF0A-F164890B219F@barrys-emacs.org> ? > On 22 Mar 2024, at 15:25, Gilmeh Serda via Python-list wrote: > >> Many if not most Linux distributions do not include pip by default. > > Really? It came with Manjaro. Debian and Ubuntu require you to install pip as a separate package. Also puts venv in its own package. Fedora leaves all the batteries intact and rhel I assume. Barry From avi.e.gross at gmail.com Fri Mar 22 14:11:00 2024 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Fri, 22 Mar 2024 14:11:00 -0400 Subject: Popping key causes dict derived from object to revert to object In-Reply-To: <87plvmr93v.fsf@zedat.fu-berlin.de> References: <87zfurgb6t.fsf@zedat.fu-berlin.de> <87plvmr93v.fsf@zedat.fu-berlin.de> Message-ID: <005a01da7c84$4b9ba5f0$e2d2f1d0$@gmail.com> Loris wrote: "Yes, I was mistakenly thinking that the popping the element would leave me with the dict minus the popped key-value pair. Seem like there is no such function." Others have tried to explain and pointed out you can del and then use the changed dict. But consider the odd concept of writing your own trivial function. def remaining(adict, anitem): _ = adict.pop(anitem) # alternatively duse del on dict and item return adict >>> remaining({"first": 1, "second": 2, "third": 3}, "second") {'first': 1, 'third': 3} Or do you want to be able to call it as in dict.remaining(key) by subclassing your own variant of dict and adding a similar method? From dieter.maurer at online.de Fri Mar 22 15:28:31 2024 From: dieter.maurer at online.de (dieter.maurer at online.de) Date: Fri, 22 Mar 2024 20:28:31 +0100 Subject: Using a background thread with asyncio/futures with flask In-Reply-To: <26109.49081.441135.764314@ixdm.fritz.box> References: <8a7231df-c30c-4fa6-9daa-546b3453b547@thomasnyberg.com> <09efe843-195e-48c0-a9cb-6fe05e3a6921@solute.de> <26109.49081.441135.764314@ixdm.fritz.box> Message-ID: <26109.56287.684110.358617@ixdm.fritz.box> dieter.maurer at online.de wrote at 2024-3-22 18:28 +0100: >Thomas Nyberg wrote at 2024-3-22 11:08 +0100: >> ... `future` use across thread boundaries ... >> Here's an example using just the standard library that >> exhibits the same issue: > ... >For use across thread boundaries, you likely will use >`concurrent.Future` (not `asyncio.Future`). >You can use `asyncio.futures._chain_futures` to associate >an `asyncio.Future` with a `concurrent.Future`. >Then the fate (result or exception set) of one will be reflected in the other. You must not set the result/exception for a future in a "foreign" thread ("foreign" here means one not associated with the future's loop). An aternative to the solution sketched above is to set the result indirectly via `loop.call_soon_threadsafe`. This way, the result is set in the futures "native" thread. From mats at wichmann.us Fri Mar 22 16:24:34 2024 From: mats at wichmann.us (Mats Wichmann) Date: Fri, 22 Mar 2024 14:24:34 -0600 Subject: the name ``wheel'' In-Reply-To: <02D89CBD-93CC-4614-AF0A-F164890B219F@barrys-emacs.org> References: <02D89CBD-93CC-4614-AF0A-F164890B219F@barrys-emacs.org> Message-ID: On 3/22/24 11:45, Barry via Python-list wrote: > ? > >> On 22 Mar 2024, at 15:25, Gilmeh Serda via Python-list wrote: >> >>> Many if not most Linux distributions do not include pip by default. >> >> Really? It came with Manjaro. > > Debian and Ubuntu require you to install pip as a separate package. > Also puts venv in its own package. > > Fedora leaves all the batteries intact and rhel I assume. pip is still a separate package in the .rpm world. which makes sense on a couple of levels: * pip releases on its own cycle, you wouldn't want to have to *force* a new release of python + python-libs + python-devel + maybe others, if you happened want to rev pip forward independently. * in a distro-packaged world, that's the primary place you get your Python packages from, and pip isn't seen as being as necessary, and potentially even as destructive. How many times have you seen an article that suggests you "sudo pip install randompackage"? Many distro setups now disallow installing like that. If you know what you're doing, and particularly if you're happy to control a specific environment by setting up a virtualenv, then fine, you'll still have access to everything you need. anyway, I seem to recall the original message (which I've since deleted) was asking about Windows anyway. There it's quite unusual to end up without pip, but not so unusual to end up without the *command* named pip - search path things, and all that. Usually if you "py -m pip --version" you'll see it's actually installed, just not accessible using the current search path. From nntp.mbourne at spamgourmet.com Fri Mar 22 16:58:31 2024 From: nntp.mbourne at spamgourmet.com (Mark Bourne) Date: Fri, 22 Mar 2024 20:58:31 +0000 Subject: Using a background thread with asyncio/futures with flask In-Reply-To: References: <8a7231df-c30c-4fa6-9daa-546b3453b547@thomasnyberg.com> <09efe843-195e-48c0-a9cb-6fe05e3a6921@solute.de> Message-ID: Thomas Nyberg wrote: > Hi, > > Yeah so flask does support async (when installed with `pip3 install > flask[async]), but you are making a good point that flask in this case > is a distraction. Here's an example using just the standard library that > exhibits the same issue: > > `app.py` > ``` > import asyncio > import threading > import time > from queue import Queue > > > in_queue = Queue() > out_queue = Queue() > > > def worker(): > ??? print("worker started running") > ??? while True: > ??????? future = in_queue.get() > ??????? print(f"worker got future: {future}") > ??????? time.sleep(5) > ??????? print("worker sleeped") > ??????? out_queue.put(future) > > > def finalizer(): > ??? print("finalizer started running") > ??? while True: > ??????? future = out_queue.get() > ??????? print(f"finalizer got future: {future}") > ??????? future.set_result("completed") > ??????? print("finalizer set result") > > > threading.Thread(target=worker).start() > threading.Thread(target=finalizer).start() > > > async def main(): > ??? future = asyncio.get_event_loop().create_future() > ??? in_queue.put(future) > ??? print(f"main put future: {future}") > ??? result = await future > ??? print(result) > > > if __name__ == "__main__": > ??? loop = asyncio.get_event_loop() > ??? loop.run_until_complete(main()) > ``` > > If I run that I see the following printed out (after which is just hangs): > > ``` > $ python3 app.py > worker started running > finalizer started running > main put future: > worker got future: > worker sleeped > finalizer got future: > finalizer set result > ``` > > I believe async uses a cooperative multitasking setup under the hood, so > I presume the way I'm doing this threading just isn't playing well with > that (and presumably some csp yield isn't happening somewhere). Anyway > at this point I feel like the easiest approach is to just throw away > threads entirely and learn how to do all I want fully in the brave new > async world, but I'm still curious why this is failing and how to make > this sort of setup work since it points to my not understanding the > basic implementation/semantics of async in python. > > Thanks for any help! > > /Thomas > > On 3/22/24 08:27, Lars Liedtke via Python-list wrote: >> Hey, >> >> As far as I know (might be old news) flask does not support asyncio. >> >> You would have to use a different framework, like e.g. FastAPI or >> similar. Maybe someone has already written "flask with asyncio" but I >> don't know about that. >> >> Cheers >> >> Lars >> >> >> Lars Liedtke >> Lead Developer >> >> [Tel.]? +49 721 98993- >> [Fax]?? +49 721 98993- >> [E-Mail]??????? lal at solute.de >> >> >> solute GmbH >> Zeppelinstra?e 15 >> 76185 Karlsruhe >> Germany >> >> [Marken] >> >> Gesch?ftsf?hrer | Managing Director: Dr. Thilo Gans, Bernd Vermaaten >> Webseite | www.solute.de >> Sitz | Registered Office: Karlsruhe >> Registergericht | Register Court: Amtsgericht Mannheim >> Registernummer | Register No.: HRB 748044 >> USt-ID | VAT ID: DE234663798 >> >> >> >> Informationen zum Datenschutz | Information about privacy policy >> https://www.solute.de/ger/datenschutz/grundsaetze-der-datenverarbeitung.php >> >> >> >> >> >> Am 20.03.24 um 09:22 schrieb Thomas Nyberg via Python-list: >> >> Hello, >> >> I have a simple (and not working) example of what I'm trying to do. >> This is a simplified version of what I'm trying to achieve (obviously >> the background workers and finalizer functions will do more later): >> >> `app.py` >> >> ``` >> import asyncio >> import threading >> import time >> from queue import Queue >> >> from flask import Flask >> >> in_queue = Queue() >> out_queue = Queue() >> >> >> def worker(): >> ??? print("worker started running") >> ??? while True: >> ??????? future = in_queue.get() >> ??????? print(f"worker got future: {future}") >> ??????? time.sleep(5) >> ??????? print("worker sleeped") >> ??????? out_queue.put(future) >> >> >> def finalizer(): >> ??? print("finalizer started running") >> ??? while True: >> ??????? future = out_queue.get() >> ??????? print(f"finalizer got future: {future}") >> ??????? future.set_result("completed") >> ??????? print("finalizer set result") >> >> >> threading.Thread(target=worker, daemon=True).start() >> threading.Thread(target=finalizer, daemon=True).start() >> >> app = Flask(__name__) >> >> >> @app.route("/") >> async def root(): >> ??? future = asyncio.get_event_loop().create_future() >> ??? in_queue.put(future) >> ??? print(f"root put future: {future}") >> ??? result = await future >> ??? return result >> >> >> if __name__ == "__main__": >> ??? app.run() >> ``` >> >> If I start up that server, and execute `curl http://localhost:5000`, >> it prints out the following in the server before hanging: >> >> ``` >> $ python3 app.py >> worker started running >> finalizer started running >> * Serving Flask app 'app' >> * Debug mode: off >> WARNING: This is a development server. Do not use it in a production >> deployment. Use a production WSGI server instead. >> * Running on http://127.0.0.1:5000 >> Press CTRL+C to quit >> root put future: >> worker got future: >> worker sleeped >> finalizer got future: >> finalizer set result >> ``` >> >> Judging by what's printing out, the `final result = await future` >> doesn't seem to be happy here. >> >> Maybe someone sees something obvious I'm doing wrong here? I presume >> I'm mixing threads and asyncio in a way I shouldn't be. Aside from possible issues mixing threads and asyncio (I'm no expert on asyncio), there's also the issue that there's nothing to cause the threads to exit. The following doesn't use asyncio, but also hangs after the main thread has got the result: ``` import queue import threading import time in_queue = queue.Queue() out_queue = queue.Queue() result_queue = queue.Queue() def worker(): print("worker started running") while True: item = in_queue.get() print(f"worker got item: {item}") time.sleep(5) print("worker sleeped") out_queue.put(item) def finalizer(): print("finalizer started running") while True: item = out_queue.get() print(f"finalizer got item: {item}") result_queue.put(item) print("finalizer set result") threading.Thread(target=worker).start() threading.Thread(target=finalizer).start() # threading.Thread(target=worker, daemon=True).start() # threading.Thread(target=finalizer, daemon=True).start() def main(): item = "Item to process" in_queue.put("Item to process") print(f"main put item: {item}") result = None while True: try: result = result_queue.get(timeout=1) except queue.Empty: # No result yet print("main waiting for result") continue break print(f"main got result {result}") if __name__ == "__main__": main() ``` By default, the main process won't exit until there are no non-daemon threads still running. You can either send some sort of signal to the threads signal the threads to exit the loop and return cleanly (you'd also need a timeout on the queue `get()` calls). Or you can create the threads as "daemon" threads (as in the commented-out lines), in which case they'll be killed when all non-daemon threads have exited. Daemon threads don't get a chance to do any cleanup, close resources, etc. when they're killed, though, so aren't always appropriate. -- Mark. From frank at chagford.com Sat Mar 23 09:25:28 2024 From: frank at chagford.com (Frank Millman) Date: Sat, 23 Mar 2024 15:25:28 +0200 Subject: Using a background thread with asyncio/futures with flask In-Reply-To: References: <8a7231df-c30c-4fa6-9daa-546b3453b547@thomasnyberg.com> <09efe843-195e-48c0-a9cb-6fe05e3a6921@solute.de> Message-ID: <1322c7e8-2153-400d-afc5-7f2993031c08@chagford.com> On 2024-03-22 12:08 PM, Thomas Nyberg via Python-list wrote: > Hi, > > Yeah so flask does support async (when installed with `pip3 install > flask[async]), but you are making a good point that flask in this case > is a distraction. Here's an example using just the standard library that > exhibits the same issue: > > `app.py` > ``` > import asyncio > import threading > import time > from queue import Queue > > > in_queue = Queue() > out_queue = Queue() > > > def worker(): > ??? print("worker started running") > ??? while True: > ??????? future = in_queue.get() > ??????? print(f"worker got future: {future}") > ??????? time.sleep(5) > ??????? print("worker sleeped") > ??????? out_queue.put(future) > > > def finalizer(): > ??? print("finalizer started running") > ??? while True: > ??????? future = out_queue.get() > ??????? print(f"finalizer got future: {future}") > ??????? future.set_result("completed") > ??????? print("finalizer set result") > > > threading.Thread(target=worker).start() > threading.Thread(target=finalizer).start() > > > async def main(): > ??? future = asyncio.get_event_loop().create_future() > ??? in_queue.put(future) > ??? print(f"main put future: {future}") > ??? result = await future > ??? print(result) > > > if __name__ == "__main__": > ??? loop = asyncio.get_event_loop() > ??? loop.run_until_complete(main()) > ``` > > If I run that I see the following printed out (after which is just hangs): > > ``` Combining Dieter's and Mark's ideas, here is a version that works. It is not pretty! call_soon_threadsafe() is a loop function, but the loop is not accessible from a different thread. Therefore I include a reference to the loop in the message passed to in_queue, which in turn passes it to out_queue. Frank ======================================================= import asyncio import threading import time from queue import Queue in_queue = Queue() out_queue = Queue() def worker(): print("worker started running") while True: loop, future = in_queue.get() print(f"worker got future: {future}") time.sleep(5) print("worker sleeped") out_queue.put((loop, future)) def finalizer(): print("finalizer started running") while True: loop, future = out_queue.get() print(f"finalizer got future: {future}") loop.call_soon_threadsafe(future.set_result, "completed") print("finalizer set result") threading.Thread(target=worker, daemon=True).start() threading.Thread(target=finalizer, daemon=True).start() async def main(): loop = asyncio.get_event_loop() future = loop.create_future() in_queue.put((loop, future)) print(f"main put future: {future}") result = await future print(result) if __name__ == "__main__": # loop = asyncio.get_event_loop() # loop.run_until_complete(main()) asyncio.run(main()) From frank at chagford.com Sun Mar 24 03:34:06 2024 From: frank at chagford.com (Frank Millman) Date: Sun, 24 Mar 2024 09:34:06 +0200 Subject: Using a background thread with asyncio/futures with flask In-Reply-To: <1322c7e8-2153-400d-afc5-7f2993031c08@chagford.com> References: <8a7231df-c30c-4fa6-9daa-546b3453b547@thomasnyberg.com> <09efe843-195e-48c0-a9cb-6fe05e3a6921@solute.de> <1322c7e8-2153-400d-afc5-7f2993031c08@chagford.com> Message-ID: On 2024-03-23 3:25 PM, Frank Millman via Python-list wrote: > > It is not pretty! call_soon_threadsafe() is a loop function, but the > loop is not accessible from a different thread. Therefore I include a > reference to the loop in the message passed to in_queue, which in turn > passes it to out_queue. > I found that you can retrieve the loop from the future using future.get_loop(), so the above is not necessary. Frank From barry at barrys-emacs.org Sun Mar 24 13:09:39 2024 From: barry at barrys-emacs.org (Barry) Date: Sun, 24 Mar 2024 17:09:39 +0000 Subject: the name ``wheel'' In-Reply-To: References: Message-ID: <2BEA61E4-C6B9-4725-93EE-693462762DBE@barrys-emacs.org> > On 22 Mar 2024, at 20:28, Mats Wichmann via Python-list wrote: > > pip is still a separate package in the .rpm world. which makes sense on a couple of levels: Yes it?s a separate package, but it?s always installed. At least on Fedora. I agree it makes sense to package it separately, which is inline with Fedora?s policy of not vendoring code into a package. Barry From max at alcyone.com Mon Mar 25 00:10:17 2024 From: max at alcyone.com (Erik Max Francis) Date: Sun, 24 Mar 2024 21:10:17 -0700 Subject: ANN: EmPy 4.1 -- a powerful, robust and mature templating system for Python Message-ID: <9700aa35-1f1f-47d3-b0b9-98a4aff41099@alcyone.com> I'm pleased to announce the release of EmPy 4.1. The 4._x_ series is a modernization of the software and a revamp of the EmPy system to update its feature set and make it more consistent with the latest Python versions and practices. EmPy 4._x_ was also relicensed to BSD. The 4._x_ series adds new markups, including inline comments, backquote literals, chained if-then-else extended expression, functional expressions, support for modern Pythonic controls, stringized and multiline significators, named escapes, diacritics, icons, and emojis. It adds support for configuration objects (replacing options dictionaries); native support for Unicode, file buffering, reference counted `sys.stdout` proxies and error dispatchers and handlers; fixes several serious bugs; has a set of full unit and system tests, an extensive builtin help system; and the online documention has been rewritten and expanded. Attempts have been made to make EmPy 4._x_ as backward compatible as is practical. Most common markup has not changed; the only changes being removal of `repr` (in favor of backquote literals) as well as literal close parenthesis, bracket and brace markup; in-place markup has changed syntax (to make way for emojis); and custom markup is now parsed more sensibly. Most backward-incompatible changes are in the embedding interface. The `Interpreter` constructor and global `expand` function now require keyword arguments to prevent further backward compatibility problems, though effort has been made to make the behavior as backward compatible as possible. The supported environment variables have changed, as well as the filter, diversion and hook APIs, and options dictionaries no longer exist (in deference to configurations). For a comprehensive list of changes from 3._x_ to 4._x_, see: ## Introduction: Welcome to EmPy! [EmPy](http://www.alcyone.com/software/empy/) is a powerful, robust and mature templating system for inserting Python code in template text. EmPy takes a source document, processes it, and produces output. This is accomplished via expansions, which are signals to the EmPy system where to act and are indicated with markup. Markup is set off by a customizable prefix (by default the at sign, `@`). EmPy can expand arbitrary Python expressions, statements and control structures in this way, as well as a variety of additional special forms. The remaining textual data is sent to the output, allowing Python to be used in effect as a markup language. EmPy also supports hooks, which can intercept and modify the behavior of a running interpreter; diversions, which allow recording and playback; filters, which are dynamic and can be chained together; and a dedicated user-customizable callback markup. The system is highly configurable via command line options, configuration files, and environment variables. An extensive API is also available for embedding EmPy functionality in your own Python programs. EmPy also has a supplemental library for additional non-essential features (`emlib`), a documentation building library used to create this documentation (`emdoc`), and an extensive help system (`emhelp`) which can be queried from the command line with the main executable `em.py` (`-h/--help`, `-H/--topics=TOPICS`). The base EmPy interpreter can function with only the `em.py`/`em` file/module available. EmPy can be used in a variety of roles, including as a templating system, a text processing system (preprocessing and/or postprocessing), a simple macro processor, a frontend for a content management system, annotating documents, for literate programming, as a souped-up text encoding converter, a text beautifier (with macros and filters), and many other purposes. ### Markup overview Expressions are embedded in text with the `@(...)` notation; variations include conditional expressions with `@(...?...!...)` and the ability to handle thrown exceptions with `@(...$...)`. As a shortcut, simple variables and expressions can be abbreviated as `@variable`, `@object.attribute`, `@sequence[index]`, `@function(arguments...)`, `@function{markup}{...}` and combinations. Full-fledged statements are embedded with `@{...}`. Control flow in terms of conditional or repeated expansion is available with `@[...]`. A `@` followed by any whitespace character (including a newline) expands to nothing, allowing string concatenations and line continuations. Line comments are indicated with `@#...` including the trailing newline. `@*...*` allows inline comments. Escapes are indicated with `@\...`; diacritics with `@^...`; icons with `@|...`; and emoji with `@:...:`. `@%...`, `@%!...`, `@%%...%%` and `@%%!...%%` indicate "significators," which are distinctive forms of variable assignment intended to specify per-document identification information in a format easy to parse externally, _e.g._, to indicate metadata. In-place expressions are specified with `@$...$...$`. Context name and line number changes can be made with `@?...` and `@!...`, respectively. `@<...>` markup is customizable by the user and can be used for any desired purpose. `` @`...` `` allows literal escaping of any EmPy markup. And finally, a `@@` sequence (the prefix repeated once) expands to a single literal at sign. The prefix defaults to `@` but can be changed with the command line option `-p/--prefix=CHAR` (_environment variable:_ `EMPY_PREFIX`, _configuration variable:_ `prefix`). ### Getting the software The current version of EmPy is 4.1. The official URL for this Web site is . The latest version of the software is available in a tarball here: . The software can be installed through PIP via this shell command: ``` % python3 -m pip install empy ``` For information about upgrading from 3._x_ to 4._x_, see . ### Requirements EmPy works with any modern version of Python. Python version 3._x_ is expected to be the default and all source file references to the Python interpreter (_e.g._, the bangpath of the .py scripts) use `python3`. EmPy also has legacy support for versions of Python going back all the way to 2.3, with special emphasis on 2.7 regardless of its end-of-life status. It has no dependency requirements on any third-party modules and can run directly off of a stock Python interpreter. EmPy will run on any operating system with a full-featured Python interpreter; this includes, but is probably not limited to, Linux, Windows, and macOS (Darwin). Using EmPy requires knowledge of the [Python language](https://www.python.org/). EmPy is also compatible with several different Python implementations: | Implementation | Supported versions | Description | | --- | --- | --- | | CPython | 2.3 to 2.7; 3.0 and up | Standard implementation in C | | PyPy | 2.7; 3.2 and up | Implementation with just-in-time compiler | | IronPython | 2.7; 3.4 and up | Implementation for .NET CLR and Mono | | Jython | 2.7 (and up?) | Implementation for JVM | It's probable that EmPy is compatible with earlier versions than those listed here (potentially going all the way back to 2.3), but this has not been tested. Only a few .py module file(s) are needed to use EmPy; they can be installed system-wide through a distribution package, a third-party module/executable, or just dropped into any desired directory in the `PYTHONPATH`. A minimal installation need only install the em.py file, either as an importable module and an executable, or both, depending on the user's needs. EmPy also has optional support for several [third-party modules](#third-party-emoji-modules); see [Emoji markup](#emoji-markup) for details. The testing system included (the test.sh script and the tests and suites directories) is intended to run on Unix-like systems with a Bourne-like shell (_e.g._, sh, bash, zsh, etc.). EmPy is routinely tested with all supported versions of all available interpreters. If you find an incompatibility with your Python interpreter or operating system, [let me know](#reporting-bugs). ### License This software is licensed under [BSD (3-Clause)](https://opensource.org/licenses/bsd-3-clause/). ### Recent release history (since 3._x_) 4.1 (2024 Mar 24) : Add support for extension markup `@((...))`, `@[[...]]`, `@{{...}}`, `@<...>`, etc., with custom callbacks retained for backward compatibility; add `@[match]` control support; add interpreter cores for overriding interpreter behavior; add more command line option toggles; add notion of verbose/brief errors; more uniform error message formatting; various documentation updates. 4.0.1 (2023 Dec 24) : Add root context argument, serializers, and idents to interpreter; fix `setContext...` methods so they also modify the currents stack; better backward compatibility for `expand` function and `CompatibilityError`; fix inconsistent stack usage with `expand` method; add error dispatchers, cleaner error handling and `ignoreErrors`; have `expand` method/function raise exceptions to caller; eliminate need for `FullContext` class distinct from `Context`; support comments in "clean" controls; add `--no-none-symbol` option; add clearer errors for removed literal markup; add `Container` support class in `emlib`; hide non-standard proxy attributes and methods; support string errors (why not); update and expand tests; help subsystem and documentation updates. 4.0 (2023 Nov 29) : A major revamp, refresh, and modernization. Major new features include inline comments `@*...*`; backquote literals `` @`...` ``; chained if-then-else expressions; functional expressions `@f{...}`; full support for `@[try]`, `@[while ...]` and `@[with ...]` control markup; `@[defined ...]` control markup; stringized and multiline significators; named escapes `@\^{...}`; diacritics `@^...`; icons `@|...`; emojis `@:...:`; configurations; full Unicode and file buffering support; proxy now reference counted; hooks can override behavior; many bug fixes; an extensive builtin help system (`emhelp`); and rewritten and expanded documentation in addition to a dedicated module (`emdoc`). Changes include relicensing to BSD, interpreter constructor now requires keyword arguments, `-d/--delete-on-error` instead of "fully buffered files"; cleaned up environment variables; "repr" markup replaced with emoji markup; remove literal markups `@)`, `@]`, `@}`; context line markup `@!...` no longer pre-adjusts line; custom markup `@<...>` now parsed more sensibly; filter shortcuts removed; context now track column and character count; auxiliary classes moved to `emlib` module; use `argv` instead of `argc` for interpreter arguments. See [Full list of changes between EmPy 3._x_ and 4._x_](http://www.alcyone.com/software/empy/ANNOUNCE.html#all-changes) for a more comprehensive list. -- Erik Max Francis && max at alcyone.com && http://www.alcyone.com/max/ San Jose, CA, USA && 37 18 N 121 57 W && Skype erikmaxfrancis Maybe this world is another planet's Hell. -- Aldous Huxley From loris.bennett at fu-berlin.de Mon Mar 25 02:45:15 2024 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Mon, 25 Mar 2024 07:45:15 +0100 Subject: Popping key causes dict derived from object to revert to object In-Reply-To: <005a01da7c84$4b9ba5f0$e2d2f1d0$@gmail.com> (avi e. gross's message of "Fri, 22 Mar 2024 14:11:00 -0400") References: <87zfurgb6t.fsf@zedat.fu-berlin.de> <87plvmr93v.fsf@zedat.fu-berlin.de> <005a01da7c84$4b9ba5f0$e2d2f1d0$@gmail.com> Message-ID: <874jcustc4.fsf@zedat.fu-berlin.de> writes: > Loris wrote: > > "Yes, I was mistakenly thinking that the popping the element would leave > me with the dict minus the popped key-value pair. Seem like there is no > such function." > > Others have tried to explain and pointed out you can del and then use the > changed dict. > > But consider the odd concept of writing your own trivial function. > > def remaining(adict, anitem): > _ = adict.pop(anitem) > # alternatively duse del on dict and item > return adict > > >>>> remaining({"first": 1, "second": 2, "third": 3}, "second") > {'first': 1, 'third': 3} > > > Or do you want to be able to call it as in dict.remaining(key) by > subclassing your own variant of dict and adding a similar method? No, 'del' does indeed do what I wanted, although I have now decided I want something else :-) Nevertheless it is good to know that 'del' exists, so that I don't have to reinvent it. Cheers, Loris -- This signature is currently under constuction. From barry at barrys-emacs.org Mon Mar 25 06:17:26 2024 From: barry at barrys-emacs.org (Barry) Date: Mon, 25 Mar 2024 10:17:26 +0000 Subject: ANN: EmPy 4.1 -- a powerful, robust and mature templating system for Python In-Reply-To: <9700aa35-1f1f-47d3-b0b9-98a4aff41099@alcyone.com> References: <9700aa35-1f1f-47d3-b0b9-98a4aff41099@alcyone.com> Message-ID: <4CD5260B-EE48-441C-9672-9F6E08B15A53@barrys-emacs.org> You should considered also announcing on https://discuss.python.org/ which is a lot more active then this list. Barry > On 25 Mar 2024, at 04:13, Erik Max Francis via Python-list wrote: > > ?I'm pleased to announce the release of EmPy 4.1. > > The 4._x_ series is a modernization of the software and a revamp of > the EmPy system to update its feature set and make it more consistent > with the latest Python versions and practices. EmPy 4._x_ was also > relicensed to BSD. > > The 4._x_ series adds new markups, including inline comments, > backquote literals, chained if-then-else extended expression, > functional expressions, support for modern Pythonic controls, > stringized and multiline significators, named escapes, diacritics, > icons, and emojis. > > It adds support for configuration objects (replacing options > dictionaries); native support for Unicode, file buffering, reference > counted `sys.stdout` proxies and error dispatchers and handlers; fixes > several serious bugs; has a set of full unit and system tests, an > extensive builtin help system; and the online documention has been > rewritten and expanded. > > Attempts have been made to make EmPy 4._x_ as backward compatible as > is practical. Most common markup has not changed; the only changes > being removal of `repr` (in favor of backquote literals) as well as > literal close parenthesis, bracket and brace markup; in-place markup > has changed syntax (to make way for emojis); and custom markup is now > parsed more sensibly. > > Most backward-incompatible changes are in the embedding interface. > The `Interpreter` constructor and global `expand` function now require > keyword arguments to prevent further backward compatibility problems, > though effort has been made to make the behavior as backward > compatible as possible. The supported environment variables have > changed, as well as the filter, diversion and hook APIs, and options > dictionaries no longer exist (in deference to configurations). > > For a comprehensive list of changes from 3._x_ to 4._x_, see: > > > > ## Introduction: Welcome to EmPy! > > [EmPy](http://www.alcyone.com/software/empy/) is a powerful, robust and mature > templating system for inserting Python code in template text. EmPy > takes a source document, processes it, and produces output. This is > accomplished via expansions, which are signals to the EmPy system > where to act and are indicated with markup. Markup is set off by a > customizable prefix (by default the at sign, `@`). EmPy can expand > arbitrary Python expressions, statements and control structures in > this way, as well as a variety of additional special forms. The > remaining textual data is sent to the output, allowing Python to be > used in effect as a markup language. > > EmPy also supports hooks, which can intercept and modify the behavior > of a running interpreter; diversions, which allow recording and > playback; filters, which are dynamic and can be chained together; and > a dedicated user-customizable callback markup. The system is highly > configurable via command line options, configuration files, and > environment variables. An extensive API is also available for > embedding EmPy functionality in your own Python programs. > > EmPy also has a supplemental library for additional non-essential > features (`emlib`), a documentation building library used to create > this documentation (`emdoc`), and an extensive help system (`emhelp`) > which can be queried from the command line with the main executable > `em.py` (`-h/--help`, `-H/--topics=TOPICS`). The base EmPy > interpreter can function with only the `em.py`/`em` file/module > available. > > EmPy can be used in a variety of roles, including as a templating > system, a text processing system (preprocessing and/or > postprocessing), a simple macro processor, a frontend for a content > management system, annotating documents, for literate programming, as > a souped-up text encoding converter, a text beautifier (with macros > and filters), and many other purposes. > > > ### Markup overview > > Expressions are embedded in text with the `@(...)` notation; > variations include conditional expressions with `@(...?...!...)` and > the ability to handle thrown exceptions with `@(...$...)`. As a > shortcut, simple variables and expressions can be abbreviated as > `@variable`, `@object.attribute`, `@sequence[index]`, > `@function(arguments...)`, `@function{markup}{...}` and > combinations. Full-fledged statements are embedded with `@{...}`. > Control flow in terms of conditional or repeated expansion is > available with `@[...]`. A `@` followed by any whitespace character > (including a newline) expands to nothing, allowing string > concatenations and line continuations. Line comments are indicated > with `@#...` including the trailing newline. `@*...*` allows inline > comments. Escapes are indicated with `@\...`; diacritics with > `@^...`; icons with `@|...`; and emoji with `@:...:`. `@%...`, > `@%!...`, `@%%...%%` and `@%%!...%%` indicate "significators," > which are distinctive forms of variable assignment intended to specify > per-document identification information in a format easy to parse > externally, _e.g._, to indicate metadata. In-place expressions are > specified with `@$...$...$`. Context name and line number changes > can be made with `@?...` and `@!...`, respectively. `@<...>` > markup is customizable by the user and can be used for any desired > purpose. `` @`...` `` allows literal escaping of any EmPy markup. > And finally, a `@@` sequence (the prefix repeated once) expands to a > single literal at sign. > > The prefix defaults to `@` but can be changed with > the command line option `-p/--prefix=CHAR` (_environment variable:_ `EMPY_PREFIX`, _configuration variable:_ `prefix`). > > > ### Getting the software > > The current version of EmPy is 4.1. > > The official URL for this Web site is . > > The latest version of the software is available in a tarball here: > . > > The software can be installed through PIP via this shell command: > > ``` > % python3 -m pip install empy > ``` > > > For information about upgrading from 3._x_ to 4._x_, see > . > > > ### Requirements > > EmPy works with any modern version of Python. Python version 3._x_ is > expected to be the default and all source file references to the > Python interpreter (_e.g._, the bangpath of the .py scripts) use > `python3`. EmPy also has legacy support for versions of Python going > back all the way to 2.3, with special emphasis on 2.7 regardless of > its end-of-life status. It has no dependency requirements on any > third-party modules and can run directly off of a stock Python > interpreter. > > EmPy will run on any operating system with a full-featured Python > interpreter; this includes, but is probably not limited to, Linux, > Windows, and macOS (Darwin). Using EmPy requires knowledge of the > [Python language](https://www.python.org/). > > EmPy is also compatible with several different Python implementations: > > | Implementation | Supported versions | Description | > | --- | --- | --- | > | CPython | 2.3 to 2.7; 3.0 and up | Standard implementation in C | > | PyPy | 2.7; 3.2 and up | Implementation with just-in-time compiler | > | IronPython | 2.7; 3.4 and up | Implementation for .NET CLR and Mono | > | Jython | 2.7 (and up?) | Implementation for JVM | > > It's probable that EmPy is compatible with earlier versions than those > listed here (potentially going all the way back to 2.3), but this has > not been tested. > > Only a few .py module file(s) are needed to use EmPy; they can be > installed system-wide through a distribution package, a third-party > module/executable, or just dropped into any desired directory in the > `PYTHONPATH`. A minimal installation need only install the em.py > file, either as an importable module and an executable, or both, > depending on the user's needs. > > EmPy also has optional support for several [third-party > modules](#third-party-emoji-modules); see [Emoji > markup](#emoji-markup) for details. > > The testing system included (the test.sh script and the tests and > suites directories) is intended to run on Unix-like systems with a > Bourne-like shell (_e.g._, sh, bash, zsh, etc.). EmPy is routinely > tested with all supported versions of all available interpreters. > > If you find an incompatibility with your Python interpreter or > operating system, [let me know](#reporting-bugs). > > > ### License > > This software is licensed under > [BSD (3-Clause)](https://opensource.org/licenses/bsd-3-clause/). > > > ### Recent release history (since 3._x_) > > 4.1 (2024 Mar 24) > > : Add support for extension markup `@((...))`, `@[[...]]`, > `@{{...}}`, `@<...>`, etc., with custom callbacks retained for > backward compatibility; add `@[match]` control support; add > interpreter cores for overriding interpreter behavior; add more > command line option toggles; add notion of verbose/brief errors; > more uniform error message formatting; various documentation > updates. > > 4.0.1 (2023 Dec 24) > > : Add root context argument, serializers, and idents to interpreter; > fix `setContext...` methods so they also modify the currents stack; > better backward compatibility for `expand` function and > `CompatibilityError`; fix inconsistent stack usage with `expand` > method; add error dispatchers, cleaner error handling and > `ignoreErrors`; have `expand` method/function raise > exceptions to caller; eliminate need for `FullContext` class > distinct from `Context`; support comments in "clean" controls; add > `--no-none-symbol` option; add clearer errors for removed literal > markup; add `Container` support class in `emlib`; hide non-standard > proxy attributes and methods; support string errors (why not); > update and expand tests; help subsystem and documentation updates. > > 4.0 (2023 Nov 29) > > : A major revamp, refresh, and modernization. Major new features > include inline comments `@*...*`; backquote literals `` @`...` > ``; chained if-then-else expressions; functional expressions > `@f{...}`; full support for `@[try]`, `@[while ...]` and `@[with > ...]` control markup; `@[defined ...]` control markup; stringized > and multiline significators; named escapes `@\^{...}`; diacritics > `@^...`; icons `@|...`; emojis `@:...:`; configurations; full > Unicode and file buffering support; proxy now reference counted; > hooks can override behavior; many bug fixes; an extensive builtin > help system (`emhelp`); and rewritten and expanded documentation in > addition to a dedicated module (`emdoc`). Changes include > relicensing to BSD, interpreter constructor now requires keyword > arguments, `-d/--delete-on-error` instead of "fully buffered files"; > cleaned up environment variables; "repr" markup replaced with emoji > markup; remove literal markups `@)`, `@]`, `@}`; context line > markup `@!...` no longer pre-adjusts line; custom markup `@<...>` > now parsed more sensibly; filter shortcuts removed; context now > track column and character count; auxiliary classes moved to `emlib` > module; use `argv` instead of `argc` for interpreter arguments. See > [Full list of changes between EmPy 3._x_ and > 4._x_](http://www.alcyone.com/software/empy/ANNOUNCE.html#all-changes) for a more > comprehensive list. > > -- > Erik Max Francis && max at alcyone.com && http://www.alcyone.com/max/ > San Jose, CA, USA && 37 18 N 121 57 W && Skype erikmaxfrancis > Maybe this world is another planet's Hell. > -- Aldous Huxley > -- > https://mail.python.org/mailman/listinfo/python-list > From loris.bennett at fu-berlin.de Mon Mar 25 02:56:16 2024 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Mon, 25 Mar 2024 07:56:16 +0100 Subject: Popping key causes dict derived from object to revert to object References: <87zfurgb6t.fsf@zedat.fu-berlin.de> <87plvmr93v.fsf@zedat.fu-berlin.de> <4V1Rrq72BnznWJ6@mail.python.org> Message-ID: <87zfumre9b.fsf@zedat.fu-berlin.de> Grant Edwards writes: > On 2024-03-22, Loris Bennett via Python-list wrote: > >> Yes, I was mistakenly thinking that the popping the element would >> leave me with the dict minus the popped key-value pair. > > It does. Indeed, but I was thinking in the context of dict_list = [d.pop('a') for d in dict_list] and incorrectly expecting to get a list of 'd' without key 'a', instead of a list of the 'd['a]'. >> Seem like there is no such function. > > Yes, there is. You can do that with either pop or del: > > >>> d = {'a':1, 'b':2, 'c':3} > >>> d > {'a': 1, 'b': 2, 'c': 3} > >>> d.pop('b') > 2 > >>> d > {'a': 1, 'c': 3} > > > >>> d = {'a':1, 'b':2, 'c':3} > >>> del d['b'] > >>> d > {'a': 1, 'c': 3} > > In both cases, you're left with the dict minus the key/value pair. > > In the first case, the deleted value printed by the REPL because it > was returned by the expression "d.pop('b')" (a method call). > > In the second case is no value shown by the REPL because "del d['b']" > is a statement not an expression. Thanks for pointing out 'del'. My main problem, however, was failing to realise that the list comprehension is populated by the return value of the 'pop', not the popped dict. Cheers, Loris -- This signature is currently under constuction. From michael.stemper at gmail.com Mon Mar 25 09:52:36 2024 From: michael.stemper at gmail.com (Michael F. Stemper) Date: Mon, 25 Mar 2024 08:52:36 -0500 Subject: Popping key causes dict derived from object to revert to object In-Reply-To: <87zfumre9b.fsf@zedat.fu-berlin.de> References: <87zfurgb6t.fsf@zedat.fu-berlin.de> <87plvmr93v.fsf@zedat.fu-berlin.de> <4V1Rrq72BnznWJ6@mail.python.org> <87zfumre9b.fsf@zedat.fu-berlin.de> Message-ID: On 25/03/2024 01.56, Loris Bennett wrote: > Grant Edwards writes: > >> On 2024-03-22, Loris Bennett via Python-list wrote: >> >>> Yes, I was mistakenly thinking that the popping the element would >>> leave me with the dict minus the popped key-value pair. >> >> It does. > > Indeed, but I was thinking in the context of > > dict_list = [d.pop('a') for d in dict_list] > > and incorrectly expecting to get a list of 'd' without key 'a', instead > of a list of the 'd['a]'. I apologize if this has already been mentioned in this thread, but are you aware of "d.keys()" and "d.values"? >>> d = {} >>> d['do'] = 'a deer, a female deer' >>> d['re'] = 'a drop of golden sunshine' >>> d['mi'] = 'a name I call myself' >>> d['fa'] = 'a long, long way to run' >>> d.keys() ['fa', 'mi', 'do', 're'] >>> d.values() ['a long, long way to run', 'a name I call myself', 'a deer, a female deer', 'a drop of golden sunshine'] >>> -- Michael F. Stemper Exodus 22:21 From jon+usenet at unequivocal.eu Mon Mar 25 11:19:17 2024 From: jon+usenet at unequivocal.eu (Jon Ribbens) Date: Mon, 25 Mar 2024 15:19:17 -0000 (UTC) Subject: Popping key causes dict derived from object to revert to object References: <87zfurgb6t.fsf@zedat.fu-berlin.de> <87plvmr93v.fsf@zedat.fu-berlin.de> <4V1Rrq72BnznWJ6@mail.python.org> <87zfumre9b.fsf@zedat.fu-berlin.de> <87le66pdcw.fsf@zedat.fu-berlin.de> Message-ID: On 2024-03-25, Loris Bennett wrote: > "Michael F. Stemper" writes: > >> On 25/03/2024 01.56, Loris Bennett wrote: >>> Grant Edwards writes: >>> >>>> On 2024-03-22, Loris Bennett via Python-list wrote: >>>> >>>>> Yes, I was mistakenly thinking that the popping the element would >>>>> leave me with the dict minus the popped key-value pair. >>>> >>>> It does. >>> Indeed, but I was thinking in the context of >>> dict_list = [d.pop('a') for d in dict_list] >>> and incorrectly expecting to get a list of 'd' without key 'a', >>> instead >>> of a list of the 'd['a]'. >> I apologize if this has already been mentioned in this thread, but are >> you aware of "d.keys()" and "d.values"? >> >> >>> d = {} >> >>> d['do'] = 'a deer, a female deer' >> >>> d['re'] = 'a drop of golden sunshine' >> >>> d['mi'] = 'a name I call myself' >> >>> d['fa'] = 'a long, long way to run' >> >>> d.keys() >> ['fa', 'mi', 'do', 're'] >> >>> d.values() >> ['a long, long way to run', 'a name I call myself', 'a deer, a female deer', 'a drop of golden sunshine'] >> >>> > > Yes, I am, thank you. However, I didn't want either the keys or the > values. Instead I wanted to remove a key within a list comprehension. Do you mean something like: [my_dict[key] for key in my_dict if key != 'a'] ? From loris.bennett at fu-berlin.de Mon Mar 25 10:58:39 2024 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Mon, 25 Mar 2024 15:58:39 +0100 Subject: Popping key causes dict derived from object to revert to object References: <87zfurgb6t.fsf@zedat.fu-berlin.de> <87plvmr93v.fsf@zedat.fu-berlin.de> <4V1Rrq72BnznWJ6@mail.python.org> <87zfumre9b.fsf@zedat.fu-berlin.de> Message-ID: <87le66pdcw.fsf@zedat.fu-berlin.de> "Michael F. Stemper" writes: > On 25/03/2024 01.56, Loris Bennett wrote: >> Grant Edwards writes: >> >>> On 2024-03-22, Loris Bennett via Python-list wrote: >>> >>>> Yes, I was mistakenly thinking that the popping the element would >>>> leave me with the dict minus the popped key-value pair. >>> >>> It does. >> Indeed, but I was thinking in the context of >> dict_list = [d.pop('a') for d in dict_list] >> and incorrectly expecting to get a list of 'd' without key 'a', >> instead >> of a list of the 'd['a]'. > I apologize if this has already been mentioned in this thread, but are > you aware of "d.keys()" and "d.values"? > > >>> d = {} > >>> d['do'] = 'a deer, a female deer' > >>> d['re'] = 'a drop of golden sunshine' > >>> d['mi'] = 'a name I call myself' > >>> d['fa'] = 'a long, long way to run' > >>> d.keys() > ['fa', 'mi', 'do', 're'] > >>> d.values() > ['a long, long way to run', 'a name I call myself', 'a deer, a female deer', 'a drop of golden sunshine'] > >>> Yes, I am, thank you. However, I didn't want either the keys or the values. Instead I wanted to remove a key within a list comprehension. Cheers, Loris PS: "a drop of golden *sun*" - rhymes with "a long, long way to run" -- This signature is currently under constuction. From grant.b.edwards at gmail.com Mon Mar 25 11:54:59 2024 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Mon, 25 Mar 2024 11:54:59 -0400 (EDT) Subject: Popping key causes dict derived from object to revert to object References: <87zfurgb6t.fsf@zedat.fu-berlin.de> <87plvmr93v.fsf@zedat.fu-berlin.de> <4V1Rrq72BnznWJ6@mail.python.org> <87zfumre9b.fsf@zedat.fu-berlin.de> Message-ID: <4V3HZC52gbznVFV@mail.python.org> On 2024-03-25, Loris Bennett via Python-list wrote: > Grant Edwards writes: > >> On 2024-03-22, Loris Bennett via Python-list wrote: >> >>> Yes, I was mistakenly thinking that the popping the element would >>> leave me with the dict minus the popped key-value pair. >> >> It does. > > Indeed, but I was thinking in the context of > > dict_list = [d.pop('a') for d in dict_list] > > and incorrectly expecting to get a list of 'd' without key 'a', instead > of a list of the 'd['a]'. So when you say "leave me with", you mean "return the same dictionary with"? There's an important difference between what a function returns and what global/local state it "leaves you with". > Thanks for pointing out 'del'. My main problem, however, was > failing to realise that the list comprehension is populated by the > return value of the 'pop', not the popped dict. OK, so perhaps you weren't execting the original dict objects to be mutated, but rather that the pop method would return a new dict object without the "popped" element. The whole point of the 'pop method is to return the popped value, otherwise it wouldn't be needed. The 'del' statement would suffice. From avi.e.gross at gmail.com Mon Mar 25 12:04:48 2024 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Mon, 25 Mar 2024 12:04:48 -0400 Subject: Popping key causes dict derived from object to revert to object In-Reply-To: <874jcustc4.fsf@zedat.fu-berlin.de> References: <87zfurgb6t.fsf@zedat.fu-berlin.de> <87plvmr93v.fsf@zedat.fu-berlin.de> <005a01da7c84$4b9ba5f0$e2d2f1d0$@gmail.com> <874jcustc4.fsf@zedat.fu-berlin.de> Message-ID: <004901da7ece$29b3e870$7d1bb950$@gmail.com> I am glad, Lori, you found a solution another way. Actually, Lori, I think you were right in looking for a built-in method that complements pop() by returning everything else other than the item mentioned. There are philosophical and practical considerations that were no doubt considered and a reality that the functionality did exist albeit not in a pipelined format. Consider a language like LISP which sort of processed lists of things in which a major concept was getting the first item or getting all the rest. Lots of LISP programs had complex structures using CAR() and CDR() nested various ways to say extract the third item as CAR(CDR(CDR(X))) to the point where some commonly used combos became functions with names like CADDR(). There are again many languages where functions or methods are available that include an exclusion variant from a collection perhaps by doing something as simple as using a minus sign to indicate what to remove, as is commonly done in R to remove a column in a data.frame while keeping the order of the remaining columns the same. Lots of languages had similar concepts about ordered data structures but dictionaries may be a bit of something else and initially in Python were not guaranteed to have any kind of order. Python dicts are more like unordered sets. So although there remains a concept of not first/rest but this/rest, I suspect there was some thought about the process that ended in deciding not to supply some functionality. When you use pop() on something like a large dictionary, the original is left intact and is ignored and a copy of a single element is made and returned. To do the opposite and return the rest has two choices. One is to make a big copy of the rest of the dictionary and the other is to use del internally and return the modified dict. The functions you could write do the latter. So why not add one or more methods to do that? Who knows? But I think some may have considered it not needed including some who felt no need for a pipeline method when del would do. Another consideration was the common idiom for iterating on a collection. Besides pop() you can get lists of dictionary entries, keys or values that you can work with and you can even iterate with "for key in dict: ..." Given how many ways common things can be done, and given that adding too many methods has costs including new users not understanding all the nuanced differences, this fairly useful functionality was left out. Unfortunately, I think they were wrong here as instead we hear often from people like you who assume things would work other ways. I still think it would be simple enough to have had a .removekeys(keys) that would work in a pipeline to modify the dict by removing one or more items and perhaps another .removevalues(values) but at some point you may keep adding methods nobody ever uses. The reality is that many trivial one-liner comprehensions can easily do many such things using iteration. But many simply do not work well in pipelined fashion and thus may need to be embedded in a method of your own by subclassing dict or rolling your own. -----Original Message----- From: Loris Bennett Sent: Monday, March 25, 2024 2:45 AM To: avi.e.gross at gmail.com Cc: python-list at python.org Subject: Re: Popping key causes dict derived from object to revert to object writes: > Loris wrote: > > "Yes, I was mistakenly thinking that the popping the element would leave > me with the dict minus the popped key-value pair. Seem like there is no > such function." > > Others have tried to explain and pointed out you can del and then use the > changed dict. > > But consider the odd concept of writing your own trivial function. > > def remaining(adict, anitem): > _ = adict.pop(anitem) > # alternatively duse del on dict and item > return adict > > >>>> remaining({"first": 1, "second": 2, "third": 3}, "second") > {'first': 1, 'third': 3} > > > Or do you want to be able to call it as in dict.remaining(key) by > subclassing your own variant of dict and adding a similar method? No, 'del' does indeed do what I wanted, although I have now decided I want something else :-) Nevertheless it is good to know that 'del' exists, so that I don't have to reinvent it. Cheers, Loris -- This signature is currently under constuction. From avi.e.gross at gmail.com Mon Mar 25 12:19:12 2024 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Mon, 25 Mar 2024 12:19:12 -0400 Subject: Popping key causes dict derived from object to revert to object In-Reply-To: <87zfumre9b.fsf@zedat.fu-berlin.de> References: <87zfurgb6t.fsf@zedat.fu-berlin.de> <87plvmr93v.fsf@zedat.fu-berlin.de> <4V1Rrq72BnznWJ6@mail.python.org> <87zfumre9b.fsf@zedat.fu-berlin.de> Message-ID: <006001da7ed0$2cb3f720$861be560$@gmail.com> Lori, The list comprehension you are thinking of does work if you change things a bit. But it is not a great idea as a main purpose of a dict is that using a hash means things are found in linear time. A comprehension iterates on all values. If you wanted to select just some items to keep in a list, your code could be modified from: dict_list = [d.pop('a') for d in dict_list] to have an IF clause that would specify something like comparing it to the item you do not want to keep. But your idiom might be better done to make another dictionaly, not list with something like: New_dict = {key:value for key in dict if key != "whatever"} Or variants on that. It builds a new dictionary, at nontrivial expense, as compared to using del on an existing dictionary. -----Original Message----- From: Python-list On Behalf Of Loris Bennett via Python-list Sent: Monday, March 25, 2024 2:56 AM To: python-list at python.org Subject: Re: Popping key causes dict derived from object to revert to object Grant Edwards writes: > On 2024-03-22, Loris Bennett via Python-list wrote: > >> Yes, I was mistakenly thinking that the popping the element would >> leave me with the dict minus the popped key-value pair. > > It does. Indeed, but I was thinking in the context of dict_list = [d.pop('a') for d in dict_list] and incorrectly expecting to get a list of 'd' without key 'a', instead of a list of the 'd['a]'. >> Seem like there is no such function. > > Yes, there is. You can do that with either pop or del: > > >>> d = {'a':1, 'b':2, 'c':3} > >>> d > {'a': 1, 'b': 2, 'c': 3} > >>> d.pop('b') > 2 > >>> d > {'a': 1, 'c': 3} > > > >>> d = {'a':1, 'b':2, 'c':3} > >>> del d['b'] > >>> d > {'a': 1, 'c': 3} > > In both cases, you're left with the dict minus the key/value pair. > > In the first case, the deleted value printed by the REPL because it > was returned by the expression "d.pop('b')" (a method call). > > In the second case is no value shown by the REPL because "del d['b']" > is a statement not an expression. Thanks for pointing out 'del'. My main problem, however, was failing to realise that the list comprehension is populated by the return value of the 'pop', not the popped dict. Cheers, Loris -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list From mikec at lancanyon.com Wed Mar 27 20:58:38 2024 From: mikec at lancanyon.com (Dr. F. M. (Mike) Covington) Date: Wed, 27 Mar 2024 19:58:38 -0500 Subject: Popping key causes dict derived from object to revert to object (3/25/2024) Message-ID: Hello, all. I guess I'm just not seeing this dictionary pop() problem you're having. What version of Python are you using? Is there something I'm missing? If I've made an obvious oversight, please forgive me and explain. Thanks! PyCharm (2023.1.1) REPL running Python 3.12: Mike ---------------------- F. M. (Mike) Covington, PhD, PCPP, MCT Automation Consultant, Developer, Instructor From perso.olivier.barthelemy at gmail.com Thu Mar 28 11:07:04 2024 From: perso.olivier.barthelemy at gmail.com (Olivier B.) Date: Thu, 28 Mar 2024 16:07:04 +0100 Subject: Making 'compiled' modules work with multiple python versions on Linux Message-ID: I have a python module that includes some C++ code that links with the Python C API I have now modified the c++ code so that it only uses the Limited API, and linked with python3.lib instead of python311.lib. I can now use that python module with different python versions on Windows But on Linux, it seems that linking to libpython3.so instead of libpython3.11.so.1.0 does not have the same effect, and results in many unresolved python symbols at link time Is this functionality only available on Windows? From none at none.fr Thu Mar 28 12:45:42 2024 From: none at none.fr (ast) Date: Thu, 28 Mar 2024 17:45:42 +0100 Subject: A missing iterator on itertools module? Message-ID: <66059eb6$0$7522$426a34cc@news.free.fr> Hello Suppose I have these 3 strings: s1 = "AZERTY" s2 = "QSDFGH" s3 = "WXCVBN" and I need an itertor who delivers A Q W Z S C E D C ... I didn't found anything in itertools to do the job. So I came up with this solution: list(chain.from_iterable(zip("AZERTY", "QSDFGH", "WXCVBN"))) ['A', 'Q', 'W', 'Z', 'S', 'X', 'E', 'D', 'C', 'R', 'F', 'V', 'T', 'G', 'B', 'Y', 'H', 'N'] Do you havbe a neat solution ? From none at none.fr Thu Mar 28 12:47:32 2024 From: none at none.fr (ast) Date: Thu, 28 Mar 2024 17:47:32 +0100 Subject: A missing iterator on itertools module? In-Reply-To: <66059eb6$0$7522$426a34cc@news.free.fr> References: <66059eb6$0$7522$426a34cc@news.free.fr> Message-ID: <66059f24$0$7522$426a34cc@news.free.fr> Le 28/03/2024 ? 17:45, ast a ?crit?: > A Q W Z S C E D C ... sorry A Q W Z S X E D C From none at none.fr Thu Mar 28 13:12:54 2024 From: none at none.fr (ast) Date: Thu, 28 Mar 2024 18:12:54 +0100 Subject: A missing iterator on itertools module? In-Reply-To: References: <66059eb6$0$7522$426a34cc@news.free.fr> Message-ID: <6605a515$0$2578$426a74cc@news.free.fr> Le 28/03/2024 ? 18:07, Stefan Ram a ?crit?: > ast wrote or quoted: >> s1 = "AZERTY" >> s2 = "QSDFGH" >> s3 = "WXCVBN" >> and I need an itertor who delivers >> A Q W Z S C E D C ... >> I didn't found anything in itertools to do the job. >> So I came up with this solution: >> list(chain.from_iterable(zip("AZERTY", "QSDFGH", "WXCVBN"))) > > Maybe you meant "zip(s1,s2,s3)" as the definition of s1, s2, > and s3 otherwise would not be required. Also the "list" is not > necessary because "chain.from_iterable" already is an iterable. > You could also use "*" instead of "list" to print it. So, > > import itertools as _itertools > s =[ "AZERTY", "QSDFGH", "WXCVBN" ] > print( *_itertools.chain.from_iterable( zip( *s ))) > > . But these are only minor nitpicks; you have found a nice solution! Why did you renamed itertools as _itertools ? From barry at barrys-emacs.org Fri Mar 29 05:10:17 2024 From: barry at barrys-emacs.org (Barry) Date: Fri, 29 Mar 2024 09:10:17 +0000 Subject: Making 'compiled' modules work with multiple python versions on Linux In-Reply-To: References: Message-ID: <72BBD020-130F-4026-9A53-F757C2576654@barrys-emacs.org> > On 28 Mar 2024, at 16:13, Olivier B. via Python-list wrote: > > But on Linux, it seems that linking to libpython3.so instead of > libpython3.11.so.1.0 does not have the same effect, and results in > many unresolved python symbols at link time > > Is this functionality only available on Windows? Python limited API works on linux, but you do not link against the .so on linux I recall. You will have missed that libpython3.so is a symlink to libpython3.11.so.10. Windows build practices do not translate one-to-one to linux, or macOS. Barry From perso.olivier.barthelemy at gmail.com Fri Mar 29 12:09:32 2024 From: perso.olivier.barthelemy at gmail.com (Olivier B.) Date: Fri, 29 Mar 2024 17:09:32 +0100 Subject: Making 'compiled' modules work with multiple python versions on Linux In-Reply-To: <72BBD020-130F-4026-9A53-F757C2576654@barrys-emacs.org> References: <72BBD020-130F-4026-9A53-F757C2576654@barrys-emacs.org> Message-ID: It is not a symlink on my system, where i built python myself, but a 15KB so file. But it seems to lack lots of python symbols. Maybe what i should do is actually make libpython.so a physical copy of libpyton311.so before linking to it, so now on any system the module would look to load libpython.so, which could be pointing to any version. I'll try that next Le ven. 29 mars 2024 ? 10:10, Barry a ?crit : > > > > > On 28 Mar 2024, at 16:13, Olivier B. via Python-list wrote: > > > > But on Linux, it seems that linking to libpython3.so instead of > > libpython3.11.so.1.0 does not have the same effect, and results in > > many unresolved python symbols at link time > > > > Is this functionality only available on Windows? > > Python limited API works on linux, but you do not link against the .so on linux I recall. > > You will have missed that libpython3.so is a symlink to libpython3.11.so.10. > > Windows build practices do not translate one-to-one to linux, or macOS. > > Barry > > From barry at barrys-emacs.org Fri Mar 29 12:45:58 2024 From: barry at barrys-emacs.org (Barry Scott) Date: Fri, 29 Mar 2024 16:45:58 +0000 Subject: Making 'compiled' modules work with multiple python versions on Linux In-Reply-To: References: <72BBD020-130F-4026-9A53-F757C2576654@barrys-emacs.org> Message-ID: <3E7DA517-1239-4BCB-816C-440F0B95A88E@barrys-emacs.org> > On 29 Mar 2024, at 16:09, Olivier B. wrote: > > It is not a symlink on my system, where i built python myself, but a > 15KB so file. But it seems to lack lots of python symbols. > > Maybe what i should do is actually make libpython.so a physical copy > of libpyton311.so before linking to it, so now on any system the > module would look to load libpython.so, which could be pointing to any > version. I'll try that next You do not link against the .so at all. All the symbols you need are defined in the python process that loads the extension. Try without the -lpython and it should just work. Barry > > Le ven. 29 mars 2024 ? 10:10, Barry a ?crit : >> >> >> >>> On 28 Mar 2024, at 16:13, Olivier B. via Python-list wrote: >>> >>> But on Linux, it seems that linking to libpython3.so instead of >>> libpython3.11.so.1.0 does not have the same effect, and results in >>> many unresolved python symbols at link time >>> >>> Is this functionality only available on Windows? >> >> Python limited API works on linux, but you do not link against the .so on linux I recall. >> >> You will have missed that libpython3.so is a symlink to libpython3.11.so.10. >> >> Windows build practices do not translate one-to-one to linux, or macOS. >> >> Barry >> >> > From greg.ewing at canterbury.ac.nz Sat Mar 30 03:04:00 2024 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 30 Mar 2024 20:04:00 +1300 Subject: xkcd.com/353 ( Flying with Python ) In-Reply-To: References: Message-ID: On 30/03/24 7:21 pm, HenHanna wrote: > https://xkcd.com/1306/ > ???????????????????????? what does? SIGIL?? mean? I think its' a Perl term, referring to the $/@/# symbols in front of identifiers. -- Greg From skip.montanaro at gmail.com Sat Mar 30 07:25:57 2024 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Sat, 30 Mar 2024 06:25:57 -0500 Subject: xkcd.com/353 ( Flying with Python ) In-Reply-To: References: Message-ID: > > > https://xkcd.com/1306/ > > what does SIGIL mean? > > I think its' a Perl term, referring to the $/@/# symbols in front of > identifiers. > I had a vague recollection of hearing it elsewhere (*Game of Thrones,* on the armies' battle flags?), but didn't know what it meant. Google tells me: *an inscribed or painted symbol considered to have magical power.* So, they're more than just line noise. They confer power on their users... Perhaps '@' in the context of decorators is the most prominent example in Python, since decorators technically don't allow the programmer to do something they couldn't before, but are now are used everywhere, a key feature of many applications and modules. Magical-ly, y'rs, Skip > From python at mrabarnett.plus.com Sat Mar 30 12:31:59 2024 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 30 Mar 2024 16:31:59 +0000 Subject: xkcd.com/353 ( Flying with Python ) In-Reply-To: References: Message-ID: On 2024-03-30 11:25, Skip Montanaro via Python-list wrote: >> >> > https://xkcd.com/1306/ >> > what does SIGIL mean? >> >> I think its' a Perl term, referring to the $/@/# symbols in front of >> identifiers. >> > > I had a vague recollection of hearing it elsewhere (*Game of Thrones,* on > the armies' battle flags?), but didn't know what it meant. Google tells me: > > *an inscribed or painted symbol considered to have magical power.* > > So, they're more than just line noise. They confer power on their users... > > Perhaps '@' in the context of decorators is the most prominent example in > Python, since decorators technically don't allow the programmer to do > something they couldn't before, but are now are used everywhere, a key > feature of many applications and modules. > > Magical-ly, y'rs, > I wouldn't consider '@' to be a sigil any more than I would a unary minus. In Perl there's the prefixes $ (scalar), @ (array) and % (hash/dictionary), but also & (function), although it's rare because there's also the () afterwards. Variables in PHP have the prefix $ and only $. In old versions of BASIC, string variables had the suffix $, and integer variables the suffix %. Some versions also had the suffix # (for double precision, I think). From learn2program at gmail.com Sat Mar 30 13:58:08 2024 From: learn2program at gmail.com (Alan Gauld) Date: Sat, 30 Mar 2024 17:58:08 +0000 Subject: xkcd.com/353 ( Flying with Python ) In-Reply-To: References: Message-ID: <9dd90b7a-d021-49bd-aa76-1a94dcff926a@yahoo.co.uk> On 30/03/2024 07:04, Greg Ewing via Python-list wrote: > On 30/03/24 7:21 pm, HenHanna wrote: >> https://xkcd.com/1306/ >> ???????????????????????? what does? SIGIL?? mean? > > I think its' a Perl term, referring to the $/@/# symbols in front of > identifiers. There seem to be several derivation sources including a fantasy world city suspended above a very thin, tall steeple.... Personally, I know SIGIL as an opensource EPUB editor! None of them seem to have any direct connection to the xkcd cartoon. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From mk1853387 at gmail.com Sat Mar 30 20:09:43 2024 From: mk1853387 at gmail.com (marc nicole) Date: Sun, 31 Mar 2024 01:09:43 +0100 Subject: Can you help me with this memoization simple example? Message-ID: I am creating a memoization example with a function that adds up / averages the elements of an array and compares it with the cached ones to retrieve them in case they are already stored. In addition, I want to store only if the result of the function differs considerably (passes a threshold e.g. 500000 below). I created an example using a decorator to do so, the results using the decorator is slightly faster than without the memoization which is OK, but is the logic of the decorator correct ? anybody can tell me ? My code is attached below: import time def memoize(f): cache = {} def g(*args): if args[1] == "avg": sum_key_arr = sum(list(args[0])) / len(list(args[0])) elif args[1] == "sum": sum_key_arr = sum(list(args[0])) if sum_key_arr not in cache: for ( key, value, ) in ( cache.items() ): # key in dict cannot be an array so I use the sum of the array as the key if ( abs(sum_key_arr - key) <= 500000 ): # threshold is great here so that all values are approximated! # print('approximated') return cache[key] else: # print('not approximated') cache[sum_key_arr] = f(args[0], args[1]) return cache[sum_key_arr] return g @memoize def aggregate(dict_list_arr, operation): if operation == "avg": return sum(list(dict_list_arr)) / len(list(dict_list_arr)) if operation == "sum": return sum(list(dict_list_arr)) return None t = time.time() for i in range(200, 15000): res = aggregate(list(range(i)), "avg") elapsed = time.time() - t print(res) print(elapsed) From python at mrabarnett.plus.com Sat Mar 30 20:39:09 2024 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 31 Mar 2024 00:39:09 +0000 Subject: Can you help me with this memoization simple example? In-Reply-To: References: Message-ID: <01e1e4ee-e770-4f69-8dfd-ba8b7bb0a6ae@mrabarnett.plus.com> On 2024-03-31 00:09, marc nicole via Python-list wrote: > I am creating a memoization example with a function that adds up / averages > the elements of an array and compares it with the cached ones to retrieve > them in case they are already stored. > > In addition, I want to store only if the result of the function differs > considerably (passes a threshold e.g. 500000 below). > > I created an example using a decorator to do so, the results using the > decorator is slightly faster than without the memoization which is OK, but > is the logic of the decorator correct ? anybody can tell me ? > > My code is attached below: > > > > import time > > > def memoize(f): > cache = {} > > def g(*args): > if args[1] == "avg": > sum_key_arr = sum(list(args[0])) / len(list(args[0])) 'list' will iterate over args[0] to make a list, and 'sum' will iterate over that list. It would be simpler to just let 'sum' iterate over args[0]. > elif args[1] == "sum": > sum_key_arr = sum(list(args[0])) > if sum_key_arr not in cache: > for ( > key, > value, > ) in ( > cache.items() > ): # key in dict cannot be an array so I use the sum of the > array as the key You can't use a list as a key, but you can use a tuple as a key, provided that the elements of the tuple are also immutable. > if ( > abs(sum_key_arr - key) <= 500000 > ): # threshold is great here so that all values are > approximated! > # print('approximated') > return cache[key] > else: > # print('not approximated') > cache[sum_key_arr] = f(args[0], args[1]) > return cache[sum_key_arr] > > return g > > > @memoize > def aggregate(dict_list_arr, operation): > if operation == "avg": > return sum(list(dict_list_arr)) / len(list(dict_list_arr)) > if operation == "sum": > return sum(list(dict_list_arr)) > return None > > > t = time.time() > for i in range(200, 15000): > res = aggregate(list(range(i)), "avg") > > elapsed = time.time() - t > print(res) > print(elapsed) From mk1853387 at gmail.com Sun Mar 31 04:04:13 2024 From: mk1853387 at gmail.com (marc nicole) Date: Sun, 31 Mar 2024 10:04:13 +0200 Subject: Can you help me with this memoization simple example? In-Reply-To: <01e1e4ee-e770-4f69-8dfd-ba8b7bb0a6ae@mrabarnett.plus.com> References: <01e1e4ee-e770-4f69-8dfd-ba8b7bb0a6ae@mrabarnett.plus.com> Message-ID: Thanks for the first comment which I incorporated but when you say "You can't use a list as a key, but you can use a tuple as a key, provided that the elements of the tuple are also immutable." does it mean the result of sum of the array is not convenient to use as key as I do? Which tuple I should use to refer to the underlying list value as you suggest? Anything else is good in my code ? Thanks Le dim. 31 mars 2024 ? 01:44, MRAB via Python-list a ?crit : > On 2024-03-31 00:09, marc nicole via Python-list wrote: > > I am creating a memoization example with a function that adds up / > averages > > the elements of an array and compares it with the cached ones to retrieve > > them in case they are already stored. > > > > In addition, I want to store only if the result of the function differs > > considerably (passes a threshold e.g. 500000 below). > > > > I created an example using a decorator to do so, the results using the > > decorator is slightly faster than without the memoization which is OK, > but > > is the logic of the decorator correct ? anybody can tell me ? > > > > My code is attached below: > > > > > > > > import time > > > > > > def memoize(f): > > cache = {} > > > > def g(*args): > > if args[1] == "avg": > > sum_key_arr = sum(list(args[0])) / len(list(args[0])) > > 'list' will iterate over args[0] to make a list, and 'sum' will iterate > over that list. > > It would be simpler to just let 'sum' iterate over args[0]. > > > elif args[1] == "sum": > > sum_key_arr = sum(list(args[0])) > > if sum_key_arr not in cache: > > for ( > > key, > > value, > > ) in ( > > cache.items() > > ): # key in dict cannot be an array so I use the sum of the > > array as the key > > You can't use a list as a key, but you can use a tuple as a key, > provided that the elements of the tuple are also immutable. > > > if ( > > abs(sum_key_arr - key) <= 500000 > > ): # threshold is great here so that all values are > > approximated! > > # print('approximated') > > return cache[key] > > else: > > # print('not approximated') > > cache[sum_key_arr] = f(args[0], args[1]) > > return cache[sum_key_arr] > > > > return g > > > > > > @memoize > > def aggregate(dict_list_arr, operation): > > if operation == "avg": > > return sum(list(dict_list_arr)) / len(list(dict_list_arr)) > > if operation == "sum": > > return sum(list(dict_list_arr)) > > return None > > > > > > t = time.time() > > for i in range(200, 15000): > > res = aggregate(list(range(i)), "avg") > > > > elapsed = time.time() - t > > print(res) > > print(elapsed) > > > -- > https://mail.python.org/mailman/listinfo/python-list > From jacob.kruger.work at gmail.com Sun Mar 31 08:20:45 2024 From: jacob.kruger.work at gmail.com (Jacob Kruger) Date: Sun, 31 Mar 2024 14:20:45 +0200 Subject: Trying to use pyinstaller under python 3.11, and, recently started receiving error message about specific module/distribution Message-ID: This started happening this past week, and, while it's worked fine in the past, the moment I try to launch the pyinstaller process at all, to generate compiled output, or even if just launch it with no command line options, I receive the following error message: pkg_resources.DistributionNotFound: The 'altgraph' distribution was not found and is required by the application The full contents of the output string when I even try to just launch pyinstaller with no commands/arguments is the following: Traceback (most recent call last): ? File "", line 198, in _run_module_as_main ? File "", line 88, in _run_code ? File "C:\pythonScripts\monitoring_nssm\venv\Scripts\pyinstaller.exe\__main__.py", line 7, in ? File "C:\pythonScripts\monitoring_nssm\venv\Lib\site-packages\PyInstaller\__main__.py", line 228, in _console_script_run run() ? File "C:\pythonScripts\monitoring_nssm\venv\Lib\site-packages\PyInstaller\__main__.py", line 170, in run ??? parser = generate_parser() ^^^^^^^^^^^^^^^^^ ? File "C:\pythonScripts\monitoring_nssm\venv\Lib\site-packages\PyInstaller\__main__.py", line 136, in generate_parser ??? import PyInstaller.building.build_main ? File "C:\pythonScripts\monitoring_nssm\venv\Lib\site-packages\PyInstaller\building\build_main.py", line 28, in ??? from PyInstaller.building.api import COLLECT, EXE, MERGE, PYZ ? File "C:\pythonScripts\monitoring_nssm\venv\Lib\site-packages\PyInstaller\building\api.py", line 32, in ??? from PyInstaller.building.splash import Splash? # argument type validation in EXE ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ? File "C:\pythonScripts\monitoring_nssm\venv\Lib\site-packages\PyInstaller\building\splash.py", line 23, in ??? from PyInstaller.depend import bindepend ? File "C:\pythonScripts\monitoring_nssm\venv\Lib\site-packages\PyInstaller\depend\bindepend.py", line 25, in ??? from PyInstaller.depend import dylib, utils ? File "C:\pythonScripts\monitoring_nssm\venv\Lib\site-packages\PyInstaller\depend\utils.py", line 31, in ??? from PyInstaller.lib.modulegraph import modulegraph ? File "C:\pythonScripts\monitoring_nssm\venv\Lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 34, in ??? from altgraph.ObjectGraph import ObjectGraph ? File "C:\pythonScripts\monitoring_nssm\venv\Lib\site-packages\altgraph\__init__.py", line 144, in ??? __version__ = pkg_resources.require("altgraph")[0].version ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ? File "C:\pythonScripts\monitoring_nssm\venv\Lib\site-packages\pkg_resources\__init__.py", line 952, in require ??? needed = self.resolve(parse_requirements(requirements)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ? File "C:\pythonScripts\monitoring_nssm\venv\Lib\site-packages\pkg_resources\__init__.py", line 813, in resolve ??? dist = self._resolve_dist( ^^^^^^^^^^^^^^^^^^^ ? File "C:\pythonScripts\monitoring_nssm\venv\Lib\site-packages\pkg_resources\__init__.py", line 854, in _resolve_dist ??? raise DistributionNotFound(req, requirers) pkg_resources.DistributionNotFound: The 'altgraph' distribution was not found and is required by the application # ---end of output--- I have tried completely removing python's installation, and, reinstalling it, but, same issue more or less immediately. If I freeze pip's installed list within this specific virtual environment, it lists the following: altgraph==0.17.4 packaging==24.0 pefile==2023.2.7 pyinstaller==6.5.0 pyinstaller-hooks-contrib==2024.3 pywin32-ctypes==0.2.2 # ---end of requirements.txt--- And, if, just for testing, I launch python interpreter, and, ask it to import altgraph, it provides the same last line of error output? If relevant, running with python 3.11.8, under windows 11 64-bit, and, can't think of anything that specifically occurred/changed this past week, besides normal things like windows updates, etc., but, don't really think that's likely to be relevant, unless something to do with pywin32 has caused an issue? Should I try installing python 3.12 version instead and see if it changes? Also, if relevant, while running under latest up-to-date version of windows 11 64 bit, have in any case enabled case-sensitivity on the folder I store all my python code in, just in case. TIA -- Jacob Kruger +2782 413 4791 "Resistance is futile!...Acceptance is versatile..." From barry at barrys-emacs.org Sun Mar 31 08:51:11 2024 From: barry at barrys-emacs.org (Barry) Date: Sun, 31 Mar 2024 13:51:11 +0100 Subject: Trying to use pyinstaller under python 3.11, and, recently started receiving error message about specific module/distribution In-Reply-To: References: Message-ID: > On 31 Mar 2024, at 13:24, Jacob Kruger via Python-list wrote: > > pkg_resources.DistributionNotFound: The 'altgraph' distribution was not found and is required by the application I think I have seen this error being discussed before? A web search for pyinstaller and that error leads to people discussing why it happens it looks like. Barry From mats at wichmann.us Sun Mar 31 14:27:34 2024 From: mats at wichmann.us (Mats Wichmann) Date: Sun, 31 Mar 2024 12:27:34 -0600 Subject: xkcd.com/353 ( Flying with Python ) In-Reply-To: References: Message-ID: On 3/30/24 10:31, MRAB via Python-list wrote: > On 2024-03-30 11:25, Skip Montanaro via Python-list wrote: >>> >>> > https://xkcd.com/1306/ >>> >?????????????????????????? what does? SIGIL?? mean? >>> >>> I think its' a Perl term, referring to the $/@/# symbols in front of >>> identifiers. > > I wouldn't consider '@' to be a sigil any more than I would a unary minus. Nonetheless, Perl folk do use that term, specifically. "One thing that distinguishes Perl from other languages is its use of sigils; the funny looking symbols placed in front of variable names. " $ Scalar $foo @ Array @foo % Hash %foo & Subroutine &foo * Typeglob *foo > Sigils have many benefits, not least of which is that variables can be interpolated into strings with no additional syntax. Perl scripts are also easy to read (for people who have bothered to learn Perl!) because the nouns stand out from verbs. And new verbs can be added to the language without breaking old scripts. > Programming Perl, Chapter 1, 4th Edition etc. From hjp-python at hjp.at Sun Mar 31 14:27:07 2024 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 31 Mar 2024 20:27:07 +0200 Subject: xkcd.com/353 ( Flying with Python ) In-Reply-To: <9dd90b7a-d021-49bd-aa76-1a94dcff926a@yahoo.co.uk> References: <9dd90b7a-d021-49bd-aa76-1a94dcff926a@yahoo.co.uk> Message-ID: <20240331182707.ptyzeimhunamw6xl@hjp.at> On 2024-03-30 17:58:08 +0000, Alan Gauld via Python-list wrote: > On 30/03/2024 07:04, Greg Ewing via Python-list wrote: > > On 30/03/24 7:21 pm, HenHanna wrote: > >> https://xkcd.com/1306/ > >> ???????????????????????? what does? SIGIL?? mean? > > > > I think its' a Perl term, referring to the $/@/# symbols in front of > > identifiers. Correct (although strictly speaking they are in front of an expression, not an identifier). > There seem to be several derivation sources including a fantasy world > city suspended above a very thin, tall steeple.... > > Personally, I know SIGIL as an opensource EPUB editor! Well, it's an ordinary English word of Latin origin (sigillum means literally "small sign") in use since the 15th century. No need to go hunting for proper names. > None of them seem to have any direct connection to the xkcd cartoon. In my opinion the connection to Perl sigils is very direct. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From hjp-python at hjp.at Sun Mar 31 14:53:41 2024 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 31 Mar 2024 20:53:41 +0200 Subject: xkcd.com/353 ( Flying with Python ) In-Reply-To: References: Message-ID: <20240331185341.h5nxxjq53q6t274p@hjp.at> On 2024-03-31 12:27:34 -0600, Mats Wichmann via Python-list wrote: > On 3/30/24 10:31, MRAB via Python-list wrote: > > On 2024-03-30 11:25, Skip Montanaro via Python-list wrote: > > > > > https://xkcd.com/1306/ > > > > >?????????????????????????? what does? SIGIL?? mean? > > > > > > > > I think its' a Perl term, referring to the $/@/# symbols in front of > > > > identifiers. [You cut out a lot of context here] > > I wouldn't consider '@' to be a sigil any more than I would a unary minus. > > Nonetheless, Perl folk do use that term, specifically. I'm pretty sure he's referring to the use of @ in python to denote a decorator here. Which is a totally different thing than a Perl sigil. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From python at mrabarnett.plus.com Sun Mar 31 15:23:50 2024 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 31 Mar 2024 20:23:50 +0100 Subject: Can you help me with this memoization simple example? In-Reply-To: References: <01e1e4ee-e770-4f69-8dfd-ba8b7bb0a6ae@mrabarnett.plus.com> Message-ID: <022811dc-dca5-400b-bf2b-3f3e3a67f151@mrabarnett.plus.com> On 2024-03-31 09:04, marc nicole wrote: > Thanks for the first comment which I incorporated > > but when you say "You can't use a list as a key, but you can use a > tuple as a key, > provided that the elements of the tuple are also immutable." > > does it mean? the result of sum of the array is not convenient to use > as key as I do? > Which tuple I should use to refer to the underlying list value as you > suggest? > I was suggesting using `tuple` on the argument: def memoize(f): ???? cache = {} ???? def g(*args): ???????? key = tuple(args[0]), args[1] ???????? if key not in cache: ???????????? cache[key] = f(args[0], args[1]) ???????? return cache[key] ???? return g > Anything else is good in my code ? > > Thanks > > Le?dim. 31 mars 2024 ??01:44, MRAB via Python-list > a ?crit?: > > On 2024-03-31 00:09, marc nicole via Python-list wrote: > > I am creating a memoization example with a function that adds up > / averages > > the elements of an array and compares it with the cached ones to > retrieve > > them in case they are already stored. > > > > In addition, I want to store only if the result of the function > differs > > considerably (passes a threshold e.g. 500000 below). > > > > I created an example using a decorator to do so, the results > using the > > decorator is slightly faster than without the memoization which > is OK, but > > is the logic of the decorator correct ? anybody can tell me ? > > > > My code is attached below: > > > > > > > > import time > > > > > > def memoize(f): > >? ? ? cache = {} > > > >? ? ? def g(*args): > >? ? ? ? ? if args[1] == "avg": > >? ? ? ? ? ? ? sum_key_arr = sum(list(args[0])) / len(list(args[0])) > > 'list' will iterate over args[0] to make a list, and 'sum' will > iterate > over that list. > > It would be simpler to just let 'sum' iterate over args[0]. > > >? ? ? ? ? elif args[1] == "sum": > >? ? ? ? ? ? ? sum_key_arr = sum(list(args[0])) > >? ? ? ? ? if sum_key_arr not in cache: > >? ? ? ? ? ? ? for ( > >? ? ? ? ? ? ? ? ? key, > >? ? ? ? ? ? ? ? ? value, > >? ? ? ? ? ? ? ) in ( > >? ? ? ? ? ? ? ? ? cache.items() > >? ? ? ? ? ? ? ):? # key in dict cannot be an array so I use the > sum of the > > array as the key > > You can't use a list as a key, but you can use a tuple as a key, > provided that the elements of the tuple are also immutable. > > >? ? ? ? ? ? ? ? ? if ( > >? ? ? ? ? ? ? ? ? ? ? abs(sum_key_arr - key) <= 500000 > >? ? ? ? ? ? ? ? ? ):? # threshold is great here so that all > values are > > approximated! > >? ? ? ? ? ? ? ? ? ? ? # print('approximated') > >? ? ? ? ? ? ? ? ? ? ? return cache[key] > >? ? ? ? ? ? ? else: > >? ? ? ? ? ? ? ? ? # print('not approximated') > >? ? ? ? ? ? ? ? ? cache[sum_key_arr] = f(args[0], args[1]) > >? ? ? ? ? return cache[sum_key_arr] > > > >? ? ? return g > > > > > > @memoize > > def aggregate(dict_list_arr, operation): > >? ? ? if operation == "avg": > >? ? ? ? ? return sum(list(dict_list_arr)) / len(list(dict_list_arr)) > >? ? ? if operation == "sum": > >? ? ? ? ? return sum(list(dict_list_arr)) > >? ? ? return None > > > > > > t = time.time() > > for i in range(200, 15000): > >? ? ? res = aggregate(list(range(i)), "avg") > > > > elapsed = time.time() - t > > print(res) > > print(elapsed) > > > -- > https://mail.python.org/mailman/listinfo/python-list > From avi.e.gross at gmail.com Sun Mar 31 16:56:19 2024 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Sun, 31 Mar 2024 16:56:19 -0400 Subject: Can you help me with this memoization simple example? In-Reply-To: <022811dc-dca5-400b-bf2b-3f3e3a67f151@mrabarnett.plus.com> References: <01e1e4ee-e770-4f69-8dfd-ba8b7bb0a6ae@mrabarnett.plus.com> <022811dc-dca5-400b-bf2b-3f3e3a67f151@mrabarnett.plus.com> Message-ID: <008901da83ad$e17541f0$a45fc5d0$@gmail.com> I am not sure if it was made clear that there is a general rule in python for what is HASHABLE and lists are changeable while tuples are not so the latter can be hashed as a simple copy of a list, albeit the contents must also be immutable. The memorize function uses a dictionary to store things and thus the things are hashed to decide how to store it in the inner representation of a dictionary and anything new that you want to look up in the dictionary has similar considerations as it is hashed to see where in the dictionary to look for it. Of course, if you add enough overhead and the memorize function you make gets relatively few requests that are identical, it may not be worthwhile. -----Original Message----- From: Python-list On Behalf Of MRAB via Python-list Sent: Sunday, March 31, 2024 3:24 PM To: python-list at python.org Subject: Re: Can you help me with this memoization simple example? On 2024-03-31 09:04, marc nicole wrote: > Thanks for the first comment which I incorporated > > but when you say "You can't use a list as a key, but you can use a > tuple as a key, > provided that the elements of the tuple are also immutable." > > does it mean the result of sum of the array is not convenient to use > as key as I do? > Which tuple I should use to refer to the underlying list value as you > suggest? > I was suggesting using `tuple` on the argument: def memoize(f): cache = {} def g(*args): key = tuple(args[0]), args[1] if key not in cache: cache[key] = f(args[0], args[1]) return cache[key] return g > Anything else is good in my code ? > > Thanks > > Le dim. 31 mars 2024 ? 01:44, MRAB via Python-list > a ?crit : > > On 2024-03-31 00:09, marc nicole via Python-list wrote: > > I am creating a memoization example with a function that adds up > / averages > > the elements of an array and compares it with the cached ones to > retrieve > > them in case they are already stored. > > > > In addition, I want to store only if the result of the function > differs > > considerably (passes a threshold e.g. 500000 below). > > > > I created an example using a decorator to do so, the results > using the > > decorator is slightly faster than without the memoization which > is OK, but > > is the logic of the decorator correct ? anybody can tell me ? > > > > My code is attached below: > > > > > > > > import time > > > > > > def memoize(f): > > cache = {} > > > > def g(*args): > > if args[1] == "avg": > > sum_key_arr = sum(list(args[0])) / len(list(args[0])) > > 'list' will iterate over args[0] to make a list, and 'sum' will > iterate > over that list. > > It would be simpler to just let 'sum' iterate over args[0]. > > > elif args[1] == "sum": > > sum_key_arr = sum(list(args[0])) > > if sum_key_arr not in cache: > > for ( > > key, > > value, > > ) in ( > > cache.items() > > ): # key in dict cannot be an array so I use the > sum of the > > array as the key > > You can't use a list as a key, but you can use a tuple as a key, > provided that the elements of the tuple are also immutable. > > > if ( > > abs(sum_key_arr - key) <= 500000 > > ): # threshold is great here so that all > values are > > approximated! > > # print('approximated') > > return cache[key] > > else: > > # print('not approximated') > > cache[sum_key_arr] = f(args[0], args[1]) > > return cache[sum_key_arr] > > > > return g > > > > > > @memoize > > def aggregate(dict_list_arr, operation): > > if operation == "avg": > > return sum(list(dict_list_arr)) / len(list(dict_list_arr)) > > if operation == "sum": > > return sum(list(dict_list_arr)) > > return None > > > > > > t = time.time() > > for i in range(200, 15000): > > res = aggregate(list(range(i)), "avg") > > > > elapsed = time.time() - t > > print(res) > > print(elapsed) > > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list From nntp.mbourne at spamgourmet.com Fri Mar 29 07:27:20 2024 From: nntp.mbourne at spamgourmet.com (Mark Bourne) Date: Fri, 29 Mar 2024 11:27:20 +0000 Subject: A missing iterator on itertools module? In-Reply-To: References: <66059eb6$0$7522$426a34cc@news.free.fr> <6605a515$0$2578$426a74cc@news.free.fr> Message-ID: Stefan Ram wrote: > ast wrote or quoted: >> Why did you renamed itertools as _itertools ? > > Assume I have a module A.py: > > import math > def f(): pass > > . Assume I have an additional module B.py: > > import A > > . Now, when I'm editing "B.py" in IDLE and type "A.", IIRC > IDLE will offer me two possible completions: "A.math" and > "A.f". The "A.math" makes no sense to me. `import math` imports the `math` module and binds it to `math` in the global namespace of the `A` module. Since it doesn't have a leading underscore, by default it's considered to be a public attribute of the `A` module, and IDLE is offering all the public attributes of the `A` module for completion. > I want it to go > away. Therefore, I rewrite A.py as: > > import math as _math > def f(): pass > > . Now, Idle will only offer the completion "A.f". > > So, I sometimes use this "import math as _math" style. But then, > it is simpler for me to /always/ use this style; after all: you > can't know whether someone eventually will import your module! You can explicitly declare the public interface of a module by defining `__all__`, listing the names which should be considered part of the module's public interface; see: - https://docs.python.org/3/reference/simple_stmts.html#the-import-statement - https://peps.python.org/pep-0008/#public-and-internal-interfaces Although `from A import *` is generally discouraged, if `A` defines `__all__` then only the names listed in `__all__` are bound in the importing module's namespace. Otherwise, all names from `A` which don't have a leading underscore are considered to be public and bound in the importing module. I don't use IDLE, but it may be that it also uses `__all__` to determine a module's public API. In that case, setting `__all__ = ["f"]` in `A` should prevent it from offering `math` as a completion (nor any other name that's not in the `__all__` list). -- Mark. From olegsivokon at gmail.com Fri Mar 29 11:37:06 2024 From: olegsivokon at gmail.com (Left Right) Date: Fri, 29 Mar 2024 16:37:06 +0100 Subject: Making 'compiled' modules work with multiple python versions on Linux In-Reply-To: <72BBD020-130F-4026-9A53-F757C2576654@barrys-emacs.org> References: <72BBD020-130F-4026-9A53-F757C2576654@barrys-emacs.org> Message-ID: It sounds weird that symbols from Limited API are _missing_ (I'd expect them to be there no matter what library version you link with). But, I haven't done this myself, so, what do I know? It would help though to see the actual error. That aside: why do you want to do this? One side effect of doing what you want will be the "weird" name of your wheel archive. Weird in a sense that virtually nobody does that. And when virtually nobody does something, you are almost guaranteed to be the first to find bugs, and then be the one whose bug reports are shoved into the backlog and never looked at again. You, kind of, are already walking into the world of pain trying to make Python binary packages, and then you also want them to be cross-platform, and then you want them to be usable by different versions of Python... Unless it's for your own amusement, I'd just have a package per version of Python. Maintenance-wise it's going to be a lot easier. On Fri, Mar 29, 2024 at 10:13?AM Barry via Python-list wrote: > > > > > On 28 Mar 2024, at 16:13, Olivier B. via Python-list wrote: > > > > But on Linux, it seems that linking to libpython3.so instead of > > libpython3.11.so.1.0 does not have the same effect, and results in > > many unresolved python symbols at link time > > > > Is this functionality only available on Windows? > > Python limited API works on linux, but you do not link against the .so on linux I recall. > > You will have missed that libpython3.so is a symlink to libpython3.11.so.10. > > Windows build practices do not translate one-to-one to linux, or macOS. > > Barry > > > -- > https://mail.python.org/mailman/listinfo/python-list From HenHanna at devnull.tb Sat Mar 30 02:21:15 2024 From: HenHanna at devnull.tb (HenHanna) Date: Fri, 29 Mar 2024 23:21:15 -0700 Subject: xkcd.com/353 ( Flying with Python ) Message-ID: https://xkcd.com/353/ ( Flying with Python ) https://xkcd.com/1306/ what does SIGIL mean? Other xkcd that you like? From HenHanna at dev.null Sat Mar 30 06:00:16 2024 From: HenHanna at dev.null (HenHanna) Date: Sat, 30 Mar 2024 10:00:16 +0000 Subject: xkcd.com/353 ( Flying with Python ) References: Message-ID: Greg Ewing wrote: > On 30/03/24 7:21 pm, HenHanna wrote: >> https://xkcd.com/1306/ >> ???????????????????????? what does? SIGIL?? mean? > I think its' a Perl term, referring to the $/@/# symbols in front of > identifiers. thanks! https://www.explainxkcd.com/wiki/index.php/1306:_Sigil_Cycle From bluemanedhawk at invalid.invalid Sat Mar 30 13:32:20 2024 From: bluemanedhawk at invalid.invalid (Blue-Maned_Hawk) Date: Sat, 30 Mar 2024 17:32:20 -0000 (UTC) Subject: xkcd.com/353 ( Flying with Python ) References: Message-ID: HenHanna wrote: > https://xkcd.com/1306/ > what does SIGIL mean? I'd define a sigil as a mandatory symbol used to indicate the properties of a name. -- Blue-Maned_Hawk?shortens to Hawk?/blu.m?in.d?ak/?he/him/his/himself/Mr. blue-maned_hawk.srht.site ?Do you know what you are?? ?Confused?? From jfairchild at tudado.org Sat Mar 30 17:09:56 2024 From: jfairchild at tudado.org (Johanne Fairchild) Date: Sat, 30 Mar 2024 18:09:56 -0300 Subject: xkcd.com/353 ( Flying with Python ) References: Message-ID: <87ttknbf4r.fsf@tudado.org> HenHanna writes: > https://xkcd.com/1306/ > what does SIGIL mean? A glyph used in magic. Or, for Perl, the symbol in front of a variable name, such as $, @, and %. Source: https://perldoc.perl.org/perlglossary#sigil Sigil is noun. Definitions: A seal; a signet. A sign or an image considered magical. A seal; a signature. Source: The American Heritage? Dictionary of the English Language, 5th Edition. From HenHanna at dev.null Sat Mar 30 17:34:03 2024 From: HenHanna at dev.null (HenHanna) Date: Sat, 30 Mar 2024 21:34:03 +0000 Subject: xkcd.com/353 ( Flying with Python ) References: <87ttknbf4r.fsf@tudado.org> Message-ID: <46f330fb23ace1d942e40bd0c7182fbc@www.novabbs.com> Johanne Fairchild wrote: > HenHanna writes: >> https://xkcd.com/1306/ >> what does SIGIL mean? > A glyph used in magic. Or, for Perl, the symbol in front of a variable > name, such as $, @, and %. > Source: https://perldoc.perl.org/perlglossary#sigil > Sigil is noun. Definitions: > A seal; a signet. > A sign or an image considered magical. > A seal; a signature. > Source: The American Heritage? Dictionary of the English Language, 5th Edition. omg... Sigil is a real word??? The word "sigil" comes from the Latin term "sigillum," which means "little sign." This Latin root is also the source of our English word "seal," making "sigil" and "seal" doublets. https://en.wiktionary.org/wiki/sigil __________________________words that we use in Programming but not Found in a real dictionary : Camel case , int, char, min, len, def, elseif cons, defun, cond, goto,