From sam.z.ezeh at gmail.com Sat Jul 2 10:27:55 2022 From: sam.z.ezeh at gmail.com (Sam Ezeh) Date: Sat, 2 Jul 2022 15:27:55 +0100 Subject: Building on Windows Message-ID: I have a Windows virtual machine and I'm following the instructions on the devguide [1] to build Python inside it. When running `PCbuild\build\bat -e -d` I get "Cannot locate MSBuild.exe on PATH or as MSBUILD variable". I've done a minimal amount of searching [2][3] but I'm not well-acquainted with Windows and don't understand the solutions. Thanks in advance. Kind regards, Sam Ezeh [1]: https://devguide.python.org/compiler/ [2]: https://bugs.python.org/issue41213 [3]: https://bugs.python.org/issue33675 From sam.z.ezeh at gmail.com Sat Jul 2 10:41:33 2022 From: sam.z.ezeh at gmail.com (Sam Ezeh) Date: Sat, 2 Jul 2022 15:41:33 +0100 Subject: Building on Windows In-Reply-To: References: Message-ID: To add to this, my process was 1. Setup the VM 2. Install Git 3. Clone CPython 4. Run `PCbuild\build.bat -d -e` 5. Notice the error, then install visual studio community 2022 6. Re-run `PCbuild\build.bat -d -e` and see the same error I can't directly copy and paste between the VM and my host desktop but in this scenario, the error was that single line and I can use paste sites where necessary. Kind regards, Sam Ezeh On Sat, 2 Jul 2022 at 15:27, Sam Ezeh wrote: > > I have a Windows virtual machine and I'm following the instructions on > the devguide [1] to build Python inside it. > > When running `PCbuild\build\bat -e -d` I get "Cannot locate > MSBuild.exe on PATH or as MSBUILD variable". I've done a minimal > amount of searching [2][3] but I'm not well-acquainted with Windows > and don't understand the solutions. > > Thanks in advance. > > Kind regards, > Sam Ezeh > > [1]: https://devguide.python.org/compiler/ > [2]: https://bugs.python.org/issue41213 > [3]: https://bugs.python.org/issue33675 From barry at barrys-emacs.org Sat Jul 2 14:19:41 2022 From: barry at barrys-emacs.org (Barry Scott) Date: Sat, 2 Jul 2022 19:19:41 +0100 Subject: Building on Windows In-Reply-To: References: Message-ID: <20512704-6D68-4730-BFE3-062B9B1560B5@barrys-emacs.org> > On 2 Jul 2022, at 15:41, Sam Ezeh wrote: > > To add to this, my process was > > 1. Setup the VM > 2. Install Git > 3. Clone CPython > 4. Run `PCbuild\build.bat -d -e` > 5. Notice the error, then install visual studio community 2022 Try running this: call "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat" I use that to get at the command line tools. If that does not work then search the disk for msbuild.exe and add its folder to your PATH. Barry > 6. Re-run `PCbuild\build.bat -d -e` and see the same error > > I can't directly copy and paste between the VM and my host desktop but > in this scenario, the error was that single line and I can use paste > sites where necessary. > > Kind regards, > Sam Ezeh > > On Sat, 2 Jul 2022 at 15:27, Sam Ezeh wrote: >> >> I have a Windows virtual machine and I'm following the instructions on >> the devguide [1] to build Python inside it. >> >> When running `PCbuild\build\bat -e -d` I get "Cannot locate >> MSBuild.exe on PATH or as MSBUILD variable". I've done a minimal >> amount of searching [2][3] but I'm not well-acquainted with Windows >> and don't understand the solutions. >> >> Thanks in advance. >> >> Kind regards, >> Sam Ezeh >> >> [1]: https://devguide.python.org/compiler/ >> [2]: https://bugs.python.org/issue41213 >> [3]: https://bugs.python.org/issue33675 > -- > https://mail.python.org/mailman/listinfo/python-list > From edmondo.giovannozzi at gmail.com Fri Jul 1 05:01:04 2022 From: edmondo.giovannozzi at gmail.com (Edmondo Giovannozzi) Date: Fri, 1 Jul 2022 02:01:04 -0700 (PDT) Subject: Byte arrays and DLLs In-Reply-To: References: <19eed94d-4f36-3171-7ce7-761850d5581a@btinternet.com> Message-ID: Il giorno venerd? 1 luglio 2022 alle 00:46:13 UTC+2 ery... at gmail.com ha scritto: > On 6/30/22, Rob Cliffe via Python-list wrote: > > > > AKAIK it is not possible to give ctypes a bytearray object and persuade > > it to give you a pointer to the actual array data, suitable for passing > > to a DLL. > You're overlooking the from_buffer() method. For example: > > >>> ba = bytearray(10) > >>> ca = (ctypes.c_char * len(ba)).from_buffer(ba) > >>> ca.value = b'spam&eggs' > >>> ba > bytearray(b'spam&eggs\x00') > > Note that the bytearray can't be resized while a view of the data is > exported. For example: > > >>> ba.append(97) > Traceback (most recent call last): > File "", line 1, in > BufferError: Existing exports of data: object cannot be re-sized > > >>> del ba[-1] > Traceback (most recent call last): > File "", line 1, in > BufferError: Existing exports of data: object cannot be re-sized Have you had a look at numpy (https://numpy.org/)? Typically, it is used for all scientific applications, supports several different kind of array, fast linear algebra, etc. And of course you can pass an array to a dynamic library with ctypes (https://numpy.org/doc/stable/reference/routines.ctypeslib.html). From rob.cliffe at btinternet.com Sun Jul 3 05:32:09 2022 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sun, 3 Jul 2022 10:32:09 +0100 Subject: Byte arrays and DLLs In-Reply-To: References: <19eed94d-4f36-3171-7ce7-761850d5581a@btinternet.com> Message-ID: That worked.? Many thanks Eryk. Rob On 30/06/2022 23:45, Eryk Sun wrote: > On 6/30/22, Rob Cliffe via Python-list wrote: >> AKAIK it is not possible to give ctypes a bytearray object and persuade >> it to give you a pointer to the actual array data, suitable for passing >> to a DLL. > You're overlooking the from_buffer() method. For example: > > >>> ba = bytearray(10) > >>> ca = (ctypes.c_char * len(ba)).from_buffer(ba) > >>> ca.value = b'spam&eggs' > >>> ba > bytearray(b'spam&eggs\x00') > > Note that the bytearray can't be resized while a view of the data is > exported. For example: > > >>> ba.append(97) > Traceback (most recent call last): > File "", line 1, in > BufferError: Existing exports of data: object cannot be re-sized > > >>> del ba[-1] > Traceback (most recent call last): > File "", line 1, in > BufferError: Existing exports of data: object cannot be re-sized From mark4asp at gmail.com Mon Jul 4 08:29:00 2022 From: mark4asp at gmail.com (Mark Pawelek) Date: Mon, 04 Jul 2022 13:29:00 +0100 Subject: Python installation References: Message-ID: <46n5chh5dtm7934g3a60bju8nd1115pvpt@4ax.com> I also have a windows installation issue on Windows 10: ISSUE: I cannot select a folder to install Python in. I want to put it under Program Files. The 'installer' only wants to put it in C:\users\Lenovo\AppData\local\Programs\Python\Python310 What do I do to alter the path to something like: C:\Programs\Python310 or C:\Program Files\Python310 ? The installer I'm using is: python-3.10.5-amd64.exe - which I downloaded today. RELEVANT INFO: It is a new Win10 PC, with i5-6500 CPU, and I added the user less than a week ago the first time I used the PC since I bought it on ebay (sold by a charity). Microsoft forced me to create a user. I had to enter an email address and (local) password. I have an online Microsoft account associated with that email address, so MS validated me by texting a code to my mobile which I entered at the PC. I didn't create the UserName: Lenovo. MS did [BTW: the PC 'Host Name' is something else, but the 'System Manufacturer' = Lenovo]. This Administrator user is the ONLY login user. The account details are: Lenovo Local User Administrator BTW: Python had already been installed 2 days ago when I modified features for another program. I deleted that install before trying this new one and YES - it was installed at: C:\users\Lenovo\AppData\local\Programs\Python\Python38 On Tue, 21 Jun 2022 14:22:29 +0300, Brian Karinga wrote: >Hello, > >I hope this email finds you well. > >I have been trying to download and install the latest version of python on >my windows device. However, when I run the program, three options arise. >These are: > >Modify >Repair >Uninstall > >I have executed the modify and repair options several times but nothing has >changed. Please advise on what the problem could be and how it can be >resolved. > >I look forward to hearing from you. > >Thank you, >Brian. From mats at wichmann.us Mon Jul 4 13:01:57 2022 From: mats at wichmann.us (Mats Wichmann) Date: Mon, 4 Jul 2022 11:01:57 -0600 Subject: Python installation In-Reply-To: <46n5chh5dtm7934g3a60bju8nd1115pvpt@4ax.com> References: <46n5chh5dtm7934g3a60bju8nd1115pvpt@4ax.com> Message-ID: On 7/4/22 06:29, Mark Pawelek wrote: > I also have a windows installation issue on Windows 10: > > ISSUE: I cannot select a folder to install Python in. I want to put > it under Program Files. The 'installer' only wants to put it in > C:\users\Lenovo\AppData\local\Programs\Python\Python310 > > What do I do to alter the path to something like: > C:\Programs\Python310 or C:\Program Files\Python310 ? > > The installer I'm using is: python-3.10.5-amd64.exe - which I > downloaded today. The second page of Advanced Options has a section to customize the install location. If some vestige of the old installation lives on, so the installer thinks it's modifying an existing install rather than a fresh one, that option would be greyed out. From wlfraed at ix.netcom.com Mon Jul 4 12:51:01 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Mon, 04 Jul 2022 12:51:01 -0400 Subject: Python installation References: <46n5chh5dtm7934g3a60bju8nd1115pvpt@4ax.com> Message-ID: On Mon, 04 Jul 2022 13:29:00 +0100, Mark Pawelek declaimed the following: > >ISSUE: I cannot select a folder to install Python in. I want to put >it under Program Files. The 'installer' only wants to put it in >C:\users\Lenovo\AppData\local\Programs\Python\Python310 > That indicates two things: One, you are logged in as "Lenovo" and Two, you are not running the installer with Admin privileges (or installed using the "current user" option). >What do I do to alter the path to something like: >C:\Programs\Python310 or C:\Program Files\Python310 ? > Install using admin privileges and select the "All Users" option. Though I'd recommend installing to a simple C:\Python310. If you install in the system default "Program Files" directory any updates (which may be PIP module installs) will need to be done under admin privileges. >RELEVANT INFO: It is a new Win10 PC, with i5-6500 CPU, and I added >the user less than a week ago the first time I used the PC since I >bought it on ebay (sold by a charity). Microsoft forced me to create a >user. I had to enter an email address and (local) password. I have an >online Microsoft account associated with that email address, so MS >validated me by texting a code to my mobile which I entered at the PC. >I didn't create the UserName: Lenovo. MS did [BTW: the PC 'Host Name' >is something else, but the 'System Manufacturer' = Lenovo]. This >Administrator user is the ONLY login user. The account details are: > Someone chose that name. If M$ were choosing user names based on manufacturer there would be thousands of "Lenovo" user names out there, and that would not be possible for a "M$ login" (it would be possible for "local machine" logins). Furthermore, one CAN set up Windows 10 WITHOUT creating a Microsoft Domain Login. Local login accounts can be created. The only reason I have a M$ login on my machine is that they may want it for their "app store" and things like Visual Studio -- my machine, however, is not logged in using the M$ account, just my local username. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From gweatherby at uchc.edu Tue Jul 5 14:47:45 2022 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Tue, 5 Jul 2022 18:47:45 +0000 Subject: Python 3.9 pidfd_open References: Message-ID: python introduced os.pidfd_open(), which works as documented. My development environment, PyCharm, complains about it being undefined. Should it be in https://raw.githubusercontent.com/python/cpython/3.9/Lib/os.py ? -- Gerard Weatherby | Application Architect NMRbox | NAN | Department of Molecular Biology and Biophysics | UConn Health 263 Farmington Avenue, Farmington, CT 06030-6406 uchc.edu From michael.stemper at gmail.com Thu Jul 7 14:26:51 2022 From: michael.stemper at gmail.com (Michael F. Stemper) Date: Thu, 7 Jul 2022 13:26:51 -0500 Subject: Why isn't there a built-in product()? Message-ID: sum() is wonderful. >>> nums = [1,2,3] >>> sum(nums) 6 >>> product(nums) Traceback (most recent call last): File "", line 1, in NameError: name 'product' is not defined >>> I understand that there is no product() or prod(). Does anybody here know why that was not included in the language? It seems as if it would be useful, so there must have been some rationale for that decision. -- Michael F. Stemper 87.3% of all statistics are made up by the person giving them. From oscar.j.benjamin at gmail.com Thu Jul 7 20:06:43 2022 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 8 Jul 2022 01:06:43 +0100 Subject: Why isn't there a built-in product()? In-Reply-To: References: Message-ID: On Thu, 7 Jul 2022 at 22:55, Michael F. Stemper wrote: > > sum() is wonderful. > > >>> nums = [1,2,3] > >>> sum(nums) > 6 > >>> product(nums) > Traceback (most recent call last): > File "", line 1, in > NameError: name 'product' is not defined > >>> > > I understand that there is no product() or prod(). Does anybody > here know why that was not included in the language? It seems > as if it would be useful, so there must have been some rationale > for that decision. There is math.prod: >>> from math import prod >>> prod([1, 2, 3, 4]) 24 https://docs.python.org/3/library/math.html#math.prod -- Oscar From barry at barrys-emacs.org Fri Jul 8 11:21:00 2022 From: barry at barrys-emacs.org (Barry Scott) Date: Fri, 8 Jul 2022 16:21:00 +0100 Subject: Python 3.9 pidfd_open In-Reply-To: References: Message-ID: <585EA9D9-7271-4810-90CF-DE148ADCA9C0@barrys-emacs.org> > On 5 Jul 2022, at 19:47, Weatherby,Gerard wrote: > > python introduced os.pidfd_open(), which works as documented. > > My development environment, PyCharm, complains about it being undefined. > > Should it be in https://raw.githubusercontent.com/python/cpython/3.9/Lib/os.py ? When faced with this type of problem I introspect from the Python REPL prompt. Run python 3.9 and see if os.pidfd_open is defined? Do not do this inside PyCharm to rule out a problem with PyCharm. Also you will need to be running a linux kernel + glibc version that is new enough to proved this to python. This is what I see with Fedora 36: $ python3.9 Python 3.9.13 (main, Jun 9 2022, 00:00:00) [GCC 12.1.1 20220507 (Red Hat 12.1.1-1)] on linux Type "help", "copyright", "credits" or "license" for more information. :>>> import os :>>> os.pidfd_open :>>> Barry > > -- > Gerard Weatherby | Application Architect > NMRbox | NAN | Department of Molecular Biology and Biophysics | UConn Health > 263 Farmington Avenue, Farmington, CT 06030-6406 > uchc.edu > -- > https://mail.python.org/mailman/listinfo/python-list > From aolima at est.itsgg.edu.ec Fri Jul 8 10:51:31 2022 From: aolima at est.itsgg.edu.ec (ANGIE ODETTE LIMA BANGUERA) Date: Fri, 8 Jul 2022 09:51:31 -0500 Subject: TENGO PROBLEMAS AL INSTALAR PYTHON Message-ID: buenos dias necesito ayuda, tengo problemas al momento de instalar python 10.0 en la aplicaci?n de visual studio code From nsh531 at gmail.com Fri Jul 8 09:06:09 2022 From: nsh531 at gmail.com (=?UTF-8?B?16DXqteZINep15jXqNef?=) Date: Fri, 8 Jul 2022 16:06:09 +0300 Subject: NILEARN - WHY THIS CODE THROWS AN ERROR????? Message-ID: fullcode: import nilearn.plotting as plot import os,gzip,io import nibabel as nib path="C:/users/administrator/desktop/nii" path2="C:/users/administrator/desktop/nii/out/" for i in os.listdir(path): if(".nii.gz" in i): pass else: if(".nii" in i): img = nib.load(path+"/"+i) data = img.get_fdata() print(data) import imageio X=0 for s in data: import numpy aleph=numpy.array(s,dtype=numpy.int8) X=X+1 plot.plot_img(aleph) imageio.imwrite("C:\\users\\administrator\\desktop\\nii\\"+i +str(X)+'.jpg', s) error: Data given cannot be loaded because it is not compatible with nibabel format From ikorot01 at gmail.com Fri Jul 8 16:15:36 2022 From: ikorot01 at gmail.com (Igor Korot) Date: Fri, 8 Jul 2022 15:15:36 -0500 Subject: TENGO PROBLEMAS AL INSTALAR PYTHON In-Reply-To: References: Message-ID: Hi, On Fri, Jul 8, 2022 at 2:54 PM ANGIE ODETTE LIMA BANGUERA wrote: > > buenos dias necesito ayuda, tengo problemas al momento de instalar python > 10.0 en la aplicaci?n de visual studio code Can you please describe what the problem is? Which python are you trying to install? Where did you get it from? Also - what platform are you using - Windows? Mac? *nix? Which version? Please post the error if you can't express it in words... Thank you. > -- > https://mail.python.org/mailman/listinfo/python-list From python at mrabarnett.plus.com Fri Jul 8 16:47:38 2022 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 8 Jul 2022 21:47:38 +0100 Subject: NILEARN - WHY THIS CODE THROWS AN ERROR????? In-Reply-To: References: Message-ID: <54a7e7f9-cf6e-2e22-d55e-d2221c9cf3b1@mrabarnett.plus.com> On 08/07/2022 14:06, ??? ???? wrote: > fullcode: > > > > import nilearn.plotting as plot > import os,gzip,io > import nibabel as nib > path="C:/users/administrator/desktop/nii" > path2="C:/users/administrator/desktop/nii/out/" > for i in os.listdir(path): > if(".nii.gz" in i): > pass > else: > > if(".nii" in i): > img = nib.load(path+"/"+i) > data = img.get_fdata() > print(data) > import imageio > X=0 > for s in data: > import numpy > aleph=numpy.array(s,dtype=numpy.int8) > X=X+1 > plot.plot_img(aleph) > > imageio.imwrite("C:\\users\\administrator\\desktop\\nii\\"+i > +str(X)+'.jpg', s) > > > > > > > error: > Data given cannot be loaded because it is not compatible with nibabel > format > What's the complete traceback? It might help you to identify the problem if you add messages to tell you what it's doing, e.g. what file it's about to load. Apparently, one of the files is not compatible with nibabel. From mats at wichmann.us Fri Jul 8 18:15:35 2022 From: mats at wichmann.us (Mats Wichmann) Date: Fri, 8 Jul 2022 16:15:35 -0600 Subject: TENGO PROBLEMAS AL INSTALAR PYTHON In-Reply-To: References: Message-ID: <1e9ba708-a0d6-2c61-1805-001d0bb2aa33@wichmann.us> On 7/8/22 14:15, Igor Korot wrote: > Hi, > > On Fri, Jul 8, 2022 at 2:54 PM ANGIE ODETTE LIMA BANGUERA > wrote: >> >> buenos dias necesito ayuda, tengo problemas al momento de instalar python >> 10.0 en la aplicaci?n de visual studio code > > Can you please describe what the problem is? > Which python are you trying to install? Where did you get it from? > Also - what platform are you using - Windows? Mac? *nix? Which version? > > Please post the error if you can't express it in words... > > Thank you. > In addition... there is no "Python 10.0" ... From avigross at verizon.net Fri Jul 8 18:20:39 2022 From: avigross at verizon.net (Avi Gross) Date: Fri, 8 Jul 2022 22:20:39 +0000 (UTC) Subject: NILEARN - WHY THIS CODE THROWS AN ERROR????? In-Reply-To: <54a7e7f9-cf6e-2e22-d55e-d2221c9cf3b1@mrabarnett.plus.com> References: <54a7e7f9-cf6e-2e22-d55e-d2221c9cf3b1@mrabarnett.plus.com> Message-ID: <461810708.1529294.1657318839437@mail.yahoo.com> Nati Stern has asked several questions here, often about relatively technical uses of python code that?many of us have never used and still is not providing more exact info that tends to be needed before anyone?can even think of diagnosing the problem. I have learned to stay away from some such questioners. But I am wondering if some people (others too)?think this forum is a generalized help desk staffed by College Professors with nothing else to do. Many questions are best handled locally where people can look over your shoulder or use the same software?and may have some fluency in your native language. And sometimes you need to do more investigating on?your own, and perhaps tell us what you tried and why it was not useful, or we end up making endless?suggestions and being told we are not working on your real issue and so on. The code below is just babel or maybe babble. Something nested in a loop had a problem. Why not?try something drastic and look at the? files and PICK ONE and use it step by step and see when it fails? It looks like the code wants to ask for all files then ignore some.? Why you would import numpy repeatedly in a loop is beyond me! LOL! But which command line failed? My GUESS is: data = img.get_fdata() If so, did you try to see the current value of the filename you call "i" in the loop and see what name was loaded in?what looks like a file ending in .nii in this code: img = nib.load(path+"/"+i) You need to proceed step by step and see if any previous steps failed.? But what is possible is you got a file with .nii in middle of the name that does not?end in .gz, or is not in the format needed. Good luck, ??? ???? -----Original Message----- From: MRAB To: python-list at python.org Sent: Fri, Jul 8, 2022 4:47 pm Subject: Re: NILEARN - WHY THIS CODE THROWS AN ERROR????? On 08/07/2022 14:06, ??? ???? wrote: > fullcode: > > > > import nilearn.plotting as plot > import os,gzip,io > import nibabel as nib > path="C:/users/administrator/desktop/nii" > path2="C:/users/administrator/desktop/nii/out/" > for i in os.listdir(path): >? ? ? if(".nii.gz" in i): >? ? ? ? ? pass >? ? ? else: > >? ? ? ? ? if(".nii" in i): >? ? ? ? ? ? ? img = nib.load(path+"/"+i) >? ? ? ? ? ? ? data = img.get_fdata() >? ? ? ? ? ? ? print(data) >? ? ? ? ? ? ? import imageio >? ? ? ? ? ? ? X=0 >? ? ? ? ? ? ? for s in data: >? ? ? ? ? ? ? ? ? import numpy >? ? ? ? ? ? ? ? ? aleph=numpy.array(s,dtype=numpy.int8) >? ? ? ? ? ? ? ? ? X=X+1 >? ? ? ? ? ? ? ? ? plot.plot_img(aleph) > >? ? ? ? ? ? ? ? ? imageio.imwrite("C:\\users\\administrator\\desktop\\nii\\"+i > +str(X)+'.jpg', s) > > > > > > > error: > Data given cannot be loaded because it is not compatible with nibabel > format > What's the complete traceback? It might help you to identify the problem if you add messages to tell you what it's doing, e.g. what file it's about to load. Apparently, one of the files is not compatible with nibabel. -- https://mail.python.org/mailman/listinfo/python-list From martinp.dipaola at gmail.com Fri Jul 8 19:07:52 2022 From: martinp.dipaola at gmail.com (Martin Di Paola) Date: Fri, 8 Jul 2022 20:07:52 -0300 Subject: TENGO PROBLEMAS AL INSTALAR PYTHON In-Reply-To: <1e9ba708-a0d6-2c61-1805-001d0bb2aa33@wichmann.us> References: <1e9ba708-a0d6-2c61-1805-001d0bb2aa33@wichmann.us> Message-ID: <20220708230752.zhltvf3ncg2tyk4m@gmail.com> On Fri, Jul 08, 2022 at 04:15:35PM -0600, Mats Wichmann wrote: > >In addition... there is no "Python 10.0" ... > Mmm, perhaps that's the problem :D @Angie Odette Lima Banguera, vamos a necesitar algun traceback o algo para guiarte. Podes tambien buscar en internet (youtube) q hay varios tutoriales para los primeros pasos. Good luck! From python at mrabarnett.plus.com Fri Jul 8 20:51:44 2022 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 9 Jul 2022 01:51:44 +0100 Subject: NILEARN - WHY THIS CODE THROWS AN ERROR????? In-Reply-To: <461810708.1529294.1657318839437@mail.yahoo.com> References: <54a7e7f9-cf6e-2e22-d55e-d2221c9cf3b1@mrabarnett.plus.com> <461810708.1529294.1657318839437@mail.yahoo.com> Message-ID: On 08/07/2022 23:20, Avi Gross via Python-list wrote: > Nati Stern has asked several questions here, often about relatively technical uses of python code that?many of us have never used and still is not providing more exact info that tends to be needed before anyone?can even think of diagnosing the problem. > > I have learned to stay away from some such questioners. But I am wondering if some people (others too)?think this forum is a generalized help desk staffed by College Professors with nothing else to do. > > Many questions are best handled locally where people can look over your shoulder or use the same software?and may have some fluency in your native language. And sometimes you need to do more investigating on?your own, and perhaps tell us what you tried and why it was not useful, or we end up making endless?suggestions and being told we are not working on your real issue and so on. > > The code below is just babel or maybe babble. Something nested in a loop had a problem. Why not?try something drastic and look at the? files and PICK ONE and use it step by step and see when it fails? > > It looks like the code wants to ask for all files then ignore some. > > Why you would import numpy repeatedly in a loop is beyond me! LOL! > > But which command line failed? My GUESS is: > > data = img.get_fdata() > > > If so, did you try to see the current value of the filename you call "i" in the loop and see what name was loaded in?what looks like a file ending in .nii in this code: > > img = nib.load(path+"/"+i) > > > You need to proceed step by step and see if any previous steps failed. > But what is possible is you got a file with .nii in middle of the name that does not?end in .gz, or is not in the format needed. Indeed, it writes JPEG files whose filename contains the original filename (with the ".nii") into the same folder, so if it has already been run and produced an output file, the next time it's run, it'll trip itself up. All this would've been clear to the OP if it had printed messages as it went. > Good luck, > ??? ???? > > -----Original Message----- > From: MRAB > To: python-list at python.org > Sent: Fri, Jul 8, 2022 4:47 pm > Subject: Re: NILEARN - WHY THIS CODE THROWS AN ERROR????? > > On 08/07/2022 14:06, ??? ???? wrote: >> fullcode: >> >> >> >> import nilearn.plotting as plot >> import os,gzip,io >> import nibabel as nib >> path="C:/users/administrator/desktop/nii" >> path2="C:/users/administrator/desktop/nii/out/" >> for i in os.listdir(path): >>? ? ? if(".nii.gz" in i): >>? ? ? ? ? pass >>? ? ? else: >> >>? ? ? ? ? if(".nii" in i): >>? ? ? ? ? ? ? img = nib.load(path+"/"+i) >>? ? ? ? ? ? ? data = img.get_fdata() >>? ? ? ? ? ? ? print(data) >>? ? ? ? ? ? ? import imageio >>? ? ? ? ? ? ? X=0 >>? ? ? ? ? ? ? for s in data: >>? ? ? ? ? ? ? ? ? import numpy >>? ? ? ? ? ? ? ? ? aleph=numpy.array(s,dtype=numpy.int8) >>? ? ? ? ? ? ? ? ? X=X+1 >>? ? ? ? ? ? ? ? ? plot.plot_img(aleph) >> >>? ? ? ? ? ? ? ? ? imageio.imwrite("C:\\users\\administrator\\desktop\\nii\\"+i >> +str(X)+'.jpg', s) >> >> >> >> >> >> >> error: >> Data given cannot be loaded because it is not compatible with nibabel >> format >> > > What's the complete traceback? > > It might help you to identify the problem if you add messages to tell > you what it's doing, e.g. what file it's about to load. > > Apparently, one of the files is not compatible with nibabel. From rosuav at gmail.com Fri Jul 8 20:58:29 2022 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 9 Jul 2022 10:58:29 +1000 Subject: NILEARN - WHY THIS CODE THROWS AN ERROR????? In-Reply-To: References: <54a7e7f9-cf6e-2e22-d55e-d2221c9cf3b1@mrabarnett.plus.com> <461810708.1529294.1657318839437@mail.yahoo.com> Message-ID: On Sat, 9 Jul 2022 at 10:57, MRAB wrote: > > On 08/07/2022 23:20, Avi Gross via Python-list wrote: > > Nati Stern has asked several questions here, often about relatively technical uses of python code that many of us have never used and still is not providing more exact info that tends to be needed before anyone can even think of diagnosing the problem. > > > > I have learned to stay away from some such questioners. But I am wondering if some people (others too) think this forum is a generalized help desk staffed by College Professors with nothing else to do. > > > > Many questions are best handled locally where people can look over your shoulder or use the same software and may have some fluency in your native language. And sometimes you need to do more investigating on your own, and perhaps tell us what you tried and why it was not useful, or we end up making endless suggestions and being told we are not working on your real issue and so on. > > > > The code below is just babel or maybe babble. Something nested in a loop had a problem. Why not try something drastic and look at the files and PICK ONE and use it step by step and see when it fails? > > > > It looks like the code wants to ask for all files then ignore some. > > > > Why you would import numpy repeatedly in a loop is beyond me! LOL! > > > > But which command line failed? My GUESS is: > > > > data = img.get_fdata() > > > > > > If so, did you try to see the current value of the filename you call "i" in the loop and see what name was loaded in what looks like a file ending in .nii in this code: > > > > img = nib.load(path+"/"+i) > > > > > > You need to proceed step by step and see if any previous steps failed. > > But what is possible is you got a file with .nii in middle of the name that does not end in .gz, or is not in the format needed. > > Indeed, it writes JPEG files whose filename contains the original > filename (with the ".nii") into the same folder, so if it has already > been run and produced an output file, the next time it's run, it'll trip > itself up. > > All this would've been clear to the OP if it had printed messages as it > went. Or if the OP had renamed them all to "shrubbery" along the way. ChrisA From mbandezi at gmail.com Sat Jul 9 06:08:55 2022 From: mbandezi at gmail.com (Mxolisi Mbandezi) Date: Sat, 9 Jul 2022 12:08:55 +0200 Subject: Include mailing list Message-ID: Dear Python developers Please,include me in your mailing list. Kind regards Mxolisi -- Mr. M.L. Mbandezi Faculty of Applied and Computer Sciences Department of Physics and Non-Destructive Testing Vaal University Of Technology Vanderbijl Park x021 Gauteng Province Cell: +27 797060639 Tel (W): +27 016 9509824 Fax: 0866516744 Email: Mxolisi at vut.ac.za ----------------------------------------------------------------------------------- " The hottest place in hell is reserved for those who remain neutral in times of great moral conflict" --------------------------------Martin Luther King.Jr------------------------------ From michael.stemper at gmail.com Sat Jul 9 09:25:40 2022 From: michael.stemper at gmail.com (Michael F. Stemper) Date: Sat, 9 Jul 2022 08:25:40 -0500 Subject: Why isn't there a built-in product()? In-Reply-To: References: Message-ID: On 07/07/2022 19.06, Oscar Benjamin wrote: > On Thu, 7 Jul 2022 at 22:55, Michael F. Stemper > wrote: >> >> sum() is wonderful. >> I understand that there is no product() or prod(). Does anybody >> here know why that was not included in the language? It seems >> as if it would be useful, so there must have been some rationale >> for that decision. > > There is math.prod: > > >>> from math import prod > >>> prod([1, 2, 3, 4]) > 24 > > https://docs.python.org/3/library/math.html#math.prod I did not know that. Thanks. -- Michael F. Stemper Outside of a dog, a book is man's best friend. Inside of a dog, it's too dark to read. From python at mrabarnett.plus.com Sat Jul 9 17:49:56 2022 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 9 Jul 2022 22:49:56 +0100 Subject: Include mailing list In-Reply-To: References: Message-ID: On 09/07/2022 11:08, Mxolisi Mbandezi wrote: > Dear Python developers > Please,include me in your mailing list. > Kind regards > Mxolisi > This tells you how to subscribe to this list: https://mail.python.org/mailman/listinfo/python-list From PythonList at DancesWithMice.info Sat Jul 9 17:51:07 2022 From: PythonList at DancesWithMice.info (dn) Date: Sun, 10 Jul 2022 09:51:07 +1200 Subject: Include mailing list In-Reply-To: References: Message-ID: <7734a98c-fec4-22ca-6678-cdfd241893b7@DancesWithMice.info> Dear Mxolisi, On 09/07/2022 22.08, Mxolisi Mbandezi wrote: > Dear Python developers > Please,include me in your mailing list. Please follow the instructions, at the link which appears below:- -- Regards, =dn From mwalmasri2003 at gmail.com Sun Jul 10 11:46:26 2022 From: mwalmasri2003 at gmail.com (Walid AlMasri) Date: Sun, 10 Jul 2022 18:46:26 +0300 Subject: Job vacancies Message-ID: Dear Python-list, Hope my email finds you well I am a postdoc in theoretical physics I has been working on python since around 3 years where I use Jupyter notebooks and pycharm in my physics research I would like to ask if there are any openings that suit my background? I want to work on python either remotely or at job site I attach my CV Thanks a lot in advance and all the best wishes to you Best Regards, Walid From pablogsal at gmail.com Mon Jul 11 14:44:15 2022 From: pablogsal at gmail.com (Pablo Galindo Salgado) Date: Mon, 11 Jul 2022 19:44:15 +0100 Subject: [RELEASE] The cursed fourth Python 3.11 beta (3.11.0b4) is available Message-ID: I cannot believe I am writing this, but Python 3.11.b4 is finally available!! [image: :scream:] [image: :tada:] [image: :tada:] [image: :tada:] https://www.python.org/downloads/release/python-3110b4/ ##[image: :warning:][image: :warning:][image: :warning:] PLEASE HELP US TO TEST THIS RELEASE [image: :warning:][image: :warning:][image: :warning:] Due to the modified release schedule and the stability concerns regarding the past beta releases, please, please, please, please, help us to test Python 3.11 by testing this beta releases. * if you maintain a library or a third-party package. Test the beta releases! * If you have code that you maintain at work/research centre/classroom/whatever. Test the beta releases! * If you are a multi-million corporation that uses Python. Test the beta releases! * If you are a single-person company that uses Python. Test the beta releases! * If you have a bunch of Python scripts. Test the beta releases! * If you use Python for work, research, teaching or literally for anything. Test the beta releases! * If you ... In summary: no matter who you are or what you do. Test the beta releases! Is **very** important for us that we identify all possible things that may break your code **before** the final release is done and we can only do this if you help us by testing the beta releases and then report anything that doesn't work! ## Credit where credit is due [image: :pray:] Lots of thanks to Christian Heimes, Brandt Bucher, Irit Katriel, Mark Shannon, Dennis Sweeney, Kumar Aditya and other contributors (sorry if I am missing any names) that worked really hard against time to help me and the release team with the release blockers. They are all awesome and we and the Python community are very lucky to have them in the team [image: :heart:] ## What happens with the next betas? As stated in [my previous communication]( https://mail.python.org/archives/list/python-dev at python.org/thread/3JWVCSBPBFWY5ZWSJ7RYB6FS5NIMCEOY/) we are in a special situation regarding beta releases. As the requirements to continue with the regular schedule are met, we are going to still target the final release of Monday, 2022-10-03. Python 3.11.0b5 was supposed to be released two days ago, so we are obviously delayed. As we are targeting the regular release schedule, I'm going to try to release 3.11.0b5 on Thursday, 2022-07-25. ## This is a beta preview of Python 3.11 Python 3.11 is still in development. 3.11.0b4 is the fourth of five planned beta release previews. Beta release previews are intended to give the wider community the opportunity to test new features and bug fixes and to prepare their projects to support the new feature release. We **strongly encourage** maintainers of third-party Python projects to **test with 3.11** during the beta phase and report issues found to [the Python bug tracker](https://github.com/python/cpython/issues) as soon as possible. While the release is planned to be feature complete entering the beta phase, it is possible that features may be modified or, in rare cases, deleted up until the start of the release candidate phase (Monday, 2021-08-02). Our goal is have no ABI changes after beta 5 and as few code changes as possible after 3.11.0rc1, the first release candidate. To achieve that, it will be **extremely important** to get as much exposure for 3.11 as possible during the beta phase. Please keep in mind that this is a preview release and its use is **not** recommended for production environments. # Major new features of the 3.11 series, compared to 3.10 Some of the new major new features and changes in Python 3.11 are: ## General changes * [PEP 657](https://www.python.org/dev/peps/pep-0657/) -- Include Fine-Grained Error Locations in Tracebacks * [PEP 654](https://www.python.org/dev/peps/pep-0654/) -- Exception Groups and except* * [PEP 680](https://www.python.org/dev/peps/pep-0680/)-- tomllib: Support for Parsing TOML in the Standard Library * [PEP 681](https://www.python.org/dev/peps/pep-0681/)-- Data Class Transforms * [bpo-46752](https://bugs.python.org/issue46752)-- Introduce task groups to asyncio * [bpo-433030](https://github.com/python/cpython/issues/34627/) -- Atomic grouping ((?>...)) and possessive quantifiers (`*+, ++, ?+, {m,n}+`) are now supported in regular expressions. * The [Faster Cpython Project](https://github.com/faster-cpython/) is already yielding some exciting results. Python 3.11 is up to 10-60% faster than Python 3.10. On average, we measured a 1.22x speedup on the standard benchmark suite. See [Faster CPython]( https://docs.python.org/3.11/whatsnew/3.11.html#faster-cpython) for details. ## Typing and typing language changes * [PEP 673](https://www.python.org/dev/peps/pep-0673/) -- Self Type * [PEP 646](https://www.python.org/dev/peps/pep-0646/)-- Variadic Generics * [PEP 675](https://www.python.org/dev/peps/pep-0675/)-- Arbitrary Literal String Type * [PEP 655](https://www.python.org/dev/peps/pep-0655/)-- Marking individual TypedDict items as required or potentially-missing (Hey, **fellow core developer,** if a feature you find important is missing from this list, [let Pablo know](mailto:pablogsal at python.org ).) The next pre-release of Python 3.11 will be 3.11.0b5, currently scheduled for Thursday, 2022-07-25. # More resources * [Online Documentation](https://docs.python.org/3.11/) * [PEP 664](https://www.python.org/dev/peps/pep-0664/), 3.11 Release Schedule * Report bugs at [https://bugs.python.org](https://bugs.python.org). * [Help fund Python and its community](/psf/donations/). # And now for something completely different The Planck temperature is 1.416784?10**32 K. At this temperature, the wavelength of light emitted by thermal radiation reaches the Planck length. There are no known physical models able to describe temperatures greater than the Planck temperature and a quantum theory of gravity would be required to model the extreme energies attained. Hypothetically, a system in thermal equilibrium at the Planck temperature might contain Planck-scale black holes, constantly being formed from thermal radiation and decaying via Hawking evaporation; adding energy to such a system might decrease its temperature by creating larger black holes, whose Hawking temperature is lower. Rumours say the Planck temperature can be reached in some of the hottest parts of Spain in summer. # We hope you enjoy the new releases! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. https://www.python.org/psf/ If you have any questions, please reach out to me or another member of the release team :) Your friendly release team, Ned Deily @nad https://discuss.python.org/u/nad Steve Dower @steve.dower https://discuss.python.org/u/steve.dower Pablo Galindo Salgado @pablogsal https://discuss.python.org/u/pablogsal From pablogsal at gmail.com Mon Jul 11 15:31:02 2022 From: pablogsal at gmail.com (Pablo Galindo Salgado) Date: Mon, 11 Jul 2022 20:31:02 +0100 Subject: [RELEASE] The cursed fourth Python 3.11 beta (3.11.0b4) is available In-Reply-To: References: Message-ID: BSD-style checksum format hashes for the release artefacts: SHA256 (python-3.11.0b4-embed-arm64.zip) = 272c6bb4948c597f6578f64c2b15a70466c5dfb49f9b84dba57a84e59e7bd4ef SHA256 (python-3.11.0b4-amd64.exe) = a3514b0401e6a85416f3e080586c86ccd9e2e62c8a54b9119d9e6415e3cadb62 SHA256 (python-3.11.0b4-macos11.pkg) = 860647775d4e6cd1a8d71412233df5dbe3aa2886fc16d82a59ab2f625464f2d7 SHA256 (python-3.11.0b4-embed-win32.zip) = 36b81da7986f8d59be61adb452681dbd3257ebb90bd89092b2fbbd9356e06425 SHA256 (python-3.11.0b4-arm64.exe) = ad0d1429682ba1edc0c0cf87f68a3d1319b887b715da70a91db41d02be4997a4 SHA256 (python-3.11.0b4-embed-amd64.zip) = 66e6bb44c36da36ecc1de64efdb92f52ba3a19221dba2a89e22e39f715bd205b SHA256 (Python-3.11.0b4.tar.xz) = 1d93b611607903e080417c1a9567f5fbbf5124cc5c86f4afbba1c8fd34c5f6fb SHA256 (python-3.11.0b4.exe) = 6febc152711840337f53e2fd5dc12bb2b1314766f591129282fd372c855fa877 SHA256 (Python-3.11.0b4.tgz) = 257e753db2294794fa8dec072c228f3f53fd541a303de9418854b3c2512ccbec From nsh531 at gmail.com Tue Jul 12 08:37:55 2022 From: nsh531 at gmail.com (=?UTF-8?B?16DXqteZINep15jXqNef?=) Date: Tue, 12 Jul 2022 15:37:55 +0300 Subject: calculate diff between dates Message-ID: code: @app.route("/hosts/last_seen") def hst(): #F1=falconpy #rr=struct_time() #rr.tm_hour=23 #F= falconpy.APIHarness(client_id=client_id,client_secret=secret) #id_list = 'ID1,ID2,ID3' # Can also pass a list here: ['ID1', 'ID2', 'ID3'] #response = F1.Hosts(client_id=client_id,client_secret=secret) d=datetime.datetime.fromtimestamp((datetime.datetime.now().utcnow(). timestamp()-86400*1.4)).isoformat() # F.command() # lasts=response.query_devices_by_filter(sort="hostname.asc") # return response.query_devices_by_filter(filter=f"hostname:{myid}*") #.QueryDevicesByFilter(filter=f"host:{myid}*") # ) j=json2html.Json2Html #f.close() # hostname_result = hosts.query_devices_by_filter(filter=f"" response = dict(a=falconpy.APIHarness(client_id=client_id,client_secret= secret).command("GetDeviceDetails", ids=list_of_aid)) # t=response["b"] # t=falconpy.APIHarness(client_id=client_id,client_secret=secret).command("getHostGroups", ids=response["a"]["body"]["resources"][0]["groups"]) #j=falconpy.APIHarness(client_id=client_id,client_secret=secret).command("query_hosts",filter='8d9285525ada4736bbd9368a2006f0ec') to=[] tc=list(range(len(response['a']['body']["resources"]))) for i in response['a']['body']["resources"]: to.append((datetime.datetime.today())-datetime.datetime(). fromisoformat(str(i["last_seen"]))) return dict(x=to,y=tc) I glad for any help From 2QdxY4RzWzUUiLuE at potatochowder.com Tue Jul 12 09:11:25 2022 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Tue, 12 Jul 2022 08:11:25 -0500 Subject: calculate diff between dates In-Reply-To: References: Message-ID: On 2022-07-12 at 15:37:55 +0300, ??? ???? wrote: > I glad for any help http://www.sscce.org/ From uri at speedy.net Tue Jul 12 13:39:29 2022 From: uri at speedy.net (=?UTF-8?B?15DXldeo15k=?=) Date: Tue, 12 Jul 2022 20:39:29 +0300 Subject: python -W Message-ID: Is `python -W all` and `python -Wa` the same? And what are the options for `python -W`? I didn't find it documented. For example, I'm using `python -W error::DeprecationWarning` and I didn't see it documented on https://docs.python.org/3/using/cmdline.html Is there a way to convert all warnings to exceptions and how? Thanks, ???? uri at speedy.net From python at mrabarnett.plus.com Tue Jul 12 15:05:32 2022 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 12 Jul 2022 20:05:32 +0100 Subject: python -W In-Reply-To: References: Message-ID: <162f7fff-6d1e-0f33-e783-6535a9cb229e@mrabarnett.plus.com> On 12/07/2022 18:39, ???? wrote: > Is `python -W all` and `python -Wa` the same? And what are the options > for `python -W`? I didn't find it documented. For example, I'm using > `python -W error::DeprecationWarning` and I didn't see it documented on > https://docs.python.org/3/using/cmdline.html > > Is there a way to convert all warnings to exceptions and how? > It's documented on that page. It says """The action names can be abbreviated as desired and the interpreter will resolve them to the appropriate action name. For example, -Wi is the same as -Wignore."", so "python -Wa" is the same as "python -Walways". It also says """-Werror # Convert to exceptions""", so "python -Werror" (or "python -We") will convert all warnings to exceptions. From uri at speedy.net Tue Jul 12 19:04:41 2022 From: uri at speedy.net (=?UTF-8?B?15DXldeo15k=?=) Date: Wed, 13 Jul 2022 02:04:41 +0300 Subject: python -W In-Reply-To: <162f7fff-6d1e-0f33-e783-6535a9cb229e@mrabarnett.plus.com> References: <162f7fff-6d1e-0f33-e783-6535a9cb229e@mrabarnett.plus.com> Message-ID: Thank you. ???? uri at speedy.net On Tue, Jul 12, 2022 at 10:09 PM MRAB wrote: > On 12/07/2022 18:39, ???? wrote: > > Is `python -W all` and `python -Wa` the same? And what are the options > > for `python -W`? I didn't find it documented. For example, I'm using > > `python -W error::DeprecationWarning` and I didn't see it documented on > > https://docs.python.org/3/using/cmdline.html > > > > Is there a way to convert all warnings to exceptions and how? > > > It's documented on that page. > > It says """The action names can be abbreviated as desired and the > interpreter will resolve them to the appropriate action name. For > example, -Wi is the same as -Wignore."", so "python -Wa" is the same as > "python -Walways". > > It also says """-Werror # Convert to exceptions""", so "python > -Werror" (or "python -We") will convert all warnings to exceptions. > -- > https://mail.python.org/mailman/listinfo/python-list > From roel at roelschroeven.net Wed Jul 13 06:50:57 2022 From: roel at roelschroeven.net (Roel Schroeven) Date: Wed, 13 Jul 2022 12:50:57 +0200 Subject: calculate diff between dates In-Reply-To: References: Message-ID: Op 12/07/2022 om 14:37 schreef ??? ????: > I glad for any help http://www.catb.org/~esr/faqs/smart-questions.html -- "Binnen een begrensde ruimte ligt een kritiek punt, waar voorbij de vrijheid afneemt naarmate het aantal individuen stijgt. Dit gaat evenzeer op voor mensen in de begrensde ruimte van een planetair ecosysteem, als voor de gasmoleculen in een hermetisch gesloten vat. Bij mensen is het niet de vraag hoeveel er maximaal in leven kunnen blijven in het systeem, maar wat voor soort bestaan mogelijk is voor diegenen die in leven blijven. -- Pardot Kynes, eerste planetoloog van Arrakis" -- Frank Herbert, Duin From nsh531 at gmail.com Wed Jul 13 14:35:18 2022 From: nsh531 at gmail.com (=?UTF-8?B?16DXqteZINep15jXqNef?=) Date: Wed, 13 Jul 2022 21:35:18 +0300 Subject: what's the problem?????? In-Reply-To: References: Message-ID: I want to set dict ?????? ??? ??, 13 ????? 2022, 20:47, ??? ??? ???? ?: > CODE: > > for nii in os.listdir("c:/users/administrator/desktop/nii"): > > from nilearn import plotting > from nilearn import datasets > atlas = datasets.fetch_atlas_msdl() > # Loading atlas image stored in 'maps' > atlas_filename = > "C:/Users/Administrator/Desktop/64/64/2mm/maps.nii.gz" > # Loading atlas data stored in 'labels' > labels = pd.read_csv( > "C:/Users/Administrator/Desktop/64/64/labels_64_dictionary.csv") > a=labels.to_dict() > b=a["Difumo_names"] > from nilearn.maskers import NiftiMapsMasker > masker = NiftiMapsMasker(maps_img=atlas_filename, standardize=True, > memory='nilearn_cache', verbose=5) > > time_series = masker.fit_transform( > "c:/users/administrator/desktop/nii/"+nii) > try: > from sklearn.covariance import GraphicalLassoCV > except ImportError: > # for Scitkit-Learn < v0.20.0 > from sklearn.covariance import GraphLassoCV as GraphicalLassoCV > > estimator = GraphicalLassoCV() > estimator.fit(time_series) > # Display the covariancec > aas={} > jsa=0 > for i in estimator.covariance_: > r=list(a["Difumo_names"].values())[jsa] > jsa=jsa+1 > a=dict() > > > for x in range(64): > g=list(a["Difumo_names"].values())[x] > > print(aas) > t= nilearn.plotting.plot_img(estimator.covariance_, labels=list(a[ > "Difumo_names"].values()), > figure=(9, 7), vmax=1, vmin=-1, > title='Covariance')# The covariance can be found > at estimator.covariance_ > > # The covariance can be found at estimator.covariance_ > t2= nilearn.plotting.plot_matrix(estimator.covariance_, labels=list(a > ["Difumo_names"].values()), > figure=(9, 7), vmax=1, vmin=-1, > title='Covariance') > > > > -- > > From avigross at verizon.net Wed Jul 13 15:01:45 2022 From: avigross at verizon.net (Avi Gross) Date: Wed, 13 Jul 2022 19:01:45 +0000 (UTC) Subject: what's the problem?????? In-Reply-To: References: Message-ID: <2061114234.599716.1657738905183@mail.yahoo.com> Nati, This is a two-way process and requires you to be very clear on what is not working or what you are trying to do or help clear away having us try to understand lots of code that is not very related to the question. Your code, as shown, makes an empty string repeatedly in a loop.? a=dict() I am guessing the code there works fine and does nothing useful. Well, what do you want in your dictionary??Most people create a dictionary outside the loop as an empty dictionary and then in the loop use one of many methods?to add key:value pairs. Earlier in the code you had another line: a=labels.to_dict() If the labels variable had a method like that, that is also a way. So be specific about what LINE or region of code and what is wrong and what you already tried or error messages you got. Avi -----Original Message----- From: ??? ???? To: Neuroimaging analysis in Python ; python-list at python.org Sent: Wed, Jul 13, 2022 2:35 pm Subject: Re: what's the problem?????? I want to set dict ?????? ??? ??, 13 ????? 2022, 20:47, ??? ??? ???? ?: > CODE: > > for nii in os.listdir("c:/users/administrator/desktop/nii"): > >? ? from nilearn import plotting >? ? from nilearn import datasets >? ? atlas = datasets.fetch_atlas_msdl() >? ? # Loading atlas image stored in 'maps' >? ? atlas_filename = > "C:/Users/Administrator/Desktop/64/64/2mm/maps.nii.gz" >? ? # Loading atlas data stored in 'labels' >? ? labels = pd.read_csv( > "C:/Users/Administrator/Desktop/64/64/labels_64_dictionary.csv") >? ? a=labels.to_dict() >? ? b=a["Difumo_names"] >? ? from nilearn.maskers import NiftiMapsMasker >? ? masker = NiftiMapsMasker(maps_img=atlas_filename, standardize=True, >? ? ? ? ? ? ? ? ? ? ? ? ? ? memory='nilearn_cache', verbose=5) > >? ? time_series = masker.fit_transform( > "c:/users/administrator/desktop/nii/"+nii) >? ? try: >? ? ? ? from sklearn.covariance import GraphicalLassoCV >? ? except ImportError: >? ? ? ? # for Scitkit-Learn < v0.20.0 >? ? ? ? from sklearn.covariance import GraphLassoCV as GraphicalLassoCV > >? ? estimator = GraphicalLassoCV() >? ? estimator.fit(time_series) > # Display the covariancec >? ? aas={} >? ? jsa=0 >? ? for i in estimator.covariance_: >? ? ? ? r=list(a["Difumo_names"].values())[jsa] >? ? ? ? jsa=jsa+1 >? ? ? ? a=dict() > > >? ? ? ? for x in range(64): >? ? ? ? ? ? g=list(a["Difumo_names"].values())[x] > >? ? print(aas) >? ? t=? nilearn.plotting.plot_img(estimator.covariance_, labels=list(a[ > "Difumo_names"].values()), >? ? ? ? ? ? ? ? ? ? ? ? figure=(9, 7), vmax=1, vmin=-1, >? ? ? ? ? ? ? ? ? ? ? ? title='Covariance')# The covariance can be found > at estimator.covariance_ > > # The covariance can be found at estimator.covariance_ >? ? t2=? nilearn.plotting.plot_matrix(estimator.covariance_, labels=list(a > ["Difumo_names"].values()), >? ? ? ? ? ? ? ? ? ? ? ? figure=(9, 7), vmax=1, vmin=-1, >? ? ? ? ? ? ? ? ? ? ? ? title='Covariance') > > > > -- > > -- https://mail.python.org/mailman/listinfo/python-list From nsh531 at gmail.com Thu Jul 14 00:18:34 2022 From: nsh531 at gmail.com (=?UTF-8?B?16DXqteZINep15jXqNef?=) Date: Thu, 14 Jul 2022 07:18:34 +0300 Subject: [Neuroimaging] what's the problem?????? In-Reply-To: References: Message-ID: I want to swap indexes of dict ?????? ??? ??, 13 ????? 2022, 21:51, ??? David Welch ?< david.m.welch at gmail.com>: > >> - Describe the research *you did* to try and understand the problem >> *before* you asked the question. >> - Describe the diagnostic steps *you took* to try and pin down the >> problem yourself *before* you asked the question. >> >> > On Wed, Jul 13, 2022 at 1:47 PM David Welch > wrote: > >> From http://catb.org/~esr/faqs/smart-questions.html#beprecise: >> >>> Be precise and informative about your problem >>> >>> - >>> >>> Describe the symptoms of your problem or bug carefully and clearly. >>> - >>> >>> Describe the environment in which it occurs (machine, OS, >>> application, whatever). Provide your vendor's distribution and release >>> level (e.g.: ?Fedora Core 7?, ?Slackware 9.1?, etc.). >>> - >>> >>> Describe the research you did to try and understand the problem >>> before you asked the question. >>> - >>> >>> Describe the diagnostic steps you took to try and pin down the >>> problem yourself before you asked the question. >>> - >>> >>> Describe any possibly relevant recent changes in your computer or >>> software configuration. >>> - >>> >>> If at all possible, provide a way to *reproduce the problem in a >>> controlled environment*. >>> >>> Do the best you can to anticipate the questions a hacker will ask, and >>> answer them in advance in your request for help. >>> >>> Giving hackers the ability to reproduce the problem in a controlled >>> environment is especially important if you are reporting something you >>> think is a bug in code. When you do this, your odds of getting a useful >>> answer and the speed with which you are likely to get that answer both >>> improve tremendously. >>> >>> Simon Tatham has written an excellent essay entitled How to Report Bugs >>> Effectively . I >>> strongly recommend that you read it. >>> >>> >> >> ---------- Forwarded message --------- >> From: ??? ???? >> Date: Wed, Jul 13, 2022 at 1:36 PM >> Subject: Re: [Neuroimaging] what's the problem?????? >> To: Neuroimaging analysis in Python , < >> python-list at python.org> >> >> >> I want to set dict >> >> ?????? ??? ??, 13 ????? 2022, 20:47, ??? ??? ???? ?: >> >>> CODE: >>> >>> for nii in os.listdir("c:/users/administrator/desktop/nii"): >>> >>> from nilearn import plotting >>> from nilearn import datasets >>> atlas = datasets.fetch_atlas_msdl() >>> # Loading atlas image stored in 'maps' >>> atlas_filename = >>> "C:/Users/Administrator/Desktop/64/64/2mm/maps.nii.gz" >>> # Loading atlas data stored in 'labels' >>> labels = pd.read_csv( >>> "C:/Users/Administrator/Desktop/64/64/labels_64_dictionary.csv") >>> a=labels.to_dict() >>> b=a["Difumo_names"] >>> from nilearn.maskers import NiftiMapsMasker >>> masker = NiftiMapsMasker(maps_img=atlas_filename, standardize=True, >>> memory='nilearn_cache', verbose=5) >>> >>> time_series = masker.fit_transform( >>> "c:/users/administrator/desktop/nii/"+nii) >>> try: >>> from sklearn.covariance import GraphicalLassoCV >>> except ImportError: >>> # for Scitkit-Learn < v0.20.0 >>> from sklearn.covariance import GraphLassoCV as GraphicalLassoCV >>> >>> estimator = GraphicalLassoCV() >>> estimator.fit(time_series) >>> # Display the covariancec >>> aas={} >>> jsa=0 >>> for i in estimator.covariance_: >>> r=list(a["Difumo_names"].values())[jsa] >>> jsa=jsa+1 >>> a=dict() >>> >>> >>> for x in range(64): >>> g=list(a["Difumo_names"].values())[x] >>> >>> print(aas) >>> t= nilearn.plotting.plot_img(estimator.covariance_, labels=list(a[ >>> "Difumo_names"].values()), >>> figure=(9, 7), vmax=1, vmin=-1, >>> title='Covariance')# The covariance can be >>> found at estimator.covariance_ >>> >>> # The covariance can be found at estimator.covariance_ >>> t2= nilearn.plotting.plot_matrix(estimator.covariance_, labels=list >>> (a["Difumo_names"].values()), >>> figure=(9, 7), vmax=1, vmin=-1, >>> title='Covariance') >>> >>> >>> >>> -- >>> >>> >> _______________________________________________ >> Neuroimaging mailing list >> Neuroimaging at python.org >> https://mail.python.org/mailman/listinfo/neuroimaging >> > From nsh531 at gmail.com Thu Jul 14 01:49:39 2022 From: nsh531 at gmail.com (=?UTF-8?B?16DXqteZINep15jXqNef?=) Date: Thu, 14 Jul 2022 08:49:39 +0300 Subject: [Neuroimaging] what's the problem?????? In-Reply-To: <006601d8973c$65707390$30515ab0$@gmail.com> References: <006601d8973c$65707390$30515ab0$@gmail.com> Message-ID: Instead of numbers, I want to enter as indexes the names of the Index as being in b variable: labels = pd.read_csv( "C:/Users/Administrator/Desktop/64/64/labels_64_dictionary.csv") a=labels.to_dict() b=a["Difumo_names"] ??????? ??? ??, 14 ????? 2022 ?-7:44 ??? :? > Nati, > > I know you think you are communicating. > > ???? >> I want to swap indexes of dict > > When you say SWAP, do you mean replace the index with something else, or > swap the key and value or something entirely else? > > You have not shared an example of what is in your dictionary, but say it > is: > > myDict = {1:"a", 2:"b", 3:"c"} > > invDict = { value:key for key, value in myDict.items() } > > print(myDict) > print(invDict) > > > The above prints out: > > print(myDict) > {1: 'a', 2: 'b', 3: 'c'} > > print(invDict) > {'a': 1, 'b': 2, 'c': 3} > > It obviously generalizes only for dictionaries where all values are > unique. And you can obviously save the results back into the same variable > if you wish. The above comprehension may, of course, not be what you want > as swapping indexes with may not mean anything to us trying > to help. > > Avi (a ???? on my mother's side.) > > > -----Original Message----- > From: Python-list > On Behalf Of ??? ???? > Sent: Thursday, July 14, 2022 12:19 AM > To: David Welch ; Neuroimaging analysis in > Python ; python-list at python.org > Subject: Re: [Neuroimaging] what's the problem?????? > > I want to swap indexes of dict > > ?????? ??? ??, 13 ????? 2022, 21:51, ??? David Welch ?< > david.m.welch at gmail.com>: > > > > >> - Describe the research *you did* to try and understand the problem > >> *before* you asked the question. > >> - Describe the diagnostic steps *you took* to try and pin down the > >> problem yourself *before* you asked the question. > >> > >> > > On Wed, Jul 13, 2022 at 1:47 PM David Welch > > wrote: > > > >> From http://catb.org/~esr/faqs/smart-questions.html#beprecise: > >> > >>> Be precise and informative about your problem > >>> > >>> - > >>> > >>> Describe the symptoms of your problem or bug carefully and clearly. > >>> - > >>> > >>> Describe the environment in which it occurs (machine, OS, > >>> application, whatever). Provide your vendor's distribution and > release > >>> level (e.g.: ?Fedora Core 7?, ?Slackware 9.1?, etc.). > >>> - > >>> > >>> Describe the research you did to try and understand the problem > >>> before you asked the question. > >>> - > >>> > >>> Describe the diagnostic steps you took to try and pin down the > >>> problem yourself before you asked the question. > >>> - > >>> > >>> Describe any possibly relevant recent changes in your computer or > >>> software configuration. > >>> - > >>> > >>> If at all possible, provide a way to *reproduce the problem in a > >>> controlled environment*. > >>> > >>> Do the best you can to anticipate the questions a hacker will ask, > >>> and answer them in advance in your request for help. > >>> > >>> Giving hackers the ability to reproduce the problem in a controlled > >>> environment is especially important if you are reporting something > >>> you think is a bug in code. When you do this, your odds of getting a > >>> useful answer and the speed with which you are likely to get that > >>> answer both improve tremendously. > >>> > >>> Simon Tatham has written an excellent essay entitled How to Report > >>> Bugs Effectively > >>> . I strongly > recommend that you read it. > >>> > >>> > >> > >> ---------- Forwarded message --------- > >> From: ??? ???? > >> Date: Wed, Jul 13, 2022 at 1:36 PM > >> Subject: Re: [Neuroimaging] what's the problem?????? > >> To: Neuroimaging analysis in Python , < > >> python-list at python.org> > >> > >> > >> I want to set dict > >> > >> ?????? ??? ??, 13 ????? 2022, 20:47, ??? ??? ???? ?: > >> > >>> CODE: > >>> > >>> for nii in os.listdir("c:/users/administrator/desktop/nii"): > >>> > >>> from nilearn import plotting > >>> from nilearn import datasets > >>> atlas = datasets.fetch_atlas_msdl() > >>> # Loading atlas image stored in 'maps' > >>> atlas_filename = > >>> "C:/Users/Administrator/Desktop/64/64/2mm/maps.nii.gz" > >>> # Loading atlas data stored in 'labels' > >>> labels = pd.read_csv( > >>> "C:/Users/Administrator/Desktop/64/64/labels_64_dictionary.csv") > >>> a=labels.to_dict() > >>> b=a["Difumo_names"] > >>> from nilearn.maskers import NiftiMapsMasker > >>> masker = NiftiMapsMasker(maps_img=atlas_filename, standardize=True, > >>> memory='nilearn_cache', verbose=5) > >>> > >>> time_series = masker.fit_transform( > >>> "c:/users/administrator/desktop/nii/"+nii) > >>> try: > >>> from sklearn.covariance import GraphicalLassoCV > >>> except ImportError: > >>> # for Scitkit-Learn < v0.20.0 > >>> from sklearn.covariance import GraphLassoCV as > >>> GraphicalLassoCV > >>> > >>> estimator = GraphicalLassoCV() > >>> estimator.fit(time_series) > >>> # Display the covariancec > >>> aas={} > >>> jsa=0 > >>> for i in estimator.covariance_: > >>> r=list(a["Difumo_names"].values())[jsa] > >>> jsa=jsa+1 > >>> a=dict() > >>> > >>> > >>> for x in range(64): > >>> g=list(a["Difumo_names"].values())[x] > >>> > >>> print(aas) > >>> t= nilearn.plotting.plot_img(estimator.covariance_, > labels=list(a[ > >>> "Difumo_names"].values()), > >>> figure=(9, 7), vmax=1, vmin=-1, > >>> title='Covariance')# The covariance can be > >>> found at estimator.covariance_ > >>> > >>> # The covariance can be found at estimator.covariance_ > >>> t2= nilearn.plotting.plot_matrix(estimator.covariance_, > >>> labels=list (a["Difumo_names"].values()), > >>> figure=(9, 7), vmax=1, vmin=-1, > >>> title='Covariance') > >>> > >>> > >>> > >>> -- > >>> > >>> > >> _______________________________________________ > >> Neuroimaging mailing list > >> Neuroimaging at python.org > >> https://mail.python.org/mailman/listinfo/neuroimaging > >> > > > -- > https://mail.python.org/mailman/listinfo/python-list > > -- From vinay_sajip at yahoo.co.uk Thu Jul 14 10:56:17 2022 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 14 Jul 2022 14:56:17 +0000 (UTC) Subject: ANN: distlib 0.3.5 released on PyPI References: <1052610260.2176373.1657810577943.ref@mail.yahoo.com> Message-ID: <1052610260.2176373.1657810577943@mail.yahoo.com> I've recently released version 0.3.5 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: * Fixed #161: Updated test case. * Fixed #164: Improved support for reproducible builds by allowing a fixed ? date/time to be inserted into created .exe files. Thanks to Somber Night for the ? patch. * Fixed #169: Removed usage of deprecated imp module in favour of importlib. * Fixed #170: Corrected implementation of ``get_required_dists()``. * Fixed #172: Compute ABI correctly for Python < 3.8. * Changed the default locator configuration. * Made updates in support of PEP 643 / Metadata 2.2. * Updated launcher executables. Thanks to Michael Bikovitsky for his help with ? the launcher changes. * Updated to write archive path of RECORD to RECORD instead of staging path. ? Thanks to Pieter Pas for the patch. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.org/project/distlib/0.3.5/ [2] https://distlib.readthedocs.io/en/0.3.5/ [3] https://github.com/pypa/distlib/issues/new/choose From wlfraed at ix.netcom.com Sun Jul 10 13:41:15 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Sun, 10 Jul 2022 13:41:15 -0400 Subject: Job vacancies References: Message-ID: On Sun, 10 Jul 2022 18:46:26 +0300, Walid AlMasri declaimed the following: > >I would like to ask if there are any openings that suit my background? >I want to work on python either remotely or at job site Most of the regulars here are just users asking for help with the language, or providing said help. Some may be developers of the Python language itself. Pretty much none that I've seen have any inputs with the hiring departments of any firms. Very few firms look explicitly for Python, in my experience. They want software engineers with experience in multiple languages and operating systems, and skill in developing software independent of language -- the language used comes down to implementation (give the design to some new-hire to hack into working code ) and may be constrained by the application domain requirements (you won't find Python in real-time avionics, for example -- but it might be used to analyze debug logs during development of said avionics). Your background appears to be research-biased, and Python (with numpy/scipy/pandas) would just be a tool for analysis... as would R, Octave, Matlab, Mathematica. Again, I wouldn't expect any such research facility to mandate Python -- they'd be more interested in the "theoretical physics" side and data collection. Analysis would come down to whatever tool is best suited (likely by a team consensus). > >I attach my CV > NO binary attachments on this forum. Plain text may make it through (since you could just include it as the message itself). But... since no hiring manager is reading here, there is no reason to include it. You need to investigate sites that manage job postings. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From michael.stemper at gmail.com Tue Jul 12 09:22:41 2022 From: michael.stemper at gmail.com (Michael F. Stemper) Date: Tue, 12 Jul 2022 08:22:41 -0500 Subject: calculate diff between dates In-Reply-To: References: Message-ID: On 12/07/2022 07.37, ??? ???? wrote: [snip ugly code that I'm not going to try to understand] > > I glad for any help There wasn't any question in your post. However, I'm going to guess that there was the implied question of "How does one find the difference between two dates?" >>> import datetime >>> d1 = datetime.date(2022,1,1) >>> d2 = datetime.date(2022,7,12) >>> d2-d1 datetime.timedelta(192) >>> -- Michael F. Stemper If it isn't running programs and it isn't fusing atoms, it's just bending space. From bart.kuijer at ziggo.nl Wed Jul 13 10:02:39 2022 From: bart.kuijer at ziggo.nl (Bart Kuijer) Date: Wed, 13 Jul 2022 16:02:39 +0200 Subject: message Message-ID: <9a5685a7-7ff0-7fa9-93dd-bf256b9a0929@ziggo.nl> In the last 10 years of my working life I worked as a freelancer, as a specialist in Microsoft Access, at Interpolis-Tilburg, Rabobank-Zeist, Abn-Amro-Amsterdam and the SVB in Amstelveen, among others. From 1999 onward, I developed an accounting program in Microsoft Access for myself, Boeket, intended for Bakker-Schmalenbach's current chart of accounts in the Netherlands. I am now 84 years old, I had a stroke 4 years ago and am paralyzed on the right side, in a wheelchair and have decided to port this package to Python Can you help me with that? I think there will be a working group with the aim of making the package available for free. To get an idea what this is all about 1. the help screen https://app.box.com/s/fiy20u5r89n4jpn0346bnxu9xq6s86lk 2, the total package https://app.box.com/s/aag348tejxgdacc00n5ri3l9txa5bwwn I am looking forward to your response. Bart Kuijer From wlfraed at ix.netcom.com Thu Jul 14 11:47:21 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Thu, 14 Jul 2022 11:47:21 -0400 Subject: [Neuroimaging] what's the problem?????? References: <006601d8973c$65707390$30515ab0$@gmail.com> Message-ID: <7fe0dhlp3ade721ngvo0s2ebtcptp5lsfn@4ax.com> On Thu, 14 Jul 2022 08:49:39 +0300, ??? ???? declaimed the following: >Instead of numbers, I want to enter as indexes the names of the Index as >being in b variable: > labels = pd.read_csv( >"C:/Users/Administrator/Desktop/64/64/labels_64_dictionary.csv") > a=labels.to_dict() > b=a["Difumo_names"] > Dicts don't have "indexes", they have "keys" (index implies a positional look-up: first, second, ..., tenth item). Show us an example of your actual data, what it looks like in Python, and (hand-edited) what your really want the data to look like. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From nsh531 at gmail.com Mon Jul 11 03:26:24 2022 From: nsh531 at gmail.com (=?UTF-8?B?16DXqteZINep15jXqNef?=) Date: Mon, 11 Jul 2022 10:26:24 +0300 Subject: Fwd: ml on medical images - keras In-Reply-To: References: Message-ID: ---------- Forwarded message --------- ???: ??? ???? ?Date: ??? ??, 10 ????? 2022, 13:01? Subject: Re: ml on medical images - keras To: Neuroimaging analysis in Python p.s. all the pictures are in PNG FORMAT ??????? ??? ??, 10 ????? 2022 ?-13:01 ??? ??? ???? :? > What's the problem on this code: > > import os > from pickletools import float8, uint8 > from PIL import Image > > import numpy as np > import tensorflow as tf > from tensorflow import keras > from tensorflow.keras import layers > inputs=[] > for i in os.listdir("c:/inetpub/wwwroot/out"): > for j in os.listdir("c:/inetpub/wwwroot/out/"+i+"/"): > A=Image.open("c:/inetpub/wwwroot/out/"+i+"/"+j) > from matplotlib import pyplot as plt > > filename = 'image-test' > > > # img = ( filename + '.png' ) > x=np.array(A,dtype=np.shape(A)) > inputs.append(x) > > simple_rnn = tf.keras.layers.SimpleRNN(4) > #np.int > output = simple_rnn(inputs=np.array(inputs,dtype=np.ndarray(260, 730, 4))) > # The output has shape `[32, 4]`. > > simple_rnn = tf.keras.layers.SimpleRNN( > 4, return_sequences=True, return_state=True) > > # whole_sequence_output has shape `[32, 10, 4]`. > # final_state has shape `[32, 4]`. > whole_sequence_output, final_state = simple_rnn(inputs) > > -- > > -- From nsh531 at gmail.com Wed Jul 13 13:47:18 2022 From: nsh531 at gmail.com (=?UTF-8?B?16DXqteZINep15jXqNef?=) Date: Wed, 13 Jul 2022 20:47:18 +0300 Subject: what's the problem?????? Message-ID: CODE: for nii in os.listdir("c:/users/administrator/desktop/nii"): from nilearn import plotting from nilearn import datasets atlas = datasets.fetch_atlas_msdl() # Loading atlas image stored in 'maps' atlas_filename = "C:/Users/Administrator/Desktop/64/64/2mm/maps.nii.gz" # Loading atlas data stored in 'labels' labels = pd.read_csv( "C:/Users/Administrator/Desktop/64/64/labels_64_dictionary.csv") a=labels.to_dict() b=a["Difumo_names"] from nilearn.maskers import NiftiMapsMasker masker = NiftiMapsMasker(maps_img=atlas_filename, standardize=True, memory='nilearn_cache', verbose=5) time_series = masker.fit_transform("c:/users/administrator/desktop/nii/" +nii) try: from sklearn.covariance import GraphicalLassoCV except ImportError: # for Scitkit-Learn < v0.20.0 from sklearn.covariance import GraphLassoCV as GraphicalLassoCV estimator = GraphicalLassoCV() estimator.fit(time_series) # Display the covariancec aas={} jsa=0 for i in estimator.covariance_: r=list(a["Difumo_names"].values())[jsa] jsa=jsa+1 a=dict() for x in range(64): g=list(a["Difumo_names"].values())[x] print(aas) t= nilearn.plotting.plot_img(estimator.covariance_, labels=list(a[ "Difumo_names"].values()), figure=(9, 7), vmax=1, vmin=-1, title='Covariance')# The covariance can be found at estimator.covariance_ # The covariance can be found at estimator.covariance_ t2= nilearn.plotting.plot_matrix(estimator.covariance_, labels=list(a[ "Difumo_names"].values()), figure=(9, 7), vmax=1, vmin=-1, title='Covariance') -- From avi.e.gross at gmail.com Thu Jul 14 01:17:10 2022 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Thu, 14 Jul 2022 01:17:10 -0400 Subject: [Neuroimaging] what's the problem?????? In-Reply-To: <006601d8973c$65707390$30515ab0$@gmail.com> References: <006601d8973c$65707390$30515ab0$@gmail.com> Message-ID: <008a01d89740$f8249dc0$e86dd940$@gmail.com> Nati, I know you think you are communicating. ???? >> I want to swap indexes of dict When you say SWAP, do you mean replace the index with something else, or swap the key and value or something entirely else? You have not shared an example of what is in your dictionary, but say it is: myDict = {1:"a", 2:"b", 3:"c"} invDict = { value:key for key, value in myDict.items() } print(myDict) print(invDict) The above prints out: print(myDict) {1: 'a', 2: 'b', 3: 'c'} print(invDict) {'a': 1, 'b': 2, 'c': 3} It obviously generalizes only for dictionaries where all values are unique. And you can obviously save the results back into the same variable if you wish. The above comprehension may, of course, not be what you want as swapping indexes with may not mean anything to us trying to help. Avi (a ???? on my mother's side.) -----Original Message----- From: Python-list On Behalf Of ??? ???? Sent: Thursday, July 14, 2022 12:19 AM To: David Welch ; Neuroimaging analysis in Python ; python-list at python.org Subject: Re: [Neuroimaging] what's the problem?????? I want to swap indexes of dict ?????? ??? ??, 13 ????? 2022, 21:51, ??? David Welch ?< david.m.welch at gmail.com>: > >> - Describe the research *you did* to try and understand the problem >> *before* you asked the question. >> - Describe the diagnostic steps *you took* to try and pin down the >> problem yourself *before* you asked the question. >> >> > On Wed, Jul 13, 2022 at 1:47 PM David Welch > wrote: > >> From http://catb.org/~esr/faqs/smart-questions.html#beprecise: >> >>> Be precise and informative about your problem >>> >>> - >>> >>> Describe the symptoms of your problem or bug carefully and clearly. >>> - >>> >>> Describe the environment in which it occurs (machine, OS, >>> application, whatever). Provide your vendor's distribution and release >>> level (e.g.: ?Fedora Core 7?, ?Slackware 9.1?, etc.). >>> - >>> >>> Describe the research you did to try and understand the problem >>> before you asked the question. >>> - >>> >>> Describe the diagnostic steps you took to try and pin down the >>> problem yourself before you asked the question. >>> - >>> >>> Describe any possibly relevant recent changes in your computer or >>> software configuration. >>> - >>> >>> If at all possible, provide a way to *reproduce the problem in a >>> controlled environment*. >>> >>> Do the best you can to anticipate the questions a hacker will ask, >>> and answer them in advance in your request for help. >>> >>> Giving hackers the ability to reproduce the problem in a controlled >>> environment is especially important if you are reporting something >>> you think is a bug in code. When you do this, your odds of getting a >>> useful answer and the speed with which you are likely to get that >>> answer both improve tremendously. >>> >>> Simon Tatham has written an excellent essay entitled How to Report >>> Bugs Effectively >>> . I strongly recommend that you read it. >>> >>> >> >> ---------- Forwarded message --------- >> From: ??? ???? >> Date: Wed, Jul 13, 2022 at 1:36 PM >> Subject: Re: [Neuroimaging] what's the problem?????? >> To: Neuroimaging analysis in Python , < >> python-list at python.org> >> >> >> I want to set dict >> >> ?????? ??? ??, 13 ????? 2022, 20:47, ??? ??? ???? ?: >> >>> CODE: >>> >>> for nii in os.listdir("c:/users/administrator/desktop/nii"): >>> >>> from nilearn import plotting >>> from nilearn import datasets >>> atlas = datasets.fetch_atlas_msdl() >>> # Loading atlas image stored in 'maps' >>> atlas_filename = >>> "C:/Users/Administrator/Desktop/64/64/2mm/maps.nii.gz" >>> # Loading atlas data stored in 'labels' >>> labels = pd.read_csv( >>> "C:/Users/Administrator/Desktop/64/64/labels_64_dictionary.csv") >>> a=labels.to_dict() >>> b=a["Difumo_names"] >>> from nilearn.maskers import NiftiMapsMasker >>> masker = NiftiMapsMasker(maps_img=atlas_filename, standardize=True, >>> memory='nilearn_cache', verbose=5) >>> >>> time_series = masker.fit_transform( >>> "c:/users/administrator/desktop/nii/"+nii) >>> try: >>> from sklearn.covariance import GraphicalLassoCV >>> except ImportError: >>> # for Scitkit-Learn < v0.20.0 >>> from sklearn.covariance import GraphLassoCV as >>> GraphicalLassoCV >>> >>> estimator = GraphicalLassoCV() >>> estimator.fit(time_series) >>> # Display the covariancec >>> aas={} >>> jsa=0 >>> for i in estimator.covariance_: >>> r=list(a["Difumo_names"].values())[jsa] >>> jsa=jsa+1 >>> a=dict() >>> >>> >>> for x in range(64): >>> g=list(a["Difumo_names"].values())[x] >>> >>> print(aas) >>> t= nilearn.plotting.plot_img(estimator.covariance_, labels=list(a[ >>> "Difumo_names"].values()), >>> figure=(9, 7), vmax=1, vmin=-1, >>> title='Covariance')# The covariance can be >>> found at estimator.covariance_ >>> >>> # The covariance can be found at estimator.covariance_ >>> t2= nilearn.plotting.plot_matrix(estimator.covariance_, >>> labels=list (a["Difumo_names"].values()), >>> figure=(9, 7), vmax=1, vmin=-1, >>> title='Covariance') >>> >>> >>> >>> -- >>> >>> >> _______________________________________________ >> Neuroimaging mailing list >> Neuroimaging at python.org >> https://mail.python.org/mailman/listinfo/neuroimaging >> > -- https://mail.python.org/mailman/listinfo/python-list From hongyi.zhao at gmail.com Thu Jul 14 05:06:43 2022 From: hongyi.zhao at gmail.com (hongy...@gmail.com) Date: Thu, 14 Jul 2022 02:06:43 -0700 (PDT) Subject: Extract the space group generators from Bilbao Crystallographic Server. Message-ID: <5b27e547-17b8-47d7-9929-7a5ecd5587afn@googlegroups.com> I'm trying to extract the matrix data of "ITA-Setting F d -3 m [origin 1]" listed here [1], and then building an augmented matrix for each of them by adding the last row as "[0, 0, 0, 1]". In short, the following form is the ultimate-desired result: [[[1, 0, 0, 0], [0, 1, 0, 0], [0, 0,1, 0], [0, 0, 0, 1]], [[-1, 0, 0, 1], [0,-1, 0, 1/2], [0, 0, 1, 1/2], [0, 0, 0, 1]], [[-1, 0, 0, 1/2], [0, 1, 0, 1/2], [0, 0,-1, 1], [0, 0, 0, 1]], [[0, 0, 1, 0], [1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 0, 1]], [[0, 1, 0, 3/4], [1, 0, 0, 1/4], [0, 0, -1, 3/4], [0, 0, 0, 1]], [[-1, 0, 0, 1/4], [0, -1, 0, 1/4], [0, 0, -1, 1/4], [0, 0, 0, 1]], [[1, 0, 0, 0], [0, 1, 0, 1/2], [0, 0, 1, 1/2], [0, 0, 0, 1 ]], [[1, 0, 0, 1/2], [0, 1, 0, 0], [0, 0, 1, 1/2], [0, 0, 0, 1]]] Any hints/tips/tricks for achieving this aim will be appreciated. [1] https://www.cryst.ehu.es/cgi-bin/cryst/programs//nph-trgen?gnum=227&what=gen&trmat=a-1/8,b-1/8,c-1/8&unconv=F%20d%20-3%20m%20:1&from=ita Regards, Zhao From avi.e.gross at gmail.com Thu Jul 14 13:24:23 2022 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Thu, 14 Jul 2022 13:24:23 -0400 Subject: [Neuroimaging] what's the problem?????? In-Reply-To: References: <006601d8973c$65707390$30515ab0$@gmail.com> Message-ID: <009d01d897a6$8fce7ff0$af6b7fd0$@gmail.com> That is more detailed, Nati, albeit I may not still understand. You are working on a detailed project you may have some understanding of and getting stuck. But the people you ask for help do not have your info and, worse, your code seems full of blunders based on making assumptions, so reading it can give us a headache when we still have no idea what you want to do and what you want to do it to. So let me make sure what I think you said and am replying to, is what you want. My guess is not. It sounds like you already have a dictionary of size N that you read in from a file I have no access to and cannot see the contents but presumably contains two columns that can be converted into dictionary format. My first guess was you had another list of something also of length N stored in Difumo_names and wanted to substitute those new keys for the old ones. If the two are of the same length that should be doable, albeit older versions of dictionaries in Python did not guarantee the order of the contents. But reading the old code and your new stuff together does not lead me to assume this is the scenario. Your English may not be up to explaining as your short phrases seem to not mean what we expect. Your code shown (way below) does not explain what "Difumo_names" used in quotes means but your code suggests it is meant as a SINGLE key as you use it like b=a["Difumo_names"] But is there any guarantee the dictionary has anything with that key? You may want to use a form that fails better such as: b=a.get("Difumo_names"],-1) That fails by assigning whatever default you replace ?-1? with. Just BTW, it might be a good idea to use names longer and more meaningful than ?a? and ?b? as following your code is harder when I have no idea what anything is about or for. I, however, think your use of ?b? later in your code makes no sense with this interpretation. What is stored in b if done as above is the arbitrary value in a[("Difumo_names"] which could be many things, including a list, but your code never explains what it is. So later you never use b in the code shown but several times use the construct below. As part of a function call, one argument is: labels=list(a["Difumo_names"].values()) The above SUGGESTS that you have a dictionary in a dictionary. Meaning, the dictionary called ?a? has a key/value pair with the key being "Difumo_names"]? and the value being another dictionary that you can ask for the values alone and then combine them into a list. But is this true? You need to know and explain what your dictionary contains and whether labels.to_dict() properly is able to read in a CSV and not only make a dictionary from it but make some of the values into sub-dictionaries. If not, you might need a better way to make your data structure. Now, IF (and I am beyond guessing at this point) there is supposed to be a data structure in the value at a["Difumo_names"] that you want to be a dictionary, but it is now stored as something like a character string, then you need to find a way to convert it to a dictionary before using it like a dictionary. I end by saying I have no clear idea what you want and expect at each step in your program. My suggestion is you rewrite your code to carefully print out info as it goes along and verify if what you are seeing makes sense at every step. Specifically, see what all the keys an values in ?a? are. And at the REPL, play around and try typing in some code and see if it fails and what error messages you see. If some of your assumptions are wrong, they need fixing in that place and not by asking mysterious questions here from people who cannot see most of the picture. From: ??? ???? Sent: Thursday, July 14, 2022 1:50 AM To: avi.e.gross at gmail.com; python-list at python.org; Neuroimaging analysis in Python Subject: Re: [Neuroimaging] what's the problem?????? Instead of numbers, I want to enter as indexes the names of the Index as being in b variable: labels = pd.read_csv("C:/Users/Administrator/Desktop/64/64/labels_64_dictionary.csv") a=labels.to_dict() b=a["Difumo_names"] ??????? ??? ??, 14 ????? 2022 ?-7:44 ??? avi.e.gross at gmail.com?>: Nati, I know you think you are communicating. ???? >> I want to swap indexes of dict When you say SWAP, do you mean replace the index with something else, or swap the key and value or something entirely else? You have not shared an example of what is in your dictionary, but say it is: myDict = {1:"a", 2:"b", 3:"c"} invDict = { value:key for key, value in myDict.items() } print(myDict) print(invDict) The above prints out: print(myDict) {1: 'a', 2: 'b', 3: 'c'} print(invDict) {'a': 1, 'b': 2, 'c': 3} It obviously generalizes only for dictionaries where all values are unique. And you can obviously save the results back into the same variable if you wish. The above comprehension may, of course, not be what you want as swapping indexes with may not mean anything to us trying to help. Avi (a ???? on my mother's side.) -----Original Message----- From: Python-list > On Behalf Of ??? ???? Sent: Thursday, July 14, 2022 12:19 AM To: David Welch >; Neuroimaging analysis in Python >; python-list at python.org Subject: Re: [Neuroimaging] what's the problem?????? I want to swap indexes of dict ?????? ??? ??, 13 ????? 2022, 21:51, ??? David Welch ?< david.m.welch at gmail.com >: > >> - Describe the research *you did* to try and understand the problem >> *before* you asked the question. >> - Describe the diagnostic steps *you took* to try and pin down the >> problem yourself *before* you asked the question. >> >> > On Wed, Jul 13, 2022 at 1:47 PM David Welch > > wrote: > >> From http://catb.org/~esr/faqs/smart-questions.html#beprecise: >> >>> Be precise and informative about your problem >>> >>> - >>> >>> Describe the symptoms of your problem or bug carefully and clearly. >>> - >>> >>> Describe the environment in which it occurs (machine, OS, >>> application, whatever). Provide your vendor's distribution and release >>> level (e.g.: ?Fedora Core 7?, ?Slackware 9.1?, etc.). >>> - >>> >>> Describe the research you did to try and understand the problem >>> before you asked the question. >>> - >>> >>> Describe the diagnostic steps you took to try and pin down the >>> problem yourself before you asked the question. >>> - >>> >>> Describe any possibly relevant recent changes in your computer or >>> software configuration. >>> - >>> >>> If at all possible, provide a way to *reproduce the problem in a >>> controlled environment*. >>> >>> Do the best you can to anticipate the questions a hacker will ask, >>> and answer them in advance in your request for help. >>> >>> Giving hackers the ability to reproduce the problem in a controlled >>> environment is especially important if you are reporting something >>> you think is a bug in code. When you do this, your odds of getting a >>> useful answer and the speed with which you are likely to get that >>> answer both improve tremendously. >>> >>> Simon Tatham has written an excellent essay entitled How to Report >>> Bugs Effectively >>> . I strongly recommend that you read it. >>> >>> >> >> ---------- Forwarded message --------- >> From: ??? ???? > >> Date: Wed, Jul 13, 2022 at 1:36 PM >> Subject: Re: [Neuroimaging] what's the problem?????? >> To: Neuroimaging analysis in Python >, < >> python-list at python.org > >> >> >> I want to set dict >> >> ?????? ??? ??, 13 ????? 2022, 20:47, ??? ??? ???? ? >: >> >>> CODE: >>> >>> for nii in os.listdir("c:/users/administrator/desktop/nii"): >>> >>> from nilearn import plotting >>> from nilearn import datasets >>> atlas = datasets.fetch_atlas_msdl() >>> # Loading atlas image stored in 'maps' >>> atlas_filename = >>> "C:/Users/Administrator/Desktop/64/64/2mm/maps.nii.gz" >>> # Loading atlas data stored in 'labels' >>> labels = pd.read_csv( >>> "C:/Users/Administrator/Desktop/64/64/labels_64_dictionary.csv") >>> a=labels.to_dict() >>> b=a["Difumo_names"] >>> from nilearn.maskers import NiftiMapsMasker >>> masker = NiftiMapsMasker(maps_img=atlas_filename, standardize=True, >>> memory='nilearn_cache', verbose=5) >>> >>> time_series = masker.fit_transform( >>> "c:/users/administrator/desktop/nii/"+nii) >>> try: >>> from sklearn.covariance import GraphicalLassoCV >>> except ImportError: >>> # for Scitkit-Learn < v0.20.0 >>> from sklearn.covariance import GraphLassoCV as >>> GraphicalLassoCV >>> >>> estimator = GraphicalLassoCV() >>> estimator.fit(time_series) >>> # Display the covariancec >>> aas={} >>> jsa=0 >>> for i in estimator.covariance_: >>> r=list(a["Difumo_names"].values())[jsa] >>> jsa=jsa+1 >>> a=dict() >>> >>> >>> for x in range(64): >>> g=list(a["Difumo_names"].values())[x] >>> >>> print(aas) >>> t= nilearn.plotting.plot_img(estimator.covariance_, labels=list(a[ >>> "Difumo_names"].values()), >>> figure=(9, 7), vmax=1, vmin=-1, >>> title='Covariance')# The covariance can be >>> found at estimator.covariance_ >>> >>> # The covariance can be found at estimator.covariance_ >>> t2= nilearn.plotting.plot_matrix(estimator.covariance_, >>> labels=list (a["Difumo_names"].values()), >>> figure=(9, 7), vmax=1, vmin=-1, >>> title='Covariance') >>> >>> >>> >>> -- >>> >>> >> _______________________________________________ >> Neuroimaging mailing list >> Neuroimaging at python.org >> https://mail.python.org/mailman/listinfo/neuroimaging >> > -- https://mail.python.org/mailman/listinfo/python-list -- From avi.e.gross at gmail.com Thu Jul 14 17:20:55 2022 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Thu, 14 Jul 2022 17:20:55 -0400 Subject: [Neuroimaging] what's the problem?????? In-Reply-To: <7fe0dhlp3ade721ngvo0s2ebtcptp5lsfn@4ax.com> References: <006601d8973c$65707390$30515ab0$@gmail.com> <7fe0dhlp3ade721ngvo0s2ebtcptp5lsfn@4ax.com> Message-ID: <009f01d897c7$9ac41f20$d04c5d60$@gmail.com> Dennis, I see Nati sent some more code without explaining again what he wants. Yes, somewhere in this stack of messages he may have said things (that we generally failed to understand) but it would be helpful to summarize WHY he sent us what he did or which lines to look at. If your suggestion is right, he may not understand dictionaries as a hashed data structure where you can access a.items() or a.keys() or a.values() if you want a list of sorts of the parts, or a[key] if you want a particular part. So do you think by b he means the names, meaning keys, so b=a.keys() might be what he wants? I am asking YOU, because I have a 99% better chance of getting a good answer from you since you know about as much as I do about his project, than from Nati who needs to learn how to ask a question in a way that others can zoom in on and answer. By now it is getting clear that he is working on a massive project and as he goes along, he gets stuck and the first reaction is to ask HERE. I think many of us are happy to chip in here and there but if it becomes a full-time job, would like to get paid! ? Avi -----Original Message----- From: Python-list On Behalf Of Dennis Lee Bieber Sent: Thursday, July 14, 2022 11:47 AM To: python-list at python.org Subject: Re: [Neuroimaging] what's the problem?????? On Thu, 14 Jul 2022 08:49:39 +0300, ??? ???? declaimed the following: >Instead of numbers, I want to enter as indexes the names of the Index >as being in b variable: > labels = pd.read_csv( >"C:/Users/Administrator/Desktop/64/64/labels_64_dictionary.csv") > a=labels.to_dict() > b=a["Difumo_names"] > Dicts don't have "indexes", they have "keys" (index implies a positional look-up: first, second, ..., tenth item). Show us an example of your actual data, what it looks like in Python, and (hand-edited) what your really want the data to look like. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ -- https://mail.python.org/mailman/listinfo/python-list From avi.e.gross at gmail.com Thu Jul 14 17:36:27 2022 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Thu, 14 Jul 2022 17:36:27 -0400 Subject: ml on medical images - keras In-Reply-To: References: Message-ID: <00a201d897c9$c68cb340$53a619c0$@gmail.com> Nati, You ask what the problem is in that code. I say there I absolutely NO PROBLEM for me in that code. Do you know why? Because even if I want to copy it and make sure I have all the modules it needs, I have no access to the files it opens, and no idea what the output of the images pumped in is supposed to be and so on. So since I won't try it, it stops being my problem. Now if you want to know if someone spied any obvious error in the code, who knows, someone might. But guess who has access to all the code and files and anything else in your environment? You! So why not use your interactive python interpreter and pause between parts of the code and insert requests to see what things look like, which is what programmers have been doing for decades, or use some debugger? Check what is being done and if something fails, trace back as to what it was being given and see if that is what the manual page suggests you should be giving it and so on. You cannot expect multiple people to keep doing the work for you and especially if they keep telling you they need more info. I know very little about you and what tasks you have agreed to do but suggest that it may end up being a lot more work than you anticipated given how many kinds of pseudo-questions you seem to have. What is wrong with your code is that someone else did not write it specifically for the purpose you want. My guess is that you copy lots of code from libraries or the internet and want to adapt it without necessarily understanding if it fits your needs or where it needs to be tweaked. If I am wrong, I apologize. But if you want help here, or in other forums, consider changing your approach and consider what you would want if anyone else asked you to help them. What you keep not wanting to do is supply a fairly simple example of inputs and outputs and error messages. Now in this case, if your inputs are images and machine learning algorithms are going to output best guesses about features such as what animal is in the picture, it may indeed be harder to give serious details. When I see a question I can answer, I may chime in but for now, this process is too frustrating. Avi -----Original Message----- From: Python-list On Behalf Of ??? ???? Sent: Monday, July 11, 2022 3:26 AM To: python-list at python.org Subject: Fwd: ml on medical images - keras ---------- Forwarded message --------- ???: ??? ???? ?Date: ??? ??, 10 ????? 2022, 13:01? Subject: Re: ml on medical images - keras To: Neuroimaging analysis in Python p.s. all the pictures are in PNG FORMAT ??????? ??? ??, 10 ????? 2022 ?-13:01 ??? ??? ???? :? > What's the problem on this code: > > import os > from pickletools import float8, uint8 > from PIL import Image > > import numpy as np > import tensorflow as tf > from tensorflow import keras > from tensorflow.keras import layers > inputs=[] > for i in os.listdir("c:/inetpub/wwwroot/out"): > for j in os.listdir("c:/inetpub/wwwroot/out/"+i+"/"): > A=Image.open("c:/inetpub/wwwroot/out/"+i+"/"+j) > from matplotlib import pyplot as plt > > filename = 'image-test' > > > # img = ( filename + '.png' ) > x=np.array(A,dtype=np.shape(A)) > inputs.append(x) > > simple_rnn = tf.keras.layers.SimpleRNN(4) #np.int output = > simple_rnn(inputs=np.array(inputs,dtype=np.ndarray(260, 730, 4))) # > The output has shape `[32, 4]`. > > simple_rnn = tf.keras.layers.SimpleRNN( > 4, return_sequences=True, return_state=True) > > # whole_sequence_output has shape `[32, 10, 4]`. > # final_state has shape `[32, 4]`. > whole_sequence_output, final_state = simple_rnn(inputs) > > -- > > -- -- https://mail.python.org/mailman/listinfo/python-list From drsalists at gmail.com Thu Jul 14 22:06:31 2022 From: drsalists at gmail.com (Dan Stromberg) Date: Thu, 14 Jul 2022 19:06:31 -0700 Subject: Extract the space group generators from Bilbao Crystallographic Server. In-Reply-To: <5b27e547-17b8-47d7-9929-7a5ecd5587afn@googlegroups.com> References: <5b27e547-17b8-47d7-9929-7a5ecd5587afn@googlegroups.com> Message-ID: It's good to include what you want to see as output, but it's important to also include what you have as input. It's also good to include what you've coded so far. It's considered good etiquette to give it a try yourself before asking the list. On Thu, Jul 14, 2022 at 1:03 PM hongy... at gmail.com wrote: > I'm trying to extract the matrix data of "ITA-Setting F d -3 m [origin 1]" > listed here [1], and then building an augmented matrix for each of them by > adding the last row as "[0, 0, 0, 1]". In short, the following form is the > ultimate-desired result: > > [[[1, 0, 0, 0], [0, 1, 0, 0], [0, 0,1, 0], [0, 0, 0, 1]], > [[-1, 0, 0, 1], [0,-1, 0, 1/2], [0, 0, 1, 1/2], [0, 0, 0, 1]], > [[-1, 0, 0, 1/2], [0, 1, 0, 1/2], [0, 0,-1, 1], [0, 0, 0, 1]], > [[0, 0, 1, 0], [1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 0, 1]], > [[0, 1, 0, 3/4], [1, 0, 0, 1/4], [0, 0, -1, 3/4], [0, 0, 0, 1]], > [[-1, 0, 0, 1/4], [0, -1, 0, 1/4], [0, 0, -1, 1/4], [0, 0, 0, 1]], > [[1, 0, 0, 0], [0, 1, 0, 1/2], [0, 0, 1, 1/2], [0, 0, 0, 1 ]], > [[1, 0, 0, 1/2], [0, 1, 0, 0], [0, 0, 1, 1/2], [0, 0, 0, 1]]] > > Any hints/tips/tricks for achieving this aim will be appreciated. > > [1] > https://www.cryst.ehu.es/cgi-bin/cryst/programs//nph-trgen?gnum=227&what=gen&trmat=a-1/8,b-1/8,c-1/8&unconv=F%20d%20-3%20m%20:1&from=ita > > Regards, > Zhao > -- > https://mail.python.org/mailman/listinfo/python-list > From avi.e.gross at gmail.com Thu Jul 14 23:08:29 2022 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Thu, 14 Jul 2022 23:08:29 -0400 Subject: Extract the space group generators from Bilbao Crystallographic Server. In-Reply-To: References: <5b27e547-17b8-47d7-9929-7a5ecd5587afn@googlegroups.com> Message-ID: <014001d897f8$292984a0$7b7c8de0$@gmail.com> I guess Dan, that he may not be seeing what he is working on as a list of lists of lists with each terminal sublist being of cardinality 4. Maybe Zhao could look up what methods a list object has that allow you to place additional items, such as a list of 4 numbers, at the beginning or end or in middle and select a method that does what he wants. But I have to ask where exactly he wants to place this: "[0, 0, 0, 1]" Unfortunately, we got no before and after picture, just after. I will explain what that means at the end but for now, I am making believe what you show is the before and see what that would mean. The list seems to be a representation for a matrix that is 8 by 4 by 4. Where do you place just the foursome above just once without breaking the matrix? I mean you can extend it at the bottom by adding four of the above as in [ [0, 0, 0, 1], [0, 0, 0, 1], [0, 0, 0, 1], [0, 0, 0, 1] ] Or you can take each internal four partner like the first one: [[1, 0, 0, 0], [0, 1, 0, 0], [0, 0,1, 0], [0, 0, 0, 1]] And extend each one at the right end by adding a fifth: [[1, 0, 0, 0], [0, 1, 0, 0], [0, 0,1, 0], [0, 0, 0, 1], [0, 0, 0, 1]] I can think of other transformations albeit I have no idea why this is being done and what makes sense. So first he needs to be clear on where and what he is adding and then think of a method. And BTW, this problem may also be looked at under a transformation. If allowed to use some modules, it might make sense to feed the data structure into something that returns a 3-D matrix directly and then use methods that allow you to tack on parts of other matrices of various dimensions including one. You can then flip it back into nested list format, if you wish. OK, as mentioned earlier, if this is the AFTER then I have to look and see if it is obvious where the "[0, 0, 0, 1]" was placed to make this: [[[1, 0, 0, 0], [0, 1, 0, 0], [0, 0,1, 0], [0, 0, 0, 1]], [[-1, 0, 0, 1], [0,-1, 0, 1/2], [0, 0, 1, 1/2], [0, 0, 0, 1]], [[-1, 0, 0, 1/2], [0, 1, 0, 1/2], [0, 0,-1, 1], [0, 0, 0, 1]], [[0, 0, 1, 0], [1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 0, 1]], [[0, 1, 0, 3/4], [1, 0, 0, 1/4], [0, 0, -1, 3/4], [0, 0, 0, 1]], [[-1, 0, 0, 1/4], [0, -1, 0, 1/4], [0, 0, -1, 1/4], [0, 0, 0, 1]], [[1, 0, 0, 0], [0, 1, 0, 1/2], [0, 0, 1, 1/2], [0, 0, 0, 1 ]], [[1, 0, 0, 1/2], [0, 1, 0, 0], [0, 0, 1, 1/2], [0, 0, 0, 1]]] It looks like you are adding not once copy but eight copies with one each at the end of the lists within the main list. That makes this easy enough so think about what it means to deal with lists and NOT matrices. Your top list contains what I will call an a,b,c,d,e,f,g,h so if you wrote code like Mylist = [ "a", "b", ..., "g", "h"] Then you can use a loop to iterate over those or perhaps a construct like this: [ item for item in Mylist ] Wil gather them together and place them back in a list, which is useless but it could be something applied to item like changing to upper case. In your case, each item will be something like this: [[1, 0, 0, 0], [0, 1, 0, 0], [0, 0,1, 0]] And you want to extend it by attaching this [0, 0, 0, 1] So how do you attach something to a list? Whatever that method is, call it append and look it up and consider a line of code like: [ item.append(something) for item in Mylist ] If done right, the above list comprehension loops over one dimension of your list version of a matrix and adds [0, 0, 0, 1] to the end and then eventually put the a/b/d/..h components back together to look like what I THINK you are asking. You can make some of the other possible changes using other tricks and gimmicks like a nested comprehension but at some point, if you work well with matrices, you may be better off converting your nested list into a numpy matrix and make your addition another matrix and then use numpy.concatenate, numpy.vstack and numpy.hstack with proper care with multiple dimensions to specify what axis they will combine on. But doing the full (home)work for you is ... -----Original Message----- From: Python-list On Behalf Of Dan Stromberg Sent: Thursday, July 14, 2022 10:07 PM To: hongy... at gmail.com Cc: Python List Subject: Re: Extract the space group generators from Bilbao Crystallographic Server. It's good to include what you want to see as output, but it's important to also include what you have as input. It's also good to include what you've coded so far. It's considered good etiquette to give it a try yourself before asking the list. On Thu, Jul 14, 2022 at 1:03 PM hongy... at gmail.com wrote: > I'm trying to extract the matrix data of "ITA-Setting F d -3 m [origin 1]" > listed here [1], and then building an augmented matrix for each of > them by adding the last row as "[0, 0, 0, 1]". In short, the following > form is the ultimate-desired result: > > [[[1, 0, 0, 0], [0, 1, 0, 0], [0, 0,1, 0], [0, 0, 0, 1]], > [[-1, 0, 0, 1], [0,-1, 0, 1/2], [0, 0, 1, 1/2], [0, 0, 0, 1]], > [[-1, 0, 0, 1/2], [0, 1, 0, 1/2], [0, 0,-1, 1], [0, 0, 0, 1]], > [[0, 0, 1, 0], [1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 0, 1]], > [[0, 1, 0, 3/4], [1, 0, 0, 1/4], [0, 0, -1, 3/4], [0, 0, 0, 1]], > [[-1, 0, 0, 1/4], [0, -1, 0, 1/4], [0, 0, -1, 1/4], [0, 0, 0, 1]], > [[1, 0, 0, 0], [0, 1, 0, 1/2], [0, 0, 1, 1/2], [0, 0, 0, 1 ]], > [[1, 0, 0, 1/2], [0, 1, 0, 0], [0, 0, 1, 1/2], [0, 0, 0, 1]]] > > Any hints/tips/tricks for achieving this aim will be appreciated. > > [1] > https://www.cryst.ehu.es/cgi-bin/cryst/programs//nph-trgen?gnum=227&wh > at=gen&trmat=a-1/8,b-1/8,c-1/8&unconv=F%20d%20-3%20m%20:1&from=ita > > Regards, > Zhao > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list From hongyi.zhao at gmail.com Fri Jul 15 05:11:36 2022 From: hongyi.zhao at gmail.com (hongy...@gmail.com) Date: Fri, 15 Jul 2022 02:11:36 -0700 (PDT) Subject: Extract the space group generators from Bilbao Crystallographic Server. In-Reply-To: References: <5b27e547-17b8-47d7-9929-7a5ecd5587afn@googlegroups.com> <014001d897f8$292984a0$7b7c8de0$@gmail.com> Message-ID: On Friday, July 15, 2022 at 11:08:55 AM UTC+8, avi.e... at gmail.com wrote: > I guess Dan, that he may not be seeing what he is working on as a list of > lists of lists with each terminal sublist being of cardinality 4. Maybe > Zhao could look up what methods a list object has that allow you to place > additional items, such as a list of 4 numbers, at the beginning or end or in > middle and select a method that does what he wants. > > But I have to ask where exactly he wants to place this: "[0, 0, 0, 1]" Yes. This must be appended to each of the subsists to construct an affine transformation matrix, as described here [1-3]. [1] https://www.cryst.ehu.es/cgi-bin/cryst/programs/nph-doc-trmat [2] https://www.gap-system.org/Manuals/pkg/crystcat/htm/CHAP001.htm#SECT002 [3] https://www.gap-system.org/Manuals/pkg/cryst/htm/CHAP002.htm > Unfortunately, we got no before and after picture, just after. I will > explain what that means at the end but for now, I am making believe what you > show is the before and see what that would mean. > > The list seems to be a representation for a matrix that is 8 by 4 by 4. > Where do you place just the foursome above just once without breaking the > matrix? I mean you can extend it at the bottom by adding four of the above > as in > [ [0, 0, 0, 1], > [0, 0, 0, 1], > [0, 0, 0, 1], > [0, 0, 0, 1] ] > > Or you can take each internal four partner like the first one: > [[1, 0, 0, 0], [0, 1, 0, 0], [0, 0,1, 0], [0, 0, 0, 1]] > And extend each one at the right end by adding a fifth: > > [[1, 0, 0, 0], [0, 1, 0, 0], [0, 0,1, 0], [0, 0, 0, 1], [0, 0, 0, 1]] > > I can think of other transformations albeit I have no idea why this is being > done and what makes sense. > > So first he needs to be clear on where and what he is adding and then think > of a method. > > And BTW, this problem may also be looked at under a transformation. If > allowed to use some modules, it might make sense to feed the data structure > into something that returns a 3-D matrix directly and then use methods that > allow you to tack on parts of other matrices of various dimensions including > one. You can then flip it back into nested list format, if you wish. > > OK, as mentioned earlier, if this is the AFTER then I have to look and see > if it is obvious where the "[0, 0, 0, 1]" was placed to make this: > [[[1, 0, 0, 0], [0, 1, 0, 0], [0, 0,1, 0], [0, 0, 0, 1]], > [[-1, 0, 0, 1], [0,-1, 0, 1/2], [0, 0, 1, 1/2], [0, 0, 0, 1]], > [[-1, 0, 0, 1/2], [0, 1, 0, 1/2], [0, 0,-1, 1], [0, 0, 0, 1]], > [[0, 0, 1, 0], [1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 0, 1]], > [[0, 1, 0, 3/4], [1, 0, 0, 1/4], [0, 0, -1, 3/4], [0, 0, 0, 1]], > [[-1, 0, 0, 1/4], [0, -1, 0, 1/4], [0, 0, -1, 1/4], [0, 0, 0, 1]], > [[1, 0, 0, 0], [0, 1, 0, 1/2], [0, 0, 1, 1/2], [0, 0, 0, 1 ]], > [[1, 0, 0, 1/2], [0, 1, 0, 0], [0, 0, 1, 1/2], [0, 0, 0, 1]]] > It looks like you are adding not once copy but eight copies with one each at > the end of the lists within the main list. > > That makes this easy enough so think about what it means to deal with lists > and NOT matrices. > > Your top list contains what I will call an a,b,c,d,e,f,g,h so if you wrote > code like > > Mylist = [ "a", "b", ..., "g", "h"] > > Then you can use a loop to iterate over those or perhaps a construct like > this: > > [ item for item in Mylist ] > > Wil gather them together and place them back in a list, which is useless but > it could be something applied to item like changing to upper case. In your > case, each item will be something like this: > > [[1, 0, 0, 0], [0, 1, 0, 0], [0, 0,1, 0]] > > And you want to extend it by attaching this [0, 0, 0, 1] > > So how do you attach something to a list? Whatever that method is, call it > append and look it up and consider a line of code like: > > [ item.append(something) for item in Mylist ] > > If done right, the above list comprehension loops over one dimension of your > list version of a matrix and adds [0, 0, 0, 1] to the end and then > eventually put the a/b/d/..h components back together to look like what I > THINK you are asking. > > You can make some of the other possible changes using other tricks and > gimmicks like a nested comprehension but at some point, if you work well > with matrices, you may be better off converting your nested list into a > numpy matrix and make your addition another matrix and then use > numpy.concatenate, numpy.vstack and numpy.hstack with proper care with > multiple dimensions to specify what axis they will combine on. > > But doing the full (home)work for you is ... Thank you for your analysis and comments. Best, Zhao > -----Original Message----- > From: Python-list On > Behalf Of Dan Stromberg > Sent: Thursday, July 14, 2022 10:07 PM > To: hongy... at gmail.com > Cc: Python List > Subject: Re: Extract the space group generators from Bilbao Crystallographic > Server. > > It's good to include what you want to see as output, but it's important to > also include what you have as input. > > It's also good to include what you've coded so far. It's considered good > etiquette to give it a try yourself before asking the list. > > On Thu, Jul 14, 2022 at 1:03 PM hongy... at gmail.com > wrote: > > I'm trying to extract the matrix data of "ITA-Setting F d -3 m [origin 1]" > > listed here [1], and then building an augmented matrix for each of > > them by adding the last row as "[0, 0, 0, 1]". In short, the following > > form is the ultimate-desired result: > > > > [[[1, 0, 0, 0], [0, 1, 0, 0], [0, 0,1, 0], [0, 0, 0, 1]], > > [[-1, 0, 0, 1], [0,-1, 0, 1/2], [0, 0, 1, 1/2], [0, 0, 0, 1]], > > [[-1, 0, 0, 1/2], [0, 1, 0, 1/2], [0, 0,-1, 1], [0, 0, 0, 1]], > > [[0, 0, 1, 0], [1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 0, 1]], > > [[0, 1, 0, 3/4], [1, 0, 0, 1/4], [0, 0, -1, 3/4], [0, 0, 0, 1]], > > [[-1, 0, 0, 1/4], [0, -1, 0, 1/4], [0, 0, -1, 1/4], [0, 0, 0, 1]], > > [[1, 0, 0, 0], [0, 1, 0, 1/2], [0, 0, 1, 1/2], [0, 0, 0, 1 ]], > > [[1, 0, 0, 1/2], [0, 1, 0, 0], [0, 0, 1, 1/2], [0, 0, 0, 1]]] > > > > Any hints/tips/tricks for achieving this aim will be appreciated. > > > > [1] > > https://www.cryst.ehu.es/cgi-bin/cryst/programs//nph-trgen?gnum=227&wh > > at=gen&trmat=a-1/8,b-1/8,c-1/8&unconv=F%20d%20-3%20m%20:1&from=ita > > > > Regards, > > Zhao > > -- > > https://mail.python.org/mailman/listinfo/python-list > > > -- > https://mail.python.org/mailman/listinfo/python-list From wlfraed at ix.netcom.com Fri Jul 15 13:11:26 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Fri, 15 Jul 2022 13:11:26 -0400 Subject: [Neuroimaging] what's the problem?????? References: <006601d8973c$65707390$30515ab0$@gmail.com> <7fe0dhlp3ade721ngvo0s2ebtcptp5lsfn@4ax.com> <009f01d897c7$9ac41f20$d04c5d60$@gmail.com> Message-ID: On Thu, 14 Jul 2022 17:20:55 -0400, declaimed the following: >Dennis, > >I see Nati sent some more code without explaining again what he wants. Yes, somewhere in this stack of messages he may have said things (that we generally failed to understand) but it would be helpful to summarize WHY he sent us what he did or which lines to look at. > And that code seems to be doing a lot of repetitive/unneeded operations... Nothing from from nilearn import ... to masker = ... appears to depend upon the file name "nii" in the loop and should be refactored to the top-level. The try/except block for sklearn looks like another candidate to be pulled out of the file loop. > If your suggestion is right, he may not understand dictionaries as a hashed data structure where you can access a.items() or a.keys() or a.values() if you want a list of sorts of the parts, or a[key] if you want a particular part. > So far as I can tell, the OP only references ONE key in the dictionary, but apparently expects that key to have multiple values. Yet there doesn't seem to be a structure /holding/ multiple values -- unless a["Difumo_names"] is, itself, returning a dictionary on which .values() can be applied. In that case, the list() call is likely redundant as .values() already returned a list (hmmm, is list(some_list) a no-op, or does it wrap some_list into another list -- in the latter case, the indexing will fail as there is only one element to access) > for i in estimator.covariance_: > r=list(a["Difumo_names"].values())[jsa] > jsa=jsa+1 > a=dict() And there is the biggest fault... The OP is completely obliterating the "a" dictionary, so subsequent passes in that loop have nothing to look up. "aas" gets initialized as a dictionary using {}, and never gets any key:value pairs assigned to it, yet there is a print(aas) in the code. And that is as far as I can guess -- given the lack of example input data, example /desired/ output... etc. There are too many apparent errors in that code sample to even attempt to think what a result could be. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From roel at roelschroeven.net Fri Jul 15 15:14:31 2022 From: roel at roelschroeven.net (Roel Schroeven) Date: Fri, 15 Jul 2022 21:14:31 +0200 Subject: [Neuroimaging] what's the problem?????? In-Reply-To: References: <006601d8973c$65707390$30515ab0$@gmail.com> <7fe0dhlp3ade721ngvo0s2ebtcptp5lsfn@4ax.com> <009f01d897c7$9ac41f20$d04c5d60$@gmail.com> Message-ID: <4811f7da-1f6c-66f3-6787-33b4baee3052@roelschroeven.net> Dennis Lee Bieber schreef op 15/07/2022 om 19:11: > ... > is, itself, returning a dictionary on which .values() can be applied. In > that case, the list() call is likely redundant as .values() already > returned a list (hmmm, is list(some_list) a no-op, or does it wrap > some_list into another list -- in the latter case, the indexing will fail > as there is only one element to access) list(some_list) returns a copy of some_list, a shallow copy to be precise. >>> a = [ [1, 2, 3], [4, 5, 6] ] >>> b = list(a) Both lists are equal: >>> a == b True But not identical: >>> a is b False The sublists are identical though: >>> a[0] is b[0] True >>> a[1] is b[1] True -- "I've come up with a set of rules that describe our reactions to technologies: 1. Anything that is in the world when you?re born is normal and ordinary and is just a natural part of the way the world works. 2. Anything that's invented between when you?re fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it. 3. Anything invented after you're thirty-five is against the natural order of things." -- Douglas Adams, The Salmon of Doubt From avi.e.gross at gmail.com Fri Jul 15 17:38:53 2022 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Fri, 15 Jul 2022 17:38:53 -0400 Subject: [Neuroimaging] what's the problem?????? In-Reply-To: References: <006601d8973c$65707390$30515ab0$@gmail.com> <7fe0dhlp3ade721ngvo0s2ebtcptp5lsfn@4ax.com> <009f01d897c7$9ac41f20$d04c5d60$@gmail.com> Message-ID: <012e01d89893$47d1dce0$d77596a0$@gmail.com> Dennis, I am replying to the group but mainly you as I doubt Nati really reads much of what we write. The use of list() is needed as what comes out of the function may be an object of type dict_keys which is iterable but if you want it in a list form all at once, the idiom he uses is common. In the code below, alternating lines are commands and output, shown by blank lines between pairs. a={"aleph":1,"bet":2, "?":3} a {'aleph': 1, 'bet': 2, '?': 3} a.keys() dict_keys(['aleph', 'bet', '?']) list(a.keys()) ['aleph', 'bet', '?'] The question you ask is what I speculated on. Is he looking for the content of a specific key in his dictionary called "a" to be something with multiple values like a list or dictionary. BUT my question is whether reading it into numpy and converting the top level to a dictionary, results in being able to store another dictionary within it, or leaves some other data structure that maybe he needs to also be converted, if that can be done. I could waste more time playing around but Nati is not an ideal partner to engage with. I experiment with embedding a dictionary within itself as the second entry and it works fine BUT I bet what Nati has is not that as it was not made like that: a["bet"] = a a {'aleph': 1, 'bet': {...}, '?': 3} a["bet"]["aleph"] 1 Now if Nati gave a damn he should do something like I am doing and run his code part way and LOOK at what he has before continuing to use it as if it is what he imagines, or worse, he is copying only part of a recipe cobbled together. And, agreed, Nati has so many things that look like errors, that it is silly to ask why the mess did not work, and better to ask how each piece starting from the top is working or should be adjusted to work. -----Original Message----- From: Python-list On Behalf Of Dennis Lee Bieber Sent: Friday, July 15, 2022 1:11 PM To: python-list at python.org Subject: Re: [Neuroimaging] what's the problem?????? On Thu, 14 Jul 2022 17:20:55 -0400, declaimed the following: >Dennis, > >I see Nati sent some more code without explaining again what he wants. Yes, somewhere in this stack of messages he may have said things (that we generally failed to understand) but it would be helpful to summarize WHY he sent us what he did or which lines to look at. > And that code seems to be doing a lot of repetitive/unneeded operations... Nothing from from nilearn import ... to masker = ... appears to depend upon the file name "nii" in the loop and should be refactored to the top-level. The try/except block for sklearn looks like another candidate to be pulled out of the file loop. > If your suggestion is right, he may not understand dictionaries as a hashed data structure where you can access a.items() or a.keys() or a.values() if you want a list of sorts of the parts, or a[key] if you want a particular part. > So far as I can tell, the OP only references ONE key in the dictionary, but apparently expects that key to have multiple values. Yet there doesn't seem to be a structure /holding/ multiple values -- unless a["Difumo_names"] is, itself, returning a dictionary on which .values() can be applied. In that case, the list() call is likely redundant as .values() already returned a list (hmmm, is list(some_list) a no-op, or does it wrap some_list into another list -- in the latter case, the indexing will fail as there is only one element to access) > for i in estimator.covariance_: > r=list(a["Difumo_names"].values())[jsa] > jsa=jsa+1 > a=dict() And there is the biggest fault... The OP is completely obliterating the "a" dictionary, so subsequent passes in that loop have nothing to look up. "aas" gets initialized as a dictionary using {}, and never gets any key:value pairs assigned to it, yet there is a print(aas) in the code. And that is as far as I can guess -- given the lack of example input data, example /desired/ output... etc. There are too many apparent errors in that code sample to even attempt to think what a result could be. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ -- https://mail.python.org/mailman/listinfo/python-list From PythonList at DancesWithMice.info Sun Jul 17 21:10:00 2022 From: PythonList at DancesWithMice.info (dn) Date: Mon, 18 Jul 2022 13:10:00 +1200 Subject: FYI: Kiwi PyCon Message-ID: The first Kiwi PyCon of the COVID-era takes place in one month's time (19~22 August), in-person at ?tautahi Christchurch, and on-line. You will be most welcome! New Zealand is in the UTC+12 time-zone. Tickets are priced according to your choice of in-person or virtual. Reduced rates are offered to students and others. Professionals and corporations are invited to provide valuable support through the Kiwi tradition of buying two tickets - donating the second to someone more 'flush' with enthusiasm for Python than with cash. More info at kiwipycon.nz. -- Regards, =dn From PythonList at DancesWithMice.info Sun Jul 17 22:56:21 2022 From: PythonList at DancesWithMice.info (dn) Date: Mon, 18 Jul 2022 14:56:21 +1200 Subject: UML, CVS, and Crafting Software: NZPUG Auckland Branch Message-ID: <27543b7c-3989-cd35-a3b2-90bd4b9c266b@DancesWithMice.info> You commit code to git (or some other CVS) all-day, every-day. Similarly, you probably work on/from/with UML diagrams. Our PUG seeks someone willing to demonstrate the very basic use of UML and/or CVS (as related to Python), please. We meet (virtually) in the UTC+12 time-zone (0600~0830 UTC) which suits 'late-owls' on the US west coast; everyone across Oceania, Asia, and the Middle East; through to folk waking-up in the UK and Europe. We are currently enjoying a bi-monthly "Software Craftsmanship" series in our monthly 'Presentation Evenings', with a theme of (raising) professional standards, led by Olaf Thielke. As you will expect, the illustration of some points uses UML diagrams. Accordingly, some of our hobbyists, 'Beginners' and 'Journeymen' will gain more - if they first understand how to read basic UML diagrams! The branch's 'Coding Evenings' (our second set of monthly meetings) currently complement the above with a 'code along at home' series called 'Crafting Software' led by dn and DJ. It aims to help coders learn some Python, acquire coding techniques, and to reflect-on and benefit-from the principles of Software Craftsmanship (ie approaching from the bottom-up or what I call: 'SOLID by stealth'). We're coding a (very) simplistic 'Business Rule', and gradually building-out and improving/refactoring as we go. At each stage we could retain the code by saving under a different module-name. Alternately, here is an ideal opportunity to illustrate how a versioning system enables one to remember, and later review and recall, 'history'! Here's where a short and practical demo of 'getting started with git'? (or ... CVS) will fit neatly into the series, perhaps on Wed 3 August. NB no need for multiple branches, diff-resolution, etc, etc. What's in it for me? In-return, the PUG's leadership-team will offer assistance including teaching you how to pull-together a talk (you know the subject but would like some help in presenting to others), thus building your competence and profile. We meet virtually, using the BigBlueButton web-conferencing software, and will be happy to extend your skills in that direction too! The PUG's Meetup group is https://www.meetup.com/nzpug-auckland/ More information about the group is available upon request. Happy to answer questions. Please consider offering a short but highly-pragmatic demonstration, contributing to our professional-improvement. Reply to me (off-list) and I'll introduce you to the wider PUG-leadership team... -- Regards, =dn From baerrs at gmail.com Mon Jul 18 00:53:34 2022 From: baerrs at gmail.com (Scott Baer) Date: Mon, 18 Jul 2022 00:53:34 -0400 Subject: script folder is empty Message-ID: I just installed Python 3.10.5 on a Windows 10 home ( Ver 21H2 OS build 1904431826). I'm logged in with admin privileges I did a custom install with python-3.10.5-amd64.exe to C:\Program Files\Python310 Installed with both For all Users & PIP selected. ; once It was done installing, I rebooted and can run python: Python 3.10.5 (tags/v3.10.5:f377153, Jun 6 2022, 16:14:13) [MSC v.1929 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> when I try to run: pip --version C:\Users\baerr>pip --version 'pip' is not recognized as an internal or external command, operable program or batch file. I've done some troubleshooting, and nothing is in the C:\Program Files\Python310\Scripts folder. I"m not a noob.. but obviously, I missing something.. I doubt this is a bug. Any help is much appreciated !! Scott baerrs at gmail.com From PythonList at DancesWithMice.info Mon Jul 18 01:42:07 2022 From: PythonList at DancesWithMice.info (dn) Date: Mon, 18 Jul 2022 17:42:07 +1200 Subject: script folder is empty In-Reply-To: References: Message-ID: On 18/07/2022 16.53, Scott Baer wrote: > I just installed Python 3.10.5 on a Windows 10 home ( Ver 21H2 OS build > 1904431826). > I'm logged in with admin privileges > I did a custom install with python-3.10.5-amd64.exe to C:\Program > Files\Python310 > Installed with both For all Users & PIP selected. > ; > once It was done installing, I rebooted and can run python: > Python 3.10.5 (tags/v3.10.5:f377153, Jun 6 2022, 16:14:13) [MSC v.1929 64 > bit (AMD64)] on win32 > Type "help", "copyright", "credits" or "license" for more information. >>>> > > when I try to run: pip --version > C:\Users\baerr>pip --version > 'pip' is not recognized as an internal or external command, > operable program or batch file. > > I've done some troubleshooting, and nothing is in the C:\Program > Files\Python310\Scripts folder. > > I"m not a noob.. but obviously, I missing something.. I doubt this is a > bug. > > Any help is much appreciated !! I don't use MS-Windows. Have you perused the docs? eg https://docs.python.org/3/using/windows.html?highlight=windows DuckDuckGo's first 'hit' for "How to Install PIP for Python on Windows" is https://www.liquidweb.com/kb/install-pip-windows/ - but please be aware that what comes-with varies according to the source used to obtain the copy of Python, and (perhaps) the version of MS-Win. YMMV! -- Regards, =dn From drsalists at gmail.com Mon Jul 18 09:07:41 2022 From: drsalists at gmail.com (Dan Stromberg) Date: Mon, 18 Jul 2022 06:07:41 -0700 Subject: script folder is empty In-Reply-To: References: Message-ID: This is another reason to use: python -m pip ... ...instead of: pip ... (Or many systems want python3 -m pip) HTH On Sun, Jul 17, 2022 at 10:42 PM dn wrote: > On 18/07/2022 16.53, Scott Baer wrote: > > I just installed Python 3.10.5 on a Windows 10 home ( Ver 21H2 OS build > > 1904431826). > > I'm logged in with admin privileges > > I did a custom install with python-3.10.5-amd64.exe to C:\Program > > Files\Python310 > > Installed with both For all Users & PIP selected. > > ; > > once It was done installing, I rebooted and can run python: > > Python 3.10.5 (tags/v3.10.5:f377153, Jun 6 2022, 16:14:13) [MSC v.1929 > 64 > > bit (AMD64)] on win32 > > Type "help", "copyright", "credits" or "license" for more information. > >>>> > > > > when I try to run: pip --version > > C:\Users\baerr>pip --version > > 'pip' is not recognized as an internal or external command, > > operable program or batch file. > > > > I've done some troubleshooting, and nothing is in the C:\Program > > Files\Python310\Scripts folder. > > > > I"m not a noob.. but obviously, I missing something.. I doubt this is a > > bug. > > > > Any help is much appreciated !! > > I don't use MS-Windows. Have you perused the docs? > eg https://docs.python.org/3/using/windows.html?highlight=windows > > DuckDuckGo's first 'hit' for "How to Install PIP for Python on Windows" > is https://www.liquidweb.com/kb/install-pip-windows/ - but please be > aware that what comes-with varies according to the source used to obtain > the copy of Python, and (perhaps) the version of MS-Win. YMMV! > -- > Regards, > =dn > -- > https://mail.python.org/mailman/listinfo/python-list > From eryksun at gmail.com Mon Jul 18 13:15:59 2022 From: eryksun at gmail.com (Eryk Sun) Date: Mon, 18 Jul 2022 12:15:59 -0500 Subject: script folder is empty In-Reply-To: References: Message-ID: On 7/17/22, Scott Baer wrote: > > I've done some troubleshooting, and nothing is in the C:\Program > Files\Python310\Scripts folder. The installer may have silently failed to install pip. Run the following command to check whether the package was installed. "C:\Program Files\Python310\python.exe" -m pip -V The standard library includes pip (a third-party package) via the "ensurepip" bundle. You can use it to install pip with the following command: "C:\Program Files\Python310\python.exe" -m ensurepip --default-pip Since your Python installation is for all users, this command must be run from an elevated shell, i.e. "run as administrator". If your user is in the administrators group and UAC is enabled (the default configuration), then the system logs you on without administrator rights and privileges. You have to elevate via the UAC consent prompt in order to run a program with administrator access. For installing other packages, in most cases you should run pip in a virtual environment, or at least just for the current user via --user. Installing to the system site packages should be limited to common packages that you need for system services and administration across multiple accounts, such as the pywin32 package. From morphex at gmail.com Mon Jul 18 12:31:28 2022 From: morphex at gmail.com (Morten W. Petersen) Date: Mon, 18 Jul 2022 18:31:28 +0200 Subject: Simple message passing system and thread safe message queue Message-ID: Hi. I wrote a couple of blog posts as I had to create a message passing system, and these posts are here: http://blogologue.com/search?category=1658082823X26 Any comments or suggestions? Regards, Morten -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ From martinp.dipaola at gmail.com Mon Jul 18 18:18:14 2022 From: martinp.dipaola at gmail.com (Martin Di Paola) Date: Mon, 18 Jul 2022 19:18:14 -0300 Subject: Simple message passing system and thread safe message queue In-Reply-To: References: Message-ID: <20220718221814.itnzodwvoaemc6hp@gmail.com> Hi, I couldn't read your posts, every time I try to open one I'm redirected to an index page. I took a look at the smps project and I as far I understand it is a SSL client that sends messages to a server that implements a store of messages. I would suggest to remove the sleep() calls and as a challenge for you, make the server single-thread using asyncio and friends. Thanks, Martin. On Mon, Jul 18, 2022 at 06:31:28PM +0200, Morten W. Petersen wrote: >Hi. > >I wrote a couple of blog posts as I had to create a message passing system, >and these posts are here: > >http://blogologue.com/search?category=1658082823X26 > >Any comments or suggestions? > >Regards, > >Morten > >-- >I am https://leavingnorway.info >Videos at https://www.youtube.com/user/TheBlogologue >Twittering at http://twitter.com/blogologue >Blogging at http://blogologue.com >Playing music at https://soundcloud.com/morten-w-petersen >Also playing music and podcasting here: >http://www.mixcloud.com/morten-w-petersen/ >On Google+ here https://plus.google.com/107781930037068750156 >On Instagram at https://instagram.com/morphexx/ >-- >https://mail.python.org/mailman/listinfo/python-list From miked at dewhirst.com.au Mon Jul 18 19:41:12 2022 From: miked at dewhirst.com.au (Mike Dewhirst) Date: Tue, 19 Jul 2022 09:41:12 +1000 Subject: Simple message passing system and thread safe message queue In-Reply-To: Message-ID: <4LmzGq735yznVDc@mail.python.org> MortenI didn't click on your link because it is a security bad practice to click on any unsolicited link.I'm not accusing you or anyone else of nefarious intent.?It could be argued that being a member of this list means links posted here are indeed solicited links.You asked for comment. Mine is that your subject line was interesting enough for me to open your email but my own security habits prevented me from learning more about your topic.CheersMike--(Unsigned mail from my phone) -------- Original message --------From: "Morten W. Petersen" Date: 19/7/22 03:35 (GMT+10:00) To: python-list Subject: Simple message passing system and thread safe message queue Hi.I wrote a couple of blog posts as I had to create a message passing system,and these posts are here:http://blogologue.com/search?category=1658082823X26Any comments or suggestions?Regards,Morten-- I am https://leavingnorway.infoVideos at https://www.youtube.com/user/TheBlogologueTwittering at http://twitter.com/blogologueBlogging at http://blogologue.comPlaying music at https://soundcloud.com/morten-w-petersenAlso playing music and podcasting here:http://www.mixcloud.com/morten-w-petersen/On Google+ here https://plus.google.com/107781930037068750156On Instagram at https://instagram.com/morphexx/-- https://mail.python.org/mailman/listinfo/python-list From morphex at gmail.com Tue Jul 19 01:53:29 2022 From: morphex at gmail.com (Morten W. Petersen) Date: Tue, 19 Jul 2022 07:53:29 +0200 Subject: Simple message passing system and thread safe message queue In-Reply-To: <20220718221814.itnzodwvoaemc6hp@gmail.com> References: <20220718221814.itnzodwvoaemc6hp@gmail.com> Message-ID: Hi Martin. Yes, I was thinking of removing the sleep calls in the client.py / making a more intensive test. No, I think I'll stick with threads, I don't see any practical benefit of such a challenge. Maybe you could say a bit about why you think that's a good idea? Regards, Morten -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Instagram at https://instagram.com/morphexx/ tir. 19. jul. 2022, 00:21 skrev Martin Di Paola : > Hi, I couldn't read your posts, every time I try to open one I'm > redirected to an index page. > > I took a look at the smps project and I as far I understand it is a > SSL client that sends messages to a server that implements a store of > messages. > > I would suggest to remove the sleep() calls and as a challenge for you, > make the server single-thread using asyncio and friends. > > Thanks, > Martin. > > On Mon, Jul 18, 2022 at 06:31:28PM +0200, Morten W. Petersen wrote: > >Hi. > > > >I wrote a couple of blog posts as I had to create a message passing > system, > >and these posts are here: > > > >http://blogologue.com/search?category=1658082823X26 > > > >Any comments or suggestions? > > > >Regards, > > > >Morten > > > >-- > >I am https://leavingnorway.info > >Videos at https://www.youtube.com/user/TheBlogologue > >Twittering at http://twitter.com/blogologue > >Blogging at http://blogologue.com > >Playing music at https://soundcloud.com/morten-w-petersen > >Also playing music and podcasting here: > >http://www.mixcloud.com/morten-w-petersen/ > >On Google+ here https://plus.google.com/107781930037068750156 > >On Instagram at https://instagram.com/morphexx/ > >-- > >https://mail.python.org/mailman/listinfo/python-list > -- > https://mail.python.org/mailman/listinfo/python-list > From antoon.pardon at vub.be Tue Jul 19 09:58:41 2022 From: antoon.pardon at vub.be (Antoon Pardon) Date: Tue, 19 Jul 2022 15:58:41 +0200 Subject: My first attempt at a package. Message-ID: I am writing a python package which has the following structure PACKAGE ?? * module1.py ?? * module2.py ?? * data.cfg However the data.cfg should be build at installation time. Can someone give advise on which packaging tool and how to use it, to accomplisch this. e.g. de data config.cfg should be build with the following linix command: $ date '+installed on %Y:%M:%d at %T' > data.cfg -- Antoon Pardon. From david at lowryduda.com Tue Jul 19 10:57:40 2022 From: david at lowryduda.com (David Lowry-Duda) Date: Tue, 19 Jul 2022 08:57:40 -0600 Subject: My first attempt at a package. In-Reply-To: References: Message-ID: On Tue, Jul 19, 2022 at 03:58:41PM +0200, Antoon Pardon wrote: >I am writing a python package which has the following structure > >PACKAGE >?? * module1.py >?? * module2.py >?? * data.cfg > >However the data.cfg should be build at installation time. > >Can someone give advise on which packaging tool and how >to use it, to accomplisch this. A lot of good packaging information can be found at https://packaging.python.org/en/latest/ and in particular at https://packaging.python.org/en/latest/tutorials/packaging-projects/ and https://packaging.python.org/en/latest/overview/ There are a couple of different ways to handle python packaging, and it can be a bit confusing. But following the tutorial on packaging is a good start. - DLD From David.Raymond at tomtom.com Tue Jul 19 14:43:52 2022 From: David.Raymond at tomtom.com (David Raymond) Date: Tue, 19 Jul 2022 18:43:52 +0000 Subject: Pip upgrade causing issues in 3.10 Message-ID: So after a long while I'm finally getting around to upgrading to 3.10 on Windows from 3.9, and my first pip upgrade is causing issues with the installation. Problem seems to be that I run pip from a command prompt in the Scripts folder, and it seems pip is trying to completely remove the Scripts folder, which it can't because the command prompt is in that folder. It successfully removes the old installation of pip before dying, leaving you without a pip to run. (Yay for ensurepip) It also creates a ~ripts folder in the main Python folder, and a few ~ip folders in the site-packages folder before it dies, which then cause warnings to be issued by pip from then on (or at least until you go and delete those folders) Upgrading pip this way worked fine in 3.9.x, so I'm curious as to why the change for 3.10? And is there a chance to go back to it not trying to completely remove that Scripts folder? Or at least have it fail a little more gracefully without creating all those ~ folders? Maybe add a hint about ensurepip when it removes the old and can't install the new? For those who will say just change how you update it: I can do that going forward, yes. But nonetheless, running it like this probably shouldn't leave me without a pip, and with extra junk folders. Fresh install here of 3.10.5. For all users, not including the py launcher thing. Below is running in an Admin Command Prompt. C:\Program Files\Python310\Scripts>pip list Package Version ---------- ------- pip 22.0.4 setuptools 58.1.0 WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available. You should consider upgrading via the 'C:\Program Files\Python310\python.exe -m pip install --upgrade pip' command. C:\Program Files\Python310\Scripts>..\python.exe -m pip install --upgrade pip Requirement already satisfied: pip in c:\program files\python310\lib\site-packages (22.0.4) Collecting pip Using cached pip-22.1.2-py3-none-any.whl (2.1 MB) Installing collected packages: pip Attempting uninstall: pip Found existing installation: pip 22.0.4 Uninstalling pip-22.0.4: ERROR: Could not install packages due to an OSError: [WinError 32] The process cannot access the file because it is being used by another process: 'c:\\program files\\python310\\scripts\\' Consider using the `--user` option or check the permissions. At this point the Scripts folder is completely empty, there's a ~ripts folder in the main python310 folder, and a few ~ip... folders in the site-packages folder C:\Program Files\Python310\Scripts>..\python.exe -m ensurepip WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) Looking in links: c:\Users\usernamehere\AppData\Local\Temp\tmpt48ibdf3 Requirement already satisfied: setuptools in c:\program files\python310\lib\site-packages (58.1.0) Processing c:\users\usernamehere\appdata\local\temp\tmpt48ibdf3\pip-22.0.4-py3-none-any.whl WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) Installing collected packages: pip WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) Successfully installed pip-22.0.4 C:\Program Files\Python310\Scripts>cd.. C:\Program Files\Python310>python.exe -m pip install -U pip WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) Requirement already satisfied: pip in c:\program files\python310\lib\site-packages (22.0.4) Collecting pip Using cached pip-22.1.2-py3-none-any.whl (2.1 MB) WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) Installing collected packages: pip Attempting uninstall: pip WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) Found existing installation: pip 22.0.4 Uninstalling pip-22.0.4: Successfully uninstalled pip-22.0.4 WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) Successfully installed pip-22.1.2 WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) C:\Program Files\Python310>cd Scripts C:\Program Files\Python310\Scripts>pip list WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) Package Version ---------- ------- pip 22.1.2 setuptools 58.1.0 WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) WARNING: Ignoring invalid distribution -ip (c:\program files\python310\lib\site-packages) From ptq008 at gmail.com Tue Jul 19 11:38:26 2022 From: ptq008 at gmail.com (Tianhe) Date: Tue, 19 Jul 2022 23:38:26 +0800 Subject: Are there any benefits of a shared build python, compared to a static build python? Message-ID: Python by default builds the library `libpythonMAJOR.MINOR.a` and statically links it into the interpreter. Also it has an `--enable-shared`, (https://docs.python.org/3/using/configure.html#cmdoption-enable-shared) flag, which will build a shared library `libpythonMAJOR.MINOR.so.1.0`, and dynamically link it to the interpreter. Are there any benefits of a shared build python, compared to a static build python? Is there any performance difference? I found a related question on SO and is it like he said ( https://stackoverflow.com/a/73035776/5983841), shared built vs statically built python only affects the "Embedding Python in Another Application" scenario ? -- Regards, Micky From miked at dewhirst.com.au Wed Jul 20 02:01:00 2022 From: miked at dewhirst.com.au (Mike Dewhirst) Date: Wed, 20 Jul 2022 16:01:00 +1000 Subject: Pip upgrade causing issues in 3.10 In-Reply-To: References: Message-ID: <5fd0bdf4-02c5-615b-c636-7dc521251a57@dewhirst.com.au> On 20/07/2022 4:43 am, David Raymond wrote: > C:\Program Files\Python310\Scripts>..\python.exe -m pip install --upgrade pip > ERROR: Could not install packages due to an OSError: [WinError 32] The process cannot access the file because it is being used by another process: 'c:\\program files\\python310\\scripts\\' There's your problem. The 'other' process is your cmd.exe within which you are typing etc. Python scripts dir should be on the path so you don't have to execute anything from within it. Windows is obviously tripping over its own toes trying to delete and install something in the same dir while you also have your foot in it. M -- Signed email is an absolute defence against phishing. This email has been signed with my private key. If you import my public key you can automatically decrypt my signature and be sure it came from me. Just ask and I'll send it to you. Your email software can handle signing. -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature Type: application/pgp-signature Size: 495 bytes Desc: OpenPGP digital signature URL: From frank at chagford.com Wed Jul 20 04:32:07 2022 From: frank at chagford.com (Frank Millman) Date: Wed, 20 Jul 2022 10:32:07 +0200 Subject: list indices must be integers or slices, not str Message-ID: <06dda811-2a19-b575-7837-e14415ea912b@chagford.com> Hi all C:\Users\E7280>python Python 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> >>> x = list(range(10)) >>> >>> '{x[1]}'.format(**vars()) '1' >>> >>> '{x[-1]}'.format(**vars()) Traceback (most recent call last): ? File "", line 1, in TypeError: list indices must be integers or slices, not str >>> Can anyone explain this error? It seems that a negative index is deemed to be a string in this case. Thanks Frank Millman From rosuav at gmail.com Wed Jul 20 05:37:06 2022 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 20 Jul 2022 19:37:06 +1000 Subject: list indices must be integers or slices, not str In-Reply-To: <06dda811-2a19-b575-7837-e14415ea912b@chagford.com> References: <06dda811-2a19-b575-7837-e14415ea912b@chagford.com> Message-ID: On Wed, 20 Jul 2022 at 18:34, Frank Millman wrote: > > Hi all > > C:\Users\E7280>python > Python 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 > bit (AMD64)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> > >>> x = list(range(10)) > >>> > >>> '{x[1]}'.format(**vars()) > '1' > >>> > >>> '{x[-1]}'.format(**vars()) > Traceback (most recent call last): > File "", line 1, in > TypeError: list indices must be integers or slices, not str > >>> > > Can anyone explain this error? It seems that a negative index is deemed > to be a string in this case. > Yeah, that does seem a little odd. What you're seeing is the same as this phenomenon: >>> "{x[1]} {x[spam]}".format(x={1: 42, "spam": "ham"}) '42 ham' >>> "{x[1]} {x[spam]}".format(x={"1": 42, "spam": "ham"}) Traceback (most recent call last): File "", line 1, in KeyError: 1 But I can't find it documented anywhere that digits-only means numeric. The best I can find is: https://docs.python.org/3/library/string.html#formatstrings """The arg_name can be followed by any number of index or attribute expressions. An expression of the form '.name' selects the named attribute using getattr(), while an expression of the form '[index]' does an index lookup using __getitem__().""" and in the corresponding grammar: field_name ::= arg_name ("." attribute_name | "[" element_index "]")* index_string ::= + In other words, any sequence of characters counts as an argument, as long as it's not ambiguous. It doesn't seem to say that "all digits is interpreted as an integer, everything else is interpreted as a string". ISTM that a negative number should be interpreted as an integer too, but that might be a backward compatibility break. ChrisA From eryksun at gmail.com Wed Jul 20 06:36:50 2022 From: eryksun at gmail.com (Eryk Sun) Date: Wed, 20 Jul 2022 05:36:50 -0500 Subject: Pip upgrade causing issues in 3.10 In-Reply-To: <5fd0bdf4-02c5-615b-c636-7dc521251a57@dewhirst.com.au> References: <5fd0bdf4-02c5-615b-c636-7dc521251a57@dewhirst.com.au> Message-ID: On 7/20/22, Mike Dewhirst wrote: > On 20/07/2022 4:43 am, David Raymond wrote: >> C:\Program Files\Python310\Scripts>..\python.exe -m pip install --upgrade >> pip >> ERROR: Could not install packages due to an OSError: [WinError 32] The >> process cannot access the file because it is being used by another >> process: 'c:\\program files\\python310\\scripts\\' > There's your problem. The 'other' process is your cmd.exe within which > you are typing etc. > > Python scripts dir should be on the path so you don't have to execute > anything from within it. Windows is obviously tripping over its own toes > trying to delete and install something in the same dir while you also > have your foot in it. This should only occur if uninstalling pip deletes all of the files in the "Scripts" directory. In this case, pip will 'compress' the individual delete operations to remove the entire directory. It begins by trying to 'stash' the "Scripts" directory, i.e. by renaming it to a temporary adjacent name. It uses shutil.move() for this, which tries os.rename() and falls back on shutil.copytree() and shutil.rmtree(). Of course this fails with a sharing violation if the directory is open as the working directory in any process, including the current Python process, because an open for a working directory doesn't share delete/rename access. pip fails to handle this error. It crashes and leaves a mess. The empty "Scripts" directory and the copytree() copy aren't rolled back properly, and neither are the stashed (renamed) "pip" and "pip*dist-info" directories in site packages. This is not user error. It is a bug in pip. It should be able to recover gracefully from failing to delete the directory. Moreover, it shouldn't even have to roll back the operations if it deleted the directory due to compression of explicit removals from the install "RECORD". In particular there is no reason to delete the "Scripts" directory, even if it's left empty. From frank at chagford.com Wed Jul 20 06:31:58 2022 From: frank at chagford.com (Frank Millman) Date: Wed, 20 Jul 2022 12:31:58 +0200 Subject: list indices must be integers or slices, not str In-Reply-To: References: <06dda811-2a19-b575-7837-e14415ea912b@chagford.com> Message-ID: <0d6e1849-1e4d-3587-7dc6-748e1b8ce17c@chagford.com> On 2022-07-20 11:37 AM, Chris Angelico wrote: > On Wed, 20 Jul 2022 at 18:34, Frank Millman wrote: >> >> Hi all >> >> C:\Users\E7280>python >> Python 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 >> bit (AMD64)] on win32 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> >> >>> x = list(range(10)) >> >>> >> >>> '{x[1]}'.format(**vars()) >> '1' >> >>> >> >>> '{x[-1]}'.format(**vars()) >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: list indices must be integers or slices, not str >> >>> >> >> Can anyone explain this error? It seems that a negative index is deemed >> to be a string in this case. >> > > Yeah, that does seem a little odd. What you're seeing is the same as > this phenomenon: > >>>> "{x[1]} {x[spam]}".format(x={1: 42, "spam": "ham"}) > '42 ham' >>>> "{x[1]} {x[spam]}".format(x={"1": 42, "spam": "ham"}) > Traceback (most recent call last): > File "", line 1, in > KeyError: 1 > > But I can't find it documented anywhere that digits-only means > numeric. The best I can find is: > > https://docs.python.org/3/library/string.html#formatstrings > """The arg_name can be followed by any number of index or attribute > expressions. An expression of the form '.name' selects the named > attribute using getattr(), while an expression of the form '[index]' > does an index lookup using __getitem__().""" > > and in the corresponding grammar: > > field_name ::= arg_name ("." attribute_name | "[" element_index "]")* > index_string ::= + > > In other words, any sequence of characters counts as an argument, as > long as it's not ambiguous. It doesn't seem to say that "all digits is > interpreted as an integer, everything else is interpreted as a > string". ISTM that a negative number should be interpreted as an > integer too, but that might be a backward compatibility break. > Thanks for investigating this further. I agree it seems odd. As quoted above, an expression of the form '[index]' does an index lookup using __getitem()__. The only __getitem__() that I can find is in the operator module, and that handles negative numbers just fine. Do you think it is worth me raising an issue, if only to find out the rationale if there is one? Frank From frank at chagford.com Wed Jul 20 07:04:45 2022 From: frank at chagford.com (Frank Millman) Date: Wed, 20 Jul 2022 13:04:45 +0200 Subject: list indices must be integers or slices, not str In-Reply-To: <0d6e1849-1e4d-3587-7dc6-748e1b8ce17c@chagford.com> References: <06dda811-2a19-b575-7837-e14415ea912b@chagford.com> <0d6e1849-1e4d-3587-7dc6-748e1b8ce17c@chagford.com> Message-ID: <6ad50524-a639-ed31-0d1d-17ac8e402ed2@chagford.com> On 2022-07-20 12:31 PM, Frank Millman wrote: > On 2022-07-20 11:37 AM, Chris Angelico wrote: >> On Wed, 20 Jul 2022 at 18:34, Frank Millman wrote: >>> >>> Hi all >>> >>> C:\Users\E7280>python >>> Python 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 >>> bit (AMD64)] on win32 >>> Type "help", "copyright", "credits" or "license" for more information. >>> ? >>> >>> ? >>> x = list(range(10)) >>> ? >>> >>> ? >>> '{x[1]}'.format(**vars()) >>> '1' >>> ? >>> >>> ? >>> '{x[-1]}'.format(**vars()) >>> Traceback (most recent call last): >>> ??? File "", line 1, in >>> TypeError: list indices must be integers or slices, not str >>> ? >>> >>> >>> Can anyone explain this error? It seems that a negative index is deemed >>> to be a string in this case. >>> >> >> Yeah, that does seem a little odd. What you're seeing is the same as >> this phenomenon: >> >>>>> "{x[1]} {x[spam]}".format(x={1: 42, "spam": "ham"}) >> '42 ham' >>>>> "{x[1]} {x[spam]}".format(x={"1": 42, "spam": "ham"}) >> Traceback (most recent call last): >> ?? File "", line 1, in >> KeyError: 1 >> >> But I can't find it documented anywhere that digits-only means >> numeric. The best I can find is: >> >> https://docs.python.org/3/library/string.html#formatstrings >> """The arg_name can be followed by any number of index or attribute >> expressions. An expression of the form '.name' selects the named >> attribute using getattr(), while an expression of the form '[index]' >> does an index lookup using __getitem__().""" >> >> and in the corresponding grammar: >> >> field_name??????? ::=? arg_name ("." attribute_name | "[" >> element_index "]")* >> index_string????? ::=? + >> >> In other words, any sequence of characters counts as an argument, as >> long as it's not ambiguous. It doesn't seem to say that "all digits is >> interpreted as an integer, everything else is interpreted as a >> string". ISTM that a negative number should be interpreted as an >> integer too, but that might be a backward compatibility break. >> > > Thanks for investigating this further. I agree it seems odd. > > As quoted above, an expression of the form '[index]' does an index > lookup using __getitem()__. > > The only __getitem__() that I can find is in the operator module, and > that handles negative numbers just fine. > > Do you think it is worth me raising an issue, if only to find out the > rationale if there is one? > > Frank I saw this from Paul Rubin - for some reason his posts appear in google groups, but not python-list. "It seems to only want integer constants. x[2+2] and x[k] where k=2 don't work either. I think the preferred style these days is f'{x[-1]}' which works." Unfortunately the 'f' option does not work for me in this case, as I am using a string object, not a string literal. Frank From rosuav at gmail.com Wed Jul 20 07:08:35 2022 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 20 Jul 2022 21:08:35 +1000 Subject: list indices must be integers or slices, not str In-Reply-To: <0d6e1849-1e4d-3587-7dc6-748e1b8ce17c@chagford.com> References: <06dda811-2a19-b575-7837-e14415ea912b@chagford.com> <0d6e1849-1e4d-3587-7dc6-748e1b8ce17c@chagford.com> Message-ID: On Wed, 20 Jul 2022 at 20:55, Frank Millman wrote: > > On 2022-07-20 11:37 AM, Chris Angelico wrote: > > On Wed, 20 Jul 2022 at 18:34, Frank Millman wrote: > >> > >> Hi all > >> > >> C:\Users\E7280>python > >> Python 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 > >> bit (AMD64)] on win32 > >> Type "help", "copyright", "credits" or "license" for more information. > >> >>> > >> >>> x = list(range(10)) > >> >>> > >> >>> '{x[1]}'.format(**vars()) > >> '1' > >> >>> > >> >>> '{x[-1]}'.format(**vars()) > >> Traceback (most recent call last): > >> File "", line 1, in > >> TypeError: list indices must be integers or slices, not str > >> >>> > >> > >> Can anyone explain this error? It seems that a negative index is deemed > >> to be a string in this case. > >> > > > > Yeah, that does seem a little odd. What you're seeing is the same as > > this phenomenon: > > > >>>> "{x[1]} {x[spam]}".format(x={1: 42, "spam": "ham"}) > > '42 ham' > >>>> "{x[1]} {x[spam]}".format(x={"1": 42, "spam": "ham"}) > > Traceback (most recent call last): > > File "", line 1, in > > KeyError: 1 > > > > But I can't find it documented anywhere that digits-only means > > numeric. The best I can find is: > > > > https://docs.python.org/3/library/string.html#formatstrings > > """The arg_name can be followed by any number of index or attribute > > expressions. An expression of the form '.name' selects the named > > attribute using getattr(), while an expression of the form '[index]' > > does an index lookup using __getitem__().""" > > > > and in the corresponding grammar: > > > > field_name ::= arg_name ("." attribute_name | "[" element_index "]")* > > index_string ::= + > > > > In other words, any sequence of characters counts as an argument, as > > long as it's not ambiguous. It doesn't seem to say that "all digits is > > interpreted as an integer, everything else is interpreted as a > > string". ISTM that a negative number should be interpreted as an > > integer too, but that might be a backward compatibility break. > > > > Thanks for investigating this further. I agree it seems odd. > > As quoted above, an expression of the form '[index]' does an index > lookup using __getitem()__. > > The only __getitem__() that I can find is in the operator module, and > that handles negative numbers just fine. In general, __getitem__ is the method used to handle those sorts of lookups: class X: def __getitem__(self, item): print("Get item", type(item), item) "{x[0]} {x[1]} {x[-1]} {x[spam]} {x[1.0]}".format(x=X()) Outside of a format directive, you'd need to quote those: x[0], x[1], x["spam"] The distinction is that unquoted bare numbers are interpreted as integers, not as strings. I'm unable to find the exact definition of that documented. > Do you think it is worth me raising an issue, if only to find out the > rationale if there is one? > I'd wait for other people's responses first, there may be a better insight to be found than what I was able to come across. ChrisA From rosuav at gmail.com Wed Jul 20 07:14:20 2022 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 20 Jul 2022 21:14:20 +1000 Subject: list indices must be integers or slices, not str In-Reply-To: <6ad50524-a639-ed31-0d1d-17ac8e402ed2@chagford.com> References: <06dda811-2a19-b575-7837-e14415ea912b@chagford.com> <0d6e1849-1e4d-3587-7dc6-748e1b8ce17c@chagford.com> <6ad50524-a639-ed31-0d1d-17ac8e402ed2@chagford.com> Message-ID: On Wed, 20 Jul 2022 at 21:06, Frank Millman wrote: > I saw this from Paul Rubin - for some reason his posts appear in google > groups, but not python-list. > > "It seems to only want integer constants. x[2+2] and x[k] where k=2 > don't work either. Yes, that's for the same reason that x[spam] can be used usefully with a dictionary. Otherwise you'd need to use quotes. It makes perfect sense that both 2+2 and k are treated as strings. > I think the preferred style these days is f'{x[-1]}' which works." Not true; there's no single "preferred style", and f-strings are absolutely NOT replacements for everything else. They have their place, as do the others. Yes, including percent formatting, it is not deprecated, and it's really tiresome when people claim that it is. > Unfortunately the 'f' option does not work for me in this case, as I am > using a string object, not a string literal. Right. An f-string uses the exact syntax of a Python expression, which is often too powerful, but also restricts it to the string literal style (since it's actual code, not a method call). For other purposes, .format() is a better choice. ChrisA From __peter__ at web.de Wed Jul 20 09:42:27 2022 From: __peter__ at web.de (Peter Otten) Date: Wed, 20 Jul 2022 15:42:27 +0200 Subject: list indices must be integers or slices, not str In-Reply-To: References: <06dda811-2a19-b575-7837-e14415ea912b@chagford.com> Message-ID: On 20/07/2022 11:37, Chris Angelico wrote: > On Wed, 20 Jul 2022 at 18:34, Frank Millman wrote: >> >> Hi all >> >> C:\Users\E7280>python >> Python 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 >> bit (AMD64)] on win32 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> >> >>> x = list(range(10)) >> >>> >> >>> '{x[1]}'.format(**vars()) >> '1' >> >>> >> >>> '{x[-1]}'.format(**vars()) >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: list indices must be integers or slices, not str >> >>> >> >> Can anyone explain this error? It seems that a negative index is deemed >> to be a string in this case. >> > > Yeah, that does seem a little odd. What you're seeing is the same as > this phenomenon: > >>>> "{x[1]} {x[spam]}".format(x={1: 42, "spam": "ham"}) > '42 ham' >>>> "{x[1]} {x[spam]}".format(x={"1": 42, "spam": "ham"}) > Traceback (most recent call last): > File "", line 1, in > KeyError: 1 > > But I can't find it documented anywhere that digits-only means > numeric. I found https://peps.python.org/pep-3101/ """ PEP 3101 ? Advanced String Formatting ... An example of the ?getitem? syntax: "My name is {0[name]}".format(dict(name='Fred')) It should be noted that the use of ?getitem? within a format string is much more limited than its conventional usage. In the above example, the string ?name? really is the literal string ?name?, not a variable named ?name?. The rules for parsing an item key are very simple. If it starts with a digit, then it is treated as a number, otherwise it is used as a string. Because keys are not quote-delimited, it is not possible to specify arbitrary dictionary keys (e.g., the strings ?10? or ?:-]?) from within a format string. """ From python at mrabarnett.plus.com Wed Jul 20 10:21:23 2022 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 20 Jul 2022 15:21:23 +0100 Subject: list indices must be integers or slices, not str In-Reply-To: References: <06dda811-2a19-b575-7837-e14415ea912b@chagford.com> <0d6e1849-1e4d-3587-7dc6-748e1b8ce17c@chagford.com> Message-ID: <12c4d762-c024-910d-081b-1d8ce075dc33@mrabarnett.plus.com> On 20/07/2022 12:08, Chris Angelico wrote: > On Wed, 20 Jul 2022 at 20:55, Frank Millman wrote: >> >> On 2022-07-20 11:37 AM, Chris Angelico wrote: >> > On Wed, 20 Jul 2022 at 18:34, Frank Millman wrote: >> >> >> >> Hi all >> >> >> >> C:\Users\E7280>python >> >> Python 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 >> >> bit (AMD64)] on win32 >> >> Type "help", "copyright", "credits" or "license" for more information. >> >> >>> >> >> >>> x = list(range(10)) >> >> >>> >> >> >>> '{x[1]}'.format(**vars()) >> >> '1' >> >> >>> >> >> >>> '{x[-1]}'.format(**vars()) >> >> Traceback (most recent call last): >> >> File "", line 1, in >> >> TypeError: list indices must be integers or slices, not str >> >> >>> >> >> >> >> Can anyone explain this error? It seems that a negative index is deemed >> >> to be a string in this case. >> >> >> > >> > Yeah, that does seem a little odd. What you're seeing is the same as >> > this phenomenon: >> > >> >>>> "{x[1]} {x[spam]}".format(x={1: 42, "spam": "ham"}) >> > '42 ham' >> >>>> "{x[1]} {x[spam]}".format(x={"1": 42, "spam": "ham"}) >> > Traceback (most recent call last): >> > File "", line 1, in >> > KeyError: 1 >> > >> > But I can't find it documented anywhere that digits-only means >> > numeric. The best I can find is: >> > >> > https://docs.python.org/3/library/string.html#formatstrings >> > """The arg_name can be followed by any number of index or attribute >> > expressions. An expression of the form '.name' selects the named >> > attribute using getattr(), while an expression of the form '[index]' >> > does an index lookup using __getitem__().""" >> > >> > and in the corresponding grammar: >> > >> > field_name ::= arg_name ("." attribute_name | "[" element_index "]")* >> > index_string ::= + >> > >> > In other words, any sequence of characters counts as an argument, as >> > long as it's not ambiguous. It doesn't seem to say that "all digits is >> > interpreted as an integer, everything else is interpreted as a >> > string". ISTM that a negative number should be interpreted as an >> > integer too, but that might be a backward compatibility break. >> > >> >> Thanks for investigating this further. I agree it seems odd. >> >> As quoted above, an expression of the form '[index]' does an index >> lookup using __getitem()__. >> >> The only __getitem__() that I can find is in the operator module, and >> that handles negative numbers just fine. > > In general, __getitem__ is the method used to handle those sorts of lookups: > > class X: > def __getitem__(self, item): > print("Get item", type(item), item) > > "{x[0]} {x[1]} {x[-1]} {x[spam]} {x[1.0]}".format(x=X()) > > Outside of a format directive, you'd need to quote those: > > x[0], x[1], x["spam"] > > The distinction is that unquoted bare numbers are interpreted as > integers, not as strings. I'm unable to find the exact definition of > that documented. > >> Do you think it is worth me raising an issue, if only to find out the >> rationale if there is one? >> > > I'd wait for other people's responses first, there may be a better > insight to be found than what I was able to come across. > It's mentioned in PEP 3101 (Advanced String Formatting): """ The rules for parsing an item key are very simple. If it starts with a digit, then it is treated as a number, otherwise it is used as a string. """ I have a vague memory of it being discussed on python-dev, but it was a long time ago. From rosuav at gmail.com Wed Jul 20 10:45:07 2022 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 21 Jul 2022 00:45:07 +1000 Subject: list indices must be integers or slices, not str In-Reply-To: References: <06dda811-2a19-b575-7837-e14415ea912b@chagford.com> Message-ID: On Wed, 20 Jul 2022 at 23:50, Peter Otten <__peter__ at web.de> wrote: > > I found > > https://peps.python.org/pep-3101/ > > """ > PEP 3101 ? Advanced String Formatting > ... > An example of the ?getitem? syntax: > > "My name is {0[name]}".format(dict(name='Fred')) > > It should be noted that the use of ?getitem? within a format string is > much more limited than its conventional usage. In the above example, the > string ?name? really is the literal string ?name?, not a variable named > ?name?. The rules for parsing an item key are very simple. If it starts > with a digit, then it is treated as a number, otherwise it is used as a > string. > Cool. I think this is a good justification for a docs patch, since that really should be mentioned somewhere other than a historical document. ChrisA From mats at wichmann.us Wed Jul 20 12:46:35 2022 From: mats at wichmann.us (Mats Wichmann) Date: Wed, 20 Jul 2022 10:46:35 -0600 Subject: list indices must be integers or slices, not str In-Reply-To: <6ad50524-a639-ed31-0d1d-17ac8e402ed2@chagford.com> References: <06dda811-2a19-b575-7837-e14415ea912b@chagford.com> <0d6e1849-1e4d-3587-7dc6-748e1b8ce17c@chagford.com> <6ad50524-a639-ed31-0d1d-17ac8e402ed2@chagford.com> Message-ID: On 7/20/22 05:04, Frank Millman wrote: > I think the preferred style these days is f'{x[-1]}' which works." > > Unfortunately the 'f' option does not work for me in this case, as I am > using a string object, not a string literal. For that you could consider https://pypi.org/project/f-yeah/ (straying a bit off thread subject by now, admittedly) From george.trojan at gmail.com Wed Jul 20 12:56:02 2022 From: george.trojan at gmail.com (george trojan) Date: Wed, 20 Jul 2022 16:56:02 +0000 Subject: exec() an locals() puzzle Message-ID: I wish I could understand the following behaviour: 1. This works as I expect it to work: def f(): i = 1 print(locals()) exec('y = i; print(y); print(locals())') print(locals()) exec('y *= 2') print('ok:', eval('y')) f() {'i': 1} 1 {'i': 1, 'y': 1} {'i': 1, 'y': 1} ok: 2 2. I can access the value of y with eval() too: def f(): i = 1 print(locals()) exec('y = i; print(y); print(locals())') print(locals()) u = eval('y') print(u) f() {'i': 1} 1 {'i': 1, 'y': 1} {'i': 1, 'y': 1} 1 3. When I change variable name u -> y, somehow locals() in the body of the function loses an entry: def f(): i = 1 print(locals()) exec('y = i; print(y); print(locals())') print(locals()) y = eval('y') print(y) f() {'i': 1} 1 {'i': 1, 'y': 1} {'i': 1} ---------------------------------------------------------------------------NameError Traceback (most recent call last) Input In [1], in () 7 print(y) 8 # y = eval('y') 9 #print('ok:', eval('y'))---> 10 f() Input In [1], in f() 4 exec('y = i; print(y); print(locals())') 5 print(locals())----> 6 y = eval('y') 7 print(y) File :1, in NameError: name 'y' is not defined1. Another thing: within the first exec(), the print order seems reversed. What is going on? BTW, I am using python 3.8.13. George From barry at barrys-emacs.org Wed Jul 20 13:50:07 2022 From: barry at barrys-emacs.org (Barry) Date: Wed, 20 Jul 2022 18:50:07 +0100 Subject: Are there any benefits of a shared build python, compared to a static build python? In-Reply-To: References: Message-ID: > On 20 Jul 2022, at 18:09, Tianhe wrote: > > ?Python by default builds the library `libpythonMAJOR.MINOR.a` and > statically links it into the interpreter. Also it has an `--enable-shared`, > (https://docs.python.org/3/using/configure.html#cmdoption-enable-shared) > flag, which will build a shared library `libpythonMAJOR.MINOR.so.1.0`, and > dynamically link it to the interpreter. > > Are there any benefits of a shared build python, compared to a static build > python? Is there any performance difference? What appears to be important is being able to do link time optimisation of python code. I think Debian starting doing this first and other distro followed after seeing the big performance improvement Debian got. Barry > > I found a related question on SO and is it like he said ( > https://stackoverflow.com/a/73035776/5983841), shared built vs statically > built python only affects the "Embedding Python in Another Application" > scenario ? > > -- > > Regards, > > Micky > -- > https://mail.python.org/mailman/listinfo/python-list > From eryksun at gmail.com Wed Jul 20 14:54:06 2022 From: eryksun at gmail.com (Eryk Sun) Date: Wed, 20 Jul 2022 13:54:06 -0500 Subject: exec() an locals() puzzle In-Reply-To: References: Message-ID: On 7/20/22, george trojan wrote: > > 1. This works as I expect it to work: > > def f(): > i = 1 > print(locals()) > exec('y = i; print(y); print(locals())') > print(locals()) > exec('y *= 2') > print('ok:', eval('y')) > f() In CPython, the locals of a function scope (as opposed to a class scope or module scope) are optimized by storing them in an array in the current frame. locals(), however, always returns the non-optimized locals mapping of the current frame, i.e. the value of sys._getframe(0).f_locals. In the case of optimized locals, the locals() call first updates this dict with a snapshot of the current values of local variables. This is useful for introspection, but modifying the snapshot cannot extend or modify the optimized local variables of the function scope. exec() and eval() default to using the global and local variable mappings that are returned by globals() and locals(). This allows them to access the snapshot of the containing function's optimized locals, but they can't extend or modify the function's optimized locals. At most they can add dynamic locals to the snapshot, as used by subsequent calls to locals(), exec() and eval(). From martinp.dipaola at gmail.com Wed Jul 20 14:59:19 2022 From: martinp.dipaola at gmail.com (Martin Di Paola) Date: Wed, 20 Jul 2022 15:59:19 -0300 Subject: list indices must be integers or slices, not str In-Reply-To: References: <06dda811-2a19-b575-7837-e14415ea912b@chagford.com> <0d6e1849-1e4d-3587-7dc6-748e1b8ce17c@chagford.com> <6ad50524-a639-ed31-0d1d-17ac8e402ed2@chagford.com> Message-ID: <20220720185919.z3mmencghjtxaxgp@gmail.com> offtopic If you want a pure-python but definitely a more hacky implementation, you can play around with inspect.stack() and get the variables from the outer frames. # code: x = 32 y = 42 printf("Hello x={x}, y={y}", x=27) # output: Hello x=27, y=42 The implementation of printf() was never released in PyPI (I guess I never saw it as more than a challenge). But the implementation is quite simple (I did a post about it): https://book-of-gehn.github.io/articles/2021/07/11/Home-Made-Python-F-String.html Thanks, Martin. On Wed, Jul 20, 2022 at 10:46:35AM -0600, Mats Wichmann wrote: >On 7/20/22 05:04, Frank Millman wrote: > >> I think the preferred style these days is f'{x[-1]}' which works." >> >> Unfortunately the 'f' option does not work for me in this case, as I am >> using a string object, not a string literal. > >For that you could consider > >https://pypi.org/project/f-yeah/ > >(straying a bit off thread subject by now, admittedly) >-- >https://mail.python.org/mailman/listinfo/python-list From martinp.dipaola at gmail.com Wed Jul 20 15:28:28 2022 From: martinp.dipaola at gmail.com (Martin Di Paola) Date: Wed, 20 Jul 2022 16:28:28 -0300 Subject: exec() an locals() puzzle In-Reply-To: References: Message-ID: <20220720192828.hmxhuhsjr25jygjf@gmail.com> I did a few tests # test 1 def f(): i = 1 print(locals()) exec('y = i; print(y); print(locals())') print(locals()) a = eval('y') print(locals()) u = a print(u) f() {'i': 1} 1 {'i': 1, 'y': 1} {'i': 1, 'y': 1} {'i': 1, 'y': 1, 'a': 1} 1 # test 2 def f(): i = 1 print(locals()) exec('y = i; print(y); print(locals())') print(locals()) a = eval('y') print(locals()) y = a print(y) f() {'i': 1} 1 {'i': 1, 'y': 1} {'i': 1} Traceback (most recent call last): NameError: name 'y' is not defined So test 1 and 2 are the same except that the variable 'y' is not present/present in the f's code. When it is not present, exec() modifies the f's locals and adds an 'y' to it but when the variable 'y' is present in the code (even if not present in the locals()), exec() does not add any 'y' (and the next eval() then fails) The interesting part is that if the 'y' variable is in the f's code *and* it is defined in the f's locals, no error occur but once again the exec() does not modify f's locals: # test 3 def f(): i = 1 y = 42 print(locals()) exec('y = i; print(y); print(locals())') print(locals()) a = eval('y') print(locals()) y = a print(y) f() {'i': 1, 'y': 42} 1 {'i': 1, 'y': 1} {'i': 1, 'y': 42} {'i': 1, 'y': 42, 'a': 42} 42 Why does this happen? No idea. I may be related with this: # test 4 def f(): i = 1 print(locals()) exec('y = i; print(y); print(locals())') print(locals()) print(y) f() Traceback (most recent call last): NameError: name 'y' is not defined Despite exec() adds the 'y' variable to f's locals, the variable is not accessible/visible from f's code. So, a few observations (by no means this is how the vm works): 1) each function has a set of variables defined by the code (let's call this "code-defined locals" or "cdef-locals"). 2) each function also has a set of "runtime locals" accessible from locals(). 3) exec() can add variables to locals() (runtime) set but it cannot add any to cdef-locals. 4) locals() may be a superset of cdef-locals (but entries in cdef-locals which value is still undefined are not shown in locals()) 5) due rule 4, exec() cannot add a variable to locals() if it is already present in the in cdef-locals. 6) when eval() runs, it uses locals() set for lookup Perhaps rule 5 is to prevent exec() to modify any arbitrary variable of the caller... Anyways, nice food for our brains. On Wed, Jul 20, 2022 at 04:56:02PM +0000, george trojan wrote: >I wish I could understand the following behaviour: > >1. This works as I expect it to work: > >def f(): > i = 1 > print(locals()) > exec('y = i; print(y); print(locals())') > print(locals()) > exec('y *= 2') > print('ok:', eval('y')) >f() > >{'i': 1} >1 >{'i': 1, 'y': 1} >{'i': 1, 'y': 1} >ok: 2 > >2. I can access the value of y with eval() too: > >def f(): > i = 1 > print(locals()) > exec('y = i; print(y); print(locals())') > print(locals()) > u = eval('y') > print(u) >f() > >{'i': 1} >1 >{'i': 1, 'y': 1} >{'i': 1, 'y': 1} >1 > >3. When I change variable name u -> y, somehow locals() in the body of >the function loses an entry: > >def f(): > i = 1 > print(locals()) > exec('y = i; print(y); print(locals())') > print(locals()) > y = eval('y') > print(y) >f() > >{'i': 1} >1 >{'i': 1, 'y': 1} >{'i': 1} > >---------------------------------------------------------------------------NameError > Traceback (most recent call last) >Input In [1], in () 7 print(y) 8 # y >= eval('y') 9 #print('ok:', eval('y'))---> 10 f() > >Input In [1], in f() 4 exec('y = i; print(y); print(locals())') > 5 print(locals())----> 6 y = eval('y') 7 print(y) > >File :1, in >NameError: name 'y' is not defined1. > >Another thing: within the first exec(), the print order seems >reversed. What is going on? > >BTW, I am using python 3.8.13. > >George >-- >https://mail.python.org/mailman/listinfo/python-list From roel at roelschroeven.net Wed Jul 20 16:59:21 2022 From: roel at roelschroeven.net (Roel Schroeven) Date: Wed, 20 Jul 2022 22:59:21 +0200 Subject: list indices must be integers or slices, not str In-Reply-To: <6ad50524-a639-ed31-0d1d-17ac8e402ed2@chagford.com> References: <06dda811-2a19-b575-7837-e14415ea912b@chagford.com> <0d6e1849-1e4d-3587-7dc6-748e1b8ce17c@chagford.com> <6ad50524-a639-ed31-0d1d-17ac8e402ed2@chagford.com> Message-ID: <32cf1828-a074-6d5c-5f60-ab9f5b880a8a@roelschroeven.net> Frank Millman schreef op 20/07/2022 om 13:04: > >> On Wed, 20 Jul 2022 at 18:34, Frank Millman wrote: > >>> ? >>> > >>> ? >>> x = list(range(10)) > >>> ? >>> > >>> ? >>> '{x[1]}'.format(**vars()) > >>> '1' > >>> ? >>> > >>> ? >>> '{x[-1]}'.format(**vars()) > >>> Traceback (most recent call last): > >>> ??? File "", line 1, in > >>> TypeError: list indices must be integers or slices, not str > >>> ? >>> > >>> > >>> Can anyone explain this error? It seems that a negative index is deemed > >>> to be a string in this case. > >>> > >> > >> [...] > >> > > "It seems to only want integer constants. x[2+2] and x[k] where k=2 > don't work either. > > I think the preferred style these days is f'{x[-1]}' which works." > > Unfortunately the 'f' option does not work for me in this case, as I am > using a string object, not a string literal. > I've always found a bit dirty to pass indiscriminate symbol tables like the one from globals() or vars() or vars(some_object) to a formatting method, leaving the formatting string free to access any of it. I prefer to make things more explicit. Can't you make a dict with a selection of things you possible want to format? In a sense I think you're looking for a more complete templating solution than what Python's formatting mechanisms are designed for. Maybe you should consider a full template engine, like Jinja? -- "There is no cause so noble that it will not attract fuggheads." -- Larry Niven From frank at chagford.com Thu Jul 21 04:32:03 2022 From: frank at chagford.com (Frank Millman) Date: Thu, 21 Jul 2022 10:32:03 +0200 Subject: list indices must be integers or slices, not str In-Reply-To: References: <06dda811-2a19-b575-7837-e14415ea912b@chagford.com> Message-ID: On 2022-07-20 4:45 PM, Chris Angelico wrote: > On Wed, 20 Jul 2022 at 23:50, Peter Otten <__peter__ at web.de> wrote: >> >> I found >> >> https://peps.python.org/pep-3101/ >> >> """ >> PEP 3101 ? Advanced String Formatting >> ... >> An example of the ?getitem? syntax: >> >> "My name is {0[name]}".format(dict(name='Fred')) >> >> It should be noted that the use of ?getitem? within a format string is >> much more limited than its conventional usage. In the above example, the >> string ?name? really is the literal string ?name?, not a variable named >> ?name?. The rules for parsing an item key are very simple. If it starts >> with a digit, then it is treated as a number, otherwise it is used as a >> string. >> > > Cool. I think this is a good justification for a docs patch, since > that really should be mentioned somewhere other than a historical > document. > > ChrisA I have submitted the following - https://github.com/python/cpython/issues/95088 Frank From george.trojan at gmail.com Thu Jul 21 15:07:12 2022 From: george.trojan at gmail.com (george trojan) Date: Thu, 21 Jul 2022 19:07:12 +0000 Subject: exec() an locals() puzzle In-Reply-To: <20220720192828.hmxhuhsjr25jygjf@gmail.com> References: <20220720192828.hmxhuhsjr25jygjf@gmail.com> Message-ID: Thanks. That cdef-locals concept is consistent with the following example: def f(): i = 1 def g(): print('i' in globals(), 'i' in locals()) def h(): print('i' in globals(), 'i' in locals()); i g() h() f() False False False True It is a mystery, which may be why the documentation for globals() and locals() is 2-line long. Le mer. 20 juill. 2022, ? 19 h 31, Martin Di Paola < martinp.dipaola at gmail.com> a ?crit : > I did a few tests > > # test 1 > def f(): > i = 1 > print(locals()) > exec('y = i; print(y); print(locals())') > print(locals()) > a = eval('y') > print(locals()) > u = a > print(u) > f() > > {'i': 1} > 1 > {'i': 1, 'y': 1} > {'i': 1, 'y': 1} > {'i': 1, 'y': 1, 'a': 1} > 1 > > # test 2 > def f(): > i = 1 > print(locals()) > exec('y = i; print(y); print(locals())') > print(locals()) > a = eval('y') > print(locals()) > y = a > print(y) > f() > {'i': 1} > 1 > {'i': 1, 'y': 1} > {'i': 1} > Traceback (most recent call last): > NameError: name 'y' is not defined > > > So test 1 and 2 are the same except that the variable 'y' is not > present/present in the f's code. > > When it is not present, exec() modifies the f's locals and adds an 'y' > to it but when the variable 'y' is present in the code (even if not > present in the locals()), exec() does not add any 'y' (and the next > eval() then fails) > > The interesting part is that if the 'y' variable is in the f's code > *and* it is defined in the f's locals, no error occur but once again the > exec() does not modify f's locals: > > # test 3 > def f(): > i = 1 > y = 42 > print(locals()) > exec('y = i; print(y); print(locals())') > print(locals()) > a = eval('y') > print(locals()) > y = a > print(y) > f() > {'i': 1, 'y': 42} > 1 > {'i': 1, 'y': 1} > {'i': 1, 'y': 42} > {'i': 1, 'y': 42, 'a': 42} > 42 > > Why does this happen? No idea. > > I may be related with this: > > # test 4 > def f(): > i = 1 > print(locals()) > exec('y = i; print(y); print(locals())') > print(locals()) > print(y) > f() > Traceback (most recent call last): > NameError: name 'y' is not defined > > Despite exec() adds the 'y' variable to f's locals, the variable is not > accessible/visible from f's code. > > So, a few observations (by no means this is how the vm works): > > 1) each function has a set of variables defined by the code (let's call > this "code-defined locals" or "cdef-locals"). > 2) each function also has a set of "runtime locals" accessible from > locals(). > 3) exec() can add variables to locals() (runtime) set but it cannot add > any to cdef-locals. > 4) locals() may be a superset of cdef-locals (but entries in cdef-locals > which value is still undefined are not shown in locals()) > 5) due rule 4, exec() cannot add a variable to locals() if it is already > present in the in cdef-locals. > 6) when eval() runs, it uses locals() set for lookup > > Perhaps rule 5 is to prevent exec() to modify any arbitrary variable of > the caller... > > Anyways, nice food for our brains. > > On Wed, Jul 20, 2022 at 04:56:02PM +0000, george trojan wrote: > >I wish I could understand the following behaviour: > > > >1. This works as I expect it to work: > > > >def f(): > > i = 1 > > print(locals()) > > exec('y = i; print(y); print(locals())') > > print(locals()) > > exec('y *= 2') > > print('ok:', eval('y')) > >f() > > > >{'i': 1} > >1 > >{'i': 1, 'y': 1} > >{'i': 1, 'y': 1} > >ok: 2 > > > >2. I can access the value of y with eval() too: > > > >def f(): > > i = 1 > > print(locals()) > > exec('y = i; print(y); print(locals())') > > print(locals()) > > u = eval('y') > > print(u) > >f() > > > >{'i': 1} > >1 > >{'i': 1, 'y': 1} > >{'i': 1, 'y': 1} > >1 > > > >3. When I change variable name u -> y, somehow locals() in the body of > >the function loses an entry: > > > >def f(): > > i = 1 > > print(locals()) > > exec('y = i; print(y); print(locals())') > > print(locals()) > > y = eval('y') > > print(y) > >f() > > > >{'i': 1} > >1 > >{'i': 1, 'y': 1} > >{'i': 1} > > > > >---------------------------------------------------------------------------NameError > > Traceback (most recent call last) > >Input In [1], in () 7 print(y) 8 # y > >= eval('y') 9 #print('ok:', eval('y'))---> 10 f() > > > >Input In [1], in f() 4 exec('y = i; print(y); print(locals())') > > 5 print(locals())----> 6 y = eval('y') 7 print(y) > > > >File :1, in > >NameError: name 'y' is not defined1. > > > >Another thing: within the first exec(), the print order seems > >reversed. What is going on? > > > >BTW, I am using python 3.8.13. > > > >George > >-- > >https://mail.python.org/mailman/listinfo/python-list > -- > https://mail.python.org/mailman/listinfo/python-list > From rambiusparkisanius at gmail.com Thu Jul 21 10:52:34 2022 From: rambiusparkisanius at gmail.com (Ivan "Rambius" Ivanov) Date: Thu, 21 Jul 2022 10:52:34 -0400 Subject: Resolving Weekday Schedules to Dates Message-ID: Hello, Do you know of a library that resolves schedules like every Wednesday at 3:00pm to absolute time, that is return the datetime of the next occurrence? Regards rambius P.S. -- Tangra Mega Rock: http://www.radiotangra.com From skip.montanaro at gmail.com Thu Jul 21 15:39:01 2022 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Thu, 21 Jul 2022 14:39:01 -0500 Subject: Resolving Weekday Schedules to Dates In-Reply-To: References: Message-ID: > Do you know of a library that resolves schedules like every Wednesday > at 3:00pm to absolute time, that is return the datetime of the next > occurrence? Take a look at the `rrule` module in the `dateutil` package: https://dateutil.readthedocs.io/en/stable/rrule.html Skip From Marco.Sulla.Python at gmail.com Thu Jul 21 15:47:13 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Thu, 21 Jul 2022 21:47:13 +0200 Subject: Why I fail so bad to check for memory leak with this code? Message-ID: I tried to check for memory leaks in a bunch of functions of mine using a simple decorator. It works, but it fails with this code, returning a random count_diff at every run. Why? import tracemalloc import gc import functools from uuid import uuid4 import pickle def getUuid(): return str(uuid4()) def trace(func): @functools.wraps(func) def inner(): tracemalloc.start() snapshot1 = tracemalloc.take_snapshot().filter_traces( (tracemalloc.Filter(True, __file__), ) ) for i in range(100): func() gc.collect() snapshot2 = tracemalloc.take_snapshot().filter_traces( (tracemalloc.Filter(True, __file__), ) ) top_stats = snapshot2.compare_to(snapshot1, 'lineno') tracemalloc.stop() for stat in top_stats: if stat.count_diff > 3: raise ValueError(f"count_diff: {stat.count_diff}") return inner dict_1 = {getUuid(): i for i in range(1000)} @trace def func_76(): pickle.dumps(iter(dict_1)) func_76() From jkn_gg at nicorp.f9.co.uk Thu Jul 21 15:38:02 2022 From: jkn_gg at nicorp.f9.co.uk (jkn) Date: Thu, 21 Jul 2022 12:38:02 -0700 (PDT) Subject: Resolving Weekday Schedules to Dates In-Reply-To: References: Message-ID: On Thursday, July 21, 2022 at 8:19:34 PM UTC+1, rambius wrote: > Hello, > > Do you know of a library that resolves schedules like every Wednesday > at 3:00pm to absolute time, that is return the datetime of the next > occurrence? > > Regards > rambius > > P.S. > > -- > Tangra Mega Rock: http://www.radiotangra.com Not exactly, and it's not a library as such; but Andy Valencia's 'mypim' http://sources.vsta.org:7100/mypim/index includes a calendar facility, so you can add repeating dates and query 'what's coming up in the next 30 days' etc. I haven't looked at this very closely, but it looks like a version of this might scratch an itch of my own. you might also look at 'rrule': https://dateutil.readthedocs.io/en/stable/rrule.html and also look at iCalendar support in general. I would be interested to learn if you come across anything to fit your needs. Regards, J^n From python at mrabarnett.plus.com Thu Jul 21 16:23:25 2022 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 21 Jul 2022 21:23:25 +0100 Subject: Why I fail so bad to check for memory leak with this code? In-Reply-To: References: Message-ID: <915eb50e-bd1f-287b-95a6-b18c7f6c2d5c@mrabarnett.plus.com> On 21/07/2022 20:47, Marco Sulla wrote: > I tried to check for memory leaks in a bunch of functions of mine using a > simple decorator. It works, but it fails with this code, returning a random > count_diff at every run. Why? > > import tracemalloc > import gc > import functools > from uuid import uuid4 > import pickle > > def getUuid(): > return str(uuid4()) > > def trace(func): > @functools.wraps(func) > def inner(): > tracemalloc.start() > > snapshot1 = tracemalloc.take_snapshot().filter_traces( > (tracemalloc.Filter(True, __file__), ) > ) > > for i in range(100): > func() > > gc.collect() > > snapshot2 = tracemalloc.take_snapshot().filter_traces( > (tracemalloc.Filter(True, __file__), ) > ) > > top_stats = snapshot2.compare_to(snapshot1, 'lineno') > tracemalloc.stop() > > for stat in top_stats: > if stat.count_diff > 3: > raise ValueError(f"count_diff: {stat.count_diff}") > > return inner > > dict_1 = {getUuid(): i for i in range(1000)} > > @trace > def func_76(): > pickle.dumps(iter(dict_1)) > > func_76() It's something to do with pickling iterators because it still occurs when I reduce func_76 to: @trace def func_76(): pickle.dumps(iter([])) From Marco.Sulla.Python at gmail.com Thu Jul 21 16:51:17 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Thu, 21 Jul 2022 22:51:17 +0200 Subject: Why I fail so bad to check for memory leak with this code? In-Reply-To: <915eb50e-bd1f-287b-95a6-b18c7f6c2d5c@mrabarnett.plus.com> References: <915eb50e-bd1f-287b-95a6-b18c7f6c2d5c@mrabarnett.plus.com> Message-ID: On Thu, 21 Jul 2022 at 22:28, MRAB wrote: > > It's something to do with pickling iterators because it still occurs > when I reduce func_76 to: > > @trace > def func_76(): > pickle.dumps(iter([])) It's too strange. I found a bunch of true memory leaks with this decorator. It seems to be reliable. It's correct with pickle and with iter, but not when pickling iters. From Marco.Sulla.Python at gmail.com Thu Jul 21 16:58:57 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Thu, 21 Jul 2022 22:58:57 +0200 Subject: Why I fail so bad to check for memory leak with this code? In-Reply-To: References: <915eb50e-bd1f-287b-95a6-b18c7f6c2d5c@mrabarnett.plus.com> Message-ID: This naif code shows no leak: import resource import pickle c = 0 while True: pickle.dumps(iter([])) if (c % 10000) == 0: max_rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss print(f"iteration: {c}, max rss: {max_rss} kb") c += 1 From avi.e.gross at gmail.com Thu Jul 21 18:02:14 2022 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Thu, 21 Jul 2022 18:02:14 -0400 Subject: Resolving Weekday Schedules to Dates In-Reply-To: References: Message-ID: <004c01d89d4d$89a5b290$9cf117b0$@gmail.com> >>>> Do you know of a library that resolves schedules like every Wednesday at 3:00pm to absolute time, that is return the datetime of the next occurrence? It may be as simple, as this to add seven days, assuming a slip of a second is not important or even a hour when things change. Enddate = Begindate + timedelta(days=7) This assumes you used code like this before: from datetime import datetime from datetime import timedelta Begindate = datetime.strptime("2020-10-11 12:34") Of course if you want to include times, you adjust to your needs. And, of course, the above can be implemented as an iterator that keeps producing as long as you need. From Marco.Sulla.Python at gmail.com Thu Jul 21 18:39:26 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Fri, 22 Jul 2022 00:39:26 +0200 Subject: Why I fail so bad to check for memory leak with this code? In-Reply-To: References: <915eb50e-bd1f-287b-95a6-b18c7f6c2d5c@mrabarnett.plus.com> Message-ID: I've done this other simple test: #!/usr/bin/env python3 import tracemalloc import gc import pickle tracemalloc.start() snapshot1 = tracemalloc.take_snapshot().filter_traces( (tracemalloc.Filter(True, __file__), ) ) for i in range(10000000): pickle.dumps(iter([])) gc.collect() snapshot2 = tracemalloc.take_snapshot().filter_traces( (tracemalloc.Filter(True, __file__), ) ) top_stats = snapshot2.compare_to(snapshot1, 'lineno') tracemalloc.stop() for stat in top_stats: print(stat) The result is: /home/marco/sources/test.py:14: size=3339 B (+3339 B), count=63 (+63), average=53 B /home/marco/sources/test.py:9: size=464 B (+464 B), count=1 (+1), average=464 B /home/marco/sources/test.py:10: size=456 B (+456 B), count=1 (+1), average=456 B /home/marco/sources/test.py:13: size=28 B (+28 B), count=1 (+1), average=28 B It seems that, after 10 million loops, only 63 have a leak, with only ~3 KB. It seems to me that we can't call it a leak, no? Probably pickle needs a lot more cycles to be sure there's actually a real leakage. From python at mrabarnett.plus.com Thu Jul 21 19:37:21 2022 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 22 Jul 2022 00:37:21 +0100 Subject: Why I fail so bad to check for memory leak with this code? In-Reply-To: References: <915eb50e-bd1f-287b-95a6-b18c7f6c2d5c@mrabarnett.plus.com> Message-ID: <2831e5ac-a119-4e5e-6a42-f4181fb397eb@mrabarnett.plus.com> On 21/07/2022 23:39, Marco Sulla wrote: > I've done this other simple test: > > #!/usr/bin/env python3 > > import tracemalloc > import gc > import pickle > > tracemalloc.start() > > snapshot1 = tracemalloc.take_snapshot().filter_traces( > ? ? (tracemalloc.Filter(True, __file__), ) > ) > > for i in range(10000000): > ? ? pickle.dumps(iter([])) > > gc.collect() > > snapshot2 = tracemalloc.take_snapshot().filter_traces( > ? ? (tracemalloc.Filter(True, __file__), ) > ) > > top_stats = snapshot2.compare_to(snapshot1, 'lineno') > tracemalloc.stop() > > for stat in top_stats: > ? ? print(stat) > > The result is: > > /home/marco/sources/test.py:14: size=3339 B (+3339 B), count=63 (+63), > average=53 B > /home/marco/sources/test.py:9: size=464 B (+464 B), count=1 (+1), > average=464 B > /home/marco/sources/test.py:10: size=456 B (+456 B), count=1 (+1), > average=456 B > /home/marco/sources/test.py:13: size=28 B (+28 B), count=1 (+1), > average=28 B > > It seems that, after 10 million loops, only 63 have a leak, with only > ~3 KB. It seems to me that we can't call it a leak, no? Probably > pickle needs a lot more cycles to be sure there's actually a real leakage. If it was a leak, then the amount of memory used or the counts would increase with increasing iterations. If that's not happening, if the memory used and the counts stay the roughly the same, then it's probably not a leak, unless it's a leak of something that happens only once, such as creating a cache or buffer on first use. From barry at barrys-emacs.org Fri Jul 22 03:00:01 2022 From: barry at barrys-emacs.org (Barry) Date: Fri, 22 Jul 2022 08:00:01 +0100 Subject: Why I fail so bad to check for memory leak with this code? Message-ID: <782CE27C-8998-4DDA-8C6B-9A00D3CD5BD8@barrys-emacs.org> ? > On 21 Jul 2022, at 21:54, Marco Sulla wrote: > ?On Thu, 21 Jul 2022 at 22:28, MRAB wrote: >> >> It's something to do with pickling iterators because it still occurs >> when I reduce func_76 to: >> >> @trace >> def func_76(): >> pickle.dumps(iter([])) > > It's too strange. I found a bunch of true memory leaks with this > decorator. It seems to be reliable. It's correct with pickle and with > iter, but not when pickling iters. With code as complex as python?s there will be memory allocations that occur that will not be directly related to the python code you test. To put it another way there is noise in your memory allocation signal. Usually the signal of a memory leak is very clear, as you noticed. For rare leaks I would use a tool like valgrind. Barry > -- > https://mail.python.org/mailman/listinfo/python-list From NOTsomeone at microsoft.invalid Fri Jul 22 14:32:52 2022 From: NOTsomeone at microsoft.invalid (GB) Date: Fri, 22 Jul 2022 19:32:52 +0100 Subject: OT: Computer vision Message-ID: I'm looking for some help getting started with a computer vision project. Can anyone here either help or point me in the direction of a better NG/forum, please? From Marco.Sulla.Python at gmail.com Fri Jul 22 15:40:47 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Fri, 22 Jul 2022 21:40:47 +0200 Subject: Why I fail so bad to check for memory leak with this code? In-Reply-To: <782CE27C-8998-4DDA-8C6B-9A00D3CD5BD8@barrys-emacs.org> References: <782CE27C-8998-4DDA-8C6B-9A00D3CD5BD8@barrys-emacs.org> Message-ID: On Fri, 22 Jul 2022 at 09:00, Barry wrote: > With code as complex as python?s there will be memory allocations that occur that will not be directly related to the python code you test. > > To put it another way there is noise in your memory allocation signal. > > Usually the signal of a memory leak is very clear, as you noticed. > > For rare leaks I would use a tool like valgrind. Thank you all, but I needed a simple decorator to automatize the memory leak (and segfault) tests. I think that this version is good enough, I hope that can be useful to someone: def trace(iterations=100): def decorator(func): def wrapper(): print( f"Loops: {iterations} - Evaluating: {func.__name__}", flush=True ) tracemalloc.start() snapshot1 = tracemalloc.take_snapshot().filter_traces( (tracemalloc.Filter(True, __file__), ) ) for i in range(iterations): func() gc.collect() snapshot2 = tracemalloc.take_snapshot().filter_traces( (tracemalloc.Filter(True, __file__), ) ) top_stats = snapshot2.compare_to(snapshot1, 'lineno') tracemalloc.stop() for stat in top_stats: if stat.count_diff * 100 > iterations: raise ValueError(f"stat: {stat}") return wrapper return decorator If the decorated function fails, you can try to raise the iterations parameter. I found that in my cases sometimes I needed a value of 200 or 300 From pjfarley3 at earthlink.net Sat Jul 23 14:01:09 2022 From: pjfarley3 at earthlink.net (pjfarley3 at earthlink.net) Date: Sat, 23 Jul 2022 14:01:09 -0400 Subject: Why do I see digest replies to posts I never saw in a digest? [was: RE: Why I fail so bad to check for memory leak with this code?] In-Reply-To: References: <782CE27C-8998-4DDA-8C6B-9A00D3CD5BD8@barrys-emacs.org> Message-ID: <002201d89ebe$3134d330$939e7990$@earthlink.net> OT to the original subject, but can anyone explain to me why in the forum digest emails I receive I often see a reply to a post that I never saw the original of in any prior digest? Peter > -----Original Message----- > From: Marco Sulla > Sent: Friday, July 22, 2022 3:41 PM > To: Barry > Cc: MRAB ; Python-list at python.org > Subject: Re: Why I fail so bad to check for memory leak with this code? > > On Fri, 22 Jul 2022 at 09:00, Barry wrote: > > With code as complex as python?s there will be memory allocations that > occur that will not be directly related to the python code you test. > > > > To put it another way there is noise in your memory allocation signal. > > > > Usually the signal of a memory leak is very clear, as you noticed. > > > > For rare leaks I would use a tool like valgrind. > > Thank you all, but I needed a simple decorator to automatize the memory leak > (and segfault) tests. I think that this version is good enough, I hope that can be > useful to someone: -- From lsvalgaard at gmail.com Sat Jul 23 13:30:09 2022 From: lsvalgaard at gmail.com (Leif Svalgaard) Date: Sat, 23 Jul 2022 10:30:09 -0700 Subject: spyder v5: invalid file name when selecting interpreter in preferences Message-ID: error message: invalid file path: C:/Users/leifs/anaconda3/python3105.exe what is wrong with that? -- Leif leif at leif.org From ksit70 at gmail.com Sat Jul 23 00:28:11 2022 From: ksit70 at gmail.com (Khairil Sitanggang) Date: Fri, 22 Jul 2022 23:28:11 -0500 Subject: Object in List : how? Message-ID: Hello Expert: I just started using python. Below is a simple code. I was trying to check if, say, NO1 is not in the NODELIST[:].NO How can I achieve this purpose? Regards, -Irfan class Node: def __init__(self): self.NO = 0 self.A = 20 NODE = Node() NODELIST = [] NODE.NO = 10 NODELIST.append(NODE) NODE.NO = 20 NODELIST.append(NODE) NODE.NO = 30 NODELIST.append(NODE) NO1 = 20 if NO1 not in NODELIST[:].NO ??? From nhlanhlah198506 at gmail.com Sat Jul 23 12:16:20 2022 From: nhlanhlah198506 at gmail.com (nhlanhlah198506) Date: Sat, 23 Jul 2022 18:16:20 +0200 Subject: Information about updating my python notebook Message-ID: <62dc1ed8.1c69fb81.acd93.741b@mx.google.com> Can I update my python account?Sent from my Galaxy From hjp-python at hjp.at Sat Jul 23 16:39:24 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sat, 23 Jul 2022 22:39:24 +0200 Subject: Why do I see digest replies to posts I never saw in a digest? In-Reply-To: <002201d89ebe$3134d330$939e7990$@earthlink.net> References: <782CE27C-8998-4DDA-8C6B-9A00D3CD5BD8@barrys-emacs.org> <002201d89ebe$3134d330$939e7990$@earthlink.net> Message-ID: <20220723203924.rjgziscaogafsxfv@hjp.at> On 2022-07-23 14:01:09 -0400, pjfarley3 at earthlink.net wrote: > OT to the original subject, but can anyone explain to me why in the > forum digest emails I receive I often see a reply to a post that I > never saw the original of in any prior digest? I think in most cases this is because both messages were posted to the usenet group but the first one didn't make it through the gateway for some reason (usually because the poster is banned on the mailinglist). In this case I'm not sure which messages you are referring to, since I see the whole chain from the one you quoted to the start of the thread (although the one by Barry has neither an In-Reply-To nor a References header so it isn't sorted in at the correct spot). hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From python at mrabarnett.plus.com Sat Jul 23 17:51:17 2022 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 23 Jul 2022 22:51:17 +0100 Subject: spyder v5: invalid file name when selecting interpreter in preferences In-Reply-To: References: Message-ID: On 23/07/2022 18:30, Leif Svalgaard wrote: > error message: invalid file path: C:/Users/leifs/anaconda3/python3105.exe > what is wrong with that? > Is there a file called python3105.exe in the folder C:/Users/leifs/anaconda3? That filename looks wrong to me because the file is usually called python.exe. This page: https://docs.anaconda.com/anaconda/user-guide/tasks/integration/python-path/ suggests to me that the path is probably more like C:\Users\leifs\anaconda3\python.exe (or, if you prefer, C:/Users/leifs/anaconda3/python.exe). From python at mrabarnett.plus.com Sat Jul 23 17:57:34 2022 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 23 Jul 2022 22:57:34 +0100 Subject: Object in List : how? In-Reply-To: References: Message-ID: <960d8db1-72d7-08b9-9bbb-9674af2f0df4@mrabarnett.plus.com> On 23/07/2022 05:28, Khairil Sitanggang wrote: > Hello Expert: > > I just started using python. Below is a simple code. I was trying to check > if, say, NO1 is not in the NODELIST[:].NO > How can I achieve this purpose? > > Regards, > -Irfan > > > class Node: > def __init__(self): > self.NO = 0 > self.A = 20 > > NODE = Node() > NODELIST = [] > > NODE.NO = 10 > NODELIST.append(NODE) > > NODE.NO = 20 > NODELIST.append(NODE) > > NODE.NO = 30 > NODELIST.append(NODE) > > > NO1 = 20 > if NO1 not in NODELIST[:].NO ??? No, you can't do it that way. You have to iterate through the list and check each member individually: if any(NO1 == N.NO for N in NODELIST): And another thing: you've created only 1 node, and you're changing it each time before adding it to the list, so the list ends up with 3 references to the _same_ object. From PythonList at DancesWithMice.info Sat Jul 23 18:48:04 2022 From: PythonList at DancesWithMice.info (dn) Date: Sun, 24 Jul 2022 10:48:04 +1200 Subject: Information about updating my python notebook In-Reply-To: <62dc1ed8.1c69fb81.acd93.741b@mx.google.com> References: <62dc1ed8.1c69fb81.acd93.741b@mx.google.com> Message-ID: <60833790-094a-f09a-082c-f1e60a89f693@DancesWithMice.info> On 24/07/2022 04.16, nhlanhlah198506 wrote: > Can I update my python account?Sent from my Galaxy How did you install Python, and on which OpSys? In what respect to you want to "update"? What do you mean by "notebook" - and "account"? -- Regards, =dn From ksit70 at gmail.com Sat Jul 23 18:50:13 2022 From: ksit70 at gmail.com (Khairil Sitanggang) Date: Sat, 23 Jul 2022 22:50:13 +0000 Subject: Object in List : how? In-Reply-To: <960d8db1-72d7-08b9-9bbb-9674af2f0df4@mrabarnett.plus.com> References: <960d8db1-72d7-08b9-9bbb-9674af2f0df4@mrabarnett.plus.com> Message-ID: Thank you. I did it as you said. Yes, I forgot to create a new object Node() for each of the 3 instances. The reason I wanted to do as I asked was that I expected Python provides that convenient feature (since it is a high level language). I use Matlab a lot and it is so rich of features that allow us to process object array in many different ways. Anyway, as I said, I just started playing with Python, and I like it so much. Thanks to Visual Studio Code making coding very enjoyable. Get Outlook for iOS ________________________________ From: Python-list on behalf of MRAB Sent: Saturday, July 23, 2022 4:57:34 PM To: python-list at python.org Subject: Re: Object in List : how? On 23/07/2022 05:28, Khairil Sitanggang wrote: > Hello Expert: > > I just started using python. Below is a simple code. I was trying to check > if, say, NO1 is not in the NODELIST[:].NO > How can I achieve this purpose? > > Regards, > -Irfan > > > class Node: > def __init__(self): > self.NO = 0 > self.A = 20 > > NODE = Node() > NODELIST = [] > > NODE.NO = 10 > NODELIST.append(NODE) > > NODE.NO = 20 > NODELIST.append(NODE) > > NODE.NO = 30 > NODELIST.append(NODE) > > > NO1 = 20 > if NO1 not in NODELIST[:].NO ??? No, you can't do it that way. You have to iterate through the list and check each member individually: if any(NO1 == N.NO for N in NODELIST): And another thing: you've created only 1 node, and you're changing it each time before adding it to the list, so the list ends up with 3 references to the _same_ object. -- https://mail.python.org/mailman/listinfo/python-list From PythonList at DancesWithMice.info Sat Jul 23 19:52:45 2022 From: PythonList at DancesWithMice.info (dn) Date: Sun, 24 Jul 2022 11:52:45 +1200 Subject: Object in List : how? In-Reply-To: <960d8db1-72d7-08b9-9bbb-9674af2f0df4@mrabarnett.plus.com> References: <960d8db1-72d7-08b9-9bbb-9674af2f0df4@mrabarnett.plus.com> Message-ID: <9f3b50e0-de32-dc21-4da0-8e351ab61caa@DancesWithMice.info> On 24/07/2022 09.57, MRAB wrote: > On 23/07/2022 05:28, Khairil Sitanggang wrote: >> Hello Expert: >> >> I just started using python. Below is a simple code.? I was trying to >> check >> if, say, NO1 is not in the NODELIST[:].NO >> How can I achieve this purpose? >> >> Regards, >> -Irfan >> >> >> class Node: >> ???? def __init__(self): >> ???????? self.NO = 0 >> ???????? self.A = 20 >> >> NODE = Node() >> NODELIST = [] >> >> NODE.NO = 10 >> NODELIST.append(NODE) >> >> NODE.NO = 20 >> NODELIST.append(NODE) >> >> NODE.NO = 30 >> NODELIST.append(NODE) >> >> >> NO1 = 20 >> if NO1 not in NODELIST[:].NO? ??? > > No, you can't do it that way. You have to iterate through the list and > check each member individually: > > ??? if any(NO1 == N.NO for N in NODELIST): > > And another thing: you've created only 1 node, and you're changing it > each time before adding it to the list, so the list ends up with 3 > references to the _same_ object. +1 Imagine the object (Node) was instead a person, and a bunch of us-people get together in a room. You could shout (above the polite conversation) "is Fred here?" and Fred will reply - or if Fred is elsewhere, silence will reign. That works - but only because everyone knows their own name. Now imagine something like (my) grade school, where the teacher takes a roll/register to note who is present for (or absent from) class. In this case another staff-member could enter the room and instead of shouting (which would be rude), can ask the teacher "is Fred here?". The teacher is able to answer from the roll. The former case is (sort of) the solution proposed above - in looking for Fred, you walk through the room, asking each person in-turn "are you Fred?". The latter is the case for Node and what you were hoping to implement. Thus, an alternate approach is to keep a register of nodes. Note that this is more than a list, because each element of the list (each node) is also identified (on the list) by its name. So, two pieces of (related) data: the id of the node, and the node itself - the name of the person and the person him-/her-self. Assuming you only have one ID that will be used to access a node, Python's built-in dict[ionary] data-structure will provide the advantage of direct-access (instead of going through (on average) half the nodes, asking each one in-turn, are you ... So: > NODE = Node() > NODELIST = [] > > NODE.NO = 10 > NODELIST.append(NODE) becomes: graph = dict{} # could be written as: graph = {} node10 = Node( 10 ) graph[ node.id ] = node10 or even: graph[ 20 ] = Node( 20 ) if you don't need lots of nodes 'hanging around' outside of the 'list' (actually a dict) representing the graph. Then: > NO1 = 20 > if NO1 not in NODELIST[:].NO ??? becomes a single line: the_node_required = graph[ 20 ] where "20" is the search-criteria. Does that make sense? If it does, and when you realise that you'd like to do more with the graph than retrieve single nodes (and more-importantly, should you want (or dare) to delve further into the depths of Object-Oriented Programming) you could declare a second class (graph) containing a number of "methods". If one of the methods implemented is __contains__() then you could indeed ask: if 20 in graph: ie "is Fred in-class today?" but just as we would ask "is Fred here?" rather than "is someone identified by the name 'Fred' here?", it would seem better form to write: if node( 20 ) in graph: ie is there a node with the id of 20, somewhere within the graph? Such might involve sub-classing OrderedDict, UserDict, or even UserList, from the Collections library in the (provided/"batteries-included") Python Standard Library: https://docs.python.org/3/library/collections.html If needed, the extra methods you choose to implement might include such functionality as connecting nodes 10 and 20 with a path/edge, being able to traverse edges, and so-on... Thus, we've covered two categories of class/object: one, a 'data-class' which contains data about a single node; the other a 'collection-class' which contains multiple nodes making-up the graph. A useful distinction and a common related-pair of data-constructs! -- Regards, =dn From ksit70 at gmail.com Sat Jul 23 21:01:18 2022 From: ksit70 at gmail.com (Khairil Sitanggang) Date: Sat, 23 Jul 2022 20:01:18 -0500 Subject: Object in List : how? In-Reply-To: <9f3b50e0-de32-dc21-4da0-8e351ab61caa@DancesWithMice.info> References: <960d8db1-72d7-08b9-9bbb-9674af2f0df4@mrabarnett.plus.com> <9f3b50e0-de32-dc21-4da0-8e351ab61caa@DancesWithMice.info> Message-ID: dn, Thanks for the good analogy and explanation. I need some time to digest it. Regards, -Irfan On Sat, Jul 23, 2022 at 6:55 PM dn wrote: > On 24/07/2022 09.57, MRAB wrote: > > On 23/07/2022 05:28, Khairil Sitanggang wrote: > >> Hello Expert: > >> > >> I just started using python. Below is a simple code. I was trying to > >> check > >> if, say, NO1 is not in the NODELIST[:].NO > >> How can I achieve this purpose? > >> > >> Regards, > >> -Irfan > >> > >> > >> class Node: > >> def __init__(self): > >> self.NO = 0 > >> self.A = 20 > >> > >> NODE = Node() > >> NODELIST = [] > >> > >> NODE.NO = 10 > >> NODELIST.append(NODE) > >> > >> NODE.NO = 20 > >> NODELIST.append(NODE) > >> > >> NODE.NO = 30 > >> NODELIST.append(NODE) > >> > >> > >> NO1 = 20 > >> if NO1 not in NODELIST[:].NO ??? > > > > No, you can't do it that way. You have to iterate through the list and > > check each member individually: > > > > if any(NO1 == N.NO for N in NODELIST): > > > > And another thing: you've created only 1 node, and you're changing it > > each time before adding it to the list, so the list ends up with 3 > > references to the _same_ object. > > +1 > > > Imagine the object (Node) was instead a person, and a bunch of us-people > get together in a room. You could shout (above the polite conversation) > "is Fred here?" and Fred will reply - or if Fred is elsewhere, silence > will reign. That works - but only because everyone knows their own name. > > Now imagine something like (my) grade school, where the teacher takes a > roll/register to note who is present for (or absent from) class. In this > case another staff-member could enter the room and instead of shouting > (which would be rude), can ask the teacher "is Fred here?". The teacher > is able to answer from the roll. > > The former case is (sort of) the solution proposed above - in looking > for Fred, you walk through the room, asking each person in-turn "are you > Fred?". > > The latter is the case for Node and what you were hoping to implement. > > Thus, an alternate approach is to keep a register of nodes. Note that > this is more than a list, because each element of the list (each node) > is also identified (on the list) by its name. So, two pieces of > (related) data: the id of the node, and the node itself - the name of > the person and the person him-/her-self. > > Assuming you only have one ID that will be used to access a node, > Python's built-in dict[ionary] data-structure will provide the advantage > of direct-access (instead of going through (on average) half the nodes, > asking each one in-turn, are you ... > > So: > > > NODE = Node() > > NODELIST = [] > > > > NODE.NO = 10 > > NODELIST.append(NODE) > > becomes: > > graph = dict{} # could be written as: graph = {} > node10 = Node( 10 ) > graph[ node.id ] = node10 > > or even: > > graph[ 20 ] = Node( 20 ) > > if you don't need lots of nodes 'hanging around' outside of the 'list' > (actually a dict) representing the graph. > > > Then: > > > NO1 = 20 > > if NO1 not in NODELIST[:].NO ??? > > becomes a single line: > > the_node_required = graph[ 20 ] > > where "20" is the search-criteria. > > Does that make sense? > > > If it does, and when you realise that you'd like to do more with the > graph than retrieve single nodes (and more-importantly, should you want > (or dare) to delve further into the depths of Object-Oriented > Programming) you could declare a second class (graph) containing a > number of "methods". If one of the methods implemented is __contains__() > then you could indeed ask: > > if 20 in graph: > > ie "is Fred in-class today?" > but just as we would ask "is Fred here?" rather than "is someone > identified by the name 'Fred' here?", it would seem better form to write: > > if node( 20 ) in graph: > > ie is there a node with the id of 20, somewhere within the graph? > > > Such might involve sub-classing OrderedDict, UserDict, or even UserList, > from the Collections library in the (provided/"batteries-included") > Python Standard Library: > https://docs.python.org/3/library/collections.html > > If needed, the extra methods you choose to implement might include such > functionality as connecting nodes 10 and 20 with a path/edge, being able > to traverse edges, and so-on... > > Thus, we've covered two categories of class/object: one, a 'data-class' > which contains data about a single node; the other a 'collection-class' > which contains multiple nodes making-up the graph. A useful distinction > and a common related-pair of data-constructs! > -- > Regards, > =dn > -- > https://mail.python.org/mailman/listinfo/python-list > From __peter__ at web.de Sun Jul 24 03:19:45 2022 From: __peter__ at web.de (Peter Otten) Date: Sun, 24 Jul 2022 09:19:45 +0200 Subject: Object in List : how? In-Reply-To: References: Message-ID: <65239cb1-4baf-bad4-d7da-4d43c77335ad@web.de> On 23/07/2022 06:28, Khairil Sitanggang wrote: > Hello Expert: > > I just started using python. Below is a simple code. I was trying to check > if, say, NO1 is not in the NODELIST[:].NO > How can I achieve this purpose? > > Regards, > -Irfan > > > class Node: > def __init__(self): > self.NO = 0 > self.A = 20 > > NODE = Node() > NODELIST = [] > > NODE.NO = 10 > NODELIST.append(NODE) > > NODE.NO = 20 This just overwrites the attribute; the previous value 10 is lost. > NODELIST.append(NODE) > > NODE.NO = 30 This againoverwrites the attribute; the previous value 20 is lost. > NODELIST.append(NODE) You are three times appending the *same* node to the list. To create a new node you need to invoke the initializer: [I'm following a common convention and use lowercase names in my examples] nodelist = [] # first node node = Node() node.no = 10 nodelist.append(node) # second node node = Node() # this is crucial node.no = 20 nodelist.append(node) ... and so on. However, usually object creation and initialization is combined by allowing arguments to the initializer: class Node: def __init__(self, no, a): self.no = no self.a = a nodelist = [] for no in [10, 20, 30]: nodelist.append(Node(no, 20)) > NO1 = 20 > if NO1 not in NODELIST[:].NO ??? You are checking if the list contains an item with a specific attribute value, so you cannot use the nodelist directly, you need an intermediate list that contains the attribute values: no1 = 20 nos = [node.no for node in nodelist] if no1 not in nos: print("not found") There's one disadvantage to this approach. If the node list is huge another huge list with the attribute values is built even though the first item in the nodelist may already have the searched-for attribute value. To avoid the overhead you could write a function: def contains_no(nodes, no): for node in nodes: if node.no == no: return True return False if not contains_no(nodelist, 20): print("not found") But Python has something more elegant, a kind of /lazy/ /list/ called "generator expression" where each item is calculated on demand. With that you can write if 20 not in (node.no for node in nodelist): print("not found") and your script will stop inspecting further nodes as soon as a matching node is found. From ksit70 at gmail.com Sun Jul 24 20:21:21 2022 From: ksit70 at gmail.com (Khairil Sitanggang) Date: Sun, 24 Jul 2022 19:21:21 -0500 Subject: Object in List : how? In-Reply-To: <65239cb1-4baf-bad4-d7da-4d43c77335ad@web.de> References: <65239cb1-4baf-bad4-d7da-4d43c77335ad@web.de> Message-ID: Thank you. On Sun, Jul 24, 2022 at 2:23 AM Peter Otten <__peter__ at web.de> wrote: > On 23/07/2022 06:28, Khairil Sitanggang wrote: > > Hello Expert: > > > > I just started using python. Below is a simple code. I was trying to > check > > if, say, NO1 is not in the NODELIST[:].NO > > How can I achieve this purpose? > > > > Regards, > > -Irfan > > > > > > class Node: > > def __init__(self): > > self.NO = 0 > > self.A = 20 > > > > NODE = Node() > > NODELIST = [] > > > > NODE.NO = 10 > > NODELIST.append(NODE) > > > > NODE.NO = 20 > > This just overwrites the attribute; the previous value 10 is lost. > > > NODELIST.append(NODE) > > > > NODE.NO = 30 > > This againoverwrites the attribute; the previous value 20 is lost. > > > NODELIST.append(NODE) > > > You are three times appending the *same* node to the list. > To create a new node you need to invoke the initializer: > > [I'm following a common convention and use lowercase names in my examples] > > nodelist = [] > > # first node > node = Node() > node.no = 10 > nodelist.append(node) > > # second node > node = Node() # this is crucial > node.no = 20 > nodelist.append(node) > > ... and so on. However, usually object creation and initialization is > combined by allowing arguments to the initializer: > > class Node: > def __init__(self, no, a): > self.no = no > self.a = a > > nodelist = [] > for no in [10, 20, 30]: > nodelist.append(Node(no, 20)) > > > NO1 = 20 > > if NO1 not in NODELIST[:].NO ??? > > You are checking if the list contains an item with a specific attribute > value, so you cannot use the nodelist directly, you need an intermediate > list that contains the attribute values: > > no1 = 20 > nos = [node.no for node in nodelist] > if no1 not in nos: > print("not found") > > There's one disadvantage to this approach. If the node list is huge > another huge list with the attribute values is built even though the > first item in the nodelist may already have the searched-for attribute > value. To avoid the overhead you could write a function: > > def contains_no(nodes, no): > for node in nodes: > if node.no == no: > return True > return False > > if not contains_no(nodelist, 20): > print("not found") > > But Python has something more elegant, a kind of /lazy/ /list/ called > "generator expression" where each item is calculated on demand. With > that you can write > > if 20 not in (node.no for node in nodelist): > print("not found") > > and your script will stop inspecting further nodes as soon as a matching > node is found. > From ksit70 at gmail.com Sun Jul 24 20:28:17 2022 From: ksit70 at gmail.com (Khairil Sitanggang) Date: Sun, 24 Jul 2022 19:28:17 -0500 Subject: Object in List : how? In-Reply-To: <65239cb1-4baf-bad4-d7da-4d43c77335ad@web.de> References: <65239cb1-4baf-bad4-d7da-4d43c77335ad@web.de> Message-ID: Peter: Thanks for the explanation. It is clear and easy to understand for a beginner like me. I highly appreciate it. Regards, -Irfan On Sun, Jul 24, 2022 at 2:23 AM Peter Otten <__peter__ at web.de> wrote: > On 23/07/2022 06:28, Khairil Sitanggang wrote: > > Hello Expert: > > > > I just started using python. Below is a simple code. I was trying to > check > > if, say, NO1 is not in the NODELIST[:].NO > > How can I achieve this purpose? > > > > Regards, > > -Irfan > > > > > > class Node: > > def __init__(self): > > self.NO = 0 > > self.A = 20 > > > > NODE = Node() > > NODELIST = [] > > > > NODE.NO = 10 > > NODELIST.append(NODE) > > > > NODE.NO = 20 > > This just overwrites the attribute; the previous value 10 is lost. > > > NODELIST.append(NODE) > > > > NODE.NO = 30 > > This againoverwrites the attribute; the previous value 20 is lost. > > > NODELIST.append(NODE) > > > You are three times appending the *same* node to the list. > To create a new node you need to invoke the initializer: > > [I'm following a common convention and use lowercase names in my examples] > > nodelist = [] > > # first node > node = Node() > node.no = 10 > nodelist.append(node) > > # second node > node = Node() # this is crucial > node.no = 20 > nodelist.append(node) > > ... and so on. However, usually object creation and initialization is > combined by allowing arguments to the initializer: > > class Node: > def __init__(self, no, a): > self.no = no > self.a = a > > nodelist = [] > for no in [10, 20, 30]: > nodelist.append(Node(no, 20)) > > > NO1 = 20 > > if NO1 not in NODELIST[:].NO ??? > > You are checking if the list contains an item with a specific attribute > value, so you cannot use the nodelist directly, you need an intermediate > list that contains the attribute values: > > no1 = 20 > nos = [node.no for node in nodelist] > if no1 not in nos: > print("not found") > > There's one disadvantage to this approach. If the node list is huge > another huge list with the attribute values is built even though the > first item in the nodelist may already have the searched-for attribute > value. To avoid the overhead you could write a function: > > def contains_no(nodes, no): > for node in nodes: > if node.no == no: > return True > return False > > if not contains_no(nodelist, 20): > print("not found") > > But Python has something more elegant, a kind of /lazy/ /list/ called > "generator expression" where each item is calculated on demand. With > that you can write > > if 20 not in (node.no for node in nodelist): > print("not found") > > and your script will stop inspecting further nodes as soon as a matching > node is found. > From ksit70 at gmail.com Sun Jul 24 20:47:38 2022 From: ksit70 at gmail.com (Khairil Sitanggang) Date: Sun, 24 Jul 2022 19:47:38 -0500 Subject: Object in List : how? In-Reply-To: <65239cb1-4baf-bad4-d7da-4d43c77335ad@web.de> References: <65239cb1-4baf-bad4-d7da-4d43c77335ad@web.de> Message-ID: Regarding your comment : " *However, usually object creation and initialization iscombined by allowing arguments to the initializer:*" , so which one of the two classes Node1, Node2 below is more common in practice? Option 2, I guess. Thanks, # option 1: class Node1: def __init__(self, a): self.a = a self.b = self.calculation() def calculation(self): r = self.a + 10 return r # option 2: class Node2: def __init__(self, a, b): self.a = a self.b = b self.b = self.calculation() def calculation(self): r = self.a + 10 return r nd1 = Node1(10) nd2 = Node2(10, 0) # 0 is dummy, will be overwritten by the call to calculation() On Sun, Jul 24, 2022 at 2:23 AM Peter Otten <__peter__ at web.de> wrote: > On 23/07/2022 06:28, Khairil Sitanggang wrote: > > Hello Expert: > > > > I just started using python. Below is a simple code. I was trying to > check > > if, say, NO1 is not in the NODELIST[:].NO > > How can I achieve this purpose? > > > > Regards, > > -Irfan > > > > > > class Node: > > def __init__(self): > > self.NO = 0 > > self.A = 20 > > > > NODE = Node() > > NODELIST = [] > > > > NODE.NO = 10 > > NODELIST.append(NODE) > > > > NODE.NO = 20 > > This just overwrites the attribute; the previous value 10 is lost. > > > NODELIST.append(NODE) > > > > NODE.NO = 30 > > This againoverwrites the attribute; the previous value 20 is lost. > > > NODELIST.append(NODE) > > > You are three times appending the *same* node to the list. > To create a new node you need to invoke the initializer: > > [I'm following a common convention and use lowercase names in my examples] > > nodelist = [] > > # first node > node = Node() > node.no = 10 > nodelist.append(node) > > # second node > node = Node() # this is crucial > node.no = 20 > nodelist.append(node) > > ... and so on. However, usually object creation and initialization is > combined by allowing arguments to the initializer: > > class Node: > def __init__(self, no, a): > self.no = no > self.a = a > > nodelist = [] > for no in [10, 20, 30]: > nodelist.append(Node(no, 20)) > > > NO1 = 20 > > if NO1 not in NODELIST[:].NO ??? > > You are checking if the list contains an item with a specific attribute > value, so you cannot use the nodelist directly, you need an intermediate > list that contains the attribute values: > > no1 = 20 > nos = [node.no for node in nodelist] > if no1 not in nos: > print("not found") > > There's one disadvantage to this approach. If the node list is huge > another huge list with the attribute values is built even though the > first item in the nodelist may already have the searched-for attribute > value. To avoid the overhead you could write a function: > > def contains_no(nodes, no): > for node in nodes: > if node.no == no: > return True > return False > > if not contains_no(nodelist, 20): > print("not found") > > But Python has something more elegant, a kind of /lazy/ /list/ called > "generator expression" where each item is calculated on demand. With > that you can write > > if 20 not in (node.no for node in nodelist): > print("not found") > > and your script will stop inspecting further nodes as soon as a matching > node is found. > From 2QdxY4RzWzUUiLuE at potatochowder.com Mon Jul 25 00:28:30 2022 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Sun, 24 Jul 2022 23:28:30 -0500 Subject: Object in List : how? In-Reply-To: References: <65239cb1-4baf-bad4-d7da-4d43c77335ad@web.de> Message-ID: On 2022-07-24 at 19:47:38 -0500, Khairil Sitanggang wrote: > Regarding [Peter Otten's] comment : " > *However, usually object creation and initialization iscombined by allowing > arguments to the initializer:*" , so which one of the two classes Node1, > Node2 below is more common in practice? Option 2, I guess. No. Please use option 1. Another option would be to expose b as an optional parameter with a default value: class Node: def __init__(self, a, b=0): self.a = a self.b = self.calculation() if b == 0 else b There are other ways to write that particular assignment to b (because the default is 0), but the relevant concept for right now is that callers *can* supply a value for b, but that they don't *have* to: n1 = Node(a) # uses a default value for b n2 = Node(a, 22) # overrides the default; use 22 instead Designing APIs can be tricky, and it's not an exact science. From PythonList at DancesWithMice.info Mon Jul 25 02:01:10 2022 From: PythonList at DancesWithMice.info (dn) Date: Mon, 25 Jul 2022 18:01:10 +1200 Subject: Object in List : how? In-Reply-To: References: <65239cb1-4baf-bad4-d7da-4d43c77335ad@web.de> Message-ID: <0bd5668a-0206-c189-2097-110bb1b07e25@DancesWithMice.info> On 25/07/2022 12.47, Khairil Sitanggang wrote: > Regarding your comment : " > *However, usually object creation and initialization iscombined by allowing > arguments to the initializer:*" , so which one of the two classes Node1, > Node2 below is more common in practice? Option 2, I guess. > Thanks, > > > # option 1: > class Node1: > def __init__(self, a): > self.a = a > self.b = self.calculation() > > def calculation(self): > r = self.a + 10 > return r > > # option 2: > class Node2: > def __init__(self, a, b): > self.a = a > self.b = b > > self.b = self.calculation() > > def calculation(self): > r = self.a + 10 > return r > > nd1 = Node1(10) > nd2 = Node2(10, 0) # 0 is dummy, will be overwritten by the call to > calculation() Let's start with calculation() - even though it is not your specific question: Given that "self" makes it into an instance-method, it will have access to self.b! Accordingly, the intermediate variable "r" and its return serves no purpose - assuming calculation() is only used to produce a value for self.b - which would leave: def calculation( self ): self.b = self.a + 10 At which point, the method becomes pointless - may as well put its single line in-line within __init__() - as I say, with above assumptions. Some languages do expect that every identifier (data-attribute in this case) be declared (as to type) and probably also initialised with a value. Some languages, and some Style Guides require that all data-attributes are declared within the constructor/initialiser. Python requires neither of these. Accordingly, if the "b" argument will only ever be a "dummy", there is absolutely no need for it - indeed one could argue that its presence is confusing because it gives the impression than it could assume any value. (see elsewhere in this thread). So, with assumptions and short of facts, "option 1" seems better (with the additional consideration regarding calculation(), as above). -- Regards, =dn From antoon.pardon at vub.be Mon Jul 25 04:39:46 2022 From: antoon.pardon at vub.be (Antoon Pardon) Date: Mon, 25 Jul 2022 10:39:46 +0200 Subject: My first attempt at a package. In-Reply-To: References: Message-ID: <46f72fdf-8fa0-ddb5-2e02-282cf6e191ea@vub.be> Op 19/07/2022 om 16:57 schreef David Lowry-Duda: > On Tue, Jul 19, 2022 at 03:58:41PM +0200, Antoon Pardon wrote: >> I am writing a python package which has the following structure >> >> PACKAGE >> ?? * module1.py >> ?? * module2.py >> ?? * data.cfg >> >> However the data.cfg should be build at installation time. >> >> Can someone give advise on which packaging tool and how >> to use it, to accomplisch this. > > A lot of good packaging information can be found at > > https://packaging.python.org/en/latest/ > > and in particular at > > https://packaging.python.org/en/latest/tutorials/packaging-projects/ > > and > > https://packaging.python.org/en/latest/overview/ > > There are a couple of different ways to handle python packaging, and > it can be a bit confusing. But following the tutorial on packaging is > a good start. > Yes it is, but it doesn't answer my question: How do I create a package in which a file is built at install time. I just want to build a configuration file that will among some other info contain the date the package was installed. The idea is that you can execute python3 -m .release to know what version of the package you installed and when you installed it. But seem unable to find a way to do this. From __peter__ at web.de Mon Jul 25 04:50:44 2022 From: __peter__ at web.de (Peter Otten) Date: Mon, 25 Jul 2022 10:50:44 +0200 Subject: Object in List : how? In-Reply-To: References: <65239cb1-4baf-bad4-d7da-4d43c77335ad@web.de> Message-ID: <26e733a3-5553-efb0-8673-ab1e54b08fc5@web.de> On 25/07/2022 02:47, Khairil Sitanggang wrote: > Regarding your comment : " > *However, usually object creation and initialization iscombined by allowing > arguments to the initializer:*" , so which one of the two classes Node1, > Node2 below is more common in practice? Option 2, I guess. > Thanks, > > > # option 1: > class Node1: > def __init__(self, a): > self.a = a > self.b = self.calculation() > > def calculation(self): > r = self.a + 10 > return r > > # option 2: > class Node2: > def __init__(self, a, b): > self.a = a > self.b = b > > self.b = self.calculation() > > def calculation(self): > r = self.a + 10 > return r > > nd1 = Node1(10) > nd2 = Node2(10, 0) # 0 is dummy, will be overwritten by the call to > calculation() An attribute that can be calculated from other attributes should never be modified through other means. Some day you may want b to become something else, write, for example, node = Node2(10, "twenty") and because by then you have forgotten about the calculation() call end up with a buggy script. But manually invoking the calculation() method is also bug prone. You have to remember to repeat it every time you change a: node = Node1(10) assert node.b == 20 # OK node.a = 20 assert node.b == 30 # fails, a and b are out of sync. The solution Python has to offer is called "property". Properties in their simplest form are calculated read-only attributes, i. e. when you write print(node.b) under the hood node.a + 10 is calculated. Here's how to change Node1 to turn b into such a property: class Node3a: def __init__(self, a): self.a = a def calculation(self): return self.a + 10 b = property(calculation) node = Node3a(42) print(node.b) # 52 node.a = 1 print(node.b) # 11 Often you are not interested in exposing both the calculation() method and the property. For cases when you only want to access the property Python provides a way to define the property with a "decorator": class Node3b: def __init__(self, a): self.a = a @property def b(self): return self.a + 10 When you compare the two classes you can see that I (1) renamed calculation() to b() and (2) replaced def b(self): ... b = property(b) with @property def b(self): ... thus avoiding the repetitons of the name. Are there any disadvantages to properties? What I presented as an advantage, that the value of the attribute is recalculated every time the attribute is accessed, may sometimes become a disadvantage, e. g. when it takes a very long time to calculate. In most cases that should not be a problem, though. From ksit70 at gmail.com Mon Jul 25 10:34:11 2022 From: ksit70 at gmail.com (Khairil Sitanggang) Date: Mon, 25 Jul 2022 09:34:11 -0500 Subject: Object in List : how? In-Reply-To: <26e733a3-5553-efb0-8673-ab1e54b08fc5@web.de> References: <65239cb1-4baf-bad4-d7da-4d43c77335ad@web.de> <26e733a3-5553-efb0-8673-ab1e54b08fc5@web.de> Message-ID: Thank you everyone. The specific requirements for that class: *(1)* Provide the values of the "input" (via constructors). *I think everyone agrees with the way it is implemented in the example. * *(2)* Provide other products such as *b* that can be of any type (array, object, etc.). It is like an "output" if you will. *I think everyone suggests that it should be designed such that people should not be able (inadvertently) to change the value from outside the class. I agree, as a matter of fact this is my intent as well.* *(3)* About the function calc(), my intent is NOT to expose it outside the class: it is "private". *And I see the way to do that from your suggestions. * Based on all your valuable suggestions, I should be able to accomplish the 3 goals. By the way, I am writing this code for finite element analysis (FEA): number crunching. Even though the final goal is to get the correct results, I still want to write it following the correct python "grammar" and style. Best regards, -Irfan On Mon, Jul 25, 2022 at 3:54 AM Peter Otten <__peter__ at web.de> wrote: > On 25/07/2022 02:47, Khairil Sitanggang wrote: > > Regarding your comment : " > > *However, usually object creation and initialization iscombined by > allowing > > arguments to the initializer:*" , so which one of the two classes Node1, > > Node2 below is more common in practice? Option 2, I guess. > > Thanks, > > > > > > # option 1: > > class Node1: > > def __init__(self, a): > > self.a = a > > self.b = self.calculation() > > > > def calculation(self): > > r = self.a + 10 > > return r > > > > # option 2: > > class Node2: > > def __init__(self, a, b): > > self.a = a > > self.b = b > > > > self.b = self.calculation() > > > > def calculation(self): > > r = self.a + 10 > > return r > > > > nd1 = Node1(10) > > nd2 = Node2(10, 0) # 0 is dummy, will be overwritten by the call to > > calculation() > > An attribute that can be calculated from other attributes should never > be modified through other means. Some day you may want b to become > something else, write, for example, > > node = Node2(10, "twenty") > > and because by then you have forgotten about the calculation() call end > up with a buggy script. But manually invoking the calculation() method > is also bug prone. You have to remember to repeat it every time you > change a: > > node = Node1(10) > assert node.b == 20 # OK > > node.a = 20 > assert node.b == 30 # fails, a and b are out of sync. > > The solution Python has to offer is called "property". Properties in > their simplest form are calculated read-only attributes, i. e. when you > write > > print(node.b) > > under the hood node.a + 10 is calculated. Here's how to change Node1 to > turn b into such a property: > > class Node3a: > def __init__(self, a): > self.a = a > def calculation(self): > return self.a + 10 > b = property(calculation) > > node = Node3a(42) > print(node.b) # 52 > > node.a = 1 > print(node.b) # 11 > > Often you are not interested in exposing both the calculation() method > and the property. For cases when you only want to access the property > Python provides a way to define the property with a "decorator": > > class Node3b: > def __init__(self, a): > self.a = a > @property > def b(self): > return self.a + 10 > > When you compare the two classes you can see that I > > (1) renamed calculation() to b() and > > (2) replaced > > def b(self): ... > b = property(b) > > with > > @property > def b(self): ... > > thus avoiding the repetitons of the name. > > Are there any disadvantages to properties? > > What I presented as an advantage, that the value of the attribute is > recalculated every time the attribute is accessed, may sometimes become > a disadvantage, e. g. when it takes a very long time to calculate. In > most cases that should not be a problem, though. > From gweatherby at uchc.edu Mon Jul 25 10:48:05 2022 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Mon, 25 Jul 2022 14:48:05 +0000 Subject: Object in List : how? In-Reply-To: References: <65239cb1-4baf-bad4-d7da-4d43c77335ad@web.de> <26e733a3-5553-efb0-8673-ab1e54b08fc5@web.de> Message-ID: <79cad1fa-a1a9-4401-837e-8607d3deacfb@Spark> ?Private? properties are more simply / commonly designated by sticking an _ in front of the name. class Node: def __init__(self,a) self._a = a I recommend you read https://docs.python.org/3/tutorial/classes.html. That?s not to say properties don?t have their uses, but making things ?private? isn?t a major one. ? Gerard Weatherby | Application Architect NMRbox | NAN | Department of Molecular Biology and Biophysics UConn Health 263 Farmington Avenue, Farmington, CT 06030-6406 uchc.edu On Jul 25, 2022, 10:36 AM -0400, Khairil Sitanggang , wrote: *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** Thank you everyone. The specific requirements for that class: *(1)* Provide the values of the "input" (via constructors). *I think everyone agrees with the way it is implemented in the example. * *(2)* Provide other products such as *b* that can be of any type (array, object, etc.). It is like an "output" if you will. *I think everyone suggests that it should be designed such that people should not be able (inadvertently) to change the value from outside the class. I agree, as a matter of fact this is my intent as well.* *(3)* About the function calc(), my intent is NOT to expose it outside the class: it is "private". *And I see the way to do that from your suggestions. * Based on all your valuable suggestions, I should be able to accomplish the 3 goals. By the way, I am writing this code for finite element analysis (FEA): number crunching. Even though the final goal is to get the correct results, I still want to write it following the correct python "grammar" and style. Best regards, -Irfan On Mon, Jul 25, 2022 at 3:54 AM Peter Otten <__peter__ at web.de> wrote: On 25/07/2022 02:47, Khairil Sitanggang wrote: Regarding your comment : " *However, usually object creation and initialization iscombined by allowing arguments to the initializer:*" , so which one of the two classes Node1, Node2 below is more common in practice? Option 2, I guess. Thanks, # option 1: class Node1: def __init__(self, a): self.a = a self.b = self.calculation() def calculation(self): r = self.a + 10 return r # option 2: class Node2: def __init__(self, a, b): self.a = a self.b = b self.b = self.calculation() def calculation(self): r = self.a + 10 return r nd1 = Node1(10) nd2 = Node2(10, 0) # 0 is dummy, will be overwritten by the call to calculation() An attribute that can be calculated from other attributes should never be modified through other means. Some day you may want b to become something else, write, for example, node = Node2(10, "twenty") and because by then you have forgotten about the calculation() call end up with a buggy script. But manually invoking the calculation() method is also bug prone. You have to remember to repeat it every time you change a: node = Node1(10) assert node.b == 20 # OK node.a = 20 assert node.b == 30 # fails, a and b are out of sync. The solution Python has to offer is called "property". Properties in their simplest form are calculated read-only attributes, i. e. when you write print(node.b) under the hood node.a + 10 is calculated. Here's how to change Node1 to turn b into such a property: class Node3a: def __init__(self, a): self.a = a def calculation(self): return self.a + 10 b = property(calculation) node = Node3a(42) print(node.b) # 52 node.a = 1 print(node.b) # 11 Often you are not interested in exposing both the calculation() method and the property. For cases when you only want to access the property Python provides a way to define the property with a "decorator": class Node3b: def __init__(self, a): self.a = a @property def b(self): return self.a + 10 When you compare the two classes you can see that I (1) renamed calculation() to b() and (2) replaced def b(self): ... b = property(b) with @property def b(self): ... thus avoiding the repetitons of the name. Are there any disadvantages to properties? What I presented as an advantage, that the value of the attribute is recalculated every time the attribute is accessed, may sometimes become a disadvantage, e. g. when it takes a very long time to calculate. In most cases that should not be a problem, though. -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!gibw_ZRRGmldOXyvOxYalA9ApszdEvnBdHnTgNVTbgvDAks80irsggsw45ZkniBHx0YRieVHhKAyn1gR$ From wlfraed at ix.netcom.com Sat Jul 23 16:55:28 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Sat, 23 Jul 2022 16:55:28 -0400 Subject: Object in List : how? References: Message-ID: On Fri, 22 Jul 2022 23:28:11 -0500, Khairil Sitanggang declaimed the following: >class Node: > def __init__(self): > self.NO = 0 > self.A = 20 > >NODE = Node() >NODELIST = [] > Comment... The convention is that ALL CAPS is used to indicate something that is to be treated as a CONSTANT. Classes get capitalized initial letters. Names of variable data is traditionally all lower case, lower case with _ between "words" (eg: lower_case), or camel case (eg: camelCase). >NODE.NO = 10 >NODELIST.append(NODE) > >NODE.NO = 20 >NODELIST.append(NODE) > >NODE.NO = 30 >NODELIST.append(NODE) > > >NO1 = 20 >if NO1 not in NODELIST[:].NO ??? The [:], in this statement, just says "make a copy of nodelist". The /list/ does not have an attribute named "NO". You have to ask for each element IN the list. One convoluted way (I've not tested it) is: if len([node for node in nodelist if node.no == no1]): print("Found at least one occurence") This is a list comprehension; it loops over each element in nodelist, making a new list if the element attribute matches the criteria. Python treats 0 as "false" and if no element matched, the list created is empty, so len() is 0. Anything else implies a match was found. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From wlfraed at ix.netcom.com Sat Jul 23 16:56:17 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Sat, 23 Jul 2022 16:56:17 -0400 Subject: Information about updating my python notebook References: <62dc1ed8.1c69fb81.acd93.741b@mx.google.com> Message-ID: On Sat, 23 Jul 2022 18:16:20 +0200, nhlanhlah198506 declaimed the following: >Can I update my python account?Sent from my Galaxy WHAT Python account? -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From wlfraed at ix.netcom.com Sat Jul 23 16:59:14 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Sat, 23 Jul 2022 16:59:14 -0400 Subject: spyder v5: invalid file name when selecting interpreter in preferences References: Message-ID: On Sat, 23 Jul 2022 10:30:09 -0700, Leif Svalgaard declaimed the following: >error message: invalid file path: C:/Users/leifs/anaconda3/python3105.exe >what is wrong with that? Please cut and paste the ENTIRE error message (not a screen image, select the TEXT) including context. If that is something being passed to a Windows command shell, the shell does not handle / (that's a command OPTION), only \ for paths. Have you navigated and proved you have such a file? -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From wlfraed at ix.netcom.com Sat Jul 23 20:49:19 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Sat, 23 Jul 2022 20:49:19 -0400 Subject: spyder v5: invalid file name when selecting interpreter in preferences References: Message-ID: On Sat, 23 Jul 2022 22:51:17 +0100, MRAB declaimed the following: >On 23/07/2022 18:30, Leif Svalgaard wrote: >> error message: invalid file path: C:/Users/leifs/anaconda3/python3105.exe >> what is wrong with that? >> >Is there a file called python3105.exe in the folder >C:/Users/leifs/anaconda3? > >That filename looks wrong to me because the file is usually called >python.exe. PS C:\Users\Wulfraed> Get-ChildItem -Path 'C:\Python38\' -Name -Filter "python*.exe" python.exe python3.8.exe python3.exe pythonservice.exe pythonw.exe PS C:\Users\Wulfraed> That's from a stale ActiveState build... Anaconda may drop the "." in the version (I'd expect it is python3.10.5) -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From wlfraed at ix.netcom.com Mon Jul 25 10:43:00 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Mon, 25 Jul 2022 10:43:00 -0400 Subject: My first attempt at a package. References: <46f72fdf-8fa0-ddb5-2e02-282cf6e191ea@vub.be> Message-ID: <6tatdhh6u3qjl83tqbcr16dh9hsj4obmue@4ax.com> On Mon, 25 Jul 2022 10:39:46 +0200, Antoon Pardon declaimed the following: >Yes it is, but it doesn't answer my question: How do I create a package >in which a file is built at install time. >I just want to build a configuration file that will among some other >info contain the date the package >was installed. The idea is that you can execute python3 -m >.release to know what version >of the package you installed and when you installed it. But seem unable >to find a way to do this. Does https://stackoverflow.com/questions/72320778/autostart-installed-package-with-python provide any hints? -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From ksit70 at gmail.com Mon Jul 25 14:29:55 2022 From: ksit70 at gmail.com (Khairil Sitanggang) Date: Mon, 25 Jul 2022 13:29:55 -0500 Subject: Object in List : how? In-Reply-To: References: Message-ID: Again, thank you so much for all your suggestions; it's even easier for me now to proceed with my coding with all the guidance in one place (this email thread). I highly appreciate all of your help. I love this python community :) Regards, -Irfan On Mon, Jul 25, 2022 at 12:38 PM Dennis Lee Bieber wrote: > On 23 Jul 2022 22:55:47 GMT, ram at zedat.fu-berlin.de (Stefan Ram) declaimed > the following: > > >Dennis Lee Bieber writes: > >> if len([node for node in nodelist if node.no == no1]): > >> print("Found at least one occurence") > > > > I'm not sure you need the "len"; an empty list already is falsy. > > > > Just wanted to be explicit for the OP. > > > -- > Wulfraed Dennis Lee Bieber AF6VN > wlfraed at ix.netcom.com > http://wlfraed.microdiversity.freeddns.org/ > -- > https://mail.python.org/mailman/listinfo/python-list > From antoon.pardon at vub.be Tue Jul 26 05:00:47 2022 From: antoon.pardon at vub.be (Antoon Pardon) Date: Tue, 26 Jul 2022 11:00:47 +0200 Subject: My first attempt at a package. In-Reply-To: <6tatdhh6u3qjl83tqbcr16dh9hsj4obmue@4ax.com> References: <46f72fdf-8fa0-ddb5-2e02-282cf6e191ea@vub.be> <6tatdhh6u3qjl83tqbcr16dh9hsj4obmue@4ax.com> Message-ID: <4e854332-ead4-0ec7-c205-eafb66a8b7e1@vub.be> Op 25/07/2022 om 16:43 schreef Dennis Lee Bieber: > On Mon, 25 Jul 2022 10:39:46 +0200, Antoon Pardon > declaimed the following: > >> Yes it is, but it doesn't answer my question: How do I create a package >> in which a file is built at install time. >> I just want to build a configuration file that will among some other >> info contain the date the package >> was installed. The idea is that you can execute python3 -m >> .release to know what version >> of the package you installed and when you installed it. But seem unable >> to find a way to do this. > Does > https://stackoverflow.com/questions/72320778/autostart-installed-package-with-python > provide any hints? Yes it does, thanks. From pablogsal at gmail.com Tue Jul 26 06:27:05 2022 From: pablogsal at gmail.com (Pablo Galindo Salgado) Date: Tue, 26 Jul 2022 11:27:05 +0100 Subject: [RELEASE] The last 3.11 beta release (3.11.0b5) is now available Message-ID: Here we are. The universe. The vastness of spacetime. At the edge. The last frontier. The last beta*(conditions apply) for Python 3.11. We have defied the powerful gods of release blockers and we have won by using the required amount of ruse and subterfuge. https://www.python.org/downloads/release/python-3110b5/ ## :warning: PLEASE HELP US TO TEST THIS RELEASE :warning: Due to the modified release schedule and the stability concerns regarding the past beta releases, please, please, please, please, help us to test Python 3.11 by testing this beta releases. * if you maintain a library or a third-party package. Test the beta releases! * If you have code that you maintain at work/research centre/classroom/whatever. Test the beta releases! * If you are a multi-million corporation that uses Python. Test the beta releases! * If you are a single-person company that uses Python. Test the beta releases! * If you have a bunch of Python scripts. Test the beta releases! * If you use Python for work, research, teaching or literally for anything. Test the beta releases! * If you ... In summary: no matter who you are of what you do. Test the beta releases! Is **very** important for us that we identify all possible things that may break your code **before** the final release is done and we can only do this if you help us by testing the beta releases and then report anything that doesn't work! ## This is a beta preview of Python 3.11 Python 3.11 is still in development. 3.11.0b5 is the last of five planned beta release previews. Beta release previews are intended to give the wider community the opportunity to test new features and bug fixes and to prepare their projects to support the new feature release. We **strongly encourage** maintainers of third-party Python projects to **test with 3.11** during the beta phase and report issues found to [the Python bug tracker](https://github.com/python/cpython/issues) as soon as possible. While the release is planned to be feature complete entering the beta phase, it is possible that features may be modified or, in rare cases, deleted up until the start of the release candidate phase (Monday, 2021-08-02). Our goal is have no ABI changes after beta 5 and as few code changes as possible after 3.11.0rc1, the first release candidate. To achieve that, it will be **extremely important** to get as much exposure for 3.11 as possible during the beta phase. Please keep in mind that this is a preview release and its use is **not** recommended for production environments. # Major new features of the 3.11 series, compared to 3.10 Some of the new major new features and changes in Python 3.11 are: ## General changes * [PEP 657](https://www.python.org/dev/peps/pep-0657/) -- Include Fine-Grained Error Locations in Tracebacks * [PEP 654](https://www.python.org/dev/peps/pep-0654/) -- Exception Groups and except* * [PEP 680](https://www.python.org/dev/peps/pep-0680/)-- tomllib: Support for Parsing TOML in the Standard Library * [PEP 681](https://www.python.org/dev/peps/pep-0681/)-- Data Class Transforms * [bpo-46752](https://github.com/python/cpython/issues/90908)-- Introduce task groups to asyncio * [bpo-433030](https://github.com/python/cpython/issues/34627/) -- Atomic grouping ((?>...)) and possessive quantifiers (`*+, ++, ?+, {m,n}+`) are now supported in regular expressions. * The [Faster Cpython Project](https://github.com/faster-cpython/) is already yielding some exciting results. Python 3.11 is up to 10-60% faster than Python 3.10. On average, we measured a 1.22x speedup on the standard benchmark suite. See [Faster CPython]( https://docs.python.org/3.11/whatsnew/3.11.html#faster-cpython) for details. ## Typing and typing language changes * [PEP 673](https://www.python.org/dev/peps/pep-0673/) -- Self Type * [PEP 646](https://www.python.org/dev/peps/pep-0646/)-- Variadic Generics * [PEP 675](https://www.python.org/dev/peps/pep-0675/)-- Arbitrary Literal String Type * [PEP 655](https://www.python.org/dev/peps/pep-0655/)-- Marking individual TypedDict items as required or potentially-missing (Hey, **fellow core developer,** if a feature you find important is missing from this list, [let Pablo know](mailto:pablogsal at python.org ).) The next pre-release of Python 3.11 will be 3.11.0rc1, currently scheduled for Monday, 2022-08-01. # More resources * [Online Documentation](https://docs.python.org/3.11/) * [PEP 664](https://www.python.org/dev/peps/pep-0664/), 3.11 Release Schedule * Report bugs at [ https://github.com/python/cpython/issues](https://github.com/python/cpython/issues) . * [Help fund Python and its community](/psf/donations/). # And now for something completely different Schwarzschild wormholes, also known as Einstein?Rosen bridges (named after Albert Einstein and Nathan Rosen), are connections between areas of space that can be modelled as vacuum solutions to the Einstein field equations, and that are now understood to be intrinsic parts of the maximally extended version of the Schwarzschild metric describing an eternal black hole with no charge and no rotation. Here, "maximally extended" refers to the idea that spacetime should not have any "edges": it should be possible to continue this path arbitrarily far into the particle's future or past for any possible trajectory of a free-falling particle (following a geodesic in the spacetime). The Einstein?Rosen bridge was discovered by Ludwig Flamm in 1916, a few months after Schwarzschild published his solution, and was rediscovered by Albert Einstein and his colleague Nathan Rosen, who published their result in 1935. However, in 1962, John Archibald Wheeler and Robert W. Fuller published a paper showing that this type of wormhole is unstable if it connects two parts of the same universe and that it will pinch off too quickly for light (or any particle moving slower than light) that falls in from one exterior region to make it to the other exterior region. Although Schwarzschild wormholes are not traversable in both directions, their existence inspired Kip Thorne to imagine traversable wormholes created by holding the "throat" of a Schwarzschild wormhole open with exotic matter (material that has negative mass/energy). # Release hashes The BSD-style checksum hashes for the release artefacts are: SHA256 (python-3.11.0b5-amd64.exe) = 0cf9d582da862f2fe207fd54b81dfca110e8f04f4b05ab8c3228ce1ea060c7af SHA256 (python-3.11.0b5-arm64.exe) = a71efd9d3835d493d8207a30916ce3417af17295c02a9b0783dc740754f6e40b SHA256 (python-3.11.0b5-embed-amd64.zip) = 5584ddbd21f45ce74ce0512eeb1d817d15374b1b7a461d79f973f6dd48ab5d9e SHA256 (python-3.11.0b5-embed-arm64.zip) = 819924f10eb08ea6322b6040a2fb953137866bb1034cd4e8fe6e93c7c0b37e31 SHA256 (python-3.11.0b5-embed-win32.zip) = 18927604bcbe3c226be7864cde0c1f25ad35c6333d9d3125dfff8ca4fc872255 SHA256 (python-3.11.0b5.exe) = 382eb4c6dc1606bd3cf6f4bdeec8e1e7dab444c5aa23b86142d608a480d7c195 SHA256 (python-3.11.0b5-macos11.pkg) = cd8e6d98e79a4adcd376c486405a535b004cf9a58a71487a11bc424acd815012 SHA256 (Python-3.11.0b5.tar.xz) = 3810bd22f7dc34a99c2a2eb4b85264a4df4f05ef59c4e0ccc2ea82ee9c491698 SHA256 (Python-3.11.0b5.tgz) = 3f7d1a4ab0e64425f4ffd92d49de192ad2ee1c62bc52e3877e9f7b254c702e60 The hashes are also attached to this email. # We hope you enjoy the new releases! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. https://www.python.org/psf/ If you have any questions, please reach out to me or another member of the release team :) Your friendly release team, Ned Deily @nad https://discuss.python.org/u/nad Steve Dower @steve.dower https://discuss.python.org/u/steve.dower Pablo Galindo Salgado @pablogsal https://discuss.python.org/u/pablogsal -------------- next part -------------- SHA256 (python-3.11.0b5-amd64.exe) = 0cf9d582da862f2fe207fd54b81dfca110e8f04f4b05ab8c3228ce1ea060c7af SHA256 (python-3.11.0b5-arm64.exe) = a71efd9d3835d493d8207a30916ce3417af17295c02a9b0783dc740754f6e40b SHA256 (python-3.11.0b5-embed-amd64.zip) = 5584ddbd21f45ce74ce0512eeb1d817d15374b1b7a461d79f973f6dd48ab5d9e SHA256 (python-3.11.0b5-embed-arm64.zip) = 819924f10eb08ea6322b6040a2fb953137866bb1034cd4e8fe6e93c7c0b37e31 SHA256 (python-3.11.0b5-embed-win32.zip) = 18927604bcbe3c226be7864cde0c1f25ad35c6333d9d3125dfff8ca4fc872255 SHA256 (python-3.11.0b5.exe) = 382eb4c6dc1606bd3cf6f4bdeec8e1e7dab444c5aa23b86142d608a480d7c195 SHA256 (python-3.11.0b5-macos11.pkg) = cd8e6d98e79a4adcd376c486405a535b004cf9a58a71487a11bc424acd815012 SHA256 (Python-3.11.0b5.tar.xz) = 3810bd22f7dc34a99c2a2eb4b85264a4df4f05ef59c4e0ccc2ea82ee9c491698 SHA256 (Python-3.11.0b5.tgz) = 3f7d1a4ab0e64425f4ffd92d49de192ad2ee1c62bc52e3877e9f7b254c702e60 From c.buhtz at posteo.jp Tue Jul 26 03:14:42 2022 From: c.buhtz at posteo.jp (c.buhtz at posteo.jp) Date: Tue, 26 Jul 2022 07:14:42 +0000 Subject: PEP about recommended project folder layout Message-ID: Hello, I am not sure if I looked into the correct sources. I was looking in "PEP 609 ? Python Packaging Authority (PyPA) Governance" [1] and the "PyPA specifications" [2]. My question in short: Is there an official document (e.g. a PEP) about a recommended layout for project folders. Looking into the wild and past there are a lot of variations of such layouts. I am far away from being a pro but depending on experience in my own projects and what I have learned from others (e.g. in blog-posts/tutorials) I recommend to have the "test" folder and the package folder side by side on the same level in the project folder (the root). my_project |- tests | ? test_*.py |- my_package | ? __init__.py ?-- README.md I sometimes add to it the so called "src"-Layout where the package folder is one level deeper in an extra "src" folder. my_project |- tests | ? test_*.py |- src | ?- my_package | ? __init__.py ?-- README.md I don't want to discuss the pros and cons of all variations. What I need is an official document I can use in discussions with other maintainers. If there is a PEP/document against my current recommendation I am also fine with this. ;) Kind Christian [1] -- [2] -- From Cecil at decebal.nl Tue Jul 26 10:38:38 2022 From: Cecil at decebal.nl (Cecil Westerhof) Date: Tue, 26 Jul 2022 16:38:38 +0200 Subject: random.SystemRandom().randint() inefficient Message-ID: <87pmhs7xch.fsf@munus.decebal.nl> I need to get a random integer. At first I tried it with: from secrets import randbelow index = randbelow(len(to_try)) This works perfectly, but it took some time. So I thought I try: from random import SystemRandom index = SystemRandom().randint(0, len(to_try) - 1) A first indication is that the second version would take about two times as much time as the first. Is there a reason for this, or should this not be happening? -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From gweatherby at uchc.edu Tue Jul 26 13:30:32 2022 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Tue, 26 Jul 2022 17:30:32 +0000 Subject: random.SystemRandom().randint() inefficient In-Reply-To: References: <87pmhs7xch.fsf@munus.decebal.nl> Message-ID: <5c3c316b-2032-4a6a-8390-786d8ce04c71@Spark> Absolutely. The task (?generate a random number?) is ill-defined. Want a fast random number? Use 552015933 (I just retrieved it from random.org). Want a true random number, use random.org API. (https://api.random.org/pricing). Something in between, follow approaches Stefan suggests. ? Gerard Weatherby | Application Architect NMRbox | NAN | Department of Molecular Biology and Biophysics UConn Health 263 Farmington Avenue, Farmington, CT 06030-6406 uchc.edu On Jul 26, 2022, 11:45 AM -0400, Stefan Ram , wrote: *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** Cecil Westerhof writes: I need to get a random integer. There are all kinds of trade-offs. When you need entropy, the function sometimes has to collect data from hardware, which takes some time. Pseudo-random integers sometimes can be calculated faster. As a compromise, you can get some entropy and use it to initialize ("seed") a pseudo-random number generator and then get some pseudo-random numbers until you seed it again. A linear-congruential generator should be fast enough, but in some cases might not be random enough. -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!kjJ7U6kkSW8uOTWJCnHeXn0Acczd619asmIrVMA_NOqvP_JMqqQ1Rpp61sptQucbOuEFGb6ot499_FqaRajiS9s$ From rosuav at gmail.com Tue Jul 26 14:01:20 2022 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 27 Jul 2022 04:01:20 +1000 Subject: random.SystemRandom().randint() inefficient In-Reply-To: <87pmhs7xch.fsf@munus.decebal.nl> References: <87pmhs7xch.fsf@munus.decebal.nl> Message-ID: On Wed, 27 Jul 2022 at 01:06, Cecil Westerhof via Python-list wrote: > > I need to get a random integer. At first I tried it with: > from secrets import randbelow > index = randbelow(len(to_try)) > > This works perfectly, but it took some time. So I thought I try: > from random import SystemRandom > index = SystemRandom().randint(0, len(to_try) - 1) > > A first indication is that the second version would take about two > times as much time as the first. Is there a reason for this, or should > this not be happening? > You're setting up a brand new SystemRandom instance just for a single random number. For a fairer comparison, set up the instance, then generate far more than just a single number, and see how that goes. ChrisA From kkchn.in at gmail.com Tue Jul 26 14:24:46 2022 From: kkchn.in at gmail.com (KK CHN) Date: Tue, 26 Jul 2022 23:54:46 +0530 Subject: Python code: brief Message-ID: List , I have come across a difficulty to understand the code in this file. I am unable to understand exactly what the code snippet is doing here. https://raw.githubusercontent.com/CODARcode/MDTrAnal/master/lib/codar/oas/MDTrSampler.py I am new to this type of scientific computing code snippets and it is coded by someone. Due to a requirement I would like to understand what these lines of code will do exactly. If someone could explain to me what the code snippets do in the code blocks, it will be a great help . Thanks in advance Krish From PythonList at DancesWithMice.info Tue Jul 26 14:51:00 2022 From: PythonList at DancesWithMice.info (dn) Date: Wed, 27 Jul 2022 06:51:00 +1200 Subject: Python code: brief In-Reply-To: References: Message-ID: On 27/07/2022 06.24, KK CHN wrote: > List , > > I have come across a difficulty to understand the code in this file. I am > unable to understand exactly what the code snippet is doing here. > > https://raw.githubusercontent.com/CODARcode/MDTrAnal/master/lib/codar/oas/MDTrSampler.py > > I am new to this type of scientific computing code snippets and it is coded > by someone. Due to a requirement I would like to understand what these > lines of code will do exactly. If someone could explain to me what the > code snippets do in the code blocks, it will be a great help . Which lines of code, specifically? Is it some Python idiom that is problematic, or the code-logic? -- Regards, =dn From Cecil at decebal.nl Tue Jul 26 15:26:11 2022 From: Cecil at decebal.nl (Cecil Westerhof) Date: Tue, 26 Jul 2022 21:26:11 +0200 Subject: random.SystemRandom().randint() inefficient References: <87pmhs7xch.fsf@munus.decebal.nl> Message-ID: <87lesf8ylo.fsf@munus.decebal.nl> Chris Angelico writes: > On Wed, 27 Jul 2022 at 01:06, Cecil Westerhof via Python-list > wrote: >> >> I need to get a random integer. At first I tried it with: >> from secrets import randbelow >> index = randbelow(len(to_try)) >> >> This works perfectly, but it took some time. So I thought I try: >> from random import SystemRandom >> index = SystemRandom().randint(0, len(to_try) - 1) >> >> A first indication is that the second version would take about two >> times as much time as the first. Is there a reason for this, or should >> this not be happening? >> > > You're setting up a brand new SystemRandom instance just for a single > random number. For a fairer comparison, set up the instance, then > generate far more than just a single number, and see how that goes. Thanks. I thought I did something wrong and I did. I will try to implement like you said and look what the result will be. (And share it.) (As I understand it both do more, or less the same and should have comparable performance.) -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From rosuav at gmail.com Tue Jul 26 16:13:24 2022 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 27 Jul 2022 06:13:24 +1000 Subject: random.SystemRandom().randint() inefficient In-Reply-To: <87lesf8ylo.fsf@munus.decebal.nl> References: <87pmhs7xch.fsf@munus.decebal.nl> <87lesf8ylo.fsf@munus.decebal.nl> Message-ID: On Wed, 27 Jul 2022 at 06:06, Cecil Westerhof via Python-list wrote: > > Chris Angelico writes: > > > On Wed, 27 Jul 2022 at 01:06, Cecil Westerhof via Python-list > > wrote: > >> > >> I need to get a random integer. At first I tried it with: > >> from secrets import randbelow > >> index = randbelow(len(to_try)) > >> > >> This works perfectly, but it took some time. So I thought I try: > >> from random import SystemRandom > >> index = SystemRandom().randint(0, len(to_try) - 1) > >> > >> A first indication is that the second version would take about two > >> times as much time as the first. Is there a reason for this, or should > >> this not be happening? > >> > > > > You're setting up a brand new SystemRandom instance just for a single > > random number. For a fairer comparison, set up the instance, then > > generate far more than just a single number, and see how that goes. > > Thanks. I thought I did something wrong and I did. > I will try to implement like you said and look what the result will > be. (And share it.) Thanks! Don't feel bad; performance testing is *hard*, getting meaningful results takes a lot of of fiddling with parameters, and getting interesting AND meaningful results can sometimes seem about impossible. > (As I understand it both do more, or less the same and should have > comparable performance.) In normal production work? Yes (the SystemRandom object doesn't have any significant state - a seeded RNG could have a lot more overhead here). But for performance testing? The work of instantiating the class could be completely irrelevant, or it could be dominating your results. It's hard to say, hence the suggestion to try it without reinstantiating. ChrisA From Cecil at decebal.nl Tue Jul 26 17:47:59 2022 From: Cecil at decebal.nl (Cecil Westerhof) Date: Tue, 26 Jul 2022 23:47:59 +0200 Subject: random.SystemRandom().randint() inefficient References: <87pmhs7xch.fsf@munus.decebal.nl> <87lesf8ylo.fsf@munus.decebal.nl> Message-ID: <87h7338s1c.fsf@munus.decebal.nl> Chris Angelico writes: > On Wed, 27 Jul 2022 at 06:06, Cecil Westerhof via Python-list > wrote: >> >> Chris Angelico writes: >> >> > On Wed, 27 Jul 2022 at 01:06, Cecil Westerhof via Python-list >> > wrote: >> >> >> >> I need to get a random integer. At first I tried it with: >> >> from secrets import randbelow >> >> index = randbelow(len(to_try)) >> >> >> >> This works perfectly, but it took some time. So I thought I try: >> >> from random import SystemRandom >> >> index = SystemRandom().randint(0, len(to_try) - 1) >> >> >> >> A first indication is that the second version would take about two >> >> times as much time as the first. Is there a reason for this, or should >> >> this not be happening? >> >> >> > >> > You're setting up a brand new SystemRandom instance just for a single >> > random number. For a fairer comparison, set up the instance, then >> > generate far more than just a single number, and see how that goes. >> >> Thanks. I thought I did something wrong and I did. >> I will try to implement like you said and look what the result will >> be. (And share it.) > > Thanks! Don't feel bad; performance testing is *hard*, getting > meaningful results takes a lot of of fiddling with parameters, and > getting interesting AND meaningful results can sometimes seem about > impossible. > >> (As I understand it both do more, or less the same and should have >> comparable performance.) > > In normal production work? Yes (the SystemRandom object doesn't have > any significant state - a seeded RNG could have a lot more overhead > here). But for performance testing? The work of instantiating the > class could be completely irrelevant, or it could be dominating your > results. It's hard to say, hence the suggestion to try it without > reinstantiating. It had a very big influence. Original it took about three times more time to run my program. (The program was still running when I posted the original post and the difference was higher as I anticipated.) Removing that did cut about 45% of the execution time of the program. (So the initiation is quit expensive.) But it still takes about 50% more time. So I am still a bit flabbergasted. The new code: from random import SystemRandom system_random = SystemRandom() index = system_random.randint(0, len(to_try) - 1) The first two statements are executed once. The last statement I think about 75 * 10 ** 6. So it seems that my first idea of using randbelow was the correct one. But if anyone could explain why SystemRandom is so much more expensive, I would be interested to know it. (Or am I still doing something wrong?) -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From rosuav at gmail.com Tue Jul 26 18:44:01 2022 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 27 Jul 2022 08:44:01 +1000 Subject: random.SystemRandom().randint() inefficient In-Reply-To: <87h7338s1c.fsf@munus.decebal.nl> References: <87pmhs7xch.fsf@munus.decebal.nl> <87lesf8ylo.fsf@munus.decebal.nl> <87h7338s1c.fsf@munus.decebal.nl> Message-ID: On Wed, 27 Jul 2022 at 08:18, Cecil Westerhof via Python-list wrote: > > Chris Angelico writes: > > > On Wed, 27 Jul 2022 at 06:06, Cecil Westerhof via Python-list > > wrote: > >> > >> Chris Angelico writes: > >> > >> > On Wed, 27 Jul 2022 at 01:06, Cecil Westerhof via Python-list > >> > wrote: > >> >> > >> >> I need to get a random integer. At first I tried it with: > >> >> from secrets import randbelow > >> >> index = randbelow(len(to_try)) > >> >> > >> >> This works perfectly, but it took some time. So I thought I try: > >> >> from random import SystemRandom > >> >> index = SystemRandom().randint(0, len(to_try) - 1) > >> >> > >> >> A first indication is that the second version would take about two > >> >> times as much time as the first. Is there a reason for this, or should > >> >> this not be happening? > >> >> > >> > > >> > You're setting up a brand new SystemRandom instance just for a single > >> > random number. For a fairer comparison, set up the instance, then > >> > generate far more than just a single number, and see how that goes. > >> > >> Thanks. I thought I did something wrong and I did. > >> I will try to implement like you said and look what the result will > >> be. (And share it.) > > > > Thanks! Don't feel bad; performance testing is *hard*, getting > > meaningful results takes a lot of of fiddling with parameters, and > > getting interesting AND meaningful results can sometimes seem about > > impossible. > > > >> (As I understand it both do more, or less the same and should have > >> comparable performance.) > > > > In normal production work? Yes (the SystemRandom object doesn't have > > any significant state - a seeded RNG could have a lot more overhead > > here). But for performance testing? The work of instantiating the > > class could be completely irrelevant, or it could be dominating your > > results. It's hard to say, hence the suggestion to try it without > > reinstantiating. > > It had a very big influence. Original it took about three times more > time to run my program. (The program was still running when I posted > the original post and the difference was higher as I anticipated.) > Removing that did cut about 45% of the execution time of the program. > (So the initiation is quit expensive.) > But it still takes about 50% more time. So I am still a bit > flabbergasted. > > The new code: > from random import SystemRandom > system_random = SystemRandom() > index = system_random.randint(0, len(to_try) - 1) > > The first two statements are executed once. > The last statement I think about 75 * 10 ** 6. > > So it seems that my first idea of using randbelow was the correct one. > But if anyone could explain why SystemRandom is so much more > expensive, I would be interested to know it. > (Or am I still doing something wrong?) Hmm. There are still a lot of differences here. Are you able to make use of randrange() instead, to make them more consistent? According to the source code, secrets.randbelow is calling on an internal method _randbelow of the SystemRandom object, but randrange (if called with only one arg) will go straight into that same method. Here's my results: rosuav at sikorsky:~$ python3 -m timeit -s 'from random import randrange' 'randrange(10000)' 1000000 loops, best of 5: 322 nsec per loop rosuav at sikorsky:~$ python3 -m timeit -s 'from random import SystemRandom; r = SystemRandom()' 'r.randint(0, 10000)' 200000 loops, best of 5: 1.92 usec per loop rosuav at sikorsky:~$ python3 -m timeit -s 'from random import SystemRandom; r = SystemRandom()' 'r.randrange(10000)' 200000 loops, best of 5: 1.87 usec per loop rosuav at sikorsky:~$ python3 -m timeit -s 'from secrets import randbelow' 'randbelow(10000)' 200000 loops, best of 5: 1.64 usec per loop (The difference with the first one is that it isn't using the system RNG, so it has the limitations of an internal PRNG.) When you call randint, what happens is (1) the endpoint is incremented to transform it from inclusive-inclusive to inclusive-exclusive; (2) randrange is called with two args; (3) fast path 1 is skipped, fast path 2 is taken, and _randbelow gets called to get an actual random number, which gets zero added to it before returning. If, instead, you use randrange(len(to_try)), what would happen is (1) fast path 1 is used, and (2) _randbelow is called to get the random number. In secrets.randbelow, it's even better: (1) _randbelow is called to get the random number. I think that might be what's going on here. You're adding some tiny amounts of extra work, but if you were to make them more similar, the distinctions would evaporate. Here's a couple of other variants that use SystemRandom: rosuav at sikorsky:~$ python3 -m timeit -s 'from random import SystemRandom; randrange = SystemRandom().randrange' 'randrange(10000)' 200000 loops, best of 5: 1.81 usec per loop rosuav at sikorsky:~$ python3 -m timeit -s 'from random import SystemRandom; randrange = SystemRandom()._randbelow' 'randrange(10000)' 200000 loops, best of 5: 1.61 usec per loop Note that we shave a few usec by avoiding the attribute lookup, but even more by directly calling _randbelow, which is why secrets.randbelow is able to save a bit. But don't use internal methods; you can probably get pretty much equivalent performance from randrange, which is public. Hope that's of some value! ChrisA From wlfraed at ix.netcom.com Tue Jul 26 18:58:37 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Tue, 26 Jul 2022 18:58:37 -0400 Subject: random.SystemRandom().randint() inefficient References: <87pmhs7xch.fsf@munus.decebal.nl> Message-ID: On Tue, 26 Jul 2022 16:38:38 +0200, Cecil Westerhof declaimed the following: >I need to get a random integer. At first I tried it with: > from secrets import randbelow > index = randbelow(len(to_try)) > >This works perfectly, but it took some time. So I thought I try: > from random import SystemRandom > index = SystemRandom().randint(0, len(to_try) - 1) > >A first indication is that the second version would take about two >times as much time as the first. Is there a reason for this, or should >this not be happening? Well, off the top of my head... For one generation of "index" you are first creating an instance of SystemRandom(), using it to generate your random integer, and then disposing of the instance. If you only need ONE random integer, the time difference probably doesn't matter. OTOH, if you need many during the run, using sr = SystemRandom() #stuff in some loop that generates multiple ints index = sr.randint(...) Hmmm, wonder if there is a speed difference between .randint(0, len(to_try) - 1) and .randint(1, len(to_try)) - 1 -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From wlfraed at ix.netcom.com Tue Jul 26 19:02:21 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Tue, 26 Jul 2022 19:02:21 -0400 Subject: random.SystemRandom().randint() inefficient References: <87pmhs7xch.fsf@munus.decebal.nl> <87lesf8ylo.fsf@munus.decebal.nl> <87h7338s1c.fsf@munus.decebal.nl> Message-ID: On Tue, 26 Jul 2022 23:47:59 +0200, Cecil Westerhof declaimed the following: >The new code: > from random import SystemRandom > system_random = SystemRandom() > index = system_random.randint(0, len(to_try) - 1) > >The first two statements are executed once. >The last statement I think about 75 * 10 ** 6. > >So it seems that my first idea of using randbelow was the correct one. >But if anyone could explain why SystemRandom is so much more >expensive, I would be interested to know it. >(Or am I still doing something wrong?) What happens with system_randint = SystemRandom().randint #no parens index = system_randint(...) which may remove the method lookup from the repetition. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From rosuav at gmail.com Tue Jul 26 19:43:20 2022 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 27 Jul 2022 09:43:20 +1000 Subject: random.SystemRandom().randint() inefficient In-Reply-To: References: <87pmhs7xch.fsf@munus.decebal.nl> Message-ID: On Wed, 27 Jul 2022 at 09:28, Dennis Lee Bieber wrote: > > On Tue, 26 Jul 2022 16:38:38 +0200, Cecil Westerhof > declaimed the following: > > >I need to get a random integer. At first I tried it with: > > from secrets import randbelow > > index = randbelow(len(to_try)) > > > >This works perfectly, but it took some time. So I thought I try: > > from random import SystemRandom > > index = SystemRandom().randint(0, len(to_try) - 1) > > > >A first indication is that the second version would take about two > >times as much time as the first. Is there a reason for this, or should > >this not be happening? > > Well, off the top of my head... > > For one generation of "index" you are first creating an instance of > SystemRandom(), using it to generate your random integer, and then > disposing of the instance. > > If you only need ONE random integer, the time difference probably > doesn't matter. OTOH, if you need many during the run, using > > sr = SystemRandom() > #stuff in some loop that generates multiple ints > index = sr.randint(...) > > Hmmm, wonder if there is a speed difference between > .randint(0, len(to_try) - 1) > and > .randint(1, len(to_try)) - 1 > Probably not significant, since the same amount of arithmetic gets done either way. But switching to single-arg randrange(len(to_try)) will definitely help, and IMO is clearer as well (since the implication is selecting one from a group of items). Incidentally - if you are actually trying to select a specific item, you may want to consider random.choice. ChrisA From barry at barrys-emacs.org Wed Jul 27 02:38:34 2022 From: barry at barrys-emacs.org (Barry) Date: Wed, 27 Jul 2022 07:38:34 +0100 Subject: random.SystemRandom().randint() inefficient In-Reply-To: <87pmhs7xch.fsf@munus.decebal.nl> References: <87pmhs7xch.fsf@munus.decebal.nl> Message-ID: <296D2D4B-F400-429A-8559-1D9AF9658D81@barrys-emacs.org> > On 26 Jul 2022, at 16:07, Cecil Westerhof via Python-list wrote: > > ?I need to get a random integer. At first I tried it with: > from secrets import randbelow > index = randbelow(len(to_try)) > > This works perfectly, but it took some time. So I thought I try: > from random import SystemRandom > index = SystemRandom().randint(0, len(to_try) - 1) > > A first indication is that the second version would take about two > times as much time as the first. Is there a reason for this, or should > this not be happening? What is the OS that you are running on and its version? If it?s linux what is the kernel version? What version of python and where from? Barry > > -- > Cecil Westerhof > Senior Software Engineer > LinkedIn: http://www.linkedin.com/in/cecilwesterhof > -- > https://mail.python.org/mailman/listinfo/python-list > From Cecil at decebal.nl Wed Jul 27 04:45:47 2022 From: Cecil at decebal.nl (Cecil Westerhof) Date: Wed, 27 Jul 2022 10:45:47 +0200 Subject: random.SystemRandom().randint() inefficient References: <87pmhs7xch.fsf@munus.decebal.nl> <296D2D4B-F400-429A-8559-1D9AF9658D81@barrys-emacs.org> Message-ID: <87czdr7xl0.fsf@munus.decebal.nl> Barry writes: >> On 26 Jul 2022, at 16:07, Cecil Westerhof via Python-list wrote: >> >> ?I need to get a random integer. At first I tried it with: >> from secrets import randbelow >> index = randbelow(len(to_try)) >> >> This works perfectly, but it took some time. So I thought I try: >> from random import SystemRandom >> index = SystemRandom().randint(0, len(to_try) - 1) >> >> A first indication is that the second version would take about two >> times as much time as the first. Is there a reason for this, or should >> this not be happening? > > What is the OS that you are running on and its version? > If it?s linux what is the kernel version? > What version of python and where from? That is always good information of-course. Debian 11.3 5.10.0-13-amd64 3.9.2 What do you mean with where the python version is from? -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From Cecil at decebal.nl Wed Jul 27 05:13:08 2022 From: Cecil at decebal.nl (Cecil Westerhof) Date: Wed, 27 Jul 2022 11:13:08 +0200 Subject: random.SystemRandom().randint() inefficient References: <87pmhs7xch.fsf@munus.decebal.nl> <87lesf8ylo.fsf@munus.decebal.nl> <87h7338s1c.fsf@munus.decebal.nl> Message-ID: <878roe9avv.fsf@munus.decebal.nl> Chris Angelico writes: > On Wed, 27 Jul 2022 at 08:18, Cecil Westerhof via Python-list > wrote: >> >> Chris Angelico writes: >> >> > On Wed, 27 Jul 2022 at 06:06, Cecil Westerhof via Python-list >> > wrote: >> >> >> >> Chris Angelico writes: >> >> >> >> > On Wed, 27 Jul 2022 at 01:06, Cecil Westerhof via Python-list >> >> > wrote: >> >> >> >> >> >> I need to get a random integer. At first I tried it with: >> >> >> from secrets import randbelow >> >> >> index = randbelow(len(to_try)) >> >> >> >> >> >> This works perfectly, but it took some time. So I thought I try: >> >> >> from random import SystemRandom >> >> >> index = SystemRandom().randint(0, len(to_try) - 1) >> >> >> >> >> >> A first indication is that the second version would take about two >> >> >> times as much time as the first. Is there a reason for this, or should >> >> >> this not be happening? >> >> >> >> >> > >> >> > You're setting up a brand new SystemRandom instance just for a single >> >> > random number. For a fairer comparison, set up the instance, then >> >> > generate far more than just a single number, and see how that goes. >> >> >> >> Thanks. I thought I did something wrong and I did. >> >> I will try to implement like you said and look what the result will >> >> be. (And share it.) >> > >> > Thanks! Don't feel bad; performance testing is *hard*, getting >> > meaningful results takes a lot of of fiddling with parameters, and >> > getting interesting AND meaningful results can sometimes seem about >> > impossible. >> > >> >> (As I understand it both do more, or less the same and should have >> >> comparable performance.) >> > >> > In normal production work? Yes (the SystemRandom object doesn't have >> > any significant state - a seeded RNG could have a lot more overhead >> > here). But for performance testing? The work of instantiating the >> > class could be completely irrelevant, or it could be dominating your >> > results. It's hard to say, hence the suggestion to try it without >> > reinstantiating. >> >> It had a very big influence. Original it took about three times more >> time to run my program. (The program was still running when I posted >> the original post and the difference was higher as I anticipated.) >> Removing that did cut about 45% of the execution time of the program. >> (So the initiation is quit expensive.) >> But it still takes about 50% more time. So I am still a bit >> flabbergasted. >> >> The new code: >> from random import SystemRandom >> system_random = SystemRandom() >> index = system_random.randint(0, len(to_try) - 1) >> >> The first two statements are executed once. >> The last statement I think about 75 * 10 ** 6. >> >> So it seems that my first idea of using randbelow was the correct one. >> But if anyone could explain why SystemRandom is so much more >> expensive, I would be interested to know it. >> (Or am I still doing something wrong?) > > Hmm. There are still a lot of differences here. Are you able to make > use of randrange() instead, to make them more consistent? > > According to the source code, secrets.randbelow is calling on an > internal method _randbelow of the SystemRandom object, but randrange > (if called with only one arg) will go straight into that same method. > Here's my results: > > rosuav at sikorsky:~$ python3 -m timeit -s 'from random import randrange' > 'randrange(10000)' > 1000000 loops, best of 5: 322 nsec per loop > rosuav at sikorsky:~$ python3 -m timeit -s 'from random import > SystemRandom; r = SystemRandom()' 'r.randint(0, 10000)' > 200000 loops, best of 5: 1.92 usec per loop > rosuav at sikorsky:~$ python3 -m timeit -s 'from random import > SystemRandom; r = SystemRandom()' 'r.randrange(10000)' > 200000 loops, best of 5: 1.87 usec per loop > rosuav at sikorsky:~$ python3 -m timeit -s 'from secrets import > randbelow' 'randbelow(10000)' > 200000 loops, best of 5: 1.64 usec per loop > > (The difference with the first one is that it isn't using the system > RNG, so it has the limitations of an internal PRNG.) > > When you call randint, what happens is (1) the endpoint is incremented > to transform it from inclusive-inclusive to inclusive-exclusive; (2) > randrange is called with two args; (3) fast path 1 is skipped, fast > path 2 is taken, and _randbelow gets called to get an actual random > number, which gets zero added to it before returning. > > If, instead, you use randrange(len(to_try)), what would happen is (1) > fast path 1 is used, and (2) _randbelow is called to get the random > number. > > In secrets.randbelow, it's even better: (1) _randbelow is called to > get the random number. > > I think that might be what's going on here. You're adding some tiny > amounts of extra work, but if you were to make them more similar, the > distinctions would evaporate. Here's a couple of other variants that > use SystemRandom: > > rosuav at sikorsky:~$ python3 -m timeit -s 'from random import > SystemRandom; randrange = SystemRandom().randrange' 'randrange(10000)' > 200000 loops, best of 5: 1.81 usec per loop > rosuav at sikorsky:~$ python3 -m timeit -s 'from random import > SystemRandom; randrange = SystemRandom()._randbelow' > 'randrange(10000)' > 200000 loops, best of 5: 1.61 usec per loop > > Note that we shave a few usec by avoiding the attribute lookup, but > even more by directly calling _randbelow, which is why > secrets.randbelow is able to save a bit. But don't use internal > methods; you can probably get pretty much equivalent performance from > randrange, which is public. The times I use are user and sys added. Real can widely differ, but user and sys added should be reasonable constant. (But is not always.) The SystemRandom versions without initiation again and again. This are my results until now for running the complete program: randbelow: 13 min randint: 21 min randrange: 15 min randrange2: 14 min randrange is: system_random = SystemRandom() index = system_random.randrange(len(to_try)) randrange2 is: randrange = SystemRandom().randrange index = randrange(len(to_try)) The strange thing is that the first time I executed randrange2 the time jumped to 20 minutes. But maybe that was because the 'entropy' was used up? I think I stick with the randbelow version. Or is there a reason that it is, or could be better to stay with the randrange version? It was introduced in python 3.6. Should I take into account that people can work with an older version? -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From Cecil at decebal.nl Wed Jul 27 05:27:59 2022 From: Cecil at decebal.nl (Cecil Westerhof) Date: Wed, 27 Jul 2022 11:27:59 +0200 Subject: random.SystemRandom().randint() inefficient References: <87pmhs7xch.fsf@munus.decebal.nl> Message-ID: <87zggu7vmo.fsf@munus.decebal.nl> Chris Angelico writes: > Incidentally - if you are actually trying to select a specific item, > you may want to consider random.choice. Yes, I try to select a random element, but it has also to be removed. An element should be used at most once. This is the code I use: # index = randbelow(len(to_try)) index = randrange(len(to_try)) found = permutation[to_try.pop(index)] -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From Cecil at decebal.nl Wed Jul 27 05:17:05 2022 From: Cecil at decebal.nl (Cecil Westerhof) Date: Wed, 27 Jul 2022 11:17:05 +0200 Subject: random.SystemRandom().randint() inefficient References: <87pmhs7xch.fsf@munus.decebal.nl> <87lesf8ylo.fsf@munus.decebal.nl> <87h7338s1c.fsf@munus.decebal.nl> Message-ID: <874jz29apa.fsf@munus.decebal.nl> Dennis Lee Bieber writes: > On Tue, 26 Jul 2022 23:47:59 +0200, Cecil Westerhof > declaimed the following: > > >>The new code: >> from random import SystemRandom >> system_random = SystemRandom() >> index = system_random.randint(0, len(to_try) - 1) >> >>The first two statements are executed once. >>The last statement I think about 75 * 10 ** 6. >> >>So it seems that my first idea of using randbelow was the correct one. >>But if anyone could explain why SystemRandom is so much more >>expensive, I would be interested to know it. >>(Or am I still doing something wrong?) > > What happens with > > system_randint = SystemRandom().randint #no parens > > index = system_randint(...) > > which may remove the method lookup from the repetition. I had already switched to randrange. This went to 15 minutes from 21 minutes. By removing the method lookup I could shave off another minute. So certainly noteworthy. (Should have thought about it myself.) -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From Cecil at decebal.nl Wed Jul 27 08:58:02 2022 From: Cecil at decebal.nl (Cecil Westerhof) Date: Wed, 27 Jul 2022 14:58:02 +0200 Subject: More efficient code, but slower program Message-ID: <87r1267lwl.fsf@munus.decebal.nl> It is not very important, but I am just curious. Original I had in a program: values = [*range(100)] But because it is done quite often I expected that initialising: range_list = [*range(100)] and then use: values = range_list.copy() Would be more efficient. So I tried: timeit('values = [*range(100)]') 1.6964535564184189 and: timeit('new_values = values.copy()', 'values = [*range(100)]') 0.6457642465829849 That showed that it should make a positive difference. But when changing the program it took a little bit more time. I find the code with the copy a little bit better, so I kept it. But I am curious why the effect is the opposite of what I expected. It does not hurt to understand optimisation better, so I can do a better job when I need it. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From Cecil at decebal.nl Wed Jul 27 11:43:27 2022 From: Cecil at decebal.nl (Cecil Westerhof) Date: Wed, 27 Jul 2022 17:43:27 +0200 Subject: random.SystemRandom().randint() inefficient References: <87pmhs7xch.fsf@munus.decebal.nl> <87lesf8ylo.fsf@munus.decebal.nl> <87h7338s1c.fsf@munus.decebal.nl> Message-ID: <87mtcu7e8w.fsf@munus.decebal.nl> "Michael F. Stemper" writes: > This is orthogonal to your question, but might be of some use to you: > > The combination of using len(to_try) as an argument to randint() and > saving the output to a variable named "index" suggests that you might > be setting up to select a random element from to_try, as in: > something = to_try[index] > > If that is the case, you might want to consider using random.choice() instead: > > >>> from random import choice > >>> to_try = [2,3,5,7,11,13,"seventeen",19] > >>> choice(to_try) > 2 > >>> choice(to_try) > 'seventeen' > >>> choice(to_try) > 13 > >>> choice(to_try) > 5 > >>> Yes, I try to select a random element, but it has also to be removed, because an element should not be used more as once. This is the code I use: # index = randbelow(len(to_try)) index = randrange(len(to_try)) found = permutation[to_try.pop(index)] -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From Cecil at decebal.nl Wed Jul 27 11:48:47 2022 From: Cecil at decebal.nl (Cecil Westerhof) Date: Wed, 27 Jul 2022 17:48:47 +0200 Subject: More efficient code, but slower program References: <87r1267lwl.fsf@munus.decebal.nl> Message-ID: <87ilni7e00.fsf@munus.decebal.nl> ram at zedat.fu-berlin.de (Stefan Ram) writes: > Cecil Westerhof writes: >>values = [*range(100)] > > In many cases, any iterable is just fine and a list is not > required, just as peudo-random numbers often are just fine and > real-world entropy is not required. In this case both are. I must select (several times) a random element from the list. So I need the list. I also want the randomness to be as good as possible to make the 'simulation' as good as possible. > Usually one wants to write code for readability, and thinking > too much about runtime efficiency optimizations is in vain, > because one might get different results with a different > version of Python or on a different machine. That is why I went for the less efficient code. ;-) -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From morphex at gmail.com Wed Jul 27 06:56:12 2022 From: morphex at gmail.com (Morten W. Petersen) Date: Wed, 27 Jul 2022 12:56:12 +0200 Subject: Simple TCP proxy Message-ID: Hi. I'd like to share with you a recent project, which is a simple TCP proxy that can stand in front of a TCP server of some sort, queueing requests and then allowing n number of connections to pass through at a time: https://github.com/morphex/stp I'll be developing it further, but the the files committed in this tree seem to be stable: https://github.com/morphex/stp/tree/9910ca8c80e9d150222b680a4967e53f0457b465 I just bombed that code with 700+ requests almost simultaneously, and STP handled it well. Regards, Morten -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ From michael.stemper at gmail.com Wed Jul 27 09:19:35 2022 From: michael.stemper at gmail.com (Michael F. Stemper) Date: Wed, 27 Jul 2022 08:19:35 -0500 Subject: random.SystemRandom().randint() inefficient In-Reply-To: <87h7338s1c.fsf@munus.decebal.nl> References: <87pmhs7xch.fsf@munus.decebal.nl> <87lesf8ylo.fsf@munus.decebal.nl> <87h7338s1c.fsf@munus.decebal.nl> Message-ID: On 26/07/2022 16.47, Cecil Westerhof wrote: > Chris Angelico writes: > >> On Wed, 27 Jul 2022 at 06:06, Cecil Westerhof via Python-list >> wrote: >>> >>> Chris Angelico writes: >>> >>>> On Wed, 27 Jul 2022 at 01:06, Cecil Westerhof via Python-list >>>> wrote: >>>>> >>>>> I need to get a random integer. At first I tried it with: >>>>> from secrets import randbelow >>>>> index = randbelow(len(to_try)) >>>>> >>>>> This works perfectly, but it took some time. So I thought I try: >>>>> from random import SystemRandom >>>>> index = SystemRandom().randint(0, len(to_try) - 1) >>>>> >>>>> A first indication is that the second version would take about two >>>>> times as much time as the first. Is there a reason for this, or should >>>>> this not be happening? >>>>> >>>> >>>> You're setting up a brand new SystemRandom instance just for a single >>>> random number. For a fairer comparison, set up the instance, then >>>> generate far more than just a single number, and see how that goes. >>> >>> Thanks. I thought I did something wrong and I did. >>> I will try to implement like you said and look what the result will >>> be. (And share it.) >> >> Thanks! Don't feel bad; performance testing is *hard*, getting >> meaningful results takes a lot of of fiddling with parameters, and >> getting interesting AND meaningful results can sometimes seem about >> impossible. >> >>> (As I understand it both do more, or less the same and should have >>> comparable performance.) >> >> In normal production work? Yes (the SystemRandom object doesn't have >> any significant state - a seeded RNG could have a lot more overhead >> here). But for performance testing? The work of instantiating the >> class could be completely irrelevant, or it could be dominating your >> results. It's hard to say, hence the suggestion to try it without >> reinstantiating. > > It had a very big influence. Original it took about three times more > time to run my program. (The program was still running when I posted > the original post and the difference was higher as I anticipated.) > Removing that did cut about 45% of the execution time of the program. > (So the initiation is quit expensive.) > But it still takes about 50% more time. So I am still a bit > flabbergasted. > > The new code: > from random import SystemRandom > system_random = SystemRandom() > index = system_random.randint(0, len(to_try) - 1) This is orthogonal to your question, but might be of some use to you: The combination of using len(to_try) as an argument to randint() and saving the output to a variable named "index" suggests that you might be setting up to select a random element from to_try, as in: something = to_try[index] If that is the case, you might want to consider using random.choice() instead: >>> from random import choice >>> to_try = [2,3,5,7,11,13,"seventeen",19] >>> choice(to_try) 2 >>> choice(to_try) 'seventeen' >>> choice(to_try) 13 >>> choice(to_try) 5 >>> -- Michael F. Stemper This sentence no verb. From 2QdxY4RzWzUUiLuE at potatochowder.com Wed Jul 27 12:24:12 2022 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Wed, 27 Jul 2022 11:24:12 -0500 Subject: More efficient code, but slower program In-Reply-To: <87ilni7e00.fsf@munus.decebal.nl> References: <87r1267lwl.fsf@munus.decebal.nl> <87ilni7e00.fsf@munus.decebal.nl> Message-ID: On 2022-07-27 at 17:48:47 +0200, Regarding "Re: More efficient code, but slower program," Cecil Westerhof via Python-list wrote: > ram at zedat.fu-berlin.de (Stefan Ram) writes: > > > Cecil Westerhof writes: > >>values = [*range(100)] > > > > In many cases, any iterable is just fine and a list is not > > required, just as peudo-random numbers often are just fine and > > real-world entropy is not required. > > In this case both are. I must select (several times) a random element > from the list. So I need the list. > I also want the randomness to be as good as possible to make the > 'simulation' as good as possible. "[A]s good as possible" for simulations and tests may not require the cryptographic quality numbers from SystemRandom. Many/most pseudo random number generators are optimized for statistically normalized outputs, and are repeatable as a bonus (again, often a requirement for certain types of simulations; YMMV). Also, what if you shuffled the list first (e.g., with random.shuffle) and then iterated through it directly? Repeatedly removing arbitrary elements from the middle of a list is potentially expensive. From python at mrabarnett.plus.com Wed Jul 27 12:30:48 2022 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 27 Jul 2022 17:30:48 +0100 Subject: random.SystemRandom().randint() inefficient In-Reply-To: <87mtcu7e8w.fsf@munus.decebal.nl> References: <87pmhs7xch.fsf@munus.decebal.nl> <87lesf8ylo.fsf@munus.decebal.nl> <87h7338s1c.fsf@munus.decebal.nl> <87mtcu7e8w.fsf@munus.decebal.nl> Message-ID: <3f359767-dffc-2984-afd3-efb8323ae3c2@mrabarnett.plus.com> On 27/07/2022 16:43, Cecil Westerhof via Python-list wrote: > "Michael F. Stemper" writes: > >> This is orthogonal to your question, but might be of some use to you: >> >> The combination of using len(to_try) as an argument to randint() and >> saving the output to a variable named "index" suggests that you might >> be setting up to select a random element from to_try, as in: >> something = to_try[index] >> >> If that is the case, you might want to consider using random.choice() instead: >> >> >>> from random import choice >> >>> to_try = [2,3,5,7,11,13,"seventeen",19] >> >>> choice(to_try) >> 2 >> >>> choice(to_try) >> 'seventeen' >> >>> choice(to_try) >> 13 >> >>> choice(to_try) >> 5 >> >>> > > Yes, I try to select a random element, but it has also to be removed, > because an element should not be used more as once. > This is the code I use: > # index = randbelow(len(to_try)) > index = randrange(len(to_try)) > found = permutation[to_try.pop(index)] > When you pop an element from the last, the elements after it need to be moved down, which takes time. Try shuffling the list and then popping the now randomly-ordered elements off the end. From wlfraed at ix.netcom.com Wed Jul 27 12:13:38 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Wed, 27 Jul 2022 12:13:38 -0400 Subject: random.SystemRandom().randint() inefficient References: <87pmhs7xch.fsf@munus.decebal.nl> <296D2D4B-F400-429A-8559-1D9AF9658D81@barrys-emacs.org> <87czdr7xl0.fsf@munus.decebal.nl> Message-ID: On Wed, 27 Jul 2022 10:45:47 +0200, Cecil Westerhof declaimed the following: >What do you mean with where the python version is from? Base Python.org download, ActiveState package download, Anaconda package download, native OS install/extra install via OS repository download (Debian/Ubuntu: apt install xxx, where xxx is not the native OS Python) -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From mats at wichmann.us Wed Jul 27 13:37:44 2022 From: mats at wichmann.us (Mats Wichmann) Date: Wed, 27 Jul 2022 11:37:44 -0600 Subject: random.SystemRandom().randint() inefficient In-Reply-To: <87czdr7xl0.fsf@munus.decebal.nl> References: <87pmhs7xch.fsf@munus.decebal.nl> <296D2D4B-F400-429A-8559-1D9AF9658D81@barrys-emacs.org> <87czdr7xl0.fsf@munus.decebal.nl> Message-ID: <6a4912ad-bd5a-f159-f413-ab3fa37eeb6e@wichmann.us> On 7/27/22 02:45, Cecil Westerhof via Python-list wrote: > Barry writes: >> What version of python and where from? > > That is always good information of-course. > Debian 11.3 > 5.10.0-13-amd64 > 3.9.2 > > What do you mean with where the python version is from? On Windows, the platform of a large proportion of people asking questions here, there are many possible Python builds (python.org, Microsoft Store, Conda, ActiveState, etc.) and it sometimes matters, thus that tends to be a standard question that gets asked here. In your case the implied answer is "distro package". From rosuav at gmail.com Wed Jul 27 13:58:19 2022 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 28 Jul 2022 03:58:19 +1000 Subject: Simple TCP proxy In-Reply-To: References: Message-ID: On Thu, 28 Jul 2022 at 02:15, Morten W. Petersen wrote: > > Hi. > > I'd like to share with you a recent project, which is a simple TCP proxy > that can stand in front of a TCP server of some sort, queueing requests and > then allowing n number of connections to pass through at a time: How's this different from what the networking subsystem already does? When you listen, you can set a queue length. Can you elaborate? ChrisA From barry at barrys-emacs.org Wed Jul 27 14:00:42 2022 From: barry at barrys-emacs.org (Barry) Date: Wed, 27 Jul 2022 19:00:42 +0100 Subject: random.SystemRandom().randint() inefficient In-Reply-To: <87czdr7xl0.fsf@munus.decebal.nl> References: <87czdr7xl0.fsf@munus.decebal.nl> Message-ID: <473B634B-494D-47BC-A2AB-1F4DD707DF03@barrys-emacs.org> > On 27 Jul 2022, at 17:09, Cecil Westerhof via Python-list wrote: > > ?Barry writes: > >>>> On 26 Jul 2022, at 16:07, Cecil Westerhof via Python-list wrote: >>> >>> ?I need to get a random integer. At first I tried it with: >>> from secrets import randbelow >>> index = randbelow(len(to_try)) >>> >>> This works perfectly, but it took some time. So I thought I try: >>> from random import SystemRandom >>> index = SystemRandom().randint(0, len(to_try) - 1) >>> >>> A first indication is that the second version would take about two >>> times as much time as the first. Is there a reason for this, or should >>> this not be happening? >> >> What is the OS that you are running on and its version? >> If it?s linux what is the kernel version? >> What version of python and where from? > > That is always good information of-course. > Debian 11.3 > 5.10.0-13-amd64 > 3.9.2 > > What do you mean with where the python version is from? Because random number generator implementation depends on the OS since it is SystemRandom() that you are comparing with python?s Implementation. Barry > > -- > Cecil Westerhof > Senior Software Engineer > LinkedIn: http://www.linkedin.com/in/cecilwesterhof > -- > https://mail.python.org/mailman/listinfo/python-list From wlfraed at ix.netcom.com Wed Jul 27 12:35:23 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Wed, 27 Jul 2022 12:35:23 -0400 Subject: More efficient code, but slower program References: <87r1267lwl.fsf@munus.decebal.nl> Message-ID: On Wed, 27 Jul 2022 14:58:02 +0200, Cecil Westerhof declaimed the following: >It is not very important, but I am just curious. > >Original I had in a program: > values = [*range(100)] > >But because it is done quite often I expected that initialising: > range_list = [*range(100)] > >and then use: > values = range_list.copy() > >Would be more efficient. So I tried: > timeit('values = [*range(100)]') > 1.6964535564184189 > >and: > timeit('new_values = values.copy()', 'values = [*range(100)]') > 0.6457642465829849 > Were these done in the same program/session? If so, the first invocation may be initializing/caching the first 100 integers (Python tends to keep some number of integers in a permanent cache to speed later access to common values). Also rather than * unpacking of the range iterator into a [] list... just... >>> list(range(100)) [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99] >>> ... will do it. Also, if your goal is to /remove/ and entry from the list via some index, you might consider if this is more effective than copying the list and THEN popping a value. >>> full = list(range(100)) >>> import random >>> idx = random.randint(0, len(full)) >>> idx 74 >>> trim = full[:idx] + full[idx+1:] >>> trim [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99] >>> trim == full False >>> or >>> trim2 = full[:] >>> del trim2[idx] >>> trim [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99] >>> The first does two partial copies skipping the item to be removed, and joins the results into a new list. The second does a full copy and DELETES the element to be removed from the copy. >>> trim3 = full[:] >>> trim3.remove(idx) >>> trim3 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99] >>> This option works because the list is sequential integers and the index matches the values in the list (.remove() removes the first MATCHING element, so if the list can have duplicates is may not remove the one AT the index position). -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From Cecil at decebal.nl Wed Jul 27 13:21:53 2022 From: Cecil at decebal.nl (Cecil Westerhof) Date: Wed, 27 Jul 2022 19:21:53 +0200 Subject: random.SystemRandom().randint() inefficient References: <87pmhs7xch.fsf@munus.decebal.nl> <296D2D4B-F400-429A-8559-1D9AF9658D81@barrys-emacs.org> <87czdr7xl0.fsf@munus.decebal.nl> Message-ID: <87edy679ou.fsf@munus.decebal.nl> Dennis Lee Bieber writes: > On Wed, 27 Jul 2022 10:45:47 +0200, Cecil Westerhof > declaimed the following: > > >>What do you mean with where the python version is from? > > Base Python.org download, ActiveState package download, Anaconda > package download, native OS install/extra install via OS repository > download (Debian/Ubuntu: apt install xxx, where xxx is not the native OS > Python) Just the default Debian install. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From Cecil at decebal.nl Wed Jul 27 13:24:14 2022 From: Cecil at decebal.nl (Cecil Westerhof) Date: Wed, 27 Jul 2022 19:24:14 +0200 Subject: random.SystemRandom().randint() inefficient References: <87pmhs7xch.fsf@munus.decebal.nl> <87lesf8ylo.fsf@munus.decebal.nl> <87h7338s1c.fsf@munus.decebal.nl> <87mtcu7e8w.fsf@munus.decebal.nl> <3f359767-dffc-2984-afd3-efb8323ae3c2@mrabarnett.plus.com> Message-ID: <87a68u79kx.fsf@munus.decebal.nl> MRAB writes: > On 27/07/2022 16:43, Cecil Westerhof via Python-list wrote: >> "Michael F. Stemper" writes: >> >>> This is orthogonal to your question, but might be of some use to you: >>> >>> The combination of using len(to_try) as an argument to randint() and >>> saving the output to a variable named "index" suggests that you might >>> be setting up to select a random element from to_try, as in: >>> something = to_try[index] >>> >>> If that is the case, you might want to consider using random.choice() instead: >>> >>> >>> from random import choice >>> >>> to_try = [2,3,5,7,11,13,"seventeen",19] >>> >>> choice(to_try) >>> 2 >>> >>> choice(to_try) >>> 'seventeen' >>> >>> choice(to_try) >>> 13 >>> >>> choice(to_try) >>> 5 >>> >>> >> Yes, I try to select a random element, but it has also to be removed, >> because an element should not be used more as once. >> This is the code I use: >> # index = randbelow(len(to_try)) >> index = randrange(len(to_try)) >> found = permutation[to_try.pop(index)] >> > > When you pop an element from the last, the elements after it need to be > moved down, which takes time. > > Try shuffling the list and then popping the now randomly-ordered > elements off the end. Would shuffling not be a lot more expensive? Especially because I do not eat the whole list. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From Cecil at decebal.nl Wed Jul 27 13:33:07 2022 From: Cecil at decebal.nl (Cecil Westerhof) Date: Wed, 27 Jul 2022 19:33:07 +0200 Subject: More efficient code, but slower program References: <87r1267lwl.fsf@munus.decebal.nl> <87ilni7e00.fsf@munus.decebal.nl> Message-ID: <875yji7964.fsf@munus.decebal.nl> 2QdxY4RzWzUUiLuE at potatochowder.com writes: > On 2022-07-27 at 17:48:47 +0200, > Regarding "Re: More efficient code, but slower program," > Cecil Westerhof via Python-list wrote: > >> ram at zedat.fu-berlin.de (Stefan Ram) writes: >> >> > Cecil Westerhof writes: >> >>values = [*range(100)] >> > >> > In many cases, any iterable is just fine and a list is not >> > required, just as peudo-random numbers often are just fine and >> > real-world entropy is not required. >> >> In this case both are. I must select (several times) a random element >> from the list. So I need the list. >> I also want the randomness to be as good as possible to make the >> 'simulation' as good as possible. > > "[A]s good as possible" for simulations and tests may not require the > cryptographic quality numbers from SystemRandom. Many/most pseudo > random number generators are optimized for statistically normalized > outputs, and are repeatable as a bonus (again, often a requirement for > certain types of simulations; YMMV). > > Also, what if you shuffled the list first (e.g., with random.shuffle) > and then iterated through it directly? Repeatedly removing arbitrary > elements from the middle of a list is potentially expensive. Someone else also mentioned that. I think I should put it on my list of things to do/try. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From morphex at gmail.com Wed Jul 27 14:32:31 2022 From: morphex at gmail.com (Morten W. Petersen) Date: Wed, 27 Jul 2022 20:32:31 +0200 Subject: Simple TCP proxy In-Reply-To: References: Message-ID: Hi Chris. You're thinking of the backlog argument of listen? Well, STP will accept all connections, but can limit how many of the accepted connections that are active at any given time. So when I bombed it with hundreds of almost simultaneous connections, all of them were accepted, but only 25 were actively sending and receiving data at any given time. First come, first served. Regards, Morten On Wed, Jul 27, 2022 at 8:00 PM Chris Angelico wrote: > On Thu, 28 Jul 2022 at 02:15, Morten W. Petersen > wrote: > > > > Hi. > > > > I'd like to share with you a recent project, which is a simple TCP proxy > > that can stand in front of a TCP server of some sort, queueing requests > and > > then allowing n number of connections to pass through at a time: > > How's this different from what the networking subsystem already does? > When you listen, you can set a queue length. Can you elaborate? > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Instagram at https://instagram.com/morphexx/ From roel at roelschroeven.net Wed Jul 27 14:44:45 2022 From: roel at roelschroeven.net (Roel Schroeven) Date: Wed, 27 Jul 2022 20:44:45 +0200 Subject: random.SystemRandom().randint() inefficient In-Reply-To: <87mtcu7e8w.fsf@munus.decebal.nl> References: <87pmhs7xch.fsf@munus.decebal.nl> <87lesf8ylo.fsf@munus.decebal.nl> <87h7338s1c.fsf@munus.decebal.nl> <87mtcu7e8w.fsf@munus.decebal.nl> Message-ID: Cecil Westerhof via Python-list schreef op 27/07/2022 om 17:43: > "Michael F. Stemper" writes: > > > This is orthogonal to your question, but might be of some use to you: > > > > The combination of using len(to_try) as an argument to randint() and > > saving the output to a variable named "index" suggests that you might > > be setting up to select a random element from to_try, as in: > > something = to_try[index] > > > > If that is the case, you might want to consider using random.choice() instead: > > > > >>> from random import choice > > >>> to_try = [2,3,5,7,11,13,"seventeen",19] > > >>> choice(to_try) > > 2 > > >>> choice(to_try) > > 'seventeen' > > >>> choice(to_try) > > 13 > > >>> choice(to_try) > > 5 > > >>> > > Yes, I try to select a random element, but it has also to be removed, > because an element should not be used more as once. > This is the code I use: > # index = randbelow(len(to_try)) > index = randrange(len(to_try)) > found = permutation[to_try.pop(index)] Do you know in advance how many items you'll need, or maybe an upper limit on the amount? In that case it might be more efficient to use random.sample(to_try, k=nr_items_needed). -- "Honest criticism is hard to take, particularly from a relative, a friend, an acquaintance, or a stranger." -- Franklin P. Jones From alan at csail.mit.edu Wed Jul 27 14:42:55 2022 From: alan at csail.mit.edu (Alan Bawden) Date: Wed, 27 Jul 2022 14:42:55 -0400 Subject: random.SystemRandom().randint() inefficient References: <87pmhs7xch.fsf@munus.decebal.nl> <87lesf8ylo.fsf@munus.decebal.nl> <87h7338s1c.fsf@munus.decebal.nl> <87mtcu7e8w.fsf@munus.decebal.nl> Message-ID: <86a68utn0w.fsf@williamsburg.bawden.org> Cecil Westerhof writes: Yes, I try to select a random element, but it has also to be removed, because an element should not be used more as once. Instead of using pop to do that why not something like: def lazy_shuffle(seq): """ Generate the elements of the given sequence in a random order. """ # Delete the next line if you want to use a different version of # randrange: from random import randrange # Delete the next line if SEQ is already a mutable sequence and you # are willing to have it destroyed by this process: seq = list(seq) n = len(seq) while n: i = randrange(n) yield seq[i] n -= 1 if i < n: seq[i] = seq[n] -- Alan Bawden From python at mrabarnett.plus.com Wed Jul 27 14:55:26 2022 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 27 Jul 2022 19:55:26 +0100 Subject: random.SystemRandom().randint() inefficient In-Reply-To: <87a68u79kx.fsf@munus.decebal.nl> References: <87pmhs7xch.fsf@munus.decebal.nl> <87lesf8ylo.fsf@munus.decebal.nl> <87h7338s1c.fsf@munus.decebal.nl> <87mtcu7e8w.fsf@munus.decebal.nl> <3f359767-dffc-2984-afd3-efb8323ae3c2@mrabarnett.plus.com> <87a68u79kx.fsf@munus.decebal.nl> Message-ID: <1599f48b-c26d-1827-653d-464d89ea566a@mrabarnett.plus.com> On 27/07/2022 18:24, Cecil Westerhof via Python-list wrote: > MRAB writes: > >> On 27/07/2022 16:43, Cecil Westerhof via Python-list wrote: >>> "Michael F. Stemper" writes: >>> >>>> This is orthogonal to your question, but might be of some use to you: >>>> >>>> The combination of using len(to_try) as an argument to randint() and >>>> saving the output to a variable named "index" suggests that you might >>>> be setting up to select a random element from to_try, as in: >>>> something = to_try[index] >>>> >>>> If that is the case, you might want to consider using random.choice() instead: >>>> >>>> >>> from random import choice >>>> >>> to_try = [2,3,5,7,11,13,"seventeen",19] >>>> >>> choice(to_try) >>>> 2 >>>> >>> choice(to_try) >>>> 'seventeen' >>>> >>> choice(to_try) >>>> 13 >>>> >>> choice(to_try) >>>> 5 >>>> >>> >>> Yes, I try to select a random element, but it has also to be removed, >>> because an element should not be used more as once. >>> This is the code I use: >>> # index = randbelow(len(to_try)) >>> index = randrange(len(to_try)) >>> found = permutation[to_try.pop(index)] >>> >> >> When you pop an element from the last, the elements after it need to be >> moved down, which takes time. >> >> Try shuffling the list and then popping the now randomly-ordered >> elements off the end. > > Would shuffling not be a lot more expensive? Especially because I do > not eat the whole list. > You won't know until you time it. From Cecil at decebal.nl Wed Jul 27 15:16:23 2022 From: Cecil at decebal.nl (Cecil Westerhof) Date: Wed, 27 Jul 2022 21:16:23 +0200 Subject: random.SystemRandom().randint() inefficient References: <87pmhs7xch.fsf@munus.decebal.nl> <87lesf8ylo.fsf@munus.decebal.nl> <87h7338s1c.fsf@munus.decebal.nl> <87mtcu7e8w.fsf@munus.decebal.nl> Message-ID: <87wnby5ptk.fsf@munus.decebal.nl> Roel Schroeven writes: > Cecil Westerhof via Python-list schreef op 27/07/2022 om 17:43: >> "Michael F. Stemper" writes: >> >> > This is orthogonal to your question, but might be of some use to you: >> > >> > The combination of using len(to_try) as an argument to randint() and >> > saving the output to a variable named "index" suggests that you might >> > be setting up to select a random element from to_try, as in: >> > something = to_try[index] >> > >> > If that is the case, you might want to consider using random.choice() instead: >> > >> > >>> from random import choice >> > >>> to_try = [2,3,5,7,11,13,"seventeen",19] >> > >>> choice(to_try) >> > 2 >> > >>> choice(to_try) >> > 'seventeen' >> > >>> choice(to_try) >> > 13 >> > >>> choice(to_try) >> > 5 >> > >>> >> >> Yes, I try to select a random element, but it has also to be removed, >> because an element should not be used more as once. >> This is the code I use: >> # index = randbelow(len(to_try)) >> index = randrange(len(to_try)) >> found = permutation[to_try.pop(index)] > Do you know in advance how many items you'll need, or maybe an upper > limit on the amount? In that case it might be more efficient to use > random.sample(to_try, k=nr_items_needed). Something else to try. :-) And yes: I will be using half of the list. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From Cecil at decebal.nl Wed Jul 27 15:17:41 2022 From: Cecil at decebal.nl (Cecil Westerhof) Date: Wed, 27 Jul 2022 21:17:41 +0200 Subject: random.SystemRandom().randint() inefficient References: <87pmhs7xch.fsf@munus.decebal.nl> <87lesf8ylo.fsf@munus.decebal.nl> <87h7338s1c.fsf@munus.decebal.nl> <87mtcu7e8w.fsf@munus.decebal.nl> <3f359767-dffc-2984-afd3-efb8323ae3c2@mrabarnett.plus.com> <87a68u79kx.fsf@munus.decebal.nl> <1599f48b-c26d-1827-653d-464d89ea566a@mrabarnett.plus.com> Message-ID: <87sfmm5pre.fsf@munus.decebal.nl> MRAB writes: >>> When you pop an element from the last, the elements after it need to be >>> moved down, which takes time. >>> >>> Try shuffling the list and then popping the now randomly-ordered >>> elements off the end. >> Would shuffling not be a lot more expensive? Especially because I do >> not eat the whole list. >> > You won't know until you time it. A first indication is that it doubles the needed time. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From Cecil at decebal.nl Wed Jul 27 15:14:29 2022 From: Cecil at decebal.nl (Cecil Westerhof) Date: Wed, 27 Jul 2022 21:14:29 +0200 Subject: random.SystemRandom().randint() inefficient References: <87pmhs7xch.fsf@munus.decebal.nl> <87lesf8ylo.fsf@munus.decebal.nl> <87h7338s1c.fsf@munus.decebal.nl> <87mtcu7e8w.fsf@munus.decebal.nl> <86a68utn0w.fsf@williamsburg.bawden.org> Message-ID: <871qu674h6.fsf@munus.decebal.nl> Alan Bawden writes: > Cecil Westerhof writes: > > Yes, I try to select a random element, but it has also to be removed, > because an element should not be used more as once. > > Instead of using pop to do that why not something like: > > def lazy_shuffle(seq): > """ > Generate the elements of the given sequence in a random order. > """ > # Delete the next line if you want to use a different version of > # randrange: > from random import randrange > # Delete the next line if SEQ is already a mutable sequence and you > # are willing to have it destroyed by this process: > seq = list(seq) > n = len(seq) > while n: > i = randrange(n) > yield seq[i] > n -= 1 > if i < n: > seq[i] = seq[n] That looks interesting. But there is on problem. (I think.) The list is only partly eaten and I will eat a lot of sequences. So a lot of sequences will be left in the runtime. Or is there a way to destroy the lazy_shuffle when it is not needed anymore? -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From alan at csail.mit.edu Wed Jul 27 15:42:25 2022 From: alan at csail.mit.edu (Alan Bawden) Date: Wed, 27 Jul 2022 15:42:25 -0400 Subject: random.SystemRandom().randint() inefficient References: <87pmhs7xch.fsf@munus.decebal.nl> <87lesf8ylo.fsf@munus.decebal.nl> <87h7338s1c.fsf@munus.decebal.nl> <87mtcu7e8w.fsf@munus.decebal.nl> <86a68utn0w.fsf@williamsburg.bawden.org> <871qu674h6.fsf@munus.decebal.nl> Message-ID: <865yjitk9q.fsf@williamsburg.bawden.org> Cecil Westerhof writes: Alan Bawden writes: > Cecil Westerhof writes: > > Yes, I try to select a random element, but it has also to be removed, > because an element should not be used more as once. > > Instead of using pop to do that why not something like: > > def lazy_shuffle(seq): > """ > Generate the elements of the given sequence in a random order. > """ > # Delete the next line if you want to use a different version of > # randrange: > from random import randrange > # Delete the next line if SEQ is already a mutable sequence and you > # are willing to have it destroyed by this process: > seq = list(seq) > n = len(seq) > while n: > i = randrange(n) > yield seq[i] > n -= 1 > if i < n: > seq[i] = seq[n] That looks interesting. But there is on problem. (I think.) The list is only partly eaten and I will eat a lot of sequences. So a lot of sequences will be left in the runtime. Or is there a way to destroy the lazy_shuffle when it is not needed anymore? You don't have to worry about that. That's what Garbage Collection is for. After you drop the last reference to the generator, the GC will destroy it for you. Welcome to Python. BTW, what I wrote might be slightly faster if you replace it with: while n: i = randrange(n) yield seq[i] n -= 1 seq[i] = seq[n] That will save you some time spent testing and branching. -- Alan Bawden From rosuav at gmail.com Wed Jul 27 15:54:48 2022 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 28 Jul 2022 05:54:48 +1000 Subject: Simple TCP proxy In-Reply-To: References: Message-ID: On Thu, 28 Jul 2022 at 04:32, Morten W. Petersen wrote: > > Hi Chris. > > You're thinking of the backlog argument of listen? Yes, precisely. > Well, STP will accept all connections, but can limit how many of the accepted connections that are active at any given time. > > So when I bombed it with hundreds of almost simultaneous connections, all of them were accepted, but only 25 were actively sending and receiving data at any given time. First come, first served. > Hmm. Okay. Not sure what the advantage is, but sure. If the server's capable of handling the total requests-per-minute, then a queueing system like this should help with burst load, although I would have thought that the listen backlog would do the same. What happens if the server actually gets overloaded though? Do connections get disconnected after appearing connected? What's the disconnect mode? BTW, you probably don't want to be using the _thread module - Python has a threading module which is better suited to this sort of work. Although you may want to consider asyncio instead, as that has far lower overhead when working with large numbers of sockets. ChrisA From rosuav at gmail.com Wed Jul 27 15:57:22 2022 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 28 Jul 2022 05:57:22 +1000 Subject: random.SystemRandom().randint() inefficient In-Reply-To: <87wnby5ptk.fsf@munus.decebal.nl> References: <87pmhs7xch.fsf@munus.decebal.nl> <87lesf8ylo.fsf@munus.decebal.nl> <87h7338s1c.fsf@munus.decebal.nl> <87mtcu7e8w.fsf@munus.decebal.nl> <87wnby5ptk.fsf@munus.decebal.nl> Message-ID: On Thu, 28 Jul 2022 at 05:36, Cecil Westerhof via Python-list wrote: > > Roel Schroeven writes: > > > Cecil Westerhof via Python-list schreef op 27/07/2022 om 17:43: > >> "Michael F. Stemper" writes: > >> > >> > This is orthogonal to your question, but might be of some use to you: > >> > > >> > The combination of using len(to_try) as an argument to randint() and > >> > saving the output to a variable named "index" suggests that you might > >> > be setting up to select a random element from to_try, as in: > >> > something = to_try[index] > >> > > >> > If that is the case, you might want to consider using random.choice() instead: > >> > > >> > >>> from random import choice > >> > >>> to_try = [2,3,5,7,11,13,"seventeen",19] > >> > >>> choice(to_try) > >> > 2 > >> > >>> choice(to_try) > >> > 'seventeen' > >> > >>> choice(to_try) > >> > 13 > >> > >>> choice(to_try) > >> > 5 > >> > >>> > >> > >> Yes, I try to select a random element, but it has also to be removed, > >> because an element should not be used more as once. > >> This is the code I use: > >> # index = randbelow(len(to_try)) > >> index = randrange(len(to_try)) > >> found = permutation[to_try.pop(index)] > > Do you know in advance how many items you'll need, or maybe an upper > > limit on the amount? In that case it might be more efficient to use > > random.sample(to_try, k=nr_items_needed). > > Something else to try. :-) > And yes: I will be using half of the list. > A perfect job for random.sample() then, if that's *exactly* half the list. But if you don't know for sure how many you'll need, an easy way to make it more efficient would be to take the n'th element, then move the last element down to the n'th position, and finally pop off the last element. That way, you only ever pop the last element away. The list will get reordered arbitrarily while you do this, but if the only purpose of it is to remove elements from random consideration, that won't be a problem. ChrisA From martinp.dipaola at gmail.com Wed Jul 27 16:58:10 2022 From: martinp.dipaola at gmail.com (Martin Di Paola) Date: Wed, 27 Jul 2022 17:58:10 -0300 Subject: Simple TCP proxy In-Reply-To: References: Message-ID: <20220727205810.55ivclftzsggrenx@gmail.com> On Wed, Jul 27, 2022 at 08:32:31PM +0200, Morten W. Petersen wrote: >You're thinking of the backlog argument of listen? From my understanding, yes, when you set up the "accepter" socket (the one that you use to listen and accept new connections), you can define the length of the queue for incoming connections that are not accepted yet. This will be the equivalent of your SimpleQueue which basically puts a limits on how many incoming connections are "accepted" to do a real job. Using skt.listen(N) the incoming connections are put on hold by the OS while in your implementation are formally accepted but they are not allowed to do any meaningful work: they are put on the SimpleQueue and only when they are popped then they will work (send/recv data). The difference then between the OS and your impl is minimal. The only case that I can think is that on the clients' side it may exist a timeout for the acceptance of the connection so your proxy server will eagerly accept these connections so no timeout is possible(*) On a side note, you implementation is too thread-naive: it uses plain Python lists, integers and boolean variables which are not thread safe. It is a matter of time until your server will start behave weird. One option is that you use thread-safe objects. I'll encourage to read about thread-safety in general and then which sync mechanisms Python offers. Another option is to remove the SimpleQueue and the background function that allows a connection to be "active". If you think, the handlers are 99% independent except that you want to allow only N of them to progress (stablish and forward the connection) and when a handler finishes, another handler "waiting" is activated, "in a queue fashion" as you said. If you allow me to not have a strict queue discipline here, you can achieve the same results coordinating the handlers using semaphores. Once again, take this email as starting point for your own research. On a second side note, the use of handlers and threads is inefficient because while you have N active handlers sending/receiving data, because you are eagerly accepting new connections you will have much more handlers created and (if I'm not wrong), each will be a thread. A more efficient solution could be 1) accept as many connections as you can, saving the socket (not the handler) in the thread-safe queue. 2) have N threads in the background popping from the queue a socket and then doing the send/recv stuff. When the thread is done, the thread closes the socket and pops another from the queue. So the queue length will be the count of accepted connections but in any moment your proxy will not activate (forward) more than N connections. This idea is thread-safe, simpler, efficient and has the queue discipline (I leave aside the usefulness). I encourage you to take time to read about the different things mentioned as concurrency and thread-related stuff is not easy to master. Thanks, Martin. (*) make your proxy server slow enough and yes, you will get timeouts anyways. > >Well, STP will accept all connections, but can limit how many of the >accepted connections that are active at any given time. > >So when I bombed it with hundreds of almost simultaneous connections, all >of them were accepted, but only 25 were actively sending and receiving data >at any given time. First come, first served. > >Regards, > >Morten > >On Wed, Jul 27, 2022 at 8:00 PM Chris Angelico wrote: > >> On Thu, 28 Jul 2022 at 02:15, Morten W. Petersen >> wrote: >> > >> > Hi. >> > >> > I'd like to share with you a recent project, which is a simple TCP proxy >> > that can stand in front of a TCP server of some sort, queueing requests >> and >> > then allowing n number of connections to pass through at a time: >> >> How's this different from what the networking subsystem already does? >> When you listen, you can set a queue length. Can you elaborate? >> >> ChrisA >> -- >> https://mail.python.org/mailman/listinfo/python-list >> > > >-- >I am https://leavingnorway.info >Videos at https://www.youtube.com/user/TheBlogologue >Twittering at http://twitter.com/blogologue >Blogging at http://blogologue.com >Playing music at https://soundcloud.com/morten-w-petersen >Also playing music and podcasting here: >http://www.mixcloud.com/morten-w-petersen/ >On Google+ here https://plus.google.com/107781930037068750156 >On Instagram at https://instagram.com/morphexx/ >-- > >I am https://leavingnorway.info > >Videos at https://www.youtube.com/user/TheBlogologue >Twittering at http://twitter.com/blogologue > >Blogging at http://blogologue.com >Playing music at https://soundcloud.com/morten-w-petersen > >Also playing music and podcasting here: >http://www.mixcloud.com/morten-w-petersen/ > >On Instagram at https://instagram.com/morphexx/ >-- >https://mail.python.org/mailman/listinfo/python-list From barry at barrys-emacs.org Thu Jul 28 02:31:49 2022 From: barry at barrys-emacs.org (Barry) Date: Thu, 28 Jul 2022 07:31:49 +0100 Subject: Simple TCP proxy In-Reply-To: References: Message-ID: > On 27 Jul 2022, at 17:16, Morten W. Petersen wrote: > > ?Hi. > > I'd like to share with you a recent project, which is a simple TCP proxy > that can stand in front of a TCP server of some sort, queueing requests and > then allowing n number of connections to pass through at a time: > > https://github.com/morphex/stp > > I'll be developing it further, but the the files committed in this tree > seem to be stable: > > https://github.com/morphex/stp/tree/9910ca8c80e9d150222b680a4967e53f0457b465 > > I just bombed that code with 700+ requests almost simultaneously, and STP > handled it well. What is the problem that this solves? Why not just increase the allowed size of the socket listen backlog if you just want to handle bursts of traffic. I do not think of this as a proxy, rather a tunnel. And the tunnel is a lot more expensive the having kernel keep the connection in the listen socket backlog. I work on a web proxy written on python that handles huge load and using backlog of the bursts. It?s async using twisted as threads are not practice at scale. Barry > > Regards, > > Morten > > -- > I am https://leavingnorway.info > Videos at https://www.youtube.com/user/TheBlogologue > Twittering at http://twitter.com/blogologue > Blogging at http://blogologue.com > Playing music at https://soundcloud.com/morten-w-petersen > Also playing music and podcasting here: > http://www.mixcloud.com/morten-w-petersen/ > On Google+ here https://plus.google.com/107781930037068750156 > On Instagram at https://instagram.com/morphexx/ > -- > https://mail.python.org/mailman/listinfo/python-list > From morphex at gmail.com Thu Jul 28 05:14:42 2022 From: morphex at gmail.com (Morten W. Petersen) Date: Thu, 28 Jul 2022 11:14:42 +0200 Subject: Simple TCP proxy In-Reply-To: References: Message-ID: OK, I'll have a look at using something else than _threading. I quickly saw a couple of points where code could be optimized for speed, the loop that transfers data back and forth also has low throughput, but first priority was getting it working and seeing that it is fairly stable. Regards, Morten -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Instagram at https://instagram.com/morphexx/ On Wed, Jul 27, 2022 at 9:57 PM Chris Angelico wrote: > On Thu, 28 Jul 2022 at 04:32, Morten W. Petersen > wrote: > > > > Hi Chris. > > > > You're thinking of the backlog argument of listen? > > Yes, precisely. > > > Well, STP will accept all connections, but can limit how many of the > accepted connections that are active at any given time. > > > > So when I bombed it with hundreds of almost simultaneous connections, > all of them were accepted, but only 25 were actively sending and receiving > data at any given time. First come, first served. > > > > Hmm. Okay. Not sure what the advantage is, but sure. > > If the server's capable of handling the total requests-per-minute, > then a queueing system like this should help with burst load, although > I would have thought that the listen backlog would do the same. What > happens if the server actually gets overloaded though? Do connections > get disconnected after appearing connected? What's the disconnect > mode? > > BTW, you probably don't want to be using the _thread module - Python > has a threading module which is better suited to this sort of work. > Although you may want to consider asyncio instead, as that has far > lower overhead when working with large numbers of sockets. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ From morphex at gmail.com Thu Jul 28 05:31:27 2022 From: morphex at gmail.com (Morten W. Petersen) Date: Thu, 28 Jul 2022 11:31:27 +0200 Subject: Simple TCP proxy In-Reply-To: References: Message-ID: Hi Barry. Well, I can agree that using backlog is an option for handling bursts. But what if that backlog number is exceeded? How easy is it to deal with such a situation? I just cloned twisted, and compared the size: morphex at morphex-Latitude-E4310:~$ du -s stp; du -s tmp/twisted/ 464 stp 98520 tmp/twisted/ morphex at morphex-Latitude-E4310:~$ du -sh stp/LICENSE 36K stp/LICENSE >>> 464/98520.0 0.004709703613479496 >>> It's quite easy to get an idea of what's going on in STP, as opposed to if something goes wrong in Twisted with the size of the codebase. I used to use emacs a lot, but then I came into a period where it was more practical to use nano, and I mostly use nano now, unless I need to for example search and replace or something like that. -Morten On Thu, Jul 28, 2022 at 8:31 AM Barry wrote: > > > > On 27 Jul 2022, at 17:16, Morten W. Petersen wrote: > > > > ?Hi. > > > > I'd like to share with you a recent project, which is a simple TCP proxy > > that can stand in front of a TCP server of some sort, queueing requests > and > > then allowing n number of connections to pass through at a time: > > > > https://github.com/morphex/stp > > > > I'll be developing it further, but the the files committed in this tree > > seem to be stable: > > > > > https://github.com/morphex/stp/tree/9910ca8c80e9d150222b680a4967e53f0457b465 > > > > I just bombed that code with 700+ requests almost simultaneously, and STP > > handled it well. > > What is the problem that this solves? > > Why not just increase the allowed size of the socket listen backlog if you > just want to handle bursts of traffic. > > I do not think of this as a proxy, rather a tunnel. > And the tunnel is a lot more expensive the having kernel keep the > connection in > the listen socket backlog. > > I work on a web proxy written on python that handles huge load and > using backlog of the bursts. > > It?s async using twisted as threads are not practice at scale. > > Barry > > > > > Regards, > > > > Morten > > > > -- > > I am https://leavingnorway.info > > Videos at https://www.youtube.com/user/TheBlogologue > > Twittering at http://twitter.com/blogologue > > Blogging at http://blogologue.com > > Playing music at https://soundcloud.com/morten-w-petersen > > Also playing music and podcasting here: > > http://www.mixcloud.com/morten-w-petersen/ > > On Google+ here https://plus.google.com/107781930037068750156 > > On Instagram at https://instagram.com/morphexx/ > > -- > > https://mail.python.org/mailman/listinfo/python-list > > > > -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ From morphex at gmail.com Thu Jul 28 05:39:42 2022 From: morphex at gmail.com (Morten W. Petersen) Date: Thu, 28 Jul 2022 11:39:42 +0200 Subject: Simple TCP proxy In-Reply-To: <20220727205810.55ivclftzsggrenx@gmail.com> References: <20220727205810.55ivclftzsggrenx@gmail.com> Message-ID: Hi Martin. I was thinking of doing something with the handle function, but just this little tweak: https://github.com/morphex/stp/commit/9910ca8c80e9d150222b680a4967e53f0457b465 made a huge difference in CPU usage. Hundreds of waiting sockets are now using 20-30% of CPU instead of 10x that. So for example making the handle function exit / stop and wait isn't necessary at this point. It also opens up the possibility of sending a noop that is appropriate for the given protocol. I've not done a lot of thread programming before, but yes, locks can be used and will be used if necessary. I wasn't sure what data types were thread safe in Python, and it might be that some variables could be off by 1 or more, if using <= >= checks is an option and that there is no risk of the variable containing "garbage". I think with a simple focus, that the project is aimed at one task, will make it easier to manage even complex matters such as concurrency and threads. -Morten On Wed, Jul 27, 2022 at 11:00 PM Martin Di Paola wrote: > > On Wed, Jul 27, 2022 at 08:32:31PM +0200, Morten W. Petersen wrote: > >You're thinking of the backlog argument of listen? > > From my understanding, yes, when you set up the "accepter" socket (the > one that you use to listen and accept new connections), you can define > the length of the queue for incoming connections that are not accepted > yet. > > This will be the equivalent of your SimpleQueue which basically puts a > limits on how many incoming connections are "accepted" to do a real job. > > Using skt.listen(N) the incoming connections are put on hold by the OS > while in your implementation are formally accepted but they are not > allowed to do any meaningful work: they are put on the SimpleQueue and > only when they are popped then they will work (send/recv data). > > The difference then between the OS and your impl is minimal. The only > case that I can think is that on the clients' side it may exist a > timeout for the acceptance of the connection so your proxy server will > eagerly accept these connections so no timeout is possible(*) > > On a side note, you implementation is too thread-naive: it uses plain > Python lists, integers and boolean variables which are not thread safe. > It is a matter of time until your server will start behave weird. > > One option is that you use thread-safe objects. I'll encourage to read > about thread-safety in general and then which sync mechanisms Python > offers. > > Another option is to remove the SimpleQueue and the background function > that allows a connection to be "active". > > If you think, the handlers are 99% independent except that you want to > allow only N of them to progress (stablish and forward the connection) > and when a handler finishes, another handler "waiting" is activated, "in > a queue fashion" as you said. > > If you allow me to not have a strict queue discipline here, you can achieve > the same results coordinating the handlers using semaphores. Once again, > take this email as starting point for your own research. > > On a second side note, the use of handlers and threads is inefficient > because while you have N active handlers sending/receiving data, because > you are eagerly accepting new connections you will have much more > handlers created and (if I'm not wrong), each will be a thread. > > A more efficient solution could be > > 1) accept as many connections as you can, saving the socket (not the > handler) in the thread-safe queue. > 2) have N threads in the background popping from the queue a socket and > then doing the send/recv stuff. When the thread is done, the thread > closes the socket and pops another from the queue. > > So the queue length will be the count of accepted connections but in any > moment your proxy will not activate (forward) more than N connections. > > This idea is thread-safe, simpler, efficient and has the queue > discipline (I leave aside the usefulness). > > I encourage you to take time to read about the different things > mentioned as concurrency and thread-related stuff is not easy to > master. > > Thanks, > Martin. > > (*) make your proxy server slow enough and yes, you will get timeouts > anyways. > > > > >Well, STP will accept all connections, but can limit how many of the > >accepted connections that are active at any given time. > > > >So when I bombed it with hundreds of almost simultaneous connections, all > >of them were accepted, but only 25 were actively sending and receiving > data > >at any given time. First come, first served. > > > >Regards, > > > >Morten > > > >On Wed, Jul 27, 2022 at 8:00 PM Chris Angelico wrote: > > > >> On Thu, 28 Jul 2022 at 02:15, Morten W. Petersen > >> wrote: > >> > > >> > Hi. > >> > > >> > I'd like to share with you a recent project, which is a simple TCP > proxy > >> > that can stand in front of a TCP server of some sort, queueing > requests > >> and > >> > then allowing n number of connections to pass through at a time: > >> > >> How's this different from what the networking subsystem already does? > >> When you listen, you can set a queue length. Can you elaborate? > >> > >> ChrisA > >> -- > >> https://mail.python.org/mailman/listinfo/python-list > >> > > > > > >-- > >I am https://leavingnorway.info > >Videos at https://www.youtube.com/user/TheBlogologue > >Twittering at http://twitter.com/blogologue > >Blogging at http://blogologue.com > >Playing music at https://soundcloud.com/morten-w-petersen > >Also playing music and podcasting here: > >http://www.mixcloud.com/morten-w-petersen/ > >On Google+ here https://plus.google.com/107781930037068750156 > >On Instagram at https://instagram.com/morphexx/ > >-- > > > >I am https://leavingnorway.info > > > >Videos at https://www.youtube.com/user/TheBlogologue > >Twittering at http://twitter.com/blogologue > > > >Blogging at http://blogologue.com > >Playing music at https://soundcloud.com/morten-w-petersen > > > >Also playing music and podcasting here: > >http://www.mixcloud.com/morten-w-petersen/ > > > >On Instagram at https://instagram.com/morphexx/ > >-- > >https://mail.python.org/mailman/listinfo/python-list > -- > https://mail.python.org/mailman/listinfo/python-list > -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ From rosuav at gmail.com Thu Jul 28 05:43:02 2022 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 28 Jul 2022 19:43:02 +1000 Subject: Simple TCP proxy In-Reply-To: References: <20220727205810.55ivclftzsggrenx@gmail.com> Message-ID: On Thu, 28 Jul 2022 at 19:41, Morten W. Petersen wrote: > > Hi Martin. > > I was thinking of doing something with the handle function, but just this > little tweak: > > https://github.com/morphex/stp/commit/9910ca8c80e9d150222b680a4967e53f0457b465 > > made a huge difference in CPU usage. Hundreds of waiting sockets are now > using 20-30% of CPU instead of 10x that. .... wait, what? Why do waiting sockets consume *any* measurable amount of CPU? Why don't the threads simply block until it's time to do something? ChrisA From morphex at gmail.com Thu Jul 28 07:01:09 2022 From: morphex at gmail.com (Morten W. Petersen) Date: Thu, 28 Jul 2022 13:01:09 +0200 Subject: Simple TCP proxy In-Reply-To: References: <20220727205810.55ivclftzsggrenx@gmail.com> Message-ID: Well, I was thinking of following the socketserver / handle layout of code and execution, for now anyway. It wouldn't be a big deal to make them block, but another option is to increase the sleep period 100% for every 200 waiting connections while waiting in handle. Another thing is that it's nice to see Python handling 500+ threads without problems. :) -Morten On Thu, Jul 28, 2022 at 11:45 AM Chris Angelico wrote: > On Thu, 28 Jul 2022 at 19:41, Morten W. Petersen > wrote: > > > > Hi Martin. > > > > I was thinking of doing something with the handle function, but just this > > little tweak: > > > > > https://github.com/morphex/stp/commit/9910ca8c80e9d150222b680a4967e53f0457b465 > > > > made a huge difference in CPU usage. Hundreds of waiting sockets are now > > using 20-30% of CPU instead of 10x that. > > .... wait, what? > > Why do waiting sockets consume *any* measurable amount of CPU? Why > don't the threads simply block until it's time to do something? > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ From rosuav at gmail.com Thu Jul 28 08:29:18 2022 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 28 Jul 2022 22:29:18 +1000 Subject: Simple TCP proxy In-Reply-To: References: <20220727205810.55ivclftzsggrenx@gmail.com> Message-ID: On Thu, 28 Jul 2022 at 21:01, Morten W. Petersen wrote: > > Well, I was thinking of following the socketserver / handle layout of code and execution, for now anyway. > > It wouldn't be a big deal to make them block, but another option is to increase the sleep period 100% for every 200 waiting connections while waiting in handle. Easy denial-of-service attack then. Spam connections and the queue starts blocking hard. The sleep loop seems like a rather inefficient way to do things. > Another thing is that it's nice to see Python handling 500+ threads without problems. :) Yeah, well, that's not all THAT many threads, ultimately :) ChrisA From barry at barrys-emacs.org Thu Jul 28 08:29:39 2022 From: barry at barrys-emacs.org (Barry) Date: Thu, 28 Jul 2022 13:29:39 +0100 Subject: Simple TCP proxy In-Reply-To: References: Message-ID: <47C575B0-1191-4B81-B146-247B7320DBE6@barrys-emacs.org> > On 28 Jul 2022, at 10:31, Morten W. Petersen wrote: > > ? > Hi Barry. > > Well, I can agree that using backlog is an option for handling bursts. But what if that backlog number is exceeded? How easy is it to deal with such a situation? You can make backlog very large, if that makes sense. But at some point you will be forced to reject connections, once you cannot keep up with the average rate of connections. > > I just cloned twisted, and compared the size: > > morphex at morphex-Latitude-E4310:~$ du -s stp; du -s tmp/twisted/ > 464 stp > 98520 tmp/twisted/ > morphex at morphex-Latitude-E4310:~$ du -sh stp/LICENSE > 36K stp/LICENSE > > >>> 464/98520.0 > 0.004709703613479496 > >>> > > It's quite easy to get an idea of what's going on in STP, as opposed to if something goes wrong in Twisted with the size of the codebase. I used to use emacs a lot, but then I came into a period where it was more practical to use nano, and I mostly use nano now, unless I need to for example search and replace or something like that. I mentioned twisted for context. Depending on yours need the built in python 3 async support may well be sufficient for you needs. Using threads is not scalable. In the places I code disk space of a few MiB is not an issue. Barry > > -Morten > >> On Thu, Jul 28, 2022 at 8:31 AM Barry wrote: >> >> >> > On 27 Jul 2022, at 17:16, Morten W. Petersen wrote: >> > >> > ?Hi. >> > >> > I'd like to share with you a recent project, which is a simple TCP proxy >> > that can stand in front of a TCP server of some sort, queueing requests and >> > then allowing n number of connections to pass through at a time: >> > >> > https://github.com/morphex/stp >> > >> > I'll be developing it further, but the the files committed in this tree >> > seem to be stable: >> > >> > https://github.com/morphex/stp/tree/9910ca8c80e9d150222b680a4967e53f0457b465 >> > >> > I just bombed that code with 700+ requests almost simultaneously, and STP >> > handled it well. >> >> What is the problem that this solves? >> >> Why not just increase the allowed size of the socket listen backlog if you just want to handle bursts of traffic. >> >> I do not think of this as a proxy, rather a tunnel. >> And the tunnel is a lot more expensive the having kernel keep the connection in >> the listen socket backlog. >> >> I work on a web proxy written on python that handles huge load and >> using backlog of the bursts. >> >> It?s async using twisted as threads are not practice at scale. >> >> Barry >> >> > >> > Regards, >> > >> > Morten >> > >> > -- >> > I am https://leavingnorway.info >> > Videos at https://www.youtube.com/user/TheBlogologue >> > Twittering at http://twitter.com/blogologue >> > Blogging at http://blogologue.com >> > Playing music at https://soundcloud.com/morten-w-petersen >> > Also playing music and podcasting here: >> > http://www.mixcloud.com/morten-w-petersen/ >> > On Google+ here https://plus.google.com/107781930037068750156 >> > On Instagram at https://instagram.com/morphexx/ >> > -- >> > https://mail.python.org/mailman/listinfo/python-list >> > >> > > > -- > I am https://leavingnorway.info > Videos at https://www.youtube.com/user/TheBlogologue > Twittering at http://twitter.com/blogologue > Blogging at http://blogologue.com > Playing music at https://soundcloud.com/morten-w-petersen > Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ > On Google+ here https://plus.google.com/107781930037068750156 > On Instagram at https://instagram.com/morphexx/ From iamgusi at gmail.com Thu Jul 28 05:54:54 2022 From: iamgusi at gmail.com (Ben Hirsig) Date: Thu, 28 Jul 2022 19:54:54 +1000 Subject: Fwd: timedelta object recursion bug Message-ID: Hi, I noticed this when using the requests library in the response.elapsed object (type timedelta). Tested using the standard datetime library alone with the example displayed on https://docs.python.org/3/library/datetime.html#examples-of-usage-timedelta It appears as though the timedelta object recursively adds its own attributes (min, max, resolution) as further timedelta objects. I?m not sure how deep they go, but presumably hitting the recursion limit. >from datetime import timedelta >year = timedelta(days=365) >print(year.max) 999999999 days, 23:59:59.999999 >print(year.max.min.max.resolution.max.min) -999999999 days, 0:00:00 I?m using 3.10.3 Cheers From loris.bennett at fu-berlin.de Thu Jul 28 09:14:29 2022 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Thu, 28 Jul 2022 15:14:29 +0200 Subject: poetry script fails to find module Message-ID: <877d3xcrbe.fsf@hornfels.zedat.fu-berlin.de> Hi, The following is a little bit involved, but I hope can make the problem clear. Using poetry I have written a dummy application which just uses to typer to illustrate a possible interface design. The directory structure is a follows: $ tree -P *.py . |-- dist |-- stoat | |-- hpc | | |-- database.py | | |-- group.py | | |-- __init__.py | | |-- main.py | | |-- owner.py | | `-- user.py | |-- __init__.py | |-- main.py | `-- storage | |-- database.py | |-- group.py | |-- __init__.py | |-- main.py | |-- owner.py | |-- share.py | `-- user.py `-- tests |-- __init__.py `-- test_stoat.py With in the poetry shell I can run the application successfully: $ python stoat/main.py hpc user --help Usage: main.py hpc user [OPTIONS] COMMAND [ARGS]... manage HPC users Options: --help Show this message and exit. Commands: add add a user remove remove a user I then install this in a non-standard path (because the OS Python3 is 3.6.8) and can run the installed version successfully: $ PYTHONPATH=/trinity/shared/zedat/lib/python3.9/site-packages python /trinity/shared/zedat/lib/python3.9/site-packages/stoat/main.py hpc user --help Usage: main.py hpc user [OPTIONS] COMMAND [ARGS]... manage HPC users Options: --help Show this message and exit. Commands: add add a user remove remove a user However, poetry creates a script 'stoat' from the entry [tool.poetry.scripts] stoat = "stoat.main:main" in pyproject.toml, which looks like #!/trinity/shared/easybuild/software/Python/3.9.6-GCCcore-11.2.0/bin/python3.9 # -*- coding: utf-8 -*- import re import sys from stoat.main import main if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) sys.exit(main()) If I run that I get $ PYTHONPATH=/trinity/shared/zedat/lib/python3.9/site-packages stoat hpc user --help Traceback (most recent call last): File "/trinity/shared/zedat/bin/stoat", line 5, in from stoat.main import main File "/trinity/shared/zedat/lib/python3.9/site-packages/stoat/main.py", line 3, in import hpc.main ModuleNotFoundError: No module named 'hpc' Why is the module 'hpc' not found by the poetry script? Cheers, Loris -- This signature is currently under construction. From loris.bennett at fu-berlin.de Thu Jul 28 10:14:37 2022 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Thu, 28 Jul 2022 16:14:37 +0200 Subject: poetry script fails to find module References: <877d3xcrbe.fsf@hornfels.zedat.fu-berlin.de> Message-ID: <87v8rhb9yq.fsf@hornfels.zedat.fu-berlin.de> "Loris Bennett" writes: > Hi, > > The following is a little bit involved, but I hope can make the problem clear. > > Using poetry I have written a dummy application which just uses to typer > to illustrate a possible interface design. The directory structure is a > follows: > > $ tree -P *.py > . > |-- dist > |-- stoat > | |-- hpc > | | |-- database.py > | | |-- group.py > | | |-- __init__.py > | | |-- main.py > | | |-- owner.py > | | `-- user.py > | |-- __init__.py > | |-- main.py > | `-- storage > | |-- database.py > | |-- group.py > | |-- __init__.py > | |-- main.py > | |-- owner.py > | |-- share.py > | `-- user.py > `-- tests > |-- __init__.py > `-- test_stoat.py > > With in the poetry shell I can run the application successfully: > > $ python stoat/main.py hpc user --help > Usage: main.py hpc user [OPTIONS] COMMAND [ARGS]... > > manage HPC users > > Options: > --help Show this message and exit. > > Commands: > add add a user > remove remove a user > > I then install this in a non-standard path (because the OS Python3 is > 3.6.8) and can run the installed version successfully: > > $ PYTHONPATH=/trinity/shared/zedat/lib/python3.9/site-packages python /trinity/shared/zedat/lib/python3.9/site-packages/stoat/main.py hpc user --help > Usage: main.py hpc user [OPTIONS] COMMAND [ARGS]... > > manage HPC users > > Options: > --help Show this message and exit. > > Commands: > add add a user > remove remove a user > > However, poetry creates a script 'stoat' from the entry > > [tool.poetry.scripts] > stoat = "stoat.main:main" > > in pyproject.toml, which looks like > > #!/trinity/shared/easybuild/software/Python/3.9.6-GCCcore-11.2.0/bin/python3.9 > # -*- coding: utf-8 -*- > import re > import sys > from stoat.main import main > if __name__ == '__main__': > sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) > sys.exit(main()) > > If I run that I get > > $ PYTHONPATH=/trinity/shared/zedat/lib/python3.9/site-packages stoat hpc user --help > Traceback (most recent call last): > File "/trinity/shared/zedat/bin/stoat", line 5, in > from stoat.main import main > File "/trinity/shared/zedat/lib/python3.9/site-packages/stoat/main.py", line 3, in > import hpc.main > ModuleNotFoundError: No module named 'hpc' > > Why is the module 'hpc' not found by the poetry script? Never mind, I worked it out. I had to replace import hpc.main with import stoat.hpc.main However, this raises the question of why it worked in the first place in the poetry shell. Cheers, Loris -- This signature is currently under construction. From python at mrabarnett.plus.com Thu Jul 28 12:18:01 2022 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 28 Jul 2022 17:18:01 +0100 Subject: Fwd: timedelta object recursion bug In-Reply-To: References: Message-ID: <897eed22-4325-f130-2687-edf7700aead2@mrabarnett.plus.com> On 28/07/2022 10:54, Ben Hirsig wrote: > Hi, I noticed this when using the requests library in the response.elapsed > object (type timedelta). Tested using the standard datetime library alone > with the example displayed on > https://docs.python.org/3/library/datetime.html#examples-of-usage-timedelta > > > > It appears as though the timedelta object recursively adds its own > attributes (min, max, resolution) as further timedelta objects. I?m not > sure how deep they go, but presumably hitting the recursion limit. > > > >>from datetime import timedelta > >>year = timedelta(days=365) > >>print(year.max) > > 999999999 days, 23:59:59.999999 > >>print(year.max.min.max.resolution.max.min) > > -999999999 days, 0:00:00 > > > > I?m using 3.10.3 > It's not recursion, it's a reference cycle. In fact, more than one: >>> from datetime import timedelta >>> year = timedelta(days=365) >>> type(year) >>> type(year.max) >>> year.max is year.max.max True >>> type(year.min) >>> year.min is year.min.min True From jon+usenet at unequivocal.eu Thu Jul 28 11:22:48 2022 From: jon+usenet at unequivocal.eu (Jon Ribbens) Date: Thu, 28 Jul 2022 15:22:48 -0000 (UTC) Subject: Fwd: timedelta object recursion bug References: Message-ID: On 2022-07-28, Ben Hirsig wrote: > Hi, I noticed this when using the requests library in the response.elapsed > object (type timedelta). Tested using the standard datetime library alone > with the example displayed on > https://docs.python.org/3/library/datetime.html#examples-of-usage-timedelta > > It appears as though the timedelta object recursively adds its own > attributes (min, max, resolution) as further timedelta objects. I?m not > sure how deep they go, but presumably hitting the recursion limit. > >>from datetime import timedelta >>year = timedelta(days=365) >>print(year.max) > 999999999 days, 23:59:59.999999 >>print(year.max.min.max.resolution.max.min) > -999999999 days, 0:00:00 Why do you think this is a bug? From dieter at handshake.de Thu Jul 28 13:54:09 2022 From: dieter at handshake.de (Dieter Maurer) Date: Thu, 28 Jul 2022 19:54:09 +0200 Subject: Fwd: timedelta object recursion bug In-Reply-To: References: Message-ID: <25314.52545.946937.264554@ixdm.fritz.box> Ben Hirsig wrote at 2022-7-28 19:54 +1000: >Hi, I noticed this when using the requests library in the response.elapsed >object (type timedelta). Tested using the standard datetime library alone >with the example displayed on >https://docs.python.org/3/library/datetime.html#examples-of-usage-timedelta > > > >It appears as though the timedelta object recursively adds its own >attributes (min, max, resolution) as further timedelta objects. I?m not >sure how deep they go, but presumably hitting the recursion limit. If you look at the source, you will see that `min`, `max`, `resolution` are class level attributes. Their values are `timedelta` instances. Therefore, you can access e.g. `timedelta(days=365).min.max.resolution`. But this is nothing to worry about. From dieter at handshake.de Thu Jul 28 17:21:16 2022 From: dieter at handshake.de (Dieter Maurer) Date: Thu, 28 Jul 2022 23:21:16 +0200 Subject: Fwd: timedelta object recursion bug In-Reply-To: References: <25314.52545.946937.264554@ixdm.fritz.box> Message-ID: <25314.64972.482381.610540@ixdm.fritz.box> Please stay on the list (such that others can help, too) Ben Hirsig wrote at 2022-7-29 06:53 +1000: >Thanks for the replies, I'm just trying to understand why this would be >useful? > >E.g. why does max need a min/max/resolution, and why would these attributes >themselves need a min/max/resolution, etc, etc? `max` is a `timedelta` and as such inherits (e.g.) `resolution` from the class (as any other `timedelta` instance). Note that `timedelta` instances do not have a `max` (`min|resolution`) slot. When `max` is looked up, it is first searched in the instance (and not found), then in the class where it is found: all `max` accesses result in the same object. From morphex at gmail.com Thu Jul 28 17:22:39 2022 From: morphex at gmail.com (Morten W. Petersen) Date: Thu, 28 Jul 2022 23:22:39 +0200 Subject: Fwd: Simple TCP proxy In-Reply-To: References: <20220727205810.55ivclftzsggrenx@gmail.com> Message-ID: Forwarding to the list as well. ---------- Forwarded message --------- From: Morten W. Petersen Date: Thu, Jul 28, 2022 at 11:22 PM Subject: Re: Simple TCP proxy To: Chris Angelico Well, an increase from 0.1 seconds to 0.2 seconds on "polling" in each thread whether or not the connection should become active doesn't seem like a big deal. And there's also some point where it is pointless to accept more connections, and where maybe remedies like accepting known good IPs, blocking IPs / IP blocks with more than 3 connections etc. should be considered. I think I'll be getting closer than most applications to an eventual ceiling for what Python can handle of threads, and that's interesting and could be beneficial for Python as well. -Morten On Thu, Jul 28, 2022 at 2:31 PM Chris Angelico wrote: > On Thu, 28 Jul 2022 at 21:01, Morten W. Petersen > wrote: > > > > Well, I was thinking of following the socketserver / handle layout of > code and execution, for now anyway. > > > > It wouldn't be a big deal to make them block, but another option is to > increase the sleep period 100% for every 200 waiting connections while > waiting in handle. > > Easy denial-of-service attack then. Spam connections and the queue > starts blocking hard. The sleep loop seems like a rather inefficient > way to do things. > > > Another thing is that it's nice to see Python handling 500+ threads > without problems. :) > > Yeah, well, that's not all THAT many threads, ultimately :) > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ From morphex at gmail.com Thu Jul 28 17:25:00 2022 From: morphex at gmail.com (Morten W. Petersen) Date: Thu, 28 Jul 2022 23:25:00 +0200 Subject: Simple TCP proxy In-Reply-To: <47C575B0-1191-4B81-B146-247B7320DBE6@barrys-emacs.org> References: <47C575B0-1191-4B81-B146-247B7320DBE6@barrys-emacs.org> Message-ID: Well, it's not just code size in terms of disk space, it is also code complexity, and the level of knowledge, skill and time it takes to make use of something. And if something fails in an unobvious way in Twisted, I imagine that requires somebody highly skilled, and that costs quite a bit of money. And people like that might also not always be available. -Morten On Thu, Jul 28, 2022 at 2:29 PM Barry wrote: > > > On 28 Jul 2022, at 10:31, Morten W. Petersen wrote: > > ? > Hi Barry. > > Well, I can agree that using backlog is an option for handling bursts. But > what if that backlog number is exceeded? How easy is it to deal with such > a situation? > > > You can make backlog very large, if that makes sense. > But at some point you will be forced to reject connections, > once you cannot keep up with the average rate of connections. > > > > I just cloned twisted, and compared the size: > > morphex at morphex-Latitude-E4310:~$ du -s stp; du -s tmp/twisted/ > 464 stp > 98520 tmp/twisted/ > morphex at morphex-Latitude-E4310:~$ du -sh stp/LICENSE > 36K stp/LICENSE > > >>> 464/98520.0 > 0.004709703613479496 > >>> > > It's quite easy to get an idea of what's going on in STP, as opposed to if > something goes wrong in Twisted with the size of the codebase. I used to > use emacs a lot, but then I came into a period where it was more practical > to use nano, and I mostly use nano now, unless I need to for example search > and replace or something like that. > > > I mentioned twisted for context. Depending on yours need the built in > python 3 async support may well be sufficient for you needs. Using threads > is not scalable. > > In the places I code disk space of a few MiB is not an issue. > > Barry > > > -Morten > > On Thu, Jul 28, 2022 at 8:31 AM Barry wrote: > >> >> >> > On 27 Jul 2022, at 17:16, Morten W. Petersen wrote: >> > >> > ?Hi. >> > >> > I'd like to share with you a recent project, which is a simple TCP proxy >> > that can stand in front of a TCP server of some sort, queueing requests >> and >> > then allowing n number of connections to pass through at a time: >> > >> > https://github.com/morphex/stp >> > >> > I'll be developing it further, but the the files committed in this tree >> > seem to be stable: >> > >> > >> https://github.com/morphex/stp/tree/9910ca8c80e9d150222b680a4967e53f0457b465 >> > >> > I just bombed that code with 700+ requests almost simultaneously, and >> STP >> > handled it well. >> >> What is the problem that this solves? >> >> Why not just increase the allowed size of the socket listen backlog if >> you just want to handle bursts of traffic. >> >> I do not think of this as a proxy, rather a tunnel. >> And the tunnel is a lot more expensive the having kernel keep the >> connection in >> the listen socket backlog. >> >> I work on a web proxy written on python that handles huge load and >> using backlog of the bursts. >> >> It?s async using twisted as threads are not practice at scale. >> >> Barry >> >> > >> > Regards, >> > >> > Morten >> > >> > -- >> > I am https://leavingnorway.info >> > Videos at https://www.youtube.com/user/TheBlogologue >> > Twittering at http://twitter.com/blogologue >> > Blogging at http://blogologue.com >> > Playing music at https://soundcloud.com/morten-w-petersen >> > Also playing music and podcasting here: >> > http://www.mixcloud.com/morten-w-petersen/ >> > On Google+ here https://plus.google.com/107781930037068750156 >> > On Instagram at https://instagram.com/morphexx/ >> > -- >> > https://mail.python.org/mailman/listinfo/python-list >> > >> >> > > -- > I am https://leavingnorway.info > Videos at https://www.youtube.com/user/TheBlogologue > Twittering at http://twitter.com/blogologue > Blogging at http://blogologue.com > Playing music at https://soundcloud.com/morten-w-petersen > Also playing music and podcasting here: > http://www.mixcloud.com/morten-w-petersen/ > On Google+ here https://plus.google.com/107781930037068750156 > On Instagram at https://instagram.com/morphexx/ > > -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ From rosuav at gmail.com Thu Jul 28 18:08:33 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 29 Jul 2022 08:08:33 +1000 Subject: Simple TCP proxy In-Reply-To: References: <20220727205810.55ivclftzsggrenx@gmail.com> Message-ID: On Fri, 29 Jul 2022 at 07:24, Morten W. Petersen wrote: > > Forwarding to the list as well. > > ---------- Forwarded message --------- > From: Morten W. Petersen > Date: Thu, Jul 28, 2022 at 11:22 PM > Subject: Re: Simple TCP proxy > To: Chris Angelico > > > Well, an increase from 0.1 seconds to 0.2 seconds on "polling" in each > thread whether or not the connection should become active doesn't seem like > a big deal. Maybe, but polling *at all* is the problem here. It shouldn't be hammering the other server. You'll quickly find that there are limits that simply shouldn't exist, because every connection is trying to check to see if it's active now. This is *completely unnecessary*. I'll reiterate the advice given earlier in this thread (of conversation): Look into the tools available for thread (of execution) synchronization, such as mutexes (in Python, threading.Lock) and events (in Python, threading.Condition). A poll interval enforces a delay before the thread notices that it's active, AND causes inactive threads to consume CPU, neither of which is a good thing. > And there's also some point where it is pointless to accept more > connections, and where maybe remedies like accepting known good IPs, > blocking IPs / IP blocks with more than 3 connections etc. should be > considered. Firewalling is its own science. Blocking IPs with too many simultaneous connections should be decided administratively, not because your proxy can't handle enough connections. > I think I'll be getting closer than most applications to an eventual > ceiling for what Python can handle of threads, and that's interesting and > could be beneficial for Python as well. Here's a quick demo of the cost of threads when they're all blocked on something. >>> import threading >>> finish = threading.Condition() >>> def thrd(cond): ... with cond: cond.wait() ... >>> threading.active_count() # Main thread only 1 >>> import time >>> def spawn(n): ... start = time.monotonic() ... for _ in range(n): ... t = threading.Thread(target=thrd, args=(finish,)) ... t.start() ... print("Spawned", n, "threads in", time.monotonic() - start, "seconds") ... >>> spawn(10000) Spawned 10000 threads in 7.548425202025101 seconds >>> threading.active_count() 10001 >>> with finish: finish.notify_all() ... >>> threading.active_count() 1 It takes a bit of time to start ten thousand threads, but after that, the system is completely idle again until I notify them all and they shut down. (Interestingly, it takes four times as long to start 20,000 threads, suggesting that something in thread spawning has O(n?) cost. Still, even that leaves the system completely idle once it's done spawning them.) If your proxy can handle 20,000 threads, I would be astonished. And this isn't even close to a thread limit. Obviously the cost is different if the threads are all doing things, but if you have thousands of active socket connections, you'll start finding that there are limitations in quite a few places, depending on how much traffic is going through them. Ultimately, yes, you will find that threads restrict you and asynchronous I/O is the only option; but you can take threads a fairly long way before they are the limiting factor. ChrisA From andymac at pcug.org.au Thu Jul 28 20:13:36 2022 From: andymac at pcug.org.au (Andrew MacIntyre) Date: Fri, 29 Jul 2022 10:13:36 +1000 Subject: Simple TCP proxy In-Reply-To: References: <20220727205810.55ivclftzsggrenx@gmail.com> Message-ID: <6cc646eb-6550-25c9-229c-e109faa84102@pcug.org.au> On 29/07/2022 8:08 am, Chris Angelico wrote: > It takes a bit of time to start ten thousand threads, but after that, > the system is completely idle again until I notify them all and they > shut down. > > (Interestingly, it takes four times as long to start 20,000 threads, > suggesting that something in thread spawning has O(n?) cost. Still, > even that leaves the system completely idle once it's done spawning > them.) Another cost of threads can be memory allocated as thread stack space, the default size of which varies by OS (see e.g. https://ariadne.space/2021/06/25/understanding-thread-stack-sizes-and-how-alpine-is-different/). threading.stack_size() can be used to check and perhaps adjust the allocation size. -- ------------------------------------------------------------------------- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac at pcug.org.au (pref) | Snail: PO Box 370 andymac at bullseye.apana.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From rosuav at gmail.com Thu Jul 28 21:48:14 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 29 Jul 2022 11:48:14 +1000 Subject: Simple TCP proxy In-Reply-To: <6cc646eb-6550-25c9-229c-e109faa84102@pcug.org.au> References: <20220727205810.55ivclftzsggrenx@gmail.com> <6cc646eb-6550-25c9-229c-e109faa84102@pcug.org.au> Message-ID: On Fri, 29 Jul 2022 at 11:42, Andrew MacIntyre wrote: > > On 29/07/2022 8:08 am, Chris Angelico wrote: > > It takes a bit of time to start ten thousand threads, but after that, > > the system is completely idle again until I notify them all and they > > shut down. > > > > (Interestingly, it takes four times as long to start 20,000 threads, > > suggesting that something in thread spawning has O(n?) cost. Still, > > even that leaves the system completely idle once it's done spawning > > them.) > > Another cost of threads can be memory allocated as thread stack space, > the default size of which varies by OS (see e.g. > https://ariadne.space/2021/06/25/understanding-thread-stack-sizes-and-how-alpine-is-different/). > > threading.stack_size() can be used to check and perhaps adjust the > allocation size. > Yeah, they do have quite a few costs, and a naive approach of "give a thread to every client", while very convenient, will end up limiting throughput. (But I'll be honest: I still have a server that's built on exactly that model, because it's much much safer than risking one client stalling out the whole server due to a small bug. But that's a MUD server.) Thing is, though, it'll most likely limit throughput to something in the order of thousands of concurrent connections (or thousands per second if it's something like HTTP where they tend to get closed again), maybe tens of thousands. So if you have something where every thread needs its own database connection, well, you're gonna have database throughput problems WAY before you actually run into thread count limitations! ChrisA From loris.bennett at fu-berlin.de Fri Jul 29 02:22:13 2022 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Fri, 29 Jul 2022 08:22:13 +0200 Subject: poetry script fails to find module References: <877d3xcrbe.fsf@hornfels.zedat.fu-berlin.de> <87v8rhb9yq.fsf@hornfels.zedat.fu-berlin.de> Message-ID: <87r1241lre.fsf@hornfels.zedat.fu-berlin.de> ram at zedat.fu-berlin.de (Stefan Ram) writes: > "Loris Bennett" writes: >>However, this raises the question of why it worked in the first place >>in the poetry shell. > > It might have had a different or extended sys.path. In the poetry shell sys.path has this additional path /home/loris/gitlab/stoat The module not being found was /home/gitlab/stoat/stoat/hpc/main.py But if I run [~/gitlab/stoat] $ python stoat/main.py hpc user --help wouldn't import hpc.main still fail? Or is it because I am calling stoat/main.py and so Python looks for 'hpc' relative to the second 'stoat' directory? Confused, loris -- This signature is currently under construction. From bursejan at gmail.com Fri Jul 29 03:53:09 2022 From: bursejan at gmail.com (Mostowski Collapse) Date: Fri, 29 Jul 2022 00:53:09 -0700 (PDT) Subject: ANN: Dogelog Player 1.0.0 In-Reply-To: <7b91484c-ebff-4015-9723-f1ee7e201fb3n@googlegroups.com> References: <3332dc1d-f8f7-4853-8d34-9912ad530825n@googlegroups.com> <7e4c3f5f-c708-40b3-a190-8bda5b54a615n@googlegroups.com> <7b91484c-ebff-4015-9723-f1ee7e201fb3n@googlegroups.com> Message-ID: <20eed721-9065-4070-9b81-54815e9de4a6n@googlegroups.com> Sofar we have resisted in adding a Next or Abort button to any of our Dogelog Player HTML pages. Behind the lack of these buttons is a method of exploring sequence meta-predicates and what they could do for user interfaces. Here is a little new article about the topic: Enumerating Solutions with Dogelog Payer https://twitter.com/dogelogch/status/1552802997247303680 Enumerating Solutions with Dogelog Payer https://www.facebook.com/groups/dogelog From loris.bennett at fu-berlin.de Fri Jul 29 04:46:43 2022 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Fri, 29 Jul 2022 10:46:43 +0200 Subject: poetry script fails to find module References: <877d3xcrbe.fsf@hornfels.zedat.fu-berlin.de> Message-ID: <87edy4qpak.fsf@hornfels.zedat.fu-berlin.de> Hi Stefan, ram at zedat.fu-berlin.de (Stefan Ram) writes: > "Loris Bennett" writes: >>Why is the module 'hpc' not found by the poetry script? > > I have tried to execute the following sequence of shell > commands to understand your problem. Here they all worked > without error messages. Warning: Some of these commands might > alter your directories or data, so only execute them if you > are aware of their meaning/consequences and agree with them! I do know what the commands do. However, just to be on the safe side, but mainly in honour of a classic dad-joke, I changed 'stoat' to 'weasel'. > mkdir stoat > mkdir stoat/hpc > echo import hpc.main >stoat/main.py > echo >stoat/hpc/main.py > python3 stoat/main.py > cd stoat > python3 main.py I guess with the production version, the package stoat is visible in sys.path and thus the subpackages have to be referred to via the main package, e.g. import stoat.hpc.main However, in the development environment, if I run python stoat/main.py hpc user --help then is stoat/hpc/main.py being found via import hpc.main because Python looks in stoat as the parent directory of stoat/main.py rather than the current working directory? That doesn't seem likely to me, but I am already confused. Cheers, Loris -- This signature is currently under construction. From loris.bennett at fu-berlin.de Fri Jul 29 05:32:12 2022 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Fri, 29 Jul 2022 11:32:12 +0200 Subject: poetry script fails to find module References: <877d3xcrbe.fsf@hornfels.zedat.fu-berlin.de> <87edy4qpak.fsf@hornfels.zedat.fu-berlin.de> Message-ID: <87a68sqn6r.fsf@hornfels.zedat.fu-berlin.de> ram at zedat.fu-berlin.de (Stefan Ram) writes: > "Loris Bennett" writes: >>However, in the development environment, if I run >> python stoat/main.py hpc user --help >>then is >> stoat/hpc/main.py >>being found via >> import hpc.main >>because Python looks in >> stoat >>as the parent directory of >> stoat/main.py >>rather than the current working directory? > > When you run "stoat/main.py", the directory of that file > "main.py" is automatically added at the front of sys.path. > (This also would happen after "cd stoat; python main.py".) OK, that explains it. I initially found that a bit odd, but thinking about it I see that "the directory containing the file being run" is a more sensible reference point than the current working directory, which is totally arbitrary. > Then, when "import hpc.main" is being executed, the system > will search for "hpc/main.py" in every entry of sys.path > and will use the first entry wherein it finds "hpc/main.py". > So it will use the directory of the file "stoat/main.py", > i.e., the directory "stoat". It finds "stoat/hpc/main.py". > > You can call "python" with "-v", and it will show some lines > with information about the imports executed, including the > directories used (albeit hidden in a lot of other lines). That's useful, although in the production case I would have to tweak the script generated by poetry. Cheers, Loris -- This signature is currently under construction. From Marco.Sulla.Python at gmail.com Fri Jul 29 14:29:40 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Fri, 29 Jul 2022 20:29:40 +0200 Subject: How to generate a .pyi file for a C Extension using stubgen Message-ID: I tried to follow the instructions here: https://mypy.readthedocs.io/en/stable/stubgen.html but the instructions about creating a stub for a C Extension are a little mysterious. I tried to use it on the .so file without luck. From morphex at gmail.com Fri Jul 29 14:46:11 2022 From: morphex at gmail.com (Morten W. Petersen) Date: Fri, 29 Jul 2022 20:46:11 +0200 Subject: Simple TCP proxy In-Reply-To: <6cc646eb-6550-25c9-229c-e109faa84102@pcug.org.au> References: <20220727205810.55ivclftzsggrenx@gmail.com> <6cc646eb-6550-25c9-229c-e109faa84102@pcug.org.au> Message-ID: OK, that's useful to know. Thanks. :) -Morten On Fri, Jul 29, 2022 at 3:43 AM Andrew MacIntyre wrote: > On 29/07/2022 8:08 am, Chris Angelico wrote: > > It takes a bit of time to start ten thousand threads, but after that, > > the system is completely idle again until I notify them all and they > > shut down. > > > > (Interestingly, it takes four times as long to start 20,000 threads, > > suggesting that something in thread spawning has O(n?) cost. Still, > > even that leaves the system completely idle once it's done spawning > > them.) > > Another cost of threads can be memory allocated as thread stack space, > the default size of which varies by OS (see e.g. > > https://ariadne.space/2021/06/25/understanding-thread-stack-sizes-and-how-alpine-is-different/ > ). > > threading.stack_size() can be used to check and perhaps adjust the > allocation size. > > -- > ------------------------------------------------------------------------- > Andrew I MacIntyre "These thoughts are mine alone..." > E-mail: andymac at pcug.org.au (pref) | Snail: PO Box 370 > andymac at bullseye.apana.org.au (alt) | Belconnen ACT 2616 > Web: http://www.andymac.org/ | Australia > -- > https://mail.python.org/mailman/listinfo/python-list > -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ From morphex at gmail.com Fri Jul 29 14:54:02 2022 From: morphex at gmail.com (Morten W. Petersen) Date: Fri, 29 Jul 2022 20:54:02 +0200 Subject: Simple TCP proxy In-Reply-To: References: <20220727205810.55ivclftzsggrenx@gmail.com> Message-ID: OK. Well, I've worked with web hosting in the past, and proxies like squid were used to lessen the load on dynamic backends. There was also a website opensourcearticles.com that we had with Firefox, Thunderbird articles etc. that got quite a bit of traffic. IIRC, that website was mostly static with some dynamic bits and heavily cached by squid. Most websites don't get a lot of traffic though, and don't have a big budget for "website system administration". So maybe that's where I'm partly going with this, just making a proxy that can be put in front and deal with a lot of common situations, in a reasonably good way. If I run into problems with threads that can't be managed, then a switch to something like the queue_manager function which has data and then functions that manage the data and connections is an option. -Morten On Fri, Jul 29, 2022 at 12:11 AM Chris Angelico wrote: > On Fri, 29 Jul 2022 at 07:24, Morten W. Petersen > wrote: > > > > Forwarding to the list as well. > > > > ---------- Forwarded message --------- > > From: Morten W. Petersen > > Date: Thu, Jul 28, 2022 at 11:22 PM > > Subject: Re: Simple TCP proxy > > To: Chris Angelico > > > > > > Well, an increase from 0.1 seconds to 0.2 seconds on "polling" in each > > thread whether or not the connection should become active doesn't seem > like > > a big deal. > > Maybe, but polling *at all* is the problem here. It shouldn't be > hammering the other server. You'll quickly find that there are limits > that simply shouldn't exist, because every connection is trying to > check to see if it's active now. This is *completely unnecessary*. > I'll reiterate the advice given earlier in this thread (of > conversation): Look into the tools available for thread (of execution) > synchronization, such as mutexes (in Python, threading.Lock) and > events (in Python, threading.Condition). A poll interval enforces a > delay before the thread notices that it's active, AND causes inactive > threads to consume CPU, neither of which is a good thing. > > > And there's also some point where it is pointless to accept more > > connections, and where maybe remedies like accepting known good IPs, > > blocking IPs / IP blocks with more than 3 connections etc. should be > > considered. > > Firewalling is its own science. Blocking IPs with too many > simultaneous connections should be decided administratively, not > because your proxy can't handle enough connections. > > > I think I'll be getting closer than most applications to an eventual > > ceiling for what Python can handle of threads, and that's interesting and > > could be beneficial for Python as well. > > Here's a quick demo of the cost of threads when they're all blocked on > something. > > >>> import threading > >>> finish = threading.Condition() > >>> def thrd(cond): > ... with cond: cond.wait() > ... > >>> threading.active_count() # Main thread only > 1 > >>> import time > >>> def spawn(n): > ... start = time.monotonic() > ... for _ in range(n): > ... t = threading.Thread(target=thrd, args=(finish,)) > ... t.start() > ... print("Spawned", n, "threads in", time.monotonic() - start, > "seconds") > ... > >>> spawn(10000) > Spawned 10000 threads in 7.548425202025101 seconds > >>> threading.active_count() > 10001 > >>> with finish: finish.notify_all() > ... > >>> threading.active_count() > 1 > > It takes a bit of time to start ten thousand threads, but after that, > the system is completely idle again until I notify them all and they > shut down. > > (Interestingly, it takes four times as long to start 20,000 threads, > suggesting that something in thread spawning has O(n?) cost. Still, > even that leaves the system completely idle once it's done spawning > them.) > > If your proxy can handle 20,000 threads, I would be astonished. And > this isn't even close to a thread limit. > > Obviously the cost is different if the threads are all doing things, > but if you have thousands of active socket connections, you'll start > finding that there are limitations in quite a few places, depending on > how much traffic is going through them. Ultimately, yes, you will find > that threads restrict you and asynchronous I/O is the only option; but > you can take threads a fairly long way before they are the limiting > factor. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ From rosuav at gmail.com Fri Jul 29 16:43:36 2022 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 30 Jul 2022 06:43:36 +1000 Subject: Simple TCP proxy In-Reply-To: References: <20220727205810.55ivclftzsggrenx@gmail.com> Message-ID: On Sat, 30 Jul 2022 at 04:54, Morten W. Petersen wrote: > > OK. > > Well, I've worked with web hosting in the past, and proxies like squid were used to lessen the load on dynamic backends. There was also a website opensourcearticles.com that we had with Firefox, Thunderbird articles etc. that got quite a bit of traffic. > > IIRC, that website was mostly static with some dynamic bits and heavily cached by squid. Yep, and squid almost certainly won't have a thread for every incoming connection, spinning and waiting for the back end server. But squid does a LOT more than simply queue connections - it'll be inspecting headers and retaining a cache of static content, so it's not really comparable. > Most websites don't get a lot of traffic though, and don't have a big budget for "website system administration". So maybe that's where I'm partly going with this, just making a proxy that can be put in front and deal with a lot of common situations, in a reasonably good way. > > If I run into problems with threads that can't be managed, then a switch to something like the queue_manager function which has data and then functions that manage the data and connections is an option. > I'll be quite frank with you: this is not production-quality code. It should not be deployed by anyone who doesn't have a big budget for "website system administration *training*". This code is good as a tool for YOU to learn how these things work; it shouldn't be a tool for anyone who actually has server load issues. I'm sorry if that sounds harsh, but the fact is, you can do a lot better by using this to learn more about networking than you'll ever do by trying to pitch it to any specific company. That said though: it's still good to know what your (theoretical) use-case is. That'll tell you what kinds of connection spam to throw at your proxy (lots of idle sockets? lots of HTTP requests? billions of half open TCP connections?) to see what it can cope with. Keep on playing with this code. There's a lot you can gain from it, still. ChrisA From morphex at gmail.com Fri Jul 29 16:59:48 2022 From: morphex at gmail.com (Morten W. Petersen) Date: Fri, 29 Jul 2022 22:59:48 +0200 Subject: Simple TCP proxy In-Reply-To: References: <20220727205810.55ivclftzsggrenx@gmail.com> Message-ID: OK, sounds like sunshine is getting the best of you. It's working with a pretty heavy load, I see ways of solving potential problems that haven't become a problem yet, and I'm enjoying it. Maybe you should tone down the coaching until someone asks for it. Regards, Morten On Fri, Jul 29, 2022 at 10:46 PM Chris Angelico wrote: > On Sat, 30 Jul 2022 at 04:54, Morten W. Petersen > wrote: > > > > OK. > > > > Well, I've worked with web hosting in the past, and proxies like squid > were used to lessen the load on dynamic backends. There was also a website > opensourcearticles.com that we had with Firefox, Thunderbird articles > etc. that got quite a bit of traffic. > > > > IIRC, that website was mostly static with some dynamic bits and heavily > cached by squid. > > Yep, and squid almost certainly won't have a thread for every incoming > connection, spinning and waiting for the back end server. But squid > does a LOT more than simply queue connections - it'll be inspecting > headers and retaining a cache of static content, so it's not really > comparable. > > > Most websites don't get a lot of traffic though, and don't have a big > budget for "website system administration". So maybe that's where I'm > partly going with this, just making a proxy that can be put in front and > deal with a lot of common situations, in a reasonably good way. > > > > If I run into problems with threads that can't be managed, then a switch > to something like the queue_manager function which has data and then > functions that manage the data and connections is an option. > > > > I'll be quite frank with you: this is not production-quality code. It > should not be deployed by anyone who doesn't have a big budget for > "website system administration *training*". This code is good as a > tool for YOU to learn how these things work; it shouldn't be a tool > for anyone who actually has server load issues. > > I'm sorry if that sounds harsh, but the fact is, you can do a lot > better by using this to learn more about networking than you'll ever > do by trying to pitch it to any specific company. > > That said though: it's still good to know what your (theoretical) > use-case is. That'll tell you what kinds of connection spam to throw > at your proxy (lots of idle sockets? lots of HTTP requests? billions > of half open TCP connections?) to see what it can cope with. > > Keep on playing with this code. There's a lot you can gain from it, still. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ From barry at barrys-emacs.org Fri Jul 29 17:22:55 2022 From: barry at barrys-emacs.org (Barry) Date: Fri, 29 Jul 2022 22:22:55 +0100 Subject: How to generate a .pyi file for a C Extension using stubgen In-Reply-To: References: Message-ID: > On 29 Jul 2022, at 19:33, Marco Sulla wrote: > > ?I tried to follow the instructions here: > > https://mypy.readthedocs.io/en/stable/stubgen.html > > but the instructions about creating a stub for a C Extension are a little > mysterious. I tried to use it on the .so file without luck. It says that stubgen works on .py files not .so files. You will need to write the .pyi for your .so manually. The docs could do with splitting the need for .pyi for .so away from the stubgen description. Barry > -- > https://mail.python.org/mailman/listinfo/python-list > From pkpearson at nowhere.invalid Fri Jul 29 19:24:57 2022 From: pkpearson at nowhere.invalid (Peter Pearson) Date: 29 Jul 2022 23:24:57 GMT Subject: mailbox misbehavior with non-ASCII Message-ID: The following code produces a nonsense result with the input described below: import mailbox box = mailbox.Maildir("/home/peter/Temp/temp",create=False) x = box.values()[0] h = x.get("X-DSPAM-Factors") print(type(h)) # The output is the desired "str" when the message file contains this: To: recipient at example.com Message-ID: <123> Date: Sun, 24 Jul 2022 15:31:19 +0000 Subject: Blah blah From: from at from.com X-DSPAM-Factors: a'b xxx ... but if the apostrophe in "a'b" is replaced with a RIGHT SINGLE QUOTATION MARK, the returned h is of type "email.header.Header", and seems to contain inscrutable garbage. I realize that one should not put non-ASCII characters in message headers, but of course I didn't put it there, it just showed up, pretty much beyond my control. And I realize that when software is given input that breaks the rules, one cannot expect optimal results, but I'd think an exception would be the right answer. Is this worth a bug report? -- To email me, substitute nowhere->runbox, invalid->com. From ethan at stoneleaf.us Fri Jul 29 19:59:29 2022 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 29 Jul 2022 16:59:29 -0700 Subject: mailbox misbehavior with non-ASCII In-Reply-To: References: Message-ID: <517babef-ce59-a98b-c508-b47541ee0315@stoneleaf.us> On 7/29/22 16:24, Peter Pearson wrote: > ... but if the apostrophe in "a'b" is replaced with a > RIGHT SINGLE QUOTATION MARK, the returned h is of type > "email.header.Header", and seems to contain inscrutable garbage. > > I'd think an exception would be the right answer. > > Is this worth a bug report? I would say yes. -- ~Ethan~ From 2QdxY4RzWzUUiLuE at potatochowder.com Fri Jul 29 20:53:09 2022 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Fri, 29 Jul 2022 19:53:09 -0500 Subject: mailbox misbehavior with non-ASCII In-Reply-To: References: Message-ID: On 2022-07-29 at 23:24:57 +0000, Peter Pearson wrote: > The following code produces a nonsense result with the input > described below: > > import mailbox > box = mailbox.Maildir("/home/peter/Temp/temp",create=False) > x = box.values()[0] > h = x.get("X-DSPAM-Factors") > print(type(h)) > # > > The output is the desired "str" when the message file contains this: > > To: recipient at example.com > Message-ID: <123> > Date: Sun, 24 Jul 2022 15:31:19 +0000 > Subject: Blah blah > From: from at from.com > X-DSPAM-Factors: a'b > > xxx > > ... but if the apostrophe in "a'b" is replaced with a > RIGHT SINGLE QUOTATION MARK, the returned h is of type > "email.header.Header", and seems to contain inscrutable garbage. > > I realize that one should not put non-ASCII characters in > message headers, but of course I didn't put it there, it > just showed up, pretty much beyond my control. And I realize > that when software is given input that breaks the rules, one > cannot expect optimal results, but I'd think an exception > would be the right answer. Be strict in what you send, but generous is what you receive. I agree that email headers are supposed to be ASCII (RFC 822, 2822, and now 5322) all say that, but always throwing an exception seems a little harsh, and arguably (I'm not arguing for or against) breaks backwards compatibility. At least let the exception contain, in its own attribute, the inscrutable garbage after the space after the colon and before next CR/LF pair. > Is this worth a bug report? If nothing else, the documentation could specify or disclaim the existing behavior. From barry at barrys-emacs.org Sat Jul 30 03:55:24 2022 From: barry at barrys-emacs.org (Barry) Date: Sat, 30 Jul 2022 08:55:24 +0100 Subject: mailbox misbehavior with non-ASCII Message-ID: <052289F0-6220-4FB0-A6FC-FF5053991C09@barrys-emacs.org> ? > On 30 Jul 2022, at 00:30, Peter Pearson wrote: > > ?The following code produces a nonsense result with the input > described below: > > import mailbox > box = mailbox.Maildir("/home/peter/Temp/temp",create=False) > x = box.values()[0] > h = x.get("X-DSPAM-Factors") > print(type(h)) > # > > The output is the desired "str" when the message file contains this: > > To: recipient at example.com > Message-ID: <123> > Date: Sun, 24 Jul 2022 15:31:19 +0000 > Subject: Blah blah > From: from at from.com > X-DSPAM-Factors: a'b > > xxx > > ... but if the apostrophe in "a'b" is replaced with a > RIGHT SINGLE QUOTATION MARK, the returned h is of type > "email.header.Header", and seems to contain inscrutable garbage. Include in any bug report the exact bytes that are in the header. In may not be utf-8 encoded it maybe windows cp1252, etc. Repr of the bytes header will show this. Barry > > I realize that one should not put non-ASCII characters in > message headers, but of course I didn't put it there, it > just showed up, pretty much beyond my control. And I realize > that when software is given input that breaks the rules, one > cannot expect optimal results, but I'd think an exception > would be the right answer. > > Is this worth a bug report? > > -- > To email me, substitute nowhere->runbox, invalid->com. > -- > https://mail.python.org/mailman/listinfo/python-list From Marco.Sulla.Python at gmail.com Sat Jul 30 05:29:40 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Sat, 30 Jul 2022 11:29:40 +0200 Subject: How to generate a .pyi file for a C Extension using stubgen In-Reply-To: References: Message-ID: On Fri, 29 Jul 2022 at 23:23, Barry wrote: > > > > > On 29 Jul 2022, at 19:33, Marco Sulla wrote: > > > > ?I tried to follow the instructions here: > > > > https://mypy.readthedocs.io/en/stable/stubgen.html > > > > but the instructions about creating a stub for a C Extension are a little > > mysterious. I tried to use it on the .so file without luck. > > It says that stubgen works on .py files not .so files. > You will need to write the .pyi for your .so manually. > > The docs could do with splitting the need for .pyi for .so > away from the stubgen description. But it says: "Mypy includes the stubgen tool that can automatically generate stub files (.pyi files) for Python modules and C extension modules." I tried stubgen -m modulename, but it generates very little code. From roel at roelschroeven.net Sat Jul 30 06:56:56 2022 From: roel at roelschroeven.net (Roel Schroeven) Date: Sat, 30 Jul 2022 12:56:56 +0200 Subject: Simple TCP proxy In-Reply-To: References: <20220727205810.55ivclftzsggrenx@gmail.com> Message-ID: <9c2f1958-fbcc-bdab-7d78-c1282a18b1dc@roelschroeven.net> Morten W. Petersen schreef op 29/07/2022 om 22:59: > OK, sounds like sunshine is getting the best of you. It has to be said: that is uncalled for. Chris gave you good advice, with the best of intentions. Sometimes we don't like good advice if it says something we don't like, but that's no reason to take it off on the messenger. -- "Iceland is the place you go to remind yourself that planet Earth is a machine... and that all organic life that has ever existed amounts to a greasy film that has survived on the exterior of that machine thanks to furious improvisation." -- Sam Hughes, Ra From barry at barrys-emacs.org Sat Jul 30 08:19:10 2022 From: barry at barrys-emacs.org (Barry Scott) Date: Sat, 30 Jul 2022 13:19:10 +0100 Subject: Simple TCP proxy In-Reply-To: References: <20220727205810.55ivclftzsggrenx@gmail.com> Message-ID: <7E5B0B14-2834-41AA-9BD7-B4282C3A8354@barrys-emacs.org> Morten, As Chris remarked you need to learn a number of networking, python, system performance and other skills to turn your project into production code. Using threads does not scale very well. Its uses a lot of memory and raises CPU used just to do the context switches. Also the GIL means that even if you are doing blocking I/O the use of threads does not scale well. Its rare to see multi threaded code, rather what you see is code that uses async I/O. At its heart async code at the low level is using a kernel interface like epoll (or on old systems select). What epoll allow you to do is wait on a sets of FDs for a range of I/O operations. Like ready to read, ready to write and other activity (like the socket closing). You could write code to use epoll your self, but while fun to write you need to know a lot about networking and linux to cover all the corner cases. Libraries like twisted, trio, uvloop and pythons selectors implemented production quality version of the required code with good APIs. Do not judge these libraries by their size. They are no bloated and only as complex as the problem they are solving requires. There is a simple example of async code using the python selectors here that shows the style of programming. https://docs.python.org/3/library/selectors.html#examples The issues that you likely need to solve and test for include: * handling unexpected socket close events. * buffering and flow control from one socket's read to the another socket's write. What if one side is reading slower then the other is writing? * timeout sockets that stop sending data and close them At some point you will exceed the capacity for one process to handle the load. The solution we used is to listen on the socket in a parent process and fork enough child processes to handle the I/O load. This avoids issues with the GIL and allows you to scale. But I am still not sure why you need to do anything more the increase the backlog on your listen socket in the main app. Set the backlog to 1,000,000 does that fix your issue? You will need on Linux to change kernel limits to allow that size. See man listen for info on what you need to change. Barry From c.buhtz at posteo.jp Sat Jul 30 08:49:05 2022 From: c.buhtz at posteo.jp (c.buhtz at posteo.jp) Date: Sat, 30 Jul 2022 12:49:05 +0000 Subject: PEP about recommended project folder layout In-Reply-To: References: Message-ID: <4Lw42Y1yPvz9rxh@submission02.posteo.de> Isn't there a PEP? On 2022-07-26 07:14 c.buhtz at posteo.jp wrote: > Hello, > > I am not sure if I looked into the correct sources. I was looking in > "PEP 609 ? Python Packaging Authority (PyPA) Governance" [1] and the > "PyPA specifications" [2]. > > My question in short: Is there an official document (e.g. a PEP) > about a recommended layout for project folders. > > Looking into the wild and past there are a lot of variations of such > layouts. I am far away from being a pro but depending on experience > in my own projects and what I have learned from others (e.g. in > blog-posts/tutorials) I recommend to have the "test" folder and the > package folder side by side on the same level in the project folder > (the root). > > my_project > |- tests > | ? test_*.py > |- my_package > | ? __init__.py > ?-- README.md > > I sometimes add to it the so called "src"-Layout where the package > folder is one level deeper in an extra "src" folder. > > my_project > |- tests > | ? test_*.py > |- src > | ?- my_package > | ? __init__.py > ?-- README.md > > I don't want to discuss the pros and cons of all variations. What I > need is an official document I can use in discussions with other > maintainers. If there is a PEP/document against my current > recommendation I am also fine with this. ;) > > Kind > Christian > > [1] -- > [2] -- From barry at barrys-emacs.org Sat Jul 30 09:59:14 2022 From: barry at barrys-emacs.org (Barry) Date: Sat, 30 Jul 2022 14:59:14 +0100 Subject: How to generate a .pyi file for a C Extension using stubgen In-Reply-To: References: Message-ID: <95C1FCDD-29FA-4CEA-9C47-419686A330AB@barrys-emacs.org> > On 30 Jul 2022, at 10:30, Marco Sulla wrote: > > ?On Fri, 29 Jul 2022 at 23:23, Barry wrote: >> >> >> >>>> On 29 Jul 2022, at 19:33, Marco Sulla wrote: >>> >>> ?I tried to follow the instructions here: >>> >>> https://mypy.readthedocs.io/en/stable/stubgen.html >>> >>> but the instructions about creating a stub for a C Extension are a little >>> mysterious. I tried to use it on the .so file without luck. >> >> It says that stubgen works on .py files not .so files. >> You will need to write the .pyi for your .so manually. >> >> The docs could do with splitting the need for .pyi for .so >> away from the stubgen description > > But it says: > > "Mypy includes the stubgen tool that can automatically generate stub > files (.pyi files) for Python modules and C extension modules." > > I tried stubgen -m modulename, but it generates very little code. Oh? From the .so I am struggling to figure out how it could ever work reliably. I cannot see that there is enough information in a useful form to allow the tool to work. Barry > From morphex at gmail.com Sat Jul 30 15:30:30 2022 From: morphex at gmail.com (Morten W. Petersen) Date: Sat, 30 Jul 2022 21:30:30 +0200 Subject: Simple TCP proxy In-Reply-To: <9c2f1958-fbcc-bdab-7d78-c1282a18b1dc@roelschroeven.net> References: <20220727205810.55ivclftzsggrenx@gmail.com> <9c2f1958-fbcc-bdab-7d78-c1282a18b1dc@roelschroeven.net> Message-ID: I thought it was a bit much. I just did a bit more testing, and saw that the throughput of wget through regular lighttpd was 1,3 GB/s, while through STP it was 122 MB/s, and using quite a bit of CPU. Then I increased the buffer size 8-fold for reading and writing in run.py, and the CPU usage went way down, and the transfer speed went up to 449 MB/s. So it would require well more than a gigabit network interface to max out STP throughput; CPU usage was around 30-40% max, on one processor core. There is good enough, and then there's general practice and/or what is regarded as an elegant solution. I'm looking for good enough, and in the process I don't mind pushing the envelope on Python threading. -Morten On Sat, Jul 30, 2022 at 12:59 PM Roel Schroeven wrote: > Morten W. Petersen schreef op 29/07/2022 om 22:59: > > OK, sounds like sunshine is getting the best of you. > It has to be said: that is uncalled for. > > Chris gave you good advice, with the best of intentions. Sometimes we > don't like good advice if it says something we don't like, but that's no > reason to take it off on the messenger. > > -- > "Iceland is the place you go to remind yourself that planet Earth is a > machine... and that all organic life that has ever existed amounts to a > greasy > film that has survived on the exterior of that machine thanks to furious > improvisation." > -- Sam Hughes, Ra > > -- > https://mail.python.org/mailman/listinfo/python-list > -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ From hjp-python at hjp.at Sat Jul 30 16:19:18 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sat, 30 Jul 2022 22:19:18 +0200 Subject: mailbox misbehavior with non-ASCII In-Reply-To: References: Message-ID: <20220730201918.cxfr5ulopa6guq43@hjp.at> On 2022-07-29 23:24:57 +0000, Peter Pearson wrote: > The following code produces a nonsense result with the input > described below: > > import mailbox > box = mailbox.Maildir("/home/peter/Temp/temp",create=False) > x = box.values()[0] > h = x.get("X-DSPAM-Factors") > print(type(h)) > # > > The output is the desired "str" when the message file contains this: > > To: recipient at example.com > Message-ID: <123> > Date: Sun, 24 Jul 2022 15:31:19 +0000 > Subject: Blah blah > From: from at from.com > X-DSPAM-Factors: a'b > > xxx > > ... but if the apostrophe in "a'b" is replaced with a > RIGHT SINGLE QUOTATION MARK, the returned h is of type > "email.header.Header", and seems to contain inscrutable garbage. It's not inscrutable to me, but then I remember when RFC 1522 was the relevant RFC. Calling h.encode() returns =?unknown-8bit?b?YeKAmWI=?= which is about the best result you can get. The character set is unknown and the content (when decoded) is the bytes 61 e2 80 99 62 which is what your file contained (assuming you used UTF-8). What would be nice if you could get at that content directly. There doesn't seem to be documented method to do that. You can use h._chunks, but as the _ in the name implies, that's implementation detail which might change in future versions (and it's not quite straightforward either, although consistent with other parts of python, I think). hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From barry at barrys-emacs.org Sat Jul 30 16:31:39 2022 From: barry at barrys-emacs.org (Barry) Date: Sat, 30 Jul 2022 21:31:39 +0100 Subject: Simple TCP proxy Message-ID: <0B1BC1CF-F92D-441F-A332-F5DBEF4271AB@barrys-emacs.org> ? > On 30 Jul 2022, at 20:33, Morten W. Petersen wrote: > ?I thought it was a bit much. > > I just did a bit more testing, and saw that the throughput of wget through > regular lighttpd was 1,3 GB/s, while through STP it was 122 MB/s, and using > quite a bit of CPU. > > Then I increased the buffer size 8-fold for reading and writing in run.py, > and the CPU usage went way down, and the transfer speed went up to 449 MB/s. You are trading latency for through put. > > So it would require well more than a gigabit network interface to max out > STP throughput; CPU usage was around 30-40% max, on one processor core. With how many connections? > > There is good enough, and then there's general practice and/or what is > regarded as an elegant solution. I'm looking for good enough, and in the > process I don't mind pushing the envelope on Python threading. You never did answer my query on why a large backlog is not good enough. Why do you need this program at all? Barry > > -Morten > > On Sat, Jul 30, 2022 at 12:59 PM Roel Schroeven > wrote: > >> Morten W. Petersen schreef op 29/07/2022 om 22:59: >>> OK, sounds like sunshine is getting the best of you. >> It has to be said: that is uncalled for. >> >> Chris gave you good advice, with the best of intentions. Sometimes we >> don't like good advice if it says something we don't like, but that's no >> reason to take it off on the messenger. >> >> -- >> "Iceland is the place you go to remind yourself that planet Earth is a >> machine... and that all organic life that has ever existed amounts to a >> greasy >> film that has survived on the exterior of that machine thanks to furious >> improvisation." >> -- Sam Hughes, Ra >> >> -- >> https://mail.python.org/mailman/listinfo/python-list > > > -- > I am https://leavingnorway.info > Videos at https://www.youtube.com/user/TheBlogologue > Twittering at http://twitter.com/blogologue > Blogging at http://blogologue.com > Playing music at https://soundcloud.com/morten-w-petersen > Also playing music and podcasting here: > http://www.mixcloud.com/morten-w-petersen/ > On Google+ here https://plus.google.com/107781930037068750156 > On Instagram at https://instagram.com/morphexx/ > -- > https://mail.python.org/mailman/listinfo/python-list From barry at barrys-emacs.org Sat Jul 30 16:34:23 2022 From: barry at barrys-emacs.org (Barry) Date: Sat, 30 Jul 2022 21:34:23 +0100 Subject: PEP about recommended project folder layout In-Reply-To: <4Lw42Y1yPvz9rxh@submission02.posteo.de> References: <4Lw42Y1yPvz9rxh@submission02.posteo.de> Message-ID: <876FD083-A333-459A-9BD1-C5D804728EC3@barrys-emacs.org> > On 30 Jul 2022, at 13:52, c.buhtz at posteo.jp wrote: > > ?Isn't there a PEP? PEP are for improving python. They are not for telling people how to use python. I would be surprised to fine a PEP that addressed this. Barry > >> On 2022-07-26 07:14 c.buhtz at posteo.jp wrote: >> Hello, >> >> I am not sure if I looked into the correct sources. I was looking in >> "PEP 609 ? Python Packaging Authority (PyPA) Governance" [1] and the >> "PyPA specifications" [2]. >> >> My question in short: Is there an official document (e.g. a PEP) >> about a recommended layout for project folders. >> >> Looking into the wild and past there are a lot of variations of such >> layouts. I am far away from being a pro but depending on experience >> in my own projects and what I have learned from others (e.g. in >> blog-posts/tutorials) I recommend to have the "test" folder and the >> package folder side by side on the same level in the project folder >> (the root). >> >> my_project >> |- tests >> | ? test_*.py >> |- my_package >> | ? __init__.py >> ?-- README.md >> >> I sometimes add to it the so called "src"-Layout where the package >> folder is one level deeper in an extra "src" folder. >> >> my_project >> |- tests >> | ? test_*.py >> |- src >> | ?- my_package >> | ? __init__.py >> ?-- README.md >> >> I don't want to discuss the pros and cons of all variations. What I >> need is an official document I can use in discussions with other >> maintainers. If there is a PEP/document against my current >> recommendation I am also fine with this. ;) >> >> Kind >> Christian >> >> [1] -- >> [2] -- > > -- > https://mail.python.org/mailman/listinfo/python-list From gweatherby at uchc.edu Sun Jul 31 06:16:35 2022 From: gweatherby at uchc.edu (Weatherby,Gerard) Date: Sun, 31 Jul 2022 10:16:35 +0000 Subject: PEP about recommended project folder layout In-Reply-To: <876FD083-A333-459A-9BD1-C5D804728EC3@barrys-emacs.org> References: <4Lw42Y1yPvz9rxh@submission02.posteo.de> <876FD083-A333-459A-9BD1-C5D804728EC3@barrys-emacs.org> Message-ID: <575980a9-c16e-43fb-8803-2e69eed00a15@Spark> I?m not aware of any standard convention for laying out packages. PEP 8 (https://peps.python.org/pep-0008/) specifies conventions for how to write Python, so a standard layout PEP would not be inconsistent. ? Gerard Weatherby | Application Architect NMRbox | NAN | Department of Molecular Biology and Biophysics UConn Health 263 Farmington Avenue, Farmington, CT 06030-6406 uchc.edu On Jul 30, 2022, 4:37 PM -0400, Barry , wrote: *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. *** On 30 Jul 2022, at 13:52, c.buhtz at posteo.jp wrote: ?Isn't there a PEP? PEP are for improving python. They are not for telling people how to use python. I would be surprised to fine a PEP that addressed this. Barry On 2022-07-26 07:14 c.buhtz at posteo.jp wrote: Hello, I am not sure if I looked into the correct sources. I was looking in "PEP 609 ? Python Packaging Authority (PyPA) Governance" [1] and the "PyPA specifications" [2]. My question in short: Is there an official document (e.g. a PEP) about a recommended layout for project folders. Looking into the wild and past there are a lot of variations of such layouts. I am far away from being a pro but depending on experience in my own projects and what I have learned from others (e.g. in blog-posts/tutorials) I recommend to have the "test" folder and the package folder side by side on the same level in the project folder (the root). my_project |- tests | ? test_*.py |- my_package | ? __init__.py ?-- README.md I sometimes add to it the so called "src"-Layout where the package folder is one level deeper in an extra "src" folder. my_project |- tests | ? test_*.py |- src | ?- my_package | ? __init__.py ?-- README.md I don't want to discuss the pros and cons of all variations. What I need is an official document I can use in discussions with other maintainers. If there is a PEP/document against my current recommendation I am also fine with this. ;) Kind Christian [1] -- [2] -- -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!ipqSQS5_ZFxO37h5h5XALf2DxE3eV-x6qNSrVnKJU3cLZIkLfKhuTzSxi1fb-BvYNVPXeT-nYFX8ogse6_J2Hg$ -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!ipqSQS5_ZFxO37h5h5XALf2DxE3eV-x6qNSrVnKJU3cLZIkLfKhuTzSxi1fb-BvYNVPXeT-nYFX8ogse6_J2Hg$ From rosuav at gmail.com Sun Jul 31 07:03:03 2022 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 31 Jul 2022 21:03:03 +1000 Subject: PEP about recommended project folder layout In-Reply-To: <575980a9-c16e-43fb-8803-2e69eed00a15@Spark> References: <4Lw42Y1yPvz9rxh@submission02.posteo.de> <876FD083-A333-459A-9BD1-C5D804728EC3@barrys-emacs.org> <575980a9-c16e-43fb-8803-2e69eed00a15@Spark> Message-ID: On Sun, 31 Jul 2022 at 20:27, Weatherby,Gerard wrote: > > I?m not aware of any standard convention for laying out packages. > > PEP 8 (https://peps.python.org/pep-0008/) specifies conventions for how to write Python, so a standard layout PEP would not be inconsistent. > PEP 8 species rules for laying out the code of the Python standard library. Its adoption by other projects does not constitute the Python developers declaring that it's a convention for how to write all Python code. A better example would be PEP 257 https://peps.python.org/pep-0257/ but even that is more for the purpose of tooling. It does at least try to describe usage conventions, though. Conventions for laying out packages (as opposed to actual requirements defined by the packaging system itself) would be better described somewhere other than a PEP. ChrisA From morphex at gmail.com Sun Jul 31 14:03:27 2022 From: morphex at gmail.com (Morten W. Petersen) Date: Sun, 31 Jul 2022 20:03:27 +0200 Subject: Simple TCP proxy In-Reply-To: <0B1BC1CF-F92D-441F-A332-F5DBEF4271AB@barrys-emacs.org> References: <0B1BC1CF-F92D-441F-A332-F5DBEF4271AB@barrys-emacs.org> Message-ID: Well, initially I was just curious. As the name implies, it's a TCP proxy, and different features could go into that. I looked at for example port knocking for hindering unauthorized access to the (protected) TCP service SMPS, but there you also have the possibility of someone eavesdropping, and learning the right handshake, if you will. So it's something that will work, until someone gets determined to make a mess. In short, it will give better control than backlog does, enabling Python-style code and logic to deal with different situations. I was about to say "deal with things intelligently"; but I think "intelligent" is a word that doesn't fit here or in many other applications. Say for example this service comes under attack for unknown reasons; it could be possible to teach the proxy to only accept connections to the backend server for IP addresses / subnets that have previously n number of transmissions back and forth. If you know that the service will have max 50 different clients. Anyway, what Chris said earlier, I think we can file that under "eagerness to tech others and show what you know". Right Chris? :) Regards, Morten On Sat, Jul 30, 2022 at 10:31 PM Barry wrote: > > > > > On 30 Jul 2022, at 20:33, Morten W. Petersen wrote: > > ?I thought it was a bit much. > > > > I just did a bit more testing, and saw that the throughput of wget > through > > regular lighttpd was 1,3 GB/s, while through STP it was 122 MB/s, and > using > > quite a bit of CPU. > > > > Then I increased the buffer size 8-fold for reading and writing in > run.py, > > and the CPU usage went way down, and the transfer speed went up to 449 > MB/s. > > You are trading latency for through put. > > > > > So it would require well more than a gigabit network interface to max out > > STP throughput; CPU usage was around 30-40% max, on one processor core. > > With how many connections? > > > > > There is good enough, and then there's general practice and/or what is > > regarded as an elegant solution. I'm looking for good enough, and in the > > process I don't mind pushing the envelope on Python threading. > > You never did answer my query on why a large backlog is not good enough. > Why do you need this program at all? > > Barry > > > > -Morten > > > > On Sat, Jul 30, 2022 at 12:59 PM Roel Schroeven > > wrote: > > > >> Morten W. Petersen schreef op 29/07/2022 om 22:59: > >>> OK, sounds like sunshine is getting the best of you. > >> It has to be said: that is uncalled for. > >> > >> Chris gave you good advice, with the best of intentions. Sometimes we > >> don't like good advice if it says something we don't like, but that's no > >> reason to take it off on the messenger. > >> > >> -- > >> "Iceland is the place you go to remind yourself that planet Earth is a > >> machine... and that all organic life that has ever existed amounts to a > >> greasy > >> film that has survived on the exterior of that machine thanks to furious > >> improvisation." > >> -- Sam Hughes, Ra > >> > >> -- > >> https://mail.python.org/mailman/listinfo/python-list > > > > > > -- > > I am https://leavingnorway.info > > Videos at https://www.youtube.com/user/TheBlogologue > > Twittering at http://twitter.com/blogologue > > Blogging at http://blogologue.com > > Playing music at https://soundcloud.com/morten-w-petersen > > Also playing music and podcasting here: > > http://www.mixcloud.com/morten-w-petersen/ > > On Google+ here https://plus.google.com/107781930037068750156 > > On Instagram at https://instagram.com/morphexx/ > > -- > > https://mail.python.org/mailman/listinfo/python-list > > -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ From sjeik_appie at hotmail.com Sun Jul 31 05:39:54 2022 From: sjeik_appie at hotmail.com (Albert-Jan Roskam) Date: Sun, 31 Jul 2022 11:39:54 +0200 Subject: Register multiple excepthooks? Message-ID: Hi, I have a function init_logging.log_uncaught_errors() that I use for sys.excepthook. Now I also want to call another function (ffi.dlclose()) upon abnormal termination. Is it possible to register multiple excepthooks, like with atexit.register? Or should I rename/redefine log_uncaught_errors() so it does both things? Thanks! Albert-Jan From avi.e.gross at gmail.com Sun Jul 31 22:43:49 2022 From: avi.e.gross at gmail.com (avi.e.gross at gmail.com) Date: Sun, 31 Jul 2022 22:43:49 -0400 Subject: Register multiple excepthooks? In-Reply-To: References: Message-ID: <017f01d8a550$87b6a3d0$9723eb70$@gmail.com> Albert-Jan, Unless there is something special in your scenario, aren't there many ways within python proper to create functions that effectively call another function or more as needed? Think decorators as one example. Of course if the caller expects some specific result when it calls your function, running a single function that returns say the value of the last thing it does, may not meet your needs. Now assuming all your functions take the same required argument and do not in any way tamper with the argument, you could indeed define a function that accepts arguments and internally calls one after another other functions in some order and intercepts anything they return (or errors) and consolidates them as needed and returns a result. I suspect there is a module that provides such functionality. -----Original Message----- From: Python-list On Behalf Of Albert-Jan Roskam Sent: Sunday, July 31, 2022 5:40 AM To: Python-list at python.org Subject: Register multiple excepthooks? Hi, I have a function init_logging.log_uncaught_errors() that I use for sys.excepthook. Now I also want to call another function (ffi.dlclose()) upon abnormal termination. Is it possible to register multiple excepthooks, like with atexit.register? Or should I rename/redefine log_uncaught_errors() so it does both things? Thanks! Albert-Jan -- https://mail.python.org/mailman/listinfo/python-list