From nbayedil at gmail.com Sun May 1 09:31:45 2022 From: nbayedil at gmail.com (Nas Bayedil) Date: Sun, 1 May 2022 19:31:45 +0600 Subject: new sorting algorithm Message-ID: *Dear, Sir/Madam* Let me first tell you briefly who we are and where we are from, what we do. My name is Nas (full name Nasipa Bayedil) from Kazakhstan. In December 2020, we registered a company online in Dover, Delaware, the United States, because all major corporations are based in the United States. Direction of the company: research and development technologies of sorting and searching in databases and in Big Data. Our research and developments should be of interest to a rapidly growing tech market. My father Nurgali has a mathematics education, he has been interested in mathematics' and physics all his life, now he is retired and continues to study his favorite mathematics, when he became a pensioner to this love an interest in programming was added. And this new interest in programming led him to invent his own method of developing sorting algorithms. *In a nutshell about what we are doing.* We have developed several variants of sorting algorithms based on new ideas and improvements to well-known ideas. Our development is carried out in the application of the idea of the article Peter McIlroy's 1993 paper "Optimistic Sorting and Information Theoretic Complexity", which was implemented by *Tim Peters TimSort in a sorting algorithm in 2002 for use in Python.* The variant implemented by Tim Peters is improved by us as it does not take full advantage of Peter McIlroy's idea. This has been achieved through the development of new methods: 1. data recognition 2. data fusion We also used the research work Sorting N-Elements Using Natural Order: A New Adaptive Sorting Approach** but added some new ideas of our own. As a result, we got a hybrid NDsort algorithm, which uses the above particular algorithms. Existing sorting algorithms that are used in practice, very fast, I will not list their advantages; for improvement let's just indicate that some of them start to work slowly on certain types of data. The author tried to deal with this from a general, *mathematical position*, and not from the point of view of programming techniques, *and achieved the goal*. Fragments of the general theory of sorting by comparison have been developed, *several facts have been discovered that are subject to patenting *and which allow developing various variants of sorting algorithms. We believe that using this method to develop completely new, fast algorithms, approaching the speed of the famous *QuickSort*, the speed of which cannot be surpassed, but its drawback can be circumvented, in the sense of stack overflow, on some data. Additionally our general sorting algorithm can be applied to design data sorting chips. This is, of course, the business of specialists in this field, but we believe that our algorithm opens up a new approach to this problem. The fact is that there are currently two ways to develop data sorting chips. The first is that data is entered in parallel; the second is that data is entered sequentially. Our sorting algorithm allows parallel-sequential data input, which would speed up the entire sorting process. I hope our research will pique your interest. The distinctive features of our sorting algorithms are the minimization of the number of comparisons, a new approach to the sorting problem, the maximum use of common sense, and the use of a principle from philosophy "Occam's razor". We use C# (Visual Studio 2019 demo version) for research and write fragments of algorithms. We would like to ask for help from Python.org engineers in implementation of our idea for the Python language together. We will be glad to hear from you any comments and questions about our idea. Kind regards, Nas Bayedil Twitter: @NasBayedil Website: www.bazony.com.kz **June 2010 Journal of Computer Science 6(2) Project: Algorithm Analysis, Authors: Shamim Akhter, International University of Business Agriculture and Technology M. Tanveer Hasan. From Marco.Sulla.Python at gmail.com Sun May 1 12:55:41 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Sun, 1 May 2022 18:55:41 +0200 Subject: tail In-Reply-To: <8690dd6e-f26d-7165-be63-d80d03e06555@DancesWithMice.info> References: <8690dd6e-f26d-7165-be63-d80d03e06555@DancesWithMice.info> Message-ID: Something like this is OK? import os def tail(f): chunk_size = 100 size = os.stat(f.fileno()).st_size positions = iter(range(size, -1, -chunk_size)) next(positions) chunk_line_pos = -1 pos = 0 for pos in positions: f.seek(pos) chars = f.read(chunk_size) chunk_line_pos = chars.rfind(b"\n") if chunk_line_pos != -1: break if chunk_line_pos == -1: nbytes = pos pos = 0 f.seek(pos) chars = f.read(nbytes) chunk_line_pos = chars.rfind(b"\n") if chunk_line_pos == -1: line_pos = pos else: line_pos = pos + chunk_line_pos + 1 f.seek(line_pos) return f.readline() This is simply for one line and for utf8. From Marco.Sulla.Python at gmail.com Sun May 1 13:04:02 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Sun, 1 May 2022 19:04:02 +0200 Subject: new sorting algorithm In-Reply-To: References: Message-ID: I suppose you should write to python-dev at python.org , or in https://discuss.python.org/ under the section Core development From drsalists at gmail.com Sun May 1 13:52:19 2022 From: drsalists at gmail.com (Dan Stromberg) Date: Sun, 1 May 2022 10:52:19 -0700 Subject: new sorting algorithm In-Reply-To: References: Message-ID: This probably should start out as a module on Pypi. Is the sorting stable? Python guarantees that. On Sun, May 1, 2022 at 8:53 AM Nas Bayedil wrote: > *Dear, Sir/Madam* > > > Let me first tell you briefly who we are and where we are from, what we do. > > My name is Nas (full name Nasipa Bayedil) from Kazakhstan. > > In December 2020, we registered a company online in Dover, Delaware, the > United States, because all major corporations are based in the United > States. Direction of the company: research and development technologies of > sorting and searching in databases and in Big Data. Our research and > developments should be of interest to a rapidly growing tech market. > > My father Nurgali has a mathematics education, he has been interested in > mathematics' and physics all his life, now he is retired and continues to > study his favorite mathematics, when he became a pensioner to this love an > interest in programming was added. And this new interest in programming led > him to invent his own method of developing sorting algorithms. > > *In a nutshell about what we are doing.* > > We have developed several variants of sorting algorithms based on new ideas > and improvements to well-known ideas. > > Our development is carried out in the application of the idea of the > article Peter McIlroy's 1993 paper "Optimistic Sorting and Information > Theoretic Complexity", which was implemented by *Tim Peters TimSort in a > sorting algorithm in 2002 for use in Python.* > > The variant implemented by Tim Peters is improved by us as it does not take > full advantage of Peter McIlroy's idea. > > This has been achieved through the development of new methods: > > 1. data recognition > > 2. data fusion > > > > We also used the research work Sorting N-Elements Using Natural Order: A > New Adaptive Sorting Approach** but added some new ideas of our own. As a > result, we got a hybrid NDsort algorithm, which uses the above particular > algorithms. > > > > Existing sorting algorithms that are used in practice, very fast, I will > not list their advantages; for improvement let's just indicate that some of > them start to work slowly on certain types of data. The author tried to > deal with this from a general, *mathematical position*, and not from the > point of view of programming techniques, *and achieved the goal*. > > Fragments of the general theory of sorting by comparison have been > developed, *several facts have been discovered that are subject to > patenting *and which allow developing various variants of sorting > algorithms. > > We believe that using this method to develop completely new, fast > algorithms, approaching the speed of the famous *QuickSort*, the speed of > which cannot be surpassed, but its drawback can be circumvented, in the > sense of stack overflow, on some data. > > Additionally our general sorting algorithm can be applied to design data > sorting chips. This is, of course, the business of specialists in this > field, but we believe that our algorithm opens up a new approach to this > problem. The fact is that there are currently two ways to develop data > sorting chips. The first is that data is entered in parallel; the second is > that data is entered sequentially. Our sorting algorithm allows > parallel-sequential data input, which would speed up the entire sorting > process. > > I hope our research will pique your interest. The distinctive features of > our sorting algorithms are the minimization of the number of comparisons, a > new approach to the sorting problem, the maximum use of common sense, and > the use of a principle from philosophy "Occam's razor". > > We use C# (Visual Studio 2019 demo version) for research and write > fragments of algorithms. We would like to ask for help from Python.org > engineers in implementation of our idea for the Python language together. > > We will be glad to hear from you any comments and questions about our idea. > > > > > > Kind regards, > > Nas Bayedil > > > > Twitter: @NasBayedil > > Website: www.bazony.com.kz > > > > **June 2010 Journal of Computer Science 6(2) Project: Algorithm Analysis, > > Authors: Shamim Akhter, International University of Business Agriculture > and Technology M. Tanveer Hasan. > -- > https://mail.python.org/mailman/listinfo/python-list > From rosuav at gmail.com Sun May 1 14:09:39 2022 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 2 May 2022 04:09:39 +1000 Subject: new sorting algorithm In-Reply-To: References: Message-ID: On Mon, 2 May 2022 at 01:53, Nas Bayedil wrote: > We believe that using this method to develop completely new, fast > algorithms, approaching the speed of the famous *QuickSort*, the speed of > which cannot be surpassed, but its drawback can be circumvented, in the > sense of stack overflow, on some data. Hmm, actually TimSort *does* exceed the speed of quicksort for a lot of real-world data. For instance, if you take a large sorted list, append a handful of (unsorted) items to it, and then sort the list, TimSort can take advantage of the fact that the bulk of the list is sorted. It ends up significantly faster than re-sorting the entire list. How well does your new variant handle this case? ChrisA From cs at cskk.id.au Sun May 1 18:18:39 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Mon, 2 May 2022 08:18:39 +1000 Subject: tail In-Reply-To: References: Message-ID: On 01May2022 18:55, Marco Sulla wrote: >Something like this is OK? [...] >def tail(f): > chunk_size = 100 > size = os.stat(f.fileno()).st_size I think you want os.fstat(). > positions = iter(range(size, -1, -chunk_size)) > next(positions) I was wondering about the iter, but this makes sense. Alternatively you could put a range check in the for-loop. > chunk_line_pos = -1 > pos = 0 > > for pos in positions: > f.seek(pos) > chars = f.read(chunk_size) > chunk_line_pos = chars.rfind(b"\n") > > if chunk_line_pos != -1: > break Normal text file _end_ in a newline. I'd expect this to stop immediately at the end of the file. > if chunk_line_pos == -1: > nbytes = pos > pos = 0 > f.seek(pos) > chars = f.read(nbytes) > chunk_line_pos = chars.rfind(b"\n") I presume this is because unless you're very lucky, 0 will not be a position in the range(). I'd be inclined to avoid duplicating this code and special case and instead maybe make the range unbounded and do something like this: if pos < 0: pos = 0 ... seek/read/etc ... if pos == 0: break around the for-loop body. > if chunk_line_pos == -1: > line_pos = pos > else: > line_pos = pos + chunk_line_pos + 1 > f.seek(line_pos) > return f.readline() > >This is simply for one line and for utf8. And anything else where a newline is just an ASCII newline byte (10) and can't be mistaken otherwise. So also ASCII and all the ISO8859-x single byte encodings. But as Chris has mentioned, not for other encodings. Seems sane. I haven't tried to run it. Cheers, Cameron Simpson From drsalists at gmail.com Sun May 1 19:17:33 2022 From: drsalists at gmail.com (Dan Stromberg) Date: Sun, 1 May 2022 16:17:33 -0700 Subject: tail In-Reply-To: References: Message-ID: On Sun, May 1, 2022 at 3:19 PM Cameron Simpson wrote: > On 01May2022 18:55, Marco Sulla wrote: > >Something like this is OK? > Scanning backward for a byte == 10 in ASCII or ISO-8859 seems fine. But what about Unicode? Are all 10 bytes newlines in Unicode encodings? If not, and you have a huge file to reverse, it might be better to use a temporary file. From drsalists at gmail.com Sun May 1 19:20:16 2022 From: drsalists at gmail.com (Dan Stromberg) Date: Sun, 1 May 2022 16:20:16 -0700 Subject: new sorting algorithm In-Reply-To: References: Message-ID: On Sun, May 1, 2022 at 1:44 PM Chris Angelico wrote: > On Mon, 2 May 2022 at 06:43, Dan Stromberg wrote: > > On Sun, May 1, 2022 at 11:10 AM Chris Angelico wrote: > >> > >> On Mon, 2 May 2022 at 01:53, Nas Bayedil wrote: > >> > We believe that using this method to develop completely new, fast > >> > algorithms, approaching the speed of the famous *QuickSort*, the > speed of > >> > which cannot be surpassed, but its drawback can be circumvented, in > the > >> > sense of stack overflow, on some data. > >> > >> Hmm, actually TimSort *does* exceed the speed of quicksort for a lot > >> of real-world data. For instance, if you take a large sorted list, > >> append a handful of (unsorted) items to it, and then sort the list, > >> TimSort can take advantage of the fact that the bulk of the list is > >> sorted. It ends up significantly faster than re-sorting the entire > >> list. > > > > > > In fact, Timsort is O(n) for already-sorted data, while many quicksorts > are O(n^2) for already-sorted data. > > > > Quicksort can be salvaged by using a median-of-3 partitioning, but it's > still O(n^2) in the (less common) worst case. > > > > This is true, but purely sorted data isn't a very practical case. The > case of mostly-sorted data IS practical, though, so it's a quite big > win that it can be close to O(n), and still faster than inserting each > item individually. > You seem to be of the impression that nearly-sorted data isn't an uphill battle with a straightforward quicksort. I'm having a hard time convincing myself of that. From rosuav at gmail.com Sun May 1 19:41:43 2022 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 2 May 2022 09:41:43 +1000 Subject: tail In-Reply-To: References: Message-ID: On Mon, 2 May 2022 at 09:19, Dan Stromberg wrote: > > On Sun, May 1, 2022 at 3:19 PM Cameron Simpson wrote: > > > On 01May2022 18:55, Marco Sulla wrote: > > >Something like this is OK? > > > > Scanning backward for a byte == 10 in ASCII or ISO-8859 seems fine. > > But what about Unicode? Are all 10 bytes newlines in Unicode encodings? Most absolutely not. "Unicode" isn't an encoding, but of the Unicode Transformation Formats and Universal Character Set encodings, most don't make that guarantee: * UTF-8 does, as mentioned. It sacrifices some efficiency and consistency for a guarantee that ASCII characters are represented by ASCII bytes, and ASCII bytes only ever represent ASCII characters. * UCS-2 and UTF-16 will both represent BMP characters with two bytes. Any character U+xx0A or U+0Axx will include an 0x0A in its representation. * UTF-16 will also encode anything U+000xxx0A with an 0x0A. (And I don't think any codepoints have been allocated that would trigger this, but UTF-16 can also use 0x0A in the high surrogate.) * UTF-32 and UCS-4 will use 0x0A for any character U+xx0A, U+0Axx, and U+Axxxx (though that plane has no characters on it either) So, of all the available Unicode standard encodings, only UTF-8 makes this guarantee. Of course, if you look at documents available on the internet, UTF-8 the encoding used by the vast majority of them (especially if you include seven-bit files, which can equally be considered ASCII, ISO-8859-x, and UTF-8), so while it might only be one encoding out of many, it's probably the most important :) In general, you can *only* make this parsing assumption IF you know for sure that your file's encoding is UTF-8, ISO-8859-x, some OEM eight-bit encoding (eg Windows-125x), or one of a handful of other compatible encodings. But it probably will be. > If not, and you have a huge file to reverse, it might be better to use a > temporary file. Yeah, or an in-memory deque if you know how many lines you want. Either way, you can read the file forwards, guaranteeing correct decoding even of a shifted character set (where a byte value can change in meaning based on arbitrarily distant context). ChrisA From rosuav at gmail.com Sun May 1 19:45:32 2022 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 2 May 2022 09:45:32 +1000 Subject: new sorting algorithm In-Reply-To: References: Message-ID: On Mon, 2 May 2022 at 09:20, Dan Stromberg wrote: > > > On Sun, May 1, 2022 at 1:44 PM Chris Angelico wrote: >> >> On Mon, 2 May 2022 at 06:43, Dan Stromberg wrote: >> > On Sun, May 1, 2022 at 11:10 AM Chris Angelico wrote: >> >> >> >> On Mon, 2 May 2022 at 01:53, Nas Bayedil wrote: >> >> > We believe that using this method to develop completely new, fast >> >> > algorithms, approaching the speed of the famous *QuickSort*, the speed of >> >> > which cannot be surpassed, but its drawback can be circumvented, in the >> >> > sense of stack overflow, on some data. >> >> >> >> Hmm, actually TimSort *does* exceed the speed of quicksort for a lot >> >> of real-world data. For instance, if you take a large sorted list, >> >> append a handful of (unsorted) items to it, and then sort the list, >> >> TimSort can take advantage of the fact that the bulk of the list is >> >> sorted. It ends up significantly faster than re-sorting the entire >> >> list. >> > >> > >> > In fact, Timsort is O(n) for already-sorted data, while many quicksorts are O(n^2) for already-sorted data. >> > >> > Quicksort can be salvaged by using a median-of-3 partitioning, but it's still O(n^2) in the (less common) worst case. >> > >> >> This is true, but purely sorted data isn't a very practical case. The >> case of mostly-sorted data IS practical, though, so it's a quite big >> win that it can be close to O(n), and still faster than inserting each >> item individually. > > > You seem to be of the impression that nearly-sorted data isn't an uphill battle with a straightforward quicksort. > > I'm having a hard time convincing myself of that. > The median-of-three partitioning technique makes that work reasonably well, so it won't be pathologically slow. It's hardly Quicksort's best feature, but it could easily be a lot worse. I'd have to check, but I think it still manages to be O(n log n). Merge sort, of course, is a lot more consistent, but the asymptotic cost is still broadly the same. But Timsort manages to be close to O(n) for sorted data, reversed data, nearly-sorted or nearly-reversed data, etc. Makes it very handy for jobs like "click a heading to sort by it", where you might add multiple sort keys. (Plus, Python's implementation has some cool tricks for small collections that make it quite efficient.) ChrisA From cs at cskk.id.au Sun May 1 21:52:53 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Mon, 2 May 2022 11:52:53 +1000 Subject: tail In-Reply-To: References: Message-ID: On 01May2022 23:30, Stefan Ram wrote: >Dan Stromberg writes: >>But what about Unicode? Are all 10 bytes newlines in Unicode encodings? > It seems in UTF-8, when a value is above U+007F, it will be > encoded with bytes that always have their high bit set. Aye. Design festure enabling easy resync-to-char-boundary at an arbitrary point in the file. > But Unicode has NEL "Next Line" U+0085 and other values that > conforming applications should recognize as line terminators. I disagree. Maybe for printing things. But textual data records? I would hope to end them with NL, and only NL (code 10). Cheers, Cameron Simpson From brent at brenthunter.tv Sun May 1 21:56:01 2022 From: brent at brenthunter.tv (Brent Hunter) Date: Mon, 2 May 2022 01:56:01 +0000 Subject: Difference in Setup Between Windows 10 Running Python 3.9 and Windows 11 Running Python 3.10 Message-ID: Hello, I was recently running a Windows 10 machine Python 3.9. I simply created a batch file titled "Start-AIG.bat" which simply contained the following: "pythonw AIG.py". It started a python program titled "AIG.py" and the Python dialog box was displayed on my screen, running all day and night. I set up Windows to run this batch file upon startup and it worked fine. I remember having to do a bunch of research before I learned that I needed to put "pythonw AIG.py" in the batch file as opposed to "python AIG.py". However, my old windows 10 desktop crashed and I have a new Windows 11 machine. I installed the latest stable version of Python, which is 3.10. Now, when I try to execute the same batch file, it doesn't launch the app. It could be because I'm using Windows 11 or could be because I now have a newer version of Python. Does anyone have any ideas what I should do to get the Python script running on my new machine? Thank you! Brent From python at mrabarnett.plus.com Sun May 1 22:16:09 2022 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 2 May 2022 03:16:09 +0100 Subject: Difference in Setup Between Windows 10 Running Python 3.9 and Windows 11 Running Python 3.10 In-Reply-To: References: Message-ID: On 2022-05-02 02:56, Brent Hunter wrote: > Hello, > > I was recently running a Windows 10 machine Python 3.9. I simply created a batch file titled "Start-AIG.bat" which simply contained the following: "pythonw AIG.py". It started a python program titled "AIG.py" and the Python dialog box was displayed on my screen, running all day and night. I set up Windows to run this batch file upon startup and it worked fine. I remember having to do a bunch of research before I learned that I needed to put "pythonw AIG.py" in the batch file as opposed to "python AIG.py". > > However, my old windows 10 desktop crashed and I have a new Windows 11 machine. I installed the latest stable version of Python, which is 3.10. Now, when I try to execute the same batch file, it doesn't launch the app. It could be because I'm using Windows 11 or could be because I now have a newer version of Python. > > Does anyone have any ideas what I should do to get the Python script running on my new machine? > The problem is probably that the Python folder isn't in Windows' search path, but the recommended thing to do nowadays on Windows is to use the Python Launcher "py.exe" (or "pyw.exe" for no console window) instead: pyw AIG.py From rosuav at gmail.com Sun May 1 22:44:19 2022 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 2 May 2022 12:44:19 +1000 Subject: tail In-Reply-To: References: Message-ID: On Mon, 2 May 2022 at 11:54, Cameron Simpson wrote: > > On 01May2022 23:30, Stefan Ram wrote: > >Dan Stromberg writes: > >>But what about Unicode? Are all 10 bytes newlines in Unicode encodings? > > It seems in UTF-8, when a value is above U+007F, it will be > > encoded with bytes that always have their high bit set. > > Aye. Design festure enabling easy resync-to-char-boundary at an > arbitrary point in the file. Yep - and there's also a distinction between "first byte of multi-byte character" and "continuation byte, keep scanning backwards". So you're guaranteed to be able to resynchronize. (If you know whether it's little-endian or big-endian, UTF-16 can also resync like that, since "high surrogate" and "low surrogate" look different.) > > But Unicode has NEL "Next Line" U+0085 and other values that > > conforming applications should recognize as line terminators. > > I disagree. Maybe for printing things. But textual data records? I would > hope to end them with NL, and only NL (code 10). > I'm with you on that - textual data records should end with 0x0A only. But if there are text entities in there, they should be allowed to include any Unicode characters, potentially including other types of whitespace. ChrisA From PythonList at DancesWithMice.info Mon May 2 04:33:02 2022 From: PythonList at DancesWithMice.info (dn) Date: Mon, 2 May 2022 20:33:02 +1200 Subject: Fwd: Python Question re Executing a Script (dn) In-Reply-To: References: Message-ID: <61055ef8-2765-c9bb-6182-60cb09b94075@DancesWithMice.info> Perhaps an MS-Win user can help the OP, please? -- Regards, =dn -------------- next part -------------- An embedded message was scrubbed... From: Brent Hunter Subject: RE: Python Question re Executing a Script (dn) Date: Mon, 2 May 2022 00:30:22 +0000 Size: 11109 URL: From rtm443x at googlemail.com Mon May 2 05:24:36 2022 From: rtm443x at googlemail.com (jan) Date: Mon, 2 May 2022 10:24:36 +0100 Subject: new sorting algorithm In-Reply-To: References: Message-ID: Hi, > The median-of-three partitioning technique makes that work reasonably well, so it won't be pathologically slow Just to be clear because I've wondered but haven't looked into it, we know naive quicksorting of already-sorted data is pathalogical, but median-of-3 is known to fix this pathology? cheers jan On 02/05/2022, Chris Angelico wrote: > On Mon, 2 May 2022 at 09:20, Dan Stromberg wrote: >> >> >> On Sun, May 1, 2022 at 1:44 PM Chris Angelico wrote: >>> >>> On Mon, 2 May 2022 at 06:43, Dan Stromberg wrote: >>> > On Sun, May 1, 2022 at 11:10 AM Chris Angelico >>> > wrote: >>> >> >>> >> On Mon, 2 May 2022 at 01:53, Nas Bayedil wrote: >>> >> > We believe that using this method to develop completely new, fast >>> >> > algorithms, approaching the speed of the famous *QuickSort*, the >>> >> > speed of >>> >> > which cannot be surpassed, but its drawback can be circumvented, in >>> >> > the >>> >> > sense of stack overflow, on some data. >>> >> >>> >> Hmm, actually TimSort *does* exceed the speed of quicksort for a lot >>> >> of real-world data. For instance, if you take a large sorted list, >>> >> append a handful of (unsorted) items to it, and then sort the list, >>> >> TimSort can take advantage of the fact that the bulk of the list is >>> >> sorted. It ends up significantly faster than re-sorting the entire >>> >> list. >>> > >>> > >>> > In fact, Timsort is O(n) for already-sorted data, while many quicksorts >>> > are O(n^2) for already-sorted data. >>> > >>> > Quicksort can be salvaged by using a median-of-3 partitioning, but it's >>> > still O(n^2) in the (less common) worst case. >>> > >>> >>> This is true, but purely sorted data isn't a very practical case. The >>> case of mostly-sorted data IS practical, though, so it's a quite big >>> win that it can be close to O(n), and still faster than inserting each >>> item individually. >> >> >> You seem to be of the impression that nearly-sorted data isn't an uphill >> battle with a straightforward quicksort. >> >> I'm having a hard time convincing myself of that. >> > > The median-of-three partitioning technique makes that work reasonably > well, so it won't be pathologically slow. It's hardly Quicksort's best > feature, but it could easily be a lot worse. I'd have to check, but I > think it still manages to be O(n log n). Merge sort, of course, is a > lot more consistent, but the asymptotic cost is still broadly the > same. > > But Timsort manages to be close to O(n) for sorted data, reversed > data, nearly-sorted or nearly-reversed data, etc. Makes it very handy > for jobs like "click a heading to sort by it", where you might add > multiple sort keys. > > (Plus, Python's implementation has some cool tricks for small > collections that make it quite efficient.) > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > From chottel at earthlink.net Mon May 2 09:09:48 2022 From: chottel at earthlink.net (charles hottel) Date: Mon, 2 May 2022 09:09:48 -0400 Subject: new sorting algorithm In-Reply-To: References: Message-ID: <6f6c624f-cc97-300a-1234-d6a0cf1d5d3b@earthlink.net> On 5/1/2022 7:45 PM, Chris Angelico wrote: > On Mon, 2 May 2022 at 09:20, Dan Stromberg wrote: >> >> >> On Sun, May 1, 2022 at 1:44 PM Chris Angelico wrote: >>> >>> On Mon, 2 May 2022 at 06:43, Dan Stromberg wrote: >>>> On Sun, May 1, 2022 at 11:10 AM Chris Angelico wrote: >>>>> >>>>> On Mon, 2 May 2022 at 01:53, Nas Bayedil wrote: >>>>>> We believe that using this method to develop completely new, fast >>>>>> algorithms, approaching the speed of the famous *QuickSort*, the speed of >>>>>> which cannot be surpassed, but its drawback can be circumvented, in the >>>>>> sense of stack overflow, on some data. >>>>> >>>>> Hmm, actually TimSort *does* exceed the speed of quicksort for a lot >>>>> of real-world data. For instance, if you take a large sorted list, >>>>> append a handful of (unsorted) items to it, and then sort the list, >>>>> TimSort can take advantage of the fact that the bulk of the list is >>>>> sorted. It ends up significantly faster than re-sorting the entire >>>>> list. >>>> >>>> >>>> In fact, Timsort is O(n) for already-sorted data, while many quicksorts are O(n^2) for already-sorted data. >>>> >>>> Quicksort can be salvaged by using a median-of-3 partitioning, but it's still O(n^2) in the (less common) worst case. >>>> >>> >>> This is true, but purely sorted data isn't a very practical case. The >>> case of mostly-sorted data IS practical, though, so it's a quite big >>> win that it can be close to O(n), and still faster than inserting each >>> item individually. >> >> >> You seem to be of the impression that nearly-sorted data isn't an uphill battle with a straightforward quicksort. >> >> I'm having a hard time convincing myself of that. >> > > The median-of-three partitioning technique makes that work reasonably > well, so it won't be pathologically slow. It's hardly Quicksort's best > feature, but it could easily be a lot worse. I'd have to check, but I > think it still manages to be O(n log n). Merge sort, of course, is a > lot more consistent, but the asymptotic cost is still broadly the > same. > > But Timsort manages to be close to O(n) for sorted data, reversed > data, nearly-sorted or nearly-reversed data, etc. Makes it very handy > for jobs like "click a heading to sort by it", where you might add > multiple sort keys. > > (Plus, Python's implementation has some cool tricks for small > collections that make it quite efficient.) > > ChrisA Some versions of Quicksort switch over to Straight Insertion Sort when the partitions become small enough. The correct size will vary depending on the hardware. I have not kept up with the latest improvements and I am not familiar with TimSort. However Heapsort should always be O(n log n) and there are modifications to Heapsort that can make it faster than vanilla Heapsort and closer to the speed of Quicksort. From wlfraed at ix.netcom.com Mon May 2 12:23:58 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Mon, 02 May 2022 12:23:58 -0400 Subject: Fwd: Python Question re Executing a Script (dn) References: <61055ef8-2765-c9bb-6182-60cb09b94075@DancesWithMice.info> Message-ID: <0euv6hdl5ccumftc56te70boi68b6r991a@4ax.com> On Mon, 2 May 2022 20:33:02 +1200, dn declaimed the following: >Perhaps an MS-Win user can help the OP, please? Not really -- at least from my viewpoint there is not enough information to perform any diagnoses... Other than to recommend that, if the OP initially posted to the list/newsgroup, they should ensure any replies also go to the same (I don't respond to what should be group replies that appear in my personal mail -- which is what the OP did for me also). {Hmmm, appears the quoted content is below your signature block, and my client doesn't quote standard signatures and below... time for some nasty cut&paste} >From: Brent Hunter >Date: Mon, 2 May 2022 00:30:22 +0000 > >I was recently running a Windows 10 machine and I believe it was running Python 3.8. All I did was create a batch file titled Start-AIG.bat which simply contained the following: "pythonw AIG.py". It started a python program titled "AIG.py" and the Python dialog box was displayed on my screen, running all day and night. I set up Windows to run this batch file upon startup and it worked fine. I remember having to do a bunch of research before I learned that I needed to put "pythonw AIG.py" in the batch file as opposed to "python AIG.py". > What in particular is a "Python dialog box"? Python doesn't, of its own, have dialog boxes. Add-on libraries: Tkinter, wxPython, the win32 extensions, et al. allow one to code scripts that produce dialogs, but the dialogs are specific to the library. "pythonw" is the name of the interpreter meant for use with programs that do not use console I/O -- IOW, pure GUI (one of the aforesaid libraries). Normally such programs are given the extension .pyw rather than .py -- and (at least on my system), .pyw files are automatically associated with pythonw. Just providing the script name with extension is sufficient. Though there is the matter that giving said name in a command line prompt means one still has a console active. .pyw files are meant to be started from the file explorer by double-clicking on them, and one doesn't want to have a console window popping up if it is never used (which is what happens if one double-clicks on a plain .py file). HOW did you "set up Windows to run this batch file upon startup"? Windows used to support (and still does if https://www.thewindowsclub.com/startup-folder-in-windows-8 is correct [ignore the -8, the site is updated for 10/11]). This is per user start up (unless one uses the "all users" path). The Windows task scheduler also allows one to create tasks that run on a schedule -- said schedule including options for specific user logging in, any user logging in, computer start-up, etc. The scheduler is probably the recommended way to set up things. >However, my old windows 10 desktop crashed and I have a new Windows 11 machine. I installed the latest stable version of Python, which is 3.10. Now, when I try to execute the same batch file, it doesn't launch the app. It could be because I'm using Windows 11 or could be because I now have a newer version of Python. > >With this additional information, do you have any ideas what I should do to get the Python script running in the same way I had it running before? As I stated above -- insufficient information. Is Python installed for all users or just one? Is "pythonw" on the system PATH? Is the script meant to only run for one user or any user? How was the script configured to start? Is the script in a valid directory (I believe in a previous post it was mentioned that you had the script in the directory used for per user START-MENU entries; that is not meant for user files!). My first approach would be to open a console and run the script manually (not the BAT file) -- does any error message appear in the console? Running using pythonw suppresses console, error messages would be lost -- GUI applications are expected to trap any runtime errors and either write to a log file or pop-up a dialog for them. Then, after getting it to run in a console, I'd try it by double-clicking on the file (rename it to .pyw so you get the no-console interpreter). If Windows says it doesn't know how to run .pyw, it implies that there is no file association set up in Windows telling it how to run that extension. NO BAT file should be required for a one-liner that does no parameter substitution. If using the "startup" directory method, create a shortcut for the script (right-click, drag, create shortcut here). Right-click/Properties. If you don't have associations set to run that extension type, prefix the Target box with the path to the python(w) executable. Move the shortcut to the proper startup directory -- Note that if you put it in the all-users startup, you have to make sure that the script and shortcut themselves are not in protected (single user) locations. Using the task scheduler, you'd have similar constraints on directories, but shouldn't need to create a shortcut file. Instead, you'd specify the path to the interpreter and the path to the script as the task ACTION, with the applicable start option as the TRIGGER. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From python at mrabarnett.plus.com Mon May 2 13:15:32 2022 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 2 May 2022 18:15:32 +0100 Subject: Fwd: Python Question re Executing a Script (dn) In-Reply-To: <0euv6hdl5ccumftc56te70boi68b6r991a@4ax.com> References: <61055ef8-2765-c9bb-6182-60cb09b94075@DancesWithMice.info> <0euv6hdl5ccumftc56te70boi68b6r991a@4ax.com> Message-ID: On 2022-05-02 17:23, Dennis Lee Bieber wrote: > On Mon, 2 May 2022 20:33:02 +1200, dn > declaimed the following: > >>Perhaps an MS-Win user can help the OP, please? > > Not really -- at least from my viewpoint there is not enough > information to perform any diagnoses... Other than to recommend that, if > the OP initially posted to the list/newsgroup, they should ensure any > replies also go to the same (I don't respond to what should be group > replies that appear in my personal mail -- which is what the OP did for me > also). > > {Hmmm, appears the quoted content is below your signature block, and my > client doesn't quote standard signatures and below... time for some nasty > cut&paste} > >>From: Brent Hunter >>Date: Mon, 2 May 2022 00:30:22 +0000 >> > >>I was recently running a Windows 10 machine and I believe it was running Python 3.8. All I did was create a batch file titled Start-AIG.bat which simply contained the following: "pythonw AIG.py". It started a python program titled "AIG.py" and the Python dialog box was displayed on my screen, running all day and night. I set up Windows to run this batch file upon startup and it worked fine. I remember having to do a bunch of research before I learned that I needed to put "pythonw AIG.py" in the batch file as opposed to "python AIG.py". >> > [snip] FTR, I've already posted the suggestion to try the Python Launcher "pyw", and the OP replied off-list that it worked. From Marco.Sulla.Python at gmail.com Mon May 2 14:36:41 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Mon, 2 May 2022 20:36:41 +0200 Subject: tail In-Reply-To: References: Message-ID: On Mon, 2 May 2022 at 18:31, Stefan Ram wrote: > > |The Unicode standard defines a number of characters that > |conforming applications should recognize as line terminators:[7] > | > |LF: Line Feed, U+000A > |VT: Vertical Tab, U+000B > |FF: Form Feed, U+000C > |CR: Carriage Return, U+000D > |CR+LF: CR (U+000D) followed by LF (U+000A) > |NEL: Next Line, U+0085 > |LS: Line Separator, U+2028 > |PS: Paragraph Separator, U+2029 > | > Wikipedia "Newline". Should I suppose that other encodings may have more line ending chars? From rosuav at gmail.com Mon May 2 14:42:24 2022 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 3 May 2022 04:42:24 +1000 Subject: tail In-Reply-To: References: Message-ID: On Tue, 3 May 2022 at 04:38, Marco Sulla wrote: > > On Mon, 2 May 2022 at 18:31, Stefan Ram wrote: > > > > |The Unicode standard defines a number of characters that > > |conforming applications should recognize as line terminators:[7] > > | > > |LF: Line Feed, U+000A > > |VT: Vertical Tab, U+000B > > |FF: Form Feed, U+000C > > |CR: Carriage Return, U+000D > > |CR+LF: CR (U+000D) followed by LF (U+000A) > > |NEL: Next Line, U+0085 > > |LS: Line Separator, U+2028 > > |PS: Paragraph Separator, U+2029 > > | > > Wikipedia "Newline". > > Should I suppose that other encodings may have more line ending chars? No, because those are Unicode characters. How they're encoded may affect the bytes you see, but those are code point values after decoding. ChrisA From Marco.Sulla.Python at gmail.com Mon May 2 15:25:00 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Mon, 2 May 2022 21:25:00 +0200 Subject: tail In-Reply-To: References: Message-ID: Ok, I suppose \n and \r are enough: ######## readline(size=- 1, /) Read and return one line from the stream. If size is specified, at most size bytes will be read. The line terminator is always b'\n' for binary files; for text files, the newline argument to open() can be used to select the line terminator(s) recognized. ######## open(file, mode='r', buffering=- 1, encoding=None, errors=None, newline=None, closefd=True, opener=None) [...] newline controls how universal newlines mode works (it only applies to text mode). It can be None, '', '\n', '\r', and '\r\n' ######## From mats at wichmann.us Mon May 2 15:18:36 2022 From: mats at wichmann.us (Mats Wichmann) Date: Mon, 2 May 2022 13:18:36 -0600 Subject: new sorting algorithm In-Reply-To: <6f6c624f-cc97-300a-1234-d6a0cf1d5d3b@earthlink.net> References: <6f6c624f-cc97-300a-1234-d6a0cf1d5d3b@earthlink.net> Message-ID: On 5/2/22 07:09, charles hottel wrote: > Some versions of Quicksort switch over to Straight Insertion Sort when > the partitions become small enough. The correct size will vary depending > on the hardware. > > I have not kept up with the latest improvements and I am not familiar > with TimSort.? However Heapsort? should always be O(n log n) and there > are modifications to Heapsort that can make it faster than vanilla > Heapsort and closer to the speed of Quicksort. A quick read might be in order then... https://en.wikipedia.org/wiki/Timsort From Marco.Sulla.Python at gmail.com Mon May 2 15:40:26 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Mon, 2 May 2022 21:40:26 +0200 Subject: tail In-Reply-To: References: Message-ID: On Mon, 2 May 2022 at 00:20, Cameron Simpson wrote: > > On 01May2022 18:55, Marco Sulla wrote: > >Something like this is OK? > [...] > >def tail(f): > > chunk_size = 100 > > size = os.stat(f.fileno()).st_size > > I think you want os.fstat(). It's the same from py 3.3 > > chunk_line_pos = -1 > > pos = 0 > > > > for pos in positions: > > f.seek(pos) > > chars = f.read(chunk_size) > > chunk_line_pos = chars.rfind(b"\n") > > > > if chunk_line_pos != -1: > > break > > Normal text file _end_ in a newline. I'd expect this to stop immediately > at the end of the file. I think it's correct. The last line in this case is an empty bytes. > > if chunk_line_pos == -1: > > nbytes = pos > > pos = 0 > > f.seek(pos) > > chars = f.read(nbytes) > > chunk_line_pos = chars.rfind(b"\n") > > I presume this is because unless you're very lucky, 0 will not be a > position in the range(). I'd be inclined to avoid duplicating this code > and special case and instead maybe make the range unbounded and do > something like this: > > if pos < 0: > pos = 0 > ... seek/read/etc ... > if pos == 0: > break > > around the for-loop body. Yes, I was not very happy to duplicate the code... I have to think about it. > Seems sane. I haven't tried to run it. Thank you ^^ From as4935 at srmist.edu.in Mon May 2 14:01:35 2022 From: as4935 at srmist.edu.in (ARRYAN SINHA (RA1811029010036)) Date: Mon, 2 May 2022 11:01:35 -0700 (PDT) Subject: error of opening Python In-Reply-To: References: Message-ID: <5067416d-7214-4808-9f2b-1430a2eefdean@googlegroups.com> On Friday, April 15, 2022 at 2:17:28 AM UTC+5:30, Andrew Hernandez wrote: > that is not an error, its simply the python console intrepeter how do I open this file From drsalists at gmail.com Mon May 2 22:11:19 2022 From: drsalists at gmail.com (Dan Stromberg) Date: Mon, 2 May 2022 19:11:19 -0700 Subject: new sorting algorithm In-Reply-To: References: Message-ID: On Mon, May 2, 2022 at 2:25 AM jan via Python-list wrote: > Hi, > > > The median-of-three partitioning technique makes that work reasonably > well, so it won't be pathologically slow > > Just to be clear because I've wondered but haven't looked into it, we > know naive quicksorting of already-sorted data is pathalogical, but > median-of-3 is known to fix this pathology? > Median-of-3 helps, but it's still possible to construct inputs that make quicksort break down to an O(n^2) algorithm in the worst case. These inputs are much less common than sorting an already sorted, or reverse-sorted list. From nouloug2 at gmail.com Tue May 3 05:31:13 2022 From: nouloug2 at gmail.com (Nicolas Formichella) Date: Tue, 3 May 2022 02:31:13 -0700 (PDT) Subject: [winreg] How to access "(Default)" key value? Message-ID: <8cd15ae0-e6d9-478c-a4be-c354d221b07dn@googlegroups.com> Hello, I am encountering an issue with winreg, I can't figure out how to read the "(Default)" of a registry value ``` def get_chrome_version() -> str: with winreg.ConnectRegistry(None, winreg.HKEY_CURRENT_USER) as registry: with winreg.OpenKeyEx(registry, r"Software\Microsoft\Windows\CurrentVersion\App Paths\chrome.exe", access=winreg.KEY_READ) as key: value, _ = winreg.QueryValueEx(key, "(default)") return value ``` Taking the example above it raises a FileNotFoundError : ``` Traceback (most recent call last): File "C:/Users/noulo/dev/py/chromedriver_dl/main.py", line 14, in get_chrome_version value, _ = winreg.QueryValueEx(key, "(default)") FileNotFoundError: [WinError 2] The system cannot find the file specified ``` (Trying "Path" instead of "(Default)" works) Although the value definitely exists as shown by Powershell ``` Get-Item -Path "HKCU:/Software/Microsoft/Windows/CurrentVersion/App Paths/chrome.exe" | Select-Object -ExpandProperty Property ``` Returns : ``` (default) Path ``` Is this a bug I should report, or if not, what is the way, using winreg, to access the "(Default)" value of a key? Regards, Nicolas FORMICHELLA From loris.bennett at fu-berlin.de Tue May 3 07:37:58 2022 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Tue, 03 May 2022 13:37:58 +0200 Subject: Match.groupdict: Meaning of default argument? References: <877d78z5us.fsf@hornfels.zedat.fu-berlin.de> Message-ID: <8735hqsv6x.fsf@hornfels.zedat.fu-berlin.de> ram at zedat.fu-berlin.de (Stefan Ram) writes: > "Loris Bennett" writes: >>I thought that 'days' would default to '0'. > > It will get the value '0' if (?P\d*) does > /not/ participate in the match. > > In your case, it /does/ participate in the match, > \d* matching the empty string. > > Try (?P\d+)?. Ah, thanks. I was misunderstanding the meaning of 'participate'. Cheers, Loris -- This signature is currently under construction. From loris.bennett at fu-berlin.de Tue May 3 07:49:47 2022 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Tue, 03 May 2022 13:49:47 +0200 Subject: Match.groupdict: Meaning of default argument? References: <877d78z5us.fsf@hornfels.zedat.fu-berlin.de> <8735hqsv6x.fsf@hornfels.zedat.fu-berlin.de> Message-ID: <87v8umrg2s.fsf@hornfels.zedat.fu-berlin.de> "Loris Bennett" writes: > ram at zedat.fu-berlin.de (Stefan Ram) writes: > >> "Loris Bennett" writes: >>>I thought that 'days' would default to '0'. >> >> It will get the value '0' if (?P\d*) does >> /not/ participate in the match. >> >> In your case, it /does/ participate in the match, >> \d* matching the empty string. >> >> Try (?P\d+)?. > > Ah, thanks. I was misunderstanding the meaning of 'participate'. What I actually need is ((?P\d+)(-?))?(?P\d\d):(?P\d\d):(?P\d\d) so that I can match both 99-11:22:33 and 11:22:33 and have 'days' be '0' in the later case. Thanks for pointing me in the right direction. Cheers, Loris -- This signature is currently under construction. From loris.bennett at fu-berlin.de Tue May 3 10:43:48 2022 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Tue, 03 May 2022 16:43:48 +0200 Subject: Match.groupdict: Meaning of default argument? References: <877d78z5us.fsf@hornfels.zedat.fu-berlin.de> <22f9a92c-3117-4f6a-820e-b7f3c44a655cn@googlegroups.com> Message-ID: <87k0b2n0bf.fsf@hornfels.zedat.fu-berlin.de> Julio Di Egidio writes: > On Friday, 29 April 2022 at 09:50:08 UTC+2, Loris Bennett wrote: >> Hi, >> >> If I do >> >> import re >> pattern = re.compile(r'(?P\d*)(-?)(?P\d\d):(?P\d\d):(?P\d\d)') >> s = '104-02:47:06' >> match = pattern.search(s) >> match_dict = match.groupdict('0') >> >> I get >> >> match_dict >> {'days': '104', 'hours': '02', 'minutes': '47', 'seconds': '06'} >> >> However, if the string has no initial part (corresponding to the number of >> days), e.g. >> >> s = '02:47:06' >> match = pattern.search(s) >> match_dict = match.groupdict('0') >> >> I get >> >> match_dict >> {'days': '', 'hours': '02', 'minutes': '47', 'seconds': '06'} >> >> I thought that 'days' would default to '0'. >> >> What am I doing wrong? > > You tell, but it's quite obvious that you (just) run a regex on a string and captures are going to be strings: indeed, '02' is not a number either... > > Julio I am not sure what you are trying to tell me. I wasn't expecting anything other than strings. The problem was, as Stefan helped me to understand, that I misunderstood what 'participating in the match' means. Cheers, Loris -- This signature is currently under construction. From python at mrabarnett.plus.com Tue May 3 12:46:12 2022 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 3 May 2022 17:46:12 +0100 Subject: [winreg] How to access "(Default)" key value? In-Reply-To: <8cd15ae0-e6d9-478c-a4be-c354d221b07dn@googlegroups.com> References: <8cd15ae0-e6d9-478c-a4be-c354d221b07dn@googlegroups.com> Message-ID: On 2022-05-03 10:31, Nicolas Formichella wrote: > Hello, > > I am encountering an issue with winreg, I can't figure out how to read the "(Default)" of a registry value > > ``` > def get_chrome_version() -> str: > with winreg.ConnectRegistry(None, winreg.HKEY_CURRENT_USER) as registry: > with winreg.OpenKeyEx(registry, r"Software\Microsoft\Windows\CurrentVersion\App Paths\chrome.exe", > access=winreg.KEY_READ) as key: > value, _ = winreg.QueryValueEx(key, "(default)") > return value > ``` > > Taking the example above it raises a FileNotFoundError : > > ``` > Traceback (most recent call last): > File "C:/Users/noulo/dev/py/chromedriver_dl/main.py", line 14, in get_chrome_version > value, _ = winreg.QueryValueEx(key, "(default)") > FileNotFoundError: [WinError 2] The system cannot find the file specified > ``` > > (Trying "Path" instead of "(Default)" works) > > Although the value definitely exists as shown by Powershell > > ``` > Get-Item -Path "HKCU:/Software/Microsoft/Windows/CurrentVersion/App Paths/chrome.exe" | Select-Object -ExpandProperty Property > ``` > > Returns : > > ``` > (default) > Path > ``` > > Is this a bug I should report, or if not, what is the way, using winreg, to access the "(Default)" value of a key? > Use 'QueryValue' with an empty string: value = winreg.QueryValue(key, "") or None: value = winreg.QueryValue(key, None) From davebromoils at gmail.com Tue May 3 15:29:05 2022 From: davebromoils at gmail.com (Dave Francis) Date: Tue, 3 May 2022 20:29:05 +0100 Subject: python 3.9.12 and python 3.10 Message-ID: Date: Tue, 3 May 2022 19:59:08 +0100 Subject: python 3.9.12 and python 3.10 I installed 3.9 and am able to enter simple code which runs. If I type an error I can highlight it but cannot copy or delete it. I looked for help & found 3.10 and installed it but when I selected the python icon 3.9 appeared with my old errors. I uninstalled 3.10 as 3.9 did not appear in control panel. Dell Inspiron 3793 Win 10 Thanks Dave Francis From eryksun at gmail.com Tue May 3 17:03:43 2022 From: eryksun at gmail.com (Eryk Sun) Date: Tue, 3 May 2022 16:03:43 -0500 Subject: Difference in Setup Between Windows 10 Running Python 3.9 and Windows 11 Running Python 3.10 In-Reply-To: References: Message-ID: On 5/1/22, Brent Hunter wrote: > > I was recently running a Windows 10 machine Python 3.9. I simply created a > batch file titled "Start-AIG.bat" which simply contained the following: > "pythonw AIG.py". It started a python program titled "AIG.py" and the > Python dialog box was displayed on my screen, running all day and night. I > set up Windows to run this batch file upon startup and it worked fine. I > remember having to do a bunch of research before I learned that I needed to > put "pythonw AIG.py" in the batch file as opposed to "python AIG.py". When running a command at startup, it's best to use the full path of the application and the full path of any files passed on the command line. For example, run something like the following: "path\to\pythonw.exe" "path\to\AIG.py" This removes any dependency on the search PATH or the current working directory. Also, the script shouldn't depend on the initial working directory. Any files that it needs should be in a well-known location, such as the script directory, os.path.dirname(os.path.abspath(__file__)). Also, there's no need to use a batch script just to run a single command. When you use a batch script, a visible console gets created for CMD. It doesn't close until CMD exits, which in this case won't be until pythonw.exe exits, unless you use `START` to run the command without waiting. It's simpler to just run the command directly using a shortcut (i.e. LNK file) or the task scheduler. From killerkomando12 at gmail.com Wed May 4 21:36:04 2022 From: killerkomando12 at gmail.com (Patrick 0511) Date: Wed, 4 May 2022 18:36:04 -0700 (PDT) Subject: Python/New/Learn Message-ID: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> Hello, I'm completely new here and don't know anything about python. Can someone tell me how best to start? So what things should I learn first? From rosuav at gmail.com Wed May 4 22:51:54 2022 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 May 2022 12:51:54 +1000 Subject: Python/New/Learn In-Reply-To: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> Message-ID: On Thu, 5 May 2022 at 12:49, Patrick 0511 wrote: > > Hello, I'm completely new here and don't know anything about python. Can someone tell me how best to start? So what things should I learn first? > I'd start right here with the tutorial! https://docs.python.org/3/tutorial/ Most important thing is: play around. Have fun. Explore! You can't really go all that far wrong, so don't be afraid to try things, even "silly" things that "don't make any sense", because trying those things is part of learning how the language works. (And you never know - sometimes you might find that it actually does work.) Oh, although, if you don't currently have Python installed, that would be a place to start. ChrisA From avigross at verizon.net Wed May 4 22:56:09 2022 From: avigross at verizon.net (Avi Gross) Date: Thu, 5 May 2022 02:56:09 +0000 (UTC) Subject: Python/New/Learn In-Reply-To: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> Message-ID: <1494828175.2619918.1651719369705@mail.yahoo.com> https://wiki.python.org/moin/PythonBooks -----Original Message----- From: Patrick 0511 To: python-list at python.org Sent: Wed, May 4, 2022 9:36 pm Subject: Python/New/Learn Hello, I'm completely new here and don't know anything about python. Can someone tell me how best to start? So what things should I learn first? -- https://mail.python.org/mailman/listinfo/python-list From rosuav at gmail.com Wed May 4 23:02:31 2022 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 May 2022 13:02:31 +1000 Subject: Python/New/Learn In-Reply-To: <1494828175.2619918.1651719369705@mail.yahoo.com> References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> <1494828175.2619918.1651719369705@mail.yahoo.com> Message-ID: On Thu, 5 May 2022 at 12:57, Avi Gross via Python-list wrote: > > https://wiki.python.org/moin/PythonBooks > That's an incredibly daunting list, and not something I'd overly strongly recommend, but yes, if you want to get a dead-tree or e-book to read, there are quite a lot of options available. ChrisA From drsalists at gmail.com Wed May 4 23:05:59 2022 From: drsalists at gmail.com (Dan Stromberg) Date: Wed, 4 May 2022 20:05:59 -0700 Subject: Python/New/Learn In-Reply-To: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> Message-ID: If you already know at least one other imperative programming language: https://wiki.python.org/moin/BeginnersGuide/Programmers If you don't: https://wiki.python.org/moin/BeginnersGuide/NonProgrammers On Wed, May 4, 2022 at 7:49 PM Patrick 0511 wrote: > Hello, I'm completely new here and don't know anything about python. Can > someone tell me how best to start? So what things should I learn first? > -- > https://mail.python.org/mailman/listinfo/python-list > From avigross at verizon.net Wed May 4 23:14:18 2022 From: avigross at verizon.net (Avi Gross) Date: Thu, 5 May 2022 03:14:18 +0000 (UTC) Subject: Python/New/Learn In-Reply-To: References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> <1494828175.2619918.1651719369705@mail.yahoo.com> Message-ID: <489353065.2659649.1651720458270@mail.yahoo.com> Chris, It was an extremely open-ended question to a forum where?most of the readers are more advanced, at least I think. My library has oodles of Python Books for free to borrow on paper and?return and I have read many of them. There are various e-books too, and?of course lots of free internet resources including videos and on-line courses. If he wants simpler books, the web pages pointed here too: https://wiki.python.org/moin/IntroductoryBooks Next time, I won't try to be helpful and brief and just be silent. -----Original Message----- From: Chris Angelico To: python-list at python.org Sent: Wed, May 4, 2022 11:02 pm Subject: Re: Python/New/Learn On Thu, 5 May 2022 at 12:57, Avi Gross via Python-list wrote: > > https://wiki.python.org/moin/PythonBooks > That's an incredibly daunting list, and not something I'd overly strongly recommend, but yes, if you want to get a dead-tree or e-book to read, there are quite a lot of options available. ChrisA -- https://mail.python.org/mailman/listinfo/python-list From rosuav at gmail.com Wed May 4 23:21:28 2022 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 May 2022 13:21:28 +1000 Subject: Python/New/Learn In-Reply-To: <489353065.2659649.1651720458270@mail.yahoo.com> References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> <1494828175.2619918.1651719369705@mail.yahoo.com> <489353065.2659649.1651720458270@mail.yahoo.com> Message-ID: On Thu, 5 May 2022 at 13:14, Avi Gross wrote: > > Chris, > > It was an extremely open-ended question to a forum where > most of the readers are more advanced, at least I think. > > > My library has oodles of Python Books for free to borrow on paper and > return and I have read many of them. There are various e-books too, and > of course lots of free internet resources including videos and on-line courses. > > > If he wants simpler books, the web pages pointed here too: > > > https://wiki.python.org/moin/IntroductoryBooks > > > Next time, I won't try to be helpful and brief and just be silent. > Being helpful is great, it's just that being brief can leave it as an incredibly scary-looking list :) If you want to recommend a couple of specific books, I think that would be a lot more helpful. ChrisA From avigross at verizon.net Wed May 4 23:46:54 2022 From: avigross at verizon.net (Avi Gross) Date: Thu, 5 May 2022 03:46:54 +0000 (UTC) Subject: Python/New/Learn In-Reply-To: References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> <1494828175.2619918.1651719369705@mail.yahoo.com> <489353065.2659649.1651720458270@mail.yahoo.com> Message-ID: <112563898.2665775.1651722414150@mail.yahoo.com> I agree Chris that the Ukrainian Python Books are daunting as I barely started learning?that language now even though my early years were just a few miles away and I might?even have relatives still there! But as has been pointed out, suggestions are more helpful if you know a bit more about the?one asking the questions or it is too broad. I do not feel qualified to suggest a book to true?beginners as I leaned Python many decades after learning so many other computer languages?and read a wide variety of books including many that assumed you already knew C or R or other?languages, as I did.? Someone new to programming may fin some resources including a tutorial handy. Someone who actually?wants to use various modules like scikit-learn or pandas, or to solve specific problems, might well want to?go straight to other resources or realize they need multiple resources over time. It can be a diversion to get someone to learn the functional programming aspects or even object-oriented?if the goal is to do simple procedural things. Python is an extremely rich language and I recall reading a?three-volume encyclopedia that took months as the books got pulled away by other readers.?So, no Mark Lutz got a bit advanced. Oddly, I am learning Julia now and cannot find a single book in my group of Libraries, not even for the?beginner I am not! But finding online resources was easy and if I have any reason to, plenty of?books can be bought. It does not work for everybody, but my personal method is to attack issues and?problems from multiple directions as each tends to reinforce and add to my knowledge. And I really?appreciate when they tell you specifically how the language is different from others you may know so?you know when not to expect it to work. A Julia documentation as an example, has a long list of places it is?not like Python, or R or C++ and so on. If the person has computer language experience, some such?resource might be better than something spending a thousand pages to teach from scratch. But, I am not volunteering to do personal tutoring. I prefer responding to specific focused questions?especially after the person seems to have done some searching and research and reading on their own?and maybe even shares some code and asks what may be wrong with it or ... And my first question would be why they chose to ask about Python. Was it their choice or required?for a course or job or ... Sometimes the answer is to use something else they already know, albeit Python is a very decent language?for many uses and well-worth learning even if you know others. -----Original Message----- From: Chris Angelico To: Avi Gross Cc: python-list at python.org Sent: Wed, May 4, 2022 11:21 pm Subject: Re: Python/New/Learn On Thu, 5 May 2022 at 13:14, Avi Gross wrote: > > Chris, > > It was an extremely open-ended question to a forum where > most of the readers are more advanced, at least I think. > > > My library has oodles of Python Books for free to borrow on paper and > return and I have read many of them. There are various e-books too, and > of course lots of free internet resources including videos and on-line courses. > > > If he wants simpler books, the web pages pointed here too: > > > https://wiki.python.org/moin/IntroductoryBooks > > > Next time, I won't try to be helpful and brief and just be silent. > Being helpful is great, it's just that being brief can leave it as an incredibly scary-looking list :) If you want to recommend a couple of specific books, I think that would be a lot more helpful. ChrisA From learn2program at gmail.com Thu May 5 03:01:12 2022 From: learn2program at gmail.com (Alan Gauld) Date: Thu, 5 May 2022 08:01:12 +0100 Subject: Python/New/Learn In-Reply-To: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> Message-ID: <6bbae59e-2c66-1803-f5f2-c422e5773e6a@gmail.com> On 05/05/2022 02:36, Patrick 0511 wrote: > Hello, I'm completely new here and don't know anything about python. Do you know any other programming languages? That makes a huge difference in what you should start with! > Can someone tell me how best to start? > So what things should I learn first? Others have already made recommendations. I'll add that there is a dedicated python ttutor list for those just learning to ask questions. It is slightly more tolerant of "silly" questions and rookie mistakes than the main list here. You'll also find most of the discussions there will be closer to your level than the more advanced topics that get addressed here. Finally, I have a tutorial aimed at complete beginners, see the link below... -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos From Joseph.Schachner at Teledyne.com Thu May 5 12:04:59 2022 From: Joseph.Schachner at Teledyne.com (Schachner, Joseph) Date: Thu, 5 May 2022 16:04:59 +0000 Subject: Python/New/Learn In-Reply-To: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> Message-ID: Buy the book "Python 101" and do the examples. When you're done with that buy the book "Python 201" and study it. There is much more than is in both those books that you could learn about Python, but that's a very good way to start. --- Joseph S. Teledyne Confidential; Commercially Sensitive Business Data -----Original Message----- From: Patrick 0511 Sent: Wednesday, May 4, 2022 9:36 PM To: python-list at python.org Subject: Python/New/Learn Hello, I'm completely new here and don't know anything about python. Can someone tell me how best to start? So what things should I learn first? From mats at wichmann.us Thu May 5 13:59:10 2022 From: mats at wichmann.us (Mats Wichmann) Date: Thu, 5 May 2022 11:59:10 -0600 Subject: Python/New/Learn In-Reply-To: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> Message-ID: On 5/4/22 19:36, Patrick 0511 wrote: > Hello, I'm completely new here and don't know anything about python. Can someone tell me how best to start? So what things should I learn first? If you know what kinds of learning experiences work best for you, that might help. For some people, books, or written tutorials are effective. For some, it's just dive in and start doing projects, referring to, say, StackOverflow when you get stuck. Without having any data at all on it, just my impressions, more people these days learn from in-person or video experiences. There are several free video courses (see for example, without any implied endorsement, https://www.freecodecamp.org/), and some that cost but not very expensive. Depends on what floats your boat. From avigross at verizon.net Thu May 5 19:48:03 2022 From: avigross at verizon.net (Avi Gross) Date: Thu, 5 May 2022 23:48:03 +0000 (UTC) Subject: Python/New/Learn In-Reply-To: References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> Message-ID: <1287201950.49622.1651794483723@mail.yahoo.com> Before more people reply to this user, I note I have not seen them reply back to the list?about any questions or comments others have taken the time to provide. My warning bells go off when I see patterns and there was a similar request from another gmail?account to an R language forum I am also on. They wanted help in learning programming (especially R)?and claimed not to have any source to study. As we keep pointing out, you can trivially find sources including?many free ones. So I wonder if there is? point being sucked in by one or more people who don't even continue a conversation?and perhaps are not even listening but just playing us to make us waste time. Or, maybe they are asking?on multiple places to decide which to choose and are not saying so. Am I paranoid? Nah! But yes, a bit wary. I get so many kinds of SPAM in mail and phone calls and lately?keep getting calls asking if I want to sell my house ... -----Original Message----- From: Schachner, Joseph To: Patrick 0511 ; python-list at python.org Sent: Thu, May 5, 2022 12:04 pm Subject: RE: Python/New/Learn Buy the book "Python 101" and do the examples.? When you're done with that buy the book "Python 201" and study it.? There is much more than is in both those books that you could learn about Python, but that's a very good way to start. --- Joseph S. Teledyne Confidential; Commercially Sensitive Business Data -----Original Message----- From: Patrick 0511 Sent: Wednesday, May 4, 2022 9:36 PM To: python-list at python.org Subject: Python/New/Learn Hello, I'm completely new here and don't know anything about python. Can someone tell me how best to start? So what things should I learn first? -- https://mail.python.org/mailman/listinfo/python-list From grant.b.edwards at gmail.com Thu May 5 19:51:49 2022 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Thu, 05 May 2022 16:51:49 -0700 (PDT) Subject: Python/New/Learn References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> Message-ID: <62746315.1c69fb81.9502b.350f@mx.google.com> On 2022-05-05, Mats Wichmann wrote: > Without having any data at all on it, just my impressions, more > people these days learn from in-person or video experiences. I've always been utterly baffled by video tutorials for programming. There must be people who prefer that format, but it seems like absolutely the worst possible option for me. You can't cut/paste snippets from the examples. You have to constantly pause them so you can try out examples. Sometimes it's not even easy to read the examples. Perhaps if there was an accompanying web page or PDF... > Depends on what floats your boat. Indeed. Of course there are the written tutorials where the examples don't actually work -- that's always fun. -- Grant From rosuav at gmail.com Thu May 5 21:55:41 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 6 May 2022 11:55:41 +1000 Subject: Python/New/Learn In-Reply-To: <62746315.1c69fb81.9502b.350f@mx.google.com> References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> <62746315.1c69fb81.9502b.350f@mx.google.com> Message-ID: On Fri, 6 May 2022 at 09:53, Grant Edwards wrote: > > On 2022-05-05, Mats Wichmann wrote: > > > Without having any data at all on it, just my impressions, more > > people these days learn from in-person or video experiences. > > I've always been utterly baffled by video tutorials for > programming. There must be people who prefer that format, but it seems > like absolutely the worst possible option for me. You can't cut/paste > snippets from the examples. You have to constantly pause them so you > can try out examples. Sometimes it's not even easy to read the > examples. Perhaps if there was an accompanying web page or PDF... > Video tutorials make GREAT sense for learning complicated programs like Adobe PhotoShop or some 3D game design engines, because (a) most of what you need is in the menus somewhere, but it's hard to find; (b) you can aim the tutorial at a specific version, and it'll be the same for all users; and (c) you can talk about it at the same speed that people can follow along. Video tutorials do NOT make sense for anything where you'll be using your own editor, typing in code, and having it behave the same way. There's nothing to point-and-click, and everything to type. But some people start making tutorials of the first kind, and then go on to make some of the second kind, thinking they'll also be useful. ChrisA From o1bigtenor at gmail.com Fri May 6 07:37:56 2022 From: o1bigtenor at gmail.com (o1bigtenor) Date: Fri, 6 May 2022 06:37:56 -0500 Subject: Python/New/Learn In-Reply-To: References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> <62746315.1c69fb81.9502b.350f@mx.google.com> Message-ID: As we're now discussing tutorial methods - - - - On Thu, May 5, 2022 at 8:57 PM Chris Angelico wrote: > > On Fri, 6 May 2022 at 09:53, Grant Edwards wrote: > > > > On 2022-05-05, Mats Wichmann wrote: > > > > > Without having any data at all on it, just my impressions, more > > > people these days learn from in-person or video experiences. > > > > I've always been utterly baffled by video tutorials for > > programming. There must be people who prefer that format, but it seems > > like absolutely the worst possible option for me. You can't cut/paste > > snippets from the examples. You have to constantly pause them so you > > can try out examples. Sometimes it's not even easy to read the > > examples. Perhaps if there was an accompanying web page or PDF... > > > > Video tutorials make GREAT sense for learning complicated programs > like Adobe PhotoShop or some 3D game design engines, because (a) most > of what you need is in the menus somewhere, but it's hard to find; (b) > you can aim the tutorial at a specific version, and it'll be the same > for all users; and (c) you can talk about it at the same speed that > people can follow along. Respectfully - - - I would disagree. FreeCAD is a mind bogglingly complex architecture (a lumping together of a lot of things without a real central vision imo in fact) and a video tutorial would seem to make sense - - - except - - - trying to see what is being done is at best tricky - - - at worse - - - impossible. If the instructions were text - - - well you could do things one step at a time and you wouldn't have to scroll back and forth 10 times trying to see exactly which part of what was the mouse applied to and then which toolbar . . . . One would think that something that you manipulate visually would be best served by video instruction. Personally I find video instruction the most difficult to follow and the most awkward. And then if the speaker is not easily understandable or is using translated terms (not necessarily the same as those in the program itself) well - - - the frustration level is most definitely NOT small and the amount of learning - - - not that large - - - especially given the effort needed to create video tutorials. IMO video is too often used because its there - - - not because this enhances the experience. (Or the instructor is an academic who is reading their video screens - - - you know - - - like the 85 or 90% of the profs at the uni - - - - ) > > Video tutorials do NOT make sense for anything where you'll be using > your own editor, typing in code, and having it behave the same way. > There's nothing to point-and-click, and everything to type. > > But some people start making tutorials of the first kind, and then go > on to make some of the second kind, thinking they'll also be useful. > I think you, that is Chris, are a very generous person. My experience has been that many consider video tutorials to be cool or sexy or of the highest art - - - and never even consider the uncool, mundane, boring, old, text option. I'm wondering if the difference is that in general education itself less and less emphasis is placed on reading (and comprehension) skills. This fits alongside a return to pictographic language supposedly to assist in multi-lingual barrier reduction. I will cease and desist - - - (thanks for even 'listening') Pace From skip.montanaro at gmail.com Fri May 6 09:06:57 2022 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Fri, 6 May 2022 08:06:57 -0500 Subject: Do projects exist to audit PyPI-hosted packages? Message-ID: I woke with a start in what amounted to the middle of the night (I really need to get about three more hours of sleep, but you'll understand why I was awake to write this). Many years ago, so as to preserve my wrists, I wrote a tool to monitor mouse and keyboard activity. It tells me when to rest. I use it when I have problems, then put it away until it's needed again. I have resurrected it a few times over the years, most recently a month or two ago. Having never been all that fond of how I tracked keyboard and mouse activity, I was happy when I stumbled upon pynput . "Yay!", I thought. My worries are over. Then extremely early this morning I woke thinking, "Damn, this runs on my computer and it can see my mouse and keyboard activity. How do I know it's not stealing my keystrokes?" Not going back to sleep after that. So, I'm going through the code (and the Xlib package on which it relies) to make myself more comfortable that there are no issues. Note: I am *most certainly not* accusing the pynput author of any mischief. In fact, I suspect there's no problem with the package. It's got a bunch of stars and plenty of forks on GitHub (for what that's worth). I suspect the code has had plenty of eyeballs looking at it. Still, I don't really know how well vetted it might be, so I have no assurances of that. I saw it mentioned somewhere (discuss I think?), checked it out, and thought it would solve my activity tracking in a cross-platform way. (I currently only use an Xorg environment, so while I am looking at the code, I'm not paying attention to the Windows or MacOS bits either.) This got me thinking. If I'm curious about pynput, might other people be as well? What about other packages? I'm actually not worried about Python proper or vulnerabilities which have already been found . PyPI currently advertises that it hosts over 373k packages. With that many hosted packages, it is almost certainly a haven for some undetected vulnerabilities. Knowing which packages have been audited ? at least in a cursory fashion ? could be used as a further criterion to use when deciding which packages to consider using on a project. So, does something already exist (pointers appreciated)? Thx... Skip From 2QdxY4RzWzUUiLuE at potatochowder.com Fri May 6 08:56:28 2022 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Fri, 6 May 2022 07:56:28 -0500 Subject: Python/New/Learn In-Reply-To: <62746315.1c69fb81.9502b.350f@mx.google.com> References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> <62746315.1c69fb81.9502b.350f@mx.google.com> Message-ID: On 2022-05-05 at 16:51:49 -0700, Grant Edwards wrote: > On 2022-05-05, Mats Wichmann wrote: > > > Without having any data at all on it, just my impressions, more > > people these days learn from in-person or video experiences. > > I've always been utterly baffled by video tutorials for > programming. There must be people who prefer that format, but it seems > like absolutely the worst possible option for me. You can't cut/paste > snippets from the examples. You have to constantly pause them so you > can try out examples. Sometimes it's not even easy to read the > examples. Perhaps if there was an accompanying web page or PDF... +1 (maybe more), except that an accompanying web page or PDF only solves the problem of copying/pasting examples badly, at the expense of the cognitive load to keep track of one more thing (because it's highly unlikely that the web page or PDF tracks the video "automatically"). As far as easy-to-read examples go, writing them down doesn't always help. One of my physics textbooks used upsilon and nu to describe some phenomenon related to lasers. IIRC, the text, the math, and the physics were pretty straightforward, until you looked at the fraction ?/? in something resembling Times Roman Italic (although, to be fair, once you got that far, it was pretty obvious that it was upsilon over nu rather than nu over upsilon). From sam.z.ezeh at gmail.com Fri May 6 11:24:33 2022 From: sam.z.ezeh at gmail.com (Sam Ezeh) Date: Fri, 6 May 2022 16:24:33 +0100 Subject: Fwd: Do projects exist to audit PyPI-hosted packages? In-Reply-To: References: Message-ID: ---------- Forwarded message --------- From: Sam Ezeh Date: Fri, 6 May 2022, 15:29 Subject: Re: Do projects exist to audit PyPI-hosted packages? To: Skip Montanaro I've had similar thoughts in the past. I don't know of anything but I wonder if repositiories for other languages might have something to deal with it. A related problem is that even if a package is maintained by somebody with good intentions, the account might be hijacked by a malicious actor and since PyPi is separate from source control, people might not be able to find out easily and malware could spread through PyPi. Kind regards, Sam Ezeh On Fri, 6 May 2022, 14:08 Skip Montanaro, wrote: > I woke with a start in what amounted to the middle of the night (I really > need to get about three more hours of sleep, but you'll understand why I > was awake to write this). > > Many years ago, so as to preserve my wrists, I wrote a tool > to > monitor mouse and keyboard activity. It tells me when to rest. I use it > when I have problems, then put it away until it's needed again. I have > resurrected it a few times over the years, most recently a month or two > ago. Having never been all that fond of how I tracked keyboard and mouse > activity, I was happy when I stumbled upon pynput > . "Yay!", I thought. My worries are > over. > > Then extremely early this morning I woke thinking, "Damn, this runs on my > computer and it can see my mouse and keyboard activity. How do I know it's > not stealing my keystrokes?" Not going back to sleep after that. So, I'm > going through the code (and the Xlib package on which it relies) to make > myself more comfortable that there are no issues. Note: I am *most > certainly not* accusing the pynput author of any mischief. In fact, I > suspect there's no problem with the package. It's got a bunch of stars and > plenty of forks on GitHub (for what that's worth). I suspect the code has > had plenty of eyeballs looking at it. Still, I don't really know how well > vetted it might be, so I have no assurances of that. I saw it mentioned > somewhere (discuss I think?), checked it out, and thought it would solve my > activity tracking in a cross-platform way. (I currently only use an Xorg > environment, so while I am looking at the code, I'm not paying attention to > the Windows or MacOS bits either.) > > This got me thinking. If I'm curious about pynput, might other people be as > well? What about other packages? I'm actually not worried about Python > proper or vulnerabilities which have already been found > . PyPI currently advertises > that > it hosts over 373k packages. With that many hosted packages, it is almost > certainly a haven for some undetected vulnerabilities. Knowing which > packages have been audited ? at least in a cursory fashion ? could be used > as a further criterion to use when deciding which packages to consider > using on a project. > > So, does something already exist (pointers appreciated)? Thx... > > Skip > -- > https://mail.python.org/mailman/listinfo/python-list > From avigross at verizon.net Fri May 6 11:59:49 2022 From: avigross at verizon.net (Avi Gross) Date: Fri, 6 May 2022 15:59:49 +0000 (UTC) Subject: Python/New/Learn In-Reply-To: References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> <62746315.1c69fb81.9502b.350f@mx.google.com> Message-ID: <1622846828.161722.1651852789296@mail.yahoo.com> This topic has rapidly shifted over way beyond Python even as the original?person has not returned to participate. There are many ways to teach anything and since the classical method was?to learn in person from someone using mainly sound or pantomime, it has?hung on. Even with the existence of writing, books were made one at a time?and were rare. In more recent times, the norm shifted gradually from lectures to individuals?and groups to include textbooks and illustrations and eventually recordings?or PowerPoint Slides with animation. Realistically, learning some things on your own is easy enough but for many?people and many subjects, you need interaction, reinforcement and more.? We have college professors who repeat almost the same identical lectures?for years to various audiences and also take few or no questions. they might?as well be recorded? and find something better to do. But how do you learn?French just from a book when it is completely not obvious how to pronounce?anything given the weird spelling and grammar rules? How do you know if?the dialect you use swallows some sounds or stretches them out a bit? For this?you need to hear and perhaps see native speakers and correlate those sounds?to the written words and learn to recognize and make them a habit. Even better,?you often want someone to listen to what you try to say and respond and help guide?you. Many Computer topics have an interesting side in that access to a computer?running whatever you are learning can give you much experience and guidance?as you can try various things and see how they work. Written text alone may be?enough to learn what is special about a language and a set of problems to work?on (or your own exploration) may be able to replace much of human interaction. You can look at learning systems such as COURSERA where they often break a?"class" into parts that can include often shorter video clips often with subtitles or?transcripts alongside it, as well as various kinds of printed material including?tests and assignments and even?ways (in some programming courses) to write?small programs that are evaluated immediately by running them through the?language program, or by having others (sometimes fellow students) grade them?and return the results to you. There are many ideas out there how to learn. One of the worst is huge?lecture halls with no rewind ... But text-only learning tools vary quite a bit and some of the better ones do not?just throw facts at you but stop periodically and give you an overview of the goals?and maybe add a touch of history that provides context on why some innovation?was such a big improvement over what had been done and help you pronounce?things when it is not obvious by saying that many people say a function name to?rhyme with this or ... I used to hate Math textbooks that used every imaginable symbol and assumed?you knew how to say every Greek letter and script L and integral symbol and?an assortment of braces and brackets in various sizes and much more.? It is?hard to memorize formulas where you call lots of items by the name of "squiggle"! Python currently sticks largely to using standard ASCII characters so it has fewer?issues to deal with. For people who are not native English speakers, though,?some things may not be intuitively obvious, let alone pronounceable. I suspect?for some purposes, a few lectures to listen to might help if well-designed.? But I noticed how in Julia, they allow all kinds of symbols but also provide a?way to make them fairly easily. Still their use of an actual lower-case epsilon?as a synonym for "in" is an example of how teaching Julia may need more thantext for some people. It uses lots of unusual symbols for operators too thatare often familiar to mathematicians and hardly anyone else. for i ? 1:10 -----Original Message----- From: 2QdxY4RzWzUUiLuE at potatochowder.com To: python-list at python.org Sent: Fri, May 6, 2022 8:56 am Subject: Re: Python/New/Learn On 2022-05-05 at 16:51:49 -0700, Grant Edwards wrote: > On 2022-05-05, Mats Wichmann wrote: > > > Without having any data at all on it, just my impressions, more > > people these days learn from in-person or video experiences. > > I've always been utterly baffled by video tutorials for > programming. There must be people who prefer that format, but it seems > like absolutely the worst possible option for me. You can't cut/paste > snippets from the examples. You have to constantly pause them so you > can try out examples. Sometimes it's not even easy to read the > examples. Perhaps if there was an accompanying web page or PDF... +1 (maybe more), except that an accompanying web page or PDF only solves the problem of copying/pasting examples badly, at the expense of the cognitive load to keep track of one more thing (because it's highly unlikely that the web page or PDF tracks the video "automatically"). As far as easy-to-read examples go, writing them down doesn't always help.? One of my physics textbooks used upsilon and nu to describe some phenomenon related to lasers.? IIRC, the text, the math, and the physics were pretty straightforward, until you looked at the fraction ?/? in something resembling Times Roman Italic (although, to be fair, once you got that far, it was pretty obvious that it was upsilon over nu rather than nu over upsilon). -- https://mail.python.org/mailman/listinfo/python-list From jonathan.kaczynski at guildeducation.com Fri May 6 08:22:03 2022 From: jonathan.kaczynski at guildeducation.com (Jonathan Kaczynski) Date: Fri, 6 May 2022 08:22:03 -0400 Subject: Seeking deeper understanding of python equality (==) Message-ID: Hi, I was recently trying to explain how python equality works and ran into a gap in my knowledge. I haven't found any good pages going beneath a surface level explanation of python equality comparison. I'll post my investigations below. What I think I'm looking for is where in the source code (https://github.com/python/cpython) does the equality comparison occur. I have an idea but wanted to ask first. Using the dis module, we see the comparison operator is a single bytecode, which is expected. ? docker run -it --rm ubuntu:jammy root at 919d94c98191:/# apt-get update root at 919d94c98191:/# apt-get --yes install python3 root at 919d94c98191:/# cat >play.py <play.py < //play.py(5)() -> x == y (Pdb) s --Call-- > /usr/lib/python3.10/uuid.py(239)__eq__() -> def __eq__(self, other): Thank you, Jonathan From loris.bennett at fu-berlin.de Fri May 6 08:11:24 2022 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Fri, 06 May 2022 14:11:24 +0200 Subject: Instatiating module / Reusing module of command-line tool References: <87o80kxcsp.fsf@hornfels.zedat.fu-berlin.de> Message-ID: <87v8uij1xv.fsf@hornfels.zedat.fu-berlin.de> ram at zedat.fu-berlin.de (Stefan Ram) writes: > "Loris Bennett" writes: >>My question: What is the analogue to initialising an object via the >>constructor for a module? > > If you need a class, you can write a class. > > When one imports a module, the module actually gets executed. > That's why people write "if __name__ == '__main__':" often. > So, everything one wants to be done at import time can be > written directly into the body of one's module. So if I have a module which relies on having internal data being set from outside, then, even though the program only ever has one instance of the module, different runs, say test and production, would require different internal data and thus different instances. Therefore a class seems more appropriate and it is more obvious to me how to initialise the objects (e.g. by having the some main function which can read command-line arguments and then just pass the arguments to the constructor. I suppose that the decisive aspect is that my module needs initialisation and thus should to be a class. Your examples in the other posting of the modules 'math' and 'string' are different, because they just contain functions and no data. Cheers, Loris -- This signature is currently under construction. From sam.z.ezeh at gmail.com Fri May 6 13:31:05 2022 From: sam.z.ezeh at gmail.com (Sam Ezeh) Date: Fri, 6 May 2022 18:31:05 +0100 Subject: Seeking deeper understanding of python equality (==) In-Reply-To: References: Message-ID: Perhaps these source references are useful: Python/ceval.c (_PyEval_EvalFrameDefault) https://github.com/python/cpython/blob/main/Python/ceval.c#L3754-L3768 Objects/object.c (do_richcompare) https://github.com/python/cpython/blob/42fee931d055a3ef8ed31abe44603b9b2856e04d/Objects/object.c#L661-L713 Kind regards, Sam Ezeh On Fri, 6 May 2022 at 18:12, Jonathan Kaczynski wrote: > > Hi, > > I was recently trying to explain how python equality works and ran into a > gap in my knowledge. I haven't found any good pages going beneath a surface > level explanation of python equality comparison. > > I'll post my investigations below. What I think I'm looking for is where in > the source code (https://github.com/python/cpython) does the equality > comparison occur. I have an idea but wanted to ask first. > > > Using the dis module, we see the comparison operator is a single bytecode, > which is expected. > > ? docker run -it --rm ubuntu:jammy > root at 919d94c98191:/# apt-get update > root at 919d94c98191:/# apt-get --yes install python3 > root at 919d94c98191:/# cat >play.py < import dis > import uuid > > def test(): > x = uuid.uuid4() > y = str(x) > x == y > return > > dis.dis(test) > EOF > root at f33b02fef026:/# python3 play.py > ... snip ... > 7 16 LOAD_FAST 0 (x) > 18 LOAD_FAST 1 (y) > 20 COMPARE_OP 2 (==) > 22 POP_TOP > ... snip ... > > > Stepping through the code with gdb, we see it jump from the compare > operator to the dunder-eq method on the UUID object. What I want to be able > to do is explain the in-between steps. Also, if you change `x == y` to `y > == x`, you still see the same behavior, which I assume has to do with > dunder-eq being defined on the UUID class and thus given priority. > > ? docker run -it --rm ubuntu:jammy > root at 919d94c98191:/# apt-get update > root at 919d94c98191:/# apt-get --yes install dpkg-source-gitarchive > root at 919d94c98191:/# sed -i 's/^# deb-src/deb-src/' /etc/apt/sources.list > root at 919d94c98191:/# apt-get update > root at 919d94c98191:/# apt-get --yes install gdb python3.10-dbg > root at 919d94c98191:/# apt-get source python3.10-dbg > root at 919d94c98191:/# cat >play.py < import uuid > x = uuid.uuid4() > y = str(x) > breakpoint() > x == y > EOF > root at 919d94c98191:/# gdb python3.10-dbg > (gdb) dir python3.10-3.10.4/Python > (gdb) run play.py > Starting program: /usr/bin/python3.10-dbg play.py > > warning: Error disabling address space randomization: Operation not > permitted > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > > //play.py(5)() > -> x == y > (Pdb) s > --Call-- > > /usr/lib/python3.10/uuid.py(239)__eq__() > -> def __eq__(self, other): > > > Thank you, > Jonathan > -- > https://mail.python.org/mailman/listinfo/python-list From skip.montanaro at gmail.com Fri May 6 13:36:26 2022 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Fri, 6 May 2022 12:36:26 -0500 Subject: Do projects exist to audit PyPI-hosted packages? In-Reply-To: References: Message-ID: > > A related problem is that even if a package is maintained by somebody with > good intentions, the account might be hijacked by a malicious actor and > since PyPi is separate from source control, people might not be able to > find out easily and malware could spread through PyPi. > I hadn't considered that. Some sort of authenticated connection between the source code hosting service and the PyPI user posting the package would be nice. Some other (only tangentially related) stuff occurs to me as I search for useful bits... I'd kinda be curious what hosting services other than GitHub or GitLab are in common use. GNU Savannah? SourceForge? PyPI relevance isn't a terrific indicator (I assume it uses Libraries.io's SourceRank to get a relevance score), but it's still some kind of indicator how useful a package is. Perhaps the PyPI BigQuery stuff has hosting info. I've not dug into it. (Thinking that obscure hosting service might be a small knock against a package, but that's just a thought. I realize not everyone is happy with corporate hosting services.) Having a decent idea what functional alternatives are out there to a particular package would be nice as well. Again, considering pynput, I hit Google up for "python packages similar to pynput" which led me here: https://www.libhunt.com/r/pynput I was unaware of its existence before. I have no idea how useful it might be for narrowly focused packages like pynput. Something with application to a much wider community, like numpy, returns a bunch more: https://www.libhunt.com/r/numpy Skip From mats at wichmann.us Fri May 6 15:00:24 2022 From: mats at wichmann.us (Mats Wichmann) Date: Fri, 6 May 2022 13:00:24 -0600 Subject: Fwd: Do projects exist to audit PyPI-hosted packages? In-Reply-To: References: Message-ID: On 5/6/22 09:24, Sam Ezeh wrote: > ---------- Forwarded message --------- > From: Sam Ezeh > Date: Fri, 6 May 2022, 15:29 > Subject: Re: Do projects exist to audit PyPI-hosted packages? > To: Skip Montanaro > > > I've had similar thoughts in the past. I don't know of anything but I > wonder if repositiories for other languages might have something to deal > with it. > > A related problem is that even if a package is maintained by somebody with > good intentions, the account might be hijacked by a malicious actor and > since PyPi is separate from source control, people might not be able to > find out easily and malware could spread through PyPi. FWIW, there's talk of mandating MFA or appropriately scoped tokens to upload from a PyPi account to cut down on hijacking chances. As I understand it, a concern that has slowed this is that sometimes a "release" involves a ton of actual package uploads and that could involve considerable manual overhead if a 2FA sequence were required for each one. Meanwhile, individual projects can now require 2FA in order for owners to do anything "administrative". Probably others understand the current state of play better here.... From p_d_a_s_i_l_v_a_ns at nonetnoaddress.pt Fri May 6 15:11:07 2022 From: p_d_a_s_i_l_v_a_ns at nonetnoaddress.pt (Paulo da Silva) Date: Fri, 6 May 2022 20:11:07 +0100 Subject: pandas (in jupyter?) problem Message-ID: Hi all! I'm having the following problem. Consider the code (the commented or the not commented which I think do the same things): #for col in missing_cols: # df[col] = np.nan df=df.copy() df[missing_cols]=np.nan df has about 20000 cols and len(missing_cols) is about 18000. I'm getting lots (1 by missing_col?) of the following message from ipykernel: "PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling `frame.insert` many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()` df[missing_cols]=np.nan" At first I didn't have df=df.copy(). I added it later, but the same problem. This slows down the code a lot, perhaps because jupyter is taking too much time issuing these messages! Thanks for any comments. From Marco.Sulla.Python at gmail.com Fri May 6 15:21:36 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Fri, 6 May 2022 21:21:36 +0200 Subject: tail In-Reply-To: References: Message-ID: I have a little problem. I tried to extend the tail function, so it can read lines from the bottom of a file object opened in text mode. The problem is it does not work. It gets a starting position that is lower than the expected by 3 characters. So the first line is read only for 2 chars, and the last line is missing. import os _lf = "\n" _cr = "\r" _lf_ord = ord(_lf) def tail(f, n=10, chunk_size=100): n_chunk_size = n * chunk_size pos = os.stat(f.fileno()).st_size chunk_line_pos = -1 lines_not_found = n binary_mode = "b" in f.mode lf = _lf_ord if binary_mode else _lf while pos != 0: pos -= n_chunk_size if pos < 0: pos = 0 f.seek(pos) chars = f.read(n_chunk_size) for i, char in enumerate(reversed(chars)): if char == lf: lines_not_found -= 1 if lines_not_found == 0: chunk_line_pos = len(chars) - i - 1 print(chunk_line_pos, i) break if lines_not_found == 0: break line_pos = pos + chunk_line_pos + 1 f.seek(line_pos) res = b"" if binary_mode else "" for i in range(n): res += f.readline() return res Maybe the problem is 1 char != 1 byte? From python at mrabarnett.plus.com Fri May 6 16:19:48 2022 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 6 May 2022 21:19:48 +0100 Subject: tail In-Reply-To: References: Message-ID: On 2022-05-06 20:21, Marco Sulla wrote: > I have a little problem. > > I tried to extend the tail function, so it can read lines from the bottom > of a file object opened in text mode. > > The problem is it does not work. It gets a starting position that is lower > than the expected by 3 characters. So the first line is read only for 2 > chars, and the last line is missing. > > import os > > _lf = "\n" > _cr = "\r" > _lf_ord = ord(_lf) > > def tail(f, n=10, chunk_size=100): > n_chunk_size = n * chunk_size > pos = os.stat(f.fileno()).st_size > chunk_line_pos = -1 > lines_not_found = n > binary_mode = "b" in f.mode > lf = _lf_ord if binary_mode else _lf > > while pos != 0: > pos -= n_chunk_size > > if pos < 0: > pos = 0 > > f.seek(pos) > chars = f.read(n_chunk_size) > > for i, char in enumerate(reversed(chars)): > if char == lf: > lines_not_found -= 1 > > if lines_not_found == 0: > chunk_line_pos = len(chars) - i - 1 > print(chunk_line_pos, i) > break > > if lines_not_found == 0: > break > > line_pos = pos + chunk_line_pos + 1 > > f.seek(line_pos) > > res = b"" if binary_mode else "" > > for i in range(n): > res += f.readline() > > return res > > Maybe the problem is 1 char != 1 byte? Is the file UTF-8? That's a variable-width encoding, so are any of the characters > U+007F? Which OS? On Windows, it's common/normal for UTF-8 files to start with a BOM/signature, which is 3 bytes/1 codepoint. From PythonList at DancesWithMice.info Fri May 6 17:52:29 2022 From: PythonList at DancesWithMice.info (dn) Date: Sat, 7 May 2022 09:52:29 +1200 Subject: Python/New/Learn In-Reply-To: References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> <62746315.1c69fb81.9502b.350f@mx.google.com> Message-ID: <47a62108-a48c-1a2f-41b0-f326a01dc0d9@DancesWithMice.info> To the OP: there are many MOOCs available on the likes of the Coursera and edX platform. (rationale, below) Disclaimer: I work on courses on the edX platform (but not Python). On 06/05/2022 23.37, o1bigtenor wrote: > As we're now discussing tutorial methods - - - - > > On Thu, May 5, 2022 at 8:57 PM Chris Angelico wrote: >> >> On Fri, 6 May 2022 at 09:53, Grant Edwards wrote: >>> >>> On 2022-05-05, Mats Wichmann wrote: >>> >>>> Without having any data at all on it, just my impressions, more >>>> people these days learn from in-person or video experiences. >>> >>> I've always been utterly baffled by video tutorials for >>> programming. There must be people who prefer that format, but it seems >>> like absolutely the worst possible option for me. You can't cut/paste >>> snippets from the examples. You have to constantly pause them so you >>> can try out examples. Sometimes it's not even easy to read the >>> examples. Perhaps if there was an accompanying web page or PDF... >>> >> >> Video tutorials make GREAT sense for learning complicated programs >> like Adobe PhotoShop or some 3D game design engines, because (a) most >> of what you need is in the menus somewhere, but it's hard to find; (b) >> you can aim the tutorial at a specific version, and it'll be the same >> for all users; and (c) you can talk about it at the same speed that >> people can follow along. > > Respectfully - - - I would disagree. FreeCAD is a mind bogglingly > complex architecture (a lumping together of a lot of things without a real > central vision imo in fact) and a video tutorial would seem to make sense > - - - except - - - trying to see what is being done is at best tricky - - - at > worse - - - impossible. If the instructions were text - - - well you could do > things one step at a time and you wouldn't have to scroll back and forth 10 > times trying to see exactly which part of what was the mouse applied > to and then which toolbar . . . . One would think that something that > you manipulate visually would be best served by video instruction. > > Personally I find video instruction the most difficult to follow and the most > awkward. And then if the speaker is not easily understandable or is using > translated terms (not necessarily the same as those in the program > itself) well - - - the frustration level is most definitely NOT small and > the amount of learning - - - not that large - - - especially given the > effort needed to create video tutorials. IMO video is too often used > because its there - - - not because this enhances the experience. > > (Or the instructor is an academic who is reading their video screens > - - - you know - - - like the 85 or 90% of the profs at the uni - - - - ) >> >> Video tutorials do NOT make sense for anything where you'll be using >> your own editor, typing in code, and having it behave the same way. >> There's nothing to point-and-click, and everything to type. >> >> But some people start making tutorials of the first kind, and then go >> on to make some of the second kind, thinking they'll also be useful. >> > I think you, that is Chris, are a very generous person. My experience > has been that many consider video tutorials to be cool or sexy or of the > highest art - - - and never even consider the uncool, mundane, boring, > old, text option. > > I'm wondering if the difference is that in general education itself less > and less emphasis is placed on reading (and comprehension) skills. > This fits alongside a return to pictographic language supposedly to > assist in multi-lingual barrier reduction. To all: The problem with some of the advice given in this thread, eg using StackOverflow or YouTube videos, is that a beginner (particularly) has no measure of the material's quality. Both platforms are riddled with utter-junk - even 'dangerous' advice. Some of such posted-content has been encouraged by 'teachers' who think the old adage "to understand something properly one must be able to explain it to someone else" can be translated into an (effective) learning practice. Which is fine, until the results are posted on a public forum without any qualitative assessment - I've even come-across 'teachers' (the quote-marks could be taken to indicate serious question or in some cases, disdain) who think that allowing/encouraging students to post such is "encouraging the student". Sadly, (IMHO) its effect is 'intellectual pollution' for those who come-after, and the generation of an unrealistic ego/self-assessment on the part of the post-er. Then there are others which have been posted by well-meaning individuals. These may be motivated, similarly, by ego or altruism. Few know what they're doing (from a training PoV) and the fact that some such contributors stand-out so far about 'the crowd' speaks to this. How many have you seen which fail to account for the differences between your system and the one the author uses? How many seem determined to show that by clicking 'here' and selecting 'there', the most complex of environments can be 'mastered' by anyone, clickety-click - without a single 'if this (step) goes wrong, here's what you might see/need to do...? The advantage of a text-book, which has been properly and competently edited and reviewed, is that the content has passed the 'many eyes' test. Each explanation has been examined critically, and is no longer (only) the author's own view-point, way of explaining, or even opinion; of how things could/should be done. There is little learning-value in asking people to learn-by-typing - although I suppose one might learn something of the importance of accurate syntax. (how could an author hope to cover all the possible errors of this type that a learner might make whilst transcribing 'my first Python program[me]'?) This is why many texts offer a download, and why my in-person classes start with trainees being given code-sets to edit/customise. When we set up our MOOC (remember, not Python, but the message is much the same) we swallowed two lots of 'Kool Aid'. Firstly, that 'the modern student' wants everything on video. Secondly, that 'long lectures' are 'out', and we should look at +/- seven minute talks. The edX platform provides excellent usage statistics, both on a cohort basis, and over time. These reveal (as a group) that most trainees either don't start, or certainly don't finish, an entire video - without regard for 'length'. Surprise! Unfortunately, the stats don't differentiate between a person watching a video for the first (and only) time, and someone who goes back to watch it again, eg after attempting a quiz/progress challenge, not doing well, and wanting to review. However, the more 'repeat views' that have taken place, imply that the average must shift to revealing more-and-more trainees who don't even open the video. The 'lecture' advice does fit with the way people organise self-paced study in-and-amongst all the other facets of life. Many working full-time, find that a 'short burst' can be managed whenever an opportunity presents itself - even during a commuter bus-ride. Whereas one would have to make a more deliberate plan to sit down for one whole hour at a time. Yes, it can be more complicated to design a sensible (alternate) presentation-structure. However, as we have discovered, this works well for 'us' as well. Replacing a long video-based lecture is much more expensive than replacing a short one, eg if a new feature has been added by a recent software-release - particularly if we are only talking about a small 'tweak'. The claimed-advantage of MOOCs is that trainees from anywhere in the world can access 'the best' in the world - usually this is a claim against the institution, but sometimes deservedly, the lecturer. I have to say though, some of these claims are not justified. A staff-member at a local-institution might have a much better explanation! I'm distressed by many (usually American) institutions claiming world-status but who use teaching-techniques that remind me of my own days as an under-grad years, um-er, many (many) decades ago. The world has turned, and moved-on! Similarly, there is a considerable difference between the teaching styles used in much of Asia, eg India and China; and the more student-centered approaches used in 'the west'. Would you prefer to write some code-examples using simple data-structures, or to sit down and be tested on your ability to recall the range of list-methods and their parameters? The 'professionals' will often have a veritable studio and likely video-staff involved in ensuring the output-standard (similar to above reference to books and 'reviewers'). This ensures that as many 'takes' as necessary are performed to ensure quality. Whereas our 'YouTubers' works are frequently punctuated by 'ers', 'ums', and re-typing errors during demos, and the like - a 'stream of consciousness approach'. Our course-structure considerations extend down to the detailed-level of 'story boards' detailing which topic will be covered 'when', how it will be covered, what the sequence will be, and so-on. This means that 'my first Python program[me]' will not be confused with 'how to install the Python interpreter' - another lesson the earnest-YouTubers could emulate to-advantage (also likely the +/- seven minutes advice). Sadly MOOCs are not the 'be-all and end-all'! For a start, there are many students who cannot gain (easy) access to the Internet - a rather basic requirement to 'attend'. Also (and as appears elsewhere in this thread) the 'lecture' approach to training is not 'the best way' - it is a 'factory approach' to teaching (one 'sage on the stage' speaking to a large number in an audience). However, the real question is: is attending a lecture the best way to learn? That said, a MOOC will usually be structured so that there are video-recorded lectures, but that much?all of the content also appears in the directive or accompanying text. Thus, both 'YouTube' and 'text book' media; combined into one presentation. In addition there are built-in 'Discussion Lists' where all the trainees working on a particular topic exchange views, ask questions, help each-other through, etc (unlike StackOverflow which is very wide and diverse collection of conversations, with no indication of 'level', competence, need...). Because a MOOC's text is both copy-paste-able and/or code-examples can be provided for download (within context, rather than as a book-wide .ZIP archive), those advantages are also available (as previously discussed 'here'). I don't see any sort of 'cut-over point' where text-books give way to, or are over-taken by videos. Complexity of the subject is not the issue or inflection-point. Some things are best presented in schematic form, for example - but although you could portray the try...except construct that way, sooner-or-later the presentation must revert to text - what is code if not 'text'? What can be a very important point, (as previously mentioned in this thread) MOOCs are produced by folk who are used to presenting information, and to the discipline of writing 'papers'. Accordingly, dates and version numbers are included as a matter of course. Thus, there is a very low possibility of finding that 'this version' is different from the one you have (assuming you installed the one the course recommends/requires). How many 'amateurs' think to do this - even bloggers who don't see themselves as primarily presenting 'learning material'. Speaking personally, I'm like the person who said that learning from videos is more difficult that text-based materials. The most obvious explanation for this relates to when/how I grew-up and enjoyed?suffered my education. We didn't have videos, or even (domestic) VCRs (until the very end of my high school years. So, everything was 'in a book'. These days it is quite a different story. Whilst I'm prepared to accept that young 'digerati' may prefer video, even that assertion is clouded by the nature of watching a video not only being a change in medium but it is also "asynchronous". Accordingly, the advantages of 'self-paced' and being able to 'watch' at a time of one's own choosing, cf attending a (synchronous) lecture at a specific time in a particular location. Interestingly, ans somewhat-related, Librarians report that there is little difference between the age-groups when library-users are offered the choice of the same material either as an eBook or in the form of 'dead trees', people select the latter/old-style more than 90% of the time! However, I may disagree (for my own case) with the person talking about 'notes'. It is easily and widely proven that people who make notes will improve their learning and "retention" rates. It is worth knowing that when we read something we are overly confident of remembering points later. Indeed, even reading and re-reading text (as many of us were taught to do for exam-prep) is a remarkably-ineffectual way to learn (for that reason). Making notes has a first-pass value of helping us to consolidate new-learning and link it to existing knowledge (a key to retaining knowledge of this 'new stuff' effectively). There is something about writing things down, even after reading them, that helps 'engage the brain'. Indeed, to the ridiculous point that when I write something down I may not ever need to go back to my notes to be able to recall the fact(s), eg my shopping list! So, why write them down then? As mentioned, it is the very act of writing which effects this 'learning'. Bad news though: typing into a computer does not create this psychomotor-benefit. There is something in the 'picture' of writing which cannot be re-created in the text-only alternative of a word processor! Secondly, if one takes notes today, and re-reads same tomorrow, and again next week; the 'learning' is not only effected, but consolidated. So, "repetition" at that level is more likely to lead to "retention". Now applying a little psychological 'warning'. We (humans) tend to approach devices in a particular mode, eg most people regard their car, bike (etc) as a means of getting from 'A' to 'B'. Yes, there are others who have a different perspective, and want to go racing, for example. The same idea of 'this is what the machine does' or 'this is what I use if for', applies to computers and smart-phones - and again, those of us who regard them as a machine to 'master' or which which to build 'apps' and develop software are akin to the 'racers'! Thus, if someone mainly uses their device to watch movies, tweet, put their face in a 'book', send 'telegrams', or record humorous events/Dad-dancing; is the 'mode' in which they habitually use their device and thus the 'state of their brain', congruent to or encouraging of "learning"? The technology one poster 'here' requested, that video and text might appear together and synchronised is (surprisingly?) available today and has been for five~ten years. It is a base-feature of HTML5's video facility. IIRC something similar is also reproduced and available through the YouTube 'engine' (I don't use that). It is not common because it is MORE WORK! In this part of the game, most people seek a technology which will take the video's audio-track and transliterate (speech-to-text). I've had to take up a practice that I never previously used, and write every single word of a presentation (like a formal speech). This forms the text component of a future video, but I (or the presenter) then have to read the words whilst making our voices sound more conversational - again, a skill most have not acquired, even having been required to read-aloud for the class in junior/grade school! This form of 'multi-media' forms one aspect of my research into Cognitive Psychology (how one learns) and tool-building to ease presentation-production and/or to promote learning. Finally, I was amused by @BigTenor's comment about "pictographic language". Some years ago, a theory was proposed that different people learn in different ways, eg some prefer to hear whereas others to read. Sadly, whilst this idea has immediate appeal because a casual introspection will have many readily saying "I'm a ... learner"), the theory has been debunked - both by primary-research and by secondary-researchers critiquing the 'evidence' which didn't actually make a proper attempt to prove the theory in-reality. However, it can be a useful 'guide' - as long as that is as far as it is ever applied. A lot of 'damage' is done by people telling themselves (or being told by their 'betters') that they can only learn in one particular style (to the exclusion of all others). Don't you believe it! As above, some things are best explained one way, and others another, eg an annotated map cf a list of directions. @Stefan's French spelling and phonetic pronunciation guide in-combination illustrates this nicely. (see also 'try...except', above) The interesting observation is that whilst I may grasp some new concept from a text explanation in the morning, I may need a diagrammatic approach to convey something similar, this afternoon, ie we don't always learn using the same approach, every time! Which leads to the idea of having multiple means of conveying information, eg the MOOC's text and video content, because we hope that if one doesn't 'do it for you', the other will! Apologies for the lack of 'Python', but thanks for the discussion (and am hoping you won't mind me passing it around amongst my colleagues - perhaps there's not much that is 'new' (for us) but the fact that it is being discussed by folk who are primarily practitioners is most helpful!) -- Regards, =dn From wlfraed at ix.netcom.com Fri May 6 17:10:12 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Fri, 06 May 2022 17:10:12 -0400 Subject: tail References: Message-ID: <343b7hlfae195bt4inm3sg9es39bd6butk@4ax.com> On Fri, 6 May 2022 21:19:48 +0100, MRAB declaimed the following: >Is the file UTF-8? That's a variable-width encoding, so are any of the >characters > U+007F? > >Which OS? On Windows, it's common/normal for UTF-8 files to start with a >BOM/signature, which is 3 bytes/1 codepoint. Windows also uses for the EOL marker, but Python's I/O system condenses that to just internally (for TEXT mode) -- so using the length of a string so read to compute a file position may be off-by-one for each EOL in the string. https://docs.python.org/3/tutorial/inputoutput.html#reading-and-writing-files """ In text mode, the default when reading is to convert platform-specific line endings (\n on Unix, \r\n on Windows) to just \n. When writing in text mode, the default is to convert occurrences of \n back to platform-specific line endings. This behind-the-scenes modification to file data is fine for text files, but will corrupt binary data like that in JPEG or EXE files. Be very careful to use binary mode when reading and writing such files. """ -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From greg.ewing at canterbury.ac.nz Fri May 6 22:07:53 2022 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 7 May 2022 14:07:53 +1200 Subject: Python/New/Learn In-Reply-To: References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> <62746315.1c69fb81.9502b.350f@mx.google.com> <1622846828.161722.1651852789296@mail.yahoo.com> Message-ID: On 7/05/22 12:27 pm, Stefan Ram wrote: > But when you read descriptions in books about phonology about > how the mouth and tongue is positioned to produce certain > sounds and see pictures of this, your faulty ears are bypassed > and you get a chance to produce the correct sounds of the > foreign language even when you might not hear the difference. > > So, one might actually be able to learn the pronunciation > of a foreign language from text in a book better than from > an audio tape (or an audio file or a video with sound)! Such books would certainly help, but I don't think there's any substitute for actually hearing the sounds if you want to be able to understand the spoken language. In my experience, you have to listen to it for quite a while to retrain your ears to the point where you can even begin to pick out words from the audio stream. I kind-of studied French for 5 years in school, with teachers to learn the pronunication from, but I never got a lot of practice at it or much chance to hear it spoken. As a result I have about a 1% success rate at understanding spoken French, even when I know all the words being used. -- Greg From greg.ewing at canterbury.ac.nz Fri May 6 22:24:03 2022 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 7 May 2022 14:24:03 +1200 Subject: Seeking deeper understanding of python equality (==) In-Reply-To: References: Message-ID: On 7/05/22 12:22 am, Jonathan Kaczynski wrote: > Stepping through the code with gdb, we see it jump from the compare > operator to the dunder-eq method on the UUID object. What I want to be able > to do is explain the in-between steps. Generally what happens with infix operators is that the interpreter first looks for a dunder method on the left operand. If that method doesn't exist or returns NotImplemented, it then looks for a dunder method on the right operand. There is an exception if the right operand is a subclass of the left operand -- in that case the right operand's dunder method takes precedence. > Also, if you change `x == y` to `y > == x`, you still see the same behavior, which I assume has to do with > dunder-eq being defined on the UUID class and thus given priority. No, in that case the conparison method of str will be getting called first, but you won't see that in pdb because it doesn't involve any Python code. Since strings don't know how to compare themselves with uuids, it will return NotImplemented and the interpreter will then call uuid's method. -- Greg From cs at cskk.id.au Sat May 7 02:28:54 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Sat, 7 May 2022 16:28:54 +1000 Subject: Instatiating module / Reusing module of command-line tool In-Reply-To: <87v8uij1xv.fsf@hornfels.zedat.fu-berlin.de> References: <87v8uij1xv.fsf@hornfels.zedat.fu-berlin.de> Message-ID: On 06May2022 14:11, Loris Bennett wrote: >ram at zedat.fu-berlin.de (Stefan Ram) writes: >> If you need a class, you can write a class. >> >> When one imports a module, the module actually gets executed. >> That's why people write "if __name__ == '__main__':" often. >> So, everything one wants to be done at import time can be >> written directly into the body of one's module. > >So if I have a module which relies on having internal data being set >from outside, then, even though the program only ever has one instance >of the module, different runs, say test and production, would require >different internal data and thus different instances. Therefore a class >seems more appropriate and it is more obvious to me how to initialise >the objects (e.g. by having the some main function which can read >command-line arguments and then just pass the arguments to the >constructor. > >I suppose that the decisive aspect is that my module needs >initialisation and thus should to be a class. Your examples in the >other posting of the modules 'math' and 'string' are different, because >they just contain functions and no data. Yeah, I do this quite a bit. So I might have the core class which does it all: class Thing: def __init__(self, whatever...): .... and if I'm exercising this from the command line I'll write a main function: def main(argv): cmd = argv.pop(0) ... use the arguments to specify data files or modes or whatever ... obj = Thing(...init the thing...) obj.do_something(...) That is usually the top thing, after the imports but before everything else. Then right down the bottom: if __name__ == '__main__': sys.exit(main(sys.argv)) for running the module in command line mode: python3 -m the.module.name args here ... That way you can import it elsewhere for the "thing" class and also do basic command line stuff with it directly. Cheers, Cameron Simpson way I'll probably write a class for a command line. From hjp-python at hjp.at Sat May 7 04:19:08 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sat, 7 May 2022 10:19:08 +0200 Subject: Python/New/Learn In-Reply-To: References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> <62746315.1c69fb81.9502b.350f@mx.google.com> <1622846828.161722.1651852789296@mail.yahoo.com> Message-ID: <20220507081908.4ibzgtnwjh7bep5e@hjp.at> On 2022-05-07 14:07:53 +1200, Greg Ewing wrote: > On 7/05/22 12:27 pm, Stefan Ram wrote: > > So, one might actually be able to learn the pronunciation > > of a foreign language from text in a book better than from > > an audio tape (or an audio file or a video with sound)! > > Such books would certainly help, but I don't think there's any > substitute for actually hearing the sounds if you want to be > able to understand the spoken language. I think "learning to understand the spoken language" and "learning to speak without a (foreign) accent" are two different things. I agree that the former needs exposure to actual people talking (preferably in real life, where people talk fast, slur endings, omit words, hem and haw, talk over each other ...). For learning to speak without an accent, just listening (or talking) to native speakers is probably not sufficient for the reasons Stefan mentioned plus another one: Outside of a classroom people usually won't correct your mistakes unless you say something truly incomprehensible or unintentionally funny. However I don't think a book is sufficient either: Most people are probably even worse at observing the position of their various mouth parts while speaking than at listening, so without feedback from a native speaker (preferably a trained voice coach) they can't really tell whether they are doing it right. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From Marco.Sulla.Python at gmail.com Sat May 7 09:21:25 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Sat, 7 May 2022 15:21:25 +0200 Subject: tail In-Reply-To: <343b7hlfae195bt4inm3sg9es39bd6butk@4ax.com> References: <343b7hlfae195bt4inm3sg9es39bd6butk@4ax.com> Message-ID: On Sat, 7 May 2022 at 01:03, Dennis Lee Bieber wrote: > > Windows also uses for the EOL marker, but Python's I/O system > condenses that to just internally (for TEXT mode) -- so using the > length of a string so read to compute a file position may be off-by-one for > each EOL in the string. So there's no way to reliably read lines in reverse in text mode using seek and read, but the only option is readlines? From avigross at verizon.net Sat May 7 10:01:04 2022 From: avigross at verizon.net (Avi Gross) Date: Sat, 7 May 2022 14:01:04 +0000 (UTC) Subject: tail In-Reply-To: References: <343b7hlfae195bt4inm3sg9es39bd6butk@4ax.com> Message-ID: <661020750.322747.1651932064781@mail.yahoo.com> Marco, I think it was made clear from the start that "text" files in the classic sense have no?random access method at any higher level than reading a byte at some offset?from the beginning of the file, or back from the end when it has not grown. The obvious fact is that most of the time the lines are not of fixed widths and?you have heard about multiple byte encodings and how the ends of lines can vary. When files get long enough that just reading them from the start as a whole, or even?in chunks, gets too expensive, some people might consider some other method.?Log files can go on for years so it is not uncommon to start a new one periodically?and have a folder with many of them in some order. To get the last few lines?simply means finding the last file and reading it, or if it is too short, getting the?penultimate one too. And obviously a database or other structure might work better which might make?each "line" a record and index them. But there are ways to create your own data that get around this such as using an?encoding with a large but fixed width for every character albeit you need more storage?space. But if the goal is a general purpose tool, internationalization from ASCII?has created a challenge for lots of such tools. -----Original Message----- From: Marco Sulla To: Dennis Lee Bieber Cc: python-list at python.org Sent: Sat, May 7, 2022 9:21 am Subject: Re: tail On Sat, 7 May 2022 at 01:03, Dennis Lee Bieber wrote: > >? ? ? ? Windows also uses for the EOL marker, but Python's I/O system > condenses that to just internally (for TEXT mode) -- so using the > length of a string so read to compute a file position may be off-by-one for > each EOL in the string. So there's no way to reliably read lines in reverse in text mode using seek and read, but the only option is readlines? -- https://mail.python.org/mailman/listinfo/python-list From barry at barrys-emacs.org Sat May 7 10:08:14 2022 From: barry at barrys-emacs.org (Barry) Date: Sat, 7 May 2022 15:08:14 +0100 Subject: tail In-Reply-To: References: Message-ID: <60454E09-0ADA-4881-A84B-6C11397D244F@barrys-emacs.org> > On 7 May 2022, at 14:24, Marco Sulla wrote: > > ?On Sat, 7 May 2022 at 01:03, Dennis Lee Bieber wrote: >> >> Windows also uses for the EOL marker, but Python's I/O system >> condenses that to just internally (for TEXT mode) -- so using the >> length of a string so read to compute a file position may be off-by-one for >> each EOL in the string. > > So there's no way to reliably read lines in reverse in text mode using > seek and read, but the only option is readlines? You need to handle the file in bin mode and do the handling of line endings and encodings yourself. It?s not that hard for the cases you wanted. Figure out which line ending is in use from the CR LF, LF, CR. Once you have a line decode it before returning it. The only OS I know that used CR was Classic Mac OS. If you do not care about that then you can split on NL and strip any trailing CR. Barry > -- > https://mail.python.org/mailman/listinfo/python-list > From o1bigtenor at gmail.com Sat May 7 12:08:41 2022 From: o1bigtenor at gmail.com (o1bigtenor) Date: Sat, 7 May 2022 11:08:41 -0500 Subject: Python/New/Learn In-Reply-To: <20220507081908.4ibzgtnwjh7bep5e@hjp.at> References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> <62746315.1c69fb81.9502b.350f@mx.google.com> <1622846828.161722.1651852789296@mail.yahoo.com> <20220507081908.4ibzgtnwjh7bep5e@hjp.at> Message-ID: On Sat, May 7, 2022 at 3:29 AM Peter J. Holzer wrote: > > On 2022-05-07 14:07:53 +1200, Greg Ewing wrote: > > On 7/05/22 12:27 pm, Stefan Ram wrote: > > > So, one might actually be able to learn the pronunciation > > > of a foreign language from text in a book better than from > > > an audio tape (or an audio file or a video with sound)! > > > > Such books would certainly help, but I don't think there's any > > substitute for actually hearing the sounds if you want to be > > able to understand the spoken language. > > I think "learning to understand the spoken language" and "learning to > speak without a (foreign) accent" are two different things. I agree that > the former needs exposure to actual people talking (preferably in real > life, where people talk fast, slur endings, omit words, hem and haw, > talk over each other ...). For learning to speak without an accent, just > listening (or talking) to native speakers is probably not sufficient for > the reasons Stefan mentioned plus another one: Outside of a classroom > people usually won't correct your mistakes unless you say something > truly incomprehensible or unintentionally funny. However I don't think a > book is sufficient either: Most people are probably even worse at > observing the position of their various mouth parts while speaking than > at listening, so without feedback from a native speaker (preferably a > trained voice coach) they can't really tell whether they are doing it > right. > > Hmmm - - - - fascinating discussion on language learning. I would suggest that adults CAN learn other languages. One factor that hasn't been mentioned is the musicality of the individual. I added 3 languages, to varying degrees, as an adult and have done some functioning in about 5 more than the 3 that I developed through childhood and early youth. the use of the IPA (international phonetics alphabet) was encouraged in my formal studies, to be done on my own time and fashion, and its use was very very helpful for much in this area. As far as learning to speak without an accent - - - that does not necessarily coincide with actual knowledge - - - imo there is a definite difference between a language 'in the ear' and 'in the mind'. I have found each language to feel different in both the mouth AND in the brain (and have found the differences quite fascinating). Thank you for very very interesting discussions - - - lol - - - which perhaps should have early on be relabelled as OT - - - grin! Pace From Marco.Sulla.Python at gmail.com Sat May 7 12:28:47 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Sat, 7 May 2022 18:28:47 +0200 Subject: tail In-Reply-To: <60454E09-0ADA-4881-A84B-6C11397D244F@barrys-emacs.org> References: <60454E09-0ADA-4881-A84B-6C11397D244F@barrys-emacs.org> Message-ID: On Sat, 7 May 2022 at 16:08, Barry wrote: > You need to handle the file in bin mode and do the handling of line endings and encodings yourself. It?s not that hard for the cases you wanted. >>> "\n".encode("utf-16") b'\xff\xfe\n\x00' >>> "".encode("utf-16") b'\xff\xfe' >>> "a\nb".encode("utf-16") b'\xff\xfea\x00\n\x00b\x00' >>> "\n".encode("utf-16").lstrip("".encode("utf-16")) b'\n\x00' Can I use the last trick to get the encoding of a LF or a CR in any encoding? From drsalists at gmail.com Sat May 7 12:54:28 2022 From: drsalists at gmail.com (Dan Stromberg) Date: Sat, 7 May 2022 09:54:28 -0700 Subject: tail In-Reply-To: References: Message-ID: I believe I'd do something like: #!/usr/local/cpython-3.10/bin/python3 """ Output the last 10 lines of a potentially-huge file. O(n). But technically so is scanning backward from the EOF. It'd be faster to use a dict, but this has the advantage of working for huge num_lines. """ import dbm import os import sys tempfile = f'/tmp/{os.path.basename(sys.argv[0])}.{os.getpid()}' db = dbm.open(tempfile, 'n') num_lines = 10 for cur_lineno, line in enumerate(sys.stdin): db[str(cur_lineno)] = line.encode('utf-8') max_lineno = cur_lineno str_age_out_lineno = str(cur_lineno - num_lines - 1) if str_age_out_lineno in db: del db[str_age_out_lineno] for lineno in range(max_lineno, max_lineno - num_lines, -1): str_lineno = str(lineno) if str_lineno not in db: break print(db[str(lineno)].decode('utf-8'), end='') db.close() os.unlink(tempfile) On Sat, Apr 23, 2022 at 11:36 AM Marco Sulla wrote: > What about introducing a method for text streams that reads the lines > from the bottom? Java has also a ReversedLinesFileReader with Apache > Commons IO. > -- > https://mail.python.org/mailman/listinfo/python-list > From python at mrabarnett.plus.com Sat May 7 12:58:05 2022 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 7 May 2022 17:58:05 +0100 Subject: tail In-Reply-To: References: <60454E09-0ADA-4881-A84B-6C11397D244F@barrys-emacs.org> Message-ID: <561ac7a8-2034-c1ce-6fca-f4280baac409@mrabarnett.plus.com> On 2022-05-07 17:28, Marco Sulla wrote: > On Sat, 7 May 2022 at 16:08, Barry wrote: >> You need to handle the file in bin mode and do the handling of line endings and encodings yourself. It?s not that hard for the cases you wanted. > >>>> "\n".encode("utf-16") > b'\xff\xfe\n\x00' >>>> "".encode("utf-16") > b'\xff\xfe' >>>> "a\nb".encode("utf-16") > b'\xff\xfea\x00\n\x00b\x00' >>>> "\n".encode("utf-16").lstrip("".encode("utf-16")) > b'\n\x00' > > Can I use the last trick to get the encoding of a LF or a CR in any encoding? In the case of UTF-16, it's 2 bytes per code unit, but those 2 bytes could be little-endian or big-endian. As you didn't specify which you wanted, it defaulted to little-endian and added a BOM (U+FEFF). If you specify which endianness you want with "utf-16le" or "utf-16be", it won't add the BOM: >>> # Little-endian. >>> "\n".encode("utf-16le") b'\n\x00' >>> # Big-endian. >>> "\n".encode("utf-16be") b'\x00\n' From Marco.Sulla.Python at gmail.com Sat May 7 14:35:34 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Sat, 7 May 2022 20:35:34 +0200 Subject: tail In-Reply-To: <561ac7a8-2034-c1ce-6fca-f4280baac409@mrabarnett.plus.com> References: <60454E09-0ADA-4881-A84B-6C11397D244F@barrys-emacs.org> <561ac7a8-2034-c1ce-6fca-f4280baac409@mrabarnett.plus.com> Message-ID: On Sat, 7 May 2022 at 19:02, MRAB wrote: > > On 2022-05-07 17:28, Marco Sulla wrote: > > On Sat, 7 May 2022 at 16:08, Barry wrote: > >> You need to handle the file in bin mode and do the handling of line endings and encodings yourself. It?s not that hard for the cases you wanted. > > > >>>> "\n".encode("utf-16") > > b'\xff\xfe\n\x00' > >>>> "".encode("utf-16") > > b'\xff\xfe' > >>>> "a\nb".encode("utf-16") > > b'\xff\xfea\x00\n\x00b\x00' > >>>> "\n".encode("utf-16").lstrip("".encode("utf-16")) > > b'\n\x00' > > > > Can I use the last trick to get the encoding of a LF or a CR in any encoding? > > In the case of UTF-16, it's 2 bytes per code unit, but those 2 bytes > could be little-endian or big-endian. > > As you didn't specify which you wanted, it defaulted to little-endian > and added a BOM (U+FEFF). > > If you specify which endianness you want with "utf-16le" or "utf-16be", > it won't add the BOM: > > >>> # Little-endian. > >>> "\n".encode("utf-16le") > b'\n\x00' > >>> # Big-endian. > >>> "\n".encode("utf-16be") > b'\x00\n' Well, ok, but I need a generic method to get LF and CR for any encoding an user can input. Do you think that "\n".encode(encoding).lstrip("".encode(encoding)) is good for any encoding? Furthermore, is there a way to get the encoding of an opened file object? From wlfraed at ix.netcom.com Sat May 7 15:08:57 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Sat, 07 May 2022 15:08:57 -0400 Subject: tail References: <60454E09-0ADA-4881-A84B-6C11397D244F@barrys-emacs.org> <561ac7a8-2034-c1ce-6fca-f4280baac409@mrabarnett.plus.com> Message-ID: <33gd7hdkl3aj1bicf9c739hdrnbqcvanro@4ax.com> On Sat, 7 May 2022 20:35:34 +0200, Marco Sulla declaimed the following: >Well, ok, but I need a generic method to get LF and CR for any >encoding an user can input. Other than EBCDIC, and AS BYTES should appear as x0A and x0D in any of the 8-bit encodings (ASCII, ISO-8859-x, CPxxxx, UTF-8). I believe those bytes also appear in UTF-16 -- BUT, they will have a null (x00) byte associated with them as padding; as a result, you can not search for just x0Dx0A (Windows line end convention -- they may be x00x0Dx00x0A or x0Dx00x0Ax00 depending on endianness cf: https://docs.microsoft.com/en-us/cpp/text/support-for-unicode?view=msvc-170 ) For EBCDIC is still x0D, but is x25 (and there is a separate [new line] at x15) -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From python at mrabarnett.plus.com Sat May 7 15:19:04 2022 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 7 May 2022 20:19:04 +0100 Subject: tail In-Reply-To: References: <60454E09-0ADA-4881-A84B-6C11397D244F@barrys-emacs.org> <561ac7a8-2034-c1ce-6fca-f4280baac409@mrabarnett.plus.com> Message-ID: On 2022-05-07 19:35, Marco Sulla wrote: > On Sat, 7 May 2022 at 19:02, MRAB wrote: > > > > On 2022-05-07 17:28, Marco Sulla wrote: > > > On Sat, 7 May 2022 at 16:08, Barry wrote: > > >> You need to handle the file in bin mode and do the handling of line endings and encodings yourself. It?s not that hard for the cases you wanted. > > > > > >>>> "\n".encode("utf-16") > > > b'\xff\xfe\n\x00' > > >>>> "".encode("utf-16") > > > b'\xff\xfe' > > >>>> "a\nb".encode("utf-16") > > > b'\xff\xfea\x00\n\x00b\x00' > > >>>> "\n".encode("utf-16").lstrip("".encode("utf-16")) > > > b'\n\x00' > > > > > > Can I use the last trick to get the encoding of a LF or a CR in any encoding? > > > > In the case of UTF-16, it's 2 bytes per code unit, but those 2 bytes > > could be little-endian or big-endian. > > > > As you didn't specify which you wanted, it defaulted to little-endian > > and added a BOM (U+FEFF). > > > > If you specify which endianness you want with "utf-16le" or "utf-16be", > > it won't add the BOM: > > > > >>> # Little-endian. > > >>> "\n".encode("utf-16le") > > b'\n\x00' > > >>> # Big-endian. > > >>> "\n".encode("utf-16be") > > b'\x00\n' > > Well, ok, but I need a generic method to get LF and CR for any > encoding an user can input. > Do you think that > > "\n".encode(encoding).lstrip("".encode(encoding)) > > is good for any encoding? '.lstrip' is the wrong method to use because it treats its argument as a set of characters, so it might strip off too many characters. A better choice is '.removeprefix'. > Furthermore, is there a way to get the encoding of an opened file object? > How was the file opened? If it was opened as a text file, use the '.encoding' attribute (which just tells you what encoding was specified when it was opened, and you'd be assuming that it's the correct one). If it was opened as a binary file, all you know is that it contains bytes, and determining the encoding (assuming that it is a text file) is down to heuristics (i.e. guesswork). From python at mrabarnett.plus.com Sat May 7 15:26:01 2022 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 7 May 2022 20:26:01 +0100 Subject: tail In-Reply-To: References: <60454E09-0ADA-4881-A84B-6C11397D244F@barrys-emacs.org> <561ac7a8-2034-c1ce-6fca-f4280baac409@mrabarnett.plus.com> Message-ID: <983cc36d-0727-23ab-f168-ae88b14e6934@mrabarnett.plus.com> On 2022-05-07 19:47, Stefan Ram wrote: > Marco Sulla writes: >>Well, ok, but I need a generic method to get LF and CR for any >>encoding an user can input. > > "LF" and "CR" come from US-ASCII. It is theoretically > possible that there might be some encodings out there > (not for Unicode) that are not based on US-ASCII and > have no LF or no CR. > >>is good for any encoding? Furthermore, is there a way to get the >>encoding of an opened file object? > > I have written a function that might be able to detect one > of few encodings based on a heuristic algorithm. > > def encoding( name ): > path = pathlib.Path( name ) > for encoding in( "utf_8", "latin_1", "cp1252" ): > try: > with path.open( encoding=encoding, errors="strict" )as file: > text = file.read() > return encoding > except UnicodeDecodeError: > pass > return "ascii" > > Yes, it's potentially slow and might be wrong. > The result "ascii" might mean it's a binary file. > "latin-1" will decode any sequence of bytes, so it'll never try "cp1252", nor fall back to "ascii", and falling back to "ascii" is wrong anyway because the file could contain 0x80..0xFF, which aren't supported by that encoding. From rosuav at gmail.com Sat May 7 16:12:01 2022 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 8 May 2022 06:12:01 +1000 Subject: tail In-Reply-To: References: <60454E09-0ADA-4881-A84B-6C11397D244F@barrys-emacs.org> <561ac7a8-2034-c1ce-6fca-f4280baac409@mrabarnett.plus.com> Message-ID: On Sun, 8 May 2022 at 04:37, Marco Sulla wrote: > > On Sat, 7 May 2022 at 19:02, MRAB wrote: > > > > On 2022-05-07 17:28, Marco Sulla wrote: > > > On Sat, 7 May 2022 at 16:08, Barry wrote: > > >> You need to handle the file in bin mode and do the handling of line endings and encodings yourself. It?s not that hard for the cases you wanted. > > > > > >>>> "\n".encode("utf-16") > > > b'\xff\xfe\n\x00' > > >>>> "".encode("utf-16") > > > b'\xff\xfe' > > >>>> "a\nb".encode("utf-16") > > > b'\xff\xfea\x00\n\x00b\x00' > > >>>> "\n".encode("utf-16").lstrip("".encode("utf-16")) > > > b'\n\x00' > > > > > > Can I use the last trick to get the encoding of a LF or a CR in any encoding? > > > > In the case of UTF-16, it's 2 bytes per code unit, but those 2 bytes > > could be little-endian or big-endian. > > > > As you didn't specify which you wanted, it defaulted to little-endian > > and added a BOM (U+FEFF). > > > > If you specify which endianness you want with "utf-16le" or "utf-16be", > > it won't add the BOM: > > > > >>> # Little-endian. > > >>> "\n".encode("utf-16le") > > b'\n\x00' > > >>> # Big-endian. > > >>> "\n".encode("utf-16be") > > b'\x00\n' > > Well, ok, but I need a generic method to get LF and CR for any > encoding an user can input. > Do you think that > > "\n".encode(encoding).lstrip("".encode(encoding)) > > is good for any encoding? No, because it is only useful for stateless encodings. Any encoding which uses "shift bytes" that cause subsequent bytes to be interpreted differently will simply not work with this naive technique. Also, you're assuming that the byte(s) you get from encoding LF will *only* represent LF, which is also not true for a number of other encodings - they might always encode LF to the same byte sequence, but could use that same byte sequence as part of a multi-byte encoding. So, no, for arbitrarily chosen encodings, this is not dependable. > Furthermore, is there a way to get the > encoding of an opened file object? Nope. That's fundamentally not possible. Unless you mean in the trivial sense of "what was the parameter passed to the open() call?", in which case f.encoding will give it to you; but to find out the actual encoding, no, you can't. The ONLY way to 100% reliably decode arbitrary text is to know, from external information, what encoding it is in. Every other scheme imposes restrictions. Trying to do something that works for absolutely any encoding is a doomed project. ChrisA From grant.b.edwards at gmail.com Sat May 7 16:37:50 2022 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Sat, 07 May 2022 13:37:50 -0700 (PDT) Subject: Python/New/Learn References: <7e7bf8e6-344c-45a9-b02e-2e0c854910bfn@googlegroups.com> <62746315.1c69fb81.9502b.350f@mx.google.com> <47a62108-a48c-1a2f-41b0-f326a01dc0d9@DancesWithMice.info> Message-ID: <6276d89e.1c69fb81.92f31.9270@mx.google.com> On 2022-05-06, dn wrote: > The problem with some of the advice given in this thread, eg using > StackOverflow or YouTube videos, is that a beginner (particularly) > has no measure of the material's quality. Both platforms are riddled > with utter-junk - even 'dangerous' advice. And the "quality level" of such online forum answers seems to vary widely by subject. They're not nearly as bad for Python as they are for PHP and Javascript. Your chances of finding a correct answer to a PHP question are virtually nil. [It dosn't help that the "correct answer" often changes between versions -- even minor ones.] -- Grant From rosuav at gmail.com Sat May 7 17:31:48 2022 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 8 May 2022 07:31:48 +1000 Subject: tail In-Reply-To: References: <60454E09-0ADA-4881-A84B-6C11397D244F@barrys-emacs.org> <561ac7a8-2034-c1ce-6fca-f4280baac409@mrabarnett.plus.com> <983cc36d-0727-23ab-f168-ae88b14e6934@mrabarnett.plus.com> Message-ID: On Sun, 8 May 2022 at 07:19, Stefan Ram wrote: > > MRAB writes: > >On 2022-05-07 19:47, Stefan Ram wrote: > ... > >>def encoding( name ): > >> path = pathlib.Path( name ) > >> for encoding in( "utf_8", "latin_1", "cp1252" ): > >> try: > >> with path.open( encoding=encoding, errors="strict" )as file: > >> text = file.read() > >> return encoding > >> except UnicodeDecodeError: > >> pass > >> return "ascii" > >>Yes, it's potentially slow and might be wrong. > >>The result "ascii" might mean it's a binary file. > >"latin-1" will decode any sequence of bytes, so it'll never try > >"cp1252", nor fall back to "ascii", and falling back to "ascii" is wrong > >anyway because the file could contain 0x80..0xFF, which aren't supported > >by that encoding. > > Thank you! It's working for my specific application where > I'm reading from a collection of text files that should be > encoded in either utf_8, latin_1, or ascii. > In that case, I'd exclude ASCII from the check, and just check UTF-8, and if that fails, decode as Latin-1. Any ASCII files will decode correctly as UTF-8, and any file will decode as Latin-1. I've used this exact fallback system when decoding raw data from Unicode-naive servers - they accept and share bytes, so it's entirely possible to have a mix of encodings in a single stream. As long as you can define the span of a single "unit" (say, a line, or a chunk in some form), you can read as bytes and do the exact same "decode as UTF-8 if possible, otherwise decode as Latin-1" dance. Sure, it's not perfectly ideal, but it's about as good as you'll get with a lot of US-based servers. (Depending on context, you might use CP-1252 instead of Latin-1, but you might need errors="replace" there, since Windows-1252 has some undefined byte values.) ChrisA From pablogsal at gmail.com Sat May 7 23:22:00 2022 From: pablogsal at gmail.com (Pablo Galindo Salgado) Date: Sun, 8 May 2022 04:22:00 +0100 Subject: [RELEASE] The first Python 3.11 beta (3.11.0b1) is available - Feature freeze is here Message-ID: We did it, team!! After quite a bumpy release process and a bunch of last-time fixes, we have reached **beta 1** and **feature freeze**. What a ride eh? You can get the shiny new release artefacts from here: https://www.python.org/downloads/release/python-3110b1/ ## This is a beta preview of Python 3.11 Python 3.11 is still in development. 3.11.0b1 is the first of four planned beta release previews. Beta release previews are intended to give the wider community the opportunity to test new features and bug fixes and to prepare their projects to support the new feature release. We **strongly encourage** maintainers of third-party Python projects to **test with 3.11** during the beta phase and report issues found to [the Python bug tracker](https://bugs.python.org) as soon as possible. While the release is planned to be feature complete entering the beta phase, it is possible that features may be modified or, in rare cases, deleted up until the start of the release candidate phase (Monday, 2021-08-02). Our goal is to have no ABI changes after beta 4 and as few code changes as possible after 3.11.0rc1, the first release candidate. To achieve that, it will be **extremely important** to get as much exposure for 3.11 as possible during the beta phase. Please keep in mind that this is a preview release and its use is **not** recommended for production environments. # Major new features of the 3.11 series, compared to 3.10 Python 3.11 is still in development. This release, 3.11.0b1 is the **first** of four beta releases. Beta release previews are intended to give the wider community the opportunity to test new features and bug fixes and to prepare their projects to support the new feature release. Many new features for Python 3.11 are still being planned and written. Among the new major new features and changes so far: * [PEP 657](https://www.python.org/dev/peps/pep-0657/) -- Include Fine-Grained Error Locations in Tracebacks * [PEP 654](https://www.python.org/dev/peps/pep-0654/) -- Exception Groups and except* * [PEP 673](https://www.python.org/dev/peps/pep-0673/) -- Self Type * [PEP 646](https://www.python.org/dev/peps/pep-0646/)-- Variadic Generics * [PEP 680](https://www.python.org/dev/peps/pep-0680/)-- tomllib: Support for Parsing TOML in the Standard Library * [PEP 675](https://www.python.org/dev/peps/pep-0675/)-- Arbitrary Literal String Type * [PEP 655](https://www.python.org/dev/peps/pep-0655/)-- Marking individual TypedDict items as required or potentially-missing * [bpo-46752](https://bugs.python.org/issue46752)-- Introduce task groups to asyncio * The Faster Cpython Project is already yielding some exciting results. Python 3.11 is up to 10-60% faster than Python 3.10. On average, we measured a 1.22x speedup on the standard benchmark suite. See Faster CPython for details. * Hey, **fellow core developer,** if a feature you find important is missing from this list, let me know. The next pre-release of Python 3.11 will be 3.11.0b2, currently scheduled for Monday, 2022-05-30. # More resources * [Online Documentation](https://docs.python.org/3.11/) * [PEP 664](https://www.python.org/dev/peps/pep-0664/), 3.11 Release Schedule * Report bugs at [https://bugs.python.org](https://bugs.python.org). * [Help fund Python and its community](/psf/donations/). # And now for something completely different The holographic principle is a tenet of string theories and a supposed property of quantum gravity that states that the description of a volume of space can be thought of as encoded on a lower-dimensional boundary to the region?such as a light-like boundary like a gravitational horizon. First proposed by Gerard 't Hooft, it was given a precise string-theory interpretation by Leonard Susskind, who combined his ideas with previous ones of 't Hooft and Charles Thorn.[ Leonard Susskind said, ?The three-dimensional world of ordinary experience??the universe filled with galaxies, stars, planets, houses, boulders, and people??is a hologram, an image of reality cited on a distant two-dimensional (2D) surface." As pointed out by Raphael Bousso, Thorn observed in 1978 that string theory admits a lower-dimensional description in which gravity emerges from it in what would now be called a holographic way. The holographic principle was inspired by black hole thermodynamics, which conjectures that the maximal entropy in any region scales with the radius squared, and not cubed as might be expected. In the case of a black hole, the insight was that the informational content of all the objects that have fallen into the hole might be entirely contained in surface fluctuations of the event horizon. The holographic principle resolves the black hole information paradox within the framework of string theory. However, there exist classical solutions to the Einstein equations that allow values of the entropy larger than those allowed by an area law, hence in principle larger than those of a black hole. These are the so-called "Wheeler's bags of gold". The existence of such solutions conflicts with the holographic interpretation, and their effects in a quantum theory of gravity including the holographic principle are not fully understood yet. # We hope you enjoy the new releases! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. https://www.python.org/psf/ Regards from chilly London, Your friendly release team, Pablo Galindo Salgado From barry at barrys-emacs.org Sun May 8 04:04:13 2022 From: barry at barrys-emacs.org (Barry) Date: Sun, 8 May 2022 09:04:13 +0100 Subject: tail In-Reply-To: References: Message-ID: > On 7 May 2022, at 17:29, Marco Sulla wrote: > > ?On Sat, 7 May 2022 at 16:08, Barry wrote: >> You need to handle the file in bin mode and do the handling of line endings and encodings yourself. It?s not that hard for the cases you wanted. > >>>> "\n".encode("utf-16") > b'\xff\xfe\n\x00' >>>> "".encode("utf-16") > b'\xff\xfe' >>>> "a\nb".encode("utf-16") > b'\xff\xfea\x00\n\x00b\x00' >>>> "\n".encode("utf-16").lstrip("".encode("utf-16")) > b'\n\x00' > > Can I use the last trick to get the encoding of a LF or a CR in any encoding? In a word no. There are cases that you just have to know the encoding you are working with. utf-16 because you have deal with the data in 2 byte units and know if it is big endian or little endian. There will be other encoding that will also be difficult. But if you are working with encoding that are using ASCII as a base, like unicode encoded as utf-8 or iso-8859 series then you can just look for NL and CR using the ASCII values of the byte. In short once you set your requirements then you can know what problems you can avoid and which you must solve. Is utf-16 important to you? If not no need to solve its issues. Barry From Marco.Sulla.Python at gmail.com Sun May 8 12:05:11 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Sun, 8 May 2022 18:05:11 +0200 Subject: tail In-Reply-To: References: Message-ID: I think I've _almost_ found a simpler, general way: import os _lf = "\n" _cr = "\r" def tail(filepath, n=10, newline=None, encoding=None, chunk_size=100): n_chunk_size = n * chunk_size pos = os.stat(filepath).st_size chunk_line_pos = -1 lines_not_found = n with open(filepath, newline=newline, encoding=encoding) as f: text = "" hard_mode = False if newline == None: newline = _lf elif newline == "": hard_mode = True if hard_mode: while pos != 0: pos -= n_chunk_size if pos < 0: pos = 0 f.seek(pos) text = f.read() lf_after = False for i, char in enumerate(reversed(text)): if char == _lf: lf_after == True elif char == _cr: lines_not_found -= 1 newline_size = 2 if lf_after else 1 lf_after = False elif lf_after: lines_not_found -= 1 newline_size = 1 lf_after = False if lines_not_found == 0: chunk_line_pos = len(text) - 1 - i + newline_size break if lines_not_found == 0: break else: while pos != 0: pos -= n_chunk_size if pos < 0: pos = 0 f.seek(pos) text = f.read() for i, char in enumerate(reversed(text)): if char == newline: lines_not_found -= 1 if lines_not_found == 0: chunk_line_pos = len(text) - 1 - i + len(newline) break if lines_not_found == 0: break if chunk_line_pos == -1: chunk_line_pos = 0 return text[chunk_line_pos:] Shortly, the file is always opened in text mode. File is read at the end in bigger and bigger chunks, until the file is finished or all the lines are found. Why? Because in encodings that have more than 1 byte per character, reading a chunk of n bytes, then reading the previous chunk, can eventually split the character between the chunks in two distinct bytes. I think one can read chunk by chunk and test the chunk junction problem. I suppose the code will be faster this way. Anyway, it seems that this trick is quite fast anyway and it's a lot simpler. The final result is read from the chunk, and not from the file, so there's no problems of misalignment of bytes and text. Furthermore, the builtin encoding parameter is used, so this should work with all the encodings (untested). Furthermore, a newline parameter can be specified, as in open(). If it's equal to the empty string, the things are a little more complicated, anyway I suppose the code is clear. It's untested too. I only tested with an utf8 linux file. Do you think there are chances to get this function as a method of the file object in CPython? The method for a file object opened in bytes mode is simpler, since there's no encoding and newline is only \n in that case. From barry at barrys-emacs.org Sun May 8 14:10:30 2022 From: barry at barrys-emacs.org (Barry Scott) Date: Sun, 8 May 2022 19:10:30 +0100 Subject: tail In-Reply-To: References: <343b7hlfae195bt4inm3sg9es39bd6butk@4ax.com> Message-ID: <6AF4F78E-5BE1-429E-9D45-197E441A02DD@barrys-emacs.org> > On 7 May 2022, at 14:40, Stefan Ram wrote: > > Marco Sulla writes: >> So there's no way to reliably read lines in reverse in text mode using >> seek and read, but the only option is readlines? > > I think, CPython is based on C. I don't know whether > Python's seek function directly calls C's fseek function, > but maybe the following parts of the C standard also are > relevant for Python? There is the posix API that and the C FILE API. I expect that the odities you about NUL chars is all about the FILE API. As far as I know its the posix API that C Python uses and it does not suffer from issues with binary files. Barry > > |Setting the file position indicator to end-of-file, as with > |fseek(file, 0, SEEK_END), has undefined behavior for a binary > |stream (because of possible trailing null characters) or for > |any stream with state-dependent encoding that does not > |assuredly end in the initial shift state. > from a footnote in a draft of a C standard > > |For a text stream, either offset shall be zero, or offset > |shall be a value returned by an earlier successful call to > |the ftell function on a stream associated with the same file > |and whence shall be SEEK_SET. > from a draft of a C standard > > |A text stream is an ordered sequence of characters composed > |into lines, each line consisting of zero or more characters > |plus a terminating new-line character. Whether the last line > |requires a terminating new-line character is implementation-defined. > from a draft of a C standard > > This might mean that reading from a text stream that is not > ending in a new-line character might have undefined behavior > (depending on the C implementation). In practice, it might > mean that some things could go wrong near the end of such > a stream. > > > -- > https://mail.python.org/mailman/listinfo/python-list > From barry at barrys-emacs.org Sun May 8 14:15:09 2022 From: barry at barrys-emacs.org (Barry Scott) Date: Sun, 8 May 2022 19:15:09 +0100 Subject: tail In-Reply-To: References: <60454E09-0ADA-4881-A84B-6C11397D244F@barrys-emacs.org> <561ac7a8-2034-c1ce-6fca-f4280baac409@mrabarnett.plus.com> <983cc36d-0727-23ab-f168-ae88b14e6934@mrabarnett.plus.com> Message-ID: <0DC070E5-2BCC-47AA-9DDE-4EE7B3F3D441@barrys-emacs.org> > On 7 May 2022, at 22:31, Chris Angelico wrote: > > On Sun, 8 May 2022 at 07:19, Stefan Ram wrote: >> >> MRAB writes: >>> On 2022-05-07 19:47, Stefan Ram wrote: >> ... >>>> def encoding( name ): >>>> path = pathlib.Path( name ) >>>> for encoding in( "utf_8", "latin_1", "cp1252" ): >>>> try: >>>> with path.open( encoding=encoding, errors="strict" )as file: >>>> text = file.read() >>>> return encoding >>>> except UnicodeDecodeError: >>>> pass >>>> return "ascii" >>>> Yes, it's potentially slow and might be wrong. >>>> The result "ascii" might mean it's a binary file. >>> "latin-1" will decode any sequence of bytes, so it'll never try >>> "cp1252", nor fall back to "ascii", and falling back to "ascii" is wrong >>> anyway because the file could contain 0x80..0xFF, which aren't supported >>> by that encoding. >> >> Thank you! It's working for my specific application where >> I'm reading from a collection of text files that should be >> encoded in either utf_8, latin_1, or ascii. >> > > In that case, I'd exclude ASCII from the check, and just check UTF-8, > and if that fails, decode as Latin-1. Any ASCII files will decode > correctly as UTF-8, and any file will decode as Latin-1. > > I've used this exact fallback system when decoding raw data from > Unicode-naive servers - they accept and share bytes, so it's entirely > possible to have a mix of encodings in a single stream. As long as you > can define the span of a single "unit" (say, a line, or a chunk in > some form), you can read as bytes and do the exact same "decode as > UTF-8 if possible, otherwise decode as Latin-1" dance. Sure, it's not > perfectly ideal, but it's about as good as you'll get with a lot of > US-based servers. (Depending on context, you might use CP-1252 instead > of Latin-1, but you might need errors="replace" there, since > Windows-1252 has some undefined byte values.) There is a very common error on Windows that files and especially web pages that claim to be utf-8 are in fact CP-1252. There is logic in the HTML standards to try utf-8 and if it fails fall back to CP-1252. Its usually the left and "smart" quote chars that cause the issue as they code as an invalid utf-8. Barry > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > From rosuav at gmail.com Sun May 8 14:27:10 2022 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 9 May 2022 04:27:10 +1000 Subject: tail In-Reply-To: <0DC070E5-2BCC-47AA-9DDE-4EE7B3F3D441@barrys-emacs.org> References: <60454E09-0ADA-4881-A84B-6C11397D244F@barrys-emacs.org> <561ac7a8-2034-c1ce-6fca-f4280baac409@mrabarnett.plus.com> <983cc36d-0727-23ab-f168-ae88b14e6934@mrabarnett.plus.com> <0DC070E5-2BCC-47AA-9DDE-4EE7B3F3D441@barrys-emacs.org> Message-ID: On Mon, 9 May 2022 at 04:15, Barry Scott wrote: > > > > > On 7 May 2022, at 22:31, Chris Angelico wrote: > > > > On Sun, 8 May 2022 at 07:19, Stefan Ram wrote: > >> > >> MRAB writes: > >>> On 2022-05-07 19:47, Stefan Ram wrote: > >> ... > >>>> def encoding( name ): > >>>> path = pathlib.Path( name ) > >>>> for encoding in( "utf_8", "latin_1", "cp1252" ): > >>>> try: > >>>> with path.open( encoding=encoding, errors="strict" )as file: > >>>> text = file.read() > >>>> return encoding > >>>> except UnicodeDecodeError: > >>>> pass > >>>> return "ascii" > >>>> Yes, it's potentially slow and might be wrong. > >>>> The result "ascii" might mean it's a binary file. > >>> "latin-1" will decode any sequence of bytes, so it'll never try > >>> "cp1252", nor fall back to "ascii", and falling back to "ascii" is wrong > >>> anyway because the file could contain 0x80..0xFF, which aren't supported > >>> by that encoding. > >> > >> Thank you! It's working for my specific application where > >> I'm reading from a collection of text files that should be > >> encoded in either utf_8, latin_1, or ascii. > >> > > > > In that case, I'd exclude ASCII from the check, and just check UTF-8, > > and if that fails, decode as Latin-1. Any ASCII files will decode > > correctly as UTF-8, and any file will decode as Latin-1. > > > > I've used this exact fallback system when decoding raw data from > > Unicode-naive servers - they accept and share bytes, so it's entirely > > possible to have a mix of encodings in a single stream. As long as you > > can define the span of a single "unit" (say, a line, or a chunk in > > some form), you can read as bytes and do the exact same "decode as > > UTF-8 if possible, otherwise decode as Latin-1" dance. Sure, it's not > > perfectly ideal, but it's about as good as you'll get with a lot of > > US-based servers. (Depending on context, you might use CP-1252 instead > > of Latin-1, but you might need errors="replace" there, since > > Windows-1252 has some undefined byte values.) > > There is a very common error on Windows that files and especially web pages that > claim to be utf-8 are in fact CP-1252. > > There is logic in the HTML standards to try utf-8 and if it fails fall back to CP-1252. > > Its usually the left and "smart" quote chars that cause the issue as they code > as an invalid utf-8. > Yeah, or sometimes, there isn't *anything* in UTF-8, and it has some sort of straight-up lie in the form of a meta tag. It's annoying. But the same logic still applies: attempt one decode (UTF-8) and if it fails, there's one fallback. Fairly simple. ChrisA From barry at barrys-emacs.org Sun May 8 14:31:45 2022 From: barry at barrys-emacs.org (Barry Scott) Date: Sun, 8 May 2022 19:31:45 +0100 Subject: tail In-Reply-To: References: Message-ID: <3848780F-83B8-4B5F-BFAD-157390288C15@barrys-emacs.org> > On 8 May 2022, at 17:05, Marco Sulla wrote: > > I think I've _almost_ found a simpler, general way: > > import os > > _lf = "\n" > _cr = "\r" > > def tail(filepath, n=10, newline=None, encoding=None, chunk_size=100): > n_chunk_size = n * chunk_size Why use tiny chunks? You can read 4KiB as fast as 100 bytes as its typically the smaller size the file system will allocate. I tend to read on multiple of MiB as its near instant. > pos = os.stat(filepath).st_size You cannot mix POSIX API with text mode. pos is in bytes from the start of the file. Textmode will be in code points. bytes != code points. > chunk_line_pos = -1 > lines_not_found = n > > with open(filepath, newline=newline, encoding=encoding) as f: > text = "" > > hard_mode = False > > if newline == None: > newline = _lf > elif newline == "": > hard_mode = True > > if hard_mode: > while pos != 0: > pos -= n_chunk_size > > if pos < 0: > pos = 0 > > f.seek(pos) In text mode you can only seek to a value return from f.tell() otherwise the behaviour is undefined. > text = f.read() You have on limit on the amount of data read. > lf_after = False > > for i, char in enumerate(reversed(text)): Simple use text.rindex('\n') or text.rfind('\n') for speed. > if char == _lf: > lf_after == True > elif char == _cr: > lines_not_found -= 1 > > newline_size = 2 if lf_after else 1 > > lf_after = False > elif lf_after: > lines_not_found -= 1 > newline_size = 1 > lf_after = False > > > if lines_not_found == 0: > chunk_line_pos = len(text) - 1 - i + newline_size > break > > if lines_not_found == 0: > break > else: > while pos != 0: > pos -= n_chunk_size > > if pos < 0: > pos = 0 > > f.seek(pos) > text = f.read() > > for i, char in enumerate(reversed(text)): > if char == newline: > lines_not_found -= 1 > > if lines_not_found == 0: > chunk_line_pos = len(text) - 1 - i + > len(newline) > break > > if lines_not_found == 0: > break > > > if chunk_line_pos == -1: > chunk_line_pos = 0 > > return text[chunk_line_pos:] > > > Shortly, the file is always opened in text mode. File is read at the end in > bigger and bigger chunks, until the file is finished or all the lines are > found. It will fail if the contents is not ASCII. > > Why? Because in encodings that have more than 1 byte per character, reading > a chunk of n bytes, then reading the previous chunk, can eventually split > the character between the chunks in two distinct bytes. No it cannot. text mode only knows how to return code points. Now if you are in binary it could be split, but you are not in binary mode so it cannot. > I think one can read chunk by chunk and test the chunk junction problem. I > suppose the code will be faster this way. Anyway, it seems that this trick > is quite fast anyway and it's a lot simpler. > The final result is read from the chunk, and not from the file, so there's > no problems of misalignment of bytes and text. Furthermore, the builtin > encoding parameter is used, so this should work with all the encodings > (untested). > > Furthermore, a newline parameter can be specified, as in open(). If it's > equal to the empty string, the things are a little more complicated, anyway > I suppose the code is clear. It's untested too. I only tested with an utf8 > linux file. > > Do you think there are chances to get this function as a method of the file > object in CPython? The method for a file object opened in bytes mode is > simpler, since there's no encoding and newline is only \n in that case. State your requirements. Then see if your implementation meets them. Barry > -- > https://mail.python.org/mailman/listinfo/python-list > From python at mrabarnett.plus.com Sun May 8 14:50:21 2022 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 8 May 2022 19:50:21 +0100 Subject: tail In-Reply-To: <0DC070E5-2BCC-47AA-9DDE-4EE7B3F3D441@barrys-emacs.org> References: <60454E09-0ADA-4881-A84B-6C11397D244F@barrys-emacs.org> <561ac7a8-2034-c1ce-6fca-f4280baac409@mrabarnett.plus.com> <983cc36d-0727-23ab-f168-ae88b14e6934@mrabarnett.plus.com> <0DC070E5-2BCC-47AA-9DDE-4EE7B3F3D441@barrys-emacs.org> Message-ID: On 2022-05-08 19:15, Barry Scott wrote: > > >> On 7 May 2022, at 22:31, Chris Angelico wrote: >> >> On Sun, 8 May 2022 at 07:19, Stefan Ram wrote: >>> >>> MRAB writes: >>>> On 2022-05-07 19:47, Stefan Ram wrote: >>> ... >>>>> def encoding( name ): >>>>> path = pathlib.Path( name ) >>>>> for encoding in( "utf_8", "latin_1", "cp1252" ): >>>>> try: >>>>> with path.open( encoding=encoding, errors="strict" )as file: >>>>> text = file.read() >>>>> return encoding >>>>> except UnicodeDecodeError: >>>>> pass >>>>> return "ascii" >>>>> Yes, it's potentially slow and might be wrong. >>>>> The result "ascii" might mean it's a binary file. >>>> "latin-1" will decode any sequence of bytes, so it'll never try >>>> "cp1252", nor fall back to "ascii", and falling back to "ascii" is wrong >>>> anyway because the file could contain 0x80..0xFF, which aren't supported >>>> by that encoding. >>> >>> Thank you! It's working for my specific application where >>> I'm reading from a collection of text files that should be >>> encoded in either utf_8, latin_1, or ascii. >>> >> >> In that case, I'd exclude ASCII from the check, and just check UTF-8, >> and if that fails, decode as Latin-1. Any ASCII files will decode >> correctly as UTF-8, and any file will decode as Latin-1. >> >> I've used this exact fallback system when decoding raw data from >> Unicode-naive servers - they accept and share bytes, so it's entirely >> possible to have a mix of encodings in a single stream. As long as you >> can define the span of a single "unit" (say, a line, or a chunk in >> some form), you can read as bytes and do the exact same "decode as >> UTF-8 if possible, otherwise decode as Latin-1" dance. Sure, it's not >> perfectly ideal, but it's about as good as you'll get with a lot of >> US-based servers. (Depending on context, you might use CP-1252 instead >> of Latin-1, but you might need errors="replace" there, since >> Windows-1252 has some undefined byte values.) > > There is a very common error on Windows that files and especially web pages that > claim to be utf-8 are in fact CP-1252. > > There is logic in the HTML standards to try utf-8 and if it fails fall back to CP-1252. > > Its usually the left and "smart" quote chars that cause the issue as they code > as an invalid utf-8. > Is it CP-1252 or ISO-8859-1 (Latin-1)? From Marco.Sulla.Python at gmail.com Sun May 8 15:47:18 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Sun, 8 May 2022 21:47:18 +0200 Subject: tail In-Reply-To: <3848780F-83B8-4B5F-BFAD-157390288C15@barrys-emacs.org> References: <3848780F-83B8-4B5F-BFAD-157390288C15@barrys-emacs.org> Message-ID: On Sun, 8 May 2022 at 20:31, Barry Scott wrote: > > > On 8 May 2022, at 17:05, Marco Sulla wrote: > > > > def tail(filepath, n=10, newline=None, encoding=None, chunk_size=100): > > n_chunk_size = n * chunk_size > > Why use tiny chunks? You can read 4KiB as fast as 100 bytes as its typically the smaller size the file system will allocate. > I tend to read on multiple of MiB as its near instant. Well, I tested on a little file, a list of my preferred pizzas, so.... > > pos = os.stat(filepath).st_size > > You cannot mix POSIX API with text mode. > pos is in bytes from the start of the file. > Textmode will be in code points. bytes != code points. > > > chunk_line_pos = -1 > > lines_not_found = n > > > > with open(filepath, newline=newline, encoding=encoding) as f: > > text = "" > > > > hard_mode = False > > > > if newline == None: > > newline = _lf > > elif newline == "": > > hard_mode = True > > > > if hard_mode: > > while pos != 0: > > pos -= n_chunk_size > > > > if pos < 0: > > pos = 0 > > > > f.seek(pos) > > In text mode you can only seek to a value return from f.tell() otherwise the behaviour is undefined. Why? I don't see any recommendation about it in the docs: https://docs.python.org/3/library/io.html#io.IOBase.seek > > text = f.read() > > You have on limit on the amount of data read. I explained that previously. Anyway, chunk_size is small, so it's not a great problem. > > lf_after = False > > > > for i, char in enumerate(reversed(text)): > > Simple use text.rindex('\n') or text.rfind('\n') for speed. I can't use them when I have to find both \n or \r. So I preferred to simplify the code and use the for cycle every time. Take into mind anyway that this is a prototype for a Python C Api implementation (builtin I hope, or a C extension if not) > > Shortly, the file is always opened in text mode. File is read at the end in > > bigger and bigger chunks, until the file is finished or all the lines are > > found. > > It will fail if the contents is not ASCII. Why? > > Why? Because in encodings that have more than 1 byte per character, reading > > a chunk of n bytes, then reading the previous chunk, can eventually split > > the character between the chunks in two distinct bytes. > > No it cannot. text mode only knows how to return code points. Now if you are in > binary it could be split, but you are not in binary mode so it cannot. >From the docs: seek(offset, whence=SEEK_SET) Change the stream position to the given byte offset. > > Do you think there are chances to get this function as a method of the file > > object in CPython? The method for a file object opened in bytes mode is > > simpler, since there's no encoding and newline is only \n in that case. > > State your requirements. Then see if your implementation meets them. The method should return the last n lines from a file object. If the file object is in text mode, the newline parameter must be honored. If the file object is in binary mode, a newline is always b"\n", to be consistent with readline. I suppose the current implementation of tail satisfies the requirements for text mode. The previous one satisfied binary mode. Anyway, apart from my implementation, I'm curious if you think a tail method is worth it to be a method of the builtin file objects in CPython. From rosuav at gmail.com Sun May 8 16:00:10 2022 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 9 May 2022 06:00:10 +1000 Subject: tail In-Reply-To: References: <3848780F-83B8-4B5F-BFAD-157390288C15@barrys-emacs.org> Message-ID: On Mon, 9 May 2022 at 05:49, Marco Sulla wrote: > Anyway, apart from my implementation, I'm curious if you think a tail > method is worth it to be a method of the builtin file objects in > CPython. Absolutely not. As has been stated multiple times in this thread, a fully general approach is extremely complicated, horrifically unreliable, and hopelessly inefficient. The ONLY way to make this sort of thing any good whatsoever is to know your own use-case and code to exactly that. Given the size of files you're working with, for instance, a simple approach of just reading the whole file would make far more sense than the complex seeking you're doing. For reading a multi-gigabyte file, the choices will be different. No, this does NOT belong in the core language. ChrisA From Marco.Sulla.Python at gmail.com Sun May 8 16:36:06 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Sun, 8 May 2022 22:36:06 +0200 Subject: tail In-Reply-To: References: <3848780F-83B8-4B5F-BFAD-157390288C15@barrys-emacs.org> Message-ID: On Sun, 8 May 2022 at 22:02, Chris Angelico wrote: > > Absolutely not. As has been stated multiple times in this thread, a > fully general approach is extremely complicated, horrifically > unreliable, and hopelessly inefficient. Well, my implementation is quite general now. It's not complicated and inefficient. About reliability, I can't say anything without a test case. > The ONLY way to make this sort > of thing any good whatsoever is to know your own use-case and code to > exactly that. Given the size of files you're working with, for > instance, a simple approach of just reading the whole file would make > far more sense than the complex seeking you're doing. For reading a > multi-gigabyte file, the choices will be different. Apart from the fact that it's very, very simple to optimize for small files: this is, IMHO, a premature optimization. The code is quite fast even if the file is small. Can it be faster? Of course, but it depends on the use case. Every optimization in CPython must pass the benchmark suite test. If there's little or no gain, the optimization is usually rejected. > No, this does NOT belong in the core language. I respect your opinion, but IMHO you think that the task is more complicated than the reality. It seems to me that the method can be quite simple and fast. From barry at barrys-emacs.org Sun May 8 16:34:04 2022 From: barry at barrys-emacs.org (Barry) Date: Sun, 8 May 2022 21:34:04 +0100 Subject: tail In-Reply-To: References: Message-ID: > On 8 May 2022, at 20:48, Marco Sulla wrote: > > ?On Sun, 8 May 2022 at 20:31, Barry Scott wrote: >> >>>> On 8 May 2022, at 17:05, Marco Sulla wrote: >>> >>> def tail(filepath, n=10, newline=None, encoding=None, chunk_size=100): >>> n_chunk_size = n * chunk_size >> >> Why use tiny chunks? You can read 4KiB as fast as 100 bytes as its typically the smaller size the file system will allocate. >> I tend to read on multiple of MiB as its near instant. > > Well, I tested on a little file, a list of my preferred pizzas, so.... Try it on a very big file. > >>> pos = os.stat(filepath).st_size >> >> You cannot mix POSIX API with text mode. >> pos is in bytes from the start of the file. >> Textmode will be in code points. bytes != code points. >> >>> chunk_line_pos = -1 >>> lines_not_found = n >>> >>> with open(filepath, newline=newline, encoding=encoding) as f: >>> text = "" >>> >>> hard_mode = False >>> >>> if newline == None: >>> newline = _lf >>> elif newline == "": >>> hard_mode = True >>> >>> if hard_mode: >>> while pos != 0: >>> pos -= n_chunk_size >>> >>> if pos < 0: >>> pos = 0 >>> >>> f.seek(pos) >> >> In text mode you can only seek to a value return from f.tell() otherwise the behaviour is undefined. > > Why? I don't see any recommendation about it in the docs: > https://docs.python.org/3/library/io.html#io.IOBase.seek What does adding 1 to a pos mean? If it?s binary it mean 1 byte further down the file but in text mode it may need to move the point 1, 2 or 3 bytes down the file. > >>> text = f.read() >> >> You have on limit on the amount of data read. > > I explained that previously. Anyway, chunk_size is small, so it's not > a great problem. Typo I meant you have no limit. You read all the data till the end of the file that might be mega bytes of data. > >>> lf_after = False >>> >>> for i, char in enumerate(reversed(text)): >> >> Simple use text.rindex('\n') or text.rfind('\n') for speed. > > I can't use them when I have to find both \n or \r. So I preferred to > simplify the code and use the for cycle every time. Take into mind > anyway that this is a prototype for a Python C Api implementation > (builtin I hope, or a C extension if not) > >>> Shortly, the file is always opened in text mode. File is read at the end in >>> bigger and bigger chunks, until the file is finished or all the lines are >>> found. >> >> It will fail if the contents is not ASCII. > > Why? > >>> Why? Because in encodings that have more than 1 byte per character, reading >>> a chunk of n bytes, then reading the previous chunk, can eventually split >>> the character between the chunks in two distinct bytes. >> >> No it cannot. text mode only knows how to return code points. Now if you are in >> binary it could be split, but you are not in binary mode so it cannot. > >> From the docs: > > seek(offset, whence=SEEK_SET) > Change the stream position to the given byte offset. > >>> Do you think there are chances to get this function as a method of the file >>> object in CPython? The method for a file object opened in bytes mode is >>> simpler, since there's no encoding and newline is only \n in that case. >> >> State your requirements. Then see if your implementation meets them. > > The method should return the last n lines from a file object. > If the file object is in text mode, the newline parameter must be honored. > If the file object is in binary mode, a newline is always b"\n", to be > consistent with readline. > > I suppose the current implementation of tail satisfies the > requirements for text mode. The previous one satisfied binary mode. > > Anyway, apart from my implementation, I'm curious if you think a tail > method is worth it to be a method of the builtin file objects in > CPython. > From Marco.Sulla.Python at gmail.com Sun May 8 16:48:32 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Sun, 8 May 2022 22:48:32 +0200 Subject: tail In-Reply-To: References: Message-ID: On Sun, 8 May 2022 at 22:34, Barry wrote: > > > On 8 May 2022, at 20:48, Marco Sulla wrote: > > > > ?On Sun, 8 May 2022 at 20:31, Barry Scott wrote: > >> > >>>> On 8 May 2022, at 17:05, Marco Sulla wrote: > >>> > >>> def tail(filepath, n=10, newline=None, encoding=None, chunk_size=100): > >>> n_chunk_size = n * chunk_size > >> > >> Why use tiny chunks? You can read 4KiB as fast as 100 bytes as its typically the smaller size the file system will allocate. > >> I tend to read on multiple of MiB as its near instant. > > > > Well, I tested on a little file, a list of my preferred pizzas, so.... > > Try it on a very big file. I'm not saying it's a good idea, it's only the value that I needed for my tests. Anyway, it's not a problem with big files. The problem is with files with long lines. > >> In text mode you can only seek to a value return from f.tell() otherwise the behaviour is undefined. > > > > Why? I don't see any recommendation about it in the docs: > > https://docs.python.org/3/library/io.html#io.IOBase.seek > > What does adding 1 to a pos mean? > If it?s binary it mean 1 byte further down the file but in text mode it may need to > move the point 1, 2 or 3 bytes down the file. Emh. I re-quote seek(offset, whence=SEEK_SET) Change the stream position to the given byte offset. And so on. No mention of differences between text and binary mode. > >> You have on limit on the amount of data read. > > > > I explained that previously. Anyway, chunk_size is small, so it's not > > a great problem. > > Typo I meant you have no limit. > > You read all the data till the end of the file that might be mega bytes of data. Yes, I already explained why and how it could be optimized. I quote myself: Shortly, the file is always opened in text mode. File is read at the end in bigger and bigger chunks, until the file is finished or all the lines are found. Why? Because in encodings that have more than 1 byte per character, reading a chunk of n bytes, then reading the previous chunk, can eventually split the character between the chunks in two distinct bytes. I think one can read chunk by chunk and test the chunk junction problem. I suppose the code will be faster this way. Anyway, it seems that this trick is quite fast anyway and it's a lot simpler. From cs at cskk.id.au Mon May 9 01:14:16 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Mon, 9 May 2022 15:14:16 +1000 Subject: tail In-Reply-To: References: Message-ID: On 08May2022 22:48, Marco Sulla wrote: >On Sun, 8 May 2022 at 22:34, Barry wrote: >> >> In text mode you can only seek to a value return from f.tell() >> >> otherwise the behaviour is undefined. >> > >> > Why? I don't see any recommendation about it in the docs: >> > https://docs.python.org/3/library/io.html#io.IOBase.seek >> >> What does adding 1 to a pos mean? >> If it?s binary it mean 1 byte further down the file but in text mode it may need to >> move the point 1, 2 or 3 bytes down the file. > >Emh. I re-quote > >seek(offset, whence=SEEK_SET) >Change the stream position to the given byte offset. > >And so on. No mention of differences between text and binary mode. You're looking at IOBase, the _binary_ basis of low level common file I/O. Compare with: https://docs.python.org/3/library/io.html#io.TextIOBase.seek The positions are "opaque numbers", which means you should not ascribe any deeper meaning to them except that they represent a point in the file. It clearly says "offset must either be a number returned by TextIOBase.tell(), or zero. Any other offset value produces undefined behaviour." The point here is that text is a very different thing. Because you cannot seek to an absolute number of characters in an encoding with variable sized characters. _If_ you did a seek to an arbitrary number you can end up in the middle of some character. And there are encodings where you cannot inspect the data to find a character boundary in the byte stream. Reading text files backwards is not a well defined thing without additional criteria: - knowing the text file actually ended on a character boundary - knowing how to find a character boundary Cheers, Cameron Simpson From greg.ewing at canterbury.ac.nz Sun May 8 19:58:32 2022 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 9 May 2022 11:58:32 +1200 Subject: tail In-Reply-To: References: <3848780F-83B8-4B5F-BFAD-157390288C15@barrys-emacs.org> Message-ID: On 9/05/22 7:47 am, Marco Sulla wrote: >> It will fail if the contents is not ASCII. > > Why? For some encodings, if you seek to an arbitrary byte position and then read, it may *appear* to succeed but give you complete gibberish. Your method might work for a certain subset of encodings (those that are self-synchronising) but it won't work for arbitrary encodings. Given that limitation, I don't think it's reliable enough to include in the standard library. -- Greg From wlfraed at ix.netcom.com Sun May 8 21:56:46 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Sun, 08 May 2022 21:56:46 -0400 Subject: tail References: Message-ID: On Sun, 8 May 2022 22:48:32 +0200, Marco Sulla declaimed the following: > >Emh. I re-quote > >seek(offset, whence=SEEK_SET) >Change the stream position to the given byte offset. > >And so on. No mention of differences between text and binary mode. You ignore that, underneath, Python is just wrapping the C API... And the documentation for C explicitly specifies that other then SEEK_END with offset 0, and SEEK_SET with offset of 0, for a text file one can only rely upon SEEK_SET using an offset previously obtained with (C) ftell() / (Python) .tell() . https://docs.python.org/3/library/io.html """ class io.IOBase The abstract base class for all I/O classes. """ seek(offset, whence=SEEK_SET) Change the stream position to the given byte offset. offset is interpreted relative to the position indicated by whence. The default value for whence is SEEK_SET. Values for whence are: """ Applicable to BINARY MODE I/O: For UTF-8 and any other multibyte encoding, this means you could end up positioning into the middle of a "character" and subsequently read garbage. It is on you to handle synchronizing on a valid character position, and also to handle different line ending conventions. """ class io.TextIOBase Base class for text streams. This class provides a character and line based interface to stream I/O. It inherits IOBase. """ seek(offset, whence=SEEK_SET) Change the stream position to the given offset. Behaviour depends on the whence parameter. The default value for whence is SEEK_SET. SEEK_SET or 0: seek from the start of the stream (the default); offset must either be a number returned by TextIOBase.tell(), or zero. Any other offset value produces undefined behaviour. SEEK_CUR or 1: ?seek? to the current position; offset must be zero, which is a no-operation (all other values are unsupported). SEEK_END or 2: seek to the end of the stream; offset must be zero (all other values are unsupported). """ EMPHASIS: "offset must either be a number returned by TextIOBase.tell(), or zero." TEXT I/O, with a specified encoding, will return Unicode data points, and will handle converting line ending to the internal ( represents new-line) format. Since your code does not specify BINARY mode in the open statement, Python should be using TEXT mode. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From nospam at please.ty Mon May 9 02:47:50 2022 From: nospam at please.ty (jak) Date: Mon, 9 May 2022 08:47:50 +0200 Subject: usb tv stick and python Message-ID: Hello everybody, I usually use vlc to watch tv and I use the w_scan program on linux to create a file (.m3u) with the list of available channels. Unfortunately I can't find an alternative to w_scan for Windows and I was wondering if you could tell me some python library that allows me, easily, to interface with the device and get the channel list. Thanks in advance for your attention. From wlfraed at ix.netcom.com Mon May 9 10:28:22 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Mon, 09 May 2022 10:28:22 -0400 Subject: usb tv stick and python References: Message-ID: <0c8i7hp8mcq34jod47ui3hv060opa8b8ip@4ax.com> On Mon, 9 May 2022 08:47:50 +0200, jak declaimed the following: >Hello everybody, >I usually use vlc to watch tv and I use the w_scan program on linux to >create a file (.m3u) with the list of available channels. Unfortunately >I can't find an alternative to w_scan for Windows and I was wondering if >you could tell me some python library that allows me, easily, to >interface with the device and get the channel list. > UNTESTED... But if it works means no change to your procedures... Presuming you are using W10 or later... Activate the Windows Subsystem for Linux ([old style] Control Panel / Programs / Programs and Features... Turn Windows Features On or Off... Scroll down, it's the third from the bottom) Windows "Microsoft Store"; Search "Debian"; Download/Install (set a log-in account/password). (Unfortunately, it's not the most recent version -- still on Buster...) Open Debian instance console (to my knowledge, no graphical applications are supported); do normal stuff to bring apt up-to-date. -=-=- wulfraed at ElusiveUnicorn:~$ apt search w_scan Sorting... Done Full Text Search... Done w-scan/oldstable 20170107-2 amd64 Channel scanning tool for DVB and ATSC channels wulfraed at ElusiveUnicorn:~$ -=-=- Install and test. Windows partitions are accessible as /mnt/ -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From nospam at please.ty Mon May 9 11:56:32 2022 From: nospam at please.ty (jak) Date: Mon, 9 May 2022 17:56:32 +0200 Subject: usb tv stick and python References: <0c8i7hp8mcq34jod47ui3hv060opa8b8ip@4ax.com> Message-ID: Il 09/05/2022 16:28, Dennis Lee Bieber ha scritto: > On Mon, 9 May 2022 08:47:50 +0200, jak declaimed the > following: > >> Hello everybody, >> I usually use vlc to watch tv and I use the w_scan program on linux to >> create a file (.m3u) with the list of available channels. Unfortunately >> I can't find an alternative to w_scan for Windows and I was wondering if >> you could tell me some python library that allows me, easily, to >> interface with the device and get the channel list. >> > > UNTESTED... But if it works means no change to your procedures... > > Presuming you are using W10 or later... > > Activate the Windows Subsystem for Linux ([old style] Control Panel / > Programs / Programs and Features... Turn Windows Features On or Off... > Scroll down, it's the third from the bottom) > > Windows "Microsoft Store"; Search "Debian"; Download/Install (set a > log-in account/password). (Unfortunately, it's not the most recent version > -- still on Buster...) > > Open Debian instance console (to my knowledge, no graphical > applications are supported); do normal stuff to bring apt up-to-date. > > -=-=- > wulfraed at ElusiveUnicorn:~$ apt search w_scan > Sorting... Done > Full Text Search... Done > w-scan/oldstable 20170107-2 amd64 > Channel scanning tool for DVB and ATSC channels > > wulfraed at ElusiveUnicorn:~$ > -=-=- > > Install and test. Windows partitions are accessible as > /mnt/ > > > First of all, thank you for your reply. Actually I already have a handy work around to use w_scan because I have a VM with linux (ubuntu) installed. I was just looking for a python package/library that would allow me to write a wrapper around. I would also be satisfied with finding documentation that describes the protocol to communicate with the dvb interface and be able to write an app that produces the list of available channels. Any advice, suggestion or pointing is welcome. From Marco.Sulla.Python at gmail.com Mon May 9 13:45:43 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Mon, 9 May 2022 19:45:43 +0200 Subject: tail In-Reply-To: References: Message-ID: On Mon, 9 May 2022 at 07:56, Cameron Simpson wrote: > > The point here is that text is a very different thing. Because you > cannot seek to an absolute number of characters in an encoding with > variable sized characters. _If_ you did a seek to an arbitrary number > you can end up in the middle of some character. And there are encodings > where you cannot inspect the data to find a character boundary in the > byte stream. Ooook, now I understand what you and Barry mean. I suppose there's no reliable way to tail a big file opened in text mode with a decent performance. Anyway, the previous-previous function I posted worked only for files opened in binary mode, and I suppose it's reliable, since it searches only for b"\n", as readline() in binary mode do. From rosuav at gmail.com Mon May 9 13:51:39 2022 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 10 May 2022 03:51:39 +1000 Subject: tail In-Reply-To: References: Message-ID: On Tue, 10 May 2022 at 03:47, Marco Sulla wrote: > > On Mon, 9 May 2022 at 07:56, Cameron Simpson wrote: > > > > The point here is that text is a very different thing. Because you > > cannot seek to an absolute number of characters in an encoding with > > variable sized characters. _If_ you did a seek to an arbitrary number > > you can end up in the middle of some character. And there are encodings > > where you cannot inspect the data to find a character boundary in the > > byte stream. > > Ooook, now I understand what you and Barry mean. I suppose there's no > reliable way to tail a big file opened in text mode with a decent performance. > > Anyway, the previous-previous function I posted worked only for files > opened in binary mode, and I suppose it's reliable, since it searches > only for b"\n", as readline() in binary mode do. It's still fundamentally impossible to solve this in a general way, so the best way to do things will always be to code for *your* specific use-case. That means that this doesn't belong in the stdlib or core language, but in your own toolkit. ChrisA From 2QdxY4RzWzUUiLuE at potatochowder.com Mon May 9 13:52:57 2022 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Mon, 9 May 2022 12:52:57 -0500 Subject: tail In-Reply-To: References: <3848780F-83B8-4B5F-BFAD-157390288C15@barrys-emacs.org> Message-ID: On 2022-05-08 at 18:52:42 +0000, Stefan Ram wrote: > Remember how recently people here talked about how you cannot copy > text from a video? Then, how did I do it? Turns out, for my > operating system, there's a screen OCR program! So I did this OCR > and then manually corrected a few wrong characters, and was done! When you're learning, and the example you tried doesn't work like it worked on the video, you probably don't know what's wrong, let alone how to correct it. From wlfraed at ix.netcom.com Mon May 9 14:25:22 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Mon, 09 May 2022 14:25:22 -0400 Subject: usb tv stick and python References: <0c8i7hp8mcq34jod47ui3hv060opa8b8ip@4ax.com> Message-ID: On Mon, 9 May 2022 17:56:32 +0200, jak declaimed the following: >First of all, thank you for your reply. Actually I already have a handy >work around to use w_scan because I have a VM with linux (ubuntu) >installed. I was just looking for a python package/library that would >allow me to write a wrapper around. I would also be satisfied with >finding documentation that describes the protocol to communicate with >the dvb interface and be able to write an app that produces the list of >available channels. Any advice, suggestion or pointing is welcome. For the protocol... You might need to locate the source code for w_scan. Perhaps https://github.com/tbsdtv/w_scan -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From Marco.Sulla.Python at gmail.com Mon May 9 15:11:23 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Mon, 9 May 2022 21:11:23 +0200 Subject: tail In-Reply-To: References: Message-ID: On Mon, 9 May 2022 at 19:53, Chris Angelico wrote: > > On Tue, 10 May 2022 at 03:47, Marco Sulla wrote: > > > > On Mon, 9 May 2022 at 07:56, Cameron Simpson wrote: > > > > > > The point here is that text is a very different thing. Because you > > > cannot seek to an absolute number of characters in an encoding with > > > variable sized characters. _If_ you did a seek to an arbitrary number > > > you can end up in the middle of some character. And there are encodings > > > where you cannot inspect the data to find a character boundary in the > > > byte stream. > > > > Ooook, now I understand what you and Barry mean. I suppose there's no > > reliable way to tail a big file opened in text mode with a decent performance. > > > > Anyway, the previous-previous function I posted worked only for files > > opened in binary mode, and I suppose it's reliable, since it searches > > only for b"\n", as readline() in binary mode do. > > It's still fundamentally impossible to solve this in a general way, so > the best way to do things will always be to code for *your* specific > use-case. That means that this doesn't belong in the stdlib or core > language, but in your own toolkit. Nevertheless, tail is a fundamental tool in *nix. It's fast and reliable. Also the tail command can't handle different encodings? From rosuav at gmail.com Mon May 9 16:47:06 2022 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 10 May 2022 06:47:06 +1000 Subject: tail In-Reply-To: References: Message-ID: On Tue, 10 May 2022 at 05:12, Marco Sulla wrote: > > On Mon, 9 May 2022 at 19:53, Chris Angelico wrote: > > > > On Tue, 10 May 2022 at 03:47, Marco Sulla wrote: > > > > > > On Mon, 9 May 2022 at 07:56, Cameron Simpson wrote: > > > > > > > > The point here is that text is a very different thing. Because you > > > > cannot seek to an absolute number of characters in an encoding with > > > > variable sized characters. _If_ you did a seek to an arbitrary number > > > > you can end up in the middle of some character. And there are encodings > > > > where you cannot inspect the data to find a character boundary in the > > > > byte stream. > > > > > > Ooook, now I understand what you and Barry mean. I suppose there's no > > > reliable way to tail a big file opened in text mode with a decent performance. > > > > > > Anyway, the previous-previous function I posted worked only for files > > > opened in binary mode, and I suppose it's reliable, since it searches > > > only for b"\n", as readline() in binary mode do. > > > > It's still fundamentally impossible to solve this in a general way, so > > the best way to do things will always be to code for *your* specific > > use-case. That means that this doesn't belong in the stdlib or core > > language, but in your own toolkit. > > Nevertheless, tail is a fundamental tool in *nix. It's fast and > reliable. Also the tail command can't handle different encodings? Like most Unix programs, it handles bytes. ChrisA From barry at barrys-emacs.org Mon May 9 16:58:45 2022 From: barry at barrys-emacs.org (Barry) Date: Mon, 9 May 2022 21:58:45 +0100 Subject: tail In-Reply-To: References: Message-ID: > On 9 May 2022, at 17:41, ram at zedat.fu-berlin.de wrote: > > ?Barry Scott writes: >> Why use tiny chunks? You can read 4KiB as fast as 100 bytes > > When optimizing code, it helps to be aware of the orders of > magnitude That is true and we?ll know to me, now show how what I said is wrong. The os is going to DMA at least 4k, with read ahead more like 64k. So I can get that into the python memory at the same scale of time as 1 byte because it?s the setup of the I/O that is expensive not the bytes transferred. Barry > . Code that is more cache-friendly is faster, that is, > code that holds data in single region of memory and that uses > regular patterns of access. Chandler Carruth talked about this, > and I made some notes when watching the video of his talk: > > CPUS HAVE A HIERARCHICAL CACHE SYSTEM > (from a 2014 talk by Chandler Carruth) > > One cycle on a 3 GHz processor 1 ns > L1 cache reference 0.5 ns > Branch mispredict 5 ns > L2 cache reference 7 ns 14x L1 cache > Mutex lock/unlock 25 ns > Main memory reference 100 ns 20xL2, 200xL1 > Compress 1K bytes with Snappy 3,000 ns > Send 1K bytes over 1 Gbps network 10,000 ns 0.01 ms > Read 4K randomly from SSD 150,000 ns 0.15 ms > Read 1 MB sequentially from memory 250,000 ns 0.25 ms > Round trip within same datacenter 500,000 ns 0.5 ms > Read 1 MB sequentially From SSD 1,000,000 ns 1 ms 4x memory > Disk seek 10,000,000 ns 10 ms 20xdatacen. RT > Read 1 MB sequentially from disk 20,000,000 ns 20 ms 80xmem.,20xSSD > Send packet CA->Netherlands->CA 150,000,000 ns 150 ms > > . Remember how recently people here talked about how you cannot > copy text from a video? Then, how did I do it? Turns out, for my > operating system, there's a screen OCR program! So I did this OCR > and then manually corrected a few wrong characters, and was done! > > > -- > https://mail.python.org/mailman/listinfo/python-list > From barry at barrys-emacs.org Mon May 9 17:07:22 2022 From: barry at barrys-emacs.org (Barry) Date: Mon, 9 May 2022 22:07:22 +0100 Subject: tail In-Reply-To: References: Message-ID: > On 9 May 2022, at 20:14, Marco Sulla wrote: > > ?On Mon, 9 May 2022 at 19:53, Chris Angelico wrote: >> >>> On Tue, 10 May 2022 at 03:47, Marco Sulla wrote: >>> >>> On Mon, 9 May 2022 at 07:56, Cameron Simpson wrote: >>>> >>>> The point here is that text is a very different thing. Because you >>>> cannot seek to an absolute number of characters in an encoding with >>>> variable sized characters. _If_ you did a seek to an arbitrary number >>>> you can end up in the middle of some character. And there are encodings >>>> where you cannot inspect the data to find a character boundary in the >>>> byte stream. >>> >>> Ooook, now I understand what you and Barry mean. I suppose there's no >>> reliable way to tail a big file opened in text mode with a decent performance. >>> >>> Anyway, the previous-previous function I posted worked only for files >>> opened in binary mode, and I suppose it's reliable, since it searches >>> only for b"\n", as readline() in binary mode do. >> >> It's still fundamentally impossible to solve this in a general way, so >> the best way to do things will always be to code for *your* specific >> use-case. That means that this doesn't belong in the stdlib or core >> language, but in your own toolkit. > > Nevertheless, tail is a fundamental tool in *nix. It's fast and > reliable. Also the tail command can't handle different encodings? POSIX tail just prints the bytes to the output that it finds between \n bytes. At no time does it need to care about encodings as that is a problem solved by the terminal software. I would not expect utf-16 to work with tail on linux systems. You could always get the source of tail and read It?s implementation. Barry > -- > https://mail.python.org/mailman/listinfo/python-list > From rosuav at gmail.com Mon May 9 17:12:53 2022 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 10 May 2022 07:12:53 +1000 Subject: tail In-Reply-To: References: Message-ID: On Tue, 10 May 2022 at 07:07, Barry wrote: > POSIX tail just prints the bytes to the output that it finds between \n bytes. > At no time does it need to care about encodings as that is a problem solved > by the terminal software. I would not expect utf-16 to work with tail on > linux systems. UTF-16 ASCII seems to work fine on my system, which probably means the terminal is just ignoring all the NUL bytes. But if there's a random 0x0A anywhere, it would probably be counted as a line break. ChrisA From wlfraed at ix.netcom.com Mon May 9 17:05:46 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Mon, 09 May 2022 17:05:46 -0400 Subject: tail References: Message-ID: On Mon, 9 May 2022 21:11:23 +0200, Marco Sulla declaimed the following: >Nevertheless, tail is a fundamental tool in *nix. It's fast and >reliable. Also the tail command can't handle different encodings? Based upon https://github.com/coreutils/coreutils/blob/master/src/tail.c the ONLY thing tail looks at is single byte "\n". It does not handle other line endings, and appears to performs BINARY I/O, not text I/O. It does nothing for bytes that are not "\n". Split multi-byte encodings are irrelevant since, if it does not find enough "\n" bytes in the buffer (chunk) it reads another binary chunk and seeks for additional "\n" bytes. Once it finds the desired amount, it is synchronized on the byte following the "\n" (which, for multi-byte encodings might be a NUL, but in any event, should be a safe location for subsequent I/O). Interpretation of encoding appears to fall to the console driver configuration when displaying the bytes output by tail. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From alan at csail.mit.edu Mon May 9 17:31:37 2022 From: alan at csail.mit.edu (Alan Bawden) Date: Mon, 09 May 2022 17:31:37 -0400 Subject: tail References: Message-ID: <86y1zatmti.fsf@williamsburg.bawden.org> Marco Sulla writes: On Mon, 9 May 2022 at 19:53, Chris Angelico wrote: ... Nevertheless, tail is a fundamental tool in *nix. It's fast and reliable. Also the tail command can't handle different encodings? It definitely can't. It works for UTF-8, and all the ASCII compatible single byte encodings, but feed it a file encoded in UTF-16, and it will sometimes screw up. (And if you don't redirect the output away from your terminal, and your terminal encoding isn't also set to UTF-16, you will likely find yourself looking at gibberish -- but that's another problem...) From rosuav at gmail.com Tue May 10 06:14:11 2022 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 10 May 2022 20:14:11 +1000 Subject: [Python-ideas] Re: New Tool Proposal In-Reply-To: <6a16e4f8-b1d7-292e-60d2-1db611456c2f@btinternet.com> References: <6a16e4f8-b1d7-292e-60d2-1db611456c2f@btinternet.com> Message-ID: On Tue, 10 May 2022 at 19:57, anthony.flury wrote: > > > On 10/05/2022 09:20, Chris Angelico wrote: > > On Tue, 10 May 2022 at 18:06, anthony.flury via Python-ideas > wrote: > > A proposal for a new tool to be implemented - > > It is often the case that developer write Code in Python and then convert to a C extension module for performance regions. > > A C extension module has a lot of boiler plate code - for instance the Structures required for each class, the functions for Module initialization etc. > > My Idea is a simple tool that uses introspection tools to take a Python module and to generate the relevant boiler plate for the module - including blank functions for the module classes and for methods. This tool would use type annotations (if given) to make sensible choices for parameter and attribute types, including using int and float directly rather than Internal objects (depending on tool options). > > Yep, that's an awesome idea! Are you aware of Cython? You might be > able to make use of that. > > Chris, Thank you. > > I am aware of Cython but that isn't quite what I had in mind. I want a tool for a developer who doesn't want to continue to support the Python 'prototype' for whatever reason, ie where they want a complete conversion to C. > > It might even be possible with inspection of the AST to write some of the code inside the C functions - but that is not for release 0.1 :-) > You may still be able to take advantage of Cython as part of the process. One thing that's really cool about source code is that, fundamentally, it's all text... and Python is *great* at manipulating text files :) It might be that you can write a script that transforms a Python module into a Cython module, which can then be compiled as-is, or further processed as needed. BTW, not sure which list you're intending to discuss this on, so I'm just replying on the same list you sent this message to. ChrisA From drsalists at gmail.com Tue May 10 10:04:51 2022 From: drsalists at gmail.com (Dan Stromberg) Date: Tue, 10 May 2022 07:04:51 -0700 Subject: [Python-ideas] Re: New Tool Proposal In-Reply-To: References: <6a16e4f8-b1d7-292e-60d2-1db611456c2f@btinternet.com> Message-ID: On Tue, May 10, 2022 at 3:15 AM Chris Angelico wrote: > > It is often the case that developer write Code in Python and then > convert to a C extension module for performance regions. > > > > A C extension module has a lot of boiler plate code - for instance the > Structures required for each class, the functions for Module initialization > etc. > > > > My Idea is a simple tool that uses introspection tools to take a Python > module and to generate the relevant boiler plate for the module - including > blank functions for the module classes and for methods. This tool would use > type annotations (if given) to make sensible choices for parameter and > attribute types, including using int and float directly rather than > Internal objects (depending on tool options). > Two things to say about this: 1) Sometimes abandoning a pure python module for a C extension for performance is a mistake - because Pypy is probably going to be much faster with the pure python module 2) I've had some luck using m4 to maintain a single source file that is used to automatically generate both pure python and cython. This is a little like using cpp in a C project. For examples of #2, perhaps see: https://stromberg.dnsalias.org/~strombrg/treap/ https://stromberg.dnsalias.org/svn/rolling_checksum_mod/trunk/ https://stromberg.dnsalias.org/~strombrg/sort-comparison/ It's often nice to keep the lines of the pure-python and cython having a 1-1 relationship, so that tracebacks report useful line numbers either way. However, in the treap example I've dispensed with that because some methods were almost identical but had some boilerplate - and m4 was able to handle that nicely at the cost of lines being 1-1. HTH From scooky2000 at gmail.com Tue May 10 12:45:20 2022 From: scooky2000 at gmail.com (Del Mervine) Date: Tue, 10 May 2022 11:45:20 -0500 Subject: [Python-ideas] Re: New Tool Proposal In-Reply-To: References: <6a16e4f8-b1d7-292e-60d2-1db611456c2f@btinternet.com> Message-ID: <0d4d27a9-154f-5fbf-47ac-e9529ea58d72@gmail.com> On 5/10/22 5:14 AM, Chris Angelico wrote: > On Tue, 10 May 2022 at 19:57, anthony.flury > wrote: >> >> On 10/05/2022 09:20, Chris Angelico wrote: >> >> On Tue, 10 May 2022 at 18:06, anthony.flury via Python-ideas >> wrote: >> >> A proposal for a new tool to be implemented - >> >> It is often the case that developer write Code in Python and then convert to a C extension module for performance regions. >> >> A C extension module has a lot of boiler plate code - for instance the Structures required for each class, the functions for Module initialization etc. >> >> My Idea is a simple tool that uses introspection tools to take a Python module and to generate the relevant boiler plate for the module - including blank functions for the module classes and for methods. This tool would use type annotations (if given) to make sensible choices for parameter and attribute types, including using int and float directly rather than Internal objects (depending on tool options). >> >> Yep, that's an awesome idea! Are you aware of Cython? You might be >> able to make use of that. >> >> Chris, Thank you. >> >> I am aware of Cython but that isn't quite what I had in mind. I want a tool for a developer who doesn't want to continue to support the Python 'prototype' for whatever reason, ie where they want a complete conversion to C. >> >> It might even be possible with inspection of the AST to write some of the code inside the C functions - but that is not for release 0.1 :-) >> > You may still be able to take advantage of Cython as part of the > process. One thing that's really cool about source code is that, > fundamentally, it's all text... and Python is *great* at manipulating > text files :) It might be that you can write a script that transforms > a Python module into a Cython module, which can then be compiled > as-is, or further processed as needed. > > BTW, not sure which list you're intending to discuss this on, so I'm > just replying on the same list you sent this message to. > > ChrisA This might be what you are looking for? From michael.stemper at gmail.com Wed May 11 09:33:27 2022 From: michael.stemper at gmail.com (Michael F. Stemper) Date: Wed, 11 May 2022 08:33:27 -0500 Subject: Changing calling sequence Message-ID: I have a function that I use to retrieve daily data from a home-brew database. Its calling sequence is; def TempsOneDay( year, month, date ): After using it (and its friends) for a few years, I've come to realize that there are times where it would be advantageous to invoke it with a datetime.date as its single argument. As far as I can tell, there are three ways for me to proceed: 1. Write a similar function that takes a single datetime.date as its argument. 2. Rewrite the existing function so that it takes a single argument, which can be either a tuple of (year,month,date) or a datetime.date argument. 3. Rewrite the existing function so that its first argument can be either an int (for year) or a datetime.date. The existing month and date arguments would be optional, with default=None. But, if the first argument is an int, and either of month or date is None, an error would be raised. The first would be the simplest. However, it is obviously WET rather than DRY. The second isn't too bad, but a change like this would require that I find all places that the function is currently used and insert a pair of parentheses. Touching this much code is risky, as well as being a bunch of work. (Admittedly, I'd only do it once.) The third is really klunky, but wouldn't need to touch anything besides this function. What are others' thoughts? Which of the approaches above looks least undesirable (and why)? Can anybody see a fourth approach? -- Michael F. Stemper This post contains greater than 95% post-consumer bytes by weight. From toby at tobiah.org Wed May 11 11:36:26 2022 From: toby at tobiah.org (Tobiah) Date: Wed, 11 May 2022 08:36:26 -0700 Subject: Changing calling sequence References: Message-ID: On 5/11/22 06:33, Michael F. Stemper wrote: > I have a function that I use to retrieve daily data from a > home-brew database. Its calling sequence is; > > def TempsOneDay( year, month, date ): > > After using it (and its friends) for a few years, I've come to > realize that there are times where it would be advantageous to > invoke it with a datetime.date as its single argument. You could just use all keyword args: def TempsOneDay(**kwargs): if 'date' in kwargs: handle_datetime(kwargs['date']) elif 'year' in kwargs and 'month' in kwargs and 'day' in kwargs: handle_args(kwargs['year'], kwargs['month'], kwargs['day']) else: raise Exception("Bad keyword args") TempsOneDay(date=datetime.datetime.now) TempsOneDay(year=2022, month=11, day=30) From 2QdxY4RzWzUUiLuE at potatochowder.com Wed May 11 13:47:18 2022 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Wed, 11 May 2022 12:47:18 -0500 Subject: Changing calling sequence In-Reply-To: References: Message-ID: On 2022-05-11 at 08:33:27 -0500, "Michael F. Stemper" wrote: > I have a function that I use to retrieve daily data from a > home-brew database. Its calling sequence is; > > def TempsOneDay( year, month, date ): > > After using it (and its friends) for a few years, I've come to > realize that there are times where it would be advantageous to > invoke it with a datetime.date as its single argument. > > As far as I can tell, there are three ways for me to proceed: > 1. Write a similar function that takes a single datetime.date > as its argument. > 2. Rewrite the existing function so that it takes a single > argument, which can be either a tuple of (year,month,date) > or a datetime.date argument. > 3. Rewrite the existing function so that its first argument > can be either an int (for year) or a datetime.date. The > existing month and date arguments would be optional, with > default=None. But, if the first argument is an int, and > either of month or date is None, an error would be raised. > > The first would be the simplest. However, it is obviously WET > rather than DRY. It's also the least disruptive to existing code and tests, and the most clear to readers (whether or not they're familiar with said existing code). What pieces, exactly, do you think you would repeat, especially after you extract the common logic into a new function that should be simpler than either API-level function. From David.Raymond at tomtom.com Wed May 11 15:01:16 2022 From: David.Raymond at tomtom.com (David Raymond) Date: Wed, 11 May 2022 19:01:16 +0000 Subject: Changing calling sequence In-Reply-To: References: Message-ID: >> I have a function that I use to retrieve daily data from a >> home-brew database. Its calling sequence is; >> >> def TempsOneDay( year, month, date ): >> >> After using it (and its friends) for a few years, I've come to >> realize that there are times where it would be advantageous to >> invoke it with a datetime.date as its single argument. > >You could just use all keyword args: > >def TempsOneDay(**kwargs): > > if 'date' in kwargs: > handle_datetime(kwargs['date']) > elif 'year' in kwargs and 'month' in kwargs and 'day' in kwargs: > handle_args(kwargs['year'], kwargs['month'], kwargs['day']) > else: > raise Exception("Bad keyword args") > >TempsOneDay(date=datetime.datetime.now) > >TempsOneDay(year=2022, month=11, day=30) > Maybe not the prettiest, but you could also define it like this, which also wouldn't require changing of any existing calls or the main body of the function past this if block. def TempsOneDay(*dateComponents): if len(dateComponents) == 3: year, month, date = dateComponents elif len(dateComponents) == 1 and isinstance(dateComponents[0], datetime.date): year, month, date = (dateComponents[0].year, dateComponents[0].month, dateComponents[0].day) else: raise Exception("Error message here") From grant.b.edwards at gmail.com Wed May 11 15:25:53 2022 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Wed, 11 May 2022 12:25:53 -0700 (PDT) Subject: Changing calling sequence References: =?utf-8?q?=3CHE1PR0701MB2857BFBEFAF93C3C3ECB2F0687C89=40HE1PR0701MB2857=2Ee?= =?utf-8?q?urprd07=2Eprod=2Eoutlook=2Ecom=3E?= Message-ID: <627c0dc1.1c69fb81.a138d.2e6e@mx.google.com> On 2022-05-11, David Raymond wrote: > Maybe not the prettiest, but you could also define it like this, > which also wouldn't require changing of any existing calls or the > main body of the function past this if block. > > def TempsOneDay(*dateComponents): > if len(dateComponents) == 3: > year, month, date = dateComponents > elif len(dateComponents) == 1 and isinstance(dateComponents[0], datetime.date): > year, month, date = (dateComponents[0].year, dateComponents[0].month, dateComponents[0].day) > else: > raise Exception("Error message here") That would be my preference were I reading the code. It makes it quite clear that there are two completely separate signatures. I think I would be a little confused by the 2nd and 3rd values with default values ? the implication would be that I can supply a datetime object as the first argument and then additional month and date values in the 2nd and 3rd args. You could try to explain it with a comment, but I tend to ignore comments... From anthony.flury at btinternet.com Wed May 11 15:58:21 2022 From: anthony.flury at btinternet.com (anthony.flury) Date: Wed, 11 May 2022 20:58:21 +0100 (BST) Subject: Changing calling sequence In-Reply-To: References: Message-ID: <45cd23fe.acec.180b4b37653.Webtop.104@btinternet.com> Why not do : def TempsOneDayDT(date:datetime.date): return TempsOneDay(date.year, date.month, date.day) No repeat of code - just a different interface to the same functionality. ------ Original Message ------ From: "Michael F. Stemper" To: python-list at python.org Sent: Wednesday, 11 May, 22 At 14:33 Subject: Changing calling sequence I have a function that I use to retrieve daily data from a home-brew database. Its calling sequence is; def TempsOneDay( year, month, date ): After using it (and its friends) for a few years, I've come to realize that there are times where it would be advantageous to invoke it with a datetime.date as its single argument. As far as I can tell, there are three ways for me to proceed: 1. Write a similar function that takes a single datetime.date as its argument. 2. Rewrite the existing function so that it takes a single argument, which can be either a tuple of (year,month,date) or a datetime.date argument. 3. Rewrite the existing function so that its first argument can be either an int (for year) or a datetime.date. The existing month and date arguments would be optional, with default=None. But, if the first argument is an int, and either of month or date is None, an error would be raised. The first would be the simplest. However, it is obviously WET rather than DRY. The second isn't too bad, but a change like this would require that I find all places that the function is currently used and insert a pair of parentheses. Touching this much code is risky, as well as being a bunch of work. (Admittedly, I'd only do it once.) The third is really klunky, but wouldn't need to touch anything besides this function. What are others' thoughts? Which of the approaches above looks least undesirable (and why)? Can anybody see a fourth approach? -- Michael F. Stemper This post contains greater than 95% post-consumer bytes by weight. -- https://mail.python.org/mailman/listinfo/python-list --
Anthony Flury
anthony.flury at btinternet.com From Marco.Sulla.Python at gmail.com Wed May 11 15:58:13 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Wed, 11 May 2022 21:58:13 +0200 Subject: tail In-Reply-To: References: Message-ID: On Mon, 9 May 2022 at 23:15, Dennis Lee Bieber wrote: > > On Mon, 9 May 2022 21:11:23 +0200, Marco Sulla > declaimed the following: > > >Nevertheless, tail is a fundamental tool in *nix. It's fast and > >reliable. Also the tail command can't handle different encodings? > > Based upon > https://github.com/coreutils/coreutils/blob/master/src/tail.c the ONLY > thing tail looks at is single byte "\n". It does not handle other line > endings, and appears to performs BINARY I/O, not text I/O. It does nothing > for bytes that are not "\n". Split multi-byte encodings are irrelevant > since, if it does not find enough "\n" bytes in the buffer (chunk) it reads > another binary chunk and seeks for additional "\n" bytes. Once it finds the > desired amount, it is synchronized on the byte following the "\n" (which, > for multi-byte encodings might be a NUL, but in any event, should be a safe > location for subsequent I/O). > > Interpretation of encoding appears to fall to the console driver > configuration when displaying the bytes output by tail. Ok, I understand. This should be a Python implementation of *nix tail: import os _lf = b"\n" _err_n = "Parameter n must be a positive integer number" _err_chunk_size = "Parameter chunk_size must be a positive integer number" def tail(filepath, n=10, chunk_size=100): if (n <= 0): raise ValueError(_err_n) if (n % 1 != 0): raise ValueError(_err_n) if (chunk_size <= 0): raise ValueError(_err_chunk_size) if (chunk_size % 1 != 0): raise ValueError(_err_chunk_size) n_chunk_size = n * chunk_size pos = os.stat(filepath).st_size chunk_line_pos = -1 lines_not_found = n with open(filepath, "rb") as f: text = bytearray() while pos != 0: pos -= n_chunk_size if pos < 0: pos = 0 f.seek(pos) chars = f.read(n_chunk_size) text[0:0] = chars search_pos = n_chunk_size while search_pos != -1: chunk_line_pos = chars.rfind(_lf, 0, search_pos) if chunk_line_pos != -1: lines_not_found -= 1 if lines_not_found == 0: break search_pos = chunk_line_pos if lines_not_found == 0: break return bytes(text[chunk_line_pos+1:]) The function opens the file in binary mode and searches only for b"\n". It returns the last n lines of the file as bytes. I suppose this function is fast. It reads the bytes from the file in chunks and stores them in a bytearray, prepending them to it. The final result is read from the bytearray and converted to bytes (to be consistent with the read method). I suppose the function is reliable. File is opened in binary mode and only b"\n" is searched as line end, as *nix tail (and python readline in binary mode) do. And bytes are returned. The caller can use them as is or convert them to a string using the encoding it wants, or do whatever its imagination can think :) Finally, it seems to me the function is quite simple. If all my affirmations are true, the three obstacles written by Chris should be passed. I'd very much like to see a CPython implementation of that function. It could be a method of a file object opened in binary mode, and *only* in binary mode. What do you think about it? From rosuav at gmail.com Wed May 11 16:07:18 2022 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 12 May 2022 06:07:18 +1000 Subject: tail In-Reply-To: References: Message-ID: On Thu, 12 May 2022 at 06:03, Marco Sulla wrote: > I suppose this function is fast. It reads the bytes from the file in chunks > and stores them in a bytearray, prepending them to it. The final result is > read from the bytearray and converted to bytes (to be consistent with the > read method). > > I suppose the function is reliable. File is opened in binary mode and only > b"\n" is searched as line end, as *nix tail (and python readline in binary > mode) do. And bytes are returned. The caller can use them as is or convert > them to a string using the encoding it wants, or do whatever its > imagination can think :) > > Finally, it seems to me the function is quite simple. > > If all my affirmations are true, the three obstacles written by Chris > should be passed. Have you actually checked those three, or do you merely suppose them to be true? > I'd very much like to see a CPython implementation of that function. It > could be a method of a file object opened in binary mode, and *only* in > binary mode. > > What do you think about it? Still not necessary. You can simply have it in your own toolkit. Why should it be part of the core language? How much benefit would it be to anyone else? All the same assumptions are still there, so it still isn't general, and you may as well just *code to your own needs* like I've been saying all along. This does not need to be in the standard library. Do what you need, assume what you can safely assume, and other people can write different code. I don't understand why this wants to be in the standard library. ChrisA From Marco.Sulla.Python at gmail.com Wed May 11 17:27:03 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Wed, 11 May 2022 23:27:03 +0200 Subject: tail In-Reply-To: References: Message-ID: On Wed, 11 May 2022 at 22:09, Chris Angelico wrote: > > Have you actually checked those three, or do you merely suppose them to be true? I only suppose, as I said. I should do some benchmark and some other tests, and, frankly, I don't want to. I don't want to because I'm quite sure the implementation is fast, since it reads by chunks and cache them. I'm not sure it's 100% free of bugs, but the concept is very simple, since it simply mimics the *nix tail, so it should be reliable. > > > I'd very much like to see a CPython implementation of that function. It > > could be a method of a file object opened in binary mode, and *only* in > > binary mode. > > > > What do you think about it? > > Still not necessary. You can simply have it in your own toolkit. Why > should it be part of the core language? Why not? > How much benefit would it be > to anyone else? I suppose that every programmer, at least one time in its life, did a tail. > All the same assumptions are still there, so it still > isn't general It's general. It mimics the *nix tail. I can't think of a more general way to implement a tail. > I don't understand why this wants to be in the standard library. Well, the answer is really simple: I needed it and if I found it in the stdlib, I used it instead of writing the first horrible function. Furthermore, tail is such a useful tool that I suppose many others are interested, based on this quick Google search: https://www.google.com/search?q=python+tail A question on Stackoverflow really much voted, many other Stackoverflow questions, a package that seems to exactly do the same thing, that is mimic *nix tail, and a blog post about how to tail in Python. Furthermore, if you search python tail pypi, you can find a bunch of other packages: https://www.google.com/search?q=python+tail+pypi It seems the subject is quite popular, and I can't imagine otherwise. From rosuav at gmail.com Wed May 11 17:31:34 2022 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 12 May 2022 07:31:34 +1000 Subject: tail In-Reply-To: References: Message-ID: On Thu, 12 May 2022 at 07:27, Marco Sulla wrote: > > On Wed, 11 May 2022 at 22:09, Chris Angelico wrote: > > > > Have you actually checked those three, or do you merely suppose them to be true? > > I only suppose, as I said. I should do some benchmark and some other > tests, and, frankly, I don't want to. I don't want to because I'm > quite sure the implementation is fast, since it reads by chunks and > cache them. I'm not sure it's 100% free of bugs, but the concept is > very simple, since it simply mimics the *nix tail, so it should be > reliable. If you don't care enough to benchmark it or even debug it, why should anyone else care? I'm done discussing. You think that someone else should have done this for you, but you aren't even willing to put in the effort to make this useful to anyone else. Just use it yourself and have done with it. ChrisA From wlfraed at ix.netcom.com Wed May 11 18:15:10 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Wed, 11 May 2022 18:15:10 -0400 Subject: tail References: Message-ID: On Thu, 12 May 2022 06:07:18 +1000, Chris Angelico declaimed the following: >I don't understand why this wants to be in the standard library. > Especially as any Linux distribution probably includes the compiled "tail" command, so this would only be of use on Windows. Under recent Windows, one has an equivalent to "tail" IFF using PowerShell rather than the "DOS" shell. https://www.middlewareinventory.com/blog/powershell-tail-file-windows-tail-command/ or install a Windows binary equivalent http://tailforwin32.sourceforge.net/ -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From avigross at verizon.net Wed May 11 19:47:56 2022 From: avigross at verizon.net (Avi Gross) Date: Wed, 11 May 2022 23:47:56 +0000 (UTC) Subject: tail In-Reply-To: References: Message-ID: <1960660253.1294402.1652312876126@mail.yahoo.com> Just FYI, UNIX had a bunch of utilities that could emulate a vanilla version of tail on a command line. You can use sed, awk and quite a few others to simply show line N to the end of a file or?other variations.? Of course the way many things were done back then had less focus on efficiency than how to stepwise?make changes in a pipeline so reading from the beginning to end was not an issue. -----Original Message----- From: Marco Sulla To: Chris Angelico Cc: python-list at python.org Sent: Wed, May 11, 2022 5:27 pm Subject: Re: tail On Wed, 11 May 2022 at 22:09, Chris Angelico wrote: > > Have you actually checked those three, or do you merely suppose them to be true? I only suppose, as I said. I should do some benchmark and some other tests, and, frankly, I don't want to. I don't want to because I'm quite sure the implementation is fast, since it reads by chunks and cache them. I'm not sure it's 100% free of bugs, but the concept is very simple, since it simply mimics the *nix tail, so it should be reliable. > > > I'd very much like to see a CPython implementation of that function. It > > could be a method of a file object opened in binary mode, and *only* in > > binary mode. > > > > What do you think about it? > > Still not necessary. You can simply have it in your own toolkit. Why > should it be part of the core language? Why not? > How much benefit would it be > to anyone else? I suppose that every programmer, at least one time in its life, did a tail. > All the same assumptions are still there, so it still > isn't general It's general. It mimics the *nix tail. I can't think of a more general way to implement a tail. > I don't understand why this wants to be in the standard library. Well, the answer is really simple: I needed it and if I found it in the stdlib, I used it instead of writing the first horrible function. Furthermore, tail is such a useful tool that I suppose many others are interested, based on this quick Google search: https://www.google.com/search?q=python+tail A question on Stackoverflow really much voted, many other Stackoverflow questions, a package that seems to exactly do the same thing, that is mimic *nix tail, and a blog post about how to tail in Python. Furthermore, if you search python tail pypi, you can find a bunch of other packages: https://www.google.com/search?q=python+tail+pypi It seems the subject is quite popular, and I can't imagine otherwise. -- https://mail.python.org/mailman/listinfo/python-list From avigross at verizon.net Wed May 11 21:27:20 2022 From: avigross at verizon.net (Avi Gross) Date: Thu, 12 May 2022 01:27:20 +0000 (UTC) Subject: tail In-Reply-To: References: Message-ID: <719424227.1299571.1652318840152@mail.yahoo.com> This seems to be a regular refrain where someone wants something as STANDARD in a?programming language or environment and others want to keep it lean and mean or do?not see THIS suggestion as particularly important or useful. Looking at the end of something is extremely common. Packages like numpy/pandas in?Python often provide functions with names like head or tail as do other languages where?data structures with names like data.frame are commonly used. These structures are in?some way indexed to make it easy to jump towards the end. Text files are not. Efficiency aside, a 3-year-old (well, certainly a 30 year old)? can cobble together a function?that takes a filename assumed to be textual and reads the file into some data structure that?stores the lines of the file and so it can be indexed by line number and also report the index of?the final line. The data structure can be a list of lists or a dictionary with line numbers as keys?or a numpy ... So the need for this functionality seems obvious but then what about someone who wants?a bunch of random lines from a file? Need we satisfy their wish to pick random offsets?from the file and get the line in which the offset is in middle of or the one about to start??Would that even be random if line lengths vary? Text files were never designed to be used?efficiently except for reading and writing and certainly not for something like sorting. Again, generally you can read in the darn file and perform the operation and free up whatever memory?you do??not need. If you have huge files, fine, but then why make a special function be part of the default?setup if it is rarely used? Why not put it in a module/package called BigFileBatches alongside?other functions useful to do things in batches? Call that when needed but for smaller files, KISS. -----Original Message----- From: Dennis Lee Bieber To: python-list at python.org Sent: Wed, May 11, 2022 6:15 pm Subject: Re: tail On Thu, 12 May 2022 06:07:18 +1000, Chris Angelico declaimed the following: >I don't understand why this wants to be in the standard library. > ??? Especially as any Linux distribution probably includes the compiled "tail" command, so this would only be of use on Windows. ??? Under recent Windows, one has an equivalent to "tail" IFF using PowerShell rather than the "DOS" shell. https://www.middlewareinventory.com/blog/powershell-tail-file-windows-tail-command/ or install a Windows binary equivalent http://tailforwin32.sourceforge.net/ -- ??? Wulfraed? ? ? ? ? ? ? ? Dennis Lee Bieber? ? ? ? AF6VN ??? wlfraed at ix.netcom.com? ? http://wlfraed.microdiversity.freeddns.org/ -- https://mail.python.org/mailman/listinfo/python-list From anthony.flury at btinternet.com Thu May 12 02:28:58 2022 From: anthony.flury at btinternet.com (anthony.flury) Date: Thu, 12 May 2022 07:28:58 +0100 Subject: [Python-ideas] Re: New Tool Proposal In-Reply-To: References: <6a16e4f8-b1d7-292e-60d2-1db611456c2f@btinternet.com> Message-ID: On 10/05/2022 15:04, Dan Stromberg wrote: > > On Tue, May 10, 2022 at 3:15 AM Chris Angelico wrote: > > > It is often the case that developer write Code in Python and > then convert to a C extension module for performance regions. > > > > A C extension module has a lot of boiler plate code - for > instance the Structures required for each class, the functions for > Module initialization etc. > > > > My Idea is a simple tool that uses introspection tools to take a > Python module and to generate the relevant boiler plate for the > module - including blank functions for the module classes and for > methods. This tool would use type annotations (if given) to make > sensible choices for parameter and attribute types, including > using int and float directly rather than Internal objects > (depending on tool options). > > > Two things to say about this: > 1) Sometimes abandoning a pure python module for a C extension for > performance is a mistake - because Pypy is probably going to be much > faster with the pure python module Dan, Thanks for your response, but I think PyPy will have a long way to go to make JIT generated code more efficient than hand crafted C extension. for instance PyPy will almost certainly be unable to determine if integers, floats etc will need to use the Python runtime or can be optimized to use direct C types, another example would be lists - there are some cases where the C extension will always be a lot more efficient using C arrays than using the C list runtime machinery. While there might be some cases where PyPy will be more efficient, I don't think that will be the case for all programs, and that ignores the fact that PyPi has major issues with the CAPI. > 2) I've had some luck using m4 to maintain a single source file that > is used to automatically generate both pure python and cython.? This > is a little like using cpp in a C project. > > For examples of #2, perhaps see: > https://stromberg.dnsalias.org/~strombrg/treap/ > https://stromberg.dnsalias.org/svn/rolling_checksum_mod/trunk/ > https://stromberg.dnsalias.org/~strombrg/sort-comparison/ > > It's often nice to keep the lines of the pure-python and cython having > a 1-1 relationship, so that tracebacks report useful line numbers > either way.? However, in the treap example I've dispensed with that > because some methods were almost identical but had some boilerplate - > and m4 was able to handle that nicely at the cost of lines being 1-1. > > HTH -- Anthony Flury *Moble*: +44 07743 282707 *Home*: +44 (0)1206 391294 *email*: anthony.flury at btinternet.com From anthony.flury at btinternet.com Sat May 7 04:21:43 2022 From: anthony.flury at btinternet.com (anthony.flury) Date: Sat, 7 May 2022 09:21:43 +0100 (BST) Subject: [docs] Reporting a Bug In-Reply-To: References: Message-ID: <4eaf116d.69ad.1809d9c3a42.Webtop.90@btinternet.com> This is exactly as expected. Strip removes any of the characters in the passed string from both the front and the end of the string being stripped. The letter 'T' from the start of 'The meaning of life' does not appear in the word 'meaning' so nothing is removed from the start of the string. The letter 'e' from the end of 'The meaning of life' does appear in the word 'meaning' so it is removed from the sentence; it will then look at the next last letter 'f', but since this doesn't appear in the word 'meaning' nothing else is removed and the new string is returned. The argument passed to strip(..) method is the set of characters to be removed - so any characters in that set are removed from the start and end of the string. ------ Original Message ------ From: "Jeff Jeffi" To: docs at python.org Cc: python-list at python.org Sent: Wednesday, 4 May, 22 At 10:36 Subject: [docs] Reporting a Bug Hello dears, First of all i am not sure about this issue please advise me if there is any mistake in my report. for example in python 3.6.3 shell: >>> x= "The meaning of life" x.strip("meaning") 'The meaning of lif' As you see the letter "e" of the word "life" is removed. Sincerely yours. J.Mohammadi _______________________________________________ docs mailing list -- docs at python.org To unsubscribe send an email to docs-leave at python.org https://mail.python.org/mailman3/lists/docs.python.org/ Member address: anthony.flury at btinternet.com --
Anthony Flury
anthony.flury at btinternet.com From David.Raymond at tomtom.com Thu May 12 08:18:54 2022 From: David.Raymond at tomtom.com (David Raymond) Date: Thu, 12 May 2022 12:18:54 +0000 Subject: Changing calling sequence In-Reply-To: References: Message-ID: >>def TempsOneDay(*dateComponents): >> if len(dateComponents) == 3: >> year, month, date = dateComponents >> elif len(dateComponents) == 1 and isinstance(dateComponents[0], datetime.date): >> year, month, date = (dateComponents[0].year, dateComponents[0].month, dateComponents[0].day) >> else: >> raise Exception("Error message here") > >|>>> help( TempsOneDay ) >|Help on function TempsOneDay in module __main__: >| >|TempsOneDay(*dateComponents) Then just add an appropriate docstring. >>> def TempsOneDay(*dateComponents): ... """Can be called either with 3 arguments: year, month, day ... or with a single datetime.date object""" ... if len(dateComponents) == 3: ... year, month, date = dateComponents ... elif len(dateComponents) == 1 and isinstance(dateComponents[0], datetime.date): ... year, month, date = (dateComponents[0].year, dateComponents[0].month, dateComponents[0].day) ... else: ... raise Exception("Error message here") ... >>> help(TempsOneDay) Help on function TempsOneDay in module __main__: TempsOneDay(*dateComponents) Can be called either with 3 arguments: year, month, day or with a single datetime.date object >>> From drsalists at gmail.com Thu May 12 12:25:24 2022 From: drsalists at gmail.com (Dan Stromberg) Date: Thu, 12 May 2022 09:25:24 -0700 Subject: "py" command for Linux and Mac? Message-ID: Hi folks. I heard there's a Windows-like "py" command for Linux (and Mac?). I'm finally getting to porting a particular project's Python 2.7 code to 3.x, and one of the first steps will probably be changing a lot of "python2 script.py" to use #!/usr/bin/env python2 and chmod +x. Then we can update the scripts one at a time to use #!/usr/bin/env python3. However, would this be Linux-and-Mac-only? I'm not at all sure this code will ever move to Windows, but in case it does, would a "py" command work on all 3 if I use #!/usr/bin/env py? And if so, where can I find that "py" command for Linux and Mac? I tried searching for it in Google and on Pypi, but unsurprisingly searching for "py" gives a buzzillion hits on other things. Thanks! From python at mrabarnett.plus.com Thu May 12 13:33:02 2022 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 12 May 2022 18:33:02 +0100 Subject: "py" command for Linux and Mac? In-Reply-To: References: Message-ID: On 2022-05-12 17:25, Dan Stromberg wrote: > Hi folks. > > I heard there's a Windows-like "py" command for Linux (and Mac?). > > I'm finally getting to porting a particular project's Python 2.7 code to > 3.x, and one of the first steps will probably be changing a lot of "python2 > script.py" to use #!/usr/bin/env python2 and chmod +x. Then we can update > the scripts one at a time to use #!/usr/bin/env python3. > > However, would this be Linux-and-Mac-only? I'm not at all sure this code > will ever move to Windows, but in case it does, would a "py" command work > on all 3 if I use #!/usr/bin/env py? > > And if so, where can I find that "py" command for Linux and Mac? > > I tried searching for it in Google and on Pypi, but unsurprisingly > searching for "py" gives a buzzillion hits on other things. > It's called the "Python Launcher"; you'll get more helpful results if you search for that. From Marco.Sulla.Python at gmail.com Thu May 12 13:48:49 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Thu, 12 May 2022 19:48:49 +0200 Subject: tail In-Reply-To: References: Message-ID: On Thu, 12 May 2022 at 00:50, Stefan Ram wrote: > > Marco Sulla writes: > >def tail(filepath, n=10, chunk_size=100): > > if (n <= 0): > > raise ValueError(_err_n) > ... > > There's no spec/doc, so one can't even test it. Excuse me, you're very right. """ A function that "tails" the file. If you don't know what that means, google "man tail" filepath: the file path of the file to be "tailed" n: the numbers of lines "tailed" chunk_size: oh don't care, use it as is """ From mats at wichmann.us Thu May 12 13:52:33 2022 From: mats at wichmann.us (Mats Wichmann) Date: Thu, 12 May 2022 11:52:33 -0600 Subject: "py" command for Linux and Mac? In-Reply-To: References: Message-ID: On 5/12/22 10:25, Dan Stromberg wrote: > Hi folks. > > I heard there's a Windows-like "py" command for Linux (and Mac?). > > I'm finally getting to porting a particular project's Python 2.7 code to > 3.x, and one of the first steps will probably be changing a lot of "python2 > script.py" to use #!/usr/bin/env python2 and chmod +x. Then we can update > the scripts one at a time to use #!/usr/bin/env python3. > > However, would this be Linux-and-Mac-only? I'm not at all sure this code > will ever move to Windows, but in case it does, would a "py" command work > on all 3 if I use #!/usr/bin/env py? The py command (python lanucher) respects shebang lines. https://docs.python.org/3/using/windows.html#python-launcher-for-windows From ongekruisigde at news.eternal-september.org Thu May 12 13:59:51 2022 From: ongekruisigde at news.eternal-september.org (De ongekruisigde) Date: Thu, 12 May 2022 17:59:51 -0000 (UTC) Subject: "py" command for Linux and Mac? References: Message-ID: On 2022-05-12, Mats Wichmann wrote: > On 5/12/22 10:25, Dan Stromberg wrote: >> Hi folks. >> >> I heard there's a Windows-like "py" command for Linux (and Mac?). >> >> I'm finally getting to porting a particular project's Python 2.7 code to >> 3.x, and one of the first steps will probably be changing a lot of "python2 >> script.py" to use #!/usr/bin/env python2 and chmod +x. Then we can update >> the scripts one at a time to use #!/usr/bin/env python3. >> >> However, would this be Linux-and-Mac-only? I'm not at all sure this code >> will ever move to Windows, but in case it does, would a "py" command work >> on all 3 if I use #!/usr/bin/env py? > > The py command (python lanucher) respects shebang lines. Linux by itself respects shebang lines, so you don't need a separate launcher program. Just put e.g.: #! /usr/bin/env python at the top of your Python file. -- In the beginning there was darkness and the darkness was without form and void. And in addition to the darkness there was also me. And I moved upon the face of the darkness and I saw that I was alone. ... ... ... Let there be light. [Bomb 20; John Carpenter's Dark Star - 1974] From michael.stemper at gmail.com Thu May 12 15:49:46 2022 From: michael.stemper at gmail.com (Michael F. Stemper) Date: Thu, 12 May 2022 14:49:46 -0500 Subject: Changing calling sequence In-Reply-To: References: <45cd23fe.acec.180b4b37653.Webtop.104@btinternet.com> Message-ID: On 11/05/2022 14.58, anthony.flury wrote: > Why not do : > > ????? def TempsOneDayDT(date:datetime.date): > ???????????? return TempsOneDay(date.year, date.month, date.day) > > No repeat of code - just a different interface to the same functionality. Yeah, a one-line wrapper around the original function seems a lot simpler that any of my ideas. I think that I'll even use the name from your example. Thanks to all who posted, as well as the many lurkers who support me in email. -- Michael F. Stemper Economists have correctly predicted seven of the last three recessions. From Marco.Sulla.Python at gmail.com Thu May 12 16:45:42 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Thu, 12 May 2022 22:45:42 +0200 Subject: tail In-Reply-To: References: Message-ID: Thank you very much. This helped me to improve the function: import os _lf = b"\n" _err_n = "Parameter n must be a positive integer number" _err_chunk_size = "Parameter chunk_size must be a positive integer number" def tail(filepath, n=10, chunk_size=100): if (n <= 0): raise ValueError(_err_n) if (n % 1 != 0): raise ValueError(_err_n) if (chunk_size <= 0): raise ValueError(_err_chunk_size) if (chunk_size % 1 != 0): raise ValueError(_err_chunk_size) n_chunk_size = n * chunk_size pos = os.stat(filepath).st_size chunk_line_pos = -1 newlines_to_find = n first_step = True with open(filepath, "rb") as f: text = bytearray() while pos != 0: pos -= n_chunk_size if pos < 0: pos = 0 f.seek(pos) chars = f.read(n_chunk_size) text[0:0] = chars search_pos = n_chunk_size while search_pos != -1: chunk_line_pos = chars.rfind(_lf, 0, search_pos) if first_step and chunk_line_pos == search_pos - 1: newlines_to_find += 1 first_step = False if chunk_line_pos != -1: newlines_to_find -= 1 if newlines_to_find == 0: break search_pos = chunk_line_pos if newlines_to_find == 0: break return bytes(text[chunk_line_pos+1:]) On Thu, 12 May 2022 at 20:29, Stefan Ram wrote: > I am not aware of a definition of "line" above, > but the PLR says: > > |A physical line is a sequence of characters terminated > |by an end-of-line sequence. > > . So 10 lines should have 10 end-of-line sequences. > Maybe. Maybe not. What if the file ends with no newline? From wlfraed at ix.netcom.com Thu May 12 17:48:25 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Thu, 12 May 2022 17:48:25 -0400 Subject: tail References: Message-ID: On Thu, 12 May 2022 22:45:42 +0200, Marco Sulla declaimed the following: > >Maybe. Maybe not. What if the file ends with no newline? https://github.com/coreutils/coreutils/blob/master/src/tail.c Lines 567-569 (also lines 550-557 for "bytes_read" determination) -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From cs at cskk.id.au Thu May 12 18:29:13 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Fri, 13 May 2022 08:29:13 +1000 Subject: tail In-Reply-To: References: Message-ID: On 12May2022 19:48, Marco Sulla wrote: >On Thu, 12 May 2022 at 00:50, Stefan Ram wrote: >> There's no spec/doc, so one can't even test it. > >Excuse me, you're very right. > >""" >A function that "tails" the file. If you don't know what that means, >google "man tail" > >filepath: the file path of the file to be "tailed" >n: the numbers of lines "tailed" >chunk_size: oh don't care, use it as is This is nearly the worst "specification" I have ever seen. Describe what your function _does_. Do not just send people to an arbitrary search engine to find possibly ephemeral web pages where someone has typed "man tail" and/or (if lucky) web pages with the output of "man tail" for any of several platforms. But this is sounding more and more like a special purpose task to be done for your particular use cases. That says it should be in your personal toolkit. If it has general applicability, _publish_ your toolkit for others to use. You can do that trivially by pushing your code repo to any of several free services like bitbucket, gitlab, sourcehut, github etc. Or you can go the extra few yards and publish a package to PyPI and see if anyone uses it. Part of your problem is that you think the term "tail" has a specific simple obvious meaning. But even to me it means at least 2 things: - to report the last "n" "lines" of a text file - to continuously report "new" data appended to a file These are different, though related, tasks. The latter one is particularly easy if done purely for bytes (on systems which allow it). As you've had explained to you, the former task is actually very fiddly. It is fiddly both in boundary conditions and complicated by being dependent on the text encoding, which you do not inherently know - that implies that you ought to (a) provide a way to specify that encoding and (b) maybe have a reasonable fallback default. But that default needs to be carefully and precisely explained. And the "find a line ending" criteria need to be explained. And the "sync to a character boundary" needs to be explained, including where it cannot be done. Cheers, Cameron Simpson From python.list at tim.thechases.com Thu May 12 19:07:02 2022 From: python.list at tim.thechases.com (Tim Chase) Date: Thu, 12 May 2022 18:07:02 -0500 Subject: Accuracy of multiprocessing.Queue.qsize before any Queue.get invocations? Message-ID: <20220512180702.41656ce3@bigbox.attlocal.net> The documentation says[1] > Return the approximate size of the queue. Because of > multithreading/multiprocessing semantics, this number is not > reliable. Are there any circumstances under which it *is* reliable? Most germane, if I've added a bunch of items to the Queue, but not yet launched any processes removing those items from the Queue, does Queue.qsize accurately (and reliably) reflect the number of items in the queue? q = Queue() for fname in os.listdir(): q.put(fname) file_count = q.qsize() # is this reliable? # since this hasn't yet started fiddling with it for _ in range(os.cpu_count()): Process(target=myfunc, args=(q, arg2, arg3)).start() I'm currently tracking the count as I add them to my Queue, file_count = 0 for fname in os.listdir(): q.put(fname) file_count += 1 but if .qsize is reliably accurate before anything has a chance to .get data from it, I'd prefer to tidy the code by removing the redunant counting code if I can. I'm just not sure what circumstances the "this number is not reliable" holds. I get that things might be asynchronously added/removed once processes are running, but is there anything that would cause unreliability *before* other processes/consumers run? Thanks, -tkc [1] https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue.qsize From miked at dewhirst.com.au Thu May 12 19:35:57 2022 From: miked at dewhirst.com.au (Mike Dewhirst) Date: Fri, 13 May 2022 09:35:57 +1000 Subject: Windows registry PermissionError Message-ID: <84e2a35c-3882-aaf5-a0a6-3efefc1ceb8d@dewhirst.com.au> I'm trying to copy a value from HKLM to HKCU for application rollout via bulk installation by an administrator but individual Windows user setup. Getting this ... Traceback (most recent call last): ? File "D:\Users\mike\envs\chemdata\registry\wreg\wreg.py", line 84, in ??? curegistry.setvalue('Country', anz) ? File "D:\Users\mike\envs\chemdata\registry\wreg\wreg.py", line 51, in setvalue ??? return wr.SetValueEx(self.select(), vname, 0, 1, value) PermissionError: [WinError 5] Access is denied ... on my very own laptop where my login has admistrator permissions ... which I realise means nothing in this case. But I would not have been surprised if it worked here but not in the field where users are firefighters and definitely not administrators and cannot install software on their workstations. import winreg as wr class Registry: def __init__(self, computer=None, hkey=None, sub_key=None): # computer is None means this computer self.computer = computer self.key = hkey self.sub_key = sub_key def connect(self): return wr.ConnectRegistry(self.computer, self.key) def select(self): # also tried OpenKeyEx() return wr.OpenKey( key=self.key, sub_key=self.sub_key, access=wr.KEY_ALL_ACCESS + wr.KEY_WRITE, ) def query(self, vname): return wr.QueryValueEx(self.select(), vname) def setvalue(self, vname, value): return wr.SetValueEx(self.select(), vname, 0, 1, value) if __name__ == "__main__": lmregistry = Registry( hkey=wr.HKEY_LOCAL_MACHINE, sub_key="SOFTWARE\WOW6432Node\XXX Technology\AppName", ) print(f"\n{lmregistry.sub_key}") anz = lmregistry.query('Country')[0] print(f"\n{anz}") # works fine curegistry = Registry( hkey=wr.HKEY_CURRENT_USER, sub_key="SOFTWARE\XXX Technology\AppName", ) curegistry.setvalue('Country', anz) <<<<< BOOM <<<<< ... Any hints appreciated. Cheers Mike -- Signed email is an absolute defence against phishing. This email has been signed with my private key. If you import my public key you can automatically decrypt my signature and be sure it came from me. Just ask and I'll send it to you. Your email software can handle signing. -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature Type: application/pgp-signature Size: 495 bytes Desc: OpenPGP digital signature URL: From eryksun at gmail.com Thu May 12 23:42:25 2022 From: eryksun at gmail.com (Eryk Sun) Date: Thu, 12 May 2022 22:42:25 -0500 Subject: Windows registry PermissionError In-Reply-To: <84e2a35c-3882-aaf5-a0a6-3efefc1ceb8d@dewhirst.com.au> References: <84e2a35c-3882-aaf5-a0a6-3efefc1ceb8d@dewhirst.com.au> Message-ID: On 5/12/22, Mike Dewhirst wrote: > > access=wr.KEY_ALL_ACCESS + wr.KEY_WRITE, The access parameter is a bit mask of access rights that combine via bitwise OR (|), not via arithmetic addition. KEY_ALL_ACCESS (0x000F_003F) is a superset of KEY_WRITE (0x0002_0006): KEY_WRITE = ( READ_CONTROL | # 0x0002_0000 KEY_SET_VALUE | # 0x0000_0002 KEY_CREATE_SUB_KEY | # 0x0000_0004 ) # 0x0002_0006 KEY_ALL_ACCESS = ( DELETE | # 0x0001_0000 READ_CONTROL | # 0x0002_0000 WRITE_DAC | # 0x0004_0000 WRITE_OWNER | # 0x0008_0000 KEY_QUERY_VALUE | # 0x0000_0001 KEY_SET_VALUE | # 0x0000_0002 KEY_CREATE_SUB_KEY | # 0x0000_0004 KEY_ENUMERATE_SUB_KEYS | # 0x0000_0008 KEY_NOTIFY | # 0x0000_0010 KEY_CREATE_LINK | # 0x0000_0020 ) # 0x000F_003F The result of the arithmetic addition `KEY_ALL_ACCESS + KEY_WRITE` is 0x0011_0045, which is wrong and meaningless. Registry key objects do not support SYNCHRONIZE (0x0010_0000) access; DELETE (0x0001_0000) access isn't needed; 0x0000_0040 is not a supported key right; KEY_CREATE_SUB_KEY (0x0000_0004) access isn't needed; and KEY_QUERY_VALUE (0x0000_0001) isn't sufficient. You should limit the requested access to the specific access rights that are required for querying and setting values in the key: access=(wr.KEY_QUERY_VALUE | wr.KEY_SET_VALUE) > def setvalue(self, vname, value): > return wr.SetValueEx(self.select(), vname, 0, 1, value) You shouldn't hard code the value of the data type constant. Use wr.REG_SZ instead of 1. The return value of self.select() is a winreg PyHKEY object that wraps the OS handle for the key object. You're relying on implicit closing of this handle based on referencing counting. It's cleaner to use it in a `with` statement, as you would for a file object returned by open(). For example: with self.select() as hkey: wr.SetValueEx(hkey, vname, 0, wr.REG_SZ, value) > lmregistry = Registry( > hkey=wr.HKEY_LOCAL_MACHINE, > sub_key="SOFTWARE\WOW6432Node\XXX Technology\AppName", You really shouldn't open the "WOW6432Node" key directly. It is an implementation detail of the WOW64 subsystem that runs 32-bit applications on a 64-bit system. If you need to operate on the registry keys of 32-bit applications from a native 64-bit process, open the normal path using the access right KEY_WOW64_32KEY (0x0000_0200). For example: hkey = wr.HKEY_LOCAL_MACHINE subkey = r"SOFTWARE\XXX Technology\AppName" access = ( wr.KEY_QUERY_VALUE | wr.KEY_SET_VALUE | wr.KEY_WOW64_32KEY ) Typically you'd first try opening the path without either KEY_WOW64_32KEY or KEY_WOW64_64KEY. The default view matches the current process. https://docs.microsoft.com/en-us/windows/win32/winprog64/accessing-an-alternate-registry-view Remember to escape the backslash separators in string literals of key paths, or use raw string literals as I used in the above example. From miked at dewhirst.com.au Fri May 13 01:36:44 2022 From: miked at dewhirst.com.au (Mike Dewhirst) Date: Fri, 13 May 2022 15:36:44 +1000 Subject: [Solved] Re: Windows registry PermissionError In-Reply-To: References: <84e2a35c-3882-aaf5-a0a6-3efefc1ceb8d@dewhirst.com.au> Message-ID: <2e535898-3aaf-8904-017e-901acc88e8ad@dewhirst.com.au> Eryk Many thanks. It is working perfectly now. See below for the reworked code. Cheers Mike On 13/05/2022 1:42 pm, Eryk Sun wrote: > On 5/12/22, Mike Dewhirst wrote: >> access=wr.KEY_ALL_ACCESS + wr.KEY_WRITE, import winreg as wr class Registry: def __init__(self, computer=None, hkey=None, sub_key=None): self.computer = computer self.key = hkey self.sub_key = sub_key def connect(self): return wr.ConnectRegistry(self.computer, self.key) def select(self, access=None): with self.connect() as hkey: return wr.OpenKeyEx(key=hkey, sub_key=self.sub_key, access=access) def count(self): access = wr.KEY_QUERY_VALUE return wr.QueryInfoKey(self.select(access=access)) def query(self, vname, access=None): if access is None: access = wr.KEY_READ | wr.KEY_WOW64_32KEY return wr.QueryValueEx(self.select(access=access), vname) def setvalue(self, vname, value, access=None): if access is None: access = wr.KEY_SET_VALUE with self.select(access=access) as hkey: return wr.SetValueEx(hkey, vname, 0, wr.REG_SZ, value) if __name__ == "__main__": lmregistry = Registry( hkey=wr.HKEY_LOCAL_MACHINE, sub_key=r"SOFTWARE\XXX Technology\AppName", ) print(f"\n{lmregistry.sub_key}") anz = lmregistry.query('Country')[0] print(f"\n{anz}") db = lmregistry.query('Database')[0] print(f"\n{db}") devref = lmregistry.query('v135')[0] print(f"\n{devref}") orgid = lmregistry.query('v136')[0] print(f"\n{orgid}") curegistry = Registry( hkey=wr.HKEY_CURRENT_USER, sub_key=r"SOFTWARE\XXX Technology\AppName", ) print(f"\n{curegistry.sub_key}") anz = curegistry.query('Country')[0] print(f"\n{anz}") db = curegistry.query('Database')[0] print(f"\n{db}") devref = curegistry.query('v135')[0] print(f"\n{devref}") orgid = curegistry.query('v136')[0] print(f"\n{orgid}") curegistry.setvalue('Country', 'nz') curegistry.setvalue('Database', '2022.2') curegistry.setvalue('v135', 'Asus10') curegistry.setvalue('v136', orgid.replace('ALL', 'Most')) anz = curegistry.query('Country')[0] print(f"\n{anz}") db = curegistry.query('Database')[0] print(f"\n{db}") devref = curegistry.query('v135')[0] print(f"\n{devref}") orgid = curegistry.query('v136')[0] print(f"\n{orgid}") Again, many thanks for putting so much effort into educating me. Cheers Mike > The access parameter is a bit mask of access rights that combine via > bitwise OR (|), not via arithmetic addition. > > KEY_ALL_ACCESS (0x000F_003F) is a superset of KEY_WRITE (0x0002_0006): > > KEY_WRITE = ( > READ_CONTROL | # 0x0002_0000 > KEY_SET_VALUE | # 0x0000_0002 > KEY_CREATE_SUB_KEY | # 0x0000_0004 > ) # 0x0002_0006 > > KEY_ALL_ACCESS = ( > DELETE | # 0x0001_0000 > READ_CONTROL | # 0x0002_0000 > WRITE_DAC | # 0x0004_0000 > WRITE_OWNER | # 0x0008_0000 > KEY_QUERY_VALUE | # 0x0000_0001 > KEY_SET_VALUE | # 0x0000_0002 > KEY_CREATE_SUB_KEY | # 0x0000_0004 > KEY_ENUMERATE_SUB_KEYS | # 0x0000_0008 > KEY_NOTIFY | # 0x0000_0010 > KEY_CREATE_LINK | # 0x0000_0020 > ) # 0x000F_003F > > The result of the arithmetic addition `KEY_ALL_ACCESS + KEY_WRITE` is > 0x0011_0045, which is wrong and meaningless. Registry key objects do > not support SYNCHRONIZE (0x0010_0000) access; DELETE (0x0001_0000) > access isn't needed; 0x0000_0040 is not a supported key right; > KEY_CREATE_SUB_KEY (0x0000_0004) access isn't needed; and > KEY_QUERY_VALUE (0x0000_0001) isn't sufficient. > > You should limit the requested access to the specific access rights > that are required for querying and setting values in the key: > > access=(wr.KEY_QUERY_VALUE | wr.KEY_SET_VALUE) > >> def setvalue(self, vname, value): >> return wr.SetValueEx(self.select(), vname, 0, 1, value) > You shouldn't hard code the value of the data type constant. Use > wr.REG_SZ instead of 1. > > The return value of self.select() is a winreg PyHKEY object that wraps > the OS handle for the key object. You're relying on implicit closing > of this handle based on referencing counting. It's cleaner to use it > in a `with` statement, as you would for a file object returned by > open(). For example: > > with self.select() as hkey: > wr.SetValueEx(hkey, vname, 0, wr.REG_SZ, value) > >> lmregistry = Registry( >> hkey=wr.HKEY_LOCAL_MACHINE, >> sub_key="SOFTWARE\WOW6432Node\XXX Technology\AppName", > You really shouldn't open the "WOW6432Node" key directly. It is an > implementation detail of the WOW64 subsystem that runs 32-bit > applications on a 64-bit system. If you need to operate on the > registry keys of 32-bit applications from a native 64-bit process, > open the normal path using the access right KEY_WOW64_32KEY > (0x0000_0200). For example: > > hkey = wr.HKEY_LOCAL_MACHINE > subkey = r"SOFTWARE\XXX Technology\AppName" > access = ( > wr.KEY_QUERY_VALUE | > wr.KEY_SET_VALUE | > wr.KEY_WOW64_32KEY > ) > > Typically you'd first try opening the path without either > KEY_WOW64_32KEY or KEY_WOW64_64KEY. The default view matches the > current process. > > https://docs.microsoft.com/en-us/windows/win32/winprog64/accessing-an-alternate-registry-view > > Remember to escape the backslash separators in string literals of key > paths, or use raw string literals as I used in the above example. -- Signed email is an absolute defence against phishing. This email has been signed with my private key. If you import my public key you can automatically decrypt my signature and be sure it came from me. Just ask and I'll send it to you. Your email software can handle signing. -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature Type: application/pgp-signature Size: 495 bytes Desc: OpenPGP digital signature URL: From eryksun at gmail.com Fri May 13 02:14:30 2022 From: eryksun at gmail.com (Eryk Sun) Date: Fri, 13 May 2022 01:14:30 -0500 Subject: [Solved] Re: Windows registry PermissionError In-Reply-To: <2e535898-3aaf-8904-017e-901acc88e8ad@dewhirst.com.au> References: <84e2a35c-3882-aaf5-a0a6-3efefc1ceb8d@dewhirst.com.au> <2e535898-3aaf-8904-017e-901acc88e8ad@dewhirst.com.au> Message-ID: Since self.connect() is always called, you should document that the initial hkey parameter has to be one of the following predefined key handles: HKEY_LOCAL_MACHINE HKEY_USERS HKEY_PERFORMANCE_DATA WinAPI RegConnectRegistryW() only matters when the target computer is a different machine, in which case an RPC proxy handle is returned. From miked at dewhirst.com.au Fri May 13 02:19:41 2022 From: miked at dewhirst.com.au (Mike Dewhirst) Date: Fri, 13 May 2022 16:19:41 +1000 Subject: [Solved] Re: Windows registry PermissionError In-Reply-To: References: <84e2a35c-3882-aaf5-a0a6-3efefc1ceb8d@dewhirst.com.au> <2e535898-3aaf-8904-017e-901acc88e8ad@dewhirst.com.au> Message-ID: <678a8c64-4d6f-e96f-c890-5b429548d036@dewhirst.com.au> On 13/05/2022 4:14 pm, Eryk Sun wrote: > Since self.connect() is always called, you should document that the > initial hkey parameter has to be one of the following predefined key > handles: > > HKEY_LOCAL_MACHINE > HKEY_USERS I'm targeting HKEY_CURRENT_USER so I assume HK_USERS includes that. Thanks again Eryk Cheers mike > HKEY_PERFORMANCE_DATA > > WinAPI RegConnectRegistryW() only matters when the target computer is > a different machine, in which case an RPC proxy handle is returned. -- Signed email is an absolute defence against phishing. This email has been signed with my private key. If you import my public key you can automatically decrypt my signature and be sure it came from me. Just ask and I'll send it to you. Your email software can handle signing. -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature Type: application/pgp-signature Size: 495 bytes Desc: OpenPGP digital signature URL: From eryksun at gmail.com Fri May 13 02:37:20 2022 From: eryksun at gmail.com (Eryk Sun) Date: Fri, 13 May 2022 01:37:20 -0500 Subject: [Solved] Re: Windows registry PermissionError In-Reply-To: <678a8c64-4d6f-e96f-c890-5b429548d036@dewhirst.com.au> References: <84e2a35c-3882-aaf5-a0a6-3efefc1ceb8d@dewhirst.com.au> <2e535898-3aaf-8904-017e-901acc88e8ad@dewhirst.com.au> <678a8c64-4d6f-e96f-c890-5b429548d036@dewhirst.com.au> Message-ID: On 5/13/22, Mike Dewhirst wrote: > On 13/05/2022 4:14 pm, Eryk Sun wrote: >> Since self.connect() is always called, you should document that the >> initial hkey parameter has to be one of the following predefined key >> handles: >> >> HKEY_LOCAL_MACHINE >> HKEY_USERS > > I'm targeting HKEY_CURRENT_USER so I assume HK_USERS includes that. Using HKEY_CURRENT_USER with RegConnectRegistryW() to access a remote registry isn't well defined and not documented as supported. If it works at all, the API probably just opens a subkey of the remote HKEY_USERS based on the string SID (security identifier string) of the current user. That may fail as not found since user SIDs are unique to machines, unless it's on a domain. Bear in mind that the remote registry service is running on the remote machine as SYSTEM (S-1-5-18) in the non-interactive services session. If it literally accessed its "current user", then it would open "HKEY_USERS\S-1-5-18". From miked at dewhirst.com.au Fri May 13 03:00:00 2022 From: miked at dewhirst.com.au (Mike Dewhirst) Date: Fri, 13 May 2022 17:00:00 +1000 Subject: [Solved] Re: Windows registry PermissionError In-Reply-To: References: <84e2a35c-3882-aaf5-a0a6-3efefc1ceb8d@dewhirst.com.au> <2e535898-3aaf-8904-017e-901acc88e8ad@dewhirst.com.au> <678a8c64-4d6f-e96f-c890-5b429548d036@dewhirst.com.au> Message-ID: On 13/05/2022 4:37 pm, Eryk Sun wrote: > On 5/13/22, Mike Dewhirst wrote: >> On 13/05/2022 4:14 pm, Eryk Sun wrote: >>> Since self.connect() is always called, you should document that the >>> initial hkey parameter has to be one of the following predefined key >>> handles: >>> >>> HKEY_LOCAL_MACHINE >>> HKEY_USERS >> I'm targeting HKEY_CURRENT_USER so I assume HK_USERS includes that. > Using HKEY_CURRENT_USER with RegConnectRegistryW() to access a remote > registry isn't well defined and not documented as supported. If it > works at all, the API probably just opens a subkey of the remote > HKEY_USERS based on the string SID (security identifier string) of the > current user. That may fail as not found since user SIDs are unique to > machines, unless it's on a domain. > > Bear in mind that the remote registry service is running on the remote > machine as SYSTEM (S-1-5-18) in the non-interactive services session. > If it literally accessed its "current user", then it would open > "HKEY_USERS\S-1-5-18". In that case, I'll remove 'computer' as a parameter and lock it in as None. I'll document why and if I ever need to access a remote registry I'll do some research along the lines you suggest. My case demands execution of this code via login script for end-user configuration after the IT department has done a bulk installation rollout. One of key items is ComputerName which is in HKLM and needs to be in HKCU for whichever user logs in. The application looks in HKCU for a unique Device Reference and ComputerName suffices and is attractive because it is also the asset number of the machine. -- Signed email is an absolute defence against phishing. This email has been signed with my private key. If you import my public key you can automatically decrypt my signature and be sure it came from me. Just ask and I'll send it to you. Your email software can handle signing. -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature Type: application/pgp-signature Size: 495 bytes Desc: OpenPGP digital signature URL: From Marco.Sulla.Python at gmail.com Fri May 13 06:16:57 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Fri, 13 May 2022 12:16:57 +0200 Subject: tail In-Reply-To: References: Message-ID: On Fri, 13 May 2022 at 00:31, Cameron Simpson wrote: > On 12May2022 19:48, Marco Sulla wrote: > >On Thu, 12 May 2022 at 00:50, Stefan Ram wrote: > >> There's no spec/doc, so one can't even test it. > > > >Excuse me, you're very right. > > > >""" > >A function that "tails" the file. If you don't know what that means, > >google "man tail" > > > >filepath: the file path of the file to be "tailed" > >n: the numbers of lines "tailed" > >chunk_size: oh don't care, use it as is > > This is nearly the worst "specification" I have ever seen. > You're lucky. I've seen much worse (or no one). From 2QdxY4RzWzUUiLuE at potatochowder.com Fri May 13 06:47:07 2022 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Fri, 13 May 2022 05:47:07 -0500 Subject: tail In-Reply-To: References: Message-ID: On 2022-05-13 at 12:16:57 +0200, Marco Sulla wrote: > On Fri, 13 May 2022 at 00:31, Cameron Simpson wrote: [...] > > This is nearly the worst "specification" I have ever seen. > You're lucky. I've seen much worse (or no one). At least with *no* documentation, the source code stands for itself. If I can execute it (whatever that entails), then I can (in theory) figure out *what* it does. I still don't what it's *supposed* to do, and therefore *cannot* know how well it does or doesn't "work,", but at least source code is deterministic and unambiguous (except when it isn't, but let's not go there). From martinp.dipaola at gmail.com Fri May 13 08:24:37 2022 From: martinp.dipaola at gmail.com (Martin Di Paola) Date: Fri, 13 May 2022 09:24:37 -0300 Subject: Accuracy of multiprocessing.Queue.qsize before any Queue.get invocations? In-Reply-To: <20220512180702.41656ce3@bigbox.attlocal.net> References: <20220512180702.41656ce3@bigbox.attlocal.net> Message-ID: <20220513122437.hev25jfdccbrnznf@gmail.com> If the queue was not shared to any other process, I would guess that its size is reliable. However, a plain counter could be much simpler/safer. The developer of multiprocessing.Queue, implemented size() thinking in how to share the size and maintain a reasonable consistency between process. He/she probably didn't care how well works in a single-process scenario as this is a very special case. Thanks, Martin. On Thu, May 12, 2022 at 06:07:02PM -0500, Tim Chase wrote: >The documentation says[1] > >> Return the approximate size of the queue. Because of >> multithreading/multiprocessing semantics, this number is not >> reliable. > >Are there any circumstances under which it *is* reliable? Most >germane, if I've added a bunch of items to the Queue, but not yet >launched any processes removing those items from the Queue, does >Queue.qsize accurately (and reliably) reflect the number of items in >the queue? > > q = Queue() > for fname in os.listdir(): > q.put(fname) > file_count = q.qsize() # is this reliable? > # since this hasn't yet started fiddling with it > for _ in range(os.cpu_count()): > Process(target=myfunc, args=(q, arg2, arg3)).start() > >I'm currently tracking the count as I add them to my Queue, > > file_count = 0 > for fname in os.listdir(): > q.put(fname) > file_count += 1 > >but if .qsize is reliably accurate before anything has a chance to >.get data from it, I'd prefer to tidy the code by removing the >redunant counting code if I can. > >I'm just not sure what circumstances the "this number is not >reliable" holds. I get that things might be asynchronously >added/removed once processes are running, but is there anything that >would cause unreliability *before* other processes/consumers run? > >Thanks, > >-tkc > >[1] >https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue.qsize > > > > > >-- >https://mail.python.org/mailman/listinfo/python-list From martinp.dipaola at gmail.com Fri May 13 08:38:44 2022 From: martinp.dipaola at gmail.com (Martin Di Paola) Date: Fri, 13 May 2022 09:38:44 -0300 Subject: Changing calling sequence In-Reply-To: References: Message-ID: <20220513123844.yjf34drpmhd5eakw@gmail.com> You probably want something like overload/multiple dispatch. I quick search on PyPI yields a 'multipledispatch' package. I never used, however. On Wed, May 11, 2022 at 08:36:26AM -0700, Tobiah wrote: >On 5/11/22 06:33, Michael F. Stemper wrote: >>I have a function that I use to retrieve daily data from a >>home-brew database. Its calling sequence is; >> >>def TempsOneDay( year, month, date ): >> >>After using it (and its friends) for a few years, I've come to >>realize that there are times where it would be advantageous to >>invoke it with a datetime.date as its single argument. > >You could just use all keyword args: > >def TempsOneDay(**kwargs): > > if 'date' in kwargs: > handle_datetime(kwargs['date']) > elif 'year' in kwargs and 'month' in kwargs and 'day' in kwargs: > handle_args(kwargs['year'], kwargs['month'], kwargs['day']) > else: > raise Exception("Bad keyword args") > >TempsOneDay(date=datetime.datetime.now) > >TempsOneDay(year=2022, month=11, day=30) > >-- >https://mail.python.org/mailman/listinfo/python-list From bryangan41 at gmail.com Fri May 13 12:37:49 2022 From: bryangan41 at gmail.com (bryangan41) Date: Sat, 14 May 2022 00:37:49 +0800 Subject: EAFP Message-ID: Is the following LBYL:foo = 123if foo < 200:? ? do()If so, how to change to EAFP?Thanks!Sent from Samsung tablet. From bryangan41 at gmail.com Fri May 13 12:47:03 2022 From: bryangan41 at gmail.com (bryangan41) Date: Sat, 14 May 2022 00:47:03 +0800 Subject: .0 in name Message-ID: May I know (1) why can the name start with a number?(2) where in the doc is it?!>>> import pdb>>> pdb.run('(a for a in "")')> (1)()(Pdb) s--Call--> (1)()(Pdb) a.0 = (Pdb) c>>>Sent from Samsung tablet. From pbryan at anode.ca Fri May 13 17:23:42 2022 From: pbryan at anode.ca (Paul Bryan) Date: Fri, 13 May 2022 14:23:42 -0700 Subject: .0 in name In-Reply-To: References: Message-ID: <964395bd7ff3ecdd4ce11de88d7b207b86d9cad0.camel@anode.ca> On Sat, 2022-05-14 at 00:47 +0800, bryangan41 wrote: > May I know (1) why can the name start with a number? The name of an attribute must be an identifier. An identifier cannot begin with a decimal number. > (2) where in the doc is it?! https://docs.python.org/3/reference/lexical_analysis.html#identifiers Paul From avigross at verizon.net Fri May 13 18:02:01 2022 From: avigross at verizon.net (Avi Gross) Date: Fri, 13 May 2022 22:02:01 +0000 (UTC) Subject: .0 in name In-Reply-To: References: Message-ID: <518526689.1761138.1652479321306@mail.yahoo.com> Bryan, As has been pointed out, it is very common in possibly all programming languages to not?allow digits at the start of many identifiers. It makes it hard to parse for numbers which tend?to start with digits. Some languages even have special rules on not starting a number with a zero?unless you mean for it to be seen as octal (or 0x for hexadecimal) and many other rules exist. There are languages where 12x means 12*x so even the lack of an operator ... There are exceptions that often are not really exceptions. You can use all kinds?of unusual variables in some quoted context. It is valid (albeit not encouraged) to use?backquoted The following is perfectly allowed in R: > `5x^2 + 2.3x` <- 666?> `+-+-+` <- 1000?> 1 + 2 * `5x^2 + 2.3x` + `+-+-+`?[1] 2333? And there are often issued when you do things like create the name of a column of?data in a data.frame with embedded spaces and other anomalies requiring special?handling. So why you wonder where it is documented that variables cannot be what you feel like is a?bit puzzling!? -----Original Message----- From: bryangan41 To: python-list at python.org Sent: Fri, May 13, 2022 12:47 pm Subject: .0 in name May I know (1) why can the name start with a number?(2) where in the doc is it?!>>> import pdb>>> pdb.run('(a for a in "")')> (1)()(Pdb) s--Call--> (1)()(Pdb) a.0 = (Pdb) c>>>Sent from Samsung tablet. -- https://mail.python.org/mailman/listinfo/python-list From pbryan at anode.ca Fri May 13 18:56:12 2022 From: pbryan at anode.ca (Paul Bryan) Date: Fri, 13 May 2022 15:56:12 -0700 Subject: .0 in name In-Reply-To: <518526689.1761138.1652479321306@mail.yahoo.com> References: <518526689.1761138.1652479321306@mail.yahoo.com> Message-ID: <623af911363afc1b3ce1536de26fb5f66a0fd250.camel@anode.ca> On Fri, 2022-05-13 at 22:02 +0000, Avi Gross via Python-list wrote: > So why you wonder where it is documented that variables cannot be > what you feel like is a?bit puzzling!? I had just assumed on good faith that the request to the documentation would be so that the OP could determine what is valid identifier syntax. Paul From avigross at verizon.net Fri May 13 21:22:00 2022 From: avigross at verizon.net (Avi Gross) Date: Sat, 14 May 2022 01:22:00 +0000 (UTC) Subject: .0 in name In-Reply-To: <518526689.1761138.1652479321306@mail.yahoo.com> References: <518526689.1761138.1652479321306@mail.yahoo.com> Message-ID: <1769701111.1786559.1652491320385@mail.yahoo.com> Boy do I hate when I see my code mangled by the stupid AOL mailer. Not that anyone cares, but the code should be read with each line starting with the "> " prompt. If I leave lots of blank lines, it may work, but as the illustration is not in python, I will now remove the prompts: `5x^2 + 2.3x` <- 666 `+-+-+` <- 1000 1 + 2 * `5x^2 + 2.3x` + `+-+-+` output:?2333? There are rarely good reasons for such silly variable names but as long as you let it?know to leave the quoted regions alone when parsing and look them us as exact entries?in various environments, it works fine. To add to what others already wrote, many languages have serious requirements you need?to be aware of and not assume otherwise. Some allow underscores in names and may limit?that in the first part or may in some cases suggest or require it. Some have rules about whether?a variable of one kind should start with an uppercase letter. Some allow periods in names?although an initial period may make it invisible for some purposes. And, some newer languages?allow all kinds of UNICODE characters and perhaps even some that can be seen as numeric?but aren't exactly 0-9.? ? ? ? ?...???? -----Original Message----- From: Avi Gross via Python-list To: python-list at python.org Sent: Fri, May 13, 2022 6:02 pm Subject: Re: .0 in name Bryan, As has been pointed out, it is very common in possibly all programming languages to not?allow digits at the start of many identifiers. It makes it hard to parse for numbers which tend?to start with digits. Some languages even have special rules on not starting a number with a zero?unless you mean for it to be seen as octal (or 0x for hexadecimal) and many other rules exist. There are languages where 12x means 12*x so even the lack of an operator ... There are exceptions that often are not really exceptions. You can use all kinds?of unusual variables in some quoted context. It is valid (albeit not encouraged) to use?backquoted The following is perfectly allowed in R: > `5x^2 + 2.3x` <- 666?> `+-+-+` <- 1000?> 1 + 2 * `5x^2 + 2.3x` + `+-+-+`?[1] 2333? And there are often issued when you do things like create the name of a column of?data in a data.frame with embedded spaces and other anomalies requiring special?handling. So why you wonder where it is documented that variables cannot be what you feel like is a?bit puzzling!? -----Original Message----- From: bryangan41 To: python-list at python.org Sent: Fri, May 13, 2022 12:47 pm Subject: .0 in name May I know (1) why can the name start with a number?(2) where in the doc is it?!>>> import pdb>>> pdb.run('(a for a in "")')> (1)()(Pdb) s--Call--> (1)()(Pdb) a.0 = (Pdb) c>>>Sent from Samsung tablet. -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list From jonathan.kaczynski at guildeducation.com Fri May 13 20:23:36 2022 From: jonathan.kaczynski at guildeducation.com (Jonathan Kaczynski) Date: Fri, 13 May 2022 20:23:36 -0400 Subject: Seeking deeper understanding of python equality (==) In-Reply-To: References: Message-ID: Thank you for your responses, Sam and Greg. The do_richcompare function is where my research originally took me, but I feel like I'm still missing some pieces to the puzzle. Here is my updated research since you posted your responses (I'll attach a pdf copy too): https://docs.google.com/document/d/10zgOMetEQtZCiYFnSS90pDnNZD7I_-MFohSy83pOieA/edit# The summary section, in the middle, is where I've summarized my reading of the source code. Greg, your response here, > Generally what happens with infix operators is that the interpreter > first looks for a dunder method on the left operand. If that method > doesn't exist or returns NotImplemented, it then looks for a dunder > method on the right operand. reads like the contents of the do_richcompare function. What I think I'm missing is how do the dunder methods relate to the tp_richcompare function? Thank you, Jonathan On Fri, May 6, 2022 at 11:55 PM Greg Ewing wrote: > On 7/05/22 12:22 am, Jonathan Kaczynski wrote: > > Stepping through the code with gdb, we see it jump from the compare > > operator to the dunder-eq method on the UUID object. What I want to be > able > > to do is explain the in-between steps. > > Generally what happens with infix operators is that the interpreter > first looks for a dunder method on the left operand. If that method > doesn't exist or returns NotImplemented, it then looks for a dunder > method on the right operand. > > There is an exception if the right operand is a subclass of the > left operand -- in that case the right operand's dunder method > takes precedence. > > > Also, if you change `x == y` to `y > > == x`, you still see the same behavior, which I assume has to do with > > dunder-eq being defined on the UUID class and thus given priority. > > No, in that case the conparison method of str will be getting > called first, but you won't see that in pdb because it doesn't > involve any Python code. Since strings don't know how to compare > themselves with uuids, it will return NotImplemented and the > interpreter will then call uuid's method. > > -- > Greg > -- > https://mail.python.org/mailman/listinfo/python-list > From PythonList at DancesWithMice.info Sat May 14 00:33:11 2022 From: PythonList at DancesWithMice.info (dn) Date: Sat, 14 May 2022 16:33:11 +1200 Subject: .0 in name In-Reply-To: <1769701111.1786559.1652491320385@mail.yahoo.com> References: <518526689.1761138.1652479321306@mail.yahoo.com> <1769701111.1786559.1652491320385@mail.yahoo.com> Message-ID: This is not what @Avi menat by "silly variable names" but: 3D_position 2nd_floor_area 3M_PostIt_size 3rd_degree_polynomial 360_degree_view 12_hours_later and ??? 2_fast_2_furious -- Regards, =dn From PythonList at DancesWithMice.info Sat May 14 01:05:19 2022 From: PythonList at DancesWithMice.info (dn) Date: Sat, 14 May 2022 17:05:19 +1200 Subject: EAFP In-Reply-To: References: Message-ID: <66493f9c-ec99-05d6-9e3f-f80270f0ac22@DancesWithMice.info> On 14/05/2022 04.37, bryangan41 wrote: > Is the following LBYL:foo = 123if foo < 200:? ? do() Yes (once formatted for Python). If so, how to change to EAFP? Not sure if can. Let's alter the code to: foo = 0 #and def do(): return 5 / foo Then, you will expect a ZeroDivisionError exception to be raised - whereas with other values, there will (likely) be no problem. Now, we have a clear identification for when 'forgiveness' will need to be requested! The 'EAFP' solution encloses the function-call: foo = 123 try: do() except ZeroDivisionError: undo() In the OP's code-snippet, the "200" criteria means there is no such clean-cut and automatic 'failure' attached to, or implied by, foo's value. However, we could define a custom-exception and 'raise the alarm' when 'forgiveness' is required: class ValueTooLargeError( ValueError): """Catch values larger than the valid range.""" def do(): if foo < 200: ... else: raise ValueTooLargeError This is a pythonic-construct (and thus a recommendable coding-pattern), in that the calling-routine can decide how to respond to the exception - which many vary according to the specifics of do()'s re-use. However, is it "EAFP"? We had to introduce the exact if-condition which makes it "LBYL"! Perhaps time for me to bow-out, and leave such to the philosophers... Sent from Samsung tablet. There are pills for that... -- Regards, =dn From avigross at verizon.net Sat May 14 01:21:53 2022 From: avigross at verizon.net (Avi Gross) Date: Sat, 14 May 2022 05:21:53 +0000 (UTC) Subject: .0 in name In-Reply-To: References: <518526689.1761138.1652479321306@mail.yahoo.com> <1769701111.1786559.1652491320385@mail.yahoo.com> Message-ID: <1176929671.1806058.1652505713504@mail.yahoo.com> You left out 3CPO, Dave. Names with numerals are not unreasonable in some circumstances. But programming languages are not a full spectrum of real life. They can have?reserved words too so you may not be able to use "while" and if you choose?to create a function that masks another, you cannot complain that you?assumed the computer would be smart enough to figure out which one?you wanted. Yes, some languages encourage multiple functions with the?same name but different signatures. Bottom line is each has RULES and?some serious SUGGESTIONS. It is what it is, not what you want it to be. But although starting with a numeral is verboten for variable names,?may I suggest that a relatively invisible character can make a name like?_3CPO that will be allowed. But be careful as Python programs often have?conventions on use of the underscore and don't even think about a dundering?name like __init__ ... -----Original Message----- From: dn To: python-list at python.org Sent: Sat, May 14, 2022 12:33 am Subject: Re: .0 in name This is not what @Avi menat by "silly variable names" but: 3D_position 2nd_floor_area 3M_PostIt_size 3rd_degree_polynomial 360_degree_view 12_hours_later and ??? 2_fast_2_furious -- Regards, =dn -- https://mail.python.org/mailman/listinfo/python-list From jonathan.kaczynski at guildeducation.com Sat May 14 08:33:55 2022 From: jonathan.kaczynski at guildeducation.com (Jonathan Kaczynski) Date: Sat, 14 May 2022 08:33:55 -0400 Subject: Seeking deeper understanding of python equality (==) In-Reply-To: References: Message-ID: Trying some new searches, I came across slotdefs in ./Objects/typeobject.c, and those are used in the resolve_slotdups function. The comment preceding the function says, "Note that multiple names may map to the same slot (e.g. __eq__, __ne__ etc. all map to tp_richcompare)". So, I'm still wondering how Py_TYPE(v)->tp_richcompare resolves to __eq__ on a user-defined class. Conversely, my understanding is, for a type defined in cpython, like str, there is usually an explicitly defined tp_richcompare function. Thank you, Jonathan On Fri, May 13, 2022 at 8:23 PM Jonathan Kaczynski < jonathan.kaczynski at guildeducation.com> wrote: > Thank you for your responses, Sam and Greg. > > The do_richcompare function is where my research originally took me, but I > feel like I'm still missing some pieces to the puzzle. > > Here is my updated research since you posted your responses (I'll attach a > pdf copy too): > https://docs.google.com/document/d/10zgOMetEQtZCiYFnSS90pDnNZD7I_-MFohSy83pOieA/edit# > The summary section, in the middle, is where I've summarized my reading of > the source code. > > Greg, your response here, > >> Generally what happens with infix operators is that the interpreter >> first looks for a dunder method on the left operand. If that method >> doesn't exist or returns NotImplemented, it then looks for a dunder >> method on the right operand. > > reads like the contents of the do_richcompare function. > > What I think I'm missing is how do the dunder methods relate to > the tp_richcompare function? > > Thank you, > Jonathan > > > On Fri, May 6, 2022 at 11:55 PM Greg Ewing > wrote: > >> On 7/05/22 12:22 am, Jonathan Kaczynski wrote: >> > Stepping through the code with gdb, we see it jump from the compare >> > operator to the dunder-eq method on the UUID object. What I want to be >> able >> > to do is explain the in-between steps. >> >> Generally what happens with infix operators is that the interpreter >> first looks for a dunder method on the left operand. If that method >> doesn't exist or returns NotImplemented, it then looks for a dunder >> method on the right operand. >> >> There is an exception if the right operand is a subclass of the >> left operand -- in that case the right operand's dunder method >> takes precedence. >> >> > Also, if you change `x == y` to `y >> > == x`, you still see the same behavior, which I assume has to do with >> > dunder-eq being defined on the UUID class and thus given priority. >> >> No, in that case the conparison method of str will be getting >> called first, but you won't see that in pdb because it doesn't >> involve any Python code. Since strings don't know how to compare >> themselves with uuids, it will return NotImplemented and the >> interpreter will then call uuid's method. >> >> -- >> Greg >> -- >> https://mail.python.org/mailman/listinfo/python-list >> > From eryksun at gmail.com Sat May 14 13:51:51 2022 From: eryksun at gmail.com (Eryk Sun) Date: Sat, 14 May 2022 12:51:51 -0500 Subject: Seeking deeper understanding of python equality (==) In-Reply-To: References: Message-ID: On 5/14/22, Jonathan Kaczynski wrote: > > So, I'm still wondering how Py_TYPE(v)->tp_richcompare resolves to __eq__ > on a user-defined class. Conversely, my understanding is, for a type > defined in cpython, like str, there is usually an explicitly > defined tp_richcompare function. Sometimes it's simplest to directly examine an object using a native debugger (e.g. gdb in Linux; cdb/windbg in Windows). With a debugger attached to the interpreter, create two classes, one that doesn't override __eq__() and one that does: >>> class C: ... pass ... >>> class D: ... __eq__ = lambda s, o: False ... In CPython, the id() of an object is its address in memory: >>> hex(id(C)) '0x2806a705790' >>> hex(id(D)) '0x2806a6bbfe0' Break into the attached debugger to examine the class objects: >>> kernel32.DebugBreak() (1968.1958): Break instruction exception - code 80000003 (first chance) KERNELBASE!wil::details::DebugBreak+0x2: 00007ffd`8818fd12 cc int 3 Class C uses the default object_richcompare(): 0:000> ?? *((python310!PyTypeObject *)0x2806a705790)->tp_richcompare 0x00007ffd`55cac288 _object* python310!object_richcompare+0( _object*, _object*, int) Class D uses slot_tp_richcompare(): 0:000> ?? *((python310!PyTypeObject *)0x2806a6bbfe0)->tp_richcompare 0x00007ffd`55cdef1c _object* python310!slot_tp_richcompare+0( _object*, _object*, int) Source code of slot_tp_richcompare(): https://github.com/python/cpython/blob/v3.10.4/Objects/typeobject.c#L7610-L7626 From PythonList at DancesWithMice.info Sat May 14 18:22:15 2022 From: PythonList at DancesWithMice.info (dn) Date: Sun, 15 May 2022 10:22:15 +1200 Subject: Changing calling sequence In-Reply-To: References: Message-ID: <671ca838-f59e-4739-f795-4f328399ab9f@DancesWithMice.info> On 12/05/2022 01.33, Michael F. Stemper wrote: > I have a function that I use to retrieve daily data from a > home-brew database. Its calling sequence is; > > def TempsOneDay( year, month, date ): > > After using it (and its friends) for a few years, I've come to > realize that there are times where it would be advantageous to > invoke it with a datetime.date as its single argument. > > As far as I can tell, there are three ways for me to proceed: > 1. Write a similar function that takes a single datetime.date > ?? as its argument. > 2. Rewrite the existing function so that it takes a single > ?? argument, which can be either a tuple of (year,month,date) > ?? or a datetime.date argument. > 3. Rewrite the existing function so that its first argument > ?? can be either an int (for year) or a datetime.date. The > ?? existing month and date arguments would be optional, with > ?? default=None. But, if the first argument is an int, and > ?? either of month or date is None, an error would be raised. > > The first would be the simplest. However, it is obviously WET > rather than DRY. > > The second isn't too bad, but a change like this would require that > I find all places that the function is currently used and insert a > pair of parentheses. Touching this much code is risky, as well > as being a bunch of work. (Admittedly, I'd only do it once.) > > The third is really klunky, but wouldn't need to touch anything > besides this function. > > What are others' thoughts? Which of the approaches above looks > least undesirable (and why)? Can anybody see a fourth approach? Reading the above, it seems that the options are limited to using positional-arguments only. Because I keep tripping-over my long, grey, beard; I'm reminded that relying upon my/human memory is, um, unreliable (at least in my case). Accordingly, by the time a function's definition reaches three parameters, I'll be converting it to use keyword-arguments as a matter of policy. YMMV! Remember: if keyword arguments are not used (ie existing/legacy code), Python will still use positional logic. Once the function's signature has been changed, we could then add another keyword-parameter to cover the datetime option. That said, a function which starts with a list of ifs-buts-and-maybes* which are only there to ascertain which set of arguments have been provided by the calling-routine; obscures the purpose/responsibility of the function and decreases its readability (perhaps not by much, but varying by situation). Accordingly, if the function is actually a method, recommend following @Stefan's approach, ie multiple-constructors. Although, this too can result in lower readability. Assuming it is a function, and that there are not many alternate APIs/approaches (here we're discussing only two), I'd probably create a wrapper-function which has the sole task of re-stating the datetime whilst calling the existing three-parameter function. The readability consideration here, is to make a good choice of (new) function-name! * Python version >= 10? Consider using match-case construct keyed on parameter-type -- Regards, =dn From 2QdxY4RzWzUUiLuE at potatochowder.com Sat May 14 19:34:23 2022 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Sat, 14 May 2022 18:34:23 -0500 Subject: Changing calling sequence In-Reply-To: <671ca838-f59e-4739-f795-4f328399ab9f@DancesWithMice.info> References: <671ca838-f59e-4739-f795-4f328399ab9f@DancesWithMice.info> Message-ID: On 2022-05-15 at 10:22:15 +1200, dn wrote: > That said, a function which starts with a list of ifs-buts-and-maybes* > which are only there to ascertain which set of arguments have been > provided by the calling-routine; obscures the purpose/responsibility > of the function and decreases its readability (perhaps not by much, > but varying by situation). Agreed. > Accordingly, if the function is actually a method, recommend following > @Stefan's approach, ie multiple-constructors. Although, this too can > result in lower readability. (Having proposed that approach myself (and having used it over the decades for functions, methods, procedures, constructors, ...), I also agree.) Assuming good names,? how can this lead to lower readability? I guess if there's too many of them, or programmers have to start wondering which one to use? Or is this in the same generally obfuscating category as the ifs-buts-and-maybes at the start of a function? ? and properly invalidated caches From drsalists at gmail.com Sat May 14 20:52:43 2022 From: drsalists at gmail.com (Dan Stromberg) Date: Sat, 14 May 2022 17:52:43 -0700 Subject: Mypy alternatives Message-ID: Hello people. I've used Mypy and liked it in combination with MonkeyType. I've heard there are alternatives to Mypy that are faster, and I'm looking at using something like this on a 457,000 line project. Are there equivalents to MonkeyType that will work with these alternatives to Mypy? And has Mypy become the defacto standard for how type annotations should look? That is, are there other tools that assume Mypy's format too, and does most doc about type annotations assume Mypy's style? Thanks! From PythonList at DancesWithMice.info Sun May 15 00:26:02 2022 From: PythonList at DancesWithMice.info (dn) Date: Sun, 15 May 2022 16:26:02 +1200 Subject: Changing calling sequence In-Reply-To: References: <671ca838-f59e-4739-f795-4f328399ab9f@DancesWithMice.info> Message-ID: On 15/05/2022 11.34, 2QdxY4RzWzUUiLuE at potatochowder.com wrote: > On 2022-05-15 at 10:22:15 +1200, > dn wrote: > >> That said, a function which starts with a list of ifs-buts-and-maybes* >> which are only there to ascertain which set of arguments have been >> provided by the calling-routine; obscures the purpose/responsibility >> of the function and decreases its readability (perhaps not by much, >> but varying by situation). > > Agreed. > >> Accordingly, if the function is actually a method, recommend following >> @Stefan's approach, ie multiple-constructors. Although, this too can >> result in lower readability. > > (Having proposed that approach myself (and having used it over the > decades for functions, methods, procedures, constructors, ...), I also > agree.) > > Assuming good names,? how can this lead to lower readability? I guess > if there's too many of them, or programmers have to start wondering > which one to use? Or is this in the same generally obfuscating category > as the ifs-buts-and-maybes at the start of a function? > > ? and properly invalidated caches Allow me to extend the term "readability" to include "comprehension". Then add the statistical expectation that a class has only __init__(). Thus, assuming this is the first time (or, ... for a while) that the class is being employed, one has to read much further to realise that there are choices of constructor. Borrowing from the earlier example: > This would be quite pythonic. For example, "datetime.date" > has .fromtimestamp(timestamp), .fromordinal(ordinal), > .fromisoformat(date_string), ... Please remember that this is only relevant if the function is actually a module - which sense does not appear from the OP (IMHO). The alternatives' names are well differentiated and (apparently#) appropriately named*. * PEP-008 hobgoblins will quote: "Function names should be lowercase, with words separated by underscores as necessary to improve readability. Variable names follow the same convention as function names." - but this is a common observation/criticism of code that has been in the PSL for a long time. # could also criticise as not following the Software Craftsmanship/Clean Code ideal of 'programming to the interface rather than the implementation' - which we see in PEP-008 as "usage rather than implementation" (but please don't ask me how to differentiate between them, given that the only reason for the different interfaces is the function's/parameters' implementation!) NB usual caveats apply to PEP-008 quotations! So, I agree with you - it comes down to those pernicious 'ifs-buts-and-maybes'. If the interface/parameter-processing starts to obfuscate the function's actual purpose, maybe it can be 'farmed-out' to a helper-function. However, that would start to look very much like the same effort (and comprehension-challenge) as having a wrapper-function! Continuing the 'have to read further' criticism (above), it could equally-well be applied to my preference for keyword-arguments, in that I've suggested defining four parameters but the user will only call the function with either three or one argument(s). Could this be described as potentially-confusing? Given that the OP wouldn't want to have to redefine the existing interface, the next comment may not be applicable - but in the interests of completeness: anyone contemplating such architecture might like to consider "Single-dispatch generic functions" (https://peps.python.org/pep-0443/). At least the decorators signal that there are alternative-choices... -- Regards, =dn From rosuav at gmail.com Sun May 15 00:44:09 2022 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 15 May 2022 14:44:09 +1000 Subject: Changing calling sequence In-Reply-To: References: <671ca838-f59e-4739-f795-4f328399ab9f@DancesWithMice.info> Message-ID: On Sun, 15 May 2022 at 14:27, dn wrote: > > On 15/05/2022 11.34, 2QdxY4RzWzUUiLuE at potatochowder.com wrote: > > On 2022-05-15 at 10:22:15 +1200, > > dn wrote: > > > >> That said, a function which starts with a list of ifs-buts-and-maybes* > >> which are only there to ascertain which set of arguments have been > >> provided by the calling-routine; obscures the purpose/responsibility > >> of the function and decreases its readability (perhaps not by much, > >> but varying by situation). > > > > Agreed. > > > >> Accordingly, if the function is actually a method, recommend following > >> @Stefan's approach, ie multiple-constructors. Although, this too can > >> result in lower readability. > > > > (Having proposed that approach myself (and having used it over the > > decades for functions, methods, procedures, constructors, ...), I also > > agree.) > > > > Assuming good names,? how can this lead to lower readability? I guess > > if there's too many of them, or programmers have to start wondering > > which one to use? Or is this in the same generally obfuscating category > > as the ifs-buts-and-maybes at the start of a function? > > > > ? and properly invalidated caches > > Allow me to extend the term "readability" to include "comprehension". > Then add the statistical expectation that a class has only __init__(). (Confusing wording here: a class usually has far more than just __init__, but I presume you mean that the signature of __init__ is the only way to construct an object of that type.) > Thus, assuming this is the first time (or, ... for a while) that the > class is being employed, one has to read much further to realise that > there are choices of constructor. Yeah. I would generally say, though, that any classmethod should be looked at as a potential alternate constructor, or at least an alternate way to obtain objects (eg preconstructed objects with commonly-used configuration - imagine a SecuritySettings class with a classmethod to get different defaults). > Borrowing from the earlier example: > > > This would be quite pythonic. For example, "datetime.date" > > has .fromtimestamp(timestamp), .fromordinal(ordinal), > > .fromisoformat(date_string), ... > > Please remember that this is only relevant if the function is actually a > module - which sense does not appear from the OP (IMHO). > > The alternatives' names are well differentiated and (apparently#) > appropriately named*. > > > * PEP-008 hobgoblins will quote: > "Function names should be lowercase, with words separated by underscores > as necessary to improve readability. Note the "as necessary". Underscores aren't required when readability is fine without them (see for instance PEP 616, which recently added two methods to strings "removeprefix" and "removesuffix", no underscores - part of the argument here was consistency with other string methods, but it's also not a major problem for readability here). > Variable names follow the same convention as function names." > - but this is a common observation/criticism of code that has been in > the PSL for a long time. > > # could also criticise as not following the Software Craftsmanship/Clean > Code ideal of 'programming to the interface rather than the > implementation' - which we see in PEP-008 as "usage rather than > implementation" > (but please don't ask me how to differentiate between them, given that > the only reason for the different interfaces is the > function's/parameters' implementation!) > > NB usual caveats apply to PEP-008 quotations! Notably here, the caveat that PEP 8 is not a permanent and unchanging document. It is advice, not rules, and not all code in the standard library fully complies with its current recommendations. > Continuing the 'have to read further' criticism (above), it could > equally-well be applied to my preference for keyword-arguments, in that > I've suggested defining four parameters but the user will only call the > function with either three or one argument(s). Could this be described > as potentially-confusing? Yes, definitely. Personally, I'd split it into two, one that takes the existing three arguments (preferably with the same name, for compatibility), and one with a different name that takes just the one arg. That could be a small wrapper that calls the original, or the original could become a wrapper that calls the new one, or the main body could be refactored into a helper that they both call. It all depends what makes the most sense internally, because that's not part of the API at that point. But it does depend on how the callers operate. Sometimes it's easier to have a single function with switchable argument forms, other times it's cleaner to separate them. ChrisA From 2QdxY4RzWzUUiLuE at potatochowder.com Sun May 15 09:20:34 2022 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Sun, 15 May 2022 08:20:34 -0500 Subject: Changing calling sequence In-Reply-To: References: <671ca838-f59e-4739-f795-4f328399ab9f@DancesWithMice.info> Message-ID: On 2022-05-15 at 14:44:09 +1000, Chris Angelico wrote: > On Sun, 15 May 2022 at 14:27, dn wrote: > > > > On 15/05/2022 11.34, 2QdxY4RzWzUUiLuE at potatochowder.com wrote: > > > On 2022-05-15 at 10:22:15 +1200, > > > dn wrote: > > > > > >> That said, a function which starts with a list of ifs-buts-and-maybes* > > >> which are only there to ascertain which set of arguments have been > > >> provided by the calling-routine; obscures the purpose/responsibility > > >> of the function and decreases its readability (perhaps not by much, > > >> but varying by situation). > > > > > > Agreed. > > > > > >> Accordingly, if the function is actually a method, recommend following > > >> @Stefan's approach, ie multiple-constructors. Although, this too can > > >> result in lower readability. > > > > > > (Having proposed that approach myself (and having used it over the > > > decades for functions, methods, procedures, constructors, ...), I also > > > agree.) > > > > > > Assuming good names,? how can this lead to lower readability? I guess > > > if there's too many of them, or programmers have to start wondering > > > which one to use? Or is this in the same generally obfuscating category > > > as the ifs-buts-and-maybes at the start of a function? > > > > > > ? and properly invalidated caches > > > > Allow me to extend the term "readability" to include "comprehension". > > Then add the statistical expectation that a class has only __init__(). Aha. In that light, yeah, in geeral, the more stuff there is, the harder it is to get your head around it. And even if I document the class (or the module), no one makes the time to read (let alone comprehend) the document, which *should* clarify all those things that are hard to discern from the code itself. > > Thus, assuming this is the first time (or, ... for a while) that the > > class is being employed, one has to read much further to realise that > > there are choices of constructor. > > Yeah. I would generally say, though, that any classmethod should be > looked at as a potential alternate constructor, or at least an > alternate way to obtain objects (eg preconstructed objects with > commonly-used configuration - imagine a SecuritySettings class with a > classmethod to get different defaults). I think opening up the class and sifting through its classmethods to find the factory functions is what dn is talking about. Such a design also means that once I have a SecuritySettings object, its (the instance's) methods include both instance and class level methods. IMO, classmethods were/are a bad idea (yes, I'm probably in the minority around here, but someone has to be). The first person to scream "but discoverability" will be severely beaten with a soft cushion. > > Borrowing from the earlier example: > > > > > This would be quite pythonic. For example, "datetime.date" > > > has .fromtimestamp(timestamp), .fromordinal(ordinal), > > > .fromisoformat(date_string), ... > > > > Please remember that this is only relevant if the function is actually a > > module - which sense does not appear from the OP (IMHO). Note that datetime.date is a class, not a module. > > The alternatives' names are well differentiated and (apparently#) > > appropriately named*. [...] > > Continuing the 'have to read further' criticism (above), it could > > equally-well be applied to my preference for keyword-arguments, in that > > I've suggested defining four parameters but the user will only call the > > function with either three or one argument(s). Could this be described > > as potentially-confusing? Potentially. :-) In a well designed *library*, common keywords across multiple functions provide consistency, which is generally good. Even a bit of redundancy can be good for the same reason. OTOH, when there's only one function, and it has a pile of keyword parameters that can only be used in certain combinations, then it definitely can be harder to read/understand/use than separate functions with simpler interfaces. > Yes, definitely. Personally, I'd split it into two, one that takes the > existing three arguments (preferably with the same name, for > compatibility), and one with a different name that takes just the one > arg. That could be a small wrapper that calls the original, or the > original could become a wrapper that calls the new one, or the main > body could be refactored into a helper that they both call. It all > depends what makes the most sense internally, because that's not part > of the API at that point. > > But it does depend on how the callers operate. Sometimes it's easier > to have a single function with switchable argument forms, other times > it's cleaner to separate them. "Easier" and "cleaner" are very often orthogonal. ;-) (Rich Hickey (creator of Clojure) talks a lot about the difference between "easy" and "simple." Arguemnts for and against Unix often involve similar terms.) And "easier" or "cleaner" for whom? The person writing the function(s), the person writing code that calls the function(s), the person reading the function(s), the person reading code that calls the function(s), all or any of that code? Testers? Maintainers? Their code? Sometimes, what's easier and cleaner for one is harder and dirtier for another. As one whose beard likely resembles dn's, I can often spot code written by developers who have never been involved in testing or maintenance, or code written by testers who have never written production code (and don't/won't realize or admit that some tests *are* production code). From rob.cliffe at btinternet.com Sun May 15 23:01:10 2022 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 16 May 2022 04:01:10 +0100 Subject: Non-deterministic set ordering Message-ID: I was shocked to discover that when repeatedly running the following program (condensed from a "real" program) under Python 3.8.3 for p in { ('x','y'), ('y','x') }: ??? print(p) the output was sometimes ('y', 'x') ('x', 'y') and sometimes ('x', 'y') ('y', 'x') Can anyone explain why running identical code should result in traversing a set in a different order? Thanks Rob Cliffe From greg.ewing at canterbury.ac.nz Sun May 15 18:18:58 2022 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 16 May 2022 10:18:58 +1200 Subject: Changing calling sequence In-Reply-To: References: <671ca838-f59e-4739-f795-4f328399ab9f@DancesWithMice.info> Message-ID: On 16/05/22 1:20 am, 2QdxY4RzWzUUiLuE at potatochowder.com wrote: > IMO, > classmethods were/are a bad idea (yes, I'm probably in the minority > around here, but someone has to be). I don't think class methods are a bad idea per se, but having them visible through instances seems unnecessary and confusing. I suspect that wasn't a deliberate design decision, but just a side effect of using a single class dict for both class and instance things. -- Greg From drsalists at gmail.com Sun May 15 23:13:07 2022 From: drsalists at gmail.com (Dan Stromberg) Date: Sun, 15 May 2022 20:13:07 -0700 Subject: Non-deterministic set ordering In-Reply-To: References: Message-ID: On Sun, May 15, 2022 at 8:01 PM Rob Cliffe via Python-list < python-list at python.org> wrote: > I was shocked to discover that when repeatedly running the following > program (condensed from a "real" program) under Python 3.8.3 > > for p in { ('x','y'), ('y','x') }: > print(p) > > the output was sometimes > > ('y', 'x') > ('x', 'y') > > and sometimes > > ('x', 'y') > ('y', 'x') > > Can anyone explain why running identical code should result in > traversing a set in a different order? > Sets are defined as unordered so that they can be hashed internally to give O(1) operations for many tasks. It wouldn't be unreasonable for sets to use a fixed-by-arbitrary ordering for a given group of set operations, but being unpredictable deters developers from mistakenly assuming they are ordered. If you need order, you should use a tuple, list, or something like https://grantjenks.com/docs/sortedcontainers/sortedset.html From rob.cliffe at btinternet.com Sun May 15 23:20:50 2022 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 16 May 2022 04:20:50 +0100 Subject: Non-deterministic set ordering In-Reply-To: References: Message-ID: On 16/05/2022 04:13, Dan Stromberg wrote: > > On Sun, May 15, 2022 at 8:01 PM Rob Cliffe via Python-list > wrote: > > I was shocked to discover that when repeatedly running the following > program (condensed from a "real" program) under Python 3.8.3 > > for p in { ('x','y'), ('y','x') }: > ???? print(p) > > the output was sometimes > > ('y', 'x') > ('x', 'y') > > and sometimes > > ('x', 'y') > ('y', 'x') > > Can anyone explain why running identical code should result in > traversing a set in a different order? > > > Sets are defined as unordered so that they can be hashed internally to > give O(1) operations for many tasks. > > It wouldn't be unreasonable for sets to use a fixed-by-arbitrary > ordering for a given group of set operations, but being unpredictable > deters developers from mistakenly assuming they are ordered. > > If you need order, you should use a tuple, list, or something like > https://grantjenks.com/docs/sortedcontainers/sortedset.html Thanks, I can work round this behaviour. But I'm curious: where does the variability come from?? Is it deliberate (as your answer seems to imply)?? AFAIK the same code within the *same run* of a program does produce identical results. Best wishes Rob Cliffe From pbryan at anode.ca Sun May 15 23:36:27 2022 From: pbryan at anode.ca (Paul Bryan) Date: Sun, 15 May 2022 20:36:27 -0700 Subject: Non-deterministic set ordering In-Reply-To: References: Message-ID: <8c947b633c736874f99a09320d0b44c4af10d686.camel@anode.ca> This may explain it: https://stackoverflow.com/questions/27522626/hash-function-in-python-3-3-returns-different-results-between-sessions On Mon, 2022-05-16 at 04:20 +0100, Rob Cliffe via Python-list wrote: > > > On 16/05/2022 04:13, Dan Stromberg wrote: > > > > On Sun, May 15, 2022 at 8:01 PM Rob Cliffe via Python-list > > wrote: > > > > ??? I was shocked to discover that when repeatedly running the > > following > > ??? program (condensed from a "real" program) under Python 3.8.3 > > > > ??? for p in { ('x','y'), ('y','x') }: > > ??? ???? print(p) > > > > ??? the output was sometimes > > > > ??? ('y', 'x') > > ??? ('x', 'y') > > > > ??? and sometimes > > > > ??? ('x', 'y') > > ??? ('y', 'x') > > > > ??? Can anyone explain why running identical code should result in > > ??? traversing a set in a different order? > > > > > > Sets are defined as unordered so that they can be hashed internally > > to > > give O(1) operations for many tasks. > > > > It wouldn't be unreasonable for sets to use a fixed-by-arbitrary > > ordering for a given group of set operations, but being > > unpredictable > > deters developers from mistakenly assuming they are ordered. > > > > If you need order, you should use a tuple, list, or something like > > https://grantjenks.com/docs/sortedcontainers/sortedset.html > Thanks, I can work round this behaviour. > But I'm curious: where does the variability come from?? Is it > deliberate > (as your answer seems to imply)?? AFAIK the same code within the > *same > run* of a program does produce identical results. > Best wishes > Rob Cliffe From rob.cliffe at btinternet.com Sun May 15 23:40:54 2022 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 16 May 2022 04:40:54 +0100 Subject: Non-deterministic set ordering In-Reply-To: <8c947b633c736874f99a09320d0b44c4af10d686.camel@anode.ca> References: <8c947b633c736874f99a09320d0b44c4af10d686.camel@anode.ca> Message-ID: <6728f650-79b3-8a9a-f001-38a0b54dd65c@btinternet.com> Thanks, Paul.? Question answered! Rob Cliffe On 16/05/2022 04:36, Paul Bryan wrote: > This may explain it: > https://stackoverflow.com/questions/27522626/hash-function-in-python-3-3-returns-different-results-between-sessions > > On Mon, 2022-05-16 at 04:20 +0100, Rob Cliffe via Python-list wrote: >> >> >> On 16/05/2022 04:13, Dan Stromberg wrote: >>> >>> On Sun, May 15, 2022 at 8:01 PM Rob Cliffe via Python-list >>> wrote: >>> >>> ??? I was shocked to discover that when repeatedly running the following >>> ??? program (condensed from a "real" program) under Python 3.8.3 >>> >>> ??? for p in { ('x','y'), ('y','x') }: >>> ??? ???? print(p) >>> >>> ??? the output was sometimes >>> >>> ??? ('y', 'x') >>> ??? ('x', 'y') >>> >>> ??? and sometimes >>> >>> ??? ('x', 'y') >>> ??? ('y', 'x') >>> >>> ??? Can anyone explain why running identical code should result in >>> ??? traversing a set in a different order? >>> >>> >>> Sets are defined as unordered so that they can be hashed internally to >>> give O(1) operations for many tasks. >>> >>> It wouldn't be unreasonable for sets to use a fixed-by-arbitrary >>> ordering for a given group of set operations, but being unpredictable >>> deters developers from mistakenly assuming they are ordered. >>> >>> If you need order, you should use a tuple, list, or something like >>> https://grantjenks.com/docs/sortedcontainers/sortedset.html >> Thanks, I can work round this behaviour. >> But I'm curious: where does the variability come from?? Is it deliberate >> (as your answer seems to imply)?? AFAIK the same code within the *same >> run* of a program does produce identical results. >> Best wishes >> Rob Cliffe > From python at mrabarnett.plus.com Sun May 15 23:40:17 2022 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 16 May 2022 04:40:17 +0100 Subject: Non-deterministic set ordering In-Reply-To: References: Message-ID: <06218b48-ecaa-24fc-3a60-5f29b81ffaff@mrabarnett.plus.com> On 2022-05-16 04:20, Rob Cliffe via Python-list wrote: > > > On 16/05/2022 04:13, Dan Stromberg wrote: >> >> On Sun, May 15, 2022 at 8:01 PM Rob Cliffe via Python-list >> wrote: >> >> I was shocked to discover that when repeatedly running the following >> program (condensed from a "real" program) under Python 3.8.3 >> >> for p in { ('x','y'), ('y','x') }: >> ???? print(p) >> >> the output was sometimes >> >> ('y', 'x') >> ('x', 'y') >> >> and sometimes >> >> ('x', 'y') >> ('y', 'x') >> >> Can anyone explain why running identical code should result in >> traversing a set in a different order? >> >> >> Sets are defined as unordered so that they can be hashed internally to >> give O(1) operations for many tasks. >> >> It wouldn't be unreasonable for sets to use a fixed-by-arbitrary >> ordering for a given group of set operations, but being unpredictable >> deters developers from mistakenly assuming they are ordered. >> >> If you need order, you should use a tuple, list, or something like >> https://grantjenks.com/docs/sortedcontainers/sortedset.html > Thanks, I can work round this behaviour. > But I'm curious: where does the variability come from?? Is it deliberate > (as your answer seems to imply)?? AFAIK the same code within the *same > run* of a program does produce identical results. Basically, Python uses hash randomisation in order to protect it against denial-of-service attacks. (Search for "PYTHONHASHSEED" in the docs.) It also applied to dicts (the code for sets was based on that for dicts), but dicts now remember their insertion order. From __peter__ at web.de Mon May 16 06:05:14 2022 From: __peter__ at web.de (Peter Otten) Date: Mon, 16 May 2022 12:05:14 +0200 Subject: EAFP In-Reply-To: References: Message-ID: <173a2a9d-962f-0606-4750-562f9984118c@web.de> On 13/05/2022 18:37, bryangan41 wrote: > Is the following LBYL:foo = 123if foo < 200:? ? do()If so, how to change to EAFP?Thanks!Sent from Samsung tablet. The distinction between look-before-you-leap and easier-to-ask-forgiveness-than-permission is weaker than yo might expect. When you write filename = ... if exists(filename): with open(filename) as instream: # do stuff else: # fallback there are two checks for the file's existence, one explicit, and one implicitly inside open() -- and worse, the first, explicit, check is unreliable because between exists() and open() there is a small delay that may be sufficient to create or delete the file. Therefore the recommended (EAFP) version of the above is filename = ... try: with open(filename) as instrem: # do stuff except FileNotFoundError # fallback or just with open(filename) as instream: # do stuff if there is no meaningful fallback. Regarding your example code, whether you can write an EAFP version for if foo < 200: do() depends on the contents of do(). If do() fails in a well-defined way, let's say by raising a FooAbove199 exception you can write just do() or, if you need a fallback try: do() except FooAbove199: # fallback Note that if you change do() from do(): # do stuff to def do(): if foo < 200: # do stuff else: raise FooAbove199(f"Invalid {foo=!r}") the do() invocation becomes EAFP even though you are actually performing the same test as before. From philippe.descharmes at sfr.fr Mon May 16 02:50:04 2022 From: philippe.descharmes at sfr.fr (philippe descharmes) Date: Mon, 16 May 2022 08:50:04 +0200 (CEST) Subject: Python Message-ID: <1904011270.13301.1652683804449@wsfrf1452.priv.atos.fr> ? ? ?Hello,? in coding, may be is a question of transivity in this code line. Thank you. Philippe From Marco.Sulla.Python at gmail.com Mon May 16 14:13:47 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Mon, 16 May 2022 20:13:47 +0200 Subject: tail In-Reply-To: References: Message-ID: On Fri, 13 May 2022 at 12:49, <2QdxY4RzWzUUiLuE at potatochowder.com> wrote: > > On 2022-05-13 at 12:16:57 +0200, > Marco Sulla wrote: > > > On Fri, 13 May 2022 at 00:31, Cameron Simpson wrote: > > [...] > > > > This is nearly the worst "specification" I have ever seen. > > > You're lucky. I've seen much worse (or no one). > > At least with *no* documentation, the source code stands for itself. So I did it well to not put one in the first time. I think that after 100 posts about tail, chunks etc it was clear what that stuff was about and how to use it. Speaking about more serious things, so far I've done a test with: * a file that does not end with \n * a file that ends with \n (after Stefan test) * a file with more than 10 lines * a file with less than 10 lines It seemed to work. I've only to benchmark it. I suppose I have to test with at least 1 GB file, a big lorem ipsum, and do an unequal comparison with Linux tail. I'll do it when I have time, so Chris will be no more angry with me. From wlfraed at ix.netcom.com Mon May 16 11:27:34 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Mon, 16 May 2022 11:27:34 -0400 Subject: Convert the decimal numbers expressed in a `numpy.ndarray` into a matrix representing elements in fractional form References: <5eb0b494-ec7a-4b6c-9966-078889dbc5aen@googlegroups.com> Message-ID: <54r48hdpj8jov1on1f9lfi4t6vl08lk6md@4ax.com> On Mon, 16 May 2022 02:03:26 -0700 (PDT), "hongy... at gmail.com" declaimed the following: >print(lst) Printing higher level structures uses the repr() of the structure and its contents -- theoretically a form that could be used within code as a literal. If you want human-readable str() you will need to write your own output loop to do the formatting of the structure, and explicitly print each item of the structure. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From hongyi.zhao at gmail.com Mon May 16 19:11:13 2022 From: hongyi.zhao at gmail.com (hongy...@gmail.com) Date: Mon, 16 May 2022 16:11:13 -0700 (PDT) Subject: Convert the decimal numbers expressed in a `numpy.ndarray` into a matrix representing elements in fractional form In-Reply-To: <54r48hdpj8jov1on1f9lfi4t6vl08lk6md@4ax.com> References: <5eb0b494-ec7a-4b6c-9966-078889dbc5aen@googlegroups.com> <54r48hdpj8jov1on1f9lfi4t6vl08lk6md@4ax.com> Message-ID: On Monday, May 16, 2022 at 11:27:58 PM UTC+8, Dennis Lee Bieber wrote: > On Mon, 16 May 2022 02:03:26 -0700 (PDT), "hongy... at gmail.com" > declaimed the following: > > > >print(lst) > > Printing higher level structures uses the repr() of the structure and > its contents -- theoretically a form that could be used within code as a > literal. If you want human-readable str() you will need to write your own > output loop to do the formatting of the structure, and explicitly print > each item of the structure. Thank you for your explanation. I have come up with the following methods: ``` b=[[0.0, -1.0, 0.0, 0.25], [1.0, 0.0, 0.0, 0.25], [0.0, 0.0, 1.0, 0.25], [0.0, 0.0, 0.0, 1.0]] import numpy as np from fractions import Fraction import re def strmat(m): if(np.array([m]).ndim==1): return str(Fraction(m)) else: return list(map(lambda L:strmat(L), np.array(m))) a=str(strmat(b)) a=re.sub(r"'","",a) repr(a) print(repr(a)) '[[0, -1, 0, 1/4], [1, 0, 0, 1/4], [0, 0, 1, 1/4], [0, 0, 0, 1]]' ``` Best, HZ From hongyi.zhao at gmail.com Mon May 16 19:31:07 2022 From: hongyi.zhao at gmail.com (hongy...@gmail.com) Date: Mon, 16 May 2022 16:31:07 -0700 (PDT) Subject: Convert the decimal numbers expressed in a `numpy.ndarray` into a matrix representing elements in fractional form In-Reply-To: References: <5eb0b494-ec7a-4b6c-9966-078889dbc5aen@googlegroups.com> <54r48hdpj8jov1on1f9lfi4t6vl08lk6md@4ax.com> Message-ID: <20220b6d-504f-4b55-ba15-609bafc71af3n@googlegroups.com> On Tuesday, May 17, 2022 at 7:11:24 AM UTC+8, hongy... at gmail.com wrote: > On Monday, May 16, 2022 at 11:27:58 PM UTC+8, Dennis Lee Bieber wrote: > > On Mon, 16 May 2022 02:03:26 -0700 (PDT), "hongy... at gmail.com" > > declaimed the following: > > > > > > >print(lst) > > > > Printing higher level structures uses the repr() of the structure and > > its contents -- theoretically a form that could be used within code as a > > literal. If you want human-readable str() you will need to write your own > > output loop to do the formatting of the structure, and explicitly print > > each item of the structure. > Thank you for your explanation. I have come up with the following methods: > ``` > b=[[0.0, -1.0, 0.0, 0.25], [1.0, 0.0, 0.0, 0.25], [0.0, 0.0, 1.0, 0.25], [0.0, 0.0, 0.0, 1.0]] > import numpy as np > from fractions import Fraction > import re > > def strmat(m): > if(np.array([m]).ndim==1): > return str(Fraction(m)) > else: return list(map(lambda L:strmat(L), np.array(m))) > > a=str(strmat(b)) > a=re.sub(r"'","",a) > repr(a) > print(repr(a)) > '[[0, -1, 0, 1/4], [1, 0, 0, 1/4], [0, 0, 1, 1/4], [0, 0, 0, 1]]' > ``` > Best, > HZ See here [1] for the related discussion. [1] https://discuss.python.org/t/convert-the-decimal-numbers-expressed-in-a-numpy-ndarray-into-a-matrix-representing-elements-in-fractional-form/15780 From hongyi.zhao at gmail.com Mon May 16 20:22:17 2022 From: hongyi.zhao at gmail.com (hongy...@gmail.com) Date: Mon, 16 May 2022 17:22:17 -0700 (PDT) Subject: Convert the decimal numbers expressed in a `numpy.ndarray` into a matrix representing elements in fractional form In-Reply-To: <54r48hdpj8jov1on1f9lfi4t6vl08lk6md@4ax.com> References: <5eb0b494-ec7a-4b6c-9966-078889dbc5aen@googlegroups.com> <54r48hdpj8jov1on1f9lfi4t6vl08lk6md@4ax.com> Message-ID: On Monday, May 16, 2022 at 11:27:58 PM UTC+8, Dennis Lee Bieber wrote: > On Mon, 16 May 2022 02:03:26 -0700 (PDT), "hongy... at gmail.com" > declaimed the following: > > > >print(lst) > > Printing higher level structures uses the repr() of the structure and > its contents -- theoretically a form that could be used within code as a > literal. I tried with the repr() method as follows, but it doesn't give any output: ``` import os,sys import numpy as np from fractions import Fraction import re from pymatgen.symmetry.analyzer import SpacegroupAnalyzer from pymatgen.core import Lattice, Structure, Molecule, IStructure def filepath(file): script_dirname=os.path.dirname(os.path.realpath(__file__)) return (script_dirname + '/' + file) s=IStructure.from_file(filepath('EntryWithCollCode136212.cif')) a = SpacegroupAnalyzer(s) SymOp=a.get_symmetry_operations() b=SymOp[1].affine_matrix.tolist() def strmat(m): if(np.array([m]).ndim==1): return str(Fraction(m)) else: return list(map(lambda L:strmat(L), np.array(m))) lst=[] for i in SymOp: lst.append(i.affine_matrix.tolist()) a=str(strmat(lst)) a=re.sub(r"'","",a) repr(a) ``` > If you want human-readable str() you will need to write your own > output loop to do the formatting of the structure, and explicitly print > each item of the structure. From wlfraed at ix.netcom.com Mon May 16 20:48:02 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Mon, 16 May 2022 20:48:02 -0400 Subject: Convert the decimal numbers expressed in a `numpy.ndarray` into a matrix representing elements in fractional form References: <5eb0b494-ec7a-4b6c-9966-078889dbc5aen@googlegroups.com> <54r48hdpj8jov1on1f9lfi4t6vl08lk6md@4ax.com> Message-ID: <27r58hhbe2ldc4f0h9tmci3rallg6cc3va@4ax.com> On Mon, 16 May 2022 17:22:17 -0700 (PDT), "hongy... at gmail.com" declaimed the following: > >I tried with the repr() method as follows, but it doesn't give any output: I have no idea what 50% of those libraries are supposed to do, and am not going to install them just to try out your posted code. If you really want such help, post the MINIMUM example code the produces your problem. >a=str(strmat(lst)) >a=re.sub(r"'","",a) Explain what you believe this operation is doing, show us the input and the output. The best I can make out of that is that it is looking for single quote characters within whatever "a" is, and replacing them with nothing. Something much more understandable, without invoking a regular expression library (especially when neither the search nor the replacement terms are regular expressions) with simple string operations... stripped = "".join(quoted.split("'")) You also don't need to specify RAW format for the "'" -- Python is quite happy mixing single and double quotes (that is: single quotes inside a string using double quotes, double quotes inside a string using single quotes, either inside strings using triply quoted delimiters) >>> "'" "'" >>> '"' '"' >>> """'"'""" '\'"\'' >>> '''"'"''' '"\'"' >>> (Note that the interactive console displays results using repr(), and hence escapes ' that are internal to avoid conflict with the ones wrapping the output) >>> repr('''"'"''') '\'"\\\'"\'' >>> str('''"'"''') '"\'"' >>> print('''"'"''') "'" >>> The print() operation does not wrap the output with extraneous quotes. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From hongyi.zhao at gmail.com Mon May 16 22:05:14 2022 From: hongyi.zhao at gmail.com (hongy...@gmail.com) Date: Mon, 16 May 2022 19:05:14 -0700 (PDT) Subject: Convert the decimal numbers expressed in a `numpy.ndarray` into a matrix representing elements in fractional form In-Reply-To: <27r58hhbe2ldc4f0h9tmci3rallg6cc3va@4ax.com> References: <5eb0b494-ec7a-4b6c-9966-078889dbc5aen@googlegroups.com> <54r48hdpj8jov1on1f9lfi4t6vl08lk6md@4ax.com> <27r58hhbe2ldc4f0h9tmci3rallg6cc3va@4ax.com> Message-ID: <710536ee-95b0-41b7-acd6-bd2d5a5eed01n@googlegroups.com> On Tuesday, May 17, 2022 at 8:48:27 AM UTC+8, Dennis Lee Bieber wrote: > On Mon, 16 May 2022 17:22:17 -0700 (PDT), "hongy... at gmail.com" > declaimed the following: > > > > > >I tried with the repr() method as follows, but it doesn't give any output: > I have no idea what 50% of those libraries are supposed to do, and am > not going to install them just to try out your posted code. If you really > want such help, post the MINIMUM example code the produces your problem. > > >a=str(strmat(lst)) > >a=re.sub(r"'","",a) > > Explain what you believe this operation is doing, show us the input and > the output. > > The best I can make out of that is that it is looking for single quote > characters within whatever "a" is, and replacing them with nothing. > Something much more understandable, without invoking a regular expression > library (especially when neither the search nor the replacement terms are > regular expressions) with simple string operations... > > stripped = "".join(quoted.split("'")) Thank you for your above trick. I tried with the following code snippet: ``` from fractions import Fraction def strmat(m): if(np.array([m]).ndim==1): return str(Fraction(m)) else: return list(map(lambda L:strmat(L), np.array(m))) # For test: b=[[0.0, -1.0, 0.0, 0.25], [1.0, 0.0, 0.0, 0.25], [0.0, 0.0, 1.0, 0.25], [0.0, 0.0, 0.0, 1.0]] a=str(strmat(b)) a1=stripped = "".join(a.split("'")) a=re.sub(r"'","",a) #repr(a) print("a1 = "+ a1) print("a = "+ a) ``` As you can see, both methods give the same results: ``` a1 = [[0, -1, 0, 1/4], [1, 0, 0, 1/4], [0, 0, 1, 1/4], [0, 0, 0, 1]] a = [[0, -1, 0, 1/4], [1, 0, 0, 1/4], [0, 0, 1, 1/4], [0, 0, 0, 1]] ``` > You also don't need to specify RAW format for the "'" -- Python is quite > happy mixing single and double quotes (that is: single quotes inside a > string using double quotes, double quotes inside a string using single > quotes, either inside strings using triply quoted delimiters) > > >>> "'" > "'" > >>> '"' > '"' > >>> """'"'""" > '\'"\'' > >>> '''"'"''' > '"\'"' > >>> > > (Note that the interactive console displays results using repr(), and hence > escapes ' that are internal to avoid conflict with the ones wrapping the > output) > > >>> repr('''"'"''') > '\'"\\\'"\'' > >>> str('''"'"''') > '"\'"' > >>> print('''"'"''') > "'" > >>> > > The print() operation does not wrap the output with extraneous quotes. Thank your insightful explanation. Regards, HZ > -- > Wulfraed Dennis Lee Bieber AF6VN > wlf... at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From barry at barrys-emacs.org Tue May 17 02:42:11 2022 From: barry at barrys-emacs.org (Barry) Date: Tue, 17 May 2022 07:42:11 +0100 Subject: Convert the decimal numbers expressed in a `numpy.ndarray` into a matrix representing elements in fractional form In-Reply-To: References: Message-ID: <9E85CF38-9A50-42DC-80B5-9EA08673AF3B@barrys-emacs.org> > On 17 May 2022, at 05:59, hongy... at gmail.com wrote: > > ?On Monday, May 16, 2022 at 11:27:58 PM UTC+8, Dennis Lee Bieber wrote: >> On Mon, 16 May 2022 02:03:26 -0700 (PDT), "hongy... at gmail.com" >> declaimed the following: >> >> >>> print(lst) >> >> Printing higher level structures uses the repr() of the structure and >> its contents -- theoretically a form that could be used within code as a >> literal. > > I tried with the repr() method as follows, but it doesn't give any output: Repr returns a string. You need to print its value to see it. > > ``` > import os,sys > import numpy as np > from fractions import Fraction > import re > from pymatgen.symmetry.analyzer import SpacegroupAnalyzer > from pymatgen.core import Lattice, Structure, Molecule, IStructure > > def filepath(file): > script_dirname=os.path.dirname(os.path.realpath(__file__)) > return (script_dirname + '/' + file) > > s=IStructure.from_file(filepath('EntryWithCollCode136212.cif')) > a = SpacegroupAnalyzer(s) > SymOp=a.get_symmetry_operations() > b=SymOp[1].affine_matrix.tolist() > > def strmat(m): > if(np.array([m]).ndim==1): > return str(Fraction(m)) > else: return list(map(lambda L:strmat(L), np.array(m))) > > lst=[] > for i in SymOp: > lst.append(i.affine_matrix.tolist()) > > a=str(strmat(lst)) > a=re.sub(r"'","",a) > repr(a) print(repr(a)) Barry > ``` > >> If you want human-readable str() you will need to write your own >> output loop to do the formatting of the structure, and explicitly print >> each item of the structure. > > > -- > https://mail.python.org/mailman/listinfo/python-list > From jonathan.kaczynski at guildeducation.com Tue May 17 07:15:24 2022 From: jonathan.kaczynski at guildeducation.com (Jonathan Kaczynski) Date: Tue, 17 May 2022 07:15:24 -0400 Subject: Seeking deeper understanding of python equality (==) In-Reply-To: References: Message-ID: Thank you for your response Eryk. I did try using gdb in my original post, but using breakpoint() left me in the python layer (pdb), not the cpython layer (gdb) and I couldn't figure out how to drop down. I see you're using the kernel32 library, so I assume you're on windows. I only have a mac and getting gdb functional on a mac seems to be onerous ( https://gist.github.com/mike-myers-tob/9a6013124bad7ff074d3297db2c98247), so I'm performing my testing on a linux container running ubuntu. The closest equivalent to putting kernel32.DebugBreak() in front of the x == y line in my test script is os.kill(os.getpid(), signal.SIGTRAP). Though, this drops me into the signal handling code in the cpython layer, and after several dozen steps I did not reach the richcompare-related functions. I'll make another attempt later today. Thanks to everyone's responses, I have a great idea of what's going on, now. I appreciate you all. My goal now is to be able to work with the debugger, like Erik is, so that next time I am able to perform this investigation in-full. Should I create a new thread for this question? Thank you, Jonathan On Sat, May 14, 2022 at 1:51 PM Eryk Sun wrote: > On 5/14/22, Jonathan Kaczynski > wrote: > > > > So, I'm still wondering how Py_TYPE(v)->tp_richcompare resolves to __eq__ > > on a user-defined class. Conversely, my understanding is, for a type > > defined in cpython, like str, there is usually an explicitly > > defined tp_richcompare function. > > Sometimes it's simplest to directly examine an object using a native > debugger (e.g. gdb in Linux; cdb/windbg in Windows). > > With a debugger attached to the interpreter, create two classes, one > that doesn't override __eq__() and one that does: > > >>> class C: > ... pass > ... > >>> class D: > ... __eq__ = lambda s, o: False > ... > > In CPython, the id() of an object is its address in memory: > > >>> hex(id(C)) > '0x2806a705790' > >>> hex(id(D)) > '0x2806a6bbfe0' > > Break into the attached debugger to examine the class objects: > > >>> kernel32.DebugBreak() > > (1968.1958): Break instruction exception - code 80000003 (first chance) > KERNELBASE!wil::details::DebugBreak+0x2: > 00007ffd`8818fd12 cc int 3 > > Class C uses the default object_richcompare(): > > 0:000> ?? *((python310!PyTypeObject *)0x2806a705790)->tp_richcompare > 0x00007ffd`55cac288 > _object* python310!object_richcompare+0( > _object*, > _object*, > int) > > Class D uses slot_tp_richcompare(): > > 0:000> ?? *((python310!PyTypeObject *)0x2806a6bbfe0)->tp_richcompare > 0x00007ffd`55cdef1c > _object* python310!slot_tp_richcompare+0( > _object*, > _object*, > int) > > Source code of slot_tp_richcompare(): > > > https://github.com/python/cpython/blob/v3.10.4/Objects/typeobject.c#L7610-L7626 > From o1bigtenor at gmail.com Tue May 17 07:20:39 2022 From: o1bigtenor at gmail.com (o1bigtenor) Date: Tue, 17 May 2022 06:20:39 -0500 Subject: Request for assistance (hopefully not OT) Message-ID: Greetings I was having space issues in my /usr directory so I deleted some programs thinking that the space taken was more an issue than having older versions of the program. So one of the programs I deleted (using rm -r) was python3.9. Python3.10 was already installed so I thought (naively!!!) that things should continue working. (Python 3.6, 3.7 and 3.8 were also part of this cleanup.) So now I have problems. Following is the system barf that I get when I run '# apt upgrade'. What can I do to correct this self-inflicted problem? (running on debian testing 5.17 Setting up python2.7-minimal (2.7.18-13.1) ... Could not find platform independent libraries Could not find platform dependent libraries Consider setting $PYTHONHOME to [:] /usr/bin/python2.7: can't open file '/usr/lib/python2.7/py_compile.py': [Errno 2] No such file or directory dpkg: error processing package python2.7-minimal (--configure): installed python2.7-minimal package post-installation script subprocess returned error exit status 2 Setting up python3.9-minimal (3.9.12-1) ... update-binfmts: warning: /usr/share/binfmts/python3.9: no executable /usr/bin/python3.9 found, but continuing anyway as you request /var/lib/dpkg/info/python3.9-minimal.postinst: 51: /usr/bin/python3.9: not found dpkg: error processing package python3.9-minimal (--configure): installed python3.9-minimal package post-installation script subprocess returned error exit status 127 dpkg: dependency problems prevent configuration of python3.9: python3.9 depends on python3.9-minimal (= 3.9.12-1); however: Package python3.9-minimal is not configured yet. dpkg: error processing package python3.9 (--configure): dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of python2.7: python2.7 depends on python2.7-minimal (= 2.7.18-13.1); however: Package python2.7-minimal is not configured yet. dpkg: error processing package python2.7 (--configure): dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of python3.9-dev: python3.9-dev depends on python3.9 (= 3.9.12-1); however: Package python3.9 is not configured yet. dpkg: error processing package python3.9-dev (--configure): dependency problems - leaving unconfigured . . . Errors were encountered while processing: python2.7-minimal python3.9-minimal python3.9 python2.7 python3.9-dev From martinp.dipaola at gmail.com Tue May 17 07:39:03 2022 From: martinp.dipaola at gmail.com (Martin Di Paola) Date: Tue, 17 May 2022 08:39:03 -0300 Subject: Request for assistance (hopefully not OT) In-Reply-To: References: Message-ID: <20220517113903.qamaodh4jub3snhj@gmail.com> Try to reinstall python and only python and if you succeeds, then try to reinstall the other tools. For this, use "apt-get" instead of "apt" $ sudo apt-get reinstall python3 When a system is heavily broken, be extra careful and read the output of the programs. If "apt-get" says than in order to reinstall python it will have to remove half of your computer, abort. Better ask than sorry. Best of the lucks. Martin. On Tue, May 17, 2022 at 06:20:39AM -0500, o1bigtenor wrote: >Greetings > >I was having space issues in my /usr directory so I deleted some >programs thinking that the space taken was more an issue than having >older versions of the program. > >So one of the programs I deleted (using rm -r) was python3.9. >Python3.10 was already installed so I thought (naively!!!) that things >should continue working. >(Python 3.6, 3.7 and 3.8 were also part of this cleanup.) > >So now I have problems. > >Following is the system barf that I get when I run '# apt upgrade'. > >What can I do to correct this self-inflicted problem? > >(running on debian testing 5.17 > >Setting up python2.7-minimal (2.7.18-13.1) ... >Could not find platform independent libraries >Could not find platform dependent libraries >Consider setting $PYTHONHOME to [:] >/usr/bin/python2.7: can't open file >'/usr/lib/python2.7/py_compile.py': [Errno 2] No such file or >directory >dpkg: error processing package python2.7-minimal (--configure): > installed python2.7-minimal package post-installation script >subprocess returned error exit status 2 >Setting up python3.9-minimal (3.9.12-1) ... >update-binfmts: warning: /usr/share/binfmts/python3.9: no executable >/usr/bin/python3.9 found, but continuing anyway as you request >/var/lib/dpkg/info/python3.9-minimal.postinst: 51: /usr/bin/python3.9: not found >dpkg: error processing package python3.9-minimal (--configure): > installed python3.9-minimal package post-installation script >subprocess returned error exit status 127 >dpkg: dependency problems prevent configuration of python3.9: > python3.9 depends on python3.9-minimal (= 3.9.12-1); however: > Package python3.9-minimal is not configured yet. > >dpkg: error processing package python3.9 (--configure): > dependency problems - leaving unconfigured >dpkg: dependency problems prevent configuration of python2.7: > python2.7 depends on python2.7-minimal (= 2.7.18-13.1); however: > Package python2.7-minimal is not configured yet. > >dpkg: error processing package python2.7 (--configure): > dependency problems - leaving unconfigured >dpkg: dependency problems prevent configuration of python3.9-dev: > python3.9-dev depends on python3.9 (= 3.9.12-1); however: > Package python3.9 is not configured yet. > >dpkg: error processing package python3.9-dev (--configure): > dependency problems - leaving unconfigured >. . . >Errors were encountered while processing: > python2.7-minimal > python3.9-minimal > python3.9 > python2.7 > python3.9-dev >-- >https://mail.python.org/mailman/listinfo/python-list From rosuav at gmail.com Tue May 17 08:48:54 2022 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 17 May 2022 22:48:54 +1000 Subject: Request for assistance (hopefully not OT) In-Reply-To: References: Message-ID: On Tue, 17 May 2022 at 21:22, o1bigtenor wrote: > > Greetings > > I was having space issues in my /usr directory so I deleted some > programs thinking that the space taken was more an issue than having > older versions of the program. > > So one of the programs I deleted (using rm -r) was python3.9. > Python3.10 was already installed so I thought (naively!!!) that things > should continue working. > (Python 3.6, 3.7 and 3.8 were also part of this cleanup.) Did you install Python 3.9 using apt? If so, you should definitely have removed it using apt - if for no reason than to find out if something's depending on it. Generally, Linux systems have just one "system Python" that other applications depend on. Any other installed version is completely independent. > So now I have problems. > > Following is the system barf that I get when I run '# apt upgrade'. > > What can I do to correct this self-inflicted problem? > > (running on debian testing 5.17 I presume 5.17 is the Linux kernel version? Depending on how up-to-date your Debian Testing is, that should theoretically mean that the system Python is 3.10, which would imply that it should have been safe to remove 3.9... but only if you had done it with apt. > Setting up python2.7-minimal (2.7.18-13.1) ... > Could not find platform independent libraries > Could not find platform dependent libraries > Consider setting $PYTHONHOME to [:] > /usr/bin/python2.7: can't open file > '/usr/lib/python2.7/py_compile.py': [Errno 2] No such file or > directory Did you also use rm to get rid of Python 2.7? > dpkg: error processing package python2.7-minimal (--configure): > installed python2.7-minimal package post-installation script > subprocess returned error exit status 2 > Setting up python3.9-minimal (3.9.12-1) ... > update-binfmts: warning: /usr/share/binfmts/python3.9: no executable > /usr/bin/python3.9 found, but continuing anyway as you request > /var/lib/dpkg/info/python3.9-minimal.postinst: 51: /usr/bin/python3.9: not found > dpkg: error processing package python3.9-minimal (--configure): > installed python3.9-minimal package post-installation script > subprocess returned error exit status 127 > dpkg: dependency problems prevent configuration of python3.9: > python3.9 depends on python3.9-minimal (= 3.9.12-1); however: > Package python3.9-minimal is not configured yet. > > dpkg: error processing package python3.9 (--configure): > dependency problems - leaving unconfigured > dpkg: dependency problems prevent configuration of python2.7: > python2.7 depends on python2.7-minimal (= 2.7.18-13.1); however: > Package python2.7-minimal is not configured yet. > > dpkg: error processing package python2.7 (--configure): > dependency problems - leaving unconfigured > dpkg: dependency problems prevent configuration of python3.9-dev: > python3.9-dev depends on python3.9 (= 3.9.12-1); however: > Package python3.9 is not configured yet. > > dpkg: error processing package python3.9-dev (--configure): > dependency problems - leaving unconfigured > . . . > Errors were encountered while processing: > python2.7-minimal > python3.9-minimal > python3.9 > python2.7 > python3.9-dev So, yeah, you're definitely going to need to reinstate some parts of Python to get this going. If you can figure out which exact Python versions you need, it might be possible to restore them manually. Download the packages from packages.debian.org, then try to manually install them with dpkg, and if that fails, unpack them and put the files into the right places. It's going to be a pain. A lot of pain. And next time, use apt to uninstall what apt installed :) Something else to consider, though: It might not be Python that's taking up all the space. On my system, /usr is dominated by /usr/lib and /usr/local/lib, and while it might look like the pythonx.y directories there are the large part, it's actually not Python itself that's so big: it's other libraries, installed using either apt or pip. So when you're trying to free up space, look to see whether you have packages installed into every version of Python you have; the largest directories in my python3.9/site-packages are scipy, plotly, numpy, pandas, speech_recognition, matplotlib, and Cython - all great tools, but if you have a copy for 3.9, a copy for 3.10, a copy for 3.11, etc, it adds up fast. "Ten minutes with a hacksaw will save you thirty with a shovel" -- Miss Pauling, discussing the art of uninstalling.... something ChrisA From hjp-python at hjp.at Tue May 17 11:21:29 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Tue, 17 May 2022 17:21:29 +0200 Subject: Convert the decimal numbers expressed in a `numpy.ndarray` into a matrix representing elements in fractional form In-Reply-To: <27r58hhbe2ldc4f0h9tmci3rallg6cc3va@4ax.com> References: <5eb0b494-ec7a-4b6c-9966-078889dbc5aen@googlegroups.com> <54r48hdpj8jov1on1f9lfi4t6vl08lk6md@4ax.com> <27r58hhbe2ldc4f0h9tmci3rallg6cc3va@4ax.com> Message-ID: <20220517152129.3d2ns6e4yrdlnbrm@hjp.at> On 2022-05-16 20:48:02 -0400, Dennis Lee Bieber wrote: > On Mon, 16 May 2022 17:22:17 -0700 (PDT), "hongy... at gmail.com" > declaimed the following: > >a=re.sub(r"'","",a) > > Explain what you believe this operation is doing, show us the input and > the output. > > The best I can make out of that is that it is looking for single quote > characters within whatever "a" is, and replacing them with nothing. > Something much more understandable, without invoking a regular expression > library (especially when neither the search nor the replacement terms are > regular expressions) with simple string operations... > > stripped = "".join(quoted.split("'")) Whether that's easier to understand it very much in the eye of the beholder. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From python at mrabarnett.plus.com Tue May 17 12:20:54 2022 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 17 May 2022 17:20:54 +0100 Subject: Convert the decimal numbers expressed in a `numpy.ndarray` into a matrix representing elements in fractional form In-Reply-To: <20220517152129.3d2ns6e4yrdlnbrm@hjp.at> References: <5eb0b494-ec7a-4b6c-9966-078889dbc5aen@googlegroups.com> <54r48hdpj8jov1on1f9lfi4t6vl08lk6md@4ax.com> <27r58hhbe2ldc4f0h9tmci3rallg6cc3va@4ax.com> <20220517152129.3d2ns6e4yrdlnbrm@hjp.at> Message-ID: On 2022-05-17 16:21, Peter J. Holzer wrote: > On 2022-05-16 20:48:02 -0400, Dennis Lee Bieber wrote: >> On Mon, 16 May 2022 17:22:17 -0700 (PDT), "hongy... at gmail.com" >> declaimed the following: >> >a=re.sub(r"'","",a) >> >> Explain what you believe this operation is doing, show us the input and >> the output. >> >> The best I can make out of that is that it is looking for single quote >> characters within whatever "a" is, and replacing them with nothing. >> Something much more understandable, without invoking a regular expression >> library (especially when neither the search nor the replacement terms are >> regular expressions) with simple string operations... >> >> stripped = "".join(quoted.split("'")) > > Whether that's easier to understand it very much in the eye of the > beholder. > As it's just a simple replacement, I would've thought that the 'obvious' solution would be: a = a.replace("'", "") From lukasz at langa.pl Tue May 17 13:36:56 2022 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Tue, 17 May 2022 19:36:56 +0200 Subject: [RELEASE] Python 3.9.13 is now available Message-ID: This is the thirteenth maintenance release of Python 3.9. Get it here: Python 3.9.13 According to the release calendar specified in PEP 596 , Python 3.9.13 is the final regular maintenance release. Starting now, the 3.9 branch will only accept security fixes and releases of those will be made in source-only form until October 2025. This is a milestone moment for me as it means that now both of my release series are security-only. My work as release manager enters its final stage. I?m not crying, you?re crying! ? Compared to the 3.8 series, this last regular bugfix release is still pretty active at 166 commits since 3.9.12. In comparison, version 3.8.10, the final regular bugfix release of Python 3.8, included only 92 commits. However, it?s likely that it was 3.8 that was special here with the governance changes occupying core developers? minds. For reference, version 3.7.8, the final regular bugfix release of Python 3.7, included 187 commits. In any case, 166 commits is quite a few changes, some of which being pretty important fixes. Take a look at the change log for details. Major new features of the 3.9 series, compared to 3.8 Some of the new major new features and changes in Python 3.9 are: PEP 573 , Module State Access from C Extension Methods PEP 584 , Union Operators in dict PEP 585 , Type Hinting Generics In Standard Collections PEP 593 , Flexible function and variable annotations PEP 602 , Python adopts a stable annual release cadence PEP 614 , Relaxing Grammar Restrictions On Decorators PEP 615 , Support for the IANA Time Zone Database in the Standard Library PEP 616 , String methods to remove prefixes and suffixes PEP 617 , New PEG parser for CPython BPO 38379 , garbage collection does not block on resurrected objects; BPO 38692 , os.pidfd_open added that allows process management without races and signals; BPO 39926 , Unicode support updated to version 13.0.0; BPO 1635741 , when Python is initialized multiple times in the same process, it does not leak memory anymore; A number of Python builtins (range, tuple, set, frozenset, list, dict) are now sped up using PEP 590 vectorcall; A number of Python modules (_abc, audioop, _bz2, _codecs, _contextvars, _crypt, _functools, _json, _locale, operator, resource, time, _weakref) now use multiphase initialization as defined by PEP 489 ; A number of standard library modules (audioop, ast, grp, _hashlib, pwd, _posixsubprocess, random, select, struct, termios, zlib) are now using the stable ABI defined by PEP 384 . You can find a more comprehensive list in this release?s ?What?s New ? document. We hope you enjoy Python 3.9! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. Your friendly release team, Ned Deily @nad Steve Dower @steve.dower ?ukasz Langa @ambv -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From loris.bennett at fu-berlin.de Tue May 17 09:00:10 2022 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Tue, 17 May 2022 15:00:10 +0200 Subject: Request for assistance (hopefully not OT) References: Message-ID: <87mtfg71r9.fsf@hornfels.zedat.fu-berlin.de> o1bigtenor writes: > Greetings > > I was having space issues in my /usr directory so I deleted some > programs thinking that the space taken was more an issue than having > older versions of the program. > > So one of the programs I deleted (using rm -r) was python3.9. Deleting anything from /usr via 'rm -r' which was installed via the package manager is an extremely bad idea. If you want to remove stuff, use the package manager. > Python3.10 was already installed so I thought (naively!!!) that things > should continue working. > (Python 3.6, 3.7 and 3.8 were also part of this cleanup.) Python 3.10 may be installed, but a significant number of packages depend on Python 3.9. That's why you should use the package manager - it knows all about the dependencies. > So now I have problems. I think you had a problem before that. Debian testing is not an operating system you should be using if you have a fairly good understanding of how Debian (or Linux in general) works. > Following is the system barf that I get when I run '# apt upgrade'. > > What can I do to correct this self-inflicted problem? > > (running on debian testing 5.17 I think you mean just 'Debian testing', which is what will become the next version of Debian, i.e. Debian 12. The '5.17' is just the kernel version, not a version of Debian. > Setting up python2.7-minimal (2.7.18-13.1) ... > Could not find platform independent libraries > Could not find platform dependent libraries > Consider setting $PYTHONHOME to [:] > /usr/bin/python2.7: can't open file > '/usr/lib/python2.7/py_compile.py': [Errno 2] No such file or > directory > dpkg: error processing package python2.7-minimal (--configure): > installed python2.7-minimal package post-installation script > subprocess returned error exit status 2 > Setting up python3.9-minimal (3.9.12-1) ... > update-binfmts: warning: /usr/share/binfmts/python3.9: no executable > /usr/bin/python3.9 found, but continuing anyway as you request > /var/lib/dpkg/info/python3.9-minimal.postinst: 51: /usr/bin/python3.9: not found > dpkg: error processing package python3.9-minimal (--configure): > installed python3.9-minimal package post-installation script > subprocess returned error exit status 127 > dpkg: dependency problems prevent configuration of python3.9: > python3.9 depends on python3.9-minimal (= 3.9.12-1); however: > Package python3.9-minimal is not configured yet. > > dpkg: error processing package python3.9 (--configure): > dependency problems - leaving unconfigured > dpkg: dependency problems prevent configuration of python2.7: > python2.7 depends on python2.7-minimal (= 2.7.18-13.1); however: > Package python2.7-minimal is not configured yet. > > dpkg: error processing package python2.7 (--configure): > dependency problems - leaving unconfigured > dpkg: dependency problems prevent configuration of python3.9-dev: > python3.9-dev depends on python3.9 (= 3.9.12-1); however: > Package python3.9 is not configured yet. > > dpkg: error processing package python3.9-dev (--configure): > dependency problems - leaving unconfigured > . . . > Errors were encountered while processing: > python2.7-minimal > python3.9-minimal > python3.9 > python2.7 > python3.9-dev It might be possible to fix the system. If will probably be fairly difficult, but you would probably learn a lot doing it. However, if I were you, I would just install Debian stable over your borked system and then learn a bit more about package management. Cheers, Loris -- This signature is currently under construction. From loris.bennett at fu-berlin.de Tue May 17 09:02:57 2022 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Tue, 17 May 2022 15:02:57 +0200 Subject: Request for assistance (hopefully not OT) References: <87mtfg71r9.fsf@hornfels.zedat.fu-berlin.de> Message-ID: <87ilq471mm.fsf@hornfels.zedat.fu-berlin.de> [snip (26 lines)] > I think you had a problem before that. Debian testing is not an > operating system you should be using if you have a fairly good > understanding of how Debian (or Linux in general) works. Should be I think you had a problem before that. Debian testing is not an operating system you should be using *unless* you have a fairly good understanding of how Debian (or Linux in general) works. [snip (62 lines)] -- This signature is currently under construction. From grant.b.edwards at gmail.com Tue May 17 14:27:29 2022 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Tue, 17 May 2022 11:27:29 -0700 (PDT) Subject: Request for assistance (hopefully not OT) References: <87mtfg71r9.fsf@hornfels.zedat.fu-berlin.de> Message-ID: <6283e911.1c69fb81.f271.5d32@mx.google.com> On 2022-05-17, Loris Bennett wrote: > It might be possible to fix the system. If will probably be fairly > difficult, but you would probably learn a lot doing it. However, if I > were you, I would just install Debian stable over your borked system and > then learn a bit more about package management. Other then reinstalling, the easiest way to fix a broken system like that requires access to a similarly configured system that isn't broken. Find one of those, and start copying binaries from the working system to the broken system. At some point, the broken system should start to work well enough that you can re-install the packages you broke by removing files behind the back of the package manager. Whether that's going to be faster/easier than backing up your configuration and data files and reinstalling Debian is another question. My guess is that reinstalling would probably be faster. FWIW, this is indeed off-topic for this group. It isn't a really Python question, it's a Debian question. -- Grant From rosuav at gmail.com Tue May 17 14:35:59 2022 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 18 May 2022 04:35:59 +1000 Subject: Request for assistance (hopefully not OT) In-Reply-To: <87mtfg71r9.fsf@hornfels.zedat.fu-berlin.de> References: <87mtfg71r9.fsf@hornfels.zedat.fu-berlin.de> Message-ID: On Wed, 18 May 2022 at 04:05, Loris Bennett wrote: > > So now I have problems. > > I think you had a problem before that. Debian testing is not an > operating system you should be using if you have a fairly good > understanding of how Debian (or Linux in general) works. I take issue with that! Debian Testing is a perfectly viable operating system! I wouldn't use it on a server, but it's perfectly fine to use it on a personal machine. You can generally consider Debian Testing to be broadly as stable as Ubuntu non-LTS releases, although in my opinion, it's actually quite a bit more dependable than them. (Perhaps you're thinking of Debian Unstable?) ChrisA From rosuav at gmail.com Tue May 17 14:37:42 2022 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 18 May 2022 04:37:42 +1000 Subject: Request for assistance (hopefully not OT) In-Reply-To: <87ilq471mm.fsf@hornfels.zedat.fu-berlin.de> References: <87mtfg71r9.fsf@hornfels.zedat.fu-berlin.de> <87ilq471mm.fsf@hornfels.zedat.fu-berlin.de> Message-ID: On Wed, 18 May 2022 at 04:05, Loris Bennett wrote: > > [snip (26 lines)] > > > I think you had a problem before that. Debian testing is not an > > operating system you should be using if you have a fairly good > > understanding of how Debian (or Linux in general) works. > > Should be > > I think you had a problem before that. Debian testing is not an > operating system you should be using *unless* you have a fairly good > understanding of how Debian (or Linux in general) works. > > [snip (62 lines)] > Oh! My bad, didn't see this correction, sorry. With this adjustment, the comment is a bit more reasonable, although I'd still say it's generally fine to run Debian Testing on a personal desktop machine; there are a number of distros that base themselves on Testing. But yes, "unless" makes much more sense there. ChrisA From torriem at gmail.com Tue May 17 21:55:58 2022 From: torriem at gmail.com (Michael Torrie) Date: Tue, 17 May 2022 19:55:58 -0600 Subject: Request for assistance (hopefully not OT) In-Reply-To: References: Message-ID: On 5/17/22 05:20, o1bigtenor wrote: > What can I do to correct this self-inflicted problem? Those are always the fun ones. Reminds me of when I was first learning Linux using Red Hat Linux 5.0 or 5.1. This was long before nice dependency-solving tools like apt. I wanted to install and run StarOffice, but it needed a newer libc (this was during the painful transition from libc5 to glibc6). I ended up removing libc which *everything depends on, trying to get the glibc update installed. Needless to say that broke the entire system. Nothing but a reinstall could be done in those days. Anyway, good luck. I think you can rescue it yet following the advice others have given. From stephen_tucker at sil.org Wed May 18 05:39:10 2022 From: stephen_tucker at sil.org (Stephen Tucker) Date: Wed, 18 May 2022 10:39:10 +0100 Subject: Discerning "Run Environment" Message-ID: Hi, I am a Windows 10 user still using Python 2.x (for good reasons, I assure you.) I have a Python 2.x module that I would like to be able to use in a variety of Python 2.x programs. The module outputs characters to the user that are only available in the Unicode character set. I have found that the selection of characters in that set that are available to my software depends on whether, for example, the program is being run during an IDLE session or at a Command Prompt. I am therefore needing to include logic in this module that (a) enables it to output appropriate characters depending on whether it is being run during an IDLE session or at a command prompt, and (b) enables it to discern which of these two "run environments" it is running in. Goal (a) is achieved easily by using string.replace to replace unavailable characters with available ones where necessary. The best way I have found so far to achieve goal (b) is to use sys.modules and ascertain whether any modules contain the string "idlelib". If they do, that I assume that the software is being run in an IDLE session. I suspect that there is a more Pythonic (and reliable?) way of achieving goal (b). Can anyone please tell me if there is, and, if there is, what it is? Thanks. Stephen Tucker. From rosuav at gmail.com Wed May 18 05:56:18 2022 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 18 May 2022 19:56:18 +1000 Subject: Discerning "Run Environment" In-Reply-To: References: Message-ID: On Wed, 18 May 2022 at 19:40, Stephen Tucker wrote: > > Hi, > > I am a Windows 10 user still using Python 2.x (for good reasons, I assure > you.) > > I have a Python 2.x module that I would like to be able to use in a variety > of Python 2.x programs. The module outputs characters to the user that are > only available in the Unicode character set. > > I have found that the selection of characters in that set that are > available to my software depends on whether, for example, the program is > being run during an IDLE session or at a Command Prompt. Real solution? Set the command prompt to codepage 65001. Then it should be able to handle all characters. (Windows-65001 is its alias for UTF-8.) > I am therefore needing to include logic in this module that (a) enables it > to output appropriate characters depending on whether it is being run > during an IDLE session or at a command prompt, and (b) enables it to > discern which of these two "run environments" it is running in. > > Goal (a) is achieved easily by using string.replace to replace unavailable > characters with available ones where necessary. > > The best way I have found so far to achieve goal (b) is to use sys.modules > and ascertain whether any modules contain the string "idlelib". If they do, > that I assume that the software is being run in an IDLE session. > > I suspect that there is a more Pythonic (and reliable?) way of achieving > goal (b). > > Can anyone please tell me if there is, and, if there is, what it is? Ultimately, it's going to depend on where your text is going: is it going to the console, or to a Tk widget? I don't have a Windows system handy to check, but I would suspect that you can distinguish these by seeing whether sys.stdout is a tty, since Idle pipes stdout into its own handler. That won't be a perfect check, as it would consider "piped into another command" to be UTF-8 compatible, but that's probably more right than wrong anyway. ChrisA From pablo.martinezulloa at ucdconnect.ie Wed May 18 10:08:39 2022 From: pablo.martinezulloa at ucdconnect.ie (Pablo Martinez Ulloa) Date: Wed, 18 May 2022 15:08:39 +0100 Subject: Issue sending data from C++ to Python Message-ID: Hello, I have been using your C++ Python API, in order to establish a bridge from C++ to Python. We want to do this, as we have a tactile sensor, which only has a library developed in C++, but we want to obtain the data in real time in Python to perform tests with a robotic arm and gripper. The problem we are facing is with transmitting the data into Python. We are using the function Py_BuildValue, and it seems to be working, but only for sending one value at a time, and we want to transmit 54 numbers. When transmitting a single value retrieved from the sensor into Python in a double or string format, it works, but when we make the string bigger it gives us an error with the utf-8 encoding. Also when sending a tuple of doubles or strings, it works when the tuple is composed of a single element, but not when we send 2 or more values. Is there any way of fixing this issue? Or is it just not possible to transmit such a large amount of data? Thank you. Best regards, *Pablo Martinez Ulloa* PhD Candidate, School of Electrical and Electronic Engineering, R324A University College Dublin, Belfield, Dublin 4, Ireland From Marco.Sulla.Python at gmail.com Tue May 17 16:45:34 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Tue, 17 May 2022 22:45:34 +0200 Subject: tail In-Reply-To: References: Message-ID: Well, I've done a benchmark. >>> timeit.timeit("tail('/home/marco/small.txt')", globals={"tail":tail}, number=100000) 1.5963431186974049 >>> timeit.timeit("tail('/home/marco/lorem.txt')", globals={"tail":tail}, number=100000) 2.5240604374557734 >>> timeit.timeit("tail('/home/marco/lorem.txt', chunk_size=1000)", globals={"tail":tail}, number=100000) 1.8944984432309866 small.txt is a text file of 1.3 KB. lorem.txt is a lorem ipsum of 1.2 GB. It seems the performance is good, thanks to the chunk suggestion. But the time of Linux tail surprise me: marco at buzz:~$ time tail lorem.txt [text] real 0m0.004s user 0m0.003s sys 0m0.001s It's strange that it's so slow. I thought it was because it decodes and print the result, but I timed timeit.timeit("print(tail('/home/marco/lorem.txt').decode('utf-8'))", globals={"tail":tail}, number=100000) and I got ~36 seconds. It seems quite strange to me. Maybe I got the benchmarks wrong at some point? From wlfraed at ix.netcom.com Tue May 17 21:11:02 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Tue, 17 May 2022 21:11:02 -0400 Subject: Convert the decimal numbers expressed in a `numpy.ndarray` into a matrix representing elements in fractional form References: <5eb0b494-ec7a-4b6c-9966-078889dbc5aen@googlegroups.com> <54r48hdpj8jov1on1f9lfi4t6vl08lk6md@4ax.com> <27r58hhbe2ldc4f0h9tmci3rallg6cc3va@4ax.com> <20220517152129.3d2ns6e4yrdlnbrm@hjp.at> Message-ID: On Tue, 17 May 2022 17:20:54 +0100, MRAB declaimed the following: >As it's just a simple replacement, I would've thought that the 'obvious' >solution would be: > a = a.replace("'", "") Mea culpa... Guess it's time for me to review the library reference for basic data types again. I'm so used to the .join(.split()) (usually for other purposes -- like a quick&dirty TSV/CSV formatting without importing the csv module). The only firm item is that I do not look at any regular expression module if I can come up with something using native data type methods -- and using the overhead of re with just simple text [ie; nothing that might be called an "expression" designed to match /varying/ content) seems really inefficient. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From loris.bennett at fu-berlin.de Wed May 18 02:02:30 2022 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Wed, 18 May 2022 08:02:30 +0200 Subject: Request for assistance (hopefully not OT) References: <87mtfg71r9.fsf@hornfels.zedat.fu-berlin.de> <87ilq471mm.fsf@hornfels.zedat.fu-berlin.de> Message-ID: <87czgbidjd.fsf@hornfels.zedat.fu-berlin.de> Chris Angelico writes: > On Wed, 18 May 2022 at 04:05, Loris Bennett wrote: >> >> [snip (26 lines)] >> >> > I think you had a problem before that. Debian testing is not an >> > operating system you should be using if you have a fairly good >> > understanding of how Debian (or Linux in general) works. >> >> Should be >> >> I think you had a problem before that. Debian testing is not an >> operating system you should be using *unless* you have a fairly good >> understanding of how Debian (or Linux in general) works. >> >> [snip (62 lines)] >> > > Oh! My bad, didn't see this correction, sorry. With this adjustment, > the comment is a bit more reasonable, although I'd still say it's > generally fine to run Debian Testing on a personal desktop machine; > there are a number of distros that base themselves on Testing. > > But yes, "unless" makes much more sense there. It's lucky I never got "if" and "unless" mixed up when I used to program in Perl ;-) Yes, there are a number of distros based on Debian Testing, but those tend to be aimed more at sysadmins (e.g. Kali and Grml) than people just starting out with Linux. However, with plain old Debian Testing you need to be able to deal with things occasionally not working properly. As the Debian people say about Testing: "If it doesn't work for you, then there's a good chance it's broken." And that's even before you delete part of the OS with 'rm'. Cheers, Loris From python at mrabarnett.plus.com Wed May 18 13:37:15 2022 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 18 May 2022 18:37:15 +0100 Subject: Issue sending data from C++ to Python In-Reply-To: References: Message-ID: On 2022-05-18 15:08, Pablo Martinez Ulloa wrote: > Hello, > > I have been using your C++ Python API, in order to establish a bridge from > C++ to Python. We want to do this, as we have a tactile sensor, which only > has a library developed in C++, but we want to obtain the data in real time > in Python to perform tests with a robotic arm and gripper. The problem we > are facing is with transmitting the data into Python. We are using the > function Py_BuildValue, and it seems to be working, but only for sending > one value at a time, and we want to transmit 54 numbers. When > transmitting a single value retrieved from the sensor into Python in a > double or string format, it works, but when we make the string bigger it > gives us an error with the utf-8 encoding. Also when sending a tuple of > doubles or strings, it works when the tuple is composed of a single > element, but not when we send 2 or more values. Is there any way of fixing > this issue? Or is it just not possible to transmit such a large amount of > data? > I find it easier if I have some code in front of me to 'criticise'! Have you thought about building and returning, say, a list instead? /* Error checking omitted. */ PyObject* list; list = PyList_New(0); PyList_Append(list, PyLong_FromSsize_t(1)); PyList_Append(list, PyFloat_FromDouble(2.0)); ... From auriocus at gmx.de Wed May 18 15:53:09 2022 From: auriocus at gmx.de (Christian Gollwitzer) Date: Wed, 18 May 2022 21:53:09 +0200 Subject: Issue sending data from C++ to Python In-Reply-To: References: Message-ID: Am 18.05.22 um 16:08 schrieb Pablo Martinez Ulloa: > I have been using your C++ Python API, in order to establish a bridge from > C++ to Python. We want to do this, as we have a tactile sensor, which only > has a library developed in C++, but we want to obtain the data in real time > in Python to perform tests with a robotic arm and gripper. The problem we > are facing is with transmitting the data into Python. We are using the > function Py_BuildValue, and it seems to be working, but only for sending > one value at a time, and we want to transmit 54 numbers The usual way to pass an array of numbers into Python is by means of a Numpy Array. In order to do that, you need to include arrayobject.h and then use PyArray_SimpleNew to create an array of numbers as a Python object which you can return https://numpy.org/devdocs/reference/c-api/array.html#c.PyArray_SimpleNew Christian From cs at cskk.id.au Wed May 18 17:30:20 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Thu, 19 May 2022 07:30:20 +1000 Subject: tail In-Reply-To: References: Message-ID: On 17May2022 22:45, Marco Sulla wrote: >Well, I've done a benchmark. >>>> timeit.timeit("tail('/home/marco/small.txt')", globals={"tail":tail}, number=100000) >1.5963431186974049 >>>> timeit.timeit("tail('/home/marco/lorem.txt')", globals={"tail":tail}, number=100000) >2.5240604374557734 >>>> timeit.timeit("tail('/home/marco/lorem.txt', chunk_size=1000)", globals={"tail":tail}, number=100000) >1.8944984432309866 This suggests that the file size does not dominate uour runtime. Ah. _Or_ that there are similar numbers of newlines vs text in the files so reading similar amounts of data from the end. If the "line desnity" of the files were similar you would hope that the runtimes would be similar. >small.txt is a text file of 1.3 KB. lorem.txt is a lorem ipsum of 1.2 >GB. It seems the performance is good, thanks to the chunk suggestion. > >But the time of Linux tail surprise me: > >marco at buzz:~$ time tail lorem.txt >[text] > >real 0m0.004s >user 0m0.003s >sys 0m0.001s > >It's strange that it's so slow. I thought it was because it decodes >and print the result, but I timed You're measuring different things. timeit() tries hard to measure just the code snippet you provide. It doesn't measure the startup cost of the whole python interpreter. Try: time python3 your-tail-prog.py /home/marco/lorem.txt BTW, does your `tail()` print output? If not, again not measuring the same thing. If you have the source of tail(1) to hand, consider getting to the core and measuring `time()` immediately before and immediately after the central tail operation and printing the result. Also: does tail(1) do character set / encoding stuff? Does your Python code do that? Might be apples and oranges. Cheers, Cameron Simpson From cousinstanley at gmail.com Wed May 18 17:26:02 2022 From: cousinstanley at gmail.com (Cousin Stanley) Date: Wed, 18 May 2022 14:26:02 -0700 Subject: Convert the decimal numbers expressed in a `numpy.ndarray` into a matrix representing elements in fractional form In-Reply-To: <5eb0b494-ec7a-4b6c-9966-078889dbc5aen@googlegroups.com> References: <5eb0b494-ec7a-4b6c-9966-078889dbc5aen@googlegroups.com> Message-ID: #!/usr/bin/env python3 ''' NewsGroup .... comp.lang.python Subject ...... Convert the decimal numbers expressed in a numpy.ndarray into a matrix representing elements in fractiona Date ......... 2022-05-16 Post_By ...... hongy... Edit_By ...... Stanley C. Kitching ''' import numpy as np from fractions import Fraction b = [ [ 0.0 , -1.0 , 0.0 , 0.25 ] , [ 1.0 , 0.0 , 0.0 , 0.25 ] , [ 0.0 , 0.0 , 1.0 , 0.25 ] , [ 0.0 , 0.0 , 0.0 , 1.0 ] ] a = [ ] print( '\n b .... \n' ) for row in b : arow = [] print( ' ' , row ) for dec_x in row : frac_x = Fraction( dec_x ) arow.append( frac_x ) a.append( arow ) # using f-string format print( '\n a .... \n' ) for row in a : for item in row : print( f' {item} ' , end = '' ) print() # ------------------------------------------ -- Stanley C. Kitching Human Being Phoenix, Arizona From gabriele1NOSPAM at hotmail.com Wed May 18 17:52:05 2022 From: gabriele1NOSPAM at hotmail.com (^Bart) Date: Wed, 18 May 2022 23:52:05 +0200 Subject: Python & nmap Message-ID: Hi guys, i need to copy some files from a Debian client to all linux embedded clients. I know the linux commands like: # scp "my_file" root at 192.168.205.x/my_directory But... I have to upload 100 devices, I have a lan and a dhcp server just for this work and I'd like to make a script by Python which can: 1) To scan the lan 2) To find which ips are "ready" 3) To send files to all of the "ready" clients 4) After I see on the display of these clients the successfully update I remove from the lan them and I put them to the box to send them to our customers. I found https://pypi.org/project/python-nmap/ and I followed the line "To check the network status" but... it doesn't work. THE INPUT ------------------------------------------------------------------------- import nmap nm.scan(hosts='192.168.205.0/24', arguments='-n -sP -PE -PA21,23,80,3389') hosts_list = [(x, nm[x]['status']['state']) for x in nm.all_hosts()] for host, status in hosts_list: print('{0}:{1}'.host) THE OUTPUT --------------------------------------------------------------------- Traceback (most recent call last): File "/home/gabriele/Documenti/Python/nmap.py", line 1, in import nmap File "/home/gabriele/Documenti/Python/nmap.py", line 2, in nm.scan(hosts='192.168.205.0/24', arguments='-n -sP -PE -PA21,23,80,3389') NameError: name 'nm' is not defined Regards. ^Bart From jon+usenet at unequivocal.eu Wed May 18 18:00:38 2022 From: jon+usenet at unequivocal.eu (Jon Ribbens) Date: Wed, 18 May 2022 22:00:38 -0000 (UTC) Subject: Python & nmap References: Message-ID: On 2022-05-18, ^Bart wrote: > THE INPUT > ------------------------------------------------------------------------- > import nmap > nm.scan(hosts='192.168.205.0/24', arguments='-n -sP -PE -PA21,23,80,3389') > hosts_list = [(x, nm[x]['status']['state']) for x in nm.all_hosts()] > for host, status in hosts_list: > print('{0}:{1}'.host) > > THE OUTPUT > --------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/gabriele/Documenti/Python/nmap.py", line 1, in > import nmap > File "/home/gabriele/Documenti/Python/nmap.py", line 2, in > nm.scan(hosts='192.168.205.0/24', arguments='-n -sP -PE > -PA21,23,80,3389') > NameError: name 'nm' is not defined You forgot the second line (after 'import nmap' and before 'nm.scan()'): nm = nmap.PortScanner() From eryksun at gmail.com Thu May 19 00:18:34 2022 From: eryksun at gmail.com (Eryk Sun) Date: Wed, 18 May 2022 23:18:34 -0500 Subject: Discerning "Run Environment" In-Reply-To: References: Message-ID: On 5/18/22, Chris Angelico wrote: > > Real solution? Set the command prompt to codepage 65001. Then it > should be able to handle all characters. (Windows-65001 is its alias > for UTF-8.) I suggest using win_unicode_console for Python versions prior to 3.6: https://pypi.org/project/win_unicode_console This package uses the console's native 16-bit character support with UTF-16 text, as does Python 3.6+. Compared to the console's incomplete and broken support for UTF-8, the console's support for UTF-16 (or just UCS-2 prior to Windows 10) is far more functional and reliable across commonly used versions of Windows 7, 8, and 10. Reading console input as UTF-8 is still limited to ASCII up to and including Windows 11, which for me is a showstopper. Non-ASCII characters are read as null bytes, which is useless. Support for writing UTF-8 to the console screen buffer is implemented correctly in recent builds of Windows 10 and 11, and mostly correct in Windows 8. Prior to Windows 8, writing UTF-8 to the console is badly broken. It returns the number of UTF-16 codes written instead of the number of bytes written, which confuses buffered writers into writing a lot of junk to the screen. From lal at solute.de Thu May 19 03:59:21 2022 From: lal at solute.de (Lars Liedtke) Date: Thu, 19 May 2022 09:59:21 +0200 Subject: Python & nmap In-Reply-To: References: Message-ID: > # scp "my_file" root at 192.168.205.x/my_directory Maybe it could be a good idea to look at Ansible for copying the Files to all the hosts, because that is one thing ansible is made for. For the nmap part: Ansible does not have a module for that (sadly) but is very extensible, so if you start developing something like that in Python, you could as well write an ansible module and combine both, because Ansible itself is written in Python. Cheers Lars -- Lars Liedtke Software Entwickler Phone: Fax: +49 721 98993- E-mail: lal at solute.de solute GmbH Zeppelinstra?e 15 76185 Karlsruhe Germany Marken der solute GmbH | brands of solute GmbH billiger.de | Shopping.de Gesch?ftsf?hrer | Managing Director: Dr. Thilo Gans, Bernd Vermaaten Webseite | www.solute.de Sitz | Registered Office: Karlsruhe Registergericht | Register Court: Amtsgericht Mannheim Registernummer | Register No.: HRB 110579 USt-ID | VAT ID: DE234663798 Informationen zum Datenschutz | Information about privacy policy http://solute.de/ger/datenschutz/grundsaetze-der-datenverarbeitung.php From alister.ware at ntlworld.com Thu May 19 05:04:13 2022 From: alister.ware at ntlworld.com (alister) Date: Thu, 19 May 2022 09:04:13 -0000 (UTC) Subject: Python & nmap References: Message-ID: On Wed, 18 May 2022 23:52:05 +0200, ^Bart wrote: > Hi guys, > > i need to copy some files from a Debian client to all linux embedded > clients. > > I know the linux commands like: > > # scp "my_file" root at 192.168.205.x/my_directory > > But... I have to upload 100 devices, I have a lan and a dhcp server just > for this work and I'd like to make a script by Python which can: > > 1) To scan the lan 2) To find which ips are "ready" > 3) To send files to all of the "ready" clients 4) After I see on the > display of these clients the successfully update I remove from the lan > them and I put them to the box to send them to our customers. > > I found https://pypi.org/project/python-nmap/ and I followed the line > "To check the network status" but... it doesn't work. > > THE INPUT > ------------------------------------------------------------------------- > import nmap nm.scan(hosts='192.168.205.0/24', arguments='-n -sP -PE > -PA21,23,80,3389') > hosts_list = [(x, nm[x]['status']['state']) for x in nm.all_hosts()] > for host, status in hosts_list: > print('{0}:{1}'.host) > > THE OUTPUT > --------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/gabriele/Documenti/Python/nmap.py", line 1, in > import nmap > File "/home/gabriele/Documenti/Python/nmap.py", line 2, in > nm.scan(hosts='192.168.205.0/24', arguments='-n -sP -PE > -PA21,23,80,3389') > NameError: name 'nm' is not defined > > Regards. > ^Bart Opbservations worth considering 1) could possibly be handled by a simple bash script (My bash skills are not great So i would probably still go python myself anyway) 2) Instead of checking availability just try to send & react appropriately if it fails (Ask for forgiveness not permission), the client could fail after test or during transfer anyway so you will still need this level of error checking -- QOTD: "What women and psychologists call `dropping your armor', we call "baring your neck." From poppy0687willia at gmail.com Thu May 19 12:01:37 2022 From: poppy0687willia at gmail.com (Poppy Thomas) Date: Thu, 19 May 2022 21:31:37 +0530 Subject: Invoice Message-ID: [image: Invoice.jpg] .............................. From Marco.Sulla.Python at gmail.com Thu May 19 13:50:16 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Thu, 19 May 2022 19:50:16 +0200 Subject: tail In-Reply-To: References: Message-ID: On Wed, 18 May 2022 at 23:32, Cameron Simpson wrote: > > On 17May2022 22:45, Marco Sulla wrote: > >Well, I've done a benchmark. > >>>> timeit.timeit("tail('/home/marco/small.txt')", globals={"tail":tail}, number=100000) > >1.5963431186974049 > >>>> timeit.timeit("tail('/home/marco/lorem.txt')", globals={"tail":tail}, number=100000) > >2.5240604374557734 > >>>> timeit.timeit("tail('/home/marco/lorem.txt', chunk_size=1000)", globals={"tail":tail}, number=100000) > >1.8944984432309866 > > This suggests that the file size does not dominate uour runtime. Yes, this is what I wanted to test and it seems good. > Ah. > _Or_ that there are similar numbers of newlines vs text in the files so > reading similar amounts of data from the end. If the "line desnity" of > the files were similar you would hope that the runtimes would be > similar. No, well, small.txt has very short lines. Lorem.txt is a lorem ipsum, so really long lines. Indeed I get better results tuning chunk_size. Anyway, also with the default value the performance is not bad at all. > >But the time of Linux tail surprise me: > > > >marco at buzz:~$ time tail lorem.txt > >[text] > > > >real 0m0.004s > >user 0m0.003s > >sys 0m0.001s > > > >It's strange that it's so slow. I thought it was because it decodes > >and print the result, but I timed > > You're measuring different things. timeit() tries hard to measure just > the code snippet you provide. It doesn't measure the startup cost of the > whole python interpreter. Try: > > time python3 your-tail-prog.py /home/marco/lorem.txt Well, I'll try it, but it's not a bit unfair to compare Python startup with C? > BTW, does your `tail()` print output? If not, again not measuring the > same thing. > [...] > Also: does tail(1) do character set / encoding stuff? Does your Python > code do that? Might be apples and oranges. Well, as I wrote I also timed timeit.timeit("print(tail('/home/marco/lorem.txt').decode('utf-8'))", globals={"tail":tail}, number=100000) and I got ~36 seconds. > If you have the source of tail(1) to hand, consider getting to the core > and measuring `time()` immediately before and immediately after the > central tail operation and printing the result. IMHO this is a very good idea, but I have to find the time(). Ahah. Emh. From dieter at handshake.de Thu May 19 13:48:45 2022 From: dieter at handshake.de (Dieter Maurer) Date: Thu, 19 May 2022 19:48:45 +0200 Subject: Issue sending data from C++ to Python In-Reply-To: References: Message-ID: <25222.33533.145548.295739@ixdm.fritz.box> Pablo Martinez Ulloa wrote at 2022-5-18 15:08 +0100: >I have been using your C++ Python API, in order to establish a bridge from >C++ to Python. Do you know `cython`? It can help very much in the implementation of bridges between Python and C/C++. From gabriele1NOSPAM at hotmail.com Thu May 19 15:16:37 2022 From: gabriele1NOSPAM at hotmail.com (^Bart) Date: Thu, 19 May 2022 21:16:37 +0200 Subject: Python & nmap References: Message-ID: > Maybe it could be a good idea to look at Ansible for copying the Files > to all the hosts, because that is one thing ansible is made for. I didn't know it... thanks to share it but... I should start to study it and I don't have not enought free time... but maybe in the future I'll do it! :) > For the nmap part: Ansible does not have a module for that (sadly) but > is very extensible, so if you start developing something like that in > Python, you could as well write an ansible module and combine both, > because Ansible itself is written in Python. Ah ok, maybe now I just start to write a bash script because I need to start this work asap, when I'll have one minute I'll try to move the script in Python and after it I could "upload" the work on Ansible! :) > Cheers > > Lars Thanks! ^Bart From gabriele1NOSPAM at hotmail.com Thu May 19 15:21:25 2022 From: gabriele1NOSPAM at hotmail.com (^Bart) Date: Thu, 19 May 2022 21:21:25 +0200 Subject: Python & nmap References: Message-ID: > Opbservations worth considering > 1) could possibly be handled by a simple bash script (My bash skills are > not great So i would probably still go python myself anyway) Like what I wrote in my last reply to another user now I need to start this work asap so maybe I'll start to write a rough bash script and I hope to manage it when I'll have free time on Python! > 2) Instead of checking availability just try to send & react appropriately > if it fails (Ask for forgiveness not permission), the client could fail > after test or during transfer anyway so you will still need this level of > error checking Sadly true... I didn't think about it but maybe I could find a solution in bash script... Thanks for your reply! :) ^Bart From gabriele1NOSPAM at hotmail.com Thu May 19 15:28:42 2022 From: gabriele1NOSPAM at hotmail.com (^Bart) Date: Thu, 19 May 2022 21:28:42 +0200 Subject: Python & nmap References: Message-ID: > You forgot the second line (after 'import nmap' and before 'nm.scan()'): > > nm = nmap.PortScanner() import nmap nm = nmap.PortScanner() nm.scan(hosts='192.168.205.0/24', arguments='-n -sP -PE -PA21,23,80,3389') hosts_list = [(x, nm[x]['status']['state']) for x in nm.all_hosts()] for host, status in hosts_list: print('{0}:{1}'.host) And the result is: Traceback (most recent call last): File "/home/gabriele/Documenti/Python/nmap.py", line 1, in import nmap File "/home/gabriele/Documenti/Python/nmap.py", line 2, in nm = nmap.PortScanner() AttributeError: partially initialized module 'nmap' has no attribute 'PortScanner' (most likely due to a circular import) >>> I'm using the IDLE Shell 3.9.2 on Debian Bullseye+KDE, if I write the script from command line it works! ^Bart From python at mrabarnett.plus.com Thu May 19 18:11:20 2022 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 19 May 2022 23:11:20 +0100 Subject: Python & nmap In-Reply-To: References: Message-ID: On 2022-05-19 20:28, ^Bart wrote: >> You forgot the second line (after 'import nmap' and before 'nm.scan()'): >> >> nm = nmap.PortScanner() > > import nmap > nm = nmap.PortScanner() > nm.scan(hosts='192.168.205.0/24', arguments='-n -sP -PE -PA21,23,80,3389') > hosts_list = [(x, nm[x]['status']['state']) for x in nm.all_hosts()] > for host, status in hosts_list: > print('{0}:{1}'.host) > > And the result is: > > Traceback (most recent call last): > File "/home/gabriele/Documenti/Python/nmap.py", line 1, in > import nmap > File "/home/gabriele/Documenti/Python/nmap.py", line 2, in > nm = nmap.PortScanner() > AttributeError: partially initialized module 'nmap' has no attribute > 'PortScanner' (most likely due to a circular import) > >>> > > I'm using the IDLE Shell 3.9.2 on Debian Bullseye+KDE, if I write the > script from command line it works! > When you installed nmap it would've been installed into site-packages, but the traceback says "/home/gabriele/Documenti/Python/nmap.py", which suggests to me that you called your script "nmap.py", so it's shadowing what you installed and is actually trying to import itself! From cs at cskk.id.au Thu May 19 21:33:29 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Fri, 20 May 2022 11:33:29 +1000 Subject: tail In-Reply-To: References: Message-ID: On 19May2022 19:50, Marco Sulla wrote: >On Wed, 18 May 2022 at 23:32, Cameron Simpson wrote: >> You're measuring different things. timeit() tries hard to measure >> just >> the code snippet you provide. It doesn't measure the startup cost of the >> whole python interpreter. Try: >> >> time python3 your-tail-prog.py /home/marco/lorem.txt > >Well, I'll try it, but it's not a bit unfair to compare Python startup with C? Yes it is. But timeit goes the other way and only measures the code. Admittedly I'd expect a C tail to be pretty quick anyway. But... even a small C programme often has a surprising degree of startup these days, what with dynamicly linked libraries, locale lookups etc etc. Try: strace tail some-empty-file.txt and see what goes on. If you're on slow hard drives what is cached in memory and what isn't can have a surprising effect. Cheers, Cameron Simpson From vinay_sajip at yahoo.co.uk Fri May 20 04:44:14 2022 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 20 May 2022 08:44:14 +0000 (UTC) Subject: ANN: A new version (0.4.9) of python-gnupg has been released. References: <1658613102.995870.1653036254293.ref@mail.yahoo.com> Message-ID: <1658613102.995870.1653036254293@mail.yahoo.com> What Changed? ============= This is an enhancement and bug-fix release, and all users are encouraged to upgrade. Brief summary: * Fixed #161: Added a status attribute to the returned object from gen_key() which ? is set to 'ok' if a key was successfully created, or 'key not created' if that ? was reported by gpg, or None in any other case. * Fixed #164: Provided the ability to add subkeys. Thanks to Daniel Kilimnik for the ? feature request and patch. * Fixed #166: Added keygrip values to the information collected when keys are listed. ? Thanks to Daniel Kilimnik for the feature request and patch. * Fixed #173: Added extra_args to send_keys(), recv_keys() and search_keys() to allow ? passing options relating to key servers. This release [2] has been signed with my code signing key: Vinay Sajip (CODE SIGNING KEY) Fingerprint: CA74 9061 914E AC13 8E66 EADB 9147 B477 339A 9B86 Recent changes to PyPI don't show the GPG signature with the download links. The source code repository is at [1]. An alternative download source where the signatures are available is at [4]. Documentation is available at [5]. As always, your feedback is most welcome (especially bug reports [3], patches and suggestions for improvement, or any other points via this group). Enjoy! Cheers Vinay Sajip [1] https://github.com/vsajip/python-gnupg [2] https://pypi.org/project/python-gnupg/0.4.9 [3] https://github.com/vsajip/python-gnupg/issues [4] https://github.com/vsajip/python-gnupg/releases/ [5] https://docs.red-dove.com/python-gnupg/ From lal at solute.de Fri May 20 05:21:18 2022 From: lal at solute.de (Lars Liedtke) Date: Fri, 20 May 2022 11:21:18 +0200 Subject: Python & nmap In-Reply-To: References: Message-ID: <85008df7-208e-057d-adf4-1548a8ff57f2@solute.de> Ansible has got a shell module, so you could run custom commands on all hosts. But it gets more difficult in parsing the output afterwards. -- Lars Liedtke Software Entwickler Phone: Fax: +49 721 98993- E-mail: lal at solute.de solute GmbH Zeppelinstra?e 15 76185 Karlsruhe Germany Marken der solute GmbH | brands of solute GmbH billiger.de | Shopping.de Gesch?ftsf?hrer | Managing Director: Dr. Thilo Gans, Bernd Vermaaten Webseite | www.solute.de Sitz | Registered Office: Karlsruhe Registergericht | Register Court: Amtsgericht Mannheim Registernummer | Register No.: HRB 110579 USt-ID | VAT ID: DE234663798 Informationen zum Datenschutz | Information about privacy policy http://solute.de/ger/datenschutz/grundsaetze-der-datenverarbeitung.php Am 19.05.22 um 21:16 schrieb ^Bart: >> Maybe it could be a good idea to look at Ansible for copying the >> Files to all the hosts, because that is one thing ansible is made for. > > I didn't know it... thanks to share it but... I should start to study > it and I don't have not enought free time... but maybe in the future > I'll do it! :) > >> For the nmap part: Ansible does not have a module for that (sadly) >> but is very extensible, so if you start developing something like >> that in Python, you could as well write an ansible module and combine >> both, because Ansible itself is written in Python. > > Ah ok, maybe now I just start to write a bash script because I need to > start this work asap, when I'll have one minute I'll try to move the > script in Python and after it I could "upload" the work on Ansible! :) > >> Cheers >> >> Lars > > Thanks! > ^Bart > From loris.bennett at fu-berlin.de Fri May 20 06:51:08 2022 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Fri, 20 May 2022 12:51:08 +0200 Subject: Python & nmap References: <85008df7-208e-057d-adf4-1548a8ff57f2@solute.de> Message-ID: <87ee0oo4tf.fsf@hornfels.zedat.fu-berlin.de> Lars Liedtke writes: > Ansible has got a shell module, so you could run custom commands on all > hosts. But it gets more difficult in parsing the output afterwards. If you just want to copy files, pdsh[1] or clush[2] might be enough. Cheers, Loris Footnotes: [1] https://github.com/chaos/pdsh [2] https://clustershell.readthedocs.io/en/latest/tools/clush.html -- This signature is currently under construction. From torriem at gmail.com Fri May 20 21:21:13 2022 From: torriem at gmail.com (Michael Torrie) Date: Fri, 20 May 2022 19:21:13 -0600 Subject: "py" command for Linux and Mac? In-Reply-To: References: Message-ID: On 5/12/22 11:59, De ongekruisigde wrote: > On 2022-05-12, Mats Wichmann wrote: >> On 5/12/22 10:25, Dan Stromberg wrote: >>> Hi folks. >>> >>> I heard there's a Windows-like "py" command for Linux (and Mac?). >>> >>> I'm finally getting to porting a particular project's Python 2.7 code to >>> 3.x, and one of the first steps will probably be changing a lot of "python2 >>> script.py" to use #!/usr/bin/env python2 and chmod +x. Then we can update >>> the scripts one at a time to use #!/usr/bin/env python3. >>> >>> However, would this be Linux-and-Mac-only? I'm not at all sure this code >>> will ever move to Windows, but in case it does, would a "py" command work >>> on all 3 if I use #!/usr/bin/env py? >> >> The py command (python lanucher) respects shebang lines. > > Linux by itself respects shebang lines, so you don't need a separate > launcher program. Just put e.g.: Dan knows this already. His question is about whether the shebang should instead refer to a py launcher so that this script will run on Windows or Linux. And of course the answer given by the grandparent is that Dan should use a normal linux shebang line in his scripts and on Windows the py launcher will read that shebang and guestimate the proper python interpreter to use and execute the script with that. Thus if I'm reading this correctly, a Linux shebang line should function as expected on Windows when python files are associated and launched with the py.exe launcher, even though there's no such thing as /usr/bin/python3 on Windows. Py launcher makes it work as if there was. From rosuav at gmail.com Fri May 20 21:50:06 2022 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 21 May 2022 11:50:06 +1000 Subject: "py" command for Linux and Mac? In-Reply-To: References: Message-ID: On Sat, 21 May 2022 at 11:22, Michael Torrie wrote: > And of course the answer given by the grandparent is that Dan should use > a normal linux shebang line in his scripts and on Windows the py > launcher will read that shebang and guestimate the proper python > interpreter to use and execute the script with that. Thus if I'm reading > this correctly, a Linux shebang line should function as expected on > Windows when python files are associated and launched with the py.exe > launcher, even though there's no such thing as /usr/bin/python3 on > Windows. Py launcher makes it work as if there was. > That's correct, and when the py.exe launcher was first, well, launched, the main thrust of it was "it uses the shebang that you already include for the sake of Unix systems". You don't need extra directives to tell it what to do. ChrisA From o1bigtenor at gmail.com Sat May 21 08:19:58 2022 From: o1bigtenor at gmail.com (o1bigtenor) Date: Sat, 21 May 2022 07:19:58 -0500 Subject: Request for assistance (hopefully not OT) In-Reply-To: References: Message-ID: On Tue, May 17, 2022 at 6:20 AM o1bigtenor wrote: > > Greetings > > I was having space issues in my /usr directory so I deleted some > programs thinking that the space taken was more an issue than having > older versions of the program. > Found the responses to my request quite interesting - - - actually fascinating. To me a computer has to date been a tool - - - - not a tool kit - - - - nor a place to spend a life. This means that I have had to rely on other more knowledgeable with the 'under the hood' material for advice. Have found that in the computer world - - - - and especially the 'Linux' world that the idea of assisting a requester, well - - - it seems to be a foreign concept. To wit - - - in this thread 50% of the posts were of the variety - - - why would you do such a stupid thing. 40% of the posts had some suggestions for at the least avenues of inquiry and only 1 post gave a carefully written response that not only gave ideas but suggested a possible process to finding a solution. It was necessary to manually install (download from the repository and then using dpkg -i to install) all of the necessary bits to each version that was listed in the complaint. I tried to use apt but that option did not produce useful results. This was a lot more involved that one might think as there is not only the pythonx.xx version but pythonx.xx-minimal and -dev and sometimes even -dbg. The order of install was also important. So even though the request for assistance was to much met by kvetching there was enough 'help' so that it was possible to 'fix' the issue. Thanks and kudus to those that did 'help' and those who found the kvetching more useful - - - - well - - - - I don't have to wonder why 'linux' is used as much by the general populace as it is. The community likes to destroy itself - - - it is a pity - - - - the community has so much to offer. Regards From torriem at gmail.com Sat May 21 10:24:50 2022 From: torriem at gmail.com (Michael Torrie) Date: Sat, 21 May 2022 08:24:50 -0600 Subject: Request for assistance (hopefully not OT) In-Reply-To: References: Message-ID: <0af858f3-ac0c-898a-da89-1cdf980cca20@gmail.com> On 5/21/22 06:19, o1bigtenor wrote: > more useful - - - - well - - - - I don't have to wonder why 'linux' is > used as much > by the general populace as it is. The community likes to destroy > itself - - - it > is a pity - - - - the community has so much to offer. As far as community goes, the Linux community (whatever that might refer to) is pretty typical of all communities, including communities that surround proprietary systems like Windows. For those that realize that communication is two-way and individual effort is required, the community is a wonderful resource of help and support. For those that approach it with impatience and demands for support without evidence of individual effort, community members respond with much less alacrity. This is true of *all* communities of all types. I think in the Windows world people don't seem to have as many community problems because most people simply aren't a part of the community--the most impatient, grumpy people seem to have enough young relatives they can coax to solve their problems for them. From skip.montanaro at gmail.com Fri May 20 12:25:57 2022 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Fri, 20 May 2022 11:25:57 -0500 Subject: Now for something completely different... Message-ID: This isn't really Python-related, but I don't wander around any more general software engineering/computer science places on the web, so I will throw this out here. I start a framebuilding class next week (lugged steel construction for those into bikes). After about three weeks of work, I should have a custom-made frame, which will also happen to have been built by me. It will need paint or powder coat and graphics. I'm looking for ideas about the graphics. Though I spent most of my professional life noodling around with Python (I'm now retired), I don't really want the graphics to be terribly Python-centric. Consequently, "import this" is probably not going to work. I have decided to go ahead and use my first name in lower case Courier as the core piece of the downtube graphic: skip That's not too informative (other than its relationship to moi), and I have room for probably four or five more characters. (I have a graphic artist in mind, so the space need not strictly be text either.) So, how would you fill the remaining space? I've considered "n++" or "n+1", but not thought of anything more than that. For those who are not into bikes, you may not have heard of the Velominati Rules . I quote Rule #12: *The correct number of bikes to own is n+1.* While the minimum number of bikes one should own is three, the correct number is n+1, where n is the number of bikes currently owned. This equation may also be re-written as s-1, where s is the number of bikes owned that would result in separation from your partner. I currently have six to ten, depending on how demanding you are that they be ready to ride with nothing more than air in the tires. This project, while definitely n+1 might well also be approaching s-1. :-) Ideas about head tube graphics are also welcome. I have no idea there at all yet. Again, some subtle reference to software engineering would be nice. So, if you'd like to play along, toss out some ideas. I'll consider everything as long as it's not NSFW. Skip From mmontgomery at levado.to Sat May 21 07:53:02 2022 From: mmontgomery at levado.to (Meredith Montgomery) Date: Sat, 21 May 2022 08:53:02 -0300 Subject: how to distinguish return from print() Message-ID: <86sfp3p0f5.fsf@levado.to> Students seeing a programming language for the first time and using Python's REPL for the first time have asked me what is the difference between a return statement and a print() call. They got a point. --8<---------------cut here---------------start------------->8--- def f(x): return x + 1 >>> f(1) 2 >>> print("hello") hello --8<---------------cut here---------------end--------------->8--- There's no visual difference. (*) My answer The way I know how to answer the difference there is to explain how a typical computer system works, which is a more or less long answer. A return statement is usually just another CPU instruction, while /print/ is a procedure that must eventually run a CPU instruction that interrupts the CPU, passes control to the operating system which, in turn, talks to the i/o devices in question (the video, for example) to get the job finally done. (We may expect a print() call, therefore, to be usually much slower than a return statement.) I also explain that if, say, math.sin(x) is going to print a message to the screen... >>> sin(pi/2) calculating your sine... hang on... 1.0 ... I might as well not use it because this will get mixed with my own print() statements and I'll be surprised about who added that calculating-your-sine message. So adding print() statements to procedures is, in practice, rare. (To them it's the most typical operation because they never do any serious programming and they learn their first steps out on the Internet. In my course, print() is only allowed after at 10th homework, after 10 weeks.) I also explain that if f(x) prints its answer to the screen instead of returning it, it will be harder to capture the answer in a variable. For example, >>> def f(x): ... print(x + 1) ... >>> f(1) 2 >>> y = f(1) 2 >>> y >>> >>> y == None True I take this opportunity to remark that Python seems to always returns something, even if it's the ``special'' value None. (*) The REPL I also explain that the REPL is just another program. Its purpose happens to be to [r]ead, [e]val, [p]rint and [l]oop, which is why we get to see return values printed to the screen. (*) My insatisfaction I wish I had simpler answers. Explaining about operating systems, the CPU, system calls... I wish there was an easier way to clarify such things. You guys might have ideas. I'm thinking about showing them the code for the simplest toy-REPL so that we can perhaps demystify the REPL. I think they see the REPL as part of the programming language, so I think it might help to figure out that nothing happens unless code is written to make it happen. If something gets displayed on the screen, there is definitely some i/o going on and print() is the one initiating that i/o, while return is just another ``arithmetic'' operation such as addition. Thank you so much. From rwloomis at fastmail.fm Sat May 21 11:14:36 2022 From: rwloomis at fastmail.fm (Robert Loomis) Date: Sat, 21 May 2022 11:14:36 -0400 Subject: Fwd: Could not load correctly In-Reply-To: <1da32579-0b74-284d-29e9-14bba9a6c6b5@fastmail.fm> References: <1da32579-0b74-284d-29e9-14bba9a6c6b5@fastmail.fm> Message-ID: <4795873a-73ff-0838-7af3-ea8f02841ceb@fastmail.fm> -------- Forwarded Message -------- Subject: Could not load correctly Date: Sat, 21 May 2022 10:58:39 -0400 From: Robert Loomis Reply-To: Bob at LoomisEngineering.com To: python-list at python.org I am new to python.I tried to download it to a virtual environment since I have been learning on another version. I downloaded version 3.10.4 into my windows10 operating system into directory c:\Users\Bob\PyVer\Py3913 and it said it was successful.I went to C:\Users\Bob\PyProj and made my environment by c:\Users\Bob\PyVer\Py3913\python -m venv my_env.I then activated it by my_env\Scripts\activate and it came back with a prompt.Then I tried to test it by typing python and I got what is below. What did I do wrong? Thank you, Bob Loomis -- This email has been checked for viruses by AVG. https://www.avg.com From hongyi.zhao at gmail.com Sat May 21 20:37:58 2022 From: hongyi.zhao at gmail.com (hongy...@gmail.com) Date: Sat, 21 May 2022 17:37:58 -0700 (PDT) Subject: Convert the decimal numbers expressed in a `numpy.ndarray` into a matrix representing elements in fractional form In-Reply-To: References: <5eb0b494-ec7a-4b6c-9966-078889dbc5aen@googlegroups.com> Message-ID: On Thursday, May 19, 2022 at 5:26:25 AM UTC+8, Cousin Stanley wrote: > #!/usr/bin/env python3 > > ''' > NewsGroup .... comp.lang.python > > Subject ...... Convert the decimal numbers > expressed in a numpy.ndarray > into a matrix representing elements > in fractiona > Date ......... 2022-05-16 > > Post_By ...... hongy... > > Edit_By ...... Stanley C. Kitching > ''' > import numpy as np > > from fractions import Fraction > b = [ > [ 0.0 , -1.0 , 0.0 , 0.25 ] , > [ 1.0 , 0.0 , 0.0 , 0.25 ] , > [ 0.0 , 0.0 , 1.0 , 0.25 ] , > [ 0.0 , 0.0 , 0.0 , 1.0 ] ] > > a = [ ] > > print( '\n b .... \n' ) > > for row in b : > arow = [] > print( ' ' , row ) > > for dec_x in row : > frac_x = Fraction( dec_x ) > arow.append( frac_x ) > > a.append( arow ) > > > # using f-string format > > print( '\n a .... \n' ) > > for row in a : > > for item in row : > > print( f' {item} ' , end = '' ) > > print() > > # ------------------------------------------ This method doesn't work, as shown below: b .... [0.0, -1.0, 0.0, 0.25] [1.0, 0.0, 0.0, 0.25] [0.0, 0.0, 1.0, 0.25] [0.0, 0.0, 0.0, 1.0] a .... 0 0 0 1 > -- > Stanley C. Kitching > Human Being > Phoenix, Arizona From rosuav at gmail.com Sun May 22 19:30:07 2022 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 23 May 2022 09:30:07 +1000 Subject: Now for something completely different... In-Reply-To: References: Message-ID: On Mon, 23 May 2022 at 09:19, Skip Montanaro wrote: > That's not too informative (other than its relationship to moi), and I have > room for probably four or five more characters. (I have a graphic artist in > mind, so the space need not strictly be text either.) Aww, not enough room to say "straight line", because (in Euclidean space) it's the fastest way from B to A. ChrisA From rosuav at gmail.com Sun May 22 19:33:56 2022 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 23 May 2022 09:33:56 +1000 Subject: how to distinguish return from print() In-Reply-To: References: <86sfp3p0f5.fsf@levado.to> Message-ID: On Mon, 23 May 2022 at 09:23, Stefan Ram wrote: > You are making it extra hard by wording the question in this > way. "What's the difference between the moon and liberty?". Uh ... > > It's much easier to explain the moon and liberty separately. "You can't tell the difference between a lump on the head and margarine. The leadership of the Conservative Party is yours for the asking!" -- Grytpype Thynne, "The Last Goon Show of All" ChrisA From python at mrabarnett.plus.com Sun May 22 21:13:07 2022 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 23 May 2022 02:13:07 +0100 Subject: Now for something completely different... In-Reply-To: References: Message-ID: On 2022-05-23 00:30, Chris Angelico wrote: > On Mon, 23 May 2022 at 09:19, Skip Montanaro wrote: >> That's not too informative (other than its relationship to moi), and I have >> room for probably four or five more characters. (I have a graphic artist in >> mind, so the space need not strictly be text either.) > > Aww, not enough room to say "straight line", because (in Euclidean > space) it's the fastest way from B to A. > Instead of "straight line", "A ? B". From mats at wichmann.us Mon May 23 09:36:33 2022 From: mats at wichmann.us (Mats Wichmann) Date: Mon, 23 May 2022 07:36:33 -0600 Subject: Fwd: Could not load correctly In-Reply-To: <4795873a-73ff-0838-7af3-ea8f02841ceb@fastmail.fm> References: <1da32579-0b74-284d-29e9-14bba9a6c6b5@fastmail.fm> <4795873a-73ff-0838-7af3-ea8f02841ceb@fastmail.fm> Message-ID: On 5/21/22 09:14, Robert Loomis wrote: > > > > -------- Forwarded Message -------- > Subject:???? Could not load correctly > Date:???? Sat, 21 May 2022 10:58:39 -0400 > From:???? Robert Loomis > Reply-To:???? Bob at LoomisEngineering.com > To:???? python-list at python.org > > > > I am new to python.I tried to download it to a virtual environment since > I have been learning on another version. I downloaded version 3.10.4 > into my windows10 operating system into directory > c:\Users\Bob\PyVer\Py3913 and it said it was successful.I went to > C:\Users\Bob\PyProj and made my environment by > c:\Users\Bob\PyVer\Py3913\python -m venv my_env.I then activated it by > my_env\Scripts\activate and it came back with a prompt.Then I tried to > test it by typing python and I got what is below. > > What did I do wrong? there's nothing below. If you tried to send a screenshot it must have gotten stripped by the mailing list. We generally hate screenshots anyway, because you can't cut and paste code from them. From ojomooluwatolami675 at gmail.com Mon May 23 15:36:12 2022 From: ojomooluwatolami675 at gmail.com (Tola Oj) Date: Mon, 23 May 2022 20:36:12 +0100 Subject: oop issue Message-ID: i am trying to print this code but it keeps giving me this typeerror, please help. the csv file format i am trying to change into a list is in a different module. class invest_crypto: crypto_current_rate = 0.05 client_list = [] def __init__(self, name, surname, amount_Deposited, amount_to_transfer): self.name = name self.surname = surname self.amount_Deposited = amount_Deposited self.amount_to_transfer = amount_to_transfer invest_crypto.client_list.append(self) def calculate_customer_transfer(self): self.customer_transfer = (self.crypto_current_rate * self. amount_Deposited) + self.amount_Deposited return self.customer_transfer @classmethod def access_client_details(cls): with open('C:\\Users\\ojomo\\OneDrive\\Desktop\\myexcel\\ oop_learn.py\\myExperiment.py\\clientDetails.csv', 'r' ) as f: reader = csv.DictReader(f) clientDetails = list(reader) for item in clientDetails: invest_crypto( name=item.get('name'), surname=item.get('surname'), amount_Deposited=item.get('amount_deposited'), amount_to_transfer=item.get('amount_to_transfer') ) @staticmethod def __repr__(self): return f"('{self.name}', '{self.surname}', '{self.amount_Deposited}', '{self.amount_to_transfer}')" invest_crypto.access_client_details() print(invest_crypto.client_list()) From python at mrabarnett.plus.com Mon May 23 16:11:05 2022 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 23 May 2022 21:11:05 +0100 Subject: oop issue In-Reply-To: References: Message-ID: <94a07924-3fd3-8bec-f592-7e21b14191ec@mrabarnett.plus.com> On 2022-05-23 20:36, Tola Oj wrote: > i am trying to print this code but it keeps giving me this typeerror, > please help. the csv file format i am trying to change into a list is in a > different module. > > class invest_crypto: > crypto_current_rate = 0.05 > client_list = [] > def __init__(self, name, surname, amount_Deposited, amount_to_transfer): > self.name = name > self.surname = surname > self.amount_Deposited = amount_Deposited > self.amount_to_transfer = amount_to_transfer > > invest_crypto.client_list.append(self) > > def calculate_customer_transfer(self): > self.customer_transfer = (self.crypto_current_rate * self. > amount_Deposited) + self.amount_Deposited > return self.customer_transfer > > @classmethod > def access_client_details(cls): > with open('C:\\Users\\ojomo\\OneDrive\\Desktop\\myexcel\\ > oop_learn.py\\myExperiment.py\\clientDetails.csv', 'r' ) as f: > reader = csv.DictReader(f) > clientDetails = list(reader) > > for item in clientDetails: > invest_crypto( > name=item.get('name'), > surname=item.get('surname'), > amount_Deposited=item.get('amount_deposited'), > amount_to_transfer=item.get('amount_to_transfer') > ) > @staticmethod > def __repr__(self): > return f"('{self.name}', '{self.surname}', '{self.amount_Deposited}', > '{self.amount_to_transfer}')" > > > invest_crypto.access_client_details() > print(invest_crypto.client_list()) "this typeerror"? What type error? You haven't shown the traceback. I'm guessing that it's complaining about the last line, where you're calling 'client_list', which is a list. Don't call something that's not callable! The last line should be: print(invest_crypto.client_list) (I'm not going to mention the way you're creating instances that append themselves onto a list that's on the class; that's just weird, IMHO...) From ojomooluwatolami675 at gmail.com Mon May 23 16:54:28 2022 From: ojomooluwatolami675 at gmail.com (Tola Oj) Date: Mon, 23 May 2022 21:54:28 +0100 Subject: oop issue Message-ID: i just finished learning oop as a beginner and trying to practice with it but i ran into this typeerror issue, help please. Traceback (most recent call last): File "c:\Users\ojomo\OneDrive\Desktop\myexcel\oop_learn.py\myExperiment.py\mainMain.py", line 36, in print(invest_crypto.client_list) TypeError: invest_crypto.__repr__() missing 1 required positional argument: 'self' this is my code below: import csv class invest_crypto: crypto_current_rate = 0.05 client_list = [] def __init__(self, name, surname, amount_Deposited, amount_to_transfer): self.name = name self.surname = surname self.amount_Deposited = amount_Deposited self.amount_to_transfer = amount_to_transfer invest_crypto.client_list.append(self) def calculate_customer_transfer(self): self.customer_transfer = (self.crypto_current_rate * self. amount_Deposited) + self.amount_Deposited return self.customer_transfer @classmethod def access_client_details(cls): with open('C:\\Users\\ojomo\\OneDrive\\Desktop\\myexcel\\ oop_learn.py\\myExperiment.py\\clientDetails.csv', 'r' ) as f: reader = csv.DictReader(f) clientDetails = list(reader) for item in clientDetails: invest_crypto( name=item.get('name'), surname=item.get('surname'), amount_Deposited=item.get('amount_deposited'), amount_to_transfer=item.get('amount_to_transfer') ) @staticmethod def __repr__(self): return f"('{self.name}', '{self.surname}', '{self.amount_Deposited}', '{self.amount_to_transfer}')" invest_crypto.access_client_details() print(invest_crypto.client_list) From cousinstanley at gmail.com Mon May 23 15:20:29 2022 From: cousinstanley at gmail.com (Cousin Stanley) Date: Mon, 23 May 2022 12:20:29 -0700 Subject: Convert the decimal numbers expressed in a `numpy.ndarray` into a matrix representing elements in fractional form In-Reply-To: References: <5eb0b494-ec7a-4b6c-9966-078889dbc5aen@googlegroups.com> Message-ID: > hongy... wrote .... > > This method doesn't work, as shown below: > ? b .... > > [0.0, -1.0, 0.0, 0.25] > [1.0, 0.0, 0.0, 0.25] > [0.0, 0.0, 1.0, 0.25] > [0.0, 0.0, 0.0, 1.0] > > a .... > > 0 0 0 1 > # ----------------------------------- Using .... debian 11.3 bullseye python 3.9 numpy 1,21,5 Code as I posted in my reply dated 2022-05-18 $ python3 np_array_to_fractions_2.py b .... [0.0, -1.0, 0.0, 0.25] [1.0, 0.0, 0.0, 0.25] [0.0, 0.0, 1.0, 0.25] [0.0, 0.0, 0.0, 1.0] a .... 0 -1 0 1/4 1 0 0 1/4 0 0 1 1/4 0 0 0 1 -- Stanley C. Kitching Human Being Phoenix, Arizona From __peter__ at web.de Mon May 23 18:35:21 2022 From: __peter__ at web.de (Peter Otten) Date: Tue, 24 May 2022 00:35:21 +0200 Subject: oop issue In-Reply-To: References: Message-ID: <61ae9daf-5073-7c2d-58c2-e8f5d2fc791e@web.de> On 23/05/2022 22:54, Tola Oj wrote: > i just finished learning oop as a beginner and trying to practice with it > but i ran into this typeerror issue, help please. > > Traceback (most recent call last): > File > "c:\Users\ojomo\OneDrive\Desktop\myexcel\oop_learn.py\myExperiment.py\mainMain.py", > line 36, in > print(invest_crypto.client_list) > TypeError: invest_crypto.__repr__() missing 1 required positional argument: > 'self' > @staticmethod > def __repr__(self): > return f"('{self.name}', '{self.surname}', '{self.amount_Deposited}', > '{self.amount_to_transfer}')" What are you trying to achieve with the staticmethod decorator? From avigross at verizon.net Mon May 23 19:57:41 2022 From: avigross at verizon.net (Avi Gross) Date: Mon, 23 May 2022 23:57:41 +0000 (UTC) Subject: oop issue In-Reply-To: References: Message-ID: <286421301.1131011.1653350261076@mail.yahoo.com> invest_crypto.client_list.append(self) I am wondering about the phrasing above. When you are in the dunder init function, you normally create and change items in YOURSELF so why is?your code not changing self.crypto_client_list? And what are you appending to before ever creating it? Would it kill you to create some form of?container in the main class definition initialized appropriately to some variation of empty? I won't claim to fully understand what the code wants to do. Adding yourself to the list of clients?may make sense to you but if you understand object oriented programming, you may have an?inkling that objects need to be CREATED somewhere before they can be used. Most of your code?looks like it is DEFINING an object. The last few lines try to access a method in an object that has never?been instantiated. Yes, you do have a way to store methods in a class and call them without any?objects but this is not a case like that. You need something like "myobj =?invest_crypto(args)" and then the rest of your code can do changes?and calculations and perhaps create other similar or different objects. You seem to be using a method that?reads in a file and dumps a version of the contents and that might work if you did a line like this next:?"myobj.access_client_details()" albeit not how I would name it or do it. And, of course, your print() again names the class, not an instance of a class.? Your code wraps in a few places and I wonder if it contains errors as in this set of lines: ? ? ? ? return f"('{self.name}', '{self.surname}', '{self.amount_Deposited}', ? ? ? ? '{self.amount_to_transfer}')" Is that really how you think you set up a formatted string?? Even if you get that working, how does the print statement know what to do with a list of objects? Some might design a method you can call to print the contents of a list which would loop over?the list. But is there ever more than one client (yourself) in the list? The above rambling is just reflecting my opinion that you have not learned enough or thought it?through and may even be copying and modifying various snippets of code perhaps from?places where it works to a place that might be better designed from scratch. Have fun. As someone else mentioned, smaller more focused examples may work better to get?you up to speed, but then again, we often find out someone is given a homework assignment ... -----Original Message----- From: Tola Oj To: python-list at python.org Sent: Mon, May 23, 2022 4:54 pm Subject: oop issue i just finished learning oop as a beginner and trying to practice with it but i ran into this typeerror issue, help please. Traceback (most recent call last): ? File "c:\Users\ojomo\OneDrive\Desktop\myexcel\oop_learn.py\myExperiment.py\mainMain.py", line 36, in ? ? print(invest_crypto.client_list) TypeError: invest_crypto.__repr__() missing 1 required positional argument: 'self' this is my code below: import csv class invest_crypto: ? ? crypto_current_rate = 0.05 ? ? client_list = [] ? ? def __init__(self, name, surname, amount_Deposited, amount_to_transfer): ? ? ? ? self.name = name ? ? ? ? self.surname = surname ? ? ? ? self.amount_Deposited = amount_Deposited ? ? ? ? self.amount_to_transfer = amount_to_transfer ? ? ? ? invest_crypto.client_list.append(self) ? ? def calculate_customer_transfer(self): ? ? ? ? self.customer_transfer = (self.crypto_current_rate * self. amount_Deposited) + self.amount_Deposited ? ? ? ? return self.customer_transfer ? ? @classmethod ? ? def access_client_details(cls): ? ? ? ? with open('C:\\Users\\ojomo\\OneDrive\\Desktop\\myexcel\\ oop_learn.py\\myExperiment.py\\clientDetails.csv', 'r' ) as f: ? ? ? ? ? ? reader = csv.DictReader(f) ? ? ? ? ? ? clientDetails = list(reader) ? ? ? ? ? ? for item in clientDetails: ? ? ? ? ? ? ? ? invest_crypto( ? ? ? ? ? ? ? ? ? ? name=item.get('name'), ? ? ? ? ? ? ? ? ? ? surname=item.get('surname'), ? ? ? ? ? ? ? ? ? ? amount_Deposited=item.get('amount_deposited'), ? ? ? ? ? ? ? ? ? ? amount_to_transfer=item.get('amount_to_transfer') ? ? ? ? ? ? ) ? ? @staticmethod ? ? def __repr__(self): ? ? ? ? return f"('{self.name}', '{self.surname}', '{self.amount_Deposited}', '{self.amount_to_transfer}')" invest_crypto.access_client_details() print(invest_crypto.client_list) -- https://mail.python.org/mailman/listinfo/python-list From kevinmwilson1956 at yahoo.com Tue May 24 17:14:29 2022 From: kevinmwilson1956 at yahoo.com (Kevin M. Wilson) Date: Tue, 24 May 2022 21:14:29 +0000 (UTC) Subject: F-string usage in a print() References: <104490624.2506310.1653426869775.ref@mail.yahoo.com> Message-ID: <104490624.2506310.1653426869775@mail.yahoo.com> future_value = 0 for i in range(years): # for i in range(months): future_value += monthly_investment future_value = round(future_value, 2) # monthly_interest_amount = future_value * monthly_interest_rate # future_value += monthly_interest_amount # display the result print(f"Year = ", years + f"Future value = \n", future_value)When joining a string with a number, use an f-string otherwise, code a str() because a implicit convert of an int to str causes a TypeError!Well...WTF! Am I not using the f-string function correctly...in the above line of code??? Caddy Man Good sense makes one slow to anger, and it is his glory tooverlook an offense. Proverbs 19:11 From python at mrabarnett.plus.com Tue May 24 17:54:10 2022 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 24 May 2022 22:54:10 +0100 Subject: F-string usage in a print() In-Reply-To: <104490624.2506310.1653426869775@mail.yahoo.com> References: <104490624.2506310.1653426869775.ref@mail.yahoo.com> <104490624.2506310.1653426869775@mail.yahoo.com> Message-ID: On 2022-05-24 22:14, Kevin M. Wilson via Python-list wrote: > future_value = 0 > for i in range(years): > # for i in range(months): > future_value += monthly_investment > future_value = round(future_value, 2) > # monthly_interest_amount = future_value * monthly_interest_rate > # future_value += monthly_interest_amount > # display the result > print(f"Year = ", years + f"Future value = \n", future_value)When joining a string with a number, use an f-string otherwise, code a str() because a implicit convert of an int to str causes a TypeError!Well...WTF! Am I not using the f-string function correctly...in the above line of code??? > There's no implicit conversion. An f-string always gives you a string. 'years' is an int, f"Future value = \n" is a str, and you can't add an int and a str. Maybe you meant this: print(f"Year = {i}, Future value = {future_value}") or this: print(f"Year = {i + 1}, Future value = {future_value}") These are equivalent to: print("Year = {}, Future value = {}".format(i, future_value)) and: print("Year = {}, Future value = {}".format(i + 1, future_value)) From pbryan at anode.ca Tue May 24 17:54:36 2022 From: pbryan at anode.ca (Paul Bryan) Date: Tue, 24 May 2022 14:54:36 -0700 Subject: F-string usage in a print() In-Reply-To: <104490624.2506310.1653426869775@mail.yahoo.com> References: <104490624.2506310.1653426869775.ref@mail.yahoo.com> <104490624.2506310.1653426869775@mail.yahoo.com> Message-ID: <12c0205eab2235ab8af778b40d6444a0c58e2d55.camel@anode.ca> Try something like: print(f"Year = {years}, Future value = {future_value}") On Tue, 2022-05-24 at 21:14 +0000, Kevin M. Wilson via Python-list wrote: > future_value = 0 > for i in range(years): > # for i in range(months): > ?? future_value += monthly_investment > ?? future_value = round(future_value, 2) > ?? # monthly_interest_amount = future_value * monthly_interest_rate > ?? # future_value += monthly_interest_amount > ?? # display the result > ?? print(f"Year = ", years + f"Future value = \n", future_value)When > joining a string with a number, use an f-string otherwise, code a > str() because a implicit convert of an int to str causes a > TypeError!Well...WTF! Am I not using the f-string function > correctly...in the above line of code??? > Caddy Man > > Good sense makes one slow to anger, and it is his glory tooverlook an > offense. > > Proverbs 19:11 > From mats at wichmann.us Tue May 24 18:21:24 2022 From: mats at wichmann.us (Mats Wichmann) Date: Tue, 24 May 2022 16:21:24 -0600 Subject: F-string usage in a print() In-Reply-To: <104490624.2506310.1653426869775@mail.yahoo.com> References: <104490624.2506310.1653426869775.ref@mail.yahoo.com> <104490624.2506310.1653426869775@mail.yahoo.com> Message-ID: On 5/24/22 15:14, Kevin M. Wilson via Python-list wrote: > future_value = 0 > for i in range(years): > # for i in range(months): > future_value += monthly_investment > future_value = round(future_value, 2) > # monthly_interest_amount = future_value * monthly_interest_rate > # future_value += monthly_interest_amount > # display the result > print(f"Year = ", years + f"Future value = \n", future_value)When joining a string with a number, use an f-string otherwise, code a str() because a implicit convert of an int to str causes a TypeError!Well...WTF! Am I not using the f-string function correctly...in the above line of code??? As noted elsewhere, your f-strings by themselves are fine, it isn't complaining about those, though they're kind of silly as they don't do any formatting that justifies entering them as f-strings. It's complaining about this piece: years + f"Future value = \n" which is the second of the three comma-separated argument to print(). That's the int + string the error is grumping about. From cs at cskk.id.au Tue May 24 18:12:09 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Wed, 25 May 2022 08:12:09 +1000 Subject: F-string usage in a print() In-Reply-To: <104490624.2506310.1653426869775@mail.yahoo.com> References: <104490624.2506310.1653426869775@mail.yahoo.com> Message-ID: On 24May2022 21:14, Kevin M. Wilson wrote: >future_value = 0 >for i in range(years): ># for i in range(months): > future_value += monthly_investment > future_value = round(future_value, 2) > # monthly_interest_amount = future_value * monthly_interest_rate > # future_value += monthly_interest_amount > # display the result > print(f"Year = ", years + f"Future value = \n", future_value) > >When joining a string with a number, use an f-string otherwise, code a >str() because a implicit convert of an int to str causes a >TypeError!Well...WTF! Am I not using the f-string function >correctly...in the above line of code??? The short answer is that you cannot add (the "+" operator) a string to an integer in Python because one is a numeric value and one is a text value. You would need to convert the int to a str first if you really wanted to. But in your code above, you don't need to. Let's pick apart that print(): print( f"Year = ", years + f"Future value = \n", future_value ) Like any function, print()'s arguments are separate by commas. So you've got 3 expressions there. The problematic expression looks like this one: years + f"Future value = \n" You didn't supply a complete runnable example so we don't see "years" get initialised. However, I assume it is an int because range() only accepts ints. You can't add an int to a str. Now, print() converts all its arguments to strings, so there's no need for f-strings anywhere in this. I'd go: print("Year =", years, "Future value =", future_value) myself. If you really want that newline, maybe: print("Year =", years, "Future value =") print(future_value) Format strings are for embedding values inside strings, which you don't need to do in your code as written. And whether you wrote: years + f"Future value = \n" or: years + "Future value = \n" (because that string has no embedded values) it is still wrong because years is an int and the string is still a str. You _could_ use an f-string to compute the whole print line in one go: print(f"Year = {years} Future value = \n{future_value}") but I can't see the point, personally. I've code from people who prefer that form though. BTW, it helps people to help you if you supply a complete example. Your example code did not initialise years or monthly_investment, and so will not run for someone else. It also helps to winnow it down to the smallest example you can which still shows the problem. FOr yoru example you could have reduced things to this, perhaps: years = 9 years + f"Future value = \n" Cheers, Cameron Simpson From cs at cskk.id.au Tue May 24 23:15:34 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Wed, 25 May 2022 13:15:34 +1000 Subject: F-string usage in a print() In-Reply-To: <655708510.2586142.1653437605039@mail.yahoo.com> References: <655708510.2586142.1653437605039@mail.yahoo.com> Message-ID: On 25May2022 00:13, Kevin M. Wilson wrote: >Cameron, I have a misunderstanding here, the 'f-string' is used when >the str() is not...isn't it! No, f-strings (format strings) are just a convenient way to embed values in a string. The result is a string. In days of yore the common formatting method was the % operator like C's printf() format. You could do stuff like this: print("%s: length is %d items" % (thing, len(thing))) which would print something like: Thing5: length is 12 items Ignore the print() itself, that's just how you might use this: format a string with 2 values, then print it out for a person to read. You can do various things with %-formatting, but it is pretty limited and a bit prone to pairing things up incorrectly (human error making the "(thing, len(thing,len(thing)))" tuple) because those values are separates from the string itself. The funky new thing is format strings, where you can write: print(f"{thing}: length is {len(thing)} items") Again ignore the print(), we're really just interested in the expression: f"{thing}: length is {len(thing)} items" which does the same thing as the %-format earlier, but more readably and conveniently. I suspect you think they have another purpose. Whereas I suspect you're actaully trying to do something inappropriate for a number. If you've come here from another language such as PHP you might expect to go: print(some_number + some_string) In Python you need compatible types - so many accidents come from mixing stuff up. You may know that print()'s operation is to take each expression you give it and call str() on that value, then print that string. f-strings have a similar operation: each {expression} inside one is converted to a string for embedding in the larger format string; that is also done with str() unless you've added something special in the {} part. Aside from output you should not expect to be adding strings and numbers; - it isn't normally a sensible thing to do. And when printing things, you need text (strings). But print() takes can of that by passing every value to str(), which returns a string (in a manner appropriate to the type of value). You only want f-strings or manual joining up if you don't want the default separator between items (a space). Can you elaborate on _why_ you wanted an f-string? What were you doing at the time? Cheers, Cameron Simpson From tdtemccna at gmail.com Wed May 25 09:30:04 2022 From: tdtemccna at gmail.com (Turritopsis Dohrnii Teo En Ming) Date: Wed, 25 May 2022 21:30:04 +0800 Subject: Popular Python Package 'ctx' Hijacked to Steal AWS Keys Message-ID: Subject: Popular Python Package 'ctx' Hijacked to Steal AWS Keys Good day from Singapore, Sharing this article for more awareness. Article: Popular PyPI Package 'ctx' and PHP Library 'phpass' Hijacked to Steal AWS Keys Link: https://thehackernews.com/2022/05/pypi-package-ctx-and-php-library-phpass.html Thank you. Regards, Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 25 May 2022 Wed From george at fischhof.hu Wed May 25 11:04:07 2022 From: george at fischhof.hu (George Fischhof) Date: Wed, 25 May 2022 17:04:07 +0200 Subject: Popular Python Package 'ctx' Hijacked to Steal AWS Keys In-Reply-To: References: Message-ID: Turritopsis Dohrnii Teo En Ming ezt ?rta (id?pont: 2022. m?j. 25., Sze, 15:49): > Subject: Popular Python Package 'ctx' Hijacked to Steal AWS Keys > > Good day from Singapore, > > Sharing this article for more awareness. > > Article: Popular PyPI Package 'ctx' and PHP Library 'phpass' Hijacked > to Steal AWS Keys > Link: > https://thehackernews.com/2022/05/pypi-package-ctx-and-php-library-phpass.html > > Thank you. > > Regards, > > Mr. Turritopsis Dohrnii Teo En Ming > Targeted Individual in Singapore > 25 May 2022 Wed > -- > https://mail.python.org/mailman/listinfo/python-list Hi All, it's got to my mind that PYPA, community, and developers should develop some mechanism to protect against similar threats. For example security checkers could be added to the upload flow, before a package appears, and becomes downloadable. Compiled parts should be allowed only in source, and security checkers would check those too, and compile from source and publish package only after these checks executed and did not found any harmful thing. BR, George From mmontgomery at levado.to Wed May 25 11:42:08 2022 From: mmontgomery at levado.to (Meredith Montgomery) Date: Wed, 25 May 2022 12:42:08 -0300 Subject: how to distinguish return from print() References: <86sfp3p0f5.fsf@levado.to> Message-ID: <86v8ttei0f.fsf@levado.to> ram at zedat.fu-berlin.de (Stefan Ram) writes: > Meredith Montgomery writes: > ... >>def f(x): >> return x + 1 > ... >>>>> print("hello") > > To me, what would make more sense would be: > > Teacher: > > |>>> def r(): > |... return 7 > |... > |>>> def p(): > |... print( 7 ) > |... > |>>> r() > |7 > |>>> p() > |7 > > Pupil: > > That was a very instructive example, teacher. Thank you! > But could you also show us some context where the calls > to p and r show some difference? > > Teacher: > > |>>> print( r() ) > |7 > |>>> print( p() ) > |7 > |None > > |>>> x = r() > |>>> x = p() > |7 > > Pupil: > > Now I'm confused. What's "None"? > > Teacher: > > ... I did do this too. I think that was helpful. I told them --- a print doesn't let you easily ``capture'' a value calculated in a procedure. Thank you! From i9shuaib1 at gmail.com Wed May 25 21:46:22 2022 From: i9shuaib1 at gmail.com (Shuaib Akhtar) Date: Wed, 25 May 2022 18:46:22 -0700 Subject: bug in python 3.10.4 Message-ID: <5C95784D-3E08-4FE2-B77D-C3A4BAB3475F@hxcore.ol> When double clicking a .py file when have python install. It run file but at a spot of the program it stop running. But using the built-in ide for python this problem does not happen also any other ide it work fine ? ? ? From PythonList at DancesWithMice.info Wed May 25 22:30:35 2022 From: PythonList at DancesWithMice.info (dn) Date: Thu, 26 May 2022 14:30:35 +1200 Subject: bug in python 3.10.4 In-Reply-To: <5C95784D-3E08-4FE2-B77D-C3A4BAB3475F@hxcore.ol> References: <5C95784D-3E08-4FE2-B77D-C3A4BAB3475F@hxcore.ol> Message-ID: On 26/05/2022 13.46, Shuaib Akhtar wrote: > When double clicking a .py file when have python install. It run file but > at a spot of the program it stop running. But using the built-in ide for > python this problem does not happen also any other ide it work fine Please provide (minimal) example code so that we can reproduce the problem and, if necessary, help find a solution... -- Regards, =dn From PythonList at DancesWithMice.info Thu May 26 03:56:16 2022 From: PythonList at DancesWithMice.info (dn) Date: Thu, 26 May 2022 19:56:16 +1200 Subject: bug in python 3.10.4 In-Reply-To: <8a199db1-5a81-24de-0e25-e3109d21b7c9@DancesWithMice.info> References: <5C95784D-3E08-4FE2-B77D-C3A4BAB3475F@hxcore.ol> <8a199db1-5a81-24de-0e25-e3109d21b7c9@DancesWithMice.info> Message-ID: <6dbb4723-3ccc-540a-dc19-0dae2f42a6a5@DancesWithMice.info> Please reply to the list. Others may be able to assist (particularly if they use MS-Windows!). > Removing the quit does not help with the problem. > > input 10 x 10 What was the result, or the exception report. Once again: did MS-Windows finish the job and close the window before you could see the result? On 26/05/2022 15.44, dn wrote: >> When double clicking a .py file when have python install. It run file but >> at a spot of the program it stop running. But using the built-in ide for >> python this problem does not happen also any other ide it work fine > > > The code looks good (although most IT-people think of * as the > multiplication operator). > > At present, the only way to stop the program is to enter two numbers and > an invalid operator. This leads to the quit(). I'm not a user of > MS-Windows, but won't that close the window without giving any > opportunity to read the screen? Can you try commenting-out the quit() > and replacing it with a print( "terminating" ) type of message? Does > this help with the "stop running"? > > To enable us to run the program[me] to produce the same results, what > combination of input produces the error, what output results, and do you > see any exception messages? Please copy-paste into your email-reply. > > (the more you help us, the better we can help you!) > > > > On 26/05/2022 15.12, Shuaib Akhtar wrote: >> ? >> >> ? >> >> number_1 = input("put a number\n") >> >> print('division is /') >> >> ? >> >> operations = input("put a operations\n") >> >> number_2 = input("put a number\n") >> >> ? >> >> number_2 = float(number_2) >> >> number_1 = float(number_1) >> >> ? >> >> if?operations == "+": >> >> ? ? print(number_1+number_2) >> >> elifoperations.lower() == "x": >> >> ? ? print(number_1 * number_2) >> >> elifoperations == "/": >> >> ? ? print(number_1/number_2) >> >> elifoperations == "-": >> >> ? ? print(number_1 - number_2) >> >> else: >> >> ? ? print("put a operation next time") >> >> ? ? quit() >> >> ? >> >> ? >> > > -- Regards, =dn From python at mrabarnett.plus.com Thu May 26 09:18:41 2022 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 26 May 2022 14:18:41 +0100 Subject: bug in python 3.10.4 In-Reply-To: <5C95784D-3E08-4FE2-B77D-C3A4BAB3475F@hxcore.ol> References: <5C95784D-3E08-4FE2-B77D-C3A4BAB3475F@hxcore.ol> Message-ID: On 2022-05-26 02:46, Shuaib Akhtar wrote: > When double clicking a .py file when have python install. It run file but > at a spot of the program it stop running. But using the built-in ide for > python this problem does not happen also any other ide it work fine > When you double-click on a .py file, Windows opens a console window and runs the program, and when the program finishes, Windows closes the console window. That's normal. From aachu at comcast.net Thu May 26 11:24:05 2022 From: aachu at comcast.net (ANTHONY CHU) Date: Thu, 26 May 2022 08:24:05 -0700 (PDT) Subject: pip does not find after Python 3.10.4 installed Message-ID: <345438250.629587.1653578645719@connect.xfinity.com> The Python 3.10.4 (64-bit) and Python Launcher had been (standard) installed successfully. But I could not find pip anywhere. I uninstalled and re-installed a couple of times, it is still the problem. I checked the installed directory C:\Users\xxxxx\AppData\Local\Programs\Python\Python310\Tools\Scripts\ and also scanned the whole drive, pip could not be found. I also checked on the web and none of them help. Did I do something wrong? Note: I did install Python 3.8 on another Windows Pro desktop without any problem. Thank very much. Subscriber: aachu From wlfraed at ix.netcom.com Thu May 26 11:35:28 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Thu, 26 May 2022 11:35:28 -0400 Subject: bug in python 3.10.4 References: <5C95784D-3E08-4FE2-B77D-C3A4BAB3475F@hxcore.ol> <8a199db1-5a81-24de-0e25-e3109d21b7c9@DancesWithMice.info> <6dbb4723-3ccc-540a-dc19-0dae2f42a6a5@DancesWithMice.info> Message-ID: On Thu, 26 May 2022 19:56:16 +1200, dn declaimed the following: Commentary meant for the OP, not "dn". >Please reply to the list. Others may be able to assist (particularly if >they use MS-Windows!). > > >> Removing the quit does not help with the problem. >> >> input 10 x 10 Cut&Paste the entire text of the console window (not a screen grab) showing the invocation and the output, if any. NOTE: This presumes you know how to use the Windows command line. Double-clicking a .py file seldom produces a usable stuff as the console opened is automatically closed when the program exits. If you need to see the output, you need to put in some code to make the program /wait/ until you signal that you are done. > >What was the result, or the exception report. > >Once again: did MS-Windows finish the job and close the window before >you could see the result? >>> number_1 = input("put a number\n") This is going to wait at the console until you enter a value. >>> if?operations == "+": >>> >>> ? ? print(number_1+number_2) >>> >>> elifoperations.lower() == "x": Is that REALLY what the code has? I'd expect "elifoperations" to produce a syntax error of some sort since there is no apparent space between the "elif" and "operations". {stylistic note: "operations" is misleading -- it is just ONE OPERATOR} -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From aachu at comcast.net Thu May 26 12:31:34 2022 From: aachu at comcast.net (ANTHONY CHU) Date: Thu, 26 May 2022 09:31:34 -0700 (PDT) Subject: pip does not find after Python 3.10.4 installed In-Reply-To: <345438250.629587.1653578645719@connect.xfinity.com> References: <345438250.629587.1653578645719@connect.xfinity.com> Message-ID: <768321690.456342.1653582694340@connect.xfinity.com> I found that RAV Antivirus complains some processes even Python 3.10.4 installed successfully. Remove RAV and re-install Python; then everything goes OK. Thanks, On 05/26/2022 8:24 AM ANTHONY CHU wrote: > > > The Python 3.10.4 (64-bit) and Python Launcher had been (standard) installed successfully. But I could not find pip anywhere. I uninstalled and re-installed a couple of times, it is still the problem. I checked the installed directory C:\Users\xxxxx\AppData\Local\Programs\Python\Python310\Tools\Scripts\ and also scanned the whole drive, pip could not be found. > I also checked on the web and none of them help. Did I do something wrong? > Note: I did install Python 3.8 on another Windows Pro desktop without any problem. > > Thank very much. > > Subscriber: aachu > From eryksun at gmail.com Thu May 26 18:13:46 2022 From: eryksun at gmail.com (Eryk Sun) Date: Thu, 26 May 2022 17:13:46 -0500 Subject: pip does not find after Python 3.10.4 installed In-Reply-To: <345438250.629587.1653578645719@connect.xfinity.com> References: <345438250.629587.1653578645719@connect.xfinity.com> Message-ID: On 5/26/22, ANTHONY CHU wrote: > The Python 3.10.4 (64-bit) and Python Launcher had been (standard) installed > successfully. But I could not find pip anywhere. I uninstalled and > re-installed a couple of times, it is still the problem. I checked the > installed directory > C:\Users\xxxxx\AppData\Local\Programs\Python\Python310\Tools\Scripts\ FYI, scripts and binaries for packages such as pip are installed in "Scripts", not in "Tools\Scripts". In Windows, it's common to run pip via `py -m pip`. Note that the latter command will use an active virtual environment, if any, else it defaults to the highest version of Python that's installed. You can use a specific installed version via `py -X.Y[-32] -m pip`. From rolfblum at ewe.net Fri May 27 12:18:42 2022 From: rolfblum at ewe.net (Rolf Blum) Date: Fri, 27 May 2022 18:18:42 +0200 Subject: starting Python 3.10.4 64-bit Message-ID: Hallo, I installed Python 3.10.4 64-bit successfully. I Also checked in the Setup to add it to the path. In the win 8.1 console python 3.7xxxx is started when I enter: python. This also happened after a reboot and after a repair with the Setup. How do I start the python console for 3.10.4? Thanks From python at mrabarnett.plus.com Fri May 27 13:27:44 2022 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 27 May 2022 18:27:44 +0100 Subject: starting Python 3.10.4 64-bit In-Reply-To: References: Message-ID: <22a112bc-1713-b4bb-869e-f9f36796430e@mrabarnett.plus.com> On 2022-05-27 17:18, Rolf Blum wrote: > Hallo, > I installed Python 3.10.4 64-bit successfully. I Also checked in the > Setup to add it to the path. > In the win 8.1 console python 3.7xxxx is started when I enter: python. > This also happened after a reboot and after a repair with the Setup. > How do I start the python console for 3.10.4? > Thanks You probably also have Python 3.8 on the path, and it's finding that one first. These days it's recommended to use the Python Launcher, "py". That lets you specify which version of Python you want to use. From larry.martell at gmail.com Fri May 27 16:14:01 2022 From: larry.martell at gmail.com (Larry Martell) Date: Fri, 27 May 2022 16:14:01 -0400 Subject: terminate called after throwing an instance of 'boost::python::error_already_set Message-ID: I have a script that has literally been running for 10 years. Suddenly, for some runs it crashes with the error: terminate called after throwing an instance of 'boost::python::error_already_set No stack trace. Anyone have any thoughts on what could cause this and/or how I can track it down? From PythonList at DancesWithMice.info Fri May 27 17:50:24 2022 From: PythonList at DancesWithMice.info (dn) Date: Sat, 28 May 2022 09:50:24 +1200 Subject: terminate called after throwing an instance of 'boost::python::error_already_set In-Reply-To: References: Message-ID: <17e157a4-20e0-a640-1fb0-7fc17fbd65b7@DancesWithMice.info> On 28/05/2022 08.14, Larry Martell wrote: > I have a script that has literally been running for 10 years. > Suddenly, for some runs it crashes with the error: > > terminate called after throwing an instance of 'boost::python::error_already_set > > No stack trace. Anyone have any thoughts on what could cause this > and/or how I can track it down? 1 a change to Python interpreter being used 2 a change to the boost library being used 3 a change to lower levels in 'the s/w stack' or h/w 4 a change to the data being passed-across -- Regards, =dn From larry.martell at gmail.com Fri May 27 19:11:45 2022 From: larry.martell at gmail.com (Larry Martell) Date: Fri, 27 May 2022 19:11:45 -0400 Subject: terminate called after throwing an instance of 'boost::python::error_already_set In-Reply-To: <17e157a4-20e0-a640-1fb0-7fc17fbd65b7@DancesWithMice.info> References: <17e157a4-20e0-a640-1fb0-7fc17fbd65b7@DancesWithMice.info> Message-ID: On Fri, May 27, 2022 at 5:51 PM dn wrote: > On 28/05/2022 08.14, Larry Martell wrote: > > I have a script that has literally been running for 10 years. > > Suddenly, for some runs it crashes with the error: > > > > terminate called after throwing an instance of > 'boost::python::error_already_set > > > > No stack trace. Anyone have any thoughts on what could cause this > > and/or how I can track it down? > > 1 a change to Python interpreter being used > > 2 a change to the boost library being used > > 3 a change to lower levels in 'the s/w stack' or h/w > > 4 a change to the data being passed-across Definitely not 1. 4 is always the case - every run is with different data. 2 and 3 I don?t know. What is boost and how does Python use it? None of my code is importing it. How can get a stack trace when it crashes with just that message? > From PythonList at DancesWithMice.info Fri May 27 19:42:16 2022 From: PythonList at DancesWithMice.info (dn) Date: Sat, 28 May 2022 11:42:16 +1200 Subject: terminate called after throwing an instance of 'boost::python::error_already_set In-Reply-To: References: <17e157a4-20e0-a640-1fb0-7fc17fbd65b7@DancesWithMice.info> Message-ID: On 28/05/2022 11.11, Larry Martell wrote: > > > On Fri, May 27, 2022 at 5:51 PM dn > wrote: > > On 28/05/2022 08.14, Larry Martell wrote: > > I have a script that has literally been running for 10 years. > > Suddenly, for some runs it crashes with the error: > > > > terminate called after throwing an instance of > 'boost::python::error_already_set > > > > No stack trace. Anyone have any thoughts on what could cause this > > and/or how I can track it down? > > 1 a change to Python interpreter being used > > 2 a change to the boost library being used > > 3 a change to lower levels in 'the s/w stack' or h/w > > 4 a change to the data being passed-across > > > Definitely not 1. 4 is always the case - every run is with different > data. 2 and 3 I don?t know. What is boost and how does Python use it? > None of my code is importing it. How can get a stack trace when it > crashes with just that message?? A possibly/hopefully helpful explanation appears at: https://misspent.wordpress.com/2009/10/11/boost-python-and-handling-python-exceptions/ -- Regards, =dn From barry at barrys-emacs.org Sat May 28 04:17:18 2022 From: barry at barrys-emacs.org (Barry) Date: Sat, 28 May 2022 09:17:18 +0100 Subject: terminate called after throwing an instance of 'boost::python::error_already_set In-Reply-To: References: Message-ID: > On 27 May 2022, at 21:17, Larry Martell wrote: > > ?I have a script that has literally been running for 10 years. > Suddenly, for some runs it crashes with the error: > > terminate called after throwing an instance of 'boost::python::error_already_set This is from an extension that is written in C++ that raised a C++exception, not a python one. The default action in C++ is to terminal the process so python does not get a chance to prints its stack. > > No stack trace. Anyone have any thoughts on what could cause this > and/or how I can track it down? You will need to use a C++ level debugger to see which extension is crashing. gdb on linux, lldb on macOs or visual studio on windows. Barry > -- > https://mail.python.org/mailman/listinfo/python-list > From Ralf_M at t-online.de Sat May 28 16:33:38 2022 From: Ralf_M at t-online.de (Ralf M.) Date: Sat, 28 May 2022 22:33:38 +0200 Subject: .0 in name In-Reply-To: <964395bd7ff3ecdd4ce11de88d7b207b86d9cad0.camel@anode.ca> References: <964395bd7ff3ecdd4ce11de88d7b207b86d9cad0.camel@anode.ca> Message-ID: <0d1baac8-b513-e416-22c0-f3cbd0eaba7a@t-online.de> Am 13.05.2022 um 23:23 schrieb Paul Bryan: > On Sat, 2022-05-14 at 00:47 +0800, bryangan41 wrote: > >> May I know (1) why can the name start with a number? > > The name of an attribute must be an identifier. An identifier cannot > begin with a decimal number. I'm not sure about the first statement. Feeding [print("locals:", locals()) or c for c in "ab"] to the REPL, the result is locals: {'.0': , 'c': 'a'} locals: {'.0': , 'c': 'b'} ['a', 'b'] i.e. there is a variable of name .0 in the local namespace within the list comprehension, and .0 is definitely not an identifier. I came across this while investigating another problem with list comprehensions, and I think the original post was about list comprehensions. There also can be non-identifier names in the global namespace and as attributes, e.g. using the REPL again: globals()["42"] = "The Answer" globals() outputs (see last entry) {'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': , '__spec__': None, '__annotations__': {}, '__builtins__': , '42': 'The Answer'} and class Cls: def __init__(self, lst): for i, e in enumerate(lst): self.__dict__[str(i)] = e obj = Cls([31, 42, 53]) getattr(obj, "1") works and outputs 42 >> (2) where in the doc is it?! > > https://docs.python.org/3/reference/lexical_analysis.html#identifiers That refers to identifiers, i.e. names that are recognised as such by the lexer, i.e. that can be written directly in Python source code. As shown above, names that are not identifiers can be used in several namespaces and as attributes. It's just a bit harder to use non-identifier names than identifiers. Whether it's a good idea to use them at all is a different question. I think the OP wondered about the .0 in the local namespace within list comprehensions. Unfortunately I cannot say much about that. > Paul Ralf M. From rosuav at gmail.com Sat May 28 17:15:41 2022 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 29 May 2022 07:15:41 +1000 Subject: .0 in name In-Reply-To: <0d1baac8-b513-e416-22c0-f3cbd0eaba7a@t-online.de> References: <964395bd7ff3ecdd4ce11de88d7b207b86d9cad0.camel@anode.ca> <0d1baac8-b513-e416-22c0-f3cbd0eaba7a@t-online.de> Message-ID: On Sun, 29 May 2022 at 06:41, Ralf M. wrote: > > Am 13.05.2022 um 23:23 schrieb Paul Bryan: > > On Sat, 2022-05-14 at 00:47 +0800, bryangan41 wrote: > > > >> May I know (1) why can the name start with a number? > > > > The name of an attribute must be an identifier. An identifier cannot > > begin with a decimal number. > > I'm not sure about the first statement. Feeding > > [print("locals:", locals()) or c for c in "ab"] > > to the REPL, the result is > > locals: {'.0': , 'c': 'a'} > locals: {'.0': , 'c': 'b'} > ['a', 'b'] > > i.e. there is a variable of name .0 in the local namespace within the > list comprehension, and .0 is definitely not an identifier. > > I came across this while investigating another problem with list > comprehensions, and I think the original post was about list comprehensions. > There are a few quirks with comprehensions, and to understand that ".0", you have to first understand two very important aspects of scoping with regard to comprehensions. (Note: For simplicity, I'm going to refer in general to "comprehensions", and I am not going to count Python 2. My example will be a list comp, but a generator expression also behaves like this, as do other comprehensions.) Consider this function: def spam(): ham = "initial" ham = [locals() for x in "q"] return ham The disassembly module can be very helpful here. The precise output will vary with Python version, but the points I'm making should be valid for all current versions. Here's how it looks in a December build of Python 3.11 (yeah, my Python's getting a bit old now, I should update at some point): >>> dis.dis(spam) 2 0 LOAD_CONST 1 ('initial') 2 STORE_FAST 0 (ham) 3 4 LOAD_CONST 2 ( at 0x7fb6a0cfa6b0, file "", line 3>) 6 MAKE_FUNCTION 0 8 LOAD_CONST 3 ('q') 10 GET_ITER 12 CALL_FUNCTION 1 14 STORE_FAST 0 (ham) 4 16 LOAD_FAST 0 (ham) 18 RETURN_VALUE Disassembly of at 0x7fb6a0cfa6b0, file "", line 3>: 3 0 BUILD_LIST 0 2 LOAD_FAST 0 (.0) >> 4 FOR_ITER 5 (to 16) 6 STORE_FAST 1 (x) 8 LOAD_GLOBAL 0 (locals) 10 CALL_FUNCTION 0 12 LIST_APPEND 2 14 JUMP_ABSOLUTE 2 (to 4) >> 16 RETURN_VALUE >>> Okay, that's a lot of raw data, but let's pull out a few useful things from it. Line 2 initializes ham in an unsurprising way. Grab a constant, store it in a local. Easy. Line three. We grab the code object for the list comp, and make a function (that's necessary for closures). Then, *still in the context of the spam function*, we grab the constant "q", and get an iterator from it. Leaving that on the top of the stack, we call the list comprehension's function, and store the result into 'ham'. The comprehension itself loads the fast local from slot zero (name ".0") and iterates over it. Slot zero is the first argument, so that's the string iterator that we left there for the function. So why IS this? There are a few reasons, but the main one is generator expressions. Replacing the list comp with a genexp gives this result: >>> spam() . at 0x7fb6a0780890> The actual iteration (row 4 in the genexp in the above disassembly of ) doesn't happen until you iterate over this value. But it would be extremely confusing if, in that situation, errors didn't show up until much later. What if, instead of iterating over a string, you tried to iterate over a number? Where should the traceback come from? Or what if you're iterating over a variable, and you change what's in that variable? def wat(): stuff = "hello" ucase = (l.upper() for l in stuff) stuff = "goodbye" return "".join(ucase) Does this return "HELLO" or "GOODBYE"? Since stuff gets evaluated immediately, it returns HELLO, and that's consistent for list comps and genexps. But because of that, there needs to be a parameter to carry that iterator through, and every parameter needs a name. If the generated name collided with any identifier that you actually wanted, it would be extremely confusing; so to keep everything safe, the interpreter generates a name you couldn't possibly want - same as for the function itself, which is named "" or "", angle brackets included. That's a fairly long-winded way to put it, but that's why you can have variables with bizarre names :) ChrisA From Gronicus at SGA.Ninja Sat May 28 17:29:10 2022 From: Gronicus at SGA.Ninja (Steve GS) Date: Sat, 28 May 2022 17:29:10 -0400 Subject: Automatic Gain Control in Python? Message-ID: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> I have an extensive Excel/VBA program that hourly calls and plays podcasts through a "smart" speaker. The output of the speaker feeds into another computer that records the m\audio using Audacity. It has become obvious that NPR does not regulate volumes for podcasts and broadcasts nor are programs in themselves regulated by volume. Audacity does not have an AGC. It has also been noted that Excel/VBA code cannot see the audio being played on the same computer. I would like to find code that will regulate the volume and give some semblance of control/moderation. Also, sometimes the Smart Speaker fails to play the program and I get an hour of silence before the next command to play happens. The code should detect that nothing is playing and send the command to the smart speaker again. Is there any Python code that I might be able to call from VBA that will monitor and regulate the volume of the audio? A few samples of code that can read/modify the audio will help me develop the final product. Suggestions appreciated. From eryksun at gmail.com Sat May 28 18:26:50 2022 From: eryksun at gmail.com (Eryk Sun) Date: Sat, 28 May 2022 17:26:50 -0500 Subject: .0 in name In-Reply-To: References: <964395bd7ff3ecdd4ce11de88d7b207b86d9cad0.camel@anode.ca> <0d1baac8-b513-e416-22c0-f3cbd0eaba7a@t-online.de> Message-ID: On 5/28/22, Chris Angelico wrote: > > be extremely confusing; so to keep everything safe, the interpreter > generates a name you couldn't possibly want - same as for the function > itself, which is named "" or "", angle brackets > included. To clarify, "" is the co_name and co_qualname value of the code object, which was compiled for the list comprehension. These names are also used as the __name__ and __qualname__ of the temporary object that's created by MAKE_FUNCTION. They are not identifiers. The code object is a constant, which is referenced solely by its index in the co_consts tuple. The temporary function is referenced on the stack. From rosuav at gmail.com Sat May 28 18:29:25 2022 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 29 May 2022 08:29:25 +1000 Subject: .0 in name In-Reply-To: References: <964395bd7ff3ecdd4ce11de88d7b207b86d9cad0.camel@anode.ca> <0d1baac8-b513-e416-22c0-f3cbd0eaba7a@t-online.de> Message-ID: On Sun, 29 May 2022 at 08:26, Eryk Sun wrote: > > On 5/28/22, Chris Angelico wrote: > > > > be extremely confusing; so to keep everything safe, the interpreter > > generates a name you couldn't possibly want - same as for the function > > itself, which is named "" or "", angle brackets > > included. > > To clarify, "" is the co_name and co_qualname value of the > code object, which was compiled for the list comprehension. These > names are also used as the __name__ and __qualname__ of the temporary > object that's created by MAKE_FUNCTION. They are not identifiers. The > code object is a constant, which is referenced solely by its index in > the co_consts tuple. The temporary function is referenced on the > stack. Correct. Every function has a name, important for tracebacks and such, but with lambda functions, the internal functions of comprehensions, and so on, there's no actual name binding for it. So the interpreter generates a name that won't collide with any actual name that you'd have assigned anything to. ChrisA From Richard at Damon-Family.org Sat May 28 18:53:13 2022 From: Richard at Damon-Family.org (Richard Damon) Date: Sat, 28 May 2022 18:53:13 -0400 Subject: Automatic Gain Control in Python? In-Reply-To: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> References: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> Message-ID: On 5/28/22 5:29 PM, Steve GS wrote: > I have an extensive Excel/VBA program that hourly calls and plays podcasts > through a "smart" speaker. The output of the speaker feeds into another > computer that records the m\audio using Audacity. > > It has become obvious that NPR does not regulate volumes for podcasts and > broadcasts nor are programs in themselves regulated by volume. Audacity > does not have an AGC. > > It has also been noted that Excel/VBA code cannot see the audio being played > on the same computer. > > I would like to find code that will regulate the volume and give some > semblance of control/moderation. Also, sometimes the Smart Speaker fails to > play the program and I get an hour of silence before the next command to > play happens. The code should detect that nothing is playing and send the > command to the smart speaker again. > > Is there any Python code that I might be able to call from VBA that will > monitor and regulate the volume of the audio? A few samples of code that can > read/modify the audio will help me develop the final product. > > Suggestions appreciated. My first thought is you are solving the wrong problem. What seems a better option would be to get your code to actually connect up to the podcast and just download the audio directly, rather than trying to get the smart speaker to play the audio and record it with a microphone. That might require finding the API for the site that hosts the podcasts, to get it to send the files to you to "play". Once you have the files, it becomes simple signal processing to go over the files and AGCing them as needed. On a side note, make sure you are within your rights within Copyright law for what you are doing. Recording for PERSONAL use is probably within the bounds of "Fair Use", but the material is surely under Copyright, so be careful what you do with it. -- Richard Damon From Gronicus at SGA.Ninja Sat May 28 20:17:02 2022 From: Gronicus at SGA.Ninja (Steve GS) Date: Sat, 28 May 2022 20:17:02 -0400 Subject: Automatic Gain Control in Python? In-Reply-To: References: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> Message-ID: <015301d872f1$6bf9afc0$43ed0f40$@SGA.Ninja> "My first thought is you are solving the wrong problem. What seems a better option would be to get your code to actually connect up to the podcast and just download the audio directly, rather than trying to get the smart speaker to play the audio and record it with a microphone." The smart-speaker is bringing in the podcast by hourly automated commands and sending by audio cable to a computer which is recording it with Audacity. This is an automated system that runs for 48 hours every weekend. Its output is played live throughout the facility and is also recorded for replay through the week. No download to use. AGC is to happen when the Smart Speaker is playing it, real time. Any post-record editing would be a horrendous task to say the least. ============================= Genie: You have three wishes. Me: I wish I had more wishes. Genie: You cannot wish for more wishes. Me: I wish I could. -----Original Message----- From: Python-list On Behalf Of Richard Damon Sent: Saturday, May 28, 2022 6:53 PM To: python-list at python.org Subject: Re: Automatic Gain Control in Python? On 5/28/22 5:29 PM, Steve GS wrote: > I have an extensive Excel/VBA program that hourly calls and plays > podcasts through a "smart" speaker. The output of the speaker feeds > into another computer that records the m\audio using Audacity. > > It has become obvious that NPR does not regulate volumes for podcasts > and broadcasts nor are programs in themselves regulated by volume. > Audacity does not have an AGC. > > It has also been noted that Excel/VBA code cannot see the audio being > played on the same computer. > > I would like to find code that will regulate the volume and give some > semblance of control/moderation. Also, sometimes the Smart Speaker > fails to play the program and I get an hour of silence before the next > command to play happens. The code should detect that nothing is > playing and send the command to the smart speaker again. > > Is there any Python code that I might be able to call from VBA that > will monitor and regulate the volume of the audio? A few samples of > code that can read/modify the audio will help me develop the final product. > > Suggestions appreciated. My first thought is you are solving the wrong problem. What seems a better option would be to get your code to actually connect up to the podcast and just download the audio directly, rather than trying to get the smart speaker to play the audio and record it with a microphone. That might require finding the API for the site that hosts the podcasts, to get it to send the files to you to "play". Once you have the files, it becomes simple signal processing to go over the files and AGCing them as needed. On a side note, make sure you are within your rights within Copyright law for what you are doing. Recording for PERSONAL use is probably within the bounds of "Fair Use", but the material is surely under Copyright, so be careful what you do with it. -- Richard Damon -- https://mail.python.org/mailman/listinfo/python-list From python at mrabarnett.plus.com Sat May 28 20:57:26 2022 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 29 May 2022 01:57:26 +0100 Subject: Automatic Gain Control in Python? In-Reply-To: <015301d872f1$6bf9afc0$43ed0f40$@SGA.Ninja> References: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> <015301d872f1$6bf9afc0$43ed0f40$@SGA.Ninja> Message-ID: <31af8984-4760-3fa7-a4cd-8eb7d5cd27c2@mrabarnett.plus.com> On 2022-05-29 01:17, Steve GS wrote: > "My first thought is you are solving the wrong problem. What seems a better > option would be to get your code to actually connect up to the podcast and > just download the audio directly, rather than trying to get the smart > speaker to play the audio and record it with a microphone." > > The smart-speaker is bringing in the podcast by hourly automated commands > and sending by audio cable to a computer which is recording it with > Audacity. This is an automated system that runs for 48 hours every weekend. > Its output is played live throughout the facility and is also recorded for > replay through the week. > > No download to use. > > AGC is to happen when the Smart Speaker is playing it, real time. > Any post-record editing would be a horrendous task to say the least. > [snip] Why would post-record editing be "horrendous"? Does it record the whole 48 hours into 1 file? If it's recording each podcast separately, it could process each one after recording it, even while the next one is being recorded, and I really doubt that processing each one while take long. From Gronicus at SGA.Ninja Sat May 28 21:22:59 2022 From: Gronicus at SGA.Ninja (Steve GS) Date: Sat, 28 May 2022 21:22:59 -0400 Subject: Automatic Gain Control in Python? In-Reply-To: <31af8984-4760-3fa7-a4cd-8eb7d5cd27c2@mrabarnett.plus.com> References: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> <015301d872f1$6bf9afc0$43ed0f40$@SGA.Ninja> <31af8984-4760-3fa7-a4cd-8eb7d5cd27c2@mrabarnett.plus.com> Message-ID: <015b01d872fa$a2fb2c70$e8f18550$@SGA.Ninja> >> Why would post-record editing be "horrendous"? This has to be done on-the-fly before it is recorded. After the AGC is applied, it will be played, live, to the community. It is played during the week to a much smaller audience, almost as background noise. Post recording editing would be a waste of time and worthless. >> Does it record the whole 48 hours into 1 file? Two files, 24 hours each, one for Saturday, the other Sunday -----Original Message----- From: Python-list On Behalf Of MRAB Sent: Saturday, May 28, 2022 8:57 PM To: python-list at python.org Subject: Re: Automatic Gain Control in Python? On 2022-05-29 01:17, Steve GS wrote: > "My first thought is you are solving the wrong problem. What seems a > better option would be to get your code to actually connect up to the > podcast and just download the audio directly, rather than trying to > get the smart speaker to play the audio and record it with a microphone." > > The smart-speaker is bringing in the podcast by hourly automated > commands and sending by audio cable to a computer which is recording > it with Audacity. This is an automated system that runs for 48 hours every weekend. > Its output is played live throughout the facility and is also recorded > for replay through the week. > > No download to use. > > AGC is to happen when the Smart Speaker is playing it, real time. > Any post-record editing would be a horrendous task to say the least. > [snip] Why would post-record editing be "horrendous"? Does it record the whole 48 hours into 1 file? If it's recording each podcast separately, it could process each one after recording it, even while the next one is being recorded, and I really doubt that processing each one while take long. -- https://mail.python.org/mailman/listinfo/python-list From Gronicus at SGA.Ninja Sun May 29 01:09:23 2022 From: Gronicus at SGA.Ninja (Steve GS) Date: Sun, 29 May 2022 01:09:23 -0400 Subject: Automatic Gain Control in Python? In-Reply-To: References: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> <015301d872f1$6bf9afc0$43ed0f40$@SGA.Ninja> Message-ID: <017501d8731a$43882750$ca9875f0$@SGA.Ninja> You really need to understand what I am trying to do. It is not a simple lesson in use of podcasts. This is an automated system. I call it my NPR Jukebox. 15 years ago, I started with hourly URL calls to a browser to record specific NPR programs. It took a lot of coordination. I had to use IE because I needed to start and stop browsers on the hour and IE was the only one that could do that. Then web sites started "upgrading" to Edge. Through some trickery I was able to get Edge to do what IE did but it was unstable. I then discovered the Echo Smart Speaker. I set my program to announce the broadcast station or podcast by speaker to the smart speaker and it cured a lot of headaches. I then was able to call podcasts because the Echo handles them through TuneIn. Unfortunately, not all broadcasts ae available as podcasts. I am not here diddling around just playing podcasts. Let me repeat what I have already said. It is an automated system. Every hour for 48 hours on every weekend, my system using a well-developed Excel/VBA program that vocally talks to the SS hourly. The SS finds the audio and sends it to my Audacity recorder on another computer through aux-out to mic-in cable. The selections of audio are also transmitted to the community at the time of recording That part of the system is almost flawless, well compared to that I had before. Although the quality, tone, and timing of the announcement, the SS still gets confused once in a while and goes silent for the hour. I need to detect this too. Ok, now back to the original question. Podcasts and broadcasts apparently do not use the Dolby tone to balance the audio levels. And I receive highly fluctuating levels of audio. Sometimes it is between sides of a conversation, sometimes it is the podcast vs station identifications, then it is great differences between one web site and another. Then there's the differences with male and female voices. Imagine that you are watching TV late night then the commercials COME IN AT FULL BLAST. The technology of the industry grew up with male voices and apparently sees no reason to adjust for female. I have worked with audio systems and making recordings for more years that I want to admit. All I want is software to detect level changes over time and attempt to equalize them. It has to be work between the SS and the recorder and is to be checking all the time. The code is to: Listen to the audio level for about 10 seconds or so and raise or lower the level in small increments. It has nothing to do with understanding how to grab podcasts. The system is working very well for that. Footnote: ?What rhymes with orange?? ?No, it doesn?t..? -----Original Message----- From: Richard Damon On Behalf Of Richard Damon Sent: Saturday, May 28, 2022 11:37 PM To: Steve GS Subject: Re: Automatic Gain Control in Python? On 5/28/22 8:17 PM, Steve GS wrote: > "My first thought is you are solving the wrong problem. What seems a > better option would be to get your code to actually connect up to the > podcast and just download the audio directly, rather than trying to > get the smart speaker to play the audio and record it with a microphone." > > The smart-speaker is bringing in the podcast by hourly automated > commands and sending by audio cable to a computer which is recording > it with Audacity. This is an automated system that runs for 48 hours every weekend. > Its output is played live throughout the facility and is also recorded > for replay through the week. > > No download to use. > > AGC is to happen when the Smart Speaker is playing it, real time. > Any post-record editing would be a horrendous task to say the least. > My guess is you don't understand how "Podcasts" work. All they are is a web resource that your Browser/Smart Device makes a request off, and the contents are streamed over the internet to that device, which then plays it. Smart Speakers just have a program that knows how to access these. Since they can be listened to on a web browser, a program can download the data. They might be doing things to make this harder, but that is a sign that you shouldn't be doing this in the first place. Often, the data format that is streamed is exactly like the file format for storing an audio file (since that is what the code in the browser is built to handle). It may be a bit of work to figure out the access methods to get the data, but this is the sort of job that computers were designed to do. Trying to make things that weren't designed to be remote controlled to be remote controlled may well be a lot more work. -- Richard Damon From Richard at Damon-Family.org Sun May 29 08:02:33 2022 From: Richard at Damon-Family.org (Richard Damon) Date: Sun, 29 May 2022 08:02:33 -0400 Subject: Automatic Gain Control in Python? In-Reply-To: <017501d8731a$43882750$ca9875f0$@SGA.Ninja> References: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> <015301d872f1$6bf9afc0$43ed0f40$@SGA.Ninja> <017501d8731a$43882750$ca9875f0$@SGA.Ninja> Message-ID: From your description, your fundamental problem is you are trying to automatically "control" things that weren't designed to be automatically controlled in the way you are attempting. The smart speaker assumes the "user" will adjust the volume either with the controls or with verbal commands, So things will be a bit "clunky" in your results as you command the smart speaker volume level. Yes, you have an automated system that does most of what you want, but it is based on pieces not designed to be automated in this way, and you are running into the limitations caused by that. Yes, you could split the aux-out to bring it into another computer to listen to the sound level, and then using a sound input package get samples of what is playing, and analyze that data to get an average volume, and then issues the command to adjust the volume level. What you seem to be missing is that you could get the podcasts from a browser, and all a browser is is a program. It isn't that much work to write a rudimentary browser in python, especially if you don't actually need to display the results to a user, but are only trying to automate a particular task. You seem to feel strongly invested in your current code base, which is understandable, but it seems you have reached a point where you don't want to live with the limitations CAUSED by that system. Yes, there is likely a way to tack on another layer of "stuff" to adjust for this issue, but it likely is going to require some real programming. It may well be the design I am suggesting, of writing a program to fetch the podcast and save it requires a bit more work to get to the level you currently are at, but the results are a system that is actually designed to be controlled by automation. Maybe it is beyond you ability, but then so might the programming to get the volume. I will also add, that the way your describe going to your "community" gives me questions if this is a violation of copyright. Maybe it is something you can "Get away with", but I am not sure what you are doing is actually legitimate. On 5/29/22 1:09 AM, Steve GS wrote: > You really need to understand what I am trying to do. > It is not a simple lesson in use of podcasts. > This is an automated system. I call it my NPR Jukebox. > > 15 years ago, I started with hourly URL calls to a browser to record specific NPR programs. It took a lot of coordination. I had to use IE because I needed to start and stop browsers on the hour and IE was the only one that could do that. Then web sites started "upgrading" to Edge. Through some trickery I was able to get Edge to do what IE did but it was unstable. > > I then discovered the Echo Smart Speaker. I set my program to announce the broadcast station or podcast by speaker to the smart speaker and it cured a lot of headaches. I then was able to call podcasts because the Echo handles them through TuneIn. Unfortunately, not all broadcasts ae available as podcasts. > > I am not here diddling around just playing podcasts. Let me repeat what I have already said. It is an automated system. Every hour for 48 hours on every weekend, my system using a well-developed Excel/VBA program that vocally talks to the SS hourly. The SS finds the audio and sends it to my Audacity recorder on another computer through aux-out to mic-in cable. The selections of audio are also transmitted to the community at the time of recording > > That part of the system is almost flawless, well compared to that I had before. Although the quality, tone, and timing of the announcement, the SS still gets confused once in a while and goes silent for the hour. I need to detect this too. > > Ok, now back to the original question. > > Podcasts and broadcasts apparently do not use the Dolby tone to balance the audio levels. And I receive highly fluctuating levels of audio. Sometimes it is between sides of a conversation, sometimes it is the podcast vs station identifications, then it is great differences between one web site and another. Then there's the differences with male and female voices. Imagine that you are watching TV late night then the commercials COME IN AT FULL BLAST. > > The technology of the industry grew up with male voices and apparently sees no reason to adjust for female. I have worked with audio systems and making recordings for more years that I want to admit. > > All I want is software to detect level changes over time and attempt to equalize them. > It has to be work between the SS and the recorder and is to be checking all the time. > > The code is to: Listen to the audio level for about 10 seconds or so and raise or lower the level in small increments. > It has nothing to do with understanding how to grab podcasts. The system is working very well for that. > > > Footnote: > ?What rhymes with orange?? > ?No, it doesn?t..? > > > > -----Original Message----- > From: Richard Damon On Behalf Of Richard Damon > Sent: Saturday, May 28, 2022 11:37 PM > To: Steve GS > Subject: Re: Automatic Gain Control in Python? > > On 5/28/22 8:17 PM, Steve GS wrote: >> "My first thought is you are solving the wrong problem. What seems a >> better option would be to get your code to actually connect up to the >> podcast and just download the audio directly, rather than trying to >> get the smart speaker to play the audio and record it with a microphone." >> >> The smart-speaker is bringing in the podcast by hourly automated >> commands and sending by audio cable to a computer which is recording >> it with Audacity. This is an automated system that runs for 48 hours every weekend. >> Its output is played live throughout the facility and is also recorded >> for replay through the week. >> >> No download to use. >> >> AGC is to happen when the Smart Speaker is playing it, real time. >> Any post-record editing would be a horrendous task to say the least. >> > My guess is you don't understand how "Podcasts" work. All they are is a web resource that your Browser/Smart Device makes a request off, and the contents are streamed over the internet to that device, which then plays it. Smart Speakers just have a program that knows how to access these. > > Since they can be listened to on a web browser, a program can download the data. They might be doing things to make this harder, but that is a sign that you shouldn't be doing this in the first place. > > Often, the data format that is streamed is exactly like the file format for storing an audio file (since that is what the code in the browser is built to handle). > > It may be a bit of work to figure out the access methods to get the data, but this is the sort of job that computers were designed to do. > Trying to make things that weren't designed to be remote controlled to be remote controlled may well be a lot more work. > > -- > Richard Damon > -- Richard Damon From Gronicus at SGA.Ninja Sun May 29 11:04:14 2022 From: Gronicus at SGA.Ninja (Steve GS) Date: Sun, 29 May 2022 11:04:14 -0400 Subject: Automatic Gain Control in Python? In-Reply-To: References: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> <015301d872f1$6bf9afc0$43ed0f40$@SGA.Ninja> <017501d8731a$43882750$ca9875f0$@SGA.Ninja> Message-ID: <003601d8736d$5cd3e610$167bb230$@SGA.Ninja> >> From your description, your fundamental problem is you are trying to automatically "control" things that weren't designed to be automatically controlled in the way you are attempting. How so? I am sending commands to a smart speaker and it plays podcasts and broadcasts. How is this a violation of SS design? =================== >> The smart speaker assumes the "user" will adjust the volume either with the controls or with verbal commands, So things will be a bit "clunky" in your results as you command the smart speaker volume level. So, you want me to sit here for every hour of the weekend and monitor the audio levels for a result that will get, at best, one replay when I believe it can be automated. =================== >> Yes, you have an automated system that does most of what you want, but it is based on pieces not designed to be automated in this way, and you are running into the limitations caused by that. Again, what limitations of the SS am I violating? It is designed to receive commands and play the audio. Also, what makes you think that you know how my program is based? =================== >> Yes, you could split the aux-out to bring it into another computer to listen to the sound level, and then using a sound input package get samples of what is playing, and analyze that data to get an average volume, and then issues the command to adjust the volume level. Hmmm, is that not my original question? Are you suggesting to monitor the audio, sense it for volume changes and apply those changes to the original audio? One thing that may have to happen is a timed-delay to all for the AGC to work. This may require a correlation circuit. ================== >> What you seem to be missing is that you could get the podcasts from a browser, and all a browser is is a program. It isn't that much work to write a rudimentary browser in python, especially if you don't actually need to display the results to a user, but are only trying to automate a particular task. Writing my own browser in Python might work. Do you have a sample one that I could twerk to fit my needs? I would have to be able to invoke it and an hour later devoke it least I end up with multiple audio channels playing. Either way, I would still need an AGC program which was my original question. =================== >> You seem to feel strongly invested in your current code base, which is understandable, but it seems you have reached a point where you don't want to live with the limitations CAUSED by that system. The changes in volume are not CAUSED by my program. The want to fix them is a new development to improve the product. The volume fluctuations are causes, or neglections, by the engineers at the sources of podcasts and broadcasts. >> Yes, there is likely a way to tack on another layer of "stuff" to adjust for this issue, but it likely is going to require some real programming. >> It may well be the design I am suggesting, of writing a program to fetch the podcast and save it requires a bit more work to get to the level you currently are at, but the results are a system that is actually designed to be controlled by automation. Maybe it is beyond you ability, but then so might the programming to get the volume. "Real programming"?, REAL PROGRAMMING? Well at this I have to say "Well duh". I have more than 40 years of programming in languages including assembly, Pascal, C, C++, BASIC, Visual BASIC, COBOL, VBA and Python for starters. I even dipped into AT&T's SNOBAL. Starting in the mid-80s, I taught Visual BASIC and VBA for 15 years in high school and at the University level. I have 2000 lines of code of Python that helped me to bring my A1c reading from 9.0 to 6.1 thank you. I have a program of similar size that tracks the 450 specific plants in my garden. It even monitors degree-days to help me know when certain insects attack and when to tend to individual needs. This Excel/Jukebox program is also about 2000 lines of code. (Seems to be a pattern here). All of these programs are decorated with numerous pops and whistles that make programming and use quite pleasurable, So, do you I think I am now ready for some "real programming". =================== >> I will also add, that the way your describe going to your "community" gives me questions if this is a violation of copyright. Maybe it is something you can "Get away with", but I am not sure what you are doing is actually legitimate. Yes, I have been through that. It is totally legal to record NPR broadcasts for replay as long as they are not retained for than a month or for multiple replays. Your suggestion to download and play a podcast or broadcast is legal only for live replay. My want to record them for one replay for my own use. Personal play is a different story. -----Original Message----- From: Richard Damon On Behalf Of Richard Damon Sent: Sunday, May 29, 2022 8:03 AM To: Steve GS ; Python Subject: Re: Automatic Gain Control in Python? From your description, your fundamental problem is you are trying to automatically "control" things that weren't designed to be automatically controlled in the way you are attempting. The smart speaker assumes the "user" will adjust the volume either with the controls or with verbal commands, So things will be a bit "clunky" in your results as you command the smart speaker volume level. Yes, you have an automated system that does most of what you want, but it is based on pieces not designed to be automated in this way, and you are running into the limitations caused by that. Yes, you could split the aux-out to bring it into another computer to listen to the sound level, and then using a sound input package get samples of what is playing, and analyze that data to get an average volume, and then issues the command to adjust the volume level. What you seem to be missing is that you could get the podcasts from a browser, and all a browser is is a program. It isn't that much work to write a rudimentary browser in python, especially if you don't actually need to display the results to a user, but are only trying to automate a particular task. You seem to feel strongly invested in your current code base, which is understandable, but it seems you have reached a point where you don't want to live with the limitations CAUSED by that system. Yes, there is likely a way to tack on another layer of "stuff" to adjust for this issue, but it likely is going to require some real programming. It may well be the design I am suggesting, of writing a program to fetch the podcast and save it requires a bit more work to get to the level you currently are at, but the results are a system that is actually designed to be controlled by automation. Maybe it is beyond you ability, but then so might the programming to get the volume. I will also add, that the way your describe going to your "community" gives me questions if this is a violation of copyright. Maybe it is something you can "Get away with", but I am not sure what you are doing is actually legitimate. On 5/29/22 1:09 AM, Steve GS wrote: > You really need to understand what I am trying to do. > It is not a simple lesson in use of podcasts. > This is an automated system. I call it my NPR Jukebox. > > 15 years ago, I started with hourly URL calls to a browser to record specific NPR programs. It took a lot of coordination. I had to use IE because I needed to start and stop browsers on the hour and IE was the only one that could do that. Then web sites started "upgrading" to Edge. Through some trickery I was able to get Edge to do what IE did but it was unstable. > > I then discovered the Echo Smart Speaker. I set my program to announce the broadcast station or podcast by speaker to the smart speaker and it cured a lot of headaches. I then was able to call podcasts because the Echo handles them through TuneIn. Unfortunately, not all broadcasts ae available as podcasts. > > I am not here diddling around just playing podcasts. Let me repeat > what I have already said. It is an automated system. Every hour for > 48 hours on every weekend, my system using a well-developed Excel/VBA > program that vocally talks to the SS hourly. The SS finds the audio > and sends it to my Audacity recorder on another computer through > aux-out to mic-in cable. The selections of audio are also transmitted > to the community at the time of recording > > That part of the system is almost flawless, well compared to that I had before. Although the quality, tone, and timing of the announcement, the SS still gets confused once in a while and goes silent for the hour. I need to detect this too. > > Ok, now back to the original question. > > Podcasts and broadcasts apparently do not use the Dolby tone to balance the audio levels. And I receive highly fluctuating levels of audio. Sometimes it is between sides of a conversation, sometimes it is the podcast vs station identifications, then it is great differences between one web site and another. Then there's the differences with male and female voices. Imagine that you are watching TV late night then the commercials COME IN AT FULL BLAST. > > The technology of the industry grew up with male voices and apparently sees no reason to adjust for female. I have worked with audio systems and making recordings for more years that I want to admit. > > All I want is software to detect level changes over time and attempt to equalize them. > It has to be work between the SS and the recorder and is to be checking all the time. > > The code is to: Listen to the audio level for about 10 seconds or so and raise or lower the level in small increments. > It has nothing to do with understanding how to grab podcasts. The system is working very well for that. > > > Footnote: > ?What rhymes with orange?? > ?No, it doesn?t..? > > > > -----Original Message----- > From: Richard Damon On Behalf Of Richard > Damon > Sent: Saturday, May 28, 2022 11:37 PM > To: Steve GS > Subject: Re: Automatic Gain Control in Python? > > On 5/28/22 8:17 PM, Steve GS wrote: >> "My first thought is you are solving the wrong problem. What seems a >> better option would be to get your code to actually connect up to the >> podcast and just download the audio directly, rather than trying to >> get the smart speaker to play the audio and record it with a microphone." >> >> The smart-speaker is bringing in the podcast by hourly automated >> commands and sending by audio cable to a computer which is recording >> it with Audacity. This is an automated system that runs for 48 hours every weekend. >> Its output is played live throughout the facility and is also >> recorded for replay through the week. >> >> No download to use. >> >> AGC is to happen when the Smart Speaker is playing it, real time. >> Any post-record editing would be a horrendous task to say the least. >> > My guess is you don't understand how "Podcasts" work. All they are is a web resource that your Browser/Smart Device makes a request off, and the contents are streamed over the internet to that device, which then plays it. Smart Speakers just have a program that knows how to access these. > > Since they can be listened to on a web browser, a program can download the data. They might be doing things to make this harder, but that is a sign that you shouldn't be doing this in the first place. > > Often, the data format that is streamed is exactly like the file format for storing an audio file (since that is what the code in the browser is built to handle). > > It may be a bit of work to figure out the access methods to get the data, but this is the sort of job that computers were designed to do. > Trying to make things that weren't designed to be remote controlled to be remote controlled may well be a lot more work. > > -- > Richard Damon > -- Richard Damon From bschollnick at schollnick.net Sun May 29 11:17:47 2022 From: bschollnick at schollnick.net (Benjamin Schollnick) Date: Sun, 29 May 2022 11:17:47 -0400 Subject: Automatic Gain Control in Python? In-Reply-To: <003601d8736d$5cd3e610$167bb230$@SGA.Ninja> References: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> <015301d872f1$6bf9afc0$43ed0f40$@SGA.Ninja> <017501d8731a$43882750$ca9875f0$@SGA.Ninja> <003601d8736d$5cd3e610$167bb230$@SGA.Ninja> Message-ID: <5148EB0B-A337-4E06-A2B6-FBDDEF0CA594@schollnick.net> Okay, you are capturing the audio stream as a digital file somewhere, correct? Why not just right a 3rd party package to normalize the audio levels in the digital file? It?ll be faster, and probably easier than trying to do it in real time? eg. https://campus.datacamp.com/courses/spoken-language-processing-in-python/manipulating-audio-files-with-pydub?ex=8 Normalizing an audio file with PyDub Sometimes you'll have audio files where the speech is loud in some portions and quiet in others. Having this variance in volume can hinder transcription. Luckily, PyDub's effects module has a function called normalize() which finds the maximum volume of an AudioSegment, then adjusts the rest of the AudioSegment to be in proportion. This means the quiet parts will get a volume boost. You can listen to an example of an audio file which starts as loud then goes quiet, loud_then_quiet.wav, here . In this exercise, you'll use normalize() to normalize the volume of our file, making it sound more like this . or https://stackoverflow.com/questions/57925304/how-to-normalize-a-raw-audio-file-with-python - Benjamin > On May 29, 2022, at 11:04 AM, Steve GS wrote: > >>> From your description, your fundamental problem is you are trying to automatically "control" things that weren't designed to be automatically controlled in the way you are attempting. > > How so? I am sending commands to a smart speaker and it plays podcasts and broadcasts. > How is this a violation of SS design? > > =================== >>> The smart speaker assumes the "user" will adjust the volume either with the controls or with verbal commands, So things will be a bit "clunky" in your results as you command the smart speaker volume level. > > So, you want me to sit here for every hour of the weekend and monitor the audio levels for a result that will get, at best, one replay when I believe it can be automated. > > =================== >>> Yes, you have an automated system that does most of what you want, but it is based on pieces not designed to be automated in this way, and you are running into the limitations caused by that. > > Again, what limitations of the SS am I violating? It is designed to receive commands and play the audio. > Also, what makes you think that you know how my program is based? > > =================== >>> Yes, you could split the aux-out to bring it into another computer to listen to the sound level, and then using a sound input package get samples of what is playing, and analyze that data to get an average volume, and then issues the command to adjust the volume level. > > Hmmm, is that not my original question? Are you suggesting to monitor the audio, sense it for volume changes and apply those changes to the original audio? One thing that may have to happen is a timed-delay to all for the AGC to work. This may require a correlation circuit. > > ================== >>> What you seem to be missing is that you could get the podcasts from a browser, and all a browser is is a program. It isn't that much work to write a rudimentary browser in python, especially if you don't actually need to display the results to a user, but are only trying to automate a particular task. > > Writing my own browser in Python might work. Do you have a sample one that I could twerk to fit my needs? > I would have to be able to invoke it and an hour later devoke it least I end up with multiple audio channels playing. > > Either way, I would still need an AGC program which was my original question. > > =================== >>> You seem to feel strongly invested in your current code base, which is understandable, but it seems you have reached a point where you don't want to live with the limitations CAUSED by that system. > > The changes in volume are not CAUSED by my program. The want to fix them is a new development to improve the product. The volume fluctuations are causes, or neglections, by the engineers at the sources of podcasts and broadcasts. > >>> Yes, there is likely a way to tack on another layer of "stuff" to adjust for this issue, but it likely is going to require some real programming. > >>> It may well be the design I am suggesting, of writing a program to fetch the podcast and save it requires a bit more work to get to the level you currently are at, but the results are a system that is actually designed to be controlled by automation. Maybe it is beyond you ability, but then so might the programming to get the volume. > > "Real programming"?, REAL PROGRAMMING? Well at this I have to say "Well duh". I have more than 40 years of programming in languages including assembly, Pascal, C, C++, BASIC, Visual BASIC, COBOL, VBA and Python for starters. I even dipped into AT&T's SNOBAL. Starting in the mid-80s, I taught Visual BASIC and VBA for 15 years in high school and at the University level. > > I have 2000 lines of code of Python that helped me to bring my A1c reading from 9.0 to 6.1 thank you. I have a program of similar size that tracks the 450 specific plants in my garden. It even monitors degree-days to help me know when certain insects attack and when to tend to individual needs. This Excel/Jukebox program is also about 2000 lines of code. (Seems to be a pattern here). All of these programs are decorated with numerous pops and whistles that make programming and use quite pleasurable, > > So, do you I think I am now ready for some "real programming". > > =================== >>> I will also add, that the way your describe going to your "community" > gives me questions if this is a violation of copyright. Maybe it is something you can "Get away with", but I am not sure what you are doing is actually legitimate. > > Yes, I have been through that. It is totally legal to record NPR broadcasts for replay as long as they are not retained for than a month or for multiple replays. Your suggestion to download and play a podcast or broadcast is legal only for live replay. My want to record them for one replay for my own use. Personal play is a different story. > > > > > -----Original Message----- > From: Richard Damon On Behalf Of Richard Damon > Sent: Sunday, May 29, 2022 8:03 AM > To: Steve GS ; Python > Subject: Re: Automatic Gain Control in Python? > > From your description, your fundamental problem is you are trying to automatically "control" things that weren't designed to be automatically controlled in the way you are attempting. > > The smart speaker assumes the "user" will adjust the volume either with the controls or with verbal commands, So things will be a bit "clunky" > in your results as you command the smart speaker volume level. > > Yes, you have an automated system that does most of what you want, but it is based on pieces not designed to be automated in this way, and you are running into the limitations caused by that. > > Yes, you could split the aux-out to bring it into another computer to listen to the sound level, and then using a sound input package get samples of what is playing, and analyze that data to get an average volume, and then issues the command to adjust the volume level. > > What you seem to be missing is that you could get the podcasts from a browser, and all a browser is is a program. It isn't that much work to write a rudimentary browser in python, especially if you don't actually need to display the results to a user, but are only trying to automate a particular task. > > You seem to feel strongly invested in your current code base, which is understandable, but it seems you have reached a point where you don't want to live with the limitations CAUSED by that system. Yes, there is likely a way to tack on another layer of "stuff" to adjust for this issue, but it likely is going to require some real programming. > > It may well be the design I am suggesting, of writing a program to fetch the podcast and save it requires a bit more work to get to the level you currently are at, but the results are a system that is actually designed to be controlled by automation. Maybe it is beyond you ability, but then so might the programming to get the volume. > > I will also add, that the way your describe going to your "community" > gives me questions if this is a violation of copyright. Maybe it is something you can "Get away with", but I am not sure what you are doing is actually legitimate. > > On 5/29/22 1:09 AM, Steve GS wrote: >> You really need to understand what I am trying to do. >> It is not a simple lesson in use of podcasts. >> This is an automated system. I call it my NPR Jukebox. >> >> 15 years ago, I started with hourly URL calls to a browser to record specific NPR programs. It took a lot of coordination. I had to use IE because I needed to start and stop browsers on the hour and IE was the only one that could do that. Then web sites started "upgrading" to Edge. Through some trickery I was able to get Edge to do what IE did but it was unstable. >> >> I then discovered the Echo Smart Speaker. I set my program to announce the broadcast station or podcast by speaker to the smart speaker and it cured a lot of headaches. I then was able to call podcasts because the Echo handles them through TuneIn. Unfortunately, not all broadcasts ae available as podcasts. >> >> I am not here diddling around just playing podcasts. Let me repeat >> what I have already said. It is an automated system. Every hour for >> 48 hours on every weekend, my system using a well-developed Excel/VBA >> program that vocally talks to the SS hourly. The SS finds the audio >> and sends it to my Audacity recorder on another computer through >> aux-out to mic-in cable. The selections of audio are also transmitted >> to the community at the time of recording >> >> That part of the system is almost flawless, well compared to that I had before. Although the quality, tone, and timing of the announcement, the SS still gets confused once in a while and goes silent for the hour. I need to detect this too. >> >> Ok, now back to the original question. >> >> Podcasts and broadcasts apparently do not use the Dolby tone to balance the audio levels. And I receive highly fluctuating levels of audio. Sometimes it is between sides of a conversation, sometimes it is the podcast vs station identifications, then it is great differences between one web site and another. Then there's the differences with male and female voices. Imagine that you are watching TV late night then the commercials COME IN AT FULL BLAST. >> >> The technology of the industry grew up with male voices and apparently sees no reason to adjust for female. I have worked with audio systems and making recordings for more years that I want to admit. >> >> All I want is software to detect level changes over time and attempt to equalize them. >> It has to be work between the SS and the recorder and is to be checking all the time. >> >> The code is to: Listen to the audio level for about 10 seconds or so and raise or lower the level in small increments. >> It has nothing to do with understanding how to grab podcasts. The system is working very well for that. >> >> >> Footnote: >> ?What rhymes with orange?? >> ?No, it doesn?t..? >> >> >> >> -----Original Message----- >> From: Richard Damon On Behalf Of Richard >> Damon >> Sent: Saturday, May 28, 2022 11:37 PM >> To: Steve GS >> Subject: Re: Automatic Gain Control in Python? >> >> On 5/28/22 8:17 PM, Steve GS wrote: >>> "My first thought is you are solving the wrong problem. What seems a >>> better option would be to get your code to actually connect up to the >>> podcast and just download the audio directly, rather than trying to >>> get the smart speaker to play the audio and record it with a microphone." >>> >>> The smart-speaker is bringing in the podcast by hourly automated >>> commands and sending by audio cable to a computer which is recording >>> it with Audacity. This is an automated system that runs for 48 hours every weekend. >>> Its output is played live throughout the facility and is also >>> recorded for replay through the week. >>> >>> No download to use. >>> >>> AGC is to happen when the Smart Speaker is playing it, real time. >>> Any post-record editing would be a horrendous task to say the least. >>> >> My guess is you don't understand how "Podcasts" work. All they are is a web resource that your Browser/Smart Device makes a request off, and the contents are streamed over the internet to that device, which then plays it. Smart Speakers just have a program that knows how to access these. >> >> Since they can be listened to on a web browser, a program can download the data. They might be doing things to make this harder, but that is a sign that you shouldn't be doing this in the first place. >> >> Often, the data format that is streamed is exactly like the file format for storing an audio file (since that is what the code in the browser is built to handle). >> >> It may be a bit of work to figure out the access methods to get the data, but this is the sort of job that computers were designed to do. >> Trying to make things that weren't designed to be remote controlled to be remote controlled may well be a lot more work. >> >> -- >> Richard Damon >> > > -- > Richard Damon > > -- > https://mail.python.org/mailman/listinfo/python-list From python at mrabarnett.plus.com Sun May 29 13:27:20 2022 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 29 May 2022 18:27:20 +0100 Subject: Automatic Gain Control in Python? In-Reply-To: <5148EB0B-A337-4E06-A2B6-FBDDEF0CA594@schollnick.net> References: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> <015301d872f1$6bf9afc0$43ed0f40$@SGA.Ninja> <017501d8731a$43882750$ca9875f0$@SGA.Ninja> <003601d8736d$5cd3e610$167bb230$@SGA.Ninja> <5148EB0B-A337-4E06-A2B6-FBDDEF0CA594@schollnick.net> Message-ID: <0ab28130-f081-ee97-5a40-5653a3e56359@mrabarnett.plus.com> On 2022-05-29 16:17, Benjamin Schollnick wrote: > Okay, you are capturing the audio stream as a digital file somewhere, correct? > > Why not just right a 3rd party package to normalize the audio levels in the digital file? It?ll be faster, and probably easier than trying to do it in real time? > > eg. https://campus.datacamp.com/courses/spoken-language-processing-in-python/manipulating-audio-files-with-pydub?ex=8 > > Normalizing an audio file with PyDub > > Sometimes you'll have audio files where the speech is loud in some portions and quiet in others. Having this variance in volume can hinder transcription. > > Luckily, PyDub's effects module has a function called normalize() which finds the maximum volume of an AudioSegment, then adjusts the rest of the AudioSegment to be in proportion. This means the quiet parts will get a volume boost. > > You can listen to an example of an audio file which starts as loud then goes quiet, loud_then_quiet.wav, here . > > In this exercise, you'll use normalize() to normalize the volume of our file, making it sound more like this . > > or > > https://stackoverflow.com/questions/57925304/how-to-normalize-a-raw-audio-file-with-python > > [snip] Here's a sample script that uses pyaudio instead of Audacity. You can check whether the podcast is playing by checking the volume soon after it should've started. Pyaudio can also read and write files. import pyaudio import time import numpy as np WIDTH = 2 CHANNELS = 2 RATE = 44100 MAX_VOL = 1024 GAIN_STEP = 0.2 LOUDER = 1 + GAIN_STEP QUIETER = 1 - GAIN_STEP gain = 1 p = pyaudio.PyAudio() def callback(data, frame_count, time_info, status): global gain # Decode the bytestream chunk = np.frombuffer(data, dtype=np.int16) # Adjust the volume. chunk = (chunk.astype(np.double) * gain).astype(np.int16) # Adjust the gain according to the current maximum volume. max_vol = max(chunk) if max_vol < MAX_VOL: gain *= LOUDER elif max_vol > MAX_VOL: gain *= QUIETER return (chunk.tobytes(), pyaudio.paContinue) stream = p.open(format=p.get_format_from_width(WIDTH), channels=CHANNELS, rate=RATE, input=True, output=True, stream_callback=callback) stream.start_stream() while stream.is_active(): time.sleep(0.1) stream.stop_stream() stream.close() p.terminate() From 00jhenryg at gmail.com Sat May 28 22:11:00 2022 From: 00jhenryg at gmail.com (Jack Gilbert) Date: Sat, 28 May 2022 21:11:00 -0500 Subject: help, please, with 3.10.4 install Message-ID: I downloaded 3.10.4 on a 64 bit , 8.1 I can see IDLE shell 3.10.1, I see Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)] on win32 also, the same line: Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)] on win32 in CMD prompt for the life of me I can't figure out how to launch python?? I did click add to path in the install thanks From auriocus at gmx.de Sun May 29 03:22:44 2022 From: auriocus at gmx.de (Christian Gollwitzer) Date: Sun, 29 May 2022 09:22:44 +0200 Subject: Automatic Gain Control in Python? In-Reply-To: References: Message-ID: Am 29.05.22 um 00:45 schrieb Stefan Ram: > "Steve GS" writes: >> Subject: Automatic Gain Control in Python? > > Automatic Gain Control in Python is trivial. You have a list > of samples and normalize them, i.e., divide by max. Slightly > simplified > > [ s/max( samples )for s in samples ] > > (where sample values are normalized to the range -1,+1.) No, that's not it. Loudness is perceived in a different way, the crudest approximation is the standard deviation averaged over small frames, better measures take into account that the ear has a strongly frequency dependent response. For similar sound files, like podcasts which are voice-heavy, the stddev works reasoably well. If the normalization only reduces the volume, then dividing by the max is sufficient. However if you also want to raise the volume, then you need dynamic range compression. If you have ever tried to record some music with a simple microphone and a computer, you would have noticed that the recording is very soft, when normalized to the max. Commercial music is incredibly loud, and you might have wondered, how they do that. Google for "Loudness war" and "dynamic range compression" if you want to understand it in detail. Christian From martin.schoon at gmail.com Sun May 29 05:19:41 2022 From: martin.schoon at gmail.com (Martin =?UTF-8?Q?Sch=C3=B6=C3=B6n?=) Date: 29 May 2022 09:19:41 GMT Subject: Automatic Gain Control in Python? References: Message-ID: Den 2022-05-29 skrev Christian Gollwitzer : > Am 29.05.22 um 00:45 schrieb Stefan Ram: >> "Steve GS" writes: >>> Subject: Automatic Gain Control in Python? >> >> Automatic Gain Control in Python is trivial. You have a list >> of samples and normalize them, i.e., divide by max. Slightly >> simplified >> >> [ s/max( samples )for s in samples ] >> >> (where sample values are normalized to the range -1,+1.) > > No, that's not it. Loudness is perceived in a different way, the crudest > music is incredibly loud, and you might have wondered, how they do that. > > Google for "Loudness war" and "dynamic range compression" if you want to > understand it in detail. > I have no suggestions for solving the problem but it struck me that you might be interested in looking up a standard called EBU R128. Start with youtube and you find lectures/demos. Python connection; There is a Python package called ffmpeg-normalize which contains an implementation of EBU R128. AFAIK it works on files, not streaming audio. /Martin From redstone-cold at 163.com Sun May 29 08:57:20 2022 From: redstone-cold at 163.com (iMath) Date: Sun, 29 May 2022 05:57:20 -0700 (PDT) Subject: matplotlib basemap colorbar exception : Given element not contained in the stack In-Reply-To: References: Message-ID: <0a4226dc-6d7c-4878-a1d6-b93507932c2bn@googlegroups.com> please see the formated code at https://stackoverflow.com/questions/72423464/matplotlib-basemap-colorbar-exception-given-element-not-contained-in-the-stack From Gronicus at SGA.Ninja Sun May 29 11:32:28 2022 From: Gronicus at SGA.Ninja (Steve GS) Date: Sun, 29 May 2022 11:32:28 -0400 Subject: Automatic Gain Control in Python? In-Reply-To: <5148EB0B-A337-4E06-A2B6-FBDDEF0CA594@schollnick.net> References: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> <015301d872f1$6bf9afc0$43ed0f40$@SGA.Ninja> <017501d8731a$43882750$ca9875f0$@SGA.Ninja> <003601d8736d$5cd3e610$167bb230$@SGA.Ninja> <5148EB0B-A337-4E06-A2B6-FBDDEF0CA594@schollnick.net> Message-ID: <004f01d87371$4e579150$eb06b3f0$@SGA.Ninja> No, not a digital file. I am playing the podcast/broadcast live to the community using a separate network of smart speakers (Amazon Echo). Commands are being sent hourly through a speaker to the SS from an excel program. I want to monitor the audio level between the line-out of the SS and the input to another computer which then records the audio using Audacity for a single replay during the week. I think my first post should have started ?Fasten your seat belts, it is going to be a bumpy night?) Genie: You have three wishes. Me: I wish I had more wishes. Genie: You cannot wish for more wishes. Me: I wish I could. From: Benjamin Schollnick Sent: Sunday, May 29, 2022 11:18 AM To: Steve GS Cc: Richard Damon ; Python Subject: Re: Automatic Gain Control in Python? Okay, you are capturing the audio stream as a digital file somewhere, correct? Why not just right a 3rd party package to normalize the audio levels in the digital file? It?ll be faster, and probably easier than trying to do it in real time? eg. https://campus.datacamp.com/courses/spoken-language-processing-in-python/manipulating-audio-files-with-pydub?ex=8 Normalizing an audio file with PyDub Sometimes you'll have audio files where the speech is loud in some portions and quiet in others. Having this variance in volume can hinder transcription. Luckily, PyDub's effects module has a function called normalize() which finds the maximum volume of an AudioSegment, then adjusts the rest of the AudioSegment to be in proportion. This means the quiet parts will get a volume boost. You can listen to an example of an audio file which starts as loud then goes quiet, loud_then_quiet.wav, here. In this exercise, you'll use normalize() to normalize the volume of our file, making it sound more like this. or https://stackoverflow.com/questions/57925304/how-to-normalize-a-raw-audio-file-with-python - Benjamin On May 29, 2022, at 11:04 AM, Steve GS > wrote: >From your description, your fundamental problem is you are trying to automatically "control" things that weren't designed to be automatically controlled in the way you are attempting. How so? I am sending commands to a smart speaker and it plays podcasts and broadcasts. How is this a violation of SS design? =================== The smart speaker assumes the "user" will adjust the volume either with the controls or with verbal commands, So things will be a bit "clunky" in your results as you command the smart speaker volume level. So, you want me to sit here for every hour of the weekend and monitor the audio levels for a result that will get, at best, one replay when I believe it can be automated. =================== Yes, you have an automated system that does most of what you want, but it is based on pieces not designed to be automated in this way, and you are running into the limitations caused by that. Again, what limitations of the SS am I violating? It is designed to receive commands and play the audio. Also, what makes you think that you know how my program is based? =================== Yes, you could split the aux-out to bring it into another computer to listen to the sound level, and then using a sound input package get samples of what is playing, and analyze that data to get an average volume, and then issues the command to adjust the volume level. Hmmm, is that not my original question? Are you suggesting to monitor the audio, sense it for volume changes and apply those changes to the original audio? One thing that may have to happen is a timed-delay to all for the AGC to work. This may require a correlation circuit. ================== What you seem to be missing is that you could get the podcasts from a browser, and all a browser is is a program. It isn't that much work to write a rudimentary browser in python, especially if you don't actually need to display the results to a user, but are only trying to automate a particular task. Writing my own browser in Python might work. Do you have a sample one that I could twerk to fit my needs? I would have to be able to invoke it and an hour later devoke it least I end up with multiple audio channels playing. Either way, I would still need an AGC program which was my original question. =================== You seem to feel strongly invested in your current code base, which is understandable, but it seems you have reached a point where you don't want to live with the limitations CAUSED by that system. The changes in volume are not CAUSED by my program. The want to fix them is a new development to improve the product. The volume fluctuations are causes, or neglections, by the engineers at the sources of podcasts and broadcasts. Yes, there is likely a way to tack on another layer of "stuff" to adjust for this issue, but it likely is going to require some real programming. It may well be the design I am suggesting, of writing a program to fetch the podcast and save it requires a bit more work to get to the level you currently are at, but the results are a system that is actually designed to be controlled by automation. Maybe it is beyond you ability, but then so might the programming to get the volume. "Real programming"?, REAL PROGRAMMING? Well at this I have to say "Well duh". I have more than 40 years of programming in languages including assembly, Pascal, C, C++, BASIC, Visual BASIC, COBOL, VBA and Python for starters. I even dipped into AT&T's SNOBAL. Starting in the mid-80s, I taught Visual BASIC and VBA for 15 years in high school and at the University level. I have 2000 lines of code of Python that helped me to bring my A1c reading from 9.0 to 6.1 thank you. I have a program of similar size that tracks the 450 specific plants in my garden. It even monitors degree-days to help me know when certain insects attack and when to tend to individual needs. This Excel/Jukebox program is also about 2000 lines of code. (Seems to be a pattern here). All of these programs are decorated with numerous pops and whistles that make programming and use quite pleasurable, So, do you I think I am now ready for some "real programming". =================== I will also add, that the way your describe going to your "community" gives me questions if this is a violation of copyright. Maybe it is something you can "Get away with", but I am not sure what you are doing is actually legitimate. Yes, I have been through that. It is totally legal to record NPR broadcasts for replay as long as they are not retained for than a month or for multiple replays. Your suggestion to download and play a podcast or broadcast is legal only for live replay. My want to record them for one replay for my own use. Personal play is a different story. -----Original Message----- From: Richard Damon > On Behalf Of Richard Damon Sent: Sunday, May 29, 2022 8:03 AM To: Steve GS >; Python > Subject: Re: Automatic Gain Control in Python? >From your description, your fundamental problem is you are trying to automatically "control" things that weren't designed to be automatically controlled in the way you are attempting. The smart speaker assumes the "user" will adjust the volume either with the controls or with verbal commands, So things will be a bit "clunky" in your results as you command the smart speaker volume level. Yes, you have an automated system that does most of what you want, but it is based on pieces not designed to be automated in this way, and you are running into the limitations caused by that. Yes, you could split the aux-out to bring it into another computer to listen to the sound level, and then using a sound input package get samples of what is playing, and analyze that data to get an average volume, and then issues the command to adjust the volume level. What you seem to be missing is that you could get the podcasts from a browser, and all a browser is is a program. It isn't that much work to write a rudimentary browser in python, especially if you don't actually need to display the results to a user, but are only trying to automate a particular task. You seem to feel strongly invested in your current code base, which is understandable, but it seems you have reached a point where you don't want to live with the limitations CAUSED by that system. Yes, there is likely a way to tack on another layer of "stuff" to adjust for this issue, but it likely is going to require some real programming. It may well be the design I am suggesting, of writing a program to fetch the podcast and save it requires a bit more work to get to the level you currently are at, but the results are a system that is actually designed to be controlled by automation. Maybe it is beyond you ability, but then so might the programming to get the volume. I will also add, that the way your describe going to your "community" gives me questions if this is a violation of copyright. Maybe it is something you can "Get away with", but I am not sure what you are doing is actually legitimate. On 5/29/22 1:09 AM, Steve GS wrote: You really need to understand what I am trying to do. It is not a simple lesson in use of podcasts. This is an automated system. I call it my NPR Jukebox. 15 years ago, I started with hourly URL calls to a browser to record specific NPR programs. It took a lot of coordination. I had to use IE because I needed to start and stop browsers on the hour and IE was the only one that could do that. Then web sites started "upgrading" to Edge. Through some trickery I was able to get Edge to do what IE did but it was unstable. I then discovered the Echo Smart Speaker. I set my program to announce the broadcast station or podcast by speaker to the smart speaker and it cured a lot of headaches. I then was able to call podcasts because the Echo handles them through TuneIn. Unfortunately, not all broadcasts ae available as podcasts. I am not here diddling around just playing podcasts. Let me repeat what I have already said. It is an automated system. Every hour for 48 hours on every weekend, my system using a well-developed Excel/VBA program that vocally talks to the SS hourly. The SS finds the audio and sends it to my Audacity recorder on another computer through aux-out to mic-in cable. The selections of audio are also transmitted to the community at the time of recording That part of the system is almost flawless, well compared to that I had before. Although the quality, tone, and timing of the announcement, the SS still gets confused once in a while and goes silent for the hour. I need to detect this too. Ok, now back to the original question. Podcasts and broadcasts apparently do not use the Dolby tone to balance the audio levels. And I receive highly fluctuating levels of audio. Sometimes it is between sides of a conversation, sometimes it is the podcast vs station identifications, then it is great differences between one web site and another. Then there's the differences with male and female voices. Imagine that you are watching TV late night then the commercials COME IN AT FULL BLAST. The technology of the industry grew up with male voices and apparently sees no reason to adjust for female. I have worked with audio systems and making recordings for more years that I want to admit. All I want is software to detect level changes over time and attempt to equalize them. It has to be work between the SS and the recorder and is to be checking all the time. The code is to: Listen to the audio level for about 10 seconds or so and raise or lower the level in small increments. It has nothing to do with understanding how to grab podcasts. The system is working very well for that. Footnote: ?What rhymes with orange?? ?No, it doesn?t..? -----Original Message----- From: Richard Damon > On Behalf Of Richard Damon Sent: Saturday, May 28, 2022 11:37 PM To: Steve GS > Subject: Re: Automatic Gain Control in Python? On 5/28/22 8:17 PM, Steve GS wrote: "My first thought is you are solving the wrong problem. What seems a better option would be to get your code to actually connect up to the podcast and just download the audio directly, rather than trying to get the smart speaker to play the audio and record it with a microphone." The smart-speaker is bringing in the podcast by hourly automated commands and sending by audio cable to a computer which is recording it with Audacity. This is an automated system that runs for 48 hours every weekend. Its output is played live throughout the facility and is also recorded for replay through the week. No download to use. AGC is to happen when the Smart Speaker is playing it, real time. Any post-record editing would be a horrendous task to say the least. My guess is you don't understand how "Podcasts" work. All they are is a web resource that your Browser/Smart Device makes a request off, and the contents are streamed over the internet to that device, which then plays it. Smart Speakers just have a program that knows how to access these. Since they can be listened to on a web browser, a program can download the data. They might be doing things to make this harder, but that is a sign that you shouldn't be doing this in the first place. Often, the data format that is streamed is exactly like the file format for storing an audio file (since that is what the code in the browser is built to handle). It may be a bit of work to figure out the access methods to get the data, but this is the sort of job that computers were designed to do. Trying to make things that weren't designed to be remote controlled to be remote controlled may well be a lot more work. -- Richard Damon -- Richard Damon -- https://mail.python.org/mailman/listinfo/python-list From michael.stemper at gmail.com Sun May 29 11:44:04 2022 From: michael.stemper at gmail.com (Michael F. Stemper) Date: Sun, 29 May 2022 10:44:04 -0500 Subject: Automatic Gain Control in Python? In-Reply-To: References: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> <015301d872f1$6bf9afc0$43ed0f40$@SGA.Ninja> <017501d8731a$43882750$ca9875f0$@SGA.Ninja> <003601d8736d$5cd3e610$167bb230$@SGA.Ninja> Message-ID: On 29/05/2022 10.04, Steve GS wrote: >>> What you seem to be missing is that you could get the podcasts from a browser, and all a browser is is a program. It isn't that much work to write a rudimentary browser in python, especially if you don't actually need to display the results to a user, but are only trying to automate a particular task. > > Writing my own browser in Python might work. Do you have a sample one that I could twerk to fit my needs? > I would have to be able to invoke it and an hour later devoke it least I end up with multiple audio channels playing. Somebody has already shown the rudiments of urllib. Another option to consider is the use of something like curl or wget to download the podcasts, which can be automated separately from your replay program. -- Michael F. Stemper This email is to be read by its intended recipient only. Any other party reading is required by the EULA to send me $500.00. From bschollnick at schollnick.net Sun May 29 11:50:39 2022 From: bschollnick at schollnick.net (Benjamin Schollnick) Date: Sun, 29 May 2022 11:50:39 -0400 Subject: Automatic Gain Control in Python? In-Reply-To: <004f01d87371$4e579150$eb06b3f0$@SGA.Ninja> References: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> <015301d872f1$6bf9afc0$43ed0f40$@SGA.Ninja> <017501d8731a$43882750$ca9875f0$@SGA.Ninja> <003601d8736d$5cd3e610$167bb230$@SGA.Ninja> <5148EB0B-A337-4E06-A2B6-FBDDEF0CA594@schollnick.net> <004f01d87371$4e579150$eb06b3f0$@SGA.Ninja> Message-ID: Well, maybe a different approach might be better? https://www.amazon.com/LEMONKTV-Control-Device-Karaoke-Machine/dp/B07H49JB8S I?m unclear on how bringing the audio track into a computer to examine detect the audio level, and then adjusting the volume out of the Smart Speaker is really doing more than adding complexity. An all analog solution might be the better route, although finding something that is inexpensive might be an issue as well. - Benjamin > On May 29, 2022, at 11:32 AM, Steve GS wrote: > > No, not a digital file. I am playing the podcast/broadcast live to the community using a separate network of smart speakers (Amazon Echo). Commands are being sent hourly through a speaker to the SS from an excel program. I want to monitor the audio level between the line-out of the SS and the input to another computer which then records the audio using Audacity for a single replay during the week. > > I think my first post should have started ?Fasten your seat belts, it is going to be a bumpy night?) > > > Genie: You have three wishes. > Me: I wish I had more wishes. > Genie: You cannot wish for more wishes. > Me: I wish I could. > > From: Benjamin Schollnick > > Sent: Sunday, May 29, 2022 11:18 AM > To: Steve GS > > Cc: Richard Damon >; Python > > Subject: Re: Automatic Gain Control in Python? > > Okay, you are capturing the audio stream as a digital file somewhere, correct? > > Why not just right a 3rd party package to normalize the audio levels in the digital file? It?ll be faster, and probably easier than trying to do it in real time? > > eg. https://campus.datacamp.com/courses/spoken-language-processing-in-python/manipulating-audio-files-with-pydub?ex=8 > >> Normalizing an audio file with PyDub >> >> Sometimes you'll have audio files where the speech is loud in some portions and quiet in others. Having this variance in volume can hinder transcription. >> Luckily, PyDub's effects module has a function called normalize() which finds the maximum volume of an AudioSegment, then adjusts the rest of the AudioSegment to be in proportion. This means the quiet parts will get a volume boost. >> You can listen to an example of an audio file which starts as loud then goes quiet, loud_then_quiet.wav, here . >> In this exercise, you'll use normalize() to normalize the volume of our file, making it sound more like this . > or > > https://stackoverflow.com/questions/57925304/how-to-normalize-a-raw-audio-file-with-python > > > > - Benjamin > > >> On May 29, 2022, at 11:04 AM, Steve GS > wrote: >> >>>> From your description, your fundamental problem is you are trying to automatically "control" things that weren't designed to be automatically controlled in the way you are attempting. >> >> How so? I am sending commands to a smart speaker and it plays podcasts and broadcasts. >> How is this a violation of SS design? >> >> =================== >> >>>> The smart speaker assumes the "user" will adjust the volume either with the controls or with verbal commands, So things will be a bit "clunky" in your results as you command the smart speaker volume level. >> >> So, you want me to sit here for every hour of the weekend and monitor the audio levels for a result that will get, at best, one replay when I believe it can be automated. >> >> =================== >> >>>> Yes, you have an automated system that does most of what you want, but it is based on pieces not designed to be automated in this way, and you are running into the limitations caused by that. >> >> Again, what limitations of the SS am I violating? It is designed to receive commands and play the audio. >> Also, what makes you think that you know how my program is based? >> >> =================== >> >>>> Yes, you could split the aux-out to bring it into another computer to listen to the sound level, and then using a sound input package get samples of what is playing, and analyze that data to get an average volume, and then issues the command to adjust the volume level. >> >> Hmmm, is that not my original question? Are you suggesting to monitor the audio, sense it for volume changes and apply those changes to the original audio? One thing that may have to happen is a timed-delay to all for the AGC to work. This may require a correlation circuit. >> >> ================== >> >>>> What you seem to be missing is that you could get the podcasts from a browser, and all a browser is is a program. It isn't that much work to write a rudimentary browser in python, especially if you don't actually need to display the results to a user, but are only trying to automate a particular task. >> >> Writing my own browser in Python might work. Do you have a sample one that I could twerk to fit my needs? >> I would have to be able to invoke it and an hour later devoke it least I end up with multiple audio channels playing. >> >> Either way, I would still need an AGC program which was my original question. >> >> =================== >> >>>> You seem to feel strongly invested in your current code base, which is understandable, but it seems you have reached a point where you don't want to live with the limitations CAUSED by that system. >> >> The changes in volume are not CAUSED by my program. The want to fix them is a new development to improve the product. The volume fluctuations are causes, or neglections, by the engineers at the sources of podcasts and broadcasts. >> >> >>>> Yes, there is likely a way to tack on another layer of "stuff" to adjust for this issue, but it likely is going to require some real programming. >> >> >>>> It may well be the design I am suggesting, of writing a program to fetch the podcast and save it requires a bit more work to get to the level you currently are at, but the results are a system that is actually designed to be controlled by automation. Maybe it is beyond you ability, but then so might the programming to get the volume. >> >> "Real programming"?, REAL PROGRAMMING? Well at this I have to say "Well duh". I have more than 40 years of programming in languages including assembly, Pascal, C, C++, BASIC, Visual BASIC, COBOL, VBA and Python for starters. I even dipped into AT&T's SNOBAL. Starting in the mid-80s, I taught Visual BASIC and VBA for 15 years in high school and at the University level. >> >> I have 2000 lines of code of Python that helped me to bring my A1c reading from 9.0 to 6.1 thank you. I have a program of similar size that tracks the 450 specific plants in my garden. It even monitors degree-days to help me know when certain insects attack and when to tend to individual needs. This Excel/Jukebox program is also about 2000 lines of code. (Seems to be a pattern here). All of these programs are decorated with numerous pops and whistles that make programming and use quite pleasurable, >> >> So, do you I think I am now ready for some "real programming". >> >> =================== >> >>>> I will also add, that the way your describe going to your "community" >> gives me questions if this is a violation of copyright. Maybe it is something you can "Get away with", but I am not sure what you are doing is actually legitimate. >> >> Yes, I have been through that. It is totally legal to record NPR broadcasts for replay as long as they are not retained for than a month or for multiple replays. Your suggestion to download and play a podcast or broadcast is legal only for live replay. My want to record them for one replay for my own use. Personal play is a different story. >> >> >> >> >> -----Original Message----- >> From: Richard Damon > On Behalf Of Richard Damon >> Sent: Sunday, May 29, 2022 8:03 AM >> To: Steve GS >; Python > >> Subject: Re: Automatic Gain Control in Python? >> >> From your description, your fundamental problem is you are trying to automatically "control" things that weren't designed to be automatically controlled in the way you are attempting. >> >> The smart speaker assumes the "user" will adjust the volume either with the controls or with verbal commands, So things will be a bit "clunky" >> in your results as you command the smart speaker volume level. >> >> Yes, you have an automated system that does most of what you want, but it is based on pieces not designed to be automated in this way, and you are running into the limitations caused by that. >> >> Yes, you could split the aux-out to bring it into another computer to listen to the sound level, and then using a sound input package get samples of what is playing, and analyze that data to get an average volume, and then issues the command to adjust the volume level. >> >> What you seem to be missing is that you could get the podcasts from a browser, and all a browser is is a program. It isn't that much work to write a rudimentary browser in python, especially if you don't actually need to display the results to a user, but are only trying to automate a particular task. >> >> You seem to feel strongly invested in your current code base, which is understandable, but it seems you have reached a point where you don't want to live with the limitations CAUSED by that system. Yes, there is likely a way to tack on another layer of "stuff" to adjust for this issue, but it likely is going to require some real programming. >> >> It may well be the design I am suggesting, of writing a program to fetch the podcast and save it requires a bit more work to get to the level you currently are at, but the results are a system that is actually designed to be controlled by automation. Maybe it is beyond you ability, but then so might the programming to get the volume. >> >> I will also add, that the way your describe going to your "community" >> gives me questions if this is a violation of copyright. Maybe it is something you can "Get away with", but I am not sure what you are doing is actually legitimate. >> >> On 5/29/22 1:09 AM, Steve GS wrote: >> >>> You really need to understand what I am trying to do. >>> It is not a simple lesson in use of podcasts. >>> This is an automated system. I call it my NPR Jukebox. >>> >>> 15 years ago, I started with hourly URL calls to a browser to record specific NPR programs. It took a lot of coordination. I had to use IE because I needed to start and stop browsers on the hour and IE was the only one that could do that. Then web sites started "upgrading" to Edge. Through some trickery I was able to get Edge to do what IE did but it was unstable. >>> >>> I then discovered the Echo Smart Speaker. I set my program to announce the broadcast station or podcast by speaker to the smart speaker and it cured a lot of headaches. I then was able to call podcasts because the Echo handles them through TuneIn. Unfortunately, not all broadcasts ae available as podcasts. >>> >>> I am not here diddling around just playing podcasts. Let me repeat >>> what I have already said. It is an automated system. Every hour for >>> 48 hours on every weekend, my system using a well-developed Excel/VBA >>> program that vocally talks to the SS hourly. The SS finds the audio >>> and sends it to my Audacity recorder on another computer through >>> aux-out to mic-in cable. The selections of audio are also transmitted >>> to the community at the time of recording >>> >>> That part of the system is almost flawless, well compared to that I had before. Although the quality, tone, and timing of the announcement, the SS still gets confused once in a while and goes silent for the hour. I need to detect this too. >>> >>> Ok, now back to the original question. >>> >>> Podcasts and broadcasts apparently do not use the Dolby tone to balance the audio levels. And I receive highly fluctuating levels of audio. Sometimes it is between sides of a conversation, sometimes it is the podcast vs station identifications, then it is great differences between one web site and another. Then there's the differences with male and female voices. Imagine that you are watching TV late night then the commercials COME IN AT FULL BLAST. >>> >>> The technology of the industry grew up with male voices and apparently sees no reason to adjust for female. I have worked with audio systems and making recordings for more years that I want to admit. >>> >>> All I want is software to detect level changes over time and attempt to equalize them. >>> It has to be work between the SS and the recorder and is to be checking all the time. >>> >>> The code is to: Listen to the audio level for about 10 seconds or so and raise or lower the level in small increments. >>> It has nothing to do with understanding how to grab podcasts. The system is working very well for that. >>> >>> >>> Footnote: >>> ?What rhymes with orange?? >>> ?No, it doesn?t..? >>> >>> >>> >>> -----Original Message----- >>> From: Richard Damon > On Behalf Of Richard >>> Damon >>> Sent: Saturday, May 28, 2022 11:37 PM >>> To: Steve GS > >>> Subject: Re: Automatic Gain Control in Python? >>> >>> On 5/28/22 8:17 PM, Steve GS wrote: >>> >>>> "My first thought is you are solving the wrong problem. What seems a >>>> better option would be to get your code to actually connect up to the >>>> podcast and just download the audio directly, rather than trying to >>>> get the smart speaker to play the audio and record it with a microphone." >>>> >>>> The smart-speaker is bringing in the podcast by hourly automated >>>> commands and sending by audio cable to a computer which is recording >>>> it with Audacity. This is an automated system that runs for 48 hours every weekend. >>>> Its output is played live throughout the facility and is also >>>> recorded for replay through the week. >>>> >>>> No download to use. >>>> >>>> AGC is to happen when the Smart Speaker is playing it, real time. >>>> Any post-record editing would be a horrendous task to say the least. >>>> >>> My guess is you don't understand how "Podcasts" work. All they are is a web resource that your Browser/Smart Device makes a request off, and the contents are streamed over the internet to that device, which then plays it. Smart Speakers just have a program that knows how to access these. >>> >>> Since they can be listened to on a web browser, a program can download the data. They might be doing things to make this harder, but that is a sign that you shouldn't be doing this in the first place. >>> >>> Often, the data format that is streamed is exactly like the file format for storing an audio file (since that is what the code in the browser is built to handle). >>> >>> It may be a bit of work to figure out the access methods to get the data, but this is the sort of job that computers were designed to do. >>> Trying to make things that weren't designed to be remote controlled to be remote controlled may well be a lot more work. >>> >>> -- >>> Richard Damon >>> >> >> -- >> Richard Damon >> >> -- >> https://mail.python.org/mailman/listinfo/python-list From grant.b.edwards at gmail.com Sun May 29 15:18:39 2022 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Sun, 29 May 2022 12:18:39 -0700 (PDT) Subject: Automatic Gain Control in Python? References: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> <015301d872f1$6bf9afc0$43ed0f40$@SGA.Ninja> <017501d8731a$43882750$ca9875f0$@SGA.Ninja> <003601d8736d$5cd3e610$167bb230$@SGA.Ninja> <5148EB0B-A337-4E06-A2B6-FBDDEF0CA594@schollnick.net> Message-ID: <6293c70f.1c69fb81.5d366.c1bc@mx.google.com> On 2022-05-29, Benjamin Schollnick wrote: > Why not just right a 3rd party package to normalize the audio levels > in the digital file? It?ll be faster, and probably easier than > trying to do it in real time? I doubt you even need to write any code to do that. Sox can normalize audio levels in files with a single command. http://sox.sourceforge.net/ -- Grant From bschollnick at schollnick.net Sun May 29 15:24:03 2022 From: bschollnick at schollnick.net (Benjamin Schollnick) Date: Sun, 29 May 2022 15:24:03 -0400 Subject: Automatic Gain Control in Python? In-Reply-To: <6293c70f.1c69fb81.5d366.c1bc@mx.google.com> References: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> <015301d872f1$6bf9afc0$43ed0f40$@SGA.Ninja> <017501d8731a$43882750$ca9875f0$@SGA.Ninja> <003601d8736d$5cd3e610$167bb230$@SGA.Ninja> <5148EB0B-A337-4E06-A2B6-FBDDEF0CA594@schollnick.net> <6293c70f.1c69fb81.5d366.c1bc@mx.google.com> Message-ID: <527C9F7E-94EF-401E-9AE2-187F7D305314@schollnick.net> > I doubt you even need to write any code to do that. Sox can normalize > audio levels in files with a single command. Correct. My phrasing was slightly misleading. There are plenty of applications that should do that. I was thinking of one that Marco Arment makes, but I can?t locate it for reference. - Benjamin > On May 29, 2022, at 3:18 PM, Grant Edwards wrote: > > On 2022-05-29, Benjamin Schollnick wrote: > >> Why not just right a 3rd party package to normalize the audio levels >> in the digital file? It?ll be faster, and probably easier than >> trying to do it in real time? > > I doubt you even need to write any code to do that. Sox can normalize > audio levels in files with a single command. > > http://sox.sourceforge.net/ > > -- > Grant > -- > https://mail.python.org/mailman/listinfo/python-list From python at mrabarnett.plus.com Sun May 29 15:29:01 2022 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 29 May 2022 20:29:01 +0100 Subject: matplotlib basemap colorbar exception : Given element not contained in the stack In-Reply-To: <0a4226dc-6d7c-4878-a1d6-b93507932c2bn@googlegroups.com> References: <0a4226dc-6d7c-4878-a1d6-b93507932c2bn@googlegroups.com> Message-ID: On 2022-05-29 13:57, iMath wrote: > please see the formated code at https://stackoverflow.com/questions/72423464/matplotlib-basemap-colorbar-exception-given-element-not-contained-in-the-stack The problem might be that you're passing "ax=self.ax" when you create the basemap, and you have: self.ax = self.map_canvas.figure.subplots() According to the docs, ".subplots" doesn't return the axes, it returns a _tuple_ of the figure and the axes. From PythonList at DancesWithMice.info Sun May 29 22:20:47 2022 From: PythonList at DancesWithMice.info (dn) Date: Mon, 30 May 2022 14:20:47 +1200 Subject: help, please, with 3.10.4 install In-Reply-To: References: Message-ID: On 29/05/2022 14.11, Jack Gilbert wrote: > I downloaded 3.10.4 on a 64 bit , 8.1 > I can see IDLE shell 3.10.1, I see Python 3.10.4 (tags/v3.10.4:9d38120, Mar > 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)] on win32 > > also, the same line: Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, > 23:13:41) [MSC v.1929 64 bit (AMD64)] on win32 in CMD prompt > > for the life of me I can't figure out how to launch python?? > > I did click add to path in the install > > thanks Please advise if https://docs.python.org/3/using/windows.html does not answer the question... -- Regards, =dn From mats at wichmann.us Mon May 30 11:32:52 2022 From: mats at wichmann.us (Mats Wichmann) Date: Mon, 30 May 2022 09:32:52 -0600 Subject: help, please, with 3.10.4 install In-Reply-To: References: Message-ID: <2984c326-6905-12f1-7648-28a12591e245@wichmann.us> On 5/28/22 20:11, Jack Gilbert wrote: > I downloaded 3.10.4 on a 64 bit , 8.1 > also, the same line: Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, > 23:13:41) [MSC v.1929 64 bit (AMD64)] on win32 in CMD prompt > > for the life of me I can't figure out how to launch python?? Sounds like you're launching it already? In a cmd shell, type: py And you should be good to go. See the page @dn pointed to. From redstone-cold at 163.com Mon May 30 02:41:35 2022 From: redstone-cold at 163.com (iMath) Date: Sun, 29 May 2022 23:41:35 -0700 (PDT) Subject: matplotlib basemap colorbar exception : Given element not contained in the stack In-Reply-To: References: <0a4226dc-6d7c-4878-a1d6-b93507932c2bn@googlegroups.com> Message-ID: ? 2022?5?30???? UTC+8 03:29:28? ??? > On 2022-05-29 13:57, iMath wrote: > > please see the formated code at https://stackoverflow.com/questions/72423464/matplotlib-basemap-colorbar-exception-given-element-not-contained-in-the-stack > The problem might be that you're passing "ax=self.ax" when you create > the basemap, and you have: > > self.ax = self.map_canvas.figure.subplots() > > According to the docs, ".subplots" doesn't return the axes, it returns a > _tuple_ of the figure and the axes. Thanks for your reply ! But as I printed the value of self.ax , it is actually an instance of AxesSubplot. From wlfraed at ix.netcom.com Mon May 30 12:19:29 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Mon, 30 May 2022 12:19:29 -0400 Subject: help, please, with 3.10.4 install References: Message-ID: On Sat, 28 May 2022 21:11:00 -0500, Jack Gilbert <00jhenryg at gmail.com> declaimed the following: >also, the same line: Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, >23:13:41) [MSC v.1929 64 bit (AMD64)] on win32 in CMD prompt > >for the life of me I can't figure out how to launch python?? > Well, what did you type in that command shell to get the line you report above? (Cut and Paste the TEXT from that command shell -- don't just transcribe by hand). That version string is only displayed when one starts Python in interactive mode. -=-=- Microsoft Windows [Version 10.0.19044.1706] (c) Microsoft Corporation. All rights reserved. C:\Users\Wulfraed>python Python ActivePython 3.8.2 (ActiveState Software Inc.) based on on win32 Type "help", "copyright", "credits" or "license" for more information. >>> -=-=- -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From pablogsal at gmail.com Tue May 31 09:31:00 2022 From: pablogsal at gmail.com (Pablo Galindo Salgado) Date: Tue, 31 May 2022 14:31:00 +0100 Subject: [RELEASE] The second Python 3.11 beta (3.11.0b2) is available Message-ID: Does anyone want bug fixes? Because we have 164 new commits fixing different things, from code to documentation. If you have reported some issue after 3.11.0b1, you should check if is fixed and if not, make sure you tell us so we can take a look. We still have two more betas to go so help us to make sure we don't miss anything so everything is ready for the final release!! https://www.python.org/downloads/release/python-3110b2/ ## This is a beta preview of Python 3.11 Python 3.11 is still in development. 3.11.0b2 is the second of four planned beta release previews. Beta release previews are intended to give the wider community the opportunity to test new features and bug fixes and to prepare their projects to support the new feature release. We **strongly encourage** maintainers of third-party Python projects to **test with 3.11** during the beta phase and report issues found to [the Python bug tracker](https://github.com/python/cpython/issues) as soon as possible. While the release is planned to be feature complete entering the beta phase, it is possible that features may be modified or, in rare cases, deleted up until the start of the release candidate phase (Monday, 2021-08-02). Our goal is to have no ABI changes after beta 4 and as few code changes as possible after 3.11.0rc1, the first release candidate. To achieve that, it will be **extremely important** to get as much exposure for 3.11 as possible during the beta phase. Please keep in mind that this is a preview release and its use is **not** recommended for production environments. # Major new features of the 3.11 series, compared to 3.10 Many new features for Python 3.11 are still being planned and written. Among the new major new features and changes so far: * [PEP 657](https://www.python.org/dev/peps/pep-0657/) -- Include Fine-Grained Error Locations in Tracebacks * [PEP 654](https://www.python.org/dev/peps/pep-0654/) -- Exception Groups and except* * [PEP 673](https://www.python.org/dev/peps/pep-0673/) -- Self Type * [PEP 646](https://www.python.org/dev/peps/pep-0646/)-- Variadic Generics * [PEP 680](https://www.python.org/dev/peps/pep-0680/)-- tomllib: Support for Parsing TOML in the Standard Library * [PEP 675](https://www.python.org/dev/peps/pep-0675/)-- Arbitrary Literal String Type * [PEP 655](https://www.python.org/dev/peps/pep-0655/)-- Marking individual TypedDict items as required or potentially-missing * [PEP 681](https://www.python.org/dev/peps/pep-0681/)-- Data Class Transforms * [bpo-46752](https://bugs.python.org/issue46752)-- Introduce task groups to asyncio * [bpo-433030](https://github.com/python/cpython/issues/34627/) -- Atomic grouping ((?>...)) and possessive quantifiers (`*+, ++, ?+, {m,n}+`) are now supported in regular expressions. * The [Faster Cpython Project](https://github.com/faster-cpython/) is already yielding some exciting results. Python 3.11 is up to 10-60% faster than Python 3.10. On average, we measured a 1.22x speedup on the standard benchmark suite. See [Faster CPython]( https://docs.python.org/3.11/whatsnew/3.11.html#faster-cpython) for details. * (Hey, **fellow core developer,** if a feature you find important is missing from this list, [let Pablo know](mailto:pablogsal at python.org ).) The next pre-release of Python 3.11 will be 3.11.0b3, currently scheduled for Thursday, 2022-06-16. # More resources * [Online Documentation](https://docs.python.org/3.11/) * [PEP 664](https://www.python.org/dev/peps/pep-0664/), 3.11 Release Schedule * Report bugs at [https://bugs.python.org](https://bugs.python.org). * [Help fund Python and its community](/psf/donations/). # And now for something completely different The Planck time is the time required for light to travel a distance of 1 Planck length in a vacuum, which is a time interval of approximately `5.39*10^(?44)` s. No current physical theory can describe timescales shorter than the Planck time, such as the earliest events after the Big Bang, and it is conjectured that the structure of time breaks down on intervals comparable to the Planck time. While there is currently no known way to measure time intervals on the scale of the Planck time, researchers in 2020 found that the accuracy of an atomic clock is constrained by quantum effects on the order of the Planck time, and for the most precise atomic clocks thus far they calculated that such effects have been ruled out to around `10^?33` s, or 10 orders of magnitude above the Planck scale. # We hope you enjoy the new releases! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. https://www.python.org/psf/ Regards from sunny London, Pablo Galindo Salgado From hasamuels1 at yahoo.com Tue May 31 08:00:29 2022 From: hasamuels1 at yahoo.com (Howard Samuels) Date: Tue, 31 May 2022 12:00:29 +0000 (UTC) Subject: Problems with Python In-Reply-To: <532840747.5099296.1653998098873@mail.yahoo.com> References: <532840747.5099296.1653998098873.ref@mail.yahoo.com> <532840747.5099296.1653998098873@mail.yahoo.com> Message-ID: <566149160.5100626.1653998429304@mail.yahoo.com> Good day I am new to programming.??I have signed up for a virtual online course and installed Python using Anaconda as well as jupyter notebook. I encountered problems & then went to YouTube tried going directly to the python website and used Pycharm. When I used pycharm and try to run the first basic program I am getting a message that it is unable to load the Virtual environment.? Also a message that ??the SDK seem to be false??? or some like wording.? I went to google which gave me stock overflow.? I am confused there as some of the responses I see there suppose that I am into programming and understand a lot of these terms. Some things I see, I understand but none of those have made a difference.? One such is to type "pip install virtenv" The system states that the requirement is already satisfied.?? In the process of trying to uninstall & reinstall and saw the invitation to write seeking for help.? I therefore cannot immediately reproduce the error message.Can you help? Howard Samuels?Are You Ready for Jesus to Come? On Tuesday, May 31, 2022, 07:54:58 AM GMT-4, Howard Samuels wrote: Good day I am new to programming.??I have signed up for a virtual online course and installed Python using Anaconda as well as jupyter notebook. I encountered problems & then went to YouTube tried going directly to the python website and used Pycharm. When I used pycharm and try to run the first basic program I am getting a message that it is unable to load the Virtual environment.? Also a message that ??the SDK seem to be false??? or some like wording.? I went to google which gave me stock overflow.? I am confused there as some of the responses I see there suppose that I am into programming and understand a lot of these terms. Some things I see, I understand but none of those have made a difference.? One such is to type "pip install virtenv" The system states that the requirement is already satisfied.?? In the process of trying to uninstall & reinstall and saw the invitation to write seeking for help.? I therefore cannot immediately reproduce the error message.Can you help? Howard Samuels? Are You Ready for Jesus to Come? From rshepard at appl-ecosys.com Tue May 31 14:47:06 2022 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Tue, 31 May 2022 11:47:06 -0700 (PDT) Subject: Tkinter: multicolumn table widget Message-ID: <6eaa6ad-c333-ec4a-c735-8da9f669f18@appl-ecosys.com> My web searches haven't helped me learn how to design a read-only scrollable table widget displaying rows retrieved from postgres database tables. This is for my business development application. I'm writing a view module that displays my contact history with a named person. The person's last and first name are passed to the database tables and all the results are displayed in a read-only view. The view has three vertical sections: person information on top, contact methods (phones, email) below that, and a scrollable, multicolumn table at the bottom. The table columns consist of three tk.StringVal() and one tk.Text() which usually has multiple rows related to that contact event. I want to learn how to create the table so each row has the contents of each of the four columns and I can scroll down and up to look at the history. All help is certainly appreciated. Rich From philb at philb.ca Tue May 31 14:51:48 2022 From: philb at philb.ca (Phil Boutros) Date: Tue, 31 May 2022 18:51:48 -0000 (UTC) Subject: Automatic Gain Control in Python? References: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> <015301d872f1$6bf9afc0$43ed0f40$@SGA.Ninja> <017501d8731a$43882750$ca9875f0$@SGA.Ninja> Message-ID: Richard Damon wrote: > From your description, your fundamental problem is you are trying to > automatically "control" things that weren't designed to be automatically > controlled in the way you are attempting. > What you seem to be missing is that you could get the podcasts from a > browser, and all a browser is is a program. It isn't that much work to > write a rudimentary browser in python, especially if you don't actually > need to display the results to a user, but are only trying to automate a > particular task. Even easier, the few NPR podcasts I just checked now have RSS feeds of their episodes (as expected). It seems it would be much easier to just download the latest episode based on the XML file, normalize, send it to play, done. Why is it so crucial that it be done "in real time", which it really isn't, unless you are listening to the live NPR feed. In the case of podcasts, *why* would you possible play them via streaming instead of downloading the publicly accessible file to manipulate however you chose? > SteveGS wrote: >> The code is to: Listen to the audio level for about 10 seconds or >> so and raise or lower the level in small increments. Based on what? Do you want a 10 second buffer to continually be analyzed in real time? From the output of a speaker, no less? And which level? Peak? Average over...10 seconds? Are you keeping track of the previous values to make it match? Where is that stored? >> It has nothing to do with understanding how to grab podcasts. The >> system is working very well for that. OK...then perhaps I/we am/are confused on the limitations of the program. What does it "grab", *exactly*? Phil -- AH#61 Wolf#14 BS#89 bus#1 CCB#1 SENS KOTC#4 philb at philb.ca http://philb.ca From PythonList at DancesWithMice.info Tue May 31 15:43:39 2022 From: PythonList at DancesWithMice.info (dn) Date: Wed, 1 Jun 2022 07:43:39 +1200 Subject: Problems with Python In-Reply-To: <566149160.5100626.1653998429304@mail.yahoo.com> References: <532840747.5099296.1653998098873.ref@mail.yahoo.com> <532840747.5099296.1653998098873@mail.yahoo.com> <566149160.5100626.1653998429304@mail.yahoo.com> Message-ID: <642d1f3d-bc57-0ba2-8f16-5f7b6b663196@DancesWithMice.info> On 01/06/2022 00.00, Howard Samuels via Python-list wrote: > Good day > I am new to programming.??I have signed up for a virtual online course and installed Python using Anaconda as well as jupyter notebook. I encountered problems & then went to YouTube tried going directly to the python website and used Pycharm. When I used pycharm and try to run the first basic program I am getting a message that it is unable to load the Virtual environment.? Also a message that ??the SDK seem to be false??? or some like wording.? I went to google which gave me stock overflow.? I am confused there as some of the responses I see there suppose that I am into programming and understand a lot of these terms. Some things I see, I understand but none of those have made a difference.? One such is to type "pip install virtenv" The system states that the requirement is already satisfied.?? > In the process of trying to uninstall & reinstall and saw the invitation to write seeking for help.? I therefore cannot immediately reproduce the error message.Can you help? So many 'moving parts' and unnecessarily-advanced features are described here. No wonder your head is spinning! Which course? How does *it* recommend setting-up the Python environment? Failing that, start PyCharm, open a new project, open a new Python file, type: a = 2 + 2 print( a ) and use the PyCharm command to "run". Build from there, one step at a time! -- Regards, =dn From python at mrabarnett.plus.com Tue May 31 16:15:04 2022 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 31 May 2022 21:15:04 +0100 Subject: Tkinter: multicolumn table widget In-Reply-To: <6eaa6ad-c333-ec4a-c735-8da9f669f18@appl-ecosys.com> References: <6eaa6ad-c333-ec4a-c735-8da9f669f18@appl-ecosys.com> Message-ID: <55a2cf6e-92b7-9953-0314-198860a0a832@mrabarnett.plus.com> On 2022-05-31 19:47, Rich Shepard wrote: > My web searches haven't helped me learn how to design a read-only scrollable > table widget displaying rows retrieved from postgres database tables. This > is for my business development application. > > I'm writing a view module that displays my contact history with a named > person. The person's last and first name are passed to the database tables > and all the results are displayed in a read-only view. > > The view has three vertical sections: person information on top, contact > methods (phones, email) below that, and a scrollable, multicolumn table > at the bottom. > > The table columns consist of three tk.StringVal() and one tk.Text() which > usually has multiple rows related to that contact event. > > I want to learn how to create the table so each row has the contents of each > of the four columns and I can scroll down and up to look at the history. > > All help is certainly appreciated. > Have a look at the tkinter.ttk.Treeview widget; it can be formatted as a tree hierarchy, its name suggests, or a multi-column tables, but it doesn't support multi-line text though, as far as I know. As with the tkinter.Text widget, you'll need to add the scrollbar(s) separately and then link them. From rshepard at appl-ecosys.com Tue May 31 16:29:23 2022 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Tue, 31 May 2022 13:29:23 -0700 (PDT) Subject: Tkinter: multicolumn table widget In-Reply-To: <55a2cf6e-92b7-9953-0314-198860a0a832@mrabarnett.plus.com> References: <6eaa6ad-c333-ec4a-c735-8da9f669f18@appl-ecosys.com> <55a2cf6e-92b7-9953-0314-198860a0a832@mrabarnett.plus.com> Message-ID: On Tue, 31 May 2022, MRAB wrote: > Have a look at the tkinter.ttk.Treeview widget; it can be formatted as a > tree hierarchy, its name suggests, or a multi-column tables, but it > doesn't support multi-line text though, as far as I know. MRAB, Thank you, I will. Each time I add a row to the contacts database table I include a note of what was discussed and what needs to be done. I'd like to be able to see the entire note with each contact event. I'm not committed to using a table so I'm totally open to other approaches. My needs are few: - The returned results are read-only. - The number of rows returned are variable. - Each row has a contact date, contact type, note, and next contact date. The last one isn't necessary to be displayed, but the first three are. - I want to be able to scroll and view all returned rows. > As with the tkinter.Text widget, you'll need to add the scrollbar(s) > separately and then link them. This I expected. Thanks, Rich From python at mrabarnett.plus.com Tue May 31 18:18:27 2022 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 31 May 2022 23:18:27 +0100 Subject: Tkinter: multicolumn table widget In-Reply-To: References: <6eaa6ad-c333-ec4a-c735-8da9f669f18@appl-ecosys.com> <55a2cf6e-92b7-9953-0314-198860a0a832@mrabarnett.plus.com> Message-ID: On 2022-05-31 21:29, Rich Shepard wrote: > On Tue, 31 May 2022, MRAB wrote: > >> Have a look at the tkinter.ttk.Treeview widget; it can be formatted as a >> tree hierarchy, its name suggests, or a multi-column tables, but it >> doesn't support multi-line text though, as far as I know. > > MRAB, > > Thank you, I will. > > Each time I add a row to the contacts database table I include a note of > what was discussed and what needs to be done. I'd like to be able to see the > entire note with each contact event. > > I'm not committed to using a table so I'm totally open to other approaches. > My needs are few: > - The returned results are read-only. > - The number of rows returned are variable. > - Each row has a contact date, contact type, note, and next contact date. > The last one isn't necessary to be displayed, but the first three are. > - I want to be able to scroll and view all returned rows. > The note could be displayed partially in the column itself, with the full text displayed either in read-only textbox nearby when the row is selected (and it's the only selected row), or in the form of a tooltip when you hover over it. There's an example of how to show a tooltip here: https://stackoverflow.com/questions/3221956/how-do-i-display-tooltips-in-tkinter >> As with the tkinter.Text widget, you'll need to add the scrollbar(s) >> separately and then link them. > > This I expected. > From rshepard at appl-ecosys.com Tue May 31 18:30:48 2022 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Tue, 31 May 2022 15:30:48 -0700 (PDT) Subject: Tkinter: multicolumn table widget In-Reply-To: References: <6eaa6ad-c333-ec4a-c735-8da9f669f18@appl-ecosys.com> <55a2cf6e-92b7-9953-0314-198860a0a832@mrabarnett.plus.com> Message-ID: <1454b4ce-a8c2-c67a-8fc-7f7ee3e7adfd@appl-ecosys.com> On Tue, 31 May 2022, MRAB wrote: > The note could be displayed partially in the column itself, with the full > text displayed either in read-only textbox nearby when the row is selected > (and it's the only selected row), or in the form of a tooltip when you > hover over it. > > There's an example of how to show a tooltip here: > https://stackoverflow.com/questions/3221956/how-do-i-display-tooltips-in-tkinter MRAB, Thank you very much. I've been reading about the TreeView and learning how to use it. Displaying the text column in a read-only textbox or tooltip would certainly do the job. Best regards, Rich From Gronicus at SGA.Ninja Tue May 31 21:03:38 2022 From: Gronicus at SGA.Ninja (Steve GS) Date: Tue, 31 May 2022 21:03:38 -0400 Subject: Automatic Gain Control in Python? In-Reply-To: References: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> <015301d872f1$6bf9afc0$43ed0f40$@SGA.Ninja> <017501d8731a$43882750$ca9875f0$@SGA.Ninja> Message-ID: <010101d87553$6daa6ad0$48ff4070$@SGA.Ninja> > Even easier, the few NPR podcasts I just checked now have RSS feeds of their episodes (as expected). It seems it would be much easier to just download the latest episode based on the XML file, normalize, send it to play, done. How can that possibly be easier? I am playing the podcast and recording it for a one-time replay. Now you want me to write a program that automatically downloads 48 files then manipulate them for equalization then replay it. It certainly doesn't sound easier to me. I already have that working using simple computer-generated vocal commands. > Why is it so crucial that it be done "in real time", which it really isn't, unless you are listening to the live NPR feed. In the case of podcasts, *why* would you possible play them via streaming instead of downloading the publicly accessible file to manipulate however you chose? It is played "real time" and yes, for the listeners in the community, it is real time no matter when it was recorded. >> The code is to: Listen to the audio level for about 10 seconds or so >> and raise or lower the level in small increments. > Based on what? A simple sample rate. > Do you want a 10 second buffer to continually be analyzed in real time? >From the output of a speaker, no less? Why should the source be of any concern? It is much more than a speaker. It is pulling in the podcasts/broadcasts and playing them. The audio is also being sent to a computer through an aux-out cable. Why is that confusing? > And which level? Peak? Average over...10 seconds? Are you keeping track of the previous values to make it match? That is what correlation is all about. >Where is that stored? In the computer that is running the AGC. >> It has nothing to do with understanding how to grab podcasts. The >> system is working very well for that. OK...then perhaps I/we am/are confused on the limitations of the program. What does it "grab", *exactly*? Maybe you do not understand smart speakers. That is exactly what they do. You tell them what podcast/broadcast to play, they get it and play it for you. It is that simple. All I want to do is change the audio levels automatically to make it easier on the ear. -- AH#61 Wolf#14 BS#89 bus#1 CCB#1 SENS KOTC#4 philb at philb.ca http://philb.ca -- https://mail.python.org/mailman/listinfo/python-list From rosuav at gmail.com Tue May 31 21:15:36 2022 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 1 Jun 2022 11:15:36 +1000 Subject: Automatic Gain Control in Python? In-Reply-To: <010101d87553$6daa6ad0$48ff4070$@SGA.Ninja> References: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> <015301d872f1$6bf9afc0$43ed0f40$@SGA.Ninja> <017501d8731a$43882750$ca9875f0$@SGA.Ninja> <010101d87553$6daa6ad0$48ff4070$@SGA.Ninja> Message-ID: On Wed, 1 Jun 2022 at 11:05, Steve GS wrote: > > > > Even easier, the few NPR podcasts I just checked now have RSS feeds of > their episodes (as expected). It seems it would be much easier to just > download the latest episode based on the XML file, normalize, send it to > play, done. > > How can that possibly be easier? I am playing the podcast and recording it > for a one-time replay. > Now you want me to write a program that automatically downloads 48 files > then manipulate them for equalization then replay it. It certainly doesn't > sound easier to me. I already have that working using simple > computer-generated vocal commands. > General principle: If you're asking someone else for help, don't tell them that your way is easier, because the obvious response is "go ahead then, do it your own way". You're technically right in a sense: something that you already have is, indeed, easier than something else. But downloading files is *easy* in Python, and audio analysis on files is FAR easier than real-time audio analysis with hysteresis avoidance. What you're doing actually reminds me of the old acoustic couplers [1], which were a messy hack brought about by monopolies that refused to allow other devices onto the network. Unless you have a really good reason for sticking to the black-box system, I would strongly recommend going for the much much easier method of simply downloading the files as they are. ChrisA [1] https://en.wikipedia.org/wiki/Acoustic_coupler From python at mrabarnett.plus.com Tue May 31 21:46:51 2022 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 1 Jun 2022 02:46:51 +0100 Subject: Automatic Gain Control in Python? In-Reply-To: <010101d87553$6daa6ad0$48ff4070$@SGA.Ninja> References: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> <015301d872f1$6bf9afc0$43ed0f40$@SGA.Ninja> <017501d8731a$43882750$ca9875f0$@SGA.Ninja> <010101d87553$6daa6ad0$48ff4070$@SGA.Ninja> Message-ID: On 2022-06-01 02:03, Steve GS wrote: [snip] > Maybe you do not understand smart speakers. That is exactly what they do. > You tell them what podcast/broadcast to play, they get it and play it for > you. It is that simple. > > All I want to do is change the audio levels automatically to make it easier > on the ear. > Did you have a look at the code I posted? I've found that pyaudio can listen to the Line In input at the same time as Audacity is recording from it, so you could might be able to use it to monitor the level and then tell the smart speaker to turn the volume up or down. From Gronicus at SGA.Ninja Tue May 31 22:01:56 2022 From: Gronicus at SGA.Ninja (Steve GS) Date: Tue, 31 May 2022 22:01:56 -0400 Subject: Automatic Gain Control in Python? In-Reply-To: References: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> <015301d872f1$6bf9afc0$43ed0f40$@SGA.Ninja> <017501d8731a$43882750$ca9875f0$@SGA.Ninja> <010101d87553$6daa6ad0$48ff4070$@SGA.Ninja> Message-ID: <011601d8755b$930f5940$b92e0bc0$@SGA.Ninja> I started but major issues prevented me this weekend (New Roof, Major limb broke for pine tree... I still have to install pyAudio. Genie: You have three wishes. Me: I wish I had more wishes. Genie: You cannot wish for more wishes. Me: I wish I could. -----Original Message----- From: Python-list On Behalf Of MRAB Sent: Tuesday, May 31, 2022 9:47 PM To: python-list at python.org Subject: Re: Automatic Gain Control in Python? On 2022-06-01 02:03, Steve GS wrote: [snip] > Maybe you do not understand smart speakers. That is exactly what they do. > You tell them what podcast/broadcast to play, they get it and play it > for you. It is that simple. > > All I want to do is change the audio levels automatically to make it > easier on the ear. > Did you have a look at the code I posted? I've found that pyaudio can listen to the Line In input at the same time as Audacity is recording from it, so you could might be able to use it to monitor the level and then tell the smart speaker to turn the volume up or down. -- https://mail.python.org/mailman/listinfo/python-list From python at mrabarnett.plus.com Tue May 31 22:34:48 2022 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 1 Jun 2022 03:34:48 +0100 Subject: Automatic Gain Control in Python? In-Reply-To: <011601d8755b$930f5940$b92e0bc0$@SGA.Ninja> References: <013801d872d9$f8d13700$ea73a500$@SGA.Ninja> <015301d872f1$6bf9afc0$43ed0f40$@SGA.Ninja> <017501d8731a$43882750$ca9875f0$@SGA.Ninja> <010101d87553$6daa6ad0$48ff4070$@SGA.Ninja> <011601d8755b$930f5940$b92e0bc0$@SGA.Ninja> Message-ID: <086f6e53-d1df-c92b-17a8-ada90a1444ae@mrabarnett.plus.com> On 2022-06-01 03:01, Steve GS wrote: > I started but major issues prevented me this weekend (New Roof, Major limb > broke for pine tree... > I still have to install pyAudio. PyPI has wheels only up to Python 3.6, but Christoph Gohlke's site has ones for more recent versions: https://www.lfd.uci.edu/~gohlke/pythonlibs/#pyaudio > -----Original Message----- > From: Python-list On > Behalf Of MRAB > Sent: Tuesday, May 31, 2022 9:47 PM > To: python-list at python.org > Subject: Re: Automatic Gain Control in Python? > > On 2022-06-01 02:03, Steve GS wrote: > [snip] > > > Maybe you do not understand smart speakers. That is exactly what they do. > > You tell them what podcast/broadcast to play, they get it and play it > > for you. It is that simple. > > > > All I want to do is change the audio levels automatically to make it > > easier on the ear. > > > Did you have a look at the code I posted? > > I've found that pyaudio can listen to the Line In input at the same time as > Audacity is recording from it, so you could might be able to use it to > monitor the level and then tell the smart speaker to turn the volume up or > down.