From morphex at gmail.com Tue Mar 1 06:53:02 2022 From: morphex at gmail.com (Morten W. Petersen) Date: Tue, 1 Mar 2022 12:53:02 +0100 Subject: Fwd: Timezone for datetime.date objects In-Reply-To: References: Message-ID: Forwarding to the list.. ---------- Forwarded message --------- From: Morten W. Petersen Date: Tue, Mar 1, 2022 at 12:52 PM Subject: Re: Timezone for datetime.date objects To: Chris Angelico On Mon, Feb 28, 2022 at 11:57 PM Chris Angelico wrote: > On Tue, 1 Mar 2022 at 09:28, Morten W. Petersen wrote: > > > > Hi Chris, Cameron. > > > > Well, let's say I specify the datetime 2022-02-22 02:02 (AM). I think > everyone could agree that it also means 2022-02-22 02:02:00:00, to > 2022-02-22 02:02:59:59. > > > > Not sure how many :59s you want there :) I'm going to assume you mean > "02:02:00" to "02:02:59". > > > And I think the same applies for a date. If the pipes are clogged and I > can't take (give) a shit, a shower or do anything else involving fluids, I > can just leave the keys under the doormat, and agree a date with the > plumber, and go off to a friend of relatives' place for a couple of days > while waiting for the plumber to do the necessary work. > > > > That is one of the fundamental differences between humans and > computers. Humans are very VERY sloppy with time descriptions. With > computers, it's much better to be clear about time ranges; a time does > not imply a specific window size. (And autistic humans are more like > computers.) > > Yep. Well, as I said, I could create some range objects myself, and even create a little module and put it up on pypi, if I couldn't find any existing module I could use. As we're discussing this, it is clear that different points can be made, and as for the Python standard library, it is what it is today, and part of the reason I was posting this email was to see if it should be changed, amended/appended to include range. I've worked a bit with dates, date searches and so on, and having existing range classes and objects to work with, defined in the standard library, seems like a natural and useful thing. Time, as humans have defined it with timezones, leap years, leap seconds, Denmark not following the atomic clock etc. is a bit messy, and it is easy to make mistakes creating custom code dealing with it. Regards, Morten -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ From rosuav at gmail.com Tue Mar 1 07:03:29 2022 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 1 Mar 2022 23:03:29 +1100 Subject: Timezone for datetime.date objects In-Reply-To: References: Message-ID: On Tue, 1 Mar 2022 at 22:52, Morten W. Petersen wrote: > Yep. Well, as I said, I could create some range objects myself, and even create a little module and put it up on pypi, if I couldn't find any existing module I could use. > > As we're discussing this, it is clear that different points can be made, and as for the Python standard library, it is what it is today, and part of the reason I was posting this email was to see if it should be changed, amended/appended to include range. > > I've worked a bit with dates, date searches and so on, and having existing range classes and objects to work with, defined in the standard library, seems like a natural and useful thing. Time, as humans have defined it with timezones, leap years, leap seconds, Denmark not following the atomic clock etc. is a bit messy, and it is easy to make mistakes creating custom code dealing with it. > I think a range object wouldn't be a bad thing, but it would absolutely have to be a datetime range, NOT a date range with timezone. For dates with timezones, the obvious interpretation to one person is equally obviously wrong to another, and vice versa. ChrisA From loris.bennett at fu-berlin.de Tue Mar 1 02:35:05 2022 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Tue, 01 Mar 2022 08:35:05 +0100 Subject: SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data References: <87o82r8ifp.fsf@hornfels.zedat.fu-berlin.de> Message-ID: <87v8wycehi.fsf@hornfels.zedat.fu-berlin.de> Cameron Simpson writes: > On 28Feb2022 10:11, Loris Bennett wrote: >>I have an SQLAlchemy class for an event: >> >> class UserEvent(Base): >> __tablename__ = "user_events" >> >> id = Column('id', Integer, primary_key=True) >> date = Column('date', Date, nullable=False) >> uid = Column('gid', String(64), ForeignKey('users.uid'), nullable=False) >> info = ?? >> >>The event may have arbitrary, but dict-like data associated with it, >>which I want to add in the field 'info'. This data never needs to be >>modified, once the event has been inserted into the DB. >> >>What type should the info field have? JSON, PickleType, String, or >>something else? > > I would use JSON, it expresses dicts well provided the dicts contain > only basic types (strings, numbers, other dicts/lists of basic types > recursively). > > I have personal problems with pickle because nonPython code can't read > it. > > Cheers, > Cameron Simpson Thanks for the various suggestions. The data I need to store is just a dict with maybe 3 or 4 keys and short string values probably of less than 32 characters each per event. The traffic on the DB is going to be very low, creating maybe a dozen events a day, mainly triggered via a command-line interface, although I will probably set up one or two cron jobs, each of which might generate another 0 to maybe 5 records a day. I could go for JSON (or rather LONGSTRING, as JSON is just an alias for LONGSTRING, but JSON is not available on the version of MariaDB I am using). However, that seems like overkill, since I am never going to have to store anything near 4 GB in the field. So I should probably in fact just use say VARCHAR(255). WDYT? Cheers, Loris From boblatest at yahoo.com Tue Mar 1 04:50:17 2022 From: boblatest at yahoo.com (Robert Latest) Date: 1 Mar 2022 09:50:17 GMT Subject: SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data References: <87o82r8ifp.fsf@hornfels.zedat.fu-berlin.de> <87v8wycehi.fsf@hornfels.zedat.fu-berlin.de> Message-ID: Loris Bennett wrote: > Thanks for the various suggestions. The data I need to store is just a > dict with maybe 3 or 4 keys and short string values probably of less > than 32 characters each per event. The traffic on the DB is going to be > very low, creating maybe a dozen events a day, mainly triggered via a > command-line interface, although I will probably set up one or two cron > jobs, each of which might generate another 0 to maybe 5 records a day. > > I could go for JSON (or rather LONGSTRING, as JSON is just an alias for > LONGSTRING, but JSON is not available on the version of MariaDB I am > using). However, that seems like overkill, since I am never going to > have to store anything near 4 GB in the field. So I should probably in > fact just use say VARCHAR(255). > > WDYT? Using TypeDecorator to transparently convert between a dict and its JSON string representation and MutableDict to track changes, you will get a completely transparent attribute that works just like a dict. Make sure to check that the generated JSON fits into your column width. I once got bitten by the fact that VARCHAR(x) can hold only x/4 characters in utf8mb4 character set. From loris.bennett at fu-berlin.de Tue Mar 1 05:15:39 2022 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Tue, 01 Mar 2022 11:15:39 +0100 Subject: SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data References: <87o82r8ifp.fsf@hornfels.zedat.fu-berlin.de> <87v8wycehi.fsf@hornfels.zedat.fu-berlin.de> Message-ID: <87fso27zck.fsf@hornfels.zedat.fu-berlin.de> Robert Latest writes: > Loris Bennett wrote: >> Thanks for the various suggestions. The data I need to store is just a >> dict with maybe 3 or 4 keys and short string values probably of less >> than 32 characters each per event. The traffic on the DB is going to be >> very low, creating maybe a dozen events a day, mainly triggered via a >> command-line interface, although I will probably set up one or two cron >> jobs, each of which might generate another 0 to maybe 5 records a day. >> >> I could go for JSON (or rather LONGSTRING, as JSON is just an alias for >> LONGSTRING, but JSON is not available on the version of MariaDB I am >> using). However, that seems like overkill, since I am never going to >> have to store anything near 4 GB in the field. So I should probably in >> fact just use say VARCHAR(255). >> >> WDYT? > > Using TypeDecorator to transparently convert between a dict and its JSON string > representation and MutableDict to track changes, you will get a completely > transparent attribute that works just like a dict. Make sure to check that the > generated JSON fits into your column width. I once got bitten by the fact that > VARCHAR(x) can hold only x/4 characters in utf8mb4 character set. Thanks for pointing out TypeDecorator - I wasn't aware of that. I won't need to track changes in the JSON data, because the events I am recording form an audit trail and so are written and read, but never modified. Cheers, Loris -- This signature is currently under construction. From wlfraed at ix.netcom.com Tue Mar 1 16:23:58 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Tue, 01 Mar 2022 16:23:58 -0500 Subject: How to solve the given problem? References: <1323924283.239248.1644533825969@mail.yahoo.com> <028b56df-5ff5-4fd7-9205-eea6bdc569d7n@googlegroups.com> <1d8c2edf-6d36-4af5-81d5-b60d74a951a3n@googlegroups.com> Message-ID: On Thu, 17 Feb 2022 02:20:53 -0800 (PST), NArshad declaimed the following: >I have completed the homework or what so ever it used to be its only I am telling the solution which looks to me as better as compared to what others have given like the one that Christian has given. At the risk of irritating the other regulars... If you've completed it, why not show us your Python code so we may review it? I presume you did do it in Python. I'm not going to show my code -- mainly as I just spent today writing a solution in REXX. But I will show the results of running said code. My solution is GENERAL in that it handles 1 variable number of periods in the plan 2 variable number of periods in the actual (partial day) 3 the case where the mis-feed is not the last item in actual 4 the case where the actual matches the plan (no mis-feed) 5 the case where the mis-feed is a shortage (underfed) 6 the case where the mis-feed is an overage (overfed) 7 the case where multiple mis-feeds cancel out (not shown below) -=-=- Your example schedule and feed amounts C:\Users\Wulfraed\Documents\_Hg-Repositories\REXX>rexx feedme.rex Enter planned feeding schedule (feed units for each time period, space separated) 150 100 30 30 30 20 20 10 5 5 Enter actual feeding schedule up to current feed period 150 60 PLAN : 10 time periods 400 total feed units ACTUAL : 2 time periods 210 total feed units dispensed MIS-FEED: 40 underfed SCHEDULE: 150 60 39 38 38 25 25 13 6 6 -=-=- Same, but the mis-feed was not discovered until after the 3rd period C:\Users\Wulfraed\Documents\_Hg-Repositories\REXX>rexx feedme.rex Enter planned feeding schedule (feed units for each time period, space separated) 150 100 30 30 30 20 20 10 5 5 Enter actual feeding schedule up to current feed period 150 60 30 PLAN : 10 time periods 400 total feed units ACTUAL : 3 time periods 240 total feed units dispensed MIS-FEED: 40 underfed SCHEDULE: 150 60 30 40 40 27 27 13 7 6 -=-=- An example were the mis-feed was in the other direction C:\Users\Wulfraed\Documents\_Hg-Repositories\REXX>rexx feedme.rex Enter planned feeding schedule (feed units for each time period, space separated) 150 100 30 30 30 20 20 10 5 5 Enter actual feeding schedule up to current feed period 150 120 PLAN : 10 time periods 400 total feed units ACTUAL : 2 time periods 270 total feed units dispensed MIS-FEED: 20 overfed SCHEDULE: 150 120 27 26 26 17 17 9 4 4 -=-=- Finally, an example where there was no mis-feed C:\Users\Wulfraed\Documents\_Hg-Repositories\REXX>rexx feedme.rex Enter planned feeding schedule (feed units for each time period, space separated) 150 100 30 30 30 20 20 10 5 5 Enter actual feeding schedule up to current feed period 150 100 PLAN : 10 time periods 400 total feed units ACTUAL : 2 time periods 250 total feed units dispensed MIS-FEED: 0 on schedule C:\Users\Wulfraed\Documents\_Hg-Repositories\REXX> -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From larry.martell at gmail.com Tue Mar 1 19:12:10 2022 From: larry.martell at gmail.com (Larry Martell) Date: Tue, 1 Mar 2022 19:12:10 -0500 Subject: All permutations from 2 lists Message-ID: If I have 2 lists, e.g.: os = ["Linux","Windows"] region = ["us-east-1", "us-east-2"] How can I get a list of tuples with all possible permutations? So for this example I'd want: [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows", "us-east-1"), "Windows", "us-east-2')] The lists can be different lengths or can be 0 length. Tried a few different things with itertools but have not got just what I need. TIA! From 2QdxY4RzWzUUiLuE at potatochowder.com Tue Mar 1 19:20:16 2022 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Tue, 1 Mar 2022 18:20:16 -0600 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: On 2022-03-01 at 19:12:10 -0500, Larry Martell wrote: > If I have 2 lists, e.g.: > > os = ["Linux","Windows"] > region = ["us-east-1", "us-east-2"] > > How can I get a list of tuples with all possible permutations? > > So for this example I'd want: > > [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows", > "us-east-1"), "Windows", "us-east-2')] > > The lists can be different lengths or can be 0 length. Tried a few > different things with itertools but have not got just what I need. [(o, r) for o in os for r in region] Feel free to import itertools, but it's not really necessary. ;-) From rob.cliffe at btinternet.com Tue Mar 1 19:32:46 2022 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Wed, 2 Mar 2022 00:32:46 +0000 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: I would not use `os` as an identifier, as it is the name of an important built-in module. I think itertools.product is what you need. Example program: import itertools opsys = ["Linux","Windows"] region = ["us-east-1", "us-east-2"] print(list(itertools.product(opsys, region))) Output: [('Linux', 'us-east-1'), ('Linux', 'us-east-2'), ('Windows', 'us-east-1'), ('Windows', 'us-east-2')] itertools.product returns an iterator (or iterable, I'm not sure of the correct technical term). If you only want to use the result once you can write e.g. ??? for ops, reg in itertools.product(opsys, region): ??? ??? etc. If you need it more than once, you can convert it to a list (or tuple), as above. Best wishes Rob Cliffe On 02/03/2022 00:12, Larry Martell wrote: > If I have 2 lists, e.g.: > > os = ["Linux","Windows"] > region = ["us-east-1", "us-east-2"] > > How can I get a list of tuples with all possible permutations? > > So for this example I'd want: > > [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows", > "us-east-1"), "Windows", "us-east-2')] > > The lists can be different lengths or can be 0 length. Tried a few > different things with itertools but have not got just what I need. > > TIA! From __peter__ at web.de Wed Mar 2 02:41:20 2022 From: __peter__ at web.de (Peter Otten) Date: Wed, 2 Mar 2022 08:41:20 +0100 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: On 02/03/2022 01:32, Rob Cliffe via Python-list wrote: > itertools.product returns an iterator (or iterable, I'm not sure of the > correct technical term). There's a simple test: iter(x) is x --> True # iterator iter(x) is x --> False # iterable So: >>> from itertools import product >>> p = product("ab", [1, 2]) >>> iter(p) is p # iterator True >>> items = [1, 2] # iterable >>> iter(items) is items False Another interesting property of (finite) iterators/iterables list(iterable) == list(iterable) --> Generally True, but not guaranteed. a = list(iterator) # whatever b = list(iterator) # [] (*) (*) Kill the coder if that doesn't hold ;) From rosuav at gmail.com Wed Mar 2 03:56:48 2022 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 2 Mar 2022 19:56:48 +1100 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: On Wed, 2 Mar 2022 at 19:34, Peter Otten <__peter__ at web.de> wrote: > > On 02/03/2022 01:32, Rob Cliffe via Python-list wrote: > > > itertools.product returns an iterator (or iterable, I'm not sure of the > > correct technical term). > > There's a simple test: > > iter(x) is x --> True # iterator > iter(x) is x --> False # iterable iter(x) --> TypeError # not iterable iter(x) is x --> True # iterator iter(x) is x --> False # iterable but not iterator All (non-broken) iterators are themselves iterable. By and large, itertools is full of classes which are their own iterators, so for instance itertools.product(...) is an iterator, not just an iterable. > So: > > >>> from itertools import product > >>> p = product("ab", [1, 2]) > >>> iter(p) is p # iterator > True > >>> items = [1, 2] # iterable > >>> iter(items) is items > False This is important, because: >>> items = [1,2,3,4,5,6] >>> i1 = iter(items); print(next(i1), next(i1)) 1 2 >>> i2 = iter(items); print(next(i2), next(i2), next(i2)) 1 2 3 >>> print(next(i2), next(i1)) 4 3 Every time you iterate over a list, you get the same items, but if you partially pump one of those iterators, it needs to remember where it was up to. In contrast, itertools.product is its own iterator, so you can't restart it. > Another interesting property of (finite) iterators/iterables > > list(iterable) == list(iterable) --> Generally True, but not guaranteed. There's been discussion now and then about giving a name to that property. I'm personally in favour of "reiterable", as in, "you can iterate over this more than once and get the same content". (Of course, mutating the list in between would mean you get different content, but it doesn't remember the position of the iterator.) > a = list(iterator) # whatever > b = list(iterator) # [] (*) > > (*) Kill the coder if that doesn't hold ;) I would call that a broken iterator. It used to be possible to have a generator function that could be broken in weird ways like that, but the worst of them now raise RuntimeError. Similarly, Python won't stop you from doing this, but it is absolutely horrid code, and something you really don't want to have to debug: >>> class BrokenIter: ... def __next__(self): ... return "spam" ... >>> class Broken: ... def __iter__(self): ... return BrokenIter() ... All it takes is forgetting the "def __iter__(self): return self" in the iterator, and you create something that LOOKS usable, but occasionally breaks. ChrisA From larry.martell at gmail.com Wed Mar 2 08:27:36 2022 From: larry.martell at gmail.com (Larry Martell) Date: Wed, 2 Mar 2022 08:27:36 -0500 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: On Tue, Mar 1, 2022 at 7:21 PM <2QdxY4RzWzUUiLuE at potatochowder.com> wrote: > > On 2022-03-01 at 19:12:10 -0500, > Larry Martell wrote: > > > If I have 2 lists, e.g.: > > > > os = ["Linux","Windows"] > > region = ["us-east-1", "us-east-2"] > > > > How can I get a list of tuples with all possible permutations? > > > > So for this example I'd want: > > > > [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows", > > "us-east-1"), "Windows", "us-east-2')] > > > > The lists can be different lengths or can be 0 length. Tried a few > > different things with itertools but have not got just what I need. > > [(o, r) for o in os for r in region] This does not work if region = []. I wrote in my question that either list could be empty. From larry.martell at gmail.com Wed Mar 2 08:29:50 2022 From: larry.martell at gmail.com (Larry Martell) Date: Wed, 2 Mar 2022 08:29:50 -0500 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: On Tue, Mar 1, 2022 at 7:32 PM Rob Cliffe wrote: > > I would not use `os` as an identifier, as it is the name of an important > built-in module. This is part of a much larger data structure, I created a simplified example. It is not actually called os. > I think itertools.product is what you need. > Example program: > > import itertools > opsys = ["Linux","Windows"] > region = ["us-east-1", "us-east-2"] > print(list(itertools.product(opsys, region))) This does not work if region = []. I wrote in question that either list could be empty. > Output: > > [('Linux', 'us-east-1'), ('Linux', 'us-east-2'), ('Windows', > 'us-east-1'), ('Windows', 'us-east-2')] > > itertools.product returns an iterator (or iterable, I'm not sure of the > correct technical term). > If you only want to use the result once you can write e.g. > > for ops, reg in itertools.product(opsys, region): > etc. > > If you need it more than once, you can convert it to a list (or tuple), > as above. > Best wishes > Rob Cliffe > > On 02/03/2022 00:12, Larry Martell wrote: > > If I have 2 lists, e.g.: > > > > os = ["Linux","Windows"] > > region = ["us-east-1", "us-east-2"] > > > > How can I get a list of tuples with all possible permutations? > > > > So for this example I'd want: > > > > [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows", > > "us-east-1"), "Windows", "us-east-2')] > > > > The lists can be different lengths or can be 0 length. Tried a few > > different things with itertools but have not got just what I need. > > > > TIA! > From antoon.pardon at vub.be Wed Mar 2 08:36:52 2022 From: antoon.pardon at vub.be (Antoon Pardon) Date: Wed, 2 Mar 2022 14:36:52 +0100 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: Op 2/03/2022 om 14:27 schreef Larry Martell: > On Tue, Mar 1, 2022 at 7:21 PM<2QdxY4RzWzUUiLuE at potatochowder.com> wrote: >> On 2022-03-01 at 19:12:10 -0500, >> Larry Martell wrote: >> >>> If I have 2 lists, e.g.: >>> >>> os = ["Linux","Windows"] >>> region = ["us-east-1", "us-east-2"] >>> >>> How can I get a list of tuples with all possible permutations? >>> >>> So for this example I'd want: >>> >>> [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows", >>> "us-east-1"), "Windows", "us-east-2')] >>> >>> The lists can be different lengths or can be 0 length. Tried a few >>> different things with itertools but have not got just what I need. >> [(o, r) for o in os for r in region] > This does not work if region = []. I wrote in my question that either > list could be empty. What do you mean it doesn't work? The result seems to be an empty list, which IMO is a perfectly valid result. All possible permutations over two collections where one collection is empty, should IMO give you an empty collection. -- Antoon Pardon. From larry.martell at gmail.com Wed Mar 2 08:44:41 2022 From: larry.martell at gmail.com (Larry Martell) Date: Wed, 2 Mar 2022 08:44:41 -0500 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: On Wed, Mar 2, 2022 at 8:37 AM Antoon Pardon wrote: > > > Op 2/03/2022 om 14:27 schreef Larry Martell: > > On Tue, Mar 1, 2022 at 7:21 PM<2QdxY4RzWzUUiLuE at potatochowder.com> wrote: > >> On 2022-03-01 at 19:12:10 -0500, > >> Larry Martell wrote: > >> > >>> If I have 2 lists, e.g.: > >>> > >>> os = ["Linux","Windows"] > >>> region = ["us-east-1", "us-east-2"] > >>> > >>> How can I get a list of tuples with all possible permutations? > >>> > >>> So for this example I'd want: > >>> > >>> [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows", > >>> "us-east-1"), "Windows", "us-east-2')] > >>> > >>> The lists can be different lengths or can be 0 length. Tried a few > >>> different things with itertools but have not got just what I need. > >> [(o, r) for o in os for r in region] > > This does not work if region = []. I wrote in my question that either > > list could be empty. > > What do you mean it doesn't work? The result seems to be an empty list, > which IMO is a perfectly valid result. > > All possible permutations over two collections where one collection is > empty, should IMO give you an empty collection. If one list is empty I want just the other list. What I am doing is building a list to pass to a mongodb query. If region is empty then I want to query for just the items in the os list. I guess I can test for the lists being empty, but I'd like a solution that handles that as down the road there could be more than just 2 lists. From joel.goldstick at gmail.com Wed Mar 2 08:53:33 2022 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Wed, 2 Mar 2022 08:53:33 -0500 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: On Wed, Mar 2, 2022 at 8:46 AM Larry Martell wrote: > > On Wed, Mar 2, 2022 at 8:37 AM Antoon Pardon wrote: > > > > > > Op 2/03/2022 om 14:27 schreef Larry Martell: > > > On Tue, Mar 1, 2022 at 7:21 PM<2QdxY4RzWzUUiLuE at potatochowder.com> wrote: > > >> On 2022-03-01 at 19:12:10 -0500, > > >> Larry Martell wrote: > > >> > > >>> If I have 2 lists, e.g.: > > >>> > > >>> os = ["Linux","Windows"] > > >>> region = ["us-east-1", "us-east-2"] > > >>> > > >>> How can I get a list of tuples with all possible permutations? > > >>> > > >>> So for this example I'd want: > > >>> > > >>> [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows", > > >>> "us-east-1"), "Windows", "us-east-2')] > > >>> > > >>> The lists can be different lengths or can be 0 length. Tried a few > > >>> different things with itertools but have not got just what I need. > > >> [(o, r) for o in os for r in region] > > > This does not work if region = []. I wrote in my question that either > > > list could be empty. > > > > What do you mean it doesn't work? The result seems to be an empty list, > > which IMO is a perfectly valid result. > > > > All possible permutations over two collections where one collection is > > empty, should IMO give you an empty collection. > > If one list is empty I want just the other list. What I am doing is > building a list to pass to a mongodb query. If region is empty then I > want to query for just the items in the os list. I guess I can test > for the lists being empty, but I'd like a solution that handles that > as down the road there could be more than just 2 lists. > -- > https://mail.python.org/mailman/listinfo/python-list Does this help you out: >>> [(o,r) for o in opsys for r in region or "x"] [('Linux', 'x'), ('Window', 'x')] -- Joel Goldstick From larry.martell at gmail.com Wed Mar 2 09:00:52 2022 From: larry.martell at gmail.com (Larry Martell) Date: Wed, 2 Mar 2022 09:00:52 -0500 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: On Wed, Mar 2, 2022 at 8:54 AM Joel Goldstick wrote: > > On Wed, Mar 2, 2022 at 8:46 AM Larry Martell wrote: > > > > On Wed, Mar 2, 2022 at 8:37 AM Antoon Pardon wrote: > > > > > > > > > Op 2/03/2022 om 14:27 schreef Larry Martell: > > > > On Tue, Mar 1, 2022 at 7:21 PM<2QdxY4RzWzUUiLuE at potatochowder.com> wrote: > > > >> On 2022-03-01 at 19:12:10 -0500, > > > >> Larry Martell wrote: > > > >> > > > >>> If I have 2 lists, e.g.: > > > >>> > > > >>> os = ["Linux","Windows"] > > > >>> region = ["us-east-1", "us-east-2"] > > > >>> > > > >>> How can I get a list of tuples with all possible permutations? > > > >>> > > > >>> So for this example I'd want: > > > >>> > > > >>> [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows", > > > >>> "us-east-1"), "Windows", "us-east-2')] > > > >>> > > > >>> The lists can be different lengths or can be 0 length. Tried a few > > > >>> different things with itertools but have not got just what I need. > > > >> [(o, r) for o in os for r in region] > > > > This does not work if region = []. I wrote in my question that either > > > > list could be empty. > > > > > > What do you mean it doesn't work? The result seems to be an empty list, > > > which IMO is a perfectly valid result. > > > > > > All possible permutations over two collections where one collection is > > > empty, should IMO give you an empty collection. > > > > If one list is empty I want just the other list. What I am doing is > > building a list to pass to a mongodb query. If region is empty then I > > want to query for just the items in the os list. I guess I can test > > for the lists being empty, but I'd like a solution that handles that > > as down the road there could be more than just 2 lists. > > -- > > https://mail.python.org/mailman/listinfo/python-list > > Does this help you out: > > >>> [(o,r) for o in opsys for r in region or "x"] > [('Linux', 'x'), ('Window', 'x')] That doesn't work if opsys = [] - either list could be empty. From joel.goldstick at gmail.com Wed Mar 2 09:05:28 2022 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Wed, 2 Mar 2022 09:05:28 -0500 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: On Wed, Mar 2, 2022 at 9:01 AM Larry Martell wrote: > > On Wed, Mar 2, 2022 at 8:54 AM Joel Goldstick wrote: > > > > On Wed, Mar 2, 2022 at 8:46 AM Larry Martell wrote: > > > > > > On Wed, Mar 2, 2022 at 8:37 AM Antoon Pardon wrote: > > > > > > > > > > > > Op 2/03/2022 om 14:27 schreef Larry Martell: > > > > > On Tue, Mar 1, 2022 at 7:21 PM<2QdxY4RzWzUUiLuE at potatochowder.com> wrote: > > > > >> On 2022-03-01 at 19:12:10 -0500, > > > > >> Larry Martell wrote: > > > > >> > > > > >>> If I have 2 lists, e.g.: > > > > >>> > > > > >>> os = ["Linux","Windows"] > > > > >>> region = ["us-east-1", "us-east-2"] > > > > >>> > > > > >>> How can I get a list of tuples with all possible permutations? > > > > >>> > > > > >>> So for this example I'd want: > > > > >>> > > > > >>> [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows", > > > > >>> "us-east-1"), "Windows", "us-east-2')] > > > > >>> > > > > >>> The lists can be different lengths or can be 0 length. Tried a few > > > > >>> different things with itertools but have not got just what I need. > > > > >> [(o, r) for o in os for r in region] > > > > > This does not work if region = []. I wrote in my question that either > > > > > list could be empty. > > > > > > > > What do you mean it doesn't work? The result seems to be an empty list, > > > > which IMO is a perfectly valid result. > > > > > > > > All possible permutations over two collections where one collection is > > > > empty, should IMO give you an empty collection. > > > > > > If one list is empty I want just the other list. What I am doing is > > > building a list to pass to a mongodb query. If region is empty then I > > > want to query for just the items in the os list. I guess I can test > > > for the lists being empty, but I'd like a solution that handles that > > > as down the road there could be more than just 2 lists. > > > -- > > > https://mail.python.org/mailman/listinfo/python-list > > > > Does this help you out: > > > > >>> [(o,r) for o in opsys for r in region or "x"] > > [('Linux', 'x'), ('Window', 'x')] > > That doesn't work if opsys = [] - either list could be empty. So, maybe what you need to do is determine the length of the longest list, and pad the other lists as required before producing the tuples? -- Joel Goldstick From antoon.pardon at vub.be Wed Mar 2 09:09:22 2022 From: antoon.pardon at vub.be (Antoon Pardon) Date: Wed, 2 Mar 2022 15:09:22 +0100 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: Op 2/03/2022 om 14:44 schreef Larry Martell: > On Wed, Mar 2, 2022 at 8:37 AM Antoon Pardon wrote: >> >> Op 2/03/2022 om 14:27 schreef Larry Martell: >>> On Tue, Mar 1, 2022 at 7:21 PM<2QdxY4RzWzUUiLuE at potatochowder.com> wrote: >>>> On 2022-03-01 at 19:12:10 -0500, >>>> Larry Martell wrote: >>>> >>>>> If I have 2 lists, e.g.: >>>>> >>>>> os = ["Linux","Windows"] >>>>> region = ["us-east-1", "us-east-2"] >>>>> >>>>> How can I get a list of tuples with all possible permutations? >>>>> >>>>> So for this example I'd want: >>>>> >>>>> [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows", >>>>> "us-east-1"), "Windows", "us-east-2')] >>>>> >>>>> The lists can be different lengths or can be 0 length. Tried a few >>>>> different things with itertools but have not got just what I need. >>>> [(o, r) for o in os for r in region] >>> This does not work if region = []. I wrote in my question that either >>> list could be empty. >> What do you mean it doesn't work? The result seems to be an empty list, >> which IMO is a perfectly valid result. >> >> All possible permutations over two collections where one collection is >> empty, should IMO give you an empty collection. > If one list is empty I want just the other list. What I am doing is > building a list to pass to a mongodb query. If region is empty then I > want to query for just the items in the os list. I guess I can test > for the lists being empty, but I'd like a solution that handles that > as down the road there could be more than just 2 lists. How about the following: Keep a list of your lists you want to permute over. Like the following: permutation_elements = [["Linux","Windows"],["us-east-1", "us-east-2"]] permutation = itertools.product(*permutation_elements) If you don't include the empty list, you will get more or less what you seem to want. Antoon Pardon. From larry.martell at gmail.com Wed Mar 2 09:29:10 2022 From: larry.martell at gmail.com (Larry Martell) Date: Wed, 2 Mar 2022 09:29:10 -0500 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: On Wed, Mar 2, 2022 at 9:10 AM Antoon Pardon wrote: > > Op 2/03/2022 om 14:44 schreef Larry Martell: > > On Wed, Mar 2, 2022 at 8:37 AM Antoon Pardon wrote: > >> > >> Op 2/03/2022 om 14:27 schreef Larry Martell: > >>> On Tue, Mar 1, 2022 at 7:21 PM<2QdxY4RzWzUUiLuE at potatochowder.com> wrote: > >>>> On 2022-03-01 at 19:12:10 -0500, > >>>> Larry Martell wrote: > >>>> > >>>>> If I have 2 lists, e.g.: > >>>>> > >>>>> os = ["Linux","Windows"] > >>>>> region = ["us-east-1", "us-east-2"] > >>>>> > >>>>> How can I get a list of tuples with all possible permutations? > >>>>> > >>>>> So for this example I'd want: > >>>>> > >>>>> [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows", > >>>>> "us-east-1"), "Windows", "us-east-2')] > >>>>> > >>>>> The lists can be different lengths or can be 0 length. Tried a few > >>>>> different things with itertools but have not got just what I need. > >>>> [(o, r) for o in os for r in region] > >>> This does not work if region = []. I wrote in my question that either > >>> list could be empty. > >> What do you mean it doesn't work? The result seems to be an empty list, > >> which IMO is a perfectly valid result. > >> > >> All possible permutations over two collections where one collection is > >> empty, should IMO give you an empty collection. > > If one list is empty I want just the other list. What I am doing is > > building a list to pass to a mongodb query. If region is empty then I > > want to query for just the items in the os list. I guess I can test > > for the lists being empty, but I'd like a solution that handles that > > as down the road there could be more than just 2 lists. > > How about the following: Keep a list of your lists you want to permute over. > Like the following: > > permutation_elements = [["Linux","Windows"],["us-east-1", "us-east-2"]] > > permutation = itertools.product(*permutation_elements) > > If you don't include the empty list, you will get more or less what you > seem to want. But I need to deal with that case. From antoon.pardon at vub.be Wed Mar 2 09:36:05 2022 From: antoon.pardon at vub.be (Antoon Pardon) Date: Wed, 2 Mar 2022 15:36:05 +0100 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: <033080a3-9327-5eb5-4b63-2261db5939d7@vub.be> Op 2/03/2022 om 15:29 schreef Larry Martell: > On Wed, Mar 2, 2022 at 9:10 AM Antoon Pardon wrote: >> Op 2/03/2022 om 14:44 schreef Larry Martell: >>> On Wed, Mar 2, 2022 at 8:37 AM Antoon Pardon wrote: >>>> Op 2/03/2022 om 14:27 schreef Larry Martell: >>>>> On Tue, Mar 1, 2022 at 7:21 PM<2QdxY4RzWzUUiLuE at potatochowder.com> wrote: >>>>>> On 2022-03-01 at 19:12:10 -0500, >>>>>> Larry Martell wrote: >>>>>> >>>>>>> If I have 2 lists, e.g.: >>>>>>> >>>>>>> os = ["Linux","Windows"] >>>>>>> region = ["us-east-1", "us-east-2"] >>>>>>> >>>>>>> How can I get a list of tuples with all possible permutations? >>>>>>> >>>>>>> So for this example I'd want: >>>>>>> >>>>>>> [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows", >>>>>>> "us-east-1"), "Windows", "us-east-2')] >>>>>>> >>>>>>> The lists can be different lengths or can be 0 length. Tried a few >>>>>>> different things with itertools but have not got just what I need. >>>>>> [(o, r) for o in os for r in region] >>>>> This does not work if region = []. I wrote in my question that either >>>>> list could be empty. >>>> What do you mean it doesn't work? The result seems to be an empty list, >>>> which IMO is a perfectly valid result. >>>> >>>> All possible permutations over two collections where one collection is >>>> empty, should IMO give you an empty collection. >>> If one list is empty I want just the other list. What I am doing is >>> building a list to pass to a mongodb query. If region is empty then I >>> want to query for just the items in the os list. I guess I can test >>> for the lists being empty, but I'd like a solution that handles that >>> as down the road there could be more than just 2 lists. >> How about the following: Keep a list of your lists you want to permute over. >> Like the following: >> >> permutation_elements = [["Linux","Windows"],["us-east-1", "us-east-2"]] >> >> permutation = itertools.product(*permutation_elements) >> >> If you don't include the empty list, you will get more or less what you >> seem to want. > But I need to deal with that case. What does that mean? How does using the above method to produce the permutations you want, prevent you from dealing with an empty list however you want when you encounter them? Just don't add them to the permutation_elements. Antoon Pardon. From larry.martell at gmail.com Wed Mar 2 09:58:19 2022 From: larry.martell at gmail.com (Larry Martell) Date: Wed, 2 Mar 2022 09:58:19 -0500 Subject: All permutations from 2 lists In-Reply-To: <033080a3-9327-5eb5-4b63-2261db5939d7@vub.be> References: <033080a3-9327-5eb5-4b63-2261db5939d7@vub.be> Message-ID: On Wed, Mar 2, 2022 at 9:37 AM Antoon Pardon wrote: > > > > Op 2/03/2022 om 15:29 schreef Larry Martell: > > On Wed, Mar 2, 2022 at 9:10 AM Antoon Pardon wrote: > >> Op 2/03/2022 om 14:44 schreef Larry Martell: > >>> On Wed, Mar 2, 2022 at 8:37 AM Antoon Pardon wrote: > >>>> Op 2/03/2022 om 14:27 schreef Larry Martell: > >>>>> On Tue, Mar 1, 2022 at 7:21 PM<2QdxY4RzWzUUiLuE at potatochowder.com> wrote: > >>>>>> On 2022-03-01 at 19:12:10 -0500, > >>>>>> Larry Martell wrote: > >>>>>> > >>>>>>> If I have 2 lists, e.g.: > >>>>>>> > >>>>>>> os = ["Linux","Windows"] > >>>>>>> region = ["us-east-1", "us-east-2"] > >>>>>>> > >>>>>>> How can I get a list of tuples with all possible permutations? > >>>>>>> > >>>>>>> So for this example I'd want: > >>>>>>> > >>>>>>> [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows", > >>>>>>> "us-east-1"), "Windows", "us-east-2')] > >>>>>>> > >>>>>>> The lists can be different lengths or can be 0 length. Tried a few > >>>>>>> different things with itertools but have not got just what I need. > >>>>>> [(o, r) for o in os for r in region] > >>>>> This does not work if region = []. I wrote in my question that either > >>>>> list could be empty. > >>>> What do you mean it doesn't work? The result seems to be an empty list, > >>>> which IMO is a perfectly valid result. > >>>> > >>>> All possible permutations over two collections where one collection is > >>>> empty, should IMO give you an empty collection. > >>> If one list is empty I want just the other list. What I am doing is > >>> building a list to pass to a mongodb query. If region is empty then I > >>> want to query for just the items in the os list. I guess I can test > >>> for the lists being empty, but I'd like a solution that handles that > >>> as down the road there could be more than just 2 lists. > >> How about the following: Keep a list of your lists you want to permute over. > >> Like the following: > >> > >> permutation_elements = [["Linux","Windows"],["us-east-1", "us-east-2"]] > >> > >> permutation = itertools.product(*permutation_elements) > >> > >> If you don't include the empty list, you will get more or less what you > >> seem to want. > > But I need to deal with that case. > > What does that mean? How does using the above method to produce the permutations > you want, prevent you from dealing with an empty list however you want when you > encounter them? Just don't add them to the permutation_elements. I need to know what items are in which position. If sometimes the regions are in one index and sometimes in another will not work for me. From antoon.pardon at vub.be Wed Mar 2 10:25:42 2022 From: antoon.pardon at vub.be (Antoon Pardon) Date: Wed, 2 Mar 2022 16:25:42 +0100 Subject: All permutations from 2 lists In-Reply-To: References: <033080a3-9327-5eb5-4b63-2261db5939d7@vub.be> Message-ID: Op 2/03/2022 om 15:58 schreef Larry Martell: > On Wed, Mar 2, 2022 at 9:37 AM Antoon Pardon wrote: >> >>>>> If one list is empty I want just the other list. What I am doing is >>>>> building a list to pass to a mongodb query. If region is empty then I >>>>> want to query for just the items in the os list. I guess I can test >>>>> for the lists being empty, but I'd like a solution that handles that >>>>> as down the road there could be more than just 2 lists. >>>> How about the following: Keep a list of your lists you want to permute over. >>>> Like the following: >>>> >>>> permutation_elements = [["Linux","Windows"],["us-east-1", "us-east-2"]] >>>> >>>> permutation = itertools.product(*permutation_elements) >>>> >>>> If you don't include the empty list, you will get more or less what you >>>> seem to want. >>> But I need to deal with that case. >> What does that mean? How does using the above method to produce the permutations >> you want, prevent you from dealing with an empty list however you want when you >> encounter them? Just don't add them to the permutation_elements. > I need to know what items are in which position. If sometimes the > regions are in one index and sometimes in another will not work for > me. I am starting to suspect you didn't think this through. What you are telling here contradicts what you told earlier that if either list was empty, you just wanted the other list. Because then you wouldn't know what items were in that list. The only solution I can see now is that if a list is empty, you either add [None] or [""] to the permutation_elements (whatever suits you better) and then use itertools.product From robin at reportlab.com Wed Mar 2 10:32:34 2022 From: robin at reportlab.com (Robin Becker) Date: Wed, 2 Mar 2022 15:32:34 +0000 Subject: lxml empty versus self closed tag Message-ID: <23c68bd2-2d86-8be6-e1e3-e6c6acc74db3@everest.reportlab.co.uk> I'm using lxml.etree.XMLParser and would like to distinguish from I seem to have e.getchildren()==[] and e.text==None for both cases. Is there a way to get the first to have e.text=='' -- Robin Becker From larry.martell at gmail.com Wed Mar 2 10:49:11 2022 From: larry.martell at gmail.com (Larry Martell) Date: Wed, 2 Mar 2022 10:49:11 -0500 Subject: All permutations from 2 lists In-Reply-To: References: <033080a3-9327-5eb5-4b63-2261db5939d7@vub.be> Message-ID: On Wed, Mar 2, 2022 at 10:26 AM Antoon Pardon wrote: > > > > Op 2/03/2022 om 15:58 schreef Larry Martell: > > On Wed, Mar 2, 2022 at 9:37 AM Antoon Pardon wrote: > >> > >>>>> If one list is empty I want just the other list. What I am doing is > >>>>> building a list to pass to a mongodb query. If region is empty then I > >>>>> want to query for just the items in the os list. I guess I can test > >>>>> for the lists being empty, but I'd like a solution that handles that > >>>>> as down the road there could be more than just 2 lists. > >>>> How about the following: Keep a list of your lists you want to permute over. > >>>> Like the following: > >>>> > >>>> permutation_elements = [["Linux","Windows"],["us-east-1", "us-east-2"]] > >>>> > >>>> permutation = itertools.product(*permutation_elements) > >>>> > >>>> If you don't include the empty list, you will get more or less what you > >>>> seem to want. > >>> But I need to deal with that case. > >> What does that mean? How does using the above method to produce the permutations > >> you want, prevent you from dealing with an empty list however you want when you > >> encounter them? Just don't add them to the permutation_elements. > > I need to know what items are in which position. If sometimes the > > regions are in one index and sometimes in another will not work for > > me. > > I am starting to suspect you didn't think this through. What you are telling here > contradicts what you told earlier that if either list was empty, you just wanted > the other list. Because then you wouldn't know what items were in that list. > > The only solution I can see now is that if a list is empty, you either add [None] or > [""] to the permutation_elements (whatever suits you better) and then use > itertools.product I found a way to pass this directly into the query: def query_lfixer(query): for k, v in query.items(): if type(v)==list: query[k] = {"$in": v} return query self._db_conn[collection_name].find(query_lfixer(query)) From hjp-python at hjp.at Wed Mar 2 12:17:54 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Wed, 2 Mar 2022 18:17:54 +0100 Subject: Timezone for datetime.date objects In-Reply-To: References: Message-ID: <20220302171754.a2w3v4b2uyryj4sz@hjp.at> On 2022-02-28 23:28:23 +0100, Morten W. Petersen wrote: > Well, let's say I specify the datetime 2022-02-22 02:02 (AM). I think > everyone could agree that it also means 2022-02-22 02:02:00:00, to > 2022-02-22 02:02:59:59. I disagree. The datetime 2022-02-22 02:02 specifies a point in time, not a time period. It means 2022-02-22 02:02:00.000000000..... In reality, a point in time may be fuzzy. "The train departs at 02:02" definitely doesn't mean that the train will depart exactly at 02:02:00, but it also doesn't mean that it will depart between 02:02 and 02:03. Rather it's a smooth probability distribution starting a bit before 02:02:00 (a train should never leave early, but sometimes clocks are wrong or somebody doesn't pay attention) a peak shortly after 02:02:00 and a very long tail. > And I think the same applies for a date. Depends on the context, but without s specific context (like business days) I would agree. A day *is* a time period with a beginning and an end. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From om+python at omajoshi.com Tue Mar 1 19:57:02 2022 From: om+python at omajoshi.com (Om Joshi) Date: Tue, 01 Mar 2022 18:57:02 -0600 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: <17f482166bc.b390544c2901434.8269986412220123355@omajoshi.com> For completeness, the itertools solution (which returns an iterator) is >>> os = ["Linux","Windows"] >>> region = ["us-east-1", "us-east-2"] >>> import itertools >>> itertools.product(os,region) >>> list(itertools.product(os,region)) [('Linux', 'us-east-1'), ('Linux', 'us-east-2'), ('Windows', 'us-east-1'), ('Windows', 'us-east-2')] There are probably use cases where you want the iterator and others where you want the list comprehension. I think the itertools looks nice and it's easier to generalize it to N=3,4,5,... lists than writing out `for a in list1 for b in list2 for c in list3` etc -- also, you can do things like itertools.product(*list_of_lists) to get the product if you have a variable number of lists. Not exactly sure about speed comparison but my instinct is that a double/triple list comprehension is going to be slower than letting itertools do its magic. ---- On Tue, 01 Mar 2022 18:20:16 -0600 <2QdxY4RzWzUUiLuE at potatochowder.com> wrote ---- > On 2022-03-01 at 19:12:10 -0500, > Larry Martell wrote: > > > If I have 2 lists, e.g.: > > > > os = ["Linux","Windows"] > > region = ["us-east-1", "us-east-2"] > > > > How can I get a list of tuples with all possible permutations? > > > > So for this example I'd want: > > > > [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows", > > "us-east-1"), "Windows", "us-east-2')] > > > > The lists can be different lengths or can be 0 length. Tried a few > > different things with itertools but have not got just what I need. > > [(o, r) for o in os for r in region] > > Feel free to import itertools, but it's not really necessary. ;-) > -- > https://mail.python.org/mailman/listinfo/python-list > From loris.bennett at fu-berlin.de Wed Mar 2 02:25:48 2022 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Wed, 02 Mar 2022 08:25:48 +0100 Subject: SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data References: <87o82r8ifp.fsf@hornfels.zedat.fu-berlin.de> <87v8wycehi.fsf@hornfels.zedat.fu-berlin.de> Message-ID: <875yowrf2b.fsf@hornfels.zedat.fu-berlin.de> Dennis Lee Bieber writes: > On Tue, 01 Mar 2022 08:35:05 +0100, Loris Bennett > declaimed the following: > >>Thanks for the various suggestions. The data I need to store is just a >>dict with maybe 3 or 4 keys and short string values probably of less >>than 32 characters each per event. The traffic on the DB is going to be >>very low, creating maybe a dozen events a day, mainly triggered via a >>command-line interface, although I will probably set up one or two cron >>jobs, each of which might generate another 0 to maybe 5 records a day. >> >>I could go for JSON (or rather LONGSTRING, as JSON is just an alias for >>LONGSTRING, but JSON is not available on the version of MariaDB I am >>using). However, that seems like overkill, since I am never going to >>have to store anything near 4 GB in the field. So I should probably in >>fact just use say VARCHAR(255). >> >>WDYT? >> > > Having taken a few on-line short courses on database normalization and > SQL during my first lay-off, my view would be to normalize everything > first... Which, in your description, means putting that dictionary into a > separate table of the form (I also tend to define an autoincrement primary > key for all tables): > > DICTDATA(*ID*, _eventID_, dictKey, dictValue) > > where * delimits primary key, _ delimits foreign key to parent (event?) > record. Ah, yes, you are right. That would indeed be the correct way to do it. I'll look into that. Up to now I was thinking I would only ever want to read out the dict in its entirety, but that's probably not correct. > Caveat: While I have a book on SQLAlchemy, I confess it makes no sense to > me -- I can code SQL joins faster than figuring out how to represent the > same join in SQLAlchemy. I currently can't code SQL joins fast anyway, so although doing it in SQLAlchemy is might be relatively slower, absolutely there's maybe not going to be much difference :-) Cheers, Loris -- This signature is currently under construction. From om+python at omajoshi.com Wed Mar 2 13:00:37 2022 From: om+python at omajoshi.com (Om Joshi) Date: Wed, 02 Mar 2022 12:00:37 -0600 Subject: All permutations from 2 lists In-Reply-To: <17f482166bc.b390544c2901434.8269986412220123355@omajoshi.com> References: <17f482166bc.b390544c2901434.8269986412220123355@omajoshi.com> Message-ID: <17f4bca84fd.127924298239816.5126658683781106724@omajoshi.com> I sent this 17hrs ago but I guess it just went through. Apologies for the redundant comments... ---- On Tue, 01 Mar 2022 18:57:02 -0600 om+python at omajoshi.com wrote ----For completeness, the itertools solution (which returns an iterator) is >>> os = ["Linux","Windows"] >>> region = ["us-east-1", "us-east-2"] >>> import itertools >>> itertools.product(os,region) >>> list(itertools.product(os,region)) [('Linux', 'us-east-1'), ('Linux', 'us-east-2'), ('Windows', 'us-east-1'), ('Windows', 'us-east-2')] There are probably use cases where you want the iterator and others where you want the list comprehension. I think the itertools looks nice and it's easier to generalize it to N=3,4,5,... lists than writing out `for a in list1 for b in list2 for c in list3` etc -- also, you can do things like itertools.product(*list_of_lists) to get the product if you have a variable number of lists. Not exactly sure about speed comparison but my instinct is that a double/triple list comprehension is going to be slower than letting itertools do its magic. ---- On Tue, 01 Mar 2022 18:20:16 -0600 <2QdxY4RzWzUUiLuE at potatochowder.com> wrote ---- > On 2022-03-01 at 19:12:10 -0500, > Larry Martell wrote: > > > If I have 2 lists, e.g.: > > > > os = ["Linux","Windows"] > > region = ["us-east-1", "us-east-2"] > > > > How can I get a list of tuples with all possible permutations? > > > > So for this example I'd want: > > > > [("Linux", "us-east-1"), ("Linux", "us-east-2"), ("Windows", > > "us-east-1"), "Windows", "us-east-2')] > > > > The lists can be different lengths or can be 0 length. Tried a few > > different things with itertools but have not got just what I need. > > [(o, r) for o in os for r in region] > > Feel free to import itertools, but it's not really necessary. ;-) > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list From om+python at omajoshi.com Wed Mar 2 13:25:24 2022 From: om+python at omajoshi.com (Om Joshi) Date: Wed, 02 Mar 2022 12:25:24 -0600 Subject: All permutations from 2 lists In-Reply-To: References: <033080a3-9327-5eb5-4b63-2261db5939d7@vub.be> Message-ID: <17f4be13516.b061fd75241559.2049818849539338487@omajoshi.com> Something like this?itertools.product(x or ("",) for x in perm_elems)Out of curiousity, how might one adapt this if x is not a list but an iterator, without doing `itertools.product(list(x) or ("",) for x in perm_elems)`?? ---- On Wed, 02 Mar 2022 09:25:42 -0600 antoon.pardon at vub.be wrote ---- Op 2/03/2022 om 15:58 schreef Larry Martell: > On Wed, Mar 2, 2022 at 9:37 AM Antoon Pardon wrote: >> >>>>> If one list is empty I want just the other list. What I am doing is >>>>> building a list to pass to a mongodb query. If region is empty then I >>>>> want to query for just the items in the os list. I guess I can test >>>>> for the lists being empty, but I'd like a solution that handles that >>>>> as down the road there could be more than just 2 lists. >>>> How about the following: Keep a list of your lists you want to permute over. >>>> Like the following: >>>> >>>> permutation_elements = [["Linux","Windows"],["us-east-1", "us-east-2"]] >>>> >>>> permutation = itertools.product(*permutation_elements) >>>> >>>> If you don't include the empty list, you will get more or less what you >>>> seem to want. >>> But I need to deal with that case. >> What does that mean? How does using the above method to produce the permutations >> you want, prevent you from dealing with an empty list however you want when you >> encounter them? Just don't add them to the permutation_elements. > I need to know what items are in which position. If sometimes the > regions are in one index and sometimes in another will not work for > me. I am starting to suspect you didn't think this through. What you are telling here contradicts what you told earlier that if either list was empty, you just wanted the other list. Because then you wouldn't know what items were in that list. The only solution I can see now is that if a list is empty, you either add [None] or [""] to the permutation_elements (whatever suits you better) and then use itertools.product -- https://mail.python.org/mailman/listinfo/python-list From dieter at handshake.de Wed Mar 2 13:39:24 2022 From: dieter at handshake.de (Dieter Maurer) Date: Wed, 2 Mar 2022 19:39:24 +0100 Subject: lxml empty versus self closed tag In-Reply-To: <23c68bd2-2d86-8be6-e1e3-e6c6acc74db3@everest.reportlab.co.uk> References: <23c68bd2-2d86-8be6-e1e3-e6c6acc74db3@everest.reportlab.co.uk> Message-ID: <25119.47580.635071.641702@ixdm.fritz.box> Robin Becker wrote at 2022-3-2 15:32 +0000: >I'm using lxml.etree.XMLParser and would like to distinguish > > > >from > > > >I seem to have e.getchildren()==[] and e.text==None for both cases. Is there a way to get the first to have e.text=='' I do not think so (at least not without a DTD): `' is just a shorthand notation for '' and the difference has no influence on the DOM. Note that `lxml` is just a Python binding for `libxml2`. All the parsing is done by this library. From cs at cskk.id.au Wed Mar 2 16:32:32 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Thu, 3 Mar 2022 08:32:32 +1100 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: On 02Mar2022 08:29, Larry Martell wrote: >On Tue, Mar 1, 2022 at 7:32 PM Rob Cliffe wrote: >> I think itertools.product is what you need. >> Example program: >> >> import itertools >> opsys = ["Linux","Windows"] >> region = ["us-east-1", "us-east-2"] >> print(list(itertools.product(opsys, region))) > >This does not work if region = []. I wrote in question that either >list could be empty. What do you want to get if a list is empty? You haven't said. My personal expectation would be an empty result. Alternatively, if you expect an empty list to imply some single default the the experession: the_list or (the_default,) might be of use. Cheers, Cameron Simpson From larry.martell at gmail.com Wed Mar 2 17:05:28 2022 From: larry.martell at gmail.com (Larry Martell) Date: Wed, 2 Mar 2022 17:05:28 -0500 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: On Wed, Mar 2, 2022 at 5:00 PM Cameron Simpson wrote: > > On 02Mar2022 08:29, Larry Martell wrote: > >On Tue, Mar 1, 2022 at 7:32 PM Rob Cliffe wrote: > >> I think itertools.product is what you need. > >> Example program: > >> > >> import itertools > >> opsys = ["Linux","Windows"] > >> region = ["us-east-1", "us-east-2"] > >> print(list(itertools.product(opsys, region))) > > > >This does not work if region = []. I wrote in question that either > >list could be empty. > > What do you want to get if a list is empty? You haven't said. My > personal expectation would be an empty result. > > Alternatively, if you expect an empty list to imply some single default > the the experession: > > the_list or (the_default,) > > might be of use. I've solved the issue. From joel.goldstick at gmail.com Wed Mar 2 17:30:27 2022 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Wed, 2 Mar 2022 17:30:27 -0500 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: On Wed, Mar 2, 2022 at 5:07 PM Larry Martell wrote: > > On Wed, Mar 2, 2022 at 5:00 PM Cameron Simpson wrote: > > > > On 02Mar2022 08:29, Larry Martell wrote: > > >On Tue, Mar 1, 2022 at 7:32 PM Rob Cliffe wrote: > > >> I think itertools.product is what you need. > > >> Example program: > > >> > > >> import itertools > > >> opsys = ["Linux","Windows"] > > >> region = ["us-east-1", "us-east-2"] > > >> print(list(itertools.product(opsys, region))) > > > > > >This does not work if region = []. I wrote in question that either > > >list could be empty. > > > > What do you want to get if a list is empty? You haven't said. My > > personal expectation would be an empty result. > > > > Alternatively, if you expect an empty list to imply some single default > > the the experession: > > > > the_list or (the_default,) > > > > might be of use. > > I've solved the issue. > -- > https://mail.python.org/mailman/listinfo/python-list Would you be so kind as to show the results of your solution? -- Joel Goldstick From larry.martell at gmail.com Wed Mar 2 17:46:49 2022 From: larry.martell at gmail.com (Larry Martell) Date: Wed, 2 Mar 2022 17:46:49 -0500 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: On Wed, Mar 2, 2022 at 5:31 PM Joel Goldstick wrote: > > On Wed, Mar 2, 2022 at 5:07 PM Larry Martell wrote: > > > > On Wed, Mar 2, 2022 at 5:00 PM Cameron Simpson wrote: > > > > > > On 02Mar2022 08:29, Larry Martell wrote: > > > >On Tue, Mar 1, 2022 at 7:32 PM Rob Cliffe wrote: > > > >> I think itertools.product is what you need. > > > >> Example program: > > > >> > > > >> import itertools > > > >> opsys = ["Linux","Windows"] > > > >> region = ["us-east-1", "us-east-2"] > > > >> print(list(itertools.product(opsys, region))) > > > > > > > >This does not work if region = []. I wrote in question that either > > > >list could be empty. > > > > > > What do you want to get if a list is empty? You haven't said. My > > > personal expectation would be an empty result. > > > > > > Alternatively, if you expect an empty list to imply some single default > > > the the experession: > > > > > > the_list or (the_default,) > > > > > > might be of use. > > > > I've solved the issue. > > -- > > https://mail.python.org/mailman/listinfo/python-list > > Would you be so kind as to show the results of your solution? I posted it at 10:49am Eastern time. From gheskett at shentel.net Wed Mar 2 21:00:48 2022 From: gheskett at shentel.net (gene heskett) Date: Wed, 02 Mar 2022 21:00:48 -0500 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: <4990559.bs1oIaI8Jt@coyote> On Wednesday, 2 March 2022 17:46:49 EST Larry Martell wrote: > On Wed, Mar 2, 2022 at 5:31 PM Joel Goldstick wrote: > > On Wed, Mar 2, 2022 at 5:07 PM Larry Martell wrote: > > > On Wed, Mar 2, 2022 at 5:00 PM Cameron Simpson wrote: > > > > On 02Mar2022 08:29, Larry Martell wrote: > > > > >On Tue, Mar 1, 2022 at 7:32 PM Rob Cliffe wrote: > > > > >> I think itertools.product is what you need. > > > > >> Example program: > > > > >> > > > > >> import itertools > > > > >> opsys = ["Linux","Windows"] > > > > >> region = ["us-east-1", "us-east-2"] > > > > >> print(list(itertools.product(opsys, region))) > > > > > > > > > >This does not work if region = []. I wrote in question that > > > > >either > > > > >list could be empty. > > > > > > > > What do you want to get if a list is empty? You haven't said. My > > > > personal expectation would be an empty result. > > > > > > > > Alternatively, if you expect an empty list to imply some single > > > > default> > > > > > > the the experession: > > > > the_list or (the_default,) > > > > > > > > might be of use. > > > > > > I've solved the issue. > > > -- > > > https://mail.python.org/mailman/listinfo/python-list > > > > Would you be so kind as to show the results of your solution? > > I posted it at 10:49am Eastern time. You may want to repost it Larry, at 9pm eastern it has not arrived here. Cheers, Gene Heskett. -- "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author, 1940) If we desire respect for the law, we must first make the law respectable. - Louis D. Brandeis Genes Web page From gheskett at shentel.net Wed Mar 2 21:04:00 2022 From: gheskett at shentel.net (gene heskett) Date: Wed, 02 Mar 2022 21:04:00 -0500 Subject: All permutations from 2 lists In-Reply-To: References: Message-ID: <2384304.IHPYU4O7oZ@coyote> On Wednesday, 2 March 2022 10:49:11 EST Larry Martell wrote: > On Wed, Mar 2, 2022 at 10:26 AM Antoon Pardon wrote: > > Op 2/03/2022 om 15:58 schreef Larry Martell: > > > On Wed, Mar 2, 2022 at 9:37 AM Antoon Pardon wrote: > > >>>>> If one list is empty I want just the other list. What I am > > >>>>> doing is > > >>>>> building a list to pass to a mongodb query. If region is empty > > >>>>> then I > > >>>>> want to query for just the items in the os list. I guess I can > > >>>>> test > > >>>>> for the lists being empty, but I'd like a solution that handles > > >>>>> that > > >>>>> as down the road there could be more than just 2 lists. > > >>>> > > >>>> How about the following: Keep a list of your lists you want to > > >>>> permute over. Like the following: > > >>>> > > >>>> permutation_elements = [["Linux","Windows"],["us-east-1", > > >>>> "us-east-2"]] > > >>>> > > >>>> permutation = itertools.product(*permutation_elements) > > >>>> > > >>>> If you don't include the empty list, you will get more or less > > >>>> what you seem to want. > > >>> > > >>> But I need to deal with that case. > > >> > > >> What does that mean? How does using the above method to produce > > >> the permutations you want, prevent you from dealing with an empty > > >> list however you want when you encounter them? Just don't add > > >> them to the permutation_elements.> > > > > I need to know what items are in which position. If sometimes the > > > regions are in one index and sometimes in another will not work for > > > me. > > > > I am starting to suspect you didn't think this through. What you are > > telling here contradicts what you told earlier that if either list > > was empty, you just wanted the other list. Because then you wouldn't > > know what items were in that list. > > > > The only solution I can see now is that if a list is empty, you > > either add [None] or [""] to the permutation_elements (whatever > > suits you better) and then use itertools.product > > I found a way to pass this directly into the query: > > def query_lfixer(query): > for k, v in query.items(): > if type(v)==list: > query[k] = {"$in": v} > return query > > self._db_conn[collection_name].find(query_lfixer(query)) I take it back, kmail5 had decided it was a different thread. My bad, no biscuit. Cheers, Gene Heskett. -- "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author, 1940) If we desire respect for the law, we must first make the law respectable. - Louis D. Brandeis Genes Web page From rosuav at gmail.com Wed Mar 2 21:06:29 2022 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 3 Mar 2022 13:06:29 +1100 Subject: All permutations from 2 lists In-Reply-To: <2384304.IHPYU4O7oZ@coyote> References: <2384304.IHPYU4O7oZ@coyote> Message-ID: On Thu, 3 Mar 2022 at 13:05, gene heskett wrote: > I take it back, kmail5 had decided it was a different thread. My bad, no > biscuit. > Awww, I was going to make a really bad joke about timezones :) ChrisA From avigross at verizon.net Wed Mar 2 21:42:00 2022 From: avigross at verizon.net (Avi Gross) Date: Thu, 3 Mar 2022 02:42:00 +0000 (UTC) Subject: All permutations from 2 lists References: <1388685953.247061.1646275320253.ref@mail.yahoo.com> Message-ID: <1388685953.247061.1646275320253@mail.yahoo.com> Larry, i waited patiently to see what others will write and perhaps see if you explain better what you need. You seem to gleefully swat down anything offered. So I am not tempted to engage. Some later messages suggest you may not be specifying quite what you want. It sounds like you are asking for something like a function that combines two things a certain way. Initially it sounded like two simple lists of any length and something like this was proposed: [(f, s) for f in os for s in region] It returns a list of tuples, but could just as easily return a list of lists or dictionary or other variants. You pointed out, validly, that if either of the original lists is empty, it fails. True. But that opens the question of what happens when any random thing is thrown there such as "hello" which Python happily treats as ['h', 'e', 'l', 'l', 'o'] for this purpose. You now seem to want a bulletproof solution that might start with a very basic function like this but add more checks and deal appropriately: def combine_2(first, second): l_first = len(first) l_second = len(second) if ( l_first + l_second == 0 ): return([]) if ( l_first == 0 ): return(second) if ( l_second == 0 ): return(first) return([(f, s) for f in first for s in second]) The above works fine if I give it [] for one or both but maybe not. It returns an empty list if both are empty, which may be what you want or you may want an error or an empty tuple, ... You wanted tuples in return. But if either one is empty, the above returns the original, a list without tuples. To return the valid list with each element as a singleton tuple could easily be done but you have not initially specified the expected behavior. And it is hard to guess as it is not clear what you will do with this. Do you want to be able to do this for more than 2 at a time? At some point, the above approach is not right and an approach that works for an arbitrary number of lists to combine may make sense. Some python modules do something similar to the R function expand.grid() that makes all combinations into rows of a data.frame such as this expand_grid in the pandas module: https://pandas.pydata.org/pandas-docs/version/0.17.1/cookbook.html#creating-example-data I assume that could be used as-is in some applications but could also be easily converted back into a list of tuples or have a generator deliver one result each time ... And, yes, the above approach needs to convert your arguments into a single dictionary but that is easy enough. Getting back a list of lists horizontally may be something like this: df.values.tolist() As others have pointed out, you are actually not stating what you want and are getting the right answer if you ALLOW any of your items to be empty. It is like any operation using an NaN or an Inf do not work well as the result is generally nonsense. If you do not want any empty sets, take care to exclude them. In my dictionary example above, you might add only non-empty lists to the dictionary, for example, and if the dictionary remains empty, then there is nothing to combine. I also see a subtle requirement that is annoying. You want the NAME of each entry preserved in order even as some may be EMPTY. So i ask where these names are stored so you can communicate them. This is trivial in languages like R but perhaps not in Python without some help. My suggestion is that numpy/pandas and their data types can help. Once you make a structure like I describe above, the columns have headers you can ask about and then you can pass the headers of surviving entities that perfectly match the columns, even if you take the columns out into a list of tuples. It looks to some like you want lots of things without being willing to earn them. Any number of methods will easily remove or filter what you keep. Once you have clean data, the remainder can be trivial. You then supplied a solution you said worked fine that looked pretty much NOTHING like what we have been discussing: def query_lfixer(query): for k, v in query.items(): if type(v)==list: query[k] = {"$in": v} return query self._db_conn[collection_name].find(query_lfixer(query)) The above code seems be taking a dictionary and modifying it? And what you needed in the end was a dictionary? So why did so many of us bother? I will keep that in mind for future questions. From philip.bloom at applovin.com Wed Mar 2 22:01:23 2022 From: philip.bloom at applovin.com (Philip Bloom) Date: Wed, 2 Mar 2022 19:01:23 -0800 Subject: Getting Syslog working on OSX Monterey In-Reply-To: <4EAF5AC0-446D-4493-AFF0-2FD8A8889439@barrys-emacs.org> References: <0D40B574-2975-49BD-9E97-9055317169E5@barrys-emacs.org> <20220228214109.ntoihdobzf5naqrx@hjp.at> <4EAF5AC0-446D-4493-AFF0-2FD8A8889439@barrys-emacs.org> Message-ID: I'm probably asking on the wrong list, and probably should bother wherever apple's ASL experts live for changes in monterey. Guess nobody else is seeing this? The same exact code is working just fine on OSX Big Sur, but on OSX Monterey it doesn't work at all. Users that haven't updated are having the program produce logs as it has for years through logging.handlers.SysLogHandler. Mac OSX definitely has a listening socket at '/var/run/syslog' which shows up in Console.app. Apologies, Barry. I'm not quite understanding your responses. When we say OSX has no listener for Syslog, what is the Apple System Log and the general output to Console.app then? I thought that was the local syslog on OSX and had, for years, been behaving as such when using logging.handlers.SysLogHandler with a config in /etc/asl/ to define how it should be routed and the rollover/cleanup frequency. Thanks for replying, just having trouble understanding. On Mon, Feb 28, 2022 at 2:07 PM Barry Scott wrote: > > > > On 28 Feb 2022, at 21:41, Peter J. Holzer wrote: > > > > On 2022-02-27 22:16:54 +0000, Barry wrote: > >> If you look at the code of the logging modules syslog handle you will > see that > >> it does not use syslog. It?s assuming that it can network to a syslog > listener. > >> Such a listener is not running on my systems as far as I know. > >> > >> I have always assumed that if I want a logger syslog handler that I > would have > >> to implement it myself. So far I have code that uses syslog directly > and have > >> not written that code yet. > > > > What do you mean by using syslog directly? The syslog(3) library > > function also just sends messages to a "syslog listener" (more commonly > > called a syslog daemon) - at least on any unix-like system I'm familiar > > with (which doesn't include MacOS). It will, however, always use the > > *local* syslog daemon - AFAIK there is no standard way to open a remote > > connection (many syslog daemons can be configured to forward messages to > > a remote server, however). > > I'm re-reading the code to check on what I'm seeing. (Its been a long > time since I last look deeply at this code). > > You can write to /dev/log if you pass that to > SysLogHandler(address='/dev/log'), but the default is to use a socket > to talk to a network listener on localhost:514. There are no deamons > listening on port 514 on my Fedora systems or mac OS. > > That is not what you would expect as the default if you are using the C > API. > > What you do not see used in the SyslogHandler() is the import syslog > and hence its nor using openlog() etc from syslog API. > > Barry > > > > > hp > > > > -- > > _ | Peter J. Holzer | Story must make more sense than reality. > > |_|_) | | > > | | | hjp at hjp.at | -- Charles Stross, "Creative writing > > __/ | http://www.hjp.at/ | challenge!" > > -- > > https://mail.python.org/mailman/listinfo/python-list > > -- > https://mail.python.org/mailman/listinfo/python-list > -- Philip Bloom Director, Services Engineering *AppLovin Corporation* M: (786)-338-1439 <786-338-1439> [image: LinkedIn] [image: Twitter] [image: Facebook] [image: Instagram] [image: AppLovin] From robin at reportlab.com Thu Mar 3 04:21:42 2022 From: robin at reportlab.com (Robin Becker) Date: Thu, 3 Mar 2022 09:21:42 +0000 Subject: lxml empty versus self closed tag In-Reply-To: <25119.47580.635071.641702@ixdm.fritz.box> References: <23c68bd2-2d86-8be6-e1e3-e6c6acc74db3@everest.reportlab.co.uk> <25119.47580.635071.641702@ixdm.fritz.box> Message-ID: <1d7700a6-b20e-252c-58d7-c3b4760408f0@everest.reportlab.co.uk> On 02/03/2022 18:39, Dieter Maurer wrote: > Robin Becker wrote at 2022-3-2 15:32 +0000: >> I'm using lxml.etree.XMLParser and would like to distinguish >> >> >> >> from >> >> >> >> I seem to have e.getchildren()==[] and e.text==None for both cases. Is there a way to get the first to have e.text=='' > > I do not think so (at least not without a DTD): I have a DTD which has so I guess the empty case is allowed as well as the self closed. I am converting from an older parser which has text=='' for and text==None for the self closed version. I don't think I really need to make the distinction. However, I wonder how lxml can present an empty string content deliberately or if that always has to be a semantic decision. > `' is just a shorthand notation for '' and > the difference has no influence on the DOM. > > Note that `lxml` is just a Python binding for `libxml2`. > All the parsing is done by this library. yes I think I knew that From dieter at handshake.de Thu Mar 3 04:55:04 2022 From: dieter at handshake.de (Dieter Maurer) Date: Thu, 3 Mar 2022 10:55:04 +0100 Subject: lxml empty versus self closed tag In-Reply-To: <1d7700a6-b20e-252c-58d7-c3b4760408f0@everest.reportlab.co.uk> References: <23c68bd2-2d86-8be6-e1e3-e6c6acc74db3@everest.reportlab.co.uk> <25119.47580.635071.641702@ixdm.fritz.box> <1d7700a6-b20e-252c-58d7-c3b4760408f0@everest.reportlab.co.uk> Message-ID: <25120.36984.342209.62224@ixdm.fritz.box> Robin Becker wrote at 2022-3-3 09:21 +0000: >On 02/03/2022 18:39, Dieter Maurer wrote: >> Robin Becker wrote at 2022-3-2 15:32 +0000: >>> I'm using lxml.etree.XMLParser and would like to distinguish >>> >>> >>> >>> from >>> >>> >>> >>> I seem to have e.getchildren()==[] and e.text==None for both cases. Is there a way to get the first to have e.text=='' >> >> I do not think so (at least not without a DTD): > >I have a DTD which has > > > >so I guess the empty case is allowed as well as the self closed. Potentially, something changes when `content` contains `PCDATA` (as one possibility) (but I doubt it). From computermaster360 at gmail.com Thu Mar 3 08:24:05 2022 From: computermaster360 at gmail.com (computermaster360) Date: Thu, 3 Mar 2022 14:24:05 +0100 Subject: Behavior of the for-else construct Message-ID: I want to make a little survey here. Do you find the for-else construct useful? Have you used it in practice? Do you even know how it works, or that there is such a thing in Python? I have used it maybe once. My issue with this construct is that calling the second block `else` doesn't make sense; a much more sensible name would be `then`. Now, imagine a parallel universe, where the for-else construct would have a different behavior: for elem in iterable: process(elem) else: # executed only when the iterable was initially empty print('Nothing to process') Wouldn't this be more natural? I think so. Also, I face this case much more often than having detect whether I broke out of a loop early (which is what the current for-else construct is for). Now someone may argue that it's easy to check whether the iterable is empty beforehand. But is it really? What if it's an iterator? Then one would have to resort to using a flag variable and set it in each iteration of the loop. An ugly alternative would be trying to retrieve the first element of the iterable separately, in a try block before the for-loop, to find out whether the iterable is empty. This would of course require making an iterator of the iterable first (since we can't be sure it is already an iterator), and then -- if there are any elements -- processing the first element separately before the for-loop, which means duplicating the loop body. You can see the whole thing gets really ugly really quickly... What are your thoughts? Do you agree? Or am I just not Dutch enough...? From larry.martell at gmail.com Thu Mar 3 09:07:42 2022 From: larry.martell at gmail.com (Larry Martell) Date: Thu, 3 Mar 2022 09:07:42 -0500 Subject: All permutations from 2 lists In-Reply-To: <1388685953.247061.1646275320253@mail.yahoo.com> References: <1388685953.247061.1646275320253.ref@mail.yahoo.com> <1388685953.247061.1646275320253@mail.yahoo.com> Message-ID: On Wed, Mar 2, 2022 at 9:42 PM Avi Gross via Python-list wrote: > > Larry, > > i waited patiently to see what others will write and perhaps see if you explain better what you need. You seem to gleefully swat down anything offered. So I am not tempted to engage. But then you gave in to the temptation. > And it is hard to guess as it is not clear what you will do with this. In the interests of presenting a minimal example I clearly oversimplified. This is my use case: I get a dict from an outside source. The dict contains key/value pairs that I need to use to query a mongodb database. When the values in the dict are all scalar I can pass the dict directly into the query, e.g.: self._db_conn[collection_name].find(query). But if any of the values are lists that does not work. I need to query with something like the cross product of all the lists. It's not a true product since if a list is empty it means no filtering on that field, not no filtering on all the fields. Originally I did not know I could generate a single query that did that. So I was trying to come up with a way to generate a list of all the permutations and was going to issue a query for each individually. Clearly that would become very inefficient if the lists were long or there were a lot of lists. I then found that I could specify a list with the "$in" clause, hence my solution. > def query_lfixer(query): > for k, v in query.items(): > if type(v)==list: > query[k] = {"$in": v} > return query > > self._db_conn[collection_name].find(query_lfixer(query)) > > > So why did so many of us bother? Indeed - so why did you bother? From grant.b.edwards at gmail.com Thu Mar 3 09:27:13 2022 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Thu, 03 Mar 2022 06:27:13 -0800 (PST) Subject: All permutations from 2 lists References: <2384304.IHPYU4O7oZ@coyote> Message-ID: <6220d041.1c69fb81.5337d.570d@mx.google.com> On 2022-03-03, Chris Angelico wrote: > On Thu, 3 Mar 2022 at 13:05, gene heskett wrote: >> I take it back, kmail5 had decided it was a different thread. My bad, no >> biscuit. >> > > Awww, I was going to make a really bad joke about timezones :) As opposed to all the really good jokes about timezones... ;) From rob.cliffe at btinternet.com Thu Mar 3 09:43:28 2022 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Thu, 3 Mar 2022 14:43:28 +0000 Subject: All permutations from 2 lists In-Reply-To: References: <1388685953.247061.1646275320253.ref@mail.yahoo.com> <1388685953.247061.1646275320253@mail.yahoo.com> Message-ID: On 03/03/2022 14:07, Larry Martell wrote: > On Wed, Mar 2, 2022 at 9:42 PM Avi Gross via Python-list > wrote: >> Larry, >> >> i waited patiently to see what others will write and perhaps see if you explain better what you need. You seem to gleefully swat down anything offered. So I am not tempted to engage. > But then you gave in to the temptation. > >> And it is hard to guess as it is not clear what you will do with this. > In the interests of presenting a minimal example I clearly > oversimplified. This is my use case: I get a dict from an outside > source. The dict contains key/value pairs that I need to use to query > a mongodb database. When the values in the dict are all scalar I can > pass the dict directly into the query, e.g.: > self._db_conn[collection_name].find(query). But if any of the values > are lists that does not work. I need to query with something like the > cross product of all the lists. It's not a true product since if a > list is empty it means no filtering on that field, not no filtering on > all the fields. Originally I did not know I could generate a single > query that did that. So I was trying to come up with a way to generate > a list of all the permutations and was going to issue a query for each > individually. Clearly that would become very inefficient if the lists > were long or there were a lot of lists. I then found that I could > specify a list with the "$in" clause, hence my solution. > >> def query_lfixer(query): >> for k, v in query.items(): >> if type(v)==list: >> query[k] = {"$in": v} >> return query >> >> self._db_conn[collection_name].find(query_lfixer(query)) >> >> >> So why did so many of us bother? > Indeed - so why did you bother? You started with a request for help that did not say what you actually wanted. When people took the time and trouble to give you answers to the question **as posed**, you repeatedly rejected them while still being unclear about what you actually wanted. Now you seem to be being rude to Avi. There are plenty of people willing to help on this list, and plenty of goodwill to those that need help - but goodwill can be used up if it is abused. Respectfully, Rob Cliffe From drsalists at gmail.com Thu Mar 3 09:52:33 2022 From: drsalists at gmail.com (Dan Stromberg) Date: Thu, 3 Mar 2022 06:52:33 -0800 Subject: Behavior of the for-else construct In-Reply-To: References: Message-ID: On Thu, Mar 3, 2022 at 5:24 AM computermaster360 < computermaster360 at gmail.com> wrote: > I want to make a little survey here. > > Do you find the for-else construct useful? Have you used it in > practice? Do you even know how it works, or that there is such a thing > in Python? > > I have used it maybe once. My issue with this construct is that > calling the second block `else` doesn't make sense; a much more > sensible name would be `then`. > I use it and like it. You need break's to make it useful - then it cuts down on unnecessary booleans. From rosuav at gmail.com Thu Mar 3 10:02:17 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 4 Mar 2022 02:02:17 +1100 Subject: Behavior of the for-else construct In-Reply-To: References: Message-ID: On Fri, 4 Mar 2022 at 00:25, computermaster360 wrote: > > I want to make a little survey here. > > Do you find the for-else construct useful? Have you used it in > practice? Do you even know how it works, or that there is such a thing > in Python? Yes, yes, and yes-yes. It's extremely useful. > I have used it maybe once. My issue with this construct is that > calling the second block `else` doesn't make sense; a much more > sensible name would be `then`. Ehh, I think "finally" would be a better keyword, but that has very close connections with exception handling. > Now, imagine a parallel universe, where the for-else construct would > have a different behavior: > > for elem in iterable: > process(elem) > else: > # executed only when the iterable was initially empty > print('Nothing to process') > > Wouldn't this be more natural? I think so. Also, I face this case much > more often than having detect whether I broke out of a loop early > (which is what the current for-else construct is for). This also has value, but not as much. > Now someone may argue that it's easy to check whether the iterable is > empty beforehand. But is it really? What if it's an iterator? > Then one would have to resort to using a flag variable and set it in > each iteration of the loop. An ugly alternative would be trying to > retrieve > the first element of the iterable separately, in a try block before > the for-loop, to find out whether the iterable is empty. This would of > course > require making an iterator of the iterable first (since we can't be > sure it is already an iterator), and then -- if there are any elements > -- processing > the first element separately before the for-loop, which means > duplicating the loop body. You can see the whole thing gets really > ugly really quickly... > > What are your thoughts? Do you agree? Or am I just not Dutch enough...? Both forms have value, and only one of them can be called for-else. Maybe if the current one had been called for-finally, then what you're proposing could have been for-else, and we could theoretically have had for-else-finally (where it goes into the for block once for each element, but if there aren't any, it goes into the else block instead, and either way, if you never break, it goes into the finally before moving on). That ship has sailed, though, and given that people would be confused very greatly by for-finally, I'm not overly sorry with the current state of affairs. ChrisA From barry at barrys-emacs.org Thu Mar 3 10:40:45 2022 From: barry at barrys-emacs.org (Barry Scott) Date: Thu, 3 Mar 2022 15:40:45 +0000 Subject: Behavior of the for-else construct In-Reply-To: References: Message-ID: <8C70C5F1-A4DA-4FD9-A841-4469ADEBE38D@barrys-emacs.org> > On 3 Mar 2022, at 13:24, computermaster360 wrote: > > I want to make a little survey here. > > Do you find the for-else construct useful? No. I always have to look up what condition the else fires on. > Have you used it in > practice? No. > Do you even know how it works, or that there is such a thing > in Python? Yes (when I read the docs) and yes. > I have used it maybe once. My issue with this construct is that > calling the second block `else` doesn't make sense; a much more > sensible name would be `then`. There was a thread on this list asking for alternative syntax for the for: else: that was not taken up. Something like nobreak: instead of else: > > Now, imagine a parallel universe, where the for-else construct would > have a different behavior: > > for elem in iterable: > process(elem) > else: > # executed only when the iterable was initially empty > print('Nothing to process') > > Wouldn't this be more natural? I think so. Also, I face this case much > more often than having detect whether I broke out of a loop early > (which is what the current for-else construct is for). > > Now someone may argue that it's easy to check whether the iterable is > empty beforehand. But is it really? What if it's an iterator? > Then one would have to resort to using a flag variable and set it in > each iteration of the loop. An ugly alternative would be trying to > retrieve > the first element of the iterable separately, in a try block before > the for-loop, to find out whether the iterable is empty. This would of > course > require making an iterator of the iterable first (since we can't be > sure it is already an iterator), and then -- if there are any elements > -- processing > the first element separately before the for-loop, which means > duplicating the loop body. You can see the whole thing gets really > ugly really quickly... > > What are your thoughts? Do you agree? Or am I just not Dutch enough...? Barry > -- > https://mail.python.org/mailman/listinfo/python-list > From python.list at tim.thechases.com Thu Mar 3 11:07:34 2022 From: python.list at tim.thechases.com (Tim Chase) Date: Thu, 3 Mar 2022 10:07:34 -0600 Subject: Timezone jokes (was: All permutations from 2 lists) In-Reply-To: <6220d041.1c69fb81.5337d.570d@mx.google.com> References: <2384304.IHPYU4O7oZ@coyote> <6220d041.1c69fb81.5337d.570d@mx.google.com> Message-ID: <20220303100734.3701ca58@bigbox.attlocal.net> On 2022-03-03 06:27, Grant Edwards wrote: > On 2022-03-03, Chris Angelico wrote: > > Awww, I was going to make a really bad joke about timezones :) > > As opposed to all the really good jokes about timezones... ;) And here I thought you were just Trolling with timezones... https://en.wikipedia.org/wiki/Troll_(research_station)#cite_ref-1 ;-) -tkc From python.list at tim.thechases.com Thu Mar 3 11:03:59 2022 From: python.list at tim.thechases.com (Tim Chase) Date: Thu, 3 Mar 2022 10:03:59 -0600 Subject: Behavior of the for-else construct In-Reply-To: References: Message-ID: <20220303100359.3050f4bb@bigbox.attlocal.net> On 2022-03-04 02:02, Chris Angelico wrote: >> I want to make a little survey here. >> >> Do you find the for-else construct useful? Have you used it in >> practice? Do you even know how it works, or that there is such a >> thing in Python? > > Yes, yes, and yes-yes. It's extremely useful. Just adding another data-point, my answer is the same as above. I use it regularly and frequently wish for it (or a similar "didn't trigger a break-condition in the loop" functionality) when using other programming languages. -tkc From grant.b.edwards at gmail.com Thu Mar 3 11:58:46 2022 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Thu, 03 Mar 2022 08:58:46 -0800 (PST) Subject: Behavior of the for-else construct References: Message-ID: <6220f3c6.1c69fb81.e7cdd.716c@mx.google.com> On 2022-03-03, computermaster360 wrote: > Do you find the for-else construct useful? Yes. > Have you used it in practice? Yes. I don't use it often, but I do use it occasionally. However, I always have to look it up the docs to confirm the logic. I always feel like the else should be executed if the for loop does _not_ terminate naturally, but it's the opposite. > Now, imagine a parallel universe, where the for-else construct would > have a different behavior: > > for elem in iterable: > process(elem) > else: > # executed only when the iterable was initially empty > print('Nothing to process') > > Wouldn't this be more natural? That also make sense. > I think so. Also, I face this case much more often than having > detect whether I broke out of a loop early (which is what the > current for-else construct is for). From avigross at verizon.net Thu Mar 3 12:00:39 2022 From: avigross at verizon.net (Avi Gross) Date: Thu, 3 Mar 2022 17:00:39 +0000 (UTC) Subject: All permutations from 2 lists In-Reply-To: References: <1388685953.247061.1646275320253.ref@mail.yahoo.com> <1388685953.247061.1646275320253@mail.yahoo.com> Message-ID: <1497529778.363953.1646326839989@mail.yahoo.com> Larry, That explanation made more sense and provided context. I fully agree with you that generating the cross product of multiple lists can be messy and large and best avoided. As an example, someone on an R forum presented their version of a way to see what are potential solutions to the game WORDLE at any given time, given the current constraints. The details are not important except that their process makes multiple vectors of the characters that can be allowed for letters "one" through "five" and then generates a data.frame of all combinations. Early in the game you know little and the number of combinations can be as high as 26*26*26*26*26 in the English version. Within a few moves, you may know more but even 15*18*... can be large. So you have data.frames with sometimes millions of rows that are then converted rowwise to five letter words to make a long vector than you query a huge dictionary for each word and produce a list of possible words. Now imagine the same game looking instead for 6 letter words and 7 letter words ... I looked at it and decided it was the wrong approach and in, brief, made a much smaller dictionary containing only the five letter words, and made a regular expression that looked like" "^[letters][^letters]S[more][^this]$" The above has 5 matches that may be for a specific letter you know is there (the S in position 3) or a sequence of letters in square brackets saying any one of those match, or the same with a leading caret saying anything except those. You then simply use the R grep() function to search the list of valid 5-letter words using that pattern and in one sweep get them all without creating humongous data structures. What you describe has some similarities as you searched for an alternate way to do something and it is now clearer why you did not immediately vocalize exactly what you anticipated. But your solution was not a solution to what anyone trying to help was working on. It was a solution to a different problem and what people would have had to know about how you were using a dictionary to pass to a mysterious function was not stated, till now. I would have appreciated it if you had simply stated you decided to use a different way and if anyone is curious, here it is. For the rest of us, I think what we got from the exchange may vary. Some saw it as a natural fit with using something like a nested comprehension, albeit empty lists might need to be dealt with. Others saw a module designed to do such things as an answer. I saw other modules in numpy/pandas as reasonable. Some thought iterators were a part of a solution. The reality is that making permutations and combinations is a fairly common occurance in computer science and it can be expected that many implement one solution or another. But looking at your code, I am amused that you seem to already have not individual lists but a dictionary of named lists. Code similar to what you show now could trivially have removed dictionary items that held only an empty list. And as I pointed out, some of the solutions we came up with that could generalize to any number of lists, happily would accept such a dictionary and generate all combinations. My frustration was not about you asking how to solve a very reasonable problem in Python. It was about the process and what was disclosed and then the expectation that we should have known about things not shared. Certainly sharing too much is a problem too. Your title alone was very concrete asking about 2 lists. It is clear that was not quite your real need. -----Original Message----- From: Larry Martell To: Avi Gross Cc: python-list at python.org Sent: Thu, Mar 3, 2022 9:07 am Subject: Re: All permutations from 2 lists On Wed, Mar 2, 2022 at 9:42 PM Avi Gross via Python-list wrote: > > Larry, > > i waited patiently to see what others will write and perhaps see if you explain better what you need. You seem to gleefully swat down anything offered. So I am not tempted to engage. But then you gave in to the temptation. > And it is hard to guess as it is not clear what you will do with this. In the interests of presenting a minimal example I clearly oversimplified. This is my use case: I get a dict from an outside source. The dict contains key/value pairs that I need to use to query a mongodb database. When the values in the dict are all scalar I can pass the dict directly into the query, e.g.: self._db_conn[collection_name].find(query). But if any of the values are lists that does not work. I need to query with something like the cross product of all the lists. It's not a true product since if a list is empty it means no filtering on that field, not no filtering on all the fields.? Originally I did not know I could generate a single query that did that. So I was trying to come up with a way to generate a list of all the permutations and was going to issue a query for each individually.? Clearly that would become very inefficient if the lists were long or there were a lot of lists. I then found that I could specify a list with the "$in" clause, hence my solution. > def query_lfixer(query): >? ? for k, v in query.items(): >? ? ? ? if type(v)==list: >? ? ? ? ? ? query[k] = {"$in": v} >? ? return query > > self._db_conn[collection_name].find(query_lfixer(query)) > > > So why did so many of us bother? Indeed - so why did you bother? From roel at roelschroeven.net Thu Mar 3 12:11:13 2022 From: roel at roelschroeven.net (Roel Schroeven) Date: Thu, 3 Mar 2022 18:11:13 +0100 Subject: Behavior of the for-else construct In-Reply-To: References: Message-ID: Op 3/03/2022 om 14:24 schreef computermaster360: > I want to make a little survey here. > > Do you find the for-else construct useful? Have you used it in > practice? Do you even know how it works, or that there is such a thing > in Python? - No, or at least not when balanced against the drawbacks as I perceive them. - I have a hard time remembering how it works, which is the major drawback for me. I simply can't seem to wrap my head around it. - Yes > I have used it maybe once. My issue with this construct is that > calling the second block `else` doesn't make sense; a much more > sensible name would be `then`. There have been multiple proposals for a better name, but they all have their own drawbacks. There doesn't seem to be a clear winner. Maybe a bit of an explanation of my dislike for the for-else construct. I can't remember what exactly it does and doesn't do, maybe because of the IMHO unfortunate naming or maybe because it's just an unusal construct, I don't know. While coding I never feel the need for for-else. I like coding in a way where "every line expresses a single thought", as much as possible. That means that I don't mine creating short functions for things that in my mind otherwise break the flow of my code. So I often have short functions with a for loop and early returns. I have never seen an example of for-else that wouldn't be at least as clear with a little helper function (granted I haven't looked all that hard). I'm googling for an example, but all I find are tutorial-type things that don't necessarily compare to real code. But it's the best I can do, so let's dive in. What they all seem to have in common is that they mix business logic and user interface (maybe precisely because of their tutorial nature), which I strive to avoid. For example, on https://www.geeksforgeeks.org/using-else-conditional-statement-with-for-loop-in-python/ I found this example: ??? # Python 3.x program to check if an array consists ??? # of even number ??? def contains_even_number(l): ????? for ele in l: ??????? if ele % 2 == 0: ????????? print ("list contains an even number") ????????? break ????? # This else executes only if break is NEVER ????? # reached and loop terminated after all iterations. ????? else: ??????? print ("list does not contain an even number") ??? # Driver code ??? print ("For List 1:") ??? contains_even_number([1, 9, 8]) ??? print (" \nFor List 2:") ??? contains_even_number([1, 3, 5]) I would always refactor this even just for that mixing of business logic and printing stuff. Perhaps like this: ??? def contains_even_number(lst): ??????? for element in lst: ??????????? if element % 2 == 0: ??????????????? return True ??????? return False ??? def print_contains_even_number(lst): ??????? if contains_even_number(lst): ??????????? print("list contains an even number") ??????? else: ??????????? print("list does not contain an even number") ??? print("For List 1:") ??? print_contains_even_number([1, 9, 8]) ??? print() ??? print("For List 2:") ??? print_contains_even_number([1, 3, 5]) Even without consciously avoiding for-else, it automatically disappears. (In real code I would actually use any(element % 2 == 0 for element in lst) for something simple like this) > Now, imagine a parallel universe, where the for-else construct would > have a different behavior: > > for elem in iterable: > process(elem) > else: > # executed only when the iterable was initially empty > print('Nothing to process') > > Wouldn't this be more natural? I think so. Also, I face this case much > more often than having detect whether I broke out of a loop early > (which is what the current for-else construct is for). That's a different construct, also possibly useful. I think this would be more useful to me than the existing for-else. -- "Man had always assumed that he was more intelligent than dolphins because he had achieved so much ? the wheel, New York, wars and so on ? whilst all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man ? for precisely the same reasons." -- Douglas Adams From rob.cliffe at btinternet.com Thu Mar 3 11:44:28 2022 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Thu, 3 Mar 2022 16:44:28 +0000 Subject: Behavior of the for-else construct In-Reply-To: References: Message-ID: <180be823-a9f6-5e16-e85a-884706a5611f@btinternet.com> It has occasional uses (I THINK I've used it myself) but spelling it `else` is very confusing.? So there have been proposals for an alternative spelling, e.g. `nobreak`. There have also been suggestions for adding other suites after `for', e.g. ??? if the loop WAS exited with `break` ??? if the loop was executed zero times but these have not been accepted. Best wishes Rob Cliffe On 03/03/2022 13:24, computermaster360 wrote: > I want to make a little survey here. > > Do you find the for-else construct useful? Have you used it in > practice? Do you even know how it works, or that there is such a thing > in Python? > > I have used it maybe once. My issue with this construct is that > calling the second block `else` doesn't make sense; a much more > sensible name would be `then`. > > Now, imagine a parallel universe, where the for-else construct would > have a different behavior: > > for elem in iterable: > process(elem) > else: > # executed only when the iterable was initially empty > print('Nothing to process') > > Wouldn't this be more natural? I think so. Also, I face this case much > more often than having detect whether I broke out of a loop early > (which is what the current for-else construct is for). > > Now someone may argue that it's easy to check whether the iterable is > empty beforehand. But is it really? What if it's an iterator? > Then one would have to resort to using a flag variable and set it in > each iteration of the loop. An ugly alternative would be trying to > retrieve > the first element of the iterable separately, in a try block before > the for-loop, to find out whether the iterable is empty. This would of > course > require making an iterator of the iterable first (since we can't be > sure it is already an iterator), and then -- if there are any elements > -- processing > the first element separately before the for-loop, which means > duplicating the loop body. You can see the whole thing gets really > ugly really quickly... > > What are your thoughts? Do you agree? Or am I just not Dutch enough...? From dieter at handshake.de Thu Mar 3 12:22:02 2022 From: dieter at handshake.de (Dieter Maurer) Date: Thu, 3 Mar 2022 18:22:02 +0100 Subject: Behavior of the for-else construct In-Reply-To: References: Message-ID: <25120.63802.741123.898664@ixdm.fritz.box> computermaster360 wrote at 2022-3-3 14:24 +0100: >Do you find the for-else construct useful? Yes. >Have you used it in practice? Yes From Joseph.Schachner at Teledyne.com Thu Mar 3 12:25:54 2022 From: Joseph.Schachner at Teledyne.com (Schachner, Joseph) Date: Thu, 3 Mar 2022 17:25:54 +0000 Subject: Behavior of the for-else construct In-Reply-To: References: Message-ID: Useful: On rare occasions (when a loop has a "break" in it) Used: Yes Know how it works: Yes Even is such a thing: Yes Your suggestion: Also useful. Will require a different keyword. I don't know what that would be. "finally" is available ? Write up a feature request. --- Joseph S. Teledyne Confidential; Commercially Sensitive Business Data -----Original Message----- From: computermaster360 Sent: Thursday, March 3, 2022 8:24 AM To: python-list at python.org Subject: Behavior of the for-else construct I want to make a little survey here. Do you find the for-else construct useful? Have you used it in practice? Do you even know how it works, or that there is such a thing in Python? I have used it maybe once. My issue with this construct is that calling the second block `else` doesn't make sense; a much more sensible name would be `then`. Now, imagine a parallel universe, where the for-else construct would have a different behavior: for elem in iterable: process(elem) else: # executed only when the iterable was initially empty print('Nothing to process') Wouldn't this be more natural? I think so. Also, I face this case much more often than having detect whether I broke out of a loop early (which is what the current for-else construct is for). Now someone may argue that it's easy to check whether the iterable is empty beforehand. But is it really? What if it's an iterator? Then one would have to resort to using a flag variable and set it in each iteration of the loop. An ugly alternative would be trying to retrieve the first element of the iterable separately, in a try block before the for-loop, to find out whether the iterable is empty. This would of course require making an iterator of the iterable first (since we can't be sure it is already an iterator), and then -- if there are any elements -- processing the first element separately before the for-loop, which means duplicating the loop body. You can see the whole thing gets really ugly really quickly... What are your thoughts? Do you agree? Or am I just not Dutch enough...? From rosuav at gmail.com Thu Mar 3 12:40:51 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 4 Mar 2022 04:40:51 +1100 Subject: Timezone jokes (was: All permutations from 2 lists) In-Reply-To: <20220303100734.3701ca58@bigbox.attlocal.net> References: <2384304.IHPYU4O7oZ@coyote> <6220d041.1c69fb81.5337d.570d@mx.google.com> <20220303100734.3701ca58@bigbox.attlocal.net> Message-ID: On Fri, 4 Mar 2022 at 03:29, Tim Chase wrote: > > On 2022-03-03 06:27, Grant Edwards wrote: > > On 2022-03-03, Chris Angelico wrote: > > > Awww, I was going to make a really bad joke about timezones :) > > > > As opposed to all the really good jokes about timezones... ;) > > And here I thought you were just Trolling with timezones... > > https://en.wikipedia.org/wiki/Troll_(research_station)#cite_ref-1 > > ;-) > Best-named research station. It's the only place in the world where, as they move into winter, the clocks move forwards. ChrisA From avigross at verizon.net Thu Mar 3 12:59:33 2022 From: avigross at verizon.net (Avi Gross) Date: Thu, 3 Mar 2022 17:59:33 +0000 (UTC) Subject: Timezone jokes (was: All permutations from 2 lists) In-Reply-To: References: <2384304.IHPYU4O7oZ@coyote> <6220d041.1c69fb81.5337d.570d@mx.google.com> <20220303100734.3701ca58@bigbox.attlocal.net> Message-ID: <980247318.383140.1646330373597@mail.yahoo.com> Chris and others, This is a bit off-topic, but for anyone who saw the recent remake of Around The World in 80 days (and of course earlier versions including the book) will see that it too is a sort of timezone joke! The timezone taketh away and then giveth. -----Original Message----- From: Chris Angelico To: python-list at python.org Sent: Thu, Mar 3, 2022 12:40 pm Subject: Re: Timezone jokes (was: All permutations from 2 lists) On Fri, 4 Mar 2022 at 03:29, Tim Chase wrote: > > On 2022-03-03 06:27, Grant Edwards wrote: > > On 2022-03-03, Chris Angelico wrote: > > > Awww, I was going to make a really bad joke about timezones :) > > > > As opposed to all the really good jokes about timezones... ;) > > And here I thought you were just Trolling with timezones... > > https://en.wikipedia.org/wiki/Troll_(research_station)#cite_ref-1 > > ;-) > Best-named research station. It's the only place in the world where, as they move into winter, the clocks move forwards. ChrisA -- https://mail.python.org/mailman/listinfo/python-list From barry at barrys-emacs.org Thu Mar 3 13:32:55 2022 From: barry at barrys-emacs.org (Barry Scott) Date: Thu, 3 Mar 2022 18:32:55 +0000 Subject: Getting Syslog working on OSX Monterey In-Reply-To: References: <0D40B574-2975-49BD-9E97-9055317169E5@barrys-emacs.org> <20220228214109.ntoihdobzf5naqrx@hjp.at> <4EAF5AC0-446D-4493-AFF0-2FD8A8889439@barrys-emacs.org> Message-ID: > On 3 Mar 2022, at 03:01, Philip Bloom wrote: > > I'm probably asking on the wrong list, and probably should bother wherever apple's ASL experts live for changes in monterey. Guess nobody else is seeing this? > > The same exact code is working just fine on OSX Big Sur, but on OSX Monterey it doesn't work at all. Users that haven't updated are having the program produce logs as it has for years through logging.handlers.SysLogHandler. Mac OSX definitely has a listening socket at '/var/run/syslog' which shows up in Console.app. > > Apologies, Barry. I'm not quite understanding your responses. This works: syslog.openlog(logoption=syslog.LOG_PID, facility=syslog.LOG_USER) syslog.syslog(syslog.LOG_NOTICE, 'QQQ test log') I see the "QQQ test log" when I run syslog command. And I can reproduce that logging.handlers.SysLogHandler does not work. I added debug to the SysLogHandler() code and see that it does indeed write into the /var/run/syslog socket. I suspect that macOS /var/run/syslog does not implement the RFC that this code depends on, but that a wild guess. I suggest that you write your own handler that uses syslog.syslog() and use that. Barry > > When we say OSX has no listener for Syslog, what is the Apple System Log and the general output to Console.app then? I thought that was the local syslog on OSX and had, for years, been behaving as such when using logging.handlers.SysLogHandler with a config in /etc/asl/ to define how it should be routed and the rollover/cleanup frequency. > > Thanks for replying, just having trouble understanding. > > > On Mon, Feb 28, 2022 at 2:07 PM Barry Scott > wrote: > > > > On 28 Feb 2022, at 21:41, Peter J. Holzer > wrote: > > > > On 2022-02-27 22:16:54 +0000, Barry wrote: > >> If you look at the code of the logging modules syslog handle you will see that > >> it does not use syslog. It?s assuming that it can network to a syslog listener. > >> Such a listener is not running on my systems as far as I know. > >> > >> I have always assumed that if I want a logger syslog handler that I would have > >> to implement it myself. So far I have code that uses syslog directly and have > >> not written that code yet. > > > > What do you mean by using syslog directly? The syslog(3) library > > function also just sends messages to a "syslog listener" (more commonly > > called a syslog daemon) - at least on any unix-like system I'm familiar > > with (which doesn't include MacOS). It will, however, always use the > > *local* syslog daemon - AFAIK there is no standard way to open a remote > > connection (many syslog daemons can be configured to forward messages to > > a remote server, however). > > I'm re-reading the code to check on what I'm seeing. (Its been a long > time since I last look deeply at this code). > > You can write to /dev/log if you pass that to > SysLogHandler(address='/dev/log'), but the default is to use a socket > to talk to a network listener on localhost:514. There are no deamons > listening on port 514 on my Fedora systems or mac OS. > > That is not what you would expect as the default if you are using the C > API. > > What you do not see used in the SyslogHandler() is the import syslog > and hence its nor using openlog() etc from syslog API. > > Barry > > > > > hp > > > > -- > > _ | Peter J. Holzer | Story must make more sense than reality. > > |_|_) | | > > | | | hjp at hjp.at | -- Charles Stross, "Creative writing > > __/ | http://www.hjp.at/ | challenge!" > > -- > > https://mail.python.org/mailman/listinfo/python-list > > -- > https://mail.python.org/mailman/listinfo/python-list > > > -- > Philip Bloom > Director, Services Engineering > AppLovin Corporation > M: (786)-338-1439 > > > > > From philip.bloom at applovin.com Thu Mar 3 14:29:22 2022 From: philip.bloom at applovin.com (Philip Bloom) Date: Thu, 3 Mar 2022 11:29:22 -0800 Subject: Getting Syslog working on OSX Monterey In-Reply-To: References: <0D40B574-2975-49BD-9E97-9055317169E5@barrys-emacs.org> <20220228214109.ntoihdobzf5naqrx@hjp.at> <4EAF5AC0-446D-4493-AFF0-2FD8A8889439@barrys-emacs.org> Message-ID: Grabbing latest python that does work. Good we're about to get out of the stone ages a bit here. So findings: Syslog - works in 3.10, broken against monterey in 3.6. Logging.Handlers.Sysloghandler - is broken in both against Monterey. Will bug it for the tracker. Thanks for the feedback. On Thu, Mar 3, 2022 at 10:32 AM Barry Scott wrote: > > > On 3 Mar 2022, at 03:01, Philip Bloom wrote: > > I'm probably asking on the wrong list, and probably should bother wherever > apple's ASL experts live for changes in monterey. Guess nobody else is > seeing this? > > The same exact code is working just fine on OSX Big Sur, but on OSX > Monterey it doesn't work at all. Users that haven't updated are having the > program produce logs as it has for years through > logging.handlers.SysLogHandler. Mac OSX definitely has a listening socket > at '/var/run/syslog' which shows up in Console.app. > > Apologies, Barry. I'm not quite understanding your responses. > > > This works: > > syslog.openlog(logoption=syslog.LOG_PID, facility=syslog.LOG_USER) > syslog.syslog(syslog.LOG_NOTICE, 'QQQ test log') > > I see the "QQQ test log" when I run syslog command. > > And I can reproduce that logging.handlers.SysLogHandler does not work. > I added debug to the SysLogHandler() code and see that it does indeed > write into the /var/run/syslog socket. > I suspect that macOS /var/run/syslog does not implement the RFC that this > code depends on, but that a wild guess. > > I suggest that you write your own handler that uses syslog.syslog() and > use that. > > Barry > > > When we say OSX has no listener for Syslog, what is the Apple System Log > and the general output to Console.app then? I thought that was the local > syslog on OSX and had, for years, been behaving as such when using > logging.handlers.SysLogHandler with a config in /etc/asl/ to define how it > should be routed and the rollover/cleanup frequency. > > Thanks for replying, just having trouble understanding. > > > On Mon, Feb 28, 2022 at 2:07 PM Barry Scott > wrote: > >> >> >> > On 28 Feb 2022, at 21:41, Peter J. Holzer wrote: >> > >> > On 2022-02-27 22:16:54 +0000, Barry wrote: >> >> If you look at the code of the logging modules syslog handle you will >> see that >> >> it does not use syslog. It?s assuming that it can network to a syslog >> listener. >> >> Such a listener is not running on my systems as far as I know. >> >> >> >> I have always assumed that if I want a logger syslog handler that I >> would have >> >> to implement it myself. So far I have code that uses syslog directly >> and have >> >> not written that code yet. >> > >> > What do you mean by using syslog directly? The syslog(3) library >> > function also just sends messages to a "syslog listener" (more commonly >> > called a syslog daemon) - at least on any unix-like system I'm familiar >> > with (which doesn't include MacOS). It will, however, always use the >> > *local* syslog daemon - AFAIK there is no standard way to open a remote >> > connection (many syslog daemons can be configured to forward messages to >> > a remote server, however). >> >> I'm re-reading the code to check on what I'm seeing. (Its been a long >> time since I last look deeply at this code). >> >> You can write to /dev/log if you pass that to >> SysLogHandler(address='/dev/log'), but the default is to use a socket >> to talk to a network listener on localhost:514. There are no deamons >> listening on port 514 on my Fedora systems or mac OS. >> >> That is not what you would expect as the default if you are using the C >> API. >> >> What you do not see used in the SyslogHandler() is the import syslog >> and hence its nor using openlog() etc from syslog API. >> >> Barry >> >> >> >> > hp >> > >> > -- >> > _ | Peter J. Holzer | Story must make more sense than reality. >> > |_|_) | | >> > | | | hjp at hjp.at | -- Charles Stross, "Creative writing >> > __/ | http://www.hjp.at/ | challenge!" >> > -- >> > https://mail.python.org/mailman/listinfo/python-list >> >> -- >> https://mail.python.org/mailman/listinfo/python-list >> > > > -- > Philip Bloom > Director, Services Engineering > *AppLovin Corporation* > M: (786)-338-1439 <786-338-1439> > [image: LinkedIn] [image: > Twitter] [image: Facebook] > [image: Instagram] > > [image: AppLovin] > > > > > > -- Philip Bloom Director, Services Engineering *AppLovin Corporation* M: (786)-338-1439 <786-338-1439> [image: LinkedIn] [image: Twitter] [image: Facebook] [image: Instagram] [image: AppLovin] From computermaster360 at gmail.com Thu Mar 3 14:38:33 2022 From: computermaster360 at gmail.com (computermaster360) Date: Thu, 3 Mar 2022 20:38:33 +0100 Subject: Behavior of the for-else construct In-Reply-To: References: Message-ID: On Thu, 3 Mar 2022 at 18:25, Schachner, Joseph wrote: > I don't know what that would be. "finally" is available ? Write up a feature request. Not sure if you mean `finally` seriously but I think that would about as confusing as the current `else`, if not even more ? Meanwhile, I found another solution which is somewhat neater than using a boolean flag: item = sentinel = object() for item in iterable: do_stuff(item) if item is sentinel: print('No items in iterable!') From michael.stemper at gmail.com Thu Mar 3 09:03:30 2022 From: michael.stemper at gmail.com (Michael F. Stemper) Date: Thu, 3 Mar 2022 08:03:30 -0600 Subject: Behavior of the for-else construct In-Reply-To: References: Message-ID: On 03/03/2022 07.24, computermaster360 wrote: > I want to make a little survey here. > > Do you find the for-else construct useful? Have you used it in > practice? Do you even know how it works, or that there is such a thing > in Python? I only found out about it within the last year or so. I've used it probably half-a-dozen times. It seems quite elegant to me. -- Michael F. Stemper A preposition is something you should never end a sentence with. From jon+usenet at unequivocal.eu Thu Mar 3 09:47:19 2022 From: jon+usenet at unequivocal.eu (Jon Ribbens) Date: Thu, 3 Mar 2022 14:47:19 -0000 (UTC) Subject: Behavior of the for-else construct References: Message-ID: On 2022-03-03, computermaster360 wrote: > Do you find the for-else construct useful? Have you used it in > practice? Yes, I use it frequently. > I have used it maybe once. My issue with this construct is that > calling the second block `else` doesn't make sense; a much more > sensible name would be `then`. You are not the only person with this opinion, although personally I have the opposite opinion. I think of 'for...else' as being a search for something that matches a condition, and the 'else' block is if no item is found that matches. If you think of it like that, the syntax makes perfect sense. > Now, imagine a parallel universe, where the for-else construct would > have a different behavior: > > for elem in iterable: > process(elem) > else: > # executed only when the iterable was initially empty > print('Nothing to process') > > Wouldn't this be more natural? I think so. Also, I face this case much > more often than having detect whether I broke out of a loop early > (which is what the current for-else construct is for). I guess peoples' needs vary. I can't even remember the last time I've needed something as you suggest above - certainly far less often than I need 'for...else' as it is now. > What are your thoughts? Do you agree? I don't agree. But it doesn't really matter if anyone agrees or not, since there is no chance whatsoever that a valid Python syntax is suddenly going to change to mean something completely different, not even in "Python 4000" or whatever far-future version we might imagine. This exact topic was discussd in November 2017 by the way, under the subject heading "Re: replacing `else` with `then` in `for` and `try`". I'm not sure any particular conclusion was reached though except that some people think 'else' is more intuitive and some people think 'then' would be more intuitive. From dciprus at cisco.com Thu Mar 3 15:50:48 2022 From: dciprus at cisco.com (Dan Ciprus (dciprus)) Date: Thu, 3 Mar 2022 20:50:48 +0000 Subject: Python In-Reply-To: <8c0a6db6-8584-e398-554c-8dcbaa123eab@wichmann.us> References: <8c0a6db6-8584-e398-554c-8dcbaa123eab@wichmann.us> Message-ID: <20220303205047.wqpzbjbjd7lvdaao@carbon-gen9> if OP formulates question the way he/she did, it's not worth to respond to it. There is plenty of similar questions in the archive. On Tue, Feb 22, 2022 at 07:07:54AM -0700, Mats Wichmann wrote: >On 2/21/22 23:17, SASI KANTH REDDY GUJJULA wrote: >> Pip files are not installing after the python 3.10.2 version installing in my devise. Please solve this for me. > >Please ask a clearer question. > >Can you tell us what "are not installing" means? Are you getting >permission errors? Are you installing and then unable to import what >you have installed? Something else? > >-- >https://mail.python.org/mailman/listinfo/python-list -- Daniel Ciprus .:|:.:|:. CONSULTING ENGINEER.CUSTOMER DELIVERY Cisco Systems Inc. [ curl -L http://git.io/unix ] -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 659 bytes Desc: not available URL: From akkana at shallowsky.com Thu Mar 3 17:33:34 2022 From: akkana at shallowsky.com (Akkana Peck) Date: Thu, 3 Mar 2022 15:33:34 -0700 Subject: Behavior of the for-else construct In-Reply-To: References: Message-ID: computermaster360 writes: > I want to make a little survey here. > > Do you find the for-else construct useful? No. > Have you used it in practice? Once or twice, but ended up removing it, see below. > Do you even know how it works, or that there is such a thing in Python? I always have to look it up, because to my mind, "else" implies it does something quite different from what it actually does. Which means that even if I worked hard at memorizing what it does, so I didn't have to look it up, I still wouldn't use it in code, because I want my code to be easily readable (including by future-me). for..else makes code difficult to understand by anyone who doesn't use for..else frequently: they might be misled into misunderstanding the control flow. ...Akkana From avigross at verizon.net Thu Mar 3 18:07:59 2022 From: avigross at verizon.net (Avi Gross) Date: Thu, 3 Mar 2022 23:07:59 +0000 (UTC) Subject: Behavior of the for-else construct In-Reply-To: References: Message-ID: <21739669.459456.1646348879560@mail.yahoo.com> The drumbeat I keep hearing is that some people hear/see the same word as implying something else. ELSE is ambiguous in the context it is used. And naturally, since nobody desperately wants to use non-reserved keywords, nobody seems ready to use a word like INSTEAD instead. Ideally, a language should be extendable and some languages like R allow you to place all kinds of things inside percent signs to make new operators like %*% or %PIPE% ... Just because some feature may be wanted is not a good reason to overly complicate a language. Can you imagine how hard it would be both to implement and read something like: ... ELSE: ... OK: ... FINALLY: ... ULTIMATELY: ... What if multiple of things like the above example need to be triggered in some particular order? I have to wonder if some new form of wrapper might have made as much sense as in you wrap your loop in something that sets up and traps various signals that are then produced under conditions specified such as the loop not being entered as the starting condition is sort of null, or an exit due to a break or simply because the code itself throws that signal to be caught ... This reminds me a bit of how some programs add so much functionality because someone thought of it without wondering if anyone (including the ones who sponsored it) would ever want to use it or remember it is there or how. I recall how a version of emacs had a transpose-letter function so after typing "teh" you could hit control-t and a little mock LISP macro would go back and co a cut and go forward and do a paste and leave the cursor where it was. That was sometimes useful, but often just as easy to backspace and retype. But I recall gleefully adding a transpose for words, sentences, paragraphs and was going to add more but I was running out of keystrokes to bind them to and besides it can be fairly easy to select items and yank them and move to where you want them and replace them. To make changes in a language is sometimes really expensive but also dangerous. A "free" language must be added to sparingly and with so many requests, perhaps only a few non bug-fixes can seriously be considered. -----Original Message----- From: Akkana Peck To: python-list at python.org Sent: Thu, Mar 3, 2022 5:33 pm Subject: Re: Behavior of the for-else construct computermaster360 writes: > I want to make a little survey here. > > Do you find the for-else construct useful? No. > Have you used it in practice? Once or twice, but ended up removing it, see below. > Do you even know how it works, or that there is such a thing in Python? I always have to look it up, because to my mind, "else" implies it does something quite different from what it actually does. Which means that even if I worked hard at memorizing what it does, so I didn't have to look it up, I still wouldn't use it in code, because I want my code to be easily readable (including by future-me). for..else makes code difficult to understand by anyone who doesn't use for..else frequently: they might be misled into misunderstanding the control flow. ? ? ? ? ...Akkana -- https://mail.python.org/mailman/listinfo/python-list From rob.cliffe at btinternet.com Thu Mar 3 19:13:00 2022 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Fri, 4 Mar 2022 00:13:00 +0000 Subject: Behavior of the for-else construct In-Reply-To: <21739669.459456.1646348879560@mail.yahoo.com> References: <21739669.459456.1646348879560@mail.yahoo.com> Message-ID: I find it so hard to remember what `for ... else` means that on the very few occasions I have used it, I ALWAYS put a comment alongside/below the `else` to remind myself (and anyone else unfortunate enough to read my code) what triggers it, e.g. ??? for item in search_list: ??? ??? ... ??? ??? ... break ??? else: # if no item in search_list matched the criteria You get the idea. If I really want to remember what this construct means, I remind myself that `else` here really means `no break`.? Would have been better if it had been spelt `nobreak` or similar in the first place. Rob Cliffe On 03/03/2022 23:07, Avi Gross via Python-list wrote: > The drumbeat I keep hearing is that some people hear/see the same word as implying something else. ELSE is ambiguous in the context it is used. > > And naturally, since nobody desperately wants to use non-reserved keywords, nobody seems ready to use a word like INSTEAD instead. > > Ideally, a language should be extendable and some languages like R allow you to place all kinds of things inside percent signs to make new operators like %*% or %PIPE% ... > > Just because some feature may be wanted is not a good reason to overly complicate a language. Can you imagine how hard it would be both to implement and read something like: > > ... > ELSE: > ... > OK: > ... > FINALLY: > ... > ULTIMATELY: > ... > > What if multiple of things like the above example need to be triggered in some particular order? > > I have to wonder if some new form of wrapper might have made as much sense as in you wrap your loop in something that sets up and traps various signals that are then produced under conditions specified such as the loop not being entered as the starting condition is sort of null, or an exit due to a break or simply because the code itself throws that signal to be caught ... > > This reminds me a bit of how some programs add so much functionality because someone thought of it without wondering if anyone (including the ones who sponsored it) would ever want to use it or remember it is there or how. I recall how a version of emacs had a transpose-letter function so after typing "teh" you could hit control-t and a little mock LISP macro would go back and co a cut and go forward and do a paste and leave the cursor where it was. That was sometimes useful, but often just as easy to backspace and retype. But I recall gleefully adding a transpose for words, sentences, paragraphs and was going to add more but I was running out of keystrokes to bind them to and besides it can be fairly easy to select items and yank them and move to where you want them and replace them. > > To make changes in a language is sometimes really expensive but also dangerous. A "free" language must be added to sparingly and with so many requests, perhaps only a few non bug-fixes can seriously be considered. > > > > -----Original Message----- > From: Akkana Peck > To: python-list at python.org > Sent: Thu, Mar 3, 2022 5:33 pm > Subject: Re: Behavior of the for-else construct > > computermaster360 writes: >> I want to make a little survey here. >> >> Do you find the for-else construct useful? > No. > >> Have you used it in practice? > Once or twice, but ended up removing it, see below. > >> Do you even know how it works, or that there is such a thing in Python? > I always have to look it up, because to my mind, "else" implies > it does something quite different from what it actually does. > > Which means that even if I worked hard at memorizing what it does, > so I didn't have to look it up, I still wouldn't use it in code, > because I want my code to be easily readable (including by future-me). > for..else makes code difficult to understand by anyone who doesn't > use for..else frequently: they might be misled into misunderstanding > the control flow. > > ? ? ? ? ...Akkana > From rosuav at gmail.com Thu Mar 3 19:34:07 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 4 Mar 2022 11:34:07 +1100 Subject: Behavior of the for-else construct In-Reply-To: <21739669.459456.1646348879560@mail.yahoo.com> References: <21739669.459456.1646348879560@mail.yahoo.com> Message-ID: On Fri, 4 Mar 2022 at 10:09, Avi Gross via Python-list wrote: > > The drumbeat I keep hearing is that some people hear/see the same word as implying something else. ELSE is ambiguous in the context it is used. > What I'm hearing is that there are, broadly speaking, two types of programmers [1]: 1) Those who think about "for-else" as a search tool and perfectly understand how it behaves 2) Those who have an incorrect idea about what for-else is supposed to do, don't understand it, and don't like it. You could easily make a similar point about a lot of other advanced constructs. Some people don't understand threading, and either dislike it or are scared of it. Some people never get their heads around asyncio and the way that yield points work. Some people can't grok operator precedence, so they parenthesize everything "just to be safe". And some people dislike exceptions so much that they warp all their functions into returning a (value,True) or (error,False) tuple instead. Does this mean that all these features are bad? No. There's no way to make every feature perfectly intuitive to every programmer. Those features are still incredibly useful to the programmers that DO use them. Maybe, with hindsight, for-finally would have been a slightly better spelling than for-else. Who knows. But people simply need to understand it, just like people need to understand how binary floating-point works, and claiming that it's "ambiguous' is simply wrong. It has one meaning in the language, and then if programmers have an incorrect expectation, they need to learn (or to not use the feature, which isn't really a problem, it's just not taking advantage of it). > And naturally, since nobody desperately wants to use non-reserved keywords, nobody seems ready to use a word like INSTEAD instead. > > Ideally, a language should be extendable and some languages like R allow you to place all kinds of things inside percent signs to make new operators like %*% or %PIPE% ... > I don't know what you mean by "extendable", but if you mean that different people should be able to change the language syntax in different ways, then absolutely not. When two different files can be completely different languages based on a few directives, it's extremely difficult to read. (Import hooks, and tools like MacroPy, can be used for this sort of effect. I do not think that we should be using them on a regular basis to change core syntax.) > Just because some feature may be wanted is not a good reason to overly complicate a language. Can you imagine how hard it would be both to implement and read something like: > > ... > ELSE: > ... > OK: > ... > FINALLY: > ... > ULTIMATELY: > ... > > What if multiple of things like the above example need to be triggered in some particular order? I don't know what they'd all mean, but if they were all in the core language, they would have to be supported in arbitrary combinations. It's possible to have a "try-except-else-finally" block in Python, for instance. But if you mean that they should all do what "else" does now, then this is a terrible idea. One way of spelling it is just fine. > This reminds me a bit of how some programs add so much functionality because someone thought of it without wondering if anyone (including the ones who sponsored it) would ever want to use it or remember it is there or how. I recall how a version of emacs had a transpose-letter function so after typing "teh" you could hit control-t and a little mock LISP macro would go back and co a cut and go forward and do a paste and leave the cursor where it was. That was sometimes useful, but often just as easy to backspace and retype. But I recall gleefully adding a transpose for words, sentences, paragraphs and was going to add more but I was running out of keystrokes to bind them to and besides it can be fairly easy to select items and yank them and move to where you want them and replace them. > SciTE has a "transpose lines" feature. I use it frequently. But editor features are quite different from language features. ChrisA [1] Something tells me I've heard this before From avigross at verizon.net Thu Mar 3 19:38:22 2022 From: avigross at verizon.net (Avi Gross) Date: Fri, 4 Mar 2022 00:38:22 +0000 (UTC) Subject: Behavior of the for-else construct In-Reply-To: References: <21739669.459456.1646348879560@mail.yahoo.com> Message-ID: <621325684.471007.1646354302946@mail.yahoo.com> Rob, I regularly code with lots of comments like the one you describe, or mark the end of a region that started on an earlier screen such as a deeply nested construct. I have had problems though when I have shared such code and the recipient strips my comments and then later wants me to make changes or even just explain it! My reply tends to be unprintable as in, well, never mind! This leads to a question I constantly ask. If you were free to design a brand new language now, what would you do different that existing languages have had to deal with the hard way? I recall when filenames and extensions had a limited number of characters allowed and embedded spaces were verboten. This regularity made lots of code possible but then some bright people insisted on allowing spaces and you can no longer easily do things like expand *.c into a long line of text and then unambiguously work on one file name at a time. You can often now create a name like "was file1.c and now is file2.c" and it seems acceptable. Yes, you can work around things and get a vector or list of strings and not a command line of text and all things considered, people can get as much or more work done. I have seen major struggles to get other character sets into languages. Any new language typically should have this built in from scratch and should consider adding non-ASCII characters into the mix. Mathematicians often use lots of weird braces/brackets as an example while normal programs are limited to [{( and maybe < and their counterparts. This leads to odd Python behavior (other languages too) where symbols are re-used ad nauseam. { can mean set or dictionary or simply some other way to group code. So I would love to see some key that allows you to do something like L* to mean the combination is a left bracket and should be treated as the beginning of a sequence expected to end in R* or perhaps *R. That would allow many other symbols to be viewed as balanced entities. Think of how Python expanded using single and double quotes (which arguably might work better if balanced this way) to sometimes using triple quotes to putting letters like "b" or "f" in front to make it a special kind of string. But I suspect programming might just get harder for those who would not appreciate a language that used (many) hundreds of symbols. I do work in many alphabets and many of them pronounce and use letters that look familiar in very different ways and sound them differently and invent new ones. Every time I learn another human language, I have to both incorporate the new symbols and rules and also segregate them a bit from identical or similar things in the languages I already speak. It can be quite a chore. But still, I suspect many people are already familiar with symbols such as from set Theory such as subset and superset that could be used as another pair of parentheses of some type Having a way to enter them using keyboards is a challenge. Back to the topic, I was thinking wickedly of a way to extend the FOR loop with existing keywords while sounding a tad ominous is not with an ELSE but a FOR ... OR ELSE ... -----Original Message----- From: Rob Cliffe via Python-list To: python-list at python.org Sent: Thu, Mar 3, 2022 7:13 pm Subject: Re: Behavior of the for-else construct I find it so hard to remember what `for ... else` means that on the very few occasions I have used it, I ALWAYS put a comment alongside/below the `else` to remind myself (and anyone else unfortunate enough to read my code) what triggers it, e.g. ??? for item in search_list: ??? ??? ... ??? ??? ... break ??? else: # if no item in search_list matched the criteria You get the idea. If I really want to remember what this construct means, I remind myself that `else` here really means `no break`.? Would have been better if it had been spelt `nobreak` or similar in the first place. Rob Cliffe On 03/03/2022 23:07, Avi Gross via Python-list wrote: > The drumbeat I keep hearing is that some people hear/see the same word as implying something else. ELSE is ambiguous in the context it is used. > > And naturally, since nobody desperately wants to use non-reserved keywords, nobody seems ready to use a word like INSTEAD instead. > > Ideally, a language should be extendable and some languages like R allow you to place all kinds of things inside percent signs to make new operators like %*% or %PIPE% ... > > Just because some feature may be wanted is not a good reason to overly complicate a language. Can you imagine how hard it would be both to implement and read something like: > > ... > ELSE: >? ? ... > OK: >? ? ... > FINALLY: >? ? ... > ULTIMATELY: >? ? ... > > What if multiple of things like the above example need to be triggered in some particular order? > > I have to wonder if some new form of wrapper might have made as much sense as in you wrap your loop in something that sets up and traps various signals that are then produced under conditions specified such as the loop not being entered as the starting condition is sort of null, or an exit due to a break or simply because the code itself throws that signal to be caught ... > > This reminds me a bit of how some programs add so much functionality because someone thought of it without wondering if anyone (including the ones who sponsored it) would ever want to use it or remember it is there or how. I recall how a version of emacs had a transpose-letter function so after typing "teh" you could hit control-t and a little mock LISP macro would go back and co a cut and go forward and do a paste and leave the cursor where it was. That was sometimes useful, but often just as easy to backspace and retype. But I recall gleefully adding a transpose for words, sentences, paragraphs and was going to add more but I was running out of keystrokes to bind them to and besides it can be fairly easy to select items and yank them and move to where you want them and replace them. > > To make changes in a language is sometimes really expensive but also dangerous. A "free" language must be added to sparingly and with so many requests, perhaps only a few non bug-fixes can seriously be considered. > > > > -----Original Message----- > From: Akkana Peck > To: python-list at python.org > Sent: Thu, Mar 3, 2022 5:33 pm > Subject: Re: Behavior of the for-else construct > > computermaster360 writes: >> I want to make a little survey here. >> >> Do you find the for-else construct useful? > No. > >> Have you used it in practice? > Once or twice, but ended up removing it, see below. > >> Do you even know how it works, or that there is such a thing in Python? > I always have to look it up, because to my mind, "else" implies > it does something quite different from what it actually does. > > Which means that even if I worked hard at memorizing what it does, > so I didn't have to look it up, I still wouldn't use it in code, > because I want my code to be easily readable (including by future-me). > for..else makes code difficult to understand by anyone who doesn't > use for..else frequently: they might be misled into misunderstanding > the control flow. > >? ? ? ? ? ...Akkana > -- https://mail.python.org/mailman/listinfo/python-list From rosuav at gmail.com Thu Mar 3 19:43:39 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 4 Mar 2022 11:43:39 +1100 Subject: Behavior of the for-else construct In-Reply-To: References: <21739669.459456.1646348879560@mail.yahoo.com> Message-ID: On Fri, 4 Mar 2022 at 11:14, Rob Cliffe via Python-list wrote: > > I find it so hard to remember what `for ... else` means that on the very > few occasions I have used it, I ALWAYS put a comment alongside/below the > `else` to remind myself (and anyone else unfortunate enough to read my > code) what triggers it, e.g. > > for item in search_list: > ... > ... break > else: # if no item in search_list matched the criteria A "break" statement always takes you to the line following the current loop construct. The "else" block is part of the current loop construct. It's no more a "no-break" block than the body of the for loop is a "no-break" body here: for item in stuff: if condition: break frobnicate(item) # if no previous item matched the condition > You get the idea. > If I really want to remember what this construct means, I remind myself > that `else` here really means `no break`. Would have been better if it > had been spelt `nobreak` or similar in the first place. Maybe, but I think that obscures the meaning of it; "finally" isn't quite right either (in a try block, you'd hit a finally block whether you raise an exception or not), but I think it's closer. Creating a new language keyword is an incredibly high cost. Think of it like this: for item in search_list: if condition: pass else: print("Condition not true for this item") for item in search_list: if condition: break else: print("Condition not true for any item") There's a parallel here. Since a for-else loop is basically useless without an if-break construct inside it, the else clause can be thought of as the else on a massive if/elif chain: if stuff[0].is_good: print("This item is good", stuff[0]) elif stuff[1].is_good: print("This item is good", stuff[1]) ... ... elif stuff[n].is_good: print("This item is good", stuff[n]) else: print("All items are bad") As a loop, this looks like this: for item in stuff: if item.is_good: print("This item is good", item) break else: print("All items are bad") The else is attached to the for so that it compasses ALL the if statements, but it's still broadly saying "do this when we don't hit the 'if' condition". Whether that's a sufficient mnemonic, I don't know, but it doesn't really matter; the construct is useful to those of us who want it, and if other people ignore it, that's fine. Nobody ever said you had to use or understand every single feature of the language you're using. ChrisA From rosuav at gmail.com Thu Mar 3 19:55:20 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 4 Mar 2022 11:55:20 +1100 Subject: Behavior of the for-else construct In-Reply-To: <621325684.471007.1646354302946@mail.yahoo.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> Message-ID: On Fri, 4 Mar 2022 at 11:39, Avi Gross via Python-list wrote: > > Rob, > > I regularly code with lots of comments like the one you describe, or mark the end of a region that started on an earlier screen such as a deeply nested construct. > > I have had problems though when I have shared such code and the recipient strips my comments and then later wants me to make changes or even just explain it! My reply tends to be unprintable as in, well, never mind! > If they strip out the comments, you can be confident that the comments were unhelpful :) > This leads to a question I constantly ask. If you were free to design a brand new language now, what would you do different that existing languages have had to deal with the hard way? > Very good way to start thinking. In fact, I'd recommend going further, and actually designing the entire language. (Don't bother actually writing an implementation, but fully lay out the syntax and semantics.) It's a great exercise, and you'll learn why things are the way they are. > I recall when filenames and extensions had a limited number of characters allowed and embedded spaces were verboten. This regularity made lots of code possible but then some bright people insisted on allowing spaces and you can no longer easily do things like expand *.c into a long line of text and then unambiguously work on one file name at a time. You can often now create a name like "was file1.c and now is file2.c" and it seems acceptable. Yes, you can work around things and get a vector or list of strings and not a command line of text and all things considered, people can get as much or more work done. > I don't remember when embedded spaces were verboten, so I'm guessing you're talking about 1970s or earlier, on mainframes? In MS-DOS, it was perfectly possible to have spaces in file names, and OS/2 also had that flexibility (and used it for special files like "EA DATA. SF" on a FAT disk). Windows forbade a bunch of characters in file names, but other systems have always been fine with them. It's not only file names that can be multiple words in a single logical argument. The Windows "start" command has a bizarreness where it takes a quoted string as a title, but a second quoted string as a file name, so <> will open that directory, <> opens a shell with a title of "c:\some_dir", and <> opens the directory, even if it has spaces in it. It's much better to treat arguments as a vector of strings rather than a single string, as the start command tries to. > I have seen major struggles to get other character sets into languages. Any new language typically should have this built in from scratch and should consider adding non-ASCII characters into the mix. Mathematicians often use lots of weird braces/brackets as an example while normal programs are limited to [{( and maybe < and their counterparts. This leads to odd Python behavior (other languages too) where symbols are re-used ad nauseam. { can mean set or dictionary or simply some other way to group code. > Tell me, which is more important: the way the code looks, or the way it is typed? Because your *editor* can control both of these. > So I would love to see some key that allows you to do something like L* to mean the combination is a left bracket and should be treated as the beginning of a sequence expected to end in R* or perhaps *R. That would allow many other symbols to be viewed as balanced entities. Think of how Python expanded using single and double quotes (which arguably might work better if balanced this way) to sometimes using triple quotes to putting letters like "b" or "f" in front to make it a special kind of string. > Okay. Design that in an editor and see if it catches on. I've seen a wide variety of bracket-matching tools (like color-coding all types of bracket according to what they pair with), but none of them really get popular. Baking it into the language means that everyone who uses the language has to be able to work with this. Flesh out this "L*" idea, and explain how it solves the problem. What does it do to the asterisk? > But I suspect programming might just get harder for those who would not appreciate a language that used (many) hundreds of symbols. I do work in many alphabets and many of them pronounce and use letters that look familiar in very different ways and sound them differently and invent new ones. Every time I learn another human language, I have to both incorporate the new symbols and rules and also segregate them a bit from identical or similar things in the languages I already speak. It can be quite a chore. But still, I suspect many people are already familiar with symbols such as from set Theory such as subset and superset that could be used as another pair of parentheses of some type Having a way to enter them using keyboards is a challenge. > Exactly. > Back to the topic, I was thinking wickedly of a way to extend the FOR loop with existing keywords while sounding a tad ominous is not with an ELSE but a FOR ... OR ELSE ... > for victim in debtors: if victim.pay(up): continue if victim.late(): break or else: victim.sleep_with(fishes) There's something in this. ChrisA From avigross at verizon.net Thu Mar 3 20:08:08 2022 From: avigross at verizon.net (Avi Gross) Date: Fri, 4 Mar 2022 01:08:08 +0000 (UTC) Subject: Behavior of the for-else construct In-Reply-To: References: <21739669.459456.1646348879560@mail.yahoo.com> Message-ID: <1277831092.475364.1646356088655@mail.yahoo.com> Chris, Much of what I intended is similar to what you say. I am simply saying the existing group of programmers seems to include people who will not see every thing the same way. There is no way to make them all happy and you have other constraints like not suddenly adding new reserved words, so you do the best you can and hope people can get educated. I have seen lots of otherwise bright people drop out of math classes not because they could not understand the concepts, but because they have trouble handling all the symbols being used, such as Greek letters. Often they get stuck simply because nobody tells them how to pronounce the letter as in epsilon or zeta or aleph. Or consider the many ways various people denote a derivative. For some f'(x) and f''(x) feel right but dy/dx and d^2y/dx^2 (assume since this is plain next, I meant a small superscript 2 representing squared) makes no sense. Others like it the other way and still others like a dot on top of something like an s or two dots. I can mix and match many such representations because I know they are just representations of an underlying thing that I do understand. I have seen people then get stuck at partial derivatives which use a new symbol instead of the d, whose name they do not know like ? Teaching APL also drove some students crazy with all the overprinted symbols. So, yes, using ELSE in what seems to them like multiple and incompatible ways can lead to frustration but is rather unavoidable. Which leads to a point you misunderstood. I was saying R reserved a namespace of sorts so that adding new keywords could be done safely. Users are not expected to make variable names like %in% so you can write code like if (var %in% listing) and you can even change the definition in a sort of overloading to do something different. YES this can lead to others puzzling over your code. But if you need a new keyword, perhaps you could expand into such a reserved corner of the namespace and avoid having to reuse existing key words in possibly incompatible, or at least for some non-intuitive, ways. It does not need to use percent signs, just some notation users would normally not already be using. I am not here to say R is better, just some ideas are very different and thus many things chosen now are not inevitable. It gives me some flexibility in say calling a function as `[`(args) instead of [args] and rewriting it. Python plays lots of similar games, such as the decorators you like to use. In many places, Python makes it easier for me to do things. There really are more like three kinds of Programmers. Some see the world one way and some another way and some are switch hitters. The fourth kind tend not to be programmers! The ones who are adaptable and simply acknowledge that a decision has been made and use the functionality as it is done, do best. No need to complain, just adapt. And, of course, you can just not use anything that does not appeal to you but do not be shocked if you encounter code by others who are using it and be ready to understand it enough for the purpose at hand. If Python was being designed TODAY, I wonder if a larger set of key words would be marked as RESERVED for future expansion including ORELSE and even NEVERTHELESS. -----Original Message----- From: Chris Angelico To: python-list at python.org Sent: Thu, Mar 3, 2022 7:34 pm Subject: Re: Behavior of the for-else construct On Fri, 4 Mar 2022 at 10:09, Avi Gross via Python-list wrote: > > The drumbeat I keep hearing is that some people hear/see the same word as implying something else. ELSE is ambiguous in the context it is used. > What I'm hearing is that there are, broadly speaking, two types of programmers [1]: 1) Those who think about "for-else" as a search tool and perfectly understand how it behaves 2) Those who have an incorrect idea about what for-else is supposed to do, don't understand it, and don't like it. You could easily make a similar point about a lot of other advanced constructs. Some people don't understand threading, and either dislike it or are scared of it. Some people never get their heads around asyncio and the way that yield points work. Some people can't grok operator precedence, so they parenthesize everything "just to be safe". And some people dislike exceptions so much that they warp all their functions into returning a (value,True) or (error,False) tuple instead. Does this mean that all these features are bad? No. There's no way to make every feature perfectly intuitive to every programmer. Those features are still incredibly useful to the programmers that DO use them. Maybe, with hindsight, for-finally would have been a slightly better spelling than for-else. Who knows. But people simply need to understand it, just like people need to understand how binary floating-point works, and claiming that it's "ambiguous' is simply wrong. It has one meaning in the language, and then if programmers have an incorrect expectation, they need to learn (or to not use the feature, which isn't really a problem, it's just not taking advantage of it). > And naturally, since nobody desperately wants to use non-reserved keywords, nobody seems ready to use a word like INSTEAD instead. > > Ideally, a language should be extendable and some languages like R allow you to place all kinds of things inside percent signs to make new operators like %*% or %PIPE% ... > I don't know what you mean by "extendable", but if you mean that different people should be able to change the language syntax in different ways, then absolutely not. When two different files can be completely different languages based on a few directives, it's extremely difficult to read. (Import hooks, and tools like MacroPy, can be used for this sort of effect. I do not think that we should be using them on a regular basis to change core syntax.) > Just because some feature may be wanted is not a good reason to overly complicate a language. Can you imagine how hard it would be both to implement and read something like: > > ... > ELSE: >? ? ... > OK: >? ? ... > FINALLY: >? ? ... > ULTIMATELY: >? ? ... > > What if multiple of things like the above example need to be triggered in some particular order? I don't know what they'd all mean, but if they were all in the core language, they would have to be supported in arbitrary combinations. It's possible to have a "try-except-else-finally" block in Python, for instance. But if you mean that they should all do what "else" does now, then this is a terrible idea. One way of spelling it is just fine. > This reminds me a bit of how some programs add so much functionality because someone thought of it without wondering if anyone (including the ones who sponsored it) would ever want to use it or remember it is there or how. I recall how a version of emacs had a transpose-letter function so after typing "teh" you could hit control-t and a little mock LISP macro would go back and co a cut and go forward and do a paste and leave the cursor where it was. That was sometimes useful, but often just as easy to backspace and retype. But I recall gleefully adding a transpose for words, sentences, paragraphs and was going to add more but I was running out of keystrokes to bind them to and besides it can be fairly easy to select items and yank them and move to where you want them and replace them. > SciTE has a "transpose lines" feature. I use it frequently. But editor features are quite different from language features. ChrisA [1] Something tells me I've heard this before -- https://mail.python.org/mailman/listinfo/python-list From rob.cliffe at btinternet.com Thu Mar 3 20:32:12 2022 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Fri, 4 Mar 2022 01:32:12 +0000 Subject: Behavior of the for-else construct In-Reply-To: References: <21739669.459456.1646348879560@mail.yahoo.com> Message-ID: <46ef1b7f-5689-0cd5-ce89-be4c453d262f@btinternet.com> On 04/03/2022 00:34, Chris Angelico wrote: > On Fri, 4 Mar 2022 at 10:09, Avi Gross via Python-list > wrote: >> The drumbeat I keep hearing is that some people hear/see the same word as implying something else. ELSE is ambiguous in the context it is used. >> > What I'm hearing is that there are, broadly speaking, two types of > programmers [1]: > > 1) Those who think about "for-else" as a search tool and perfectly > understand how it behaves > 2) Those who have an incorrect idea about what for-else is supposed to > do, don't understand it, and don't like it. Or those who have a vague fuzzy idea about it and have trouble remembering what it does. > > You could easily make a similar point about a lot of other advanced > constructs. Some people don't understand threading, and either dislike > it or are scared of it. Some people never get their heads around > asyncio and the way that yield points work. Some people can't grok > operator precedence, so they parenthesize everything "just to be > safe". And some people dislike exceptions so much that they warp all > their functions into returning a (value,True) or (error,False) tuple > instead. Does this mean that all these features are bad? No. You could add examples ad nauseam.? I submit that for-else is a special case.? As evidenced by the number of people (including me) who say they have trouble grokking it. > > There's no way to make every feature perfectly intuitive to every > programmer. Those features are still incredibly useful to the > programmers that DO use them. > > Maybe, with hindsight, for-finally would have been a slightly better > spelling than for-else. No.? "finally" suggests (as analogy to try...finally) that the "finally" suit body is always executed.? Presumably even if an untrapped exception occurs in the for-loop body (or even in the for-loop iterator).? A for-loop can be terminated with "break" for many? conceptually different reasons e.g. ??? A search for a suitable item has found one. ??? Something unexpected has happened. ??? A pre-set allowed execution time has been exceeded. "nobreak"/"no_break" etc. is explicit and conceptually neutral. > Who knows. But people simply need to > understand it, just like people need to understand how binary > floating-point works, and claiming that it's "ambiguous' is simply > wrong. It has one meaning in the language, and then if programmers > have an incorrect expectation, they need to learn (or to not use the > feature, which isn't really a problem, it's just not taking advantage > of it). > >> And naturally, since nobody desperately wants to use non-reserved keywords, nobody seems ready to use a word like INSTEAD instead. >> >> Ideally, a language should be extendable and some languages like R allow you to place all kinds of things inside percent signs to make new operators like %*% or %PIPE% ... >> > I don't know what you mean by "extendable", but if you mean that > different people should be able to change the language syntax in > different ways, then absolutely not. When two different files can be > completely different languages based on a few directives, it's > extremely difficult to read. +0.9, although I do sometimes wish for a macro feature in Python. Like, say, one that would translate "nobreak" into "else". ? > > (Import hooks, and tools like MacroPy, can be used for this sort of > effect. I haven't tried MacroPy yet, maybe someday I will. > I do not think that we should be using them on a regular basis > to change core syntax.) > >> Just because some feature may be wanted is not a good reason to overly complicate a language. Can you imagine how hard it would be both to implement and read something like: >> >> ... >> ELSE: >> ... >> OK: >> ... >> FINALLY: >> ... >> ULTIMATELY: >> ... >> >> What if multiple of things like the above example need to be triggered in some particular order? It would be easy to read if they were spelt sensibly, e.g. ??? if_no_iterations: ??? if_one_iteration: ??? if_multiple_iterations: ??? if_any_iterations: ??? if_break: ??? if_no_break: (I'm not saying that all of these are desirable, just conveying the idea.) If multiple clauses were triggered, they should be executed in the order in which they occur in the code. > I don't know what they'd all mean, but if they were all in the core > language, they would have to be supported in arbitrary combinations. > It's possible to have a "try-except-else-finally" block in Python, for > instance. But if you mean that they should all do what "else" does > now, then this is a terrible idea. One way of spelling it is just > fine. > >> This reminds me a bit of how some programs add so much functionality because someone thought of it without wondering if anyone (including the ones who sponsored it) would ever want to use it or remember it is there or how. I recall how a version of emacs had a transpose-letter function so after typing "teh" you could hit control-t and a little mock LISP macro would go back and co a cut and go forward and do a paste and leave the cursor where it was. That was sometimes useful, but often just as easy to backspace and retype. But I recall gleefully adding a transpose for words, sentences, paragraphs and was going to add more but I was running out of keystrokes to bind them to and besides it can be fairly easy to select items and yank them and move to where you want them and replace them. >> > SciTE has a "transpose lines" feature. I use it frequently. But editor > features are quite different from language features. > > ChrisA > > [1] Something tells me I've heard this before Of course you have.? There are three types of programmer: those that can count, and those that can't. Rob Cliffe From rob.cliffe at btinternet.com Thu Mar 3 20:41:18 2022 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Fri, 4 Mar 2022 01:41:18 +0000 Subject: Behavior of the for-else construct In-Reply-To: <621325684.471007.1646354302946@mail.yahoo.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> Message-ID: <77101e25-a341-ee34-ab6b-abc60f858430@btinternet.com> On 04/03/2022 00:38, Avi Gross via Python-list wrote: > Rob, > > I regularly code with lots of comments like the one you describe, or mark the end of a region that started on an earlier screen such as a deeply nested construct. So do I (and not just in Python).? It's good practice. > > I have had problems though when I have shared such code and the recipient strips my comments and then later wants me to make changes or even just explain it! My reply tends to be unprintable as in, well, never mind! Quite justified.? But why not make changes to/explain from *your* version, not his? > > This leads to a question I constantly ask. If you were free to design a brand new language now, what would you do different that existing languages have had to deal with the hard way? That's such a big question that I can't give an adequate answer. > > I recall when filenames and extensions had a limited number of characters allowed and embedded spaces were verboten. This regularity made lots of code possible but then some bright people insisted on allowing spaces and you can no longer easily do things like expand *.c into a long line of text and then unambiguously work on one file name at a time. You can often now create a name like "was file1.c and now is file2.c" and it seems acceptable. Yes, you can work around things and get a vector or list of strings and not a command line of text and all things considered, people can get as much or more work done. > > I have seen major struggles to get other character sets into languages. Any new language typically should have this built in from scratch and should consider adding non-ASCII characters into the mix. Mathematicians often use lots of weird braces/brackets as an example while normal programs are limited to [{( and maybe < and their counterparts. This leads to odd Python behavior (other languages too) where symbols are re-used ad nauseam. { can mean set or dictionary or simply some other way to group code. > > So I would love to see some key that allows you to do something like L* to mean the combination is a left bracket and should be treated as the beginning of a sequence expected to end in R* or perhaps *R. That would allow many other symbols to be viewed as balanced entities. Think of how Python expanded using single and double quotes (which arguably might work better if balanced this way) to sometimes using triple quotes to putting letters like "b" or "f" in front to make it a special kind of string. > > But I suspect programming might just get harder for those who would not appreciate a language that used (many) hundreds of symbols. +1.? Just remembering how to type them all would be a burden. > I do work in many alphabets and many of them pronounce and use letters that look familiar in very different ways and sound them differently and invent new ones. Every time I learn another human language, I have to both incorporate the new symbols and rules and also segregate them a bit from identical or similar things in the languages I already speak. It can be quite a chore. But still, I suspect many people are already familiar with symbols such as from set Theory such as subset and superset that could be used as another pair of parentheses of some type Having a way to enter them using keyboards is a challenge. > > Back to the topic, I was thinking wickedly of a way to extend the FOR loop with existing keywords while sounding a tad ominous is not with an ELSE but a FOR ... OR ELSE ... > > > -----Original Message----- > From: Rob Cliffe via Python-list > To: python-list at python.org > Sent: Thu, Mar 3, 2022 7:13 pm > Subject: Re: Behavior of the for-else construct > > > I find it so hard to remember what `for ... else` means that on the very > few occasions I have used it, I ALWAYS put a comment alongside/below the > `else` to remind myself (and anyone else unfortunate enough to read my > code) what triggers it, e.g. > > ??? for item in search_list: > ??? ??? ... > ??? ??? ... break > ??? else: # if no item in search_list matched the criteria > > You get the idea. > If I really want to remember what this construct means, I remind myself > that `else` here really means `no break`.? Would have been better if it > had been spelt `nobreak` or similar in the first place. > Rob Cliffe > > > On 03/03/2022 23:07, Avi Gross via Python-list wrote: >> The drumbeat I keep hearing is that some people hear/see the same word as implying something else. ELSE is ambiguous in the context it is used. >> >> And naturally, since nobody desperately wants to use non-reserved keywords, nobody seems ready to use a word like INSTEAD instead. >> >> Ideally, a language should be extendable and some languages like R allow you to place all kinds of things inside percent signs to make new operators like %*% or %PIPE% ... >> >> Just because some feature may be wanted is not a good reason to overly complicate a language. Can you imagine how hard it would be both to implement and read something like: >> >> ... >> ELSE: >> ? ? ... >> OK: >> ? ? ... >> FINALLY: >> ? ? ... >> ULTIMATELY: >> ? ? ... >> >> What if multiple of things like the above example need to be triggered in some particular order? >> >> I have to wonder if some new form of wrapper might have made as much sense as in you wrap your loop in something that sets up and traps various signals that are then produced under conditions specified such as the loop not being entered as the starting condition is sort of null, or an exit due to a break or simply because the code itself throws that signal to be caught ... >> >> This reminds me a bit of how some programs add so much functionality because someone thought of it without wondering if anyone (including the ones who sponsored it) would ever want to use it or remember it is there or how. I recall how a version of emacs had a transpose-letter function so after typing "teh" you could hit control-t and a little mock LISP macro would go back and co a cut and go forward and do a paste and leave the cursor where it was. That was sometimes useful, but often just as easy to backspace and retype. But I recall gleefully adding a transpose for words, sentences, paragraphs and was going to add more but I was running out of keystrokes to bind them to and besides it can be fairly easy to select items and yank them and move to where you want them and replace them. >> >> To make changes in a language is sometimes really expensive but also dangerous. A "free" language must be added to sparingly and with so many requests, perhaps only a few non bug-fixes can seriously be considered. >> >> >> >> -----Original Message----- >> From: Akkana Peck >> To: python-list at python.org >> Sent: Thu, Mar 3, 2022 5:33 pm >> Subject: Re: Behavior of the for-else construct >> >> computermaster360 writes: >>> I want to make a little survey here. >>> >>> Do you find the for-else construct useful? >> No. >> >>> Have you used it in practice? >> Once or twice, but ended up removing it, see below. >> >>> Do you even know how it works, or that there is such a thing in Python? >> I always have to look it up, because to my mind, "else" implies >> it does something quite different from what it actually does. >> >> Which means that even if I worked hard at memorizing what it does, >> so I didn't have to look it up, I still wouldn't use it in code, >> because I want my code to be easily readable (including by future-me). >> for..else makes code difficult to understand by anyone who doesn't >> use for..else frequently: they might be misled into misunderstanding >> the control flow. >> >> ? ? ? ? ? ...Akkana >> From rob.cliffe at btinternet.com Thu Mar 3 20:43:38 2022 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Fri, 4 Mar 2022 01:43:38 +0000 Subject: Behavior of the for-else construct In-Reply-To: References: <21739669.459456.1646348879560@mail.yahoo.com> Message-ID: <5d362e45-1b83-f7a0-8c46-e3b8e720d366@btinternet.com> On 04/03/2022 00:43, Chris Angelico wrote: > On Fri, 4 Mar 2022 at 11:14, Rob Cliffe via Python-list > wrote: >> I find it so hard to remember what `for ... else` means that on the very >> few occasions I have used it, I ALWAYS put a comment alongside/below the >> `else` to remind myself (and anyone else unfortunate enough to read my >> code) what triggers it, e.g. >> >> for item in search_list: >> ... >> ... break >> else: # if no item in search_list matched the criteria > A "break" statement always takes you to the line following the current > loop construct. The "else" block is part of the current loop > construct. It's no more a "no-break" block than the body of the for > loop is a "no-break" body here: > > for item in stuff: > if condition: break > frobnicate(item) # if no previous item matched the condition > >> You get the idea. >> If I really want to remember what this construct means, I remind myself >> that `else` here really means `no break`. Would have been better if it >> had been spelt `nobreak` or similar in the first place. > Maybe, but I think that obscures the meaning of it; "finally" isn't > quite right either (in a try block, you'd hit a finally block whether > you raise an exception or not), but I think it's closer. Creating a > new language keyword is an incredibly high cost. > > Think of it like this: > > for item in search_list: > if condition: pass > else: > print("Condition not true for this item") > > for item in search_list: > if condition: break > else: > print("Condition not true for any item") > > There's a parallel here. Since a for-else loop is basically useless > without an if-break construct inside it, Yes but you have to remember what for-else means even to grasp that point. > the else clause can be > thought of as the else on a massive if/elif chain: > > if stuff[0].is_good: > print("This item is good", stuff[0]) > elif stuff[1].is_good: > print("This item is good", stuff[1]) > ... > ... > elif stuff[n].is_good: > print("This item is good", stuff[n]) > else: > print("All items are bad") > > As a loop, this looks like this: > > for item in stuff: > if item.is_good: > print("This item is good", item) > break > else: > print("All items are bad") > > The else is attached to the for so that it compasses ALL the if > statements, but it's still broadly saying "do this when we don't hit > the 'if' condition". > > Whether that's a sufficient mnemonic, I don't know, Not for me, I'm afraid. > but it doesn't > really matter; the construct is useful to those of us who want it, and > if other people ignore it, that's fine. Nobody ever said you had to > use or understand every single feature of the language you're using. > > ChrisA From ethan at stoneleaf.us Thu Mar 3 20:44:22 2022 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 3 Mar 2022 17:44:22 -0800 Subject: Behavior of the for-else construct In-Reply-To: <46ef1b7f-5689-0cd5-ce89-be4c453d262f@btinternet.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <46ef1b7f-5689-0cd5-ce89-be4c453d262f@btinternet.com> Message-ID: On 3/3/22 5:32 PM, Rob Cliffe via Python-list wrote: > There are three types of programmer: those that can count, and those that can't. Actually, there are 10 types of programmer: those that can count in binary, and those that can't. -- ~Ethan~ From greg.ewing at canterbury.ac.nz Thu Mar 3 20:45:12 2022 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 4 Mar 2022 14:45:12 +1300 Subject: Behavior of the for-else construct In-Reply-To: References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> Message-ID: On 4/03/22 1:55 pm, Chris Angelico wrote: > It's much better to treat arguments as a vector of strings > rather than a single string, as the start command tries to. It would be nice if you could, but as I understand it, Windows always passes arguments to a program as a single string, and then it's up to the program to split it up how it wants. Different programs do that in different ways, hence the inconsistencies in how quoting and whitespace is handled. -- Greg From rob.cliffe at btinternet.com Thu Mar 3 20:54:03 2022 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Fri, 4 Mar 2022 01:54:03 +0000 Subject: Behavior of the for-else construct In-Reply-To: References: <21739669.459456.1646348879560@mail.yahoo.com> <46ef1b7f-5689-0cd5-ce89-be4c453d262f@btinternet.com> Message-ID: <08edeac4-1d59-ce5c-663b-42d75e6d8da4@btinternet.com> On 04/03/2022 01:44, Ethan Furman wrote: > On 3/3/22 5:32 PM, Rob Cliffe via Python-list wrote: > > > There are three types of programmer: those that can count, and those > that can't. > > Actually, there are 10 types of programmer:? those that can count in > binary, and those that can't. 1, 10, many. No problem. > > -- > ~Ethan~ From rob.cliffe at btinternet.com Thu Mar 3 20:52:38 2022 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Fri, 4 Mar 2022 01:52:38 +0000 Subject: Behavior of the for-else construct In-Reply-To: References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> Message-ID: <51597e7e-22a7-aa75-7a89-6f8057487523@btinternet.com> On 04/03/2022 00:55, Chris Angelico wrote: > > for victim in debtors: > if victim.pay(up): continue > if victim.late(): break > or else: > victim.sleep_with(fishes) If you mean "or else" to be synonymous with "else", then only the last debtor is killed, whether he has paid up or not, which seems a tad unfair (except that if there are no debtors either you will crash with a NameError or some random victim will be killed, which seems consistent with Mafia modus operandi while also being a trifle unfair. If (as I suspect) you mean "or else" to mean 'if a break occurred', then at least only one debtor is killed, as an example to the others, and no Exception will occur in the unlikely event of "debtors" being empty. Happy fund-raising! Rob Cliffe > > There's something in this. > > ChrisA From jfong at ms4.hinet.net Thu Mar 3 21:22:01 2022 From: jfong at ms4.hinet.net (Jach Feng) Date: Thu, 3 Mar 2022 18:22:01 -0800 (PST) Subject: Behavior of the for-else construct In-Reply-To: References: <21739669.459456.1646348879560@mail.yahoo.com> <46ef1b7f-5689-0cd5-ce89-be4c453d262f@btinternet.com> <08edeac4-1d59-ce5c-663b-42d75e6d8da4@btinternet.com> Message-ID: <57410dc1-a2cd-45c5-8212-0df598b1f1bbn@googlegroups.com> I never feel confused by "else" because I always think it in "break...else", not "for...else". For those who always think in "for...else" deserves this confusion and it can't be just escaped by replacing with another magic word such as "then" or "finally" etc:-) --Jach From avigross at verizon.net Thu Mar 3 22:02:14 2022 From: avigross at verizon.net (Avi Gross) Date: Fri, 4 Mar 2022 03:02:14 +0000 (UTC) Subject: Behavior of the for-else construct In-Reply-To: <77101e25-a341-ee34-ab6b-abc60f858430@btinternet.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <77101e25-a341-ee34-ab6b-abc60f858430@btinternet.com> Message-ID: <459805806.495251.1646362934339@mail.yahoo.com> Rob, I consider my comments in code I write to be a (silent) part of the code, or worse, some code I commented out but want to be able to look at. I often have code like: # NOTE you can turn on one of the following that applies to your situation # and comment out the others but leave them in place. FILENAME = "C: ..." #FILENAME = "I: ... The second line is uncommented if this is run on another machine where the file is on the I: drive. Often code I write is sent to someone who has to make it work on their machine or selectively has regions turned off. # Decide if the next section should be run by selecting the line that represents your needed. NORMALIZE = True # NORMALIZE = False if (NORMALIZE): ... In cases like the above, which are not the majority of how I use comments, keeping it as I defined it makes sense even if some of the commented code is not running. It makes it easier to make customizations. I often have requests for example to read in a .CSV and depending on the situation, make multiple transformations to it that can include removing columns or filtering out rows, normalizing one or more columns, dealing with outliers beyond some designated level, creating new derived columns based on existing ones, or a Boolean reflecting some complex condition, or grouping it some way or merging it with other data and so on. If writing for myself, I might do it all in one pipeline. But if writing it for someone who plays games and tries this versus that, then something like the above, with suitable comments, lets them experiment by turning sections on and off and other forms of customization. I consider the presence of comments a serious part of what i deliver. Yes, the interpreter (and sometimes compiler) strips them but leaving them in the source file does not strike me as expensive. Once removed, I consider the person who did it suspect and am less inclined to work with them. Consider code with an embedded copyright of some sort (or apparently GNU has a copyleft) and whether it is valid to separate that from the code? And note that some code we write may be quite elegant but also rather mysterious and seeing it again a year later it may not be easy to see why we did that and whether some other way might not work as well. Decent comments explaining the algorithm and why choices were made may make all the difference. Or, they may make it easy to see how to convert the code to deal with one more or one less variable by changing it in the right places consistently. To answer something Chris said and was also mentioned here, I do not consider language design to be easy let alone implementing it. Not at all. BUT I think some changes can be straightforward. Having a symbol like a curly brace mean three new things may be tough to implement. Allowing a new symbol in an expanded set of characters seems more straightforward. Consider an arrow symbol ? pointing to the right and another pointing the other way. Could we add the symbol to the language as a single character, albeit implemented using multiple bytes? If my editor let me insert the darn thing, it might then be a reasonable use for some construct in the language unique and new. Maybe the language would use the notation to hold objects holding a set and not confuse the notations for sets and dictionaries as Python ended up doing. (Yes, I know it is NOT confusing in some ways as one holds key:value pairs and the other just value, but making an empty set now requires the notation of set() while an empty dictionary is {} right? So how hard is it for a newly designed language to recognize any use of one arrow and expect everything up to the next arrow (pointing back) to be the contents of a set? It sounds a tad easier than now when Python interpreters have to pause when they see an open bracket and read what follows to see if everything beyond uses dictionary notation before it decides. But NO, I am not volunteering to do any of that. A language with too many symbols may be far worse. We cannot give every single object type their own symbols. But a few more than we have now might make it easier to create objects Python omitted and numpy and pandas and other modules had to add back the hard way. The only reason this is coming up is teh discussion of how various people react to the exact choice of how to add a new feature. I doubt people will like many choices in a new language created sort of like I describe, either. And nobody wants a new keyboard with a thousand keys, even if their language is Chinese. But there are days I want one a bit like that as I often write, as mentioned, in languages with additional characters or entirely other alphabets. Sigh. Life is complicated, then you die and it simplifies. -----Original Message----- From: Rob Cliffe via Python-list To: python-list at python.org Sent: Thu, Mar 3, 2022 8:41 pm Subject: Re: Behavior of the for-else construct On 04/03/2022 00:38, Avi Gross via Python-list wrote: > Rob, > > I regularly code with lots of comments like the one you describe, or mark the end of a region that started on an earlier screen such as a deeply nested construct. So do I (and not just in Python).? It's good practice. > > I have had problems though when I have shared such code and the recipient strips my comments and then later wants me to make changes or even just explain it! My reply tends to be unprintable as in, well, never mind! Quite justified.? But why not make changes to/explain from *your* version, not his? > > This leads to a question I constantly ask. If you were free to design a brand new language now, what would you do different that existing languages have had to deal with the hard way? That's such a big question that I can't give an adequate answer. > > I recall when filenames and extensions had a limited number of characters allowed and embedded spaces were verboten. This regularity made lots of code possible but then some bright people insisted on allowing spaces and you can no longer easily do things like expand *.c into a long line of text and then unambiguously work on one file name at a time. You can often now create a name like "was file1.c and now is file2.c" and it seems acceptable. Yes, you can work around things and get a vector or list of strings and not a command line of text and all things considered, people can get as much or more work done. > > I have seen major struggles to get other character sets into languages. Any new language typically should have this built in from scratch and should consider adding non-ASCII characters into the mix. Mathematicians often use lots of weird braces/brackets as an example while normal programs are limited to [{( and maybe < and their counterparts. This leads to odd Python behavior (other languages too) where symbols are re-used ad nauseam. { can mean set or dictionary or simply some other way to group code. > > So I would love to see some key that allows you to do something like L* to mean the combination is a left bracket and should be treated as the beginning of a sequence expected to end in R* or perhaps *R. That would allow many other symbols to be viewed as balanced entities. Think of how Python expanded using single and double quotes (which arguably might work better if balanced this way) to sometimes using triple quotes to putting letters like "b" or "f" in front to make it a special kind of string. > > But I suspect programming might just get harder for those who would not appreciate a language that used (many) hundreds of symbols. +1.? Just remembering how to type them all would be a burden. >? I do work in many alphabets and many of them pronounce and use letters that look familiar in very different ways and sound them differently and invent new ones. Every time I learn another human language, I have to both incorporate the new symbols and rules and also segregate them a bit from identical or similar things in the languages I already speak. It can be quite a chore. But still, I suspect many people are already familiar with symbols such as from set Theory such as subset and superset that could be used as another pair of parentheses of some type Having a way to enter them using keyboards is a challenge. > > Back to the topic, I was thinking wickedly of a way to extend the FOR loop with existing keywords while sounding a tad ominous is not with? an ELSE but a FOR ... OR ELSE ... > > > -----Original Message----- > From: Rob Cliffe via Python-list > To: python-list at python.org > Sent: Thu, Mar 3, 2022 7:13 pm > Subject: Re: Behavior of the for-else construct > > > I find it so hard to remember what `for ... else` means that on the very > few occasions I have used it, I ALWAYS put a comment alongside/below the > `else` to remind myself (and anyone else unfortunate enough to read my > code) what triggers it, e.g. > >? ??? for item in search_list: >? ??? ??? ... >? ??? ??? ... break >? ??? else: # if no item in search_list matched the criteria > > You get the idea. > If I really want to remember what this construct means, I remind myself > that `else` here really means `no break`.? Would have been better if it > had been spelt `nobreak` or similar in the first place. > Rob Cliffe > > > On 03/03/2022 23:07, Avi Gross via Python-list wrote: >> The drumbeat I keep hearing is that some people hear/see the same word as implying something else. ELSE is ambiguous in the context it is used. >> >> And naturally, since nobody desperately wants to use non-reserved keywords, nobody seems ready to use a word like INSTEAD instead. >> >> Ideally, a language should be extendable and some languages like R allow you to place all kinds of things inside percent signs to make new operators like %*% or %PIPE% ... >> >> Just because some feature may be wanted is not a good reason to overly complicate a language. Can you imagine how hard it would be both to implement and read something like: >> >> ... >> ELSE: >>? ? ?? ... >> OK: >>? ? ?? ... >> FINALLY: >>? ? ?? ... >> ULTIMATELY: >>? ? ?? ... >> >> What if multiple of things like the above example need to be triggered in some particular order? >> >> I have to wonder if some new form of wrapper might have made as much sense as in you wrap your loop in something that sets up and traps various signals that are then produced under conditions specified such as the loop not being entered as the starting condition is sort of null, or an exit due to a break or simply because the code itself throws that signal to be caught ... >> >> This reminds me a bit of how some programs add so much functionality because someone thought of it without wondering if anyone (including the ones who sponsored it) would ever want to use it or remember it is there or how. I recall how a version of emacs had a transpose-letter function so after typing "teh" you could hit control-t and a little mock LISP macro would go back and co a cut and go forward and do a paste and leave the cursor where it was. That was sometimes useful, but often just as easy to backspace and retype. But I recall gleefully adding a transpose for words, sentences, paragraphs and was going to add more but I was running out of keystrokes to bind them to and besides it can be fairly easy to select items and yank them and move to where you want them and replace them. >> >> To make changes in a language is sometimes really expensive but also dangerous. A "free" language must be added to sparingly and with so many requests, perhaps only a few non bug-fixes can seriously be considered. >> >> >> >> -----Original Message----- >> From: Akkana Peck >> To: python-list at python.org >> Sent: Thu, Mar 3, 2022 5:33 pm >> Subject: Re: Behavior of the for-else construct >> >> computermaster360 writes: >>> I want to make a little survey here. >>> >>> Do you find the for-else construct useful? >> No. >> >>> Have you used it in practice? >> Once or twice, but ended up removing it, see below. >> >>> Do you even know how it works, or that there is such a thing in Python? >> I always have to look it up, because to my mind, "else" implies >> it does something quite different from what it actually does. >> >> Which means that even if I worked hard at memorizing what it does, >> so I didn't have to look it up, I still wouldn't use it in code, >> because I want my code to be easily readable (including by future-me). >> for..else makes code difficult to understand by anyone who doesn't >> use for..else frequently: they might be misled into misunderstanding >> the control flow. >> >>? ? ? ? ? ? ...Akkana >> -- https://mail.python.org/mailman/listinfo/python-list From rosuav at gmail.com Thu Mar 3 22:23:09 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 4 Mar 2022 14:23:09 +1100 Subject: Behavior of the for-else construct In-Reply-To: <459805806.495251.1646362934339@mail.yahoo.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <77101e25-a341-ee34-ab6b-abc60f858430@btinternet.com> <459805806.495251.1646362934339@mail.yahoo.com> Message-ID: On Fri, 4 Mar 2022 at 14:05, Avi Gross via Python-list wrote: > To answer something Chris said and was also mentioned here, I do not consider language design to be easy let alone implementing it. Not at all. BUT I think some changes can be straightforward. Having a symbol like a curly brace mean three new things may be tough to implement. Allowing a new symbol in an expanded set of characters seems more straightforward. > > Consider an arrow symbol ? pointing to the right and another pointing the other way. Could we add the symbol to the language as a single character, albeit implemented using multiple bytes? If my editor let me insert the darn thing, it might then be a reasonable use for some construct in the language unique and new. Maybe the language would use the notation to hold objects holding a set and not confuse the notations for sets and dictionaries as Python ended up doing. (Yes, I know it is NOT confusing in some ways as one holds key:value pairs and the other just value, but making an empty set now requires the notation of set() while an empty dictionary is {} right? > > So how hard is it for a newly designed language to recognize any use of one arrow and expect everything up to the next arrow (pointing back) to be the contents of a set? It sounds a tad easier than now when Python interpreters have to pause when they see an open bracket and read what follows to see if everything beyond uses dictionary notation before it decides. > > But NO, I am not volunteering to do any of that. That is exactly why (well, one of the reasons why) I recommended going the whole way and designing a language in full. You will know exactly how hard it actually is. In fact, "pause when they see an open bracket" is the least of your worries :) ChrisA From avigross at verizon.net Thu Mar 3 22:35:38 2022 From: avigross at verizon.net (Avi Gross) Date: Fri, 4 Mar 2022 03:35:38 +0000 (UTC) Subject: Behavior of the for-else construct In-Reply-To: <57410dc1-a2cd-45c5-8212-0df598b1f1bbn@googlegroups.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <46ef1b7f-5689-0cd5-ce89-be4c453d262f@btinternet.com> <08edeac4-1d59-ce5c-663b-42d75e6d8da4@btinternet.com> <57410dc1-a2cd-45c5-8212-0df598b1f1bbn@googlegroups.com> Message-ID: <855798084.499768.1646364938449@mail.yahoo.com> That is one way to look at it, Jach. Of course, a particular loop may have multiple break statements each meaning something else. The current implementation makes all of them jump to the same ELSE statement so in one sense, I consider the ELSE to be associated with the loop as a whole. Sometimes the loop may not even have a break statement and yet the dangling ELSE seems to be accepted anyway. Some languages allow you to break out of deeply nested loops by jumping up two or more levels, perhaps to a label and are more of a goto. I shudder to think how that would work if each loop had an ELSE dangling. -----Original Message----- From: Jach Feng To: python-list at python.org Sent: Thu, Mar 3, 2022 9:22 pm Subject: Re: Behavior of the for-else construct I never feel confused by "else" because I always think it in "break...else", not "for...else". For those who always think in "for...else" deserves this confusion and it can't be just escaped by replacing with another magic word such as "then" or "finally" etc:-) --Jach -- https://mail.python.org/mailman/listinfo/python-list From rosuav at gmail.com Thu Mar 3 22:41:32 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 4 Mar 2022 14:41:32 +1100 Subject: Behavior of the for-else construct In-Reply-To: <855798084.499768.1646364938449@mail.yahoo.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <46ef1b7f-5689-0cd5-ce89-be4c453d262f@btinternet.com> <08edeac4-1d59-ce5c-663b-42d75e6d8da4@btinternet.com> <57410dc1-a2cd-45c5-8212-0df598b1f1bbn@googlegroups.com> <855798084.499768.1646364938449@mail.yahoo.com> Message-ID: On Fri, 4 Mar 2022 at 14:37, Avi Gross via Python-list wrote: > > That is one way to look at it, Jach. Of course, a particular loop may have multiple break statements each meaning something else. The current implementation makes all of them jump to the same ELSE statement so in one sense, I consider the ELSE to be associated with the loop as a whole. Sometimes the loop may not even have a break statement and yet the dangling ELSE seems to be accepted anyway. > > Some languages allow you to break out of deeply nested loops by jumping up two or more levels, perhaps to a label and are more of a goto. I shudder to think how that would work if each loop had an ELSE dangling. > It would happen exactly the same way. I don't know why you're so afraid of the else clause. It's simply part of the loop body, and if you break out of the loop body, you skip it. If Python had a way to break out of multiple loop bodies at once, it would skip the else clauses of any that you break out of. This is not complicated. You can argue that it could be better named (but honestly, I'm not a fan of any of the other proposed names either), but the concept, at its heart, is a simple one. ChrisA From drsalists at gmail.com Fri Mar 4 00:06:11 2022 From: drsalists at gmail.com (Dan Stromberg) Date: Thu, 3 Mar 2022 21:06:11 -0800 Subject: Python In-Reply-To: <20220303205047.wqpzbjbjd7lvdaao@carbon-gen9> References: <8c0a6db6-8584-e398-554c-8dcbaa123eab@wichmann.us> <20220303205047.wqpzbjbjd7lvdaao@carbon-gen9> Message-ID: Whatever happened to sending a URL to a specific answer in a FAQ list? On Thu, Mar 3, 2022 at 12:52 PM Dan Ciprus (dciprus) via Python-list < python-list at python.org> wrote: > if OP formulates question the way he/she did, it's not worth to respond to > it. > There is plenty of similar questions in the archive. > > On Tue, Feb 22, 2022 at 07:07:54AM -0700, Mats Wichmann wrote: > >On 2/21/22 23:17, SASI KANTH REDDY GUJJULA wrote: > >> Pip files are not installing after the python 3.10.2 version installing > in my devise. Please solve this for me. > > > >Please ask a clearer question. > > > >Can you tell us what "are not installing" means? Are you getting > >permission errors? Are you installing and then unable to import what > >you have installed? Something else? > > > >-- > >https://mail.python.org/mailman/listinfo/python-list > > -- > Daniel Ciprus .:|:.:|:. > CONSULTING ENGINEER.CUSTOMER DELIVERY Cisco Systems Inc. > > [ curl -L http://git.io/unix ] > -- > https://mail.python.org/mailman/listinfo/python-list > From dieter at handshake.de Fri Mar 4 02:12:29 2022 From: dieter at handshake.de (Dieter Maurer) Date: Fri, 4 Mar 2022 08:12:29 +0100 Subject: Behavior of the for-else construct In-Reply-To: References: <21739669.459456.1646348879560@mail.yahoo.com> Message-ID: <25121.48093.26762.592197@ixdm.fritz.box> Rob Cliffe wrote at 2022-3-4 00:13 +0000: >I find it so hard to remember what `for ... else` means that on the very >few occasions I have used it, I ALWAYS put a comment alongside/below the >`else` to remind myself (and anyone else unfortunate enough to read my >code) what triggers it, e.g. > > ??? for item in search_list: > ??? ??? ... > ??? ??? ... break > ??? else: # if no item in search_list matched the criteria > >You get the idea. >If I really want to remember what this construct means, I remind myself >that `else` here really means `no break`.? Would have been better if it >had been spelt `nobreak` or similar in the first place. One of my use cases for `for - else` does not involve a `break`: the initialization of the loop variable when the sequence is empty. It is demonstrated by the following transscript: ```pycon >>> for i in range(0): ... pass ... >>> i Traceback (most recent call last): File "", line 1, in NameError: name 'i' is not defined >>> for i in range(0): ... pass ... else: i = None ... >>> i >>> ``` For this use case, `else` is perfectly named. From rosuav at gmail.com Fri Mar 4 02:18:51 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 4 Mar 2022 18:18:51 +1100 Subject: Behavior of the for-else construct In-Reply-To: <25121.48093.26762.592197@ixdm.fritz.box> References: <21739669.459456.1646348879560@mail.yahoo.com> <25121.48093.26762.592197@ixdm.fritz.box> Message-ID: On Fri, 4 Mar 2022 at 18:13, Dieter Maurer wrote: > > Rob Cliffe wrote at 2022-3-4 00:13 +0000: > >I find it so hard to remember what `for ... else` means that on the very > >few occasions I have used it, I ALWAYS put a comment alongside/below the > >`else` to remind myself (and anyone else unfortunate enough to read my > >code) what triggers it, e.g. > > > > for item in search_list: > > ... > > ... break > > else: # if no item in search_list matched the criteria > > > >You get the idea. > >If I really want to remember what this construct means, I remind myself > >that `else` here really means `no break`. Would have been better if it > >had been spelt `nobreak` or similar in the first place. > > One of my use cases for `for - else` does not involve a `break`: > the initialization of the loop variable when the sequence is empty. > It is demonstrated by the following transscript: > > ```pycon > >>> for i in range(0): > ... pass > ... > >>> i > Traceback (most recent call last): > File "", line 1, in > NameError: name 'i' is not defined > >>> for i in range(0): > ... pass > ... else: i = None > ... > >>> i > >>> > ``` > > For this use case, `else` is perfectly named. What's the point of this? Why not just put "i = None" after the loop? ChrisA From roel at roelschroeven.net Fri Mar 4 03:54:11 2022 From: roel at roelschroeven.net (Roel Schroeven) Date: Fri, 4 Mar 2022 09:54:11 +0100 Subject: Behavior of the for-else construct In-Reply-To: References: <21739669.459456.1646348879560@mail.yahoo.com> <25121.48093.26762.592197@ixdm.fritz.box> Message-ID: <34231cf9-118a-d0ea-1a41-4e4bd80aa92e@roelschroeven.net> Op 4/03/2022 om 8:18 schreef Chris Angelico: > On Fri, 4 Mar 2022 at 18:13, Dieter Maurer wrote: > > One of my use cases for `for - else` does not involve a `break`: > > the initialization of the loop variable when the sequence is empty. > > It is demonstrated by the following transscript: > > > > ```pycon > > >>> for i in range(0): > > ... pass > > ... > > >>> i > > Traceback (most recent call last): > > File "", line 1, in > > NameError: name 'i' is not defined > > >>> for i in range(0): > > ... pass > > ... else: i = None > > ... > > >>> i > > >>> > > ``` > > > > For this use case, `else` is perfectly named. > > What's the point of this? Why not just put "i = None" after the loop? As I understand it: range(0) is just a (bad) example, it's actually about any arbitrary iterable. You don't know in advance if it's empty or not, and you want i to be initialized in all cases. (I don't think I have ever encountered that situation.) You could easily solve this by placing "i = None" before the loop, but I guess in situations where you want to initialize i with the result of an expensive operation Dieter's method could be a reasonable solution. This would actually be a good use case for computermaster360's proposal. -- "Honest criticism is hard to take, particularly from a relative, a friend, an acquaintance, or a stranger." -- Franklin P. Jones From roel at roelschroeven.net Fri Mar 4 04:05:44 2022 From: roel at roelschroeven.net (Roel Schroeven) Date: Fri, 4 Mar 2022 10:05:44 +0100 Subject: Behavior of the for-else construct In-Reply-To: References: <21739669.459456.1646348879560@mail.yahoo.com> Message-ID: Op 4/03/2022 om 1:43 schreef Chris Angelico: > Think of it like this: > > for item in search_list: > if condition: pass > else: > print("Condition not true for this item") > > for item in search_list: > if condition: break > else: > print("Condition not true for any item") > > There's a parallel here. Since a for-else loop is basically useless > without an if-break construct inside it, the else clause can be > thought of as the else on a massive if/elif chain: > > if stuff[0].is_good: > print("This item is good", stuff[0]) > elif stuff[1].is_good: > print("This item is good", stuff[1]) > ... > ... > elif stuff[n].is_good: > print("This item is good", stuff[n]) > else: > print("All items are bad") > > As a loop, this looks like this: > > for item in stuff: > if item.is_good: > print("This item is good", item) > break > else: > print("All items are bad") > > The else is attached to the for so that it compasses ALL the if > statements, but it's still broadly saying "do this when we don't hit > the 'if' condition". > > Whether that's a sufficient mnemonic, I don't know [...] Certainly not for me. It could work for the specific use case you mention, but nothing in the definition restricts it to be used for that use case. As Rob Cliffe says in one of his posts, a for-loop can be terminated with "break" for many? conceptually different reasons. A mnemonic that only works for one specific use case, even if it's the most used one, doesn't work as a mnemonic for the general concept for me. > [...], but it doesn't > really matter; the construct is useful to those of us who want it, and > if other people ignore it, that's fine. Nobody ever said you had to > use or understand every single feature of the language you're using. True, still, I agree with people who think nobreak would have been better because it better conveys the intention. IIRC Raymond Hettinger once said something like that too, so I'm in good company I guess. -- "Honest criticism is hard to take, particularly from a relative, a friend, an acquaintance, or a stranger." -- Franklin P. Jones From gheskett at shentel.net Fri Mar 4 05:01:09 2022 From: gheskett at shentel.net (gene heskett) Date: Fri, 04 Mar 2022 05:01:09 -0500 Subject: Behavior of the for-else construct In-Reply-To: References: <25121.48093.26762.592197@ixdm.fritz.box> Message-ID: <4392817.Tn9u4OIKmF@coyote> On Friday, 4 March 2022 02:18:51 EST Chris Angelico wrote: > On Fri, 4 Mar 2022 at 18:13, Dieter Maurer wrote: > > Rob Cliffe wrote at 2022-3-4 00:13 +0000: > > >I find it so hard to remember what `for ... else` means that on the > > >very few occasions I have used it, I ALWAYS put a comment > > >alongside/below the `else` to remind myself (and anyone else > > >unfortunate enough to read my code) what triggers it, e.g. > > > > > > for item in search_list: > > > ... > > > ... break > > > > > > else: # if no item in search_list matched the criteria > > > > > >You get the idea. > > >If I really want to remember what this construct means, I remind > > >myself that `else` here really means `no break`. Would have been > > >better if it had been spelt `nobreak` or similar in the first > > >place. > > > > One of my use cases for `for - else` does not involve a `break`: > > the initialization of the loop variable when the sequence is empty. > > It is demonstrated by the following transscript: > > > > ```pycon > > > > >>> for i in range(0): > > ... pass > > ... > > > > >>> i > > > > Traceback (most recent call last): > > File "", line 1, in > > > > NameError: name 'i' is not defined > > > > >>> for i in range(0): > > ... pass > > ... else: i = None > > ... > > > > >>> i > > > > ``` > > > > For this use case, `else` is perfectly named. > > What's the point of this? Why not just put "i = None" after the loop? > That makes the logic work, but who then cleans up the trash on the stack. Thats a memory leak. > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > . Cheers, Gene Heskett. -- "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author, 1940) If we desire respect for the law, we must first make the law respectable. - Louis D. Brandeis From rosuav at gmail.com Fri Mar 4 06:11:03 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 4 Mar 2022 22:11:03 +1100 Subject: Behavior of the for-else construct In-Reply-To: <4392817.Tn9u4OIKmF@coyote> References: <25121.48093.26762.592197@ixdm.fritz.box> <4392817.Tn9u4OIKmF@coyote> Message-ID: On Fri, 4 Mar 2022 at 21:02, gene heskett wrote: > That makes the logic work, but who then cleans up the trash on the stack. > Thats a memory leak. > Not sure I follow? ChrisA From h.goebel at goebel-consult.de Fri Mar 4 08:03:50 2022 From: h.goebel at goebel-consult.de (Hartmut Goebel) Date: Fri, 4 Mar 2022 14:03:50 +0100 Subject: virtualenv and make DESTDIR= Message-ID: <05a5b570-d917-2b25-33ea-560fcbd76ce8@goebel-consult.de> Hi, How can I make installing a virtual environment honor DESTDIR? How can I install a virtual environment in $(DESTDIR)$(PREFIX), which behaves as being set-up in $(PREFIX)? (Of course, this virtual environment can not be used. My aim is to ship it as part of a rpm package) In Makefiles is good practice to honor DESTDIR in the "install" target, like this install: ??? install -t $(DESTDIR)$(PREFIX)/bin build/bin/my-tool Now when running ??? python3 -m venv $(DESTDIR)$(PREFIX) all paths in this virtual environment refer to $(DESTDIR)$(PREFIX) instead of just $$(PREFIX) Any ideas? Background: More about DESTDIR: https://www.gnu.org/prep/standards/html_node/DESTDIR.html Following Redhat's commendations, I want to install my (somewhat complex) software into /opt/my-software. To make it easier for users to use the software, my idea was to setup a virtual environment in /opt/my-software. Thus users can easily use /opt/my-software/bin/python and have the library provided by my-software available. My Software also includes some scripts, which will also reside in /opt/my-software/bin and refer to /opt/my-software/bin/python. This will avoid to require users to set up PYTHONPATH when thy want to use MY Software. -- Sch?nen Gru? Hartmut Goebel Dipl.-Informatiker (univ), CISSP, CSSLP, ISO 27001 Lead Implementer Information Security Management, Security Governance, Secure Software Development Goebel Consult, Landshut http://www.goebel-consult.de Blog: https://www.goe-con.de/blog/why-a-pki-is-barely-trustworthy Kolumne: https://www.goe-con.de/hartmut-goebel/cissp-gefluester/2011-11-in-troja-nichts-neues From python.list at tim.thechases.com Fri Mar 4 09:38:52 2022 From: python.list at tim.thechases.com (Tim Chase) Date: Fri, 4 Mar 2022 08:38:52 -0600 Subject: Behavior of the for-else construct In-Reply-To: References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> Message-ID: <20220304083852.28142cce@bigbox.attlocal.net> On 2022-03-04 11:55, Chris Angelico wrote: > In MS-DOS, it was perfectly possible to have spaces in file names DOS didn't allow space (0x20) in filenames unless you hacked it by hex-editing your filesystem (which I may have done a couple times). However it did allow you to use 0xFF in filenames which *appeared* as a space in most character-sets. I may have caused a mild bit of consternation in school computer labs doing this. ;-) > Windows forbade a bunch of characters in file names Both DOS and Windows also had certain reserved filenames https://www.howtogeek.com/fyi/windows-10-still-wont-let-you-use-these-file-names-reserved-in-1974/ that could cause issues if passed to programs. To this day, if you poke around on microsoft.com and change random bits of URLs to include one of those reserved filenames in the GET path, you'll often trigger a 5xx error rather than a 404 that you receive with random jibberish in the same place. https://microsoft.com/?/asdfjkl ? 404 https://microsoft.com/?/lpt1 ? 5xx https://microsoft.com/?/asdfjkl/some/path ? 404 https://microsoft.com/?/lpt1/some/path ? 5xx Just in case you aspire to stir up some trouble. -tkc From avigross at verizon.net Fri Mar 4 11:43:09 2022 From: avigross at verizon.net (Avi Gross) Date: Fri, 4 Mar 2022 16:43:09 +0000 (UTC) Subject: Behavior of the for-else construct In-Reply-To: <25121.48093.26762.592197@ixdm.fritz.box> References: <21739669.459456.1646348879560@mail.yahoo.com> <25121.48093.26762.592197@ixdm.fritz.box> Message-ID: <1277767760.379444.1646412189311@mail.yahoo.com> Dieter, Your use is creative albeit it is not "needed" since all it does is make sure your variable is initialized to something, specifically None. So would this not do the same thing? eye = None for eye in range(0): print(eye) eye If I understand it, your example depends on a variable that is not yet initialized to be used in a loop and retain the last value after the loop. You then set it to None if it is not used as the loop is skipped. Others have shown an example similar to the above of using a sentinel that lets you know if the loop is skipped. Of course, there are some advantages in making it clear by doing it you way that the loop (for example if copied and used elsewhere) needs to include the else statement as an integral part. I would like to suggest a slight modification to the above as in if you are searching for something in either seq1 and if not found in seq2. Call it looking for your green shirt in the closet and if not found, looking in the attic. Would this code make sense as such a use in several ways? In English, look here first and if there is NOTHING there, look in the second place? closet = [] attic = ["Costumes", "Sheets", "Shirts" ] for item in closet: print(item) if item == "Shirts" : print("FOUND in closet!!") else: for item in attic: print(item) if item == "Shirts" : print("FOUND in attic!!") Yes, as discussed, you could do an IF statement to check if closet is empty but for iterators, it gets ... -----Original Message----- From: Dieter Maurer To: Rob Cliffe Cc: python-list at python.org Sent: Fri, Mar 4, 2022 2:12 am Subject: Re: Behavior of the for-else construct Rob Cliffe wrote at 2022-3-4 00:13 +0000: >I find it so hard to remember what `for ... else` means that on the very >few occasions I have used it, I ALWAYS put a comment alongside/below the >`else` to remind myself (and anyone else unfortunate enough to read my >code) what triggers it, e.g. > > ??? for item in search_list: > ??? ??? ... > ??? ??? ... break > ??? else: # if no item in search_list matched the criteria > >You get the idea. >If I really want to remember what this construct means, I remind myself >that `else` here really means `no break`.? Would have been better if it >had been spelt `nobreak` or similar in the first place. One of my use cases for `for - else` does not involve a `break`: the initialization of the loop variable when the sequence is empty. It is demonstrated by the following transscript: ```pycon >>> for i in range(0): ...? pass ... >>> i Traceback (most recent call last): ? File "", line 1, in NameError: name 'i' is not defined >>> for i in range(0): ...? pass ... else: i = None ... >>> i >>> ``` For this use case, `else` is perfectly named. -- https://mail.python.org/mailman/listinfo/python-list From dieter at handshake.de Fri Mar 4 12:08:42 2022 From: dieter at handshake.de (Dieter Maurer) Date: Fri, 4 Mar 2022 18:08:42 +0100 Subject: Behavior of the for-else construct In-Reply-To: <1277767760.379444.1646412189311@mail.yahoo.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <25121.48093.26762.592197@ixdm.fritz.box> <1277767760.379444.1646412189311@mail.yahoo.com> Message-ID: <25122.18330.429979.400137@ixdm.fritz.box> Avi Gross wrote at 2022-3-4 16:43 +0000: >Your use is creative albeit it is not "needed" since all it does is make sure your variable is initialized to something, specifically None. > >So would this not do the same thing? > > eye = None > > for eye in range(0): > print(eye) > > eye It would. But, the `for` construct's main purpose is often to determine a value for the loop variable and in this case, I like that it itself can ensure that it gets a value. From rosuav at gmail.com Fri Mar 4 12:46:14 2022 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 5 Mar 2022 04:46:14 +1100 Subject: Behavior of the for-else construct In-Reply-To: <20220304083852.28142cce@bigbox.attlocal.net> References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304083852.28142cce@bigbox.attlocal.net> Message-ID: On Sat, 5 Mar 2022 at 02:02, Tim Chase wrote: > > On 2022-03-04 11:55, Chris Angelico wrote: > > In MS-DOS, it was perfectly possible to have spaces in file names > > DOS didn't allow space (0x20) in filenames unless you hacked it by > hex-editing your filesystem (which I may have done a couple times). > However it did allow you to use 0xFF in filenames which *appeared* as > a space in most character-sets. Hmm, I'm not sure which APIs worked which way, but I do believe that I messed something up at one point and made a file with an included space (not FF, an actual 20) in it. Maybe it's something to do with the (ancient) FCB-based calls. It was tricky to get rid of that file, though I think it turned out that it could be removed by globbing, putting a question mark where the space was. (Of course, internally, MS-DOS considered that the base name was padded to eight with spaces, and the extension padded to three with spaces, so "READ.ME" would be "READ\x20\x20\x20\x20ME\x20", but that doesn't count, since anything that enumerates the contents of a directory would translate that into the way humans think of it.) > I may have caused a mild bit of consternation in school computer labs > doing this. ;-) Nice :) > > Windows forbade a bunch of characters in file names > > Both DOS and Windows also had certain reserved filenames > > https://www.howtogeek.com/fyi/windows-10-still-wont-let-you-use-these-file-names-reserved-in-1974/ > > that could cause issues if passed to programs. Yup. All because, way back in the day, they didn't want to demand the colon. If you actually *want* to use the printer device, for instance, you could get a hard copy of a directory listing like this: DIR >LPT1: and it's perfectly clear that you don't want to create a file called "LPT1", you want to send it to the printer. But noooooo it had to be that you could just write "LPT1" and it would go to the printer. > To this day, if you poke around on microsoft.com and change random > bits of URLs to include one of those reserved filenames in the GET > path, you'll often trigger a 5xx error rather than a 404 that you > receive with random jibberish in the same place. > > https://microsoft.com/?/asdfjkl ? 404 > https://microsoft.com/?/lpt1 ? 5xx > https://microsoft.com/?/asdfjkl/some/path ? 404 > https://microsoft.com/?/lpt1/some/path ? 5xx > > Just in case you aspire to stir up some trouble. > In theory, file system based URLs could be parsed such that, if you ever hit one of those, it returns "Directory not found". In practice... apparently they didn't do that. As a side point, I've been increasingly avoiding any sort of system whereby I take anything from the user and hand it to the file system. The logic is usually more like: If path matches "/static/%s": 1) Get a full directory listing of the declared static-files directory 2) Search that for the token given 3) If not found, return 404 4) Return the contents of the file, with cache markers Since Windows will never return "lpt1" in that directory listing, I would simply never find it, never even try to open it. This MIGHT be an issue with something that accepts file *uploads*, but I've been getting paranoid about those too, so, uhh... my file upload system now creates URLs that look like this: https://sikorsky.rosuav.com/static/upload-49497888-6bede802d13c8d2f7b92ca9fac7c That was uploaded as "pie.gif" but stored on the file system as ~/stillebot/httpstatic/uploads/49497888-6bede802d13c8d2f7b92ca9fac7c with some metadata stored elsewhere about the user-specified file name. So hey, if you were to try to upload a file that had an NTFS invalid character in it, I wouldn't even notice. Maybe I'm *too* paranoid, but at least I don't have to worry about file system attacks. ChrisA From rosuav at gmail.com Fri Mar 4 12:47:37 2022 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 5 Mar 2022 04:47:37 +1100 Subject: Behavior of the for-else construct In-Reply-To: <1277767760.379444.1646412189311@mail.yahoo.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <25121.48093.26762.592197@ixdm.fritz.box> <1277767760.379444.1646412189311@mail.yahoo.com> Message-ID: On Sat, 5 Mar 2022 at 03:44, Avi Gross via Python-list wrote: > > Dieter, > > Your use is creative albeit it is not "needed" since all it does is make sure your variable is initialized to something, specifically None. > > So would this not do the same thing? > > eye = None > > for eye in range(0): > print(eye) > > eye > > If I understand it, your example depends on a variable that is not yet initialized to be used in a loop and retain the last value after the loop. You then set it to None if it is not used as the loop is skipped. Others have shown an example similar to the above of using a sentinel that lets you know if the loop is skipped. > > Of course, there are some advantages in making it clear by doing it you way that the loop (for example if copied and used elsewhere) needs to include the else statement as an integral part. > > I would like to suggest a slight modification to the above as in if you are searching for something in either seq1 and if not found in seq2. Call it looking for your green shirt in the closet and if not found, looking in the attic. Would this code make sense as such a use in several ways? In English, look here first and if there is NOTHING there, look in the second place? > > closet = [] > > attic = ["Costumes", "Sheets", "Shirts" ] > > for item in closet: > print(item) > if item == "Shirts" : print("FOUND in closet!!") > else: > for item in attic: > print(item) > if item == "Shirts" : print("FOUND in attic!!") > > Yes, as discussed, you could do an IF statement to check if closet is empty but for iterators, it gets ... > Make sure you 'break' after finding it. Otherwise, you'll keep searching the rest of your closet, and then still go on to search your attic. The "else:" clause doesn't help you here unless that break is present. ChrisA From wlfraed at ix.netcom.com Fri Mar 4 11:14:29 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Fri, 04 Mar 2022 11:14:29 -0500 Subject: Behavior of the for-else construct References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304083852.28142cce@bigbox.attlocal.net> Message-ID: On Fri, 4 Mar 2022 08:38:52 -0600, Tim Chase declaimed the following: >DOS didn't allow space (0x20) in filenames unless you hacked it by >hex-editing your filesystem (which I may have done a couple times). >However it did allow you to use 0xFF in filenames which *appeared* as >a space in most character-sets. > >I may have caused a mild bit of consternation in school computer labs >doing this. ;-) > Xerox CP/V actually allowed to be embedded in file names... Try to tell the difference between afile and afile when doing a directory listing. {Though the BASIC interpreter gave it away -- doing a directory from within the interpreter resulted in a hex representation of names with non-graphic characters... In EBCDIC of course} -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From avigross at verizon.net Fri Mar 4 14:45:21 2022 From: avigross at verizon.net (Avi Gross) Date: Fri, 4 Mar 2022 19:45:21 +0000 (UTC) Subject: Behavior of the for-else construct In-Reply-To: <998429825.641242.1646423036979@mail.yahoo.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304083852.28142cce@bigbox.attlocal.net> <998429825.641242.1646423036979@mail.yahoo.com> Message-ID: <1017363129.639920.1646423121331@mail.yahoo.com> {NOTE, after some diversion, this long message does revert a bit to the topic.} Ah, Chris, the games we played when we were young and relatively immature! Has anyone else played with embedding "escape sequences" or other gimmicks in unexpected places like filenames so that on the right terminals, such as the VT100, it displayed oddly or was hard to open or delete as what showed was not quite the underlying representation? The main reason for the restrictions in olden days was cost. Everything was so costly including storage/memory and CPU time. Clearly it is a lot easier to have fixed-length filenames that fit into say 16 bytes, or storing multiple flags about file permissions as single bits, even if it meant lots of bit-twiddling or using masks to retrieve their values. We think nothing of creating structures that have many others embedded in them as attributes or function calls that allow a hundred optional arguments so that the function spends much of the time used just figuring out what was set before doing whatever calculation is required to fulfill the request. I was reading a novel recently (Jack Reacher Series) where the main character is noticing how much technology has changed as they have been ignoring it for a decade or so. Everything seems to be coming up faster. My view was that if something seems ten times as fast as it was, it also probably is doing a hundred or ten thousand times as much to get that result.? The real speed changes are often counterbalanced by expecting to do more. A web page served may display a screen of text but to transmit it may include not just lots of padding in the HTML, but have all kinds of code such as in Java or JavaScript or lots of back and forth with the server to keep something like a graph displayed being refreshed ... So back to filenames, the concept of having to search for long filenames that may not even be stored sequentially in large blocks that can be read (ahead) efficiently, may have seemed to be so illogical as not to be considered. So given that the shorter ones were not allowed to have embedded spaces, it made sense to treat them like tokens that could be broken up at whitespace. As mentioned, languages (or other programs) would often parse a command line and create something like this for the main program in C with suitable version in Python and other languages: ? main(int argc, char *argv[]) The code variations on the above do suppose that something has already parsed the command line that invoked them and partitioned it properly into individual strings placed in an array of such strings and also returned how many arguments it saw. Users invoking the program needed to be careful such as using double quotes around anything with embedded spaces, where allowed. But like many paradigms, there can be a shift. Consider the fact that languages like Python are constantly parsing things like names. Can you create a variable name like "me first" with an embedded space or even other symbols normally reserved such as parentheses? Most languages do not like such things. It makes it hard to parse if not quoted in some unique way. Yet languages like R happily allow such constructs if placed in back quotes (grave accents?) as in `me & you` as a variable name or the name of a function. Of course, you then never use the darn name without the extra quotes. Similarly, when you make an object like a DataFrame, can you include spaces and other things in the names of columns (or sometimes rows)? If so, is there only access some ways and not others? The answer often is not simple. As Chris repeatedly highlights, making a language consistent as you play with features can be VERY hard and sometimes not quite possible without relaxing some rules or making exceptions. Sometimes the answer varies. In base R a data.frame can be given a column name like "me + you" which it then stores as "me...you" leading to odd results. But it happily returns that result if you ask for mydf$me using auto-completion. Spell it out fully and it won't find it! A later package added on makes modified data.frame objects called tibbles which do not autocomplete but do completely store and let you access the name so mydf$me fails and mydf$"me + you" or mydf$`me + you` works but oddly an alternative format like mydf[, "me + you"] works while the similar mydf[, `me + you`] fails! My point is not about R but a more general one. I can rant about many other languages, LOL! Allowing spaces or other characters in what used to be a more easily viewable name that can be parsed easier, can lead to having to find every place such things are used and seeing if they can be made to work consistently. I show an example above where it is not consistent, in my view. But when humans view things and their perceptions differ, you are inviting disagreements about whatever you implement. You may end up having to make people do more than they would prefer such as quoting all variable names even if they do not "need" it. Wouldn't it be nice to be able to name a variable 1time at least if you felt like it? Many people have passwords like that. I think the answer is NO, not if it meant quoting every variable because there was no longer any reasonable way to parse programs. The issue that started the discussion was different but in a sense similar. If you want to extend the functionality of a "for" loop in one of many possible ways, how do you design a way to specify it so it can both be unambiguously parsed and implemented while at the same time making sense to humans reading it and interpreting it using their human language skills. I happen to think this is even harder for some who speak in languages other than English and have to program in languages loosely based on English. I am happy that I seem to think in English now but I was seven when I first encountered it after thinking in others. People who program do not all speak English or are more fluent in other languages. T may may be used to other word orders for example. They may move verbs to the end of a sentence? or place adjectives or other modifiers after versus before a word and forget about all the other games played where the same word means something completely different. To them ELSE may either mean nothing or the phrase IF ... ELSE may be said differently or adding a clause after the construct is not seen as natural. So was this way of doing FOR ... ELSE the only or even best way, is what some of this debate is about. I am thinking of a function in some languages that lets you specify what should happen in a later circumstance. In a language like R, you can specify one or more instances of on.exit(...) that are not run immediately. Each one can replace the commands in the previous one or add on to it. When the function they are defined in exits for any reason, it pauses to run any uncleared such commands. Clearly this works better if a language has a way to defer evaluation of code so there are no side effects. So consider the suggestion of code that should be run if you have a loop and you break out of it. Could you design an alternate way to handle that other than adding an ELSE clause after the loop? Clearly you could simply add a function called on.break() that can be used as described but only within the body of that loop. It might be something that can be set and unset as needed and when the loop is exited, the program implicitly checks to see if any code has been dynamically set and executes it. This clearly is not necessarily a good way or better way, but is an example of how you can implement something without using any key words. No need to think about forcing the use of ELSE versus a new keyword that may conflict with existing code. Yes, the name on.break may conflict but that is trivially handled in Python by invoking it with a full name that includes what module it is in or by creating an alias. So what about considering an alternate approach that does handle a for loop that does nothing? Would it create huge incompatibilities for something like: for eye in range(0), on.empty=... : ? ? pass In some languages, arbitrary additional arguments are allowed, and if not understood, are ignored. Python does not allow anything like the above. And in this case, the entire body of the for loop is never evaluated so no gimmicks inside the body are possible. A gimmick before it might work and I even wonder if there is room here for a decorator concept like: @on.empty(...) for eye in range(0): ? ? pass I am ending with a reminder. NOTHING I am writing here is meant to be taken seriously but merely as part of a well-intentioned debate to share ideas and not to win or lose but learn. Python is more than a language but also has aspects of a culture and we sometimes talk about whether something has a pythonic flavor or is pythonic versus translating it literally from a language like C rather than using the ideas common in python. The method chosen to implement the ELSE clause here may well be Pythonic and some of my attempts to show other ways may well not be. I am not one of those that find the current implementation to be the wrong one and will happily use it when I have code that can be done well that way. I am just discussing the issue and wider ones. Languages have an amazing variety of designs that fascinate me. -----Original Message----- From: Chris Angelico To: python-list at python.org Sent: Fri, Mar 4, 2022 12:46 pm Subject: Re: Behavior of the for-else construct On Sat, 5 Mar 2022 at 02:02, Tim Chase wrote: > > On 2022-03-04 11:55, Chris Angelico wrote: > > In MS-DOS, it was perfectly possible to have spaces in file names > > DOS didn't allow space (0x20) in filenames unless you hacked it by > hex-editing your filesystem (which I may have done a couple times). > However it did allow you to use 0xFF in filenames which *appeared* as > a space in most character-sets. Hmm, I'm not sure which APIs worked which way, but I do believe that I messed something up at one point and made a file with an included space (not FF, an actual 20) in it. Maybe it's something to do with the (ancient) FCB-based calls. It was tricky to get rid of that file, though I think it turned out that it could be removed by globbing, putting a question mark where the space was. (Of course, internally, MS-DOS considered that the base name was padded to eight with spaces, and the extension padded to three with spaces, so "READ.ME" would be "READ\x20\x20\x20\x20ME\x20", but that doesn't count, since anything that enumerates the contents of a directory would translate that into the way humans think of it.) > I may have caused a mild bit of consternation in school computer labs > doing this. ;-) Nice :) > > Windows forbade a bunch of characters in file names > > Both DOS and Windows also had certain reserved filenames > > https://www.howtogeek.com/fyi/windows-10-still-wont-let-you-use-these-file-names-reserved-in-1974/ > > that could cause issues if passed to programs. Yup. All because, way back in the day, they didn't want to demand the colon. If you actually *want* to use the printer device, for instance, you could get a hard copy of a directory listing like this: DIR >LPT1: and it's perfectly clear that you don't want to create a file called "LPT1", you want to send it to the printer. But noooooo it had to be that you could just write "LPT1" and it would go to the printer. > To this day, if you poke around on microsoft.com and change random > bits of URLs to include one of those reserved filenames in the GET > path, you'll often trigger a 5xx error rather than a 404 that you > receive with random jibberish in the same place. > >?? https://microsoft.com/?/asdfjkl ? 404 >?? https://microsoft.com/?/lpt1 ? 5xx >?? https://microsoft.com/?/asdfjkl/some/path ? 404 >?? https://microsoft.com/?/lpt1/some/path ? 5xx > > Just in case you aspire to stir up some trouble. > In theory, file system based URLs could be parsed such that, if you ever hit one of those, it returns "Directory not found". In practice... apparently they didn't do that. As a side point, I've been increasingly avoiding any sort of system whereby I take anything from the user and hand it to the file system. The logic is usually more like: If path matches "/static/%s": 1) Get a full directory listing of the declared static-files directory 2) Search that for the token given 3) If not found, return 404 4) Return the contents of the file, with cache markers Since Windows will never return "lpt1" in that directory listing, I would simply never find it, never even try to open it. This MIGHT be an issue with something that accepts file *uploads*, but I've been getting paranoid about those too, so, uhh... my file upload system now creates URLs that look like this: https://sikorsky.rosuav.com/static/upload-49497888-6bede802d13c8d2f7b92ca9fac7c That was uploaded as "pie.gif" but stored on the file system as ~/stillebot/httpstatic/uploads/49497888-6bede802d13c8d2f7b92ca9fac7c with some metadata stored elsewhere about the user-specified file name. So hey, if you were to try to upload a file that had an NTFS invalid character in it, I wouldn't even notice. Maybe I'm *too* paranoid, but at least I don't have to worry about file system attacks. ChrisA -- https://mail.python.org/mailman/listinfo/python-list From om+python at omajoshi.com Fri Mar 4 15:04:48 2022 From: om+python at omajoshi.com (Om Joshi) Date: Fri, 04 Mar 2022 14:04:48 -0600 Subject: Behavior of the for-else construct In-Reply-To: <1017363129.639920.1646423121331@mail.yahoo.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304083852.28142cce@bigbox.attlocal.net> <998429825.641242.1646423036979@mail.yahoo.com> <1017363129.639920.1646423121331@mail.yahoo.com> Message-ID: <17f5688eb9e.1035c37393512819.582614813492286794@omajoshi.com> I'm not sure if anyone has mentioned it on this thread, but with respect to your comment about adding either on.empty or a decorator, the Django template syntax uses {% for x in iterator %}

{{ x }}

{% empty %}

Empty

{% endfor %} and this seems to work quite well and be incredibly intuitive, at least for Django html templates. ---- On Fri, 04 Mar 2022 13:45:21 -0600 Avi Gross via Python-list wrote ---- > {NOTE, after some diversion, this long message does revert a bit to the topic.} > > Ah, Chris, the games we played when we were young and relatively immature! > > Has anyone else played with embedding "escape sequences" or other gimmicks in unexpected places like filenames so that on the right terminals, such as the VT100, it displayed oddly or was hard to open or delete as what showed was not quite the underlying representation? > > The main reason for the restrictions in olden days was cost. Everything was so costly including storage/memory and CPU time. Clearly it is a lot easier to have fixed-length filenames that fit into say 16 bytes, or storing multiple flags about file permissions as single bits, even if it meant lots of bit-twiddling or using masks to retrieve their values. We think nothing of creating structures that have many others embedded in them as attributes or function calls that allow a hundred optional arguments so that the function spends much of the time used just figuring out what was set before doing whatever calculation is required to fulfill the request. > > I was reading a novel recently (Jack Reacher Series) where the main character is noticing how much technology has changed as they have been ignoring it for a decade or so. Everything seems to be coming up faster. My view was that if something seems ten times as fast as it was, it also probably is doing a hundred or ten thousand times as much to get that result. The real speed changes are often counterbalanced by expecting to do more. A web page served may display a screen of text but to transmit it may include not just lots of padding in the HTML, but have all kinds of code such as in Java or JavaScript or lots of back and forth with the server to keep something like a graph displayed being refreshed ... > > So back to filenames, the concept of having to search for long filenames that may not even be stored sequentially in large blocks that can be read (ahead) efficiently, may have seemed to be so illogical as not to be considered. So given that the shorter ones were not allowed to have embedded spaces, it made sense to treat them like tokens that could be broken up at whitespace. As mentioned, languages (or other programs) would often parse a command line and create something like this for the main program in C with suitable version in Python and other languages: > > main(int argc, char *argv[]) > > The code variations on the above do suppose that something has already parsed the command line that invoked them and partitioned it properly into individual strings placed in an array of such strings and also returned how many arguments it saw. Users invoking the program needed to be careful such as using double quotes around anything with embedded spaces, where allowed. > > But like many paradigms, there can be a shift. Consider the fact that languages like Python are constantly parsing things like names. Can you create a variable name like "me first" with an embedded space or even other symbols normally reserved such as parentheses? Most languages do not like such things. It makes it hard to parse if not quoted in some unique way. Yet languages like R happily allow such constructs if placed in back quotes (grave accents?) as in `me & you` as a variable name or the name of a function. Of course, you then never use the darn name without the extra quotes. > > Similarly, when you make an object like a DataFrame, can you include spaces and other things in the names of columns (or sometimes rows)? If so, is there only access some ways and not others? > > The answer often is not simple. As Chris repeatedly highlights, making a language consistent as you play with features can be VERY hard and sometimes not quite possible without relaxing some rules or making exceptions. Sometimes the answer varies. In base R a data.frame can be given a column name like "me + you" which it then stores as "me...you" leading to odd results. But it happily returns that result if you ask for mydf$me using auto-completion. Spell it out fully and it won't find it! A later package added on makes modified data.frame objects called tibbles which do not autocomplete but do completely store and let you access the name so mydf$me fails and mydf$"me + you" or mydf ---- On Fri, 04 Mar 2022 13:45:21 -0600 Avi Gross via Python-list wrote ---- me + you` works but oddly an alternative format like mydf[, "me + you"] works while the similar mydf[, `me + you`] fails! > > My point is not about R but a more general one. I can rant about many other languages, LOL! Allowing spaces or other characters in what used to be a more easily viewable name that can be parsed easier, can lead to having to find every place such things are used and seeing if they can be made to work consistently. I show an example above where it is not consistent, in my view. > > But when humans view things and their perceptions differ, you are inviting disagreements about whatever you implement. You may end up having to make people do more than they would prefer such as quoting all variable names even if they do not "need" it. Wouldn't it be nice to be able to name a variable 1time at least if you felt like it? Many people have passwords like that. I think the answer is NO, not if it meant quoting every variable because there was no longer any reasonable way to parse programs. > > The issue that started the discussion was different but in a sense similar. If you want to extend the functionality of a "for" loop in one of many possible ways, how do you design a way to specify it so it can both be unambiguously parsed and implemented while at the same time making sense to humans reading it and interpreting it using their human language skills. > > I happen to think this is even harder for some who speak in languages other than English and have to program in languages loosely based on English. I am happy that I seem to think in English now but I was seven when I first encountered it after thinking in others. People who program do not all speak English or are more fluent in other languages. T may may be used to other word orders for example. They may move verbs to the end of a sentence or place adjectives or other modifiers after versus before a word and forget about all the other games played where the same word means something completely different. To them ELSE may either mean nothing or the phrase IF ... ELSE may be said differently or adding a clause after the construct is not seen as natural. > > So was this way of doing FOR ... ELSE the only or even best way, is what some of this debate is about. > > I am thinking of a function in some languages that lets you specify what should happen in a later circumstance. In a language like R, you can specify one or more instances of on.exit(...) that are not run immediately. Each one can replace the commands in the previous one or add on to it. When the function they are defined in exits for any reason, it pauses to run any uncleared such commands. Clearly this works better if a language has a way to defer evaluation of code so there are no side effects. > > So consider the suggestion of code that should be run if you have a loop and you break out of it. Could you design an alternate way to handle that other than adding an ELSE clause after the loop? > > Clearly you could simply add a function called on.break() that can be used as described but only within the body of that loop. It might be something that can be set and unset as needed and when the loop is exited, the program implicitly checks to see if any code has been dynamically set and executes it. This clearly is not necessarily a good way or better way, but is an example of how you can implement something without using any key words. No need to think about forcing the use of ELSE versus a new keyword that may conflict with existing code. Yes, the name on.break may conflict but that is trivially handled in Python by invoking it with a full name that includes what module it is in or by creating an alias. > > So what about considering an alternate approach that does handle a for loop that does nothing? Would it create huge incompatibilities for something like: > > for eye in range(0), on.empty=... : > pass > > In some languages, arbitrary additional arguments are allowed, and if not understood, are ignored. Python does not allow anything like the above. And in this case, the entire body of the for loop is never evaluated so no gimmicks inside the body are possible. A gimmick before it might work and I even wonder if there is room here for a decorator concept like: > > @on.empty(...) > for eye in range(0): > pass > > I am ending with a reminder. NOTHING I am writing here is meant to be taken seriously but merely as part of a well-intentioned debate to share ideas and not to win or lose but learn. Python is more than a language but also has aspects of a culture and we sometimes talk about whether something has a pythonic flavor or is pythonic versus translating it literally from a language like C rather than using the ideas common in python. The method chosen to implement the ELSE clause here may well be Pythonic and some of my attempts to show other ways may well not be. I am not one of those that find the current implementation to be the wrong one and will happily use it when I have code that can be done well that way. I am just discussing the issue and wider ones. Languages have an amazing variety of designs that fascinate me. > > > > -----Original Message----- > From: Chris Angelico > To: python-list at python.org > Sent: Fri, Mar 4, 2022 12:46 pm > Subject: Re: Behavior of the for-else construct > > > On Sat, 5 Mar 2022 at 02:02, Tim Chase wrote: > > > > On 2022-03-04 11:55, Chris Angelico wrote: > > > In MS-DOS, it was perfectly possible to have spaces in file names > > > > DOS didn't allow space (0x20) in filenames unless you hacked it by > > hex-editing your filesystem (which I may have done a couple times). > > However it did allow you to use 0xFF in filenames which *appeared* as > > a space in most character-sets. > > Hmm, I'm not sure which APIs worked which way, but I do believe that I > messed something up at one point and made a file with an included > space (not FF, an actual 20) in it. Maybe it's something to do with > the (ancient) FCB-based calls. It was tricky to get rid of that file, > though I think it turned out that it could be removed by globbing, > putting a question mark where the space was. > > (Of course, internally, MS-DOS considered that the base name was > padded to eight with spaces, and the extension padded to three with > spaces, so "READ.ME" would be "READ\x20\x20\x20\x20ME\x20", but that > doesn't count, since anything that enumerates the contents of a > directory would translate that into the way humans think of it.) > > > I may have caused a mild bit of consternation in school computer labs > > doing this. ;-) > > Nice :) > > > > Windows forbade a bunch of characters in file names > > > > Both DOS and Windows also had certain reserved filenames > > > > https://www.howtogeek.com/fyi/windows-10-still-wont-let-you-use-these-file-names-reserved-in-1974/ > > > > that could cause issues if passed to programs. > > Yup. All because, way back in the day, they didn't want to demand the > colon. If you actually *want* to use the printer device, for instance, > you could get a hard copy of a directory listing like this: > > DIR >LPT1: > > and it's perfectly clear that you don't want to create a file called > "LPT1", you want to send it to the printer. But noooooo it had to be > that you could just write "LPT1" and it would go to the printer. > > > To this day, if you poke around on microsoft.com and change random > > bits of URLs to include one of those reserved filenames in the GET > > path, you'll often trigger a 5xx error rather than a 404 that you > > receive with random jibberish in the same place. > > > > https://microsoft.com/?/asdfjkl ? 404 > > https://microsoft.com/?/lpt1 ? 5xx > > https://microsoft.com/?/asdfjkl/some/path ? 404 > > https://microsoft.com/?/lpt1/some/path ? 5xx > > > > Just in case you aspire to stir up some trouble. > > > > In theory, file system based URLs could be parsed such that, if you > ever hit one of those, it returns "Directory not found". In > practice... apparently they didn't do that. > > As a side point, I've been increasingly avoiding any sort of system > whereby I take anything from the user and hand it to the file system. > The logic is usually more like: > > If path matches "/static/%s": > 1) Get a full directory listing of the declared static-files directory > 2) Search that for the token given > 3) If not found, return 404 > 4) Return the contents of the file, with cache markers > > Since Windows will never return "lpt1" in that directory listing, I > would simply never find it, never even try to open it. This MIGHT be > an issue with something that accepts file *uploads*, but I've been > getting paranoid about those too, so, uhh... my file upload system now > creates URLs that look like this: > > https://sikorsky.rosuav.com/static/upload-49497888-6bede802d13c8d2f7b92ca9fac7c > > That was uploaded as "pie.gif" but stored on the file system as > ~/stillebot/httpstatic/uploads/49497888-6bede802d13c8d2f7b92ca9fac7c > with some metadata stored elsewhere about the user-specified file > name. So hey, if you were to try to upload a file that had an NTFS > invalid character in it, I wouldn't even notice. > > Maybe I'm *too* paranoid, but at least I don't have to worry about > file system attacks. > > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > > > -- > https://mail.python.org/mailman/listinfo/python-list > From 2QdxY4RzWzUUiLuE at potatochowder.com Fri Mar 4 15:32:17 2022 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Fri, 4 Mar 2022 14:32:17 -0600 Subject: Behavior of the for-else construct In-Reply-To: References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304083852.28142cce@bigbox.attlocal.net> Message-ID: On 2022-03-04 at 11:14:29 -0500, Dennis Lee Bieber wrote: > Try to tell the difference between > > afile and > afile > > when doing a directory listing. Easy: log in over a 110 baud modem, where the characters take almost as much time as the beep. ;-) From abdelkader.belahcene at enst.dz Fri Mar 4 15:48:17 2022 From: abdelkader.belahcene at enst.dz (BELAHCENE Abdelkader) Date: Fri, 4 Mar 2022 21:48:17 +0100 Subject: [python-uk] Fwd: It Will Never Work in Theory: live! In-Reply-To: References: <67cc82b7-a223-74d8-b414-bd22aa6e2629@third-bit.com> Message-ID: hi, I receive a lot of email from the python-list, I want to disable it, when I want to read the email, I want to go to he List. Please How to disable it. Regards Le jeu. 3 mars 2022 ? 18:05, Steve Holden a ?crit : > A communication from my good friend Greg Wilsin (instigator of the > Software Carpentry workshops) which may be of use to some. It certainly > looks like great value for money. > > Kind regards, > Steve > > > ---------- Forwarded message --------- > From: Greg Wilson > Date: Tue, Mar 1, 2022 at 5:11 PM > Subject: It Will Never Work in Theory: live! > To: Steve Holden > > > Hi Steve, > > On April 27, It Will Never Work in Theory is running two sets of online > lightning talks from leading software engineering researchers in which > they?ll summarize actionable findings from their work for practitioners. > Tickets are now on sale at https://neverworkintheory.org/, and all money > raised will to go Books for Africa. I hope to see you there, and if you > could help spread the word or help sponsor it by matching money raised > from ticket sales, we'd be very grateful. > > Cheers, > > Greg > > _______________________________________________ > python-uk mailing list > python-uk at python.org > https://mail.python.org/mailman/listinfo/python-uk > From avigross at verizon.net Fri Mar 4 15:52:43 2022 From: avigross at verizon.net (Avi Gross) Date: Fri, 4 Mar 2022 20:52:43 +0000 (UTC) Subject: Behavior of the for-else construct In-Reply-To: References: <21739669.459456.1646348879560@mail.yahoo.com> <25121.48093.26762.592197@ixdm.fritz.box> <1277767760.379444.1646412189311@mail.yahoo.com> Message-ID: <1957563644.11129.1646427163819@mail.yahoo.com> Chris, My example was precisely what to do when it is an empty closet: > closet = [] > > attic = ["Costumes", "Sheets", "Shirts" ] > > for item in closet: >? ? print(item) >? ? if item == "Shirts" : print("FOUND in closet!!") > else: >? ? for item in attic: >? ? ? ? print(item) >? ? ? ? if item == "Shirts" : print("FOUND in attic!!") Of course, for a more complete program, there is a reason to break out of either loop if you find what you are looking for. I was illustrating an alternate use in what I agree is a very incomplete example as it was meant only to illustrate one point. And my example may be implemented very differently if you need to search in dozens of places. Consider the ubiquitous PATH variable. Many programs search in one after another of the components for something like a filename. Doing that with a series of nested FOR loops with each one nested in the ELSE of the previous one might be a tad ridiculous! I have an observation about exception handling in general. Some people use exceptions, including ones they create and throw, for a similar idea. You might for example try to use an exception if your first attempt fails that specifies trying the next attempt. In my example of the PATH variable, you might use something like calling a function and giving it what you are looking for as well as a local copy of the PATH variable content and the exception would be to call the function again with the first component of PATH removed until you fail with an empty PATH. Yes, this is similar to just having a recursive function. But I am hoping to get across my view that there are many ways to do things but sometimes a new feature is useful because it is more direct or other reasons. in the case we are discussing, what are the scope issues in: eye = 1 for eye in ... : pass else: eye = None It looks like all references to eye refer to the same object. Some other methods we have touched on (or others may bring up) may result in different results as variables may be in other name spaces. My understanding of the PYTHON-only implementation of the above is something like this. Imagine a FOR loop that does not ignore the contents no matter what and the body looks like: if SOMETHING_IS_BEING_DONE: main code processing eye sequentially else: # only used if nothing is being done normally eye = None The above made-up scenario would allow the external ELSE to be seen as the implied else to an if that tests if the loop was actually entered. Of course it is not implemented this way, but it would show why variable eye is in the same scope in the several parts. Once you play with exceptions and other functions, this may change. The alternative suggestions about ELSE being associated confuse this issue so I want to be clear. The main meaning of the ELSE clause does not sound quite like this. It is actually about what to do with a loop that terminates normally, as I think Chris pointed out. This example finishes the loop normally and then does the ELSE: for x in range(6): print(x) else: print("Finally finished!") The following example is similar but quits early: for x in range(6): if x < 4: print(x) else: break else: print("Finally finished!") The (second) else clause is not run. So the example tossed at us looks a bit more like this and it does run the ELSE not because the loop is not done but because the loop never calls a break: for x in range(0): print(x) else: print("Finally finished!") So unless I am missing something, the code would destroy the last value of the variable even in a valid case when it looks like: for aye in range(0): pass else: aye = None I mean if you used range(5) you would expect the value of aye==4 after the loop. The above loop would leave it as aye==None which keeps it from being undefined. To decide if the loop ran at all would thus require further code such as: if aye == None: ... Which leads me right back to wondering why the sentinel approach is so bad! -----Original Message----- From: Chris Angelico To: python-list at python.org Sent: Fri, Mar 4, 2022 12:47 pm Subject: Re: Behavior of the for-else construct On Sat, 5 Mar 2022 at 03:44, Avi Gross via Python-list wrote: > > Dieter, > > Your use is creative albeit it is not "needed" since all it does is make sure your variable is initialized to something, specifically None. > > So would this not do the same thing? > >? eye = None > >? for eye in range(0): >? ? ? print(eye) > >? eye > > If I understand it, your example depends on a variable that is not yet initialized to be used in a loop and retain the last value after the loop. You then set it to None if it is not used as the loop is skipped. Others have shown an example similar to the above of using a sentinel that lets you know if the loop is skipped. > > Of course, there are some advantages in making it clear by doing it you way that the loop (for example if copied and used elsewhere) needs to include the else statement as an integral part. > > I would like to suggest a slight modification to the above as in if you are searching for something in either seq1 and if not found in seq2. Call it looking for your green shirt in the closet and if not found, looking in the attic. Would this code make sense as such a use in several ways? In English, look here first and if there is NOTHING there, look in the second place? > > closet = [] > > attic = ["Costumes", "Sheets", "Shirts" ] > > for item in closet: >? ? print(item) >? ? if item == "Shirts" : print("FOUND in closet!!") > else: >? ? for item in attic: >? ? ? ? print(item) >? ? ? ? if item == "Shirts" : print("FOUND in attic!!") > > Yes, as discussed, you could do an IF statement to check if closet is empty but for iterators, it gets ... > Make sure you 'break' after finding it. Otherwise, you'll keep searching the rest of your closet, and then still go on to search your attic. The "else:" clause doesn't help you here unless that break is present. ChrisA -- https://mail.python.org/mailman/listinfo/python-list From rosuav at gmail.com Fri Mar 4 16:01:36 2022 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 5 Mar 2022 08:01:36 +1100 Subject: Behavior of the for-else construct In-Reply-To: <1957563644.11129.1646427163819@mail.yahoo.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <25121.48093.26762.592197@ixdm.fritz.box> <1277767760.379444.1646412189311@mail.yahoo.com> <1957563644.11129.1646427163819@mail.yahoo.com> Message-ID: On Sat, 5 Mar 2022 at 07:52, Avi Gross wrote: > > Chris, > > My example was precisely what to do when it is an empty closet: Does it correctly handle a closet with shirts in it, though? There's not a lot of point demonstrating an alternate use for the "else" clause when it is *absolutely identical* to simply placing two loops one after another. ChrisA From michael.stemper at gmail.com Fri Mar 4 15:54:12 2022 From: michael.stemper at gmail.com (Michael F. Stemper) Date: Fri, 4 Mar 2022 14:54:12 -0600 Subject: Behavior of the for-else construct In-Reply-To: References: <21739669.459456.1646348879560@mail.yahoo.com> <46ef1b7f-5689-0cd5-ce89-be4c453d262f@btinternet.com> <08edeac4-1d59-ce5c-663b-42d75e6d8da4@btinternet.com> Message-ID: On 03/03/2022 19.54, Rob Cliffe wrote: > On 04/03/2022 01:44, Ethan Furman wrote: >> On 3/3/22 5:32 PM, Rob Cliffe via Python-list wrote: >> >> > There are three types of programmer: those that can count, and those that can't. >> >> Actually, there are 10 types of programmer:? those that can count in binary, and those that can't. > 1, 10, many. > No problem. Ah, a George Gamow fan. -- Michael F. Stemper Psalm 82:3-4 From python at mrabarnett.plus.com Fri Mar 4 16:16:22 2022 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 4 Mar 2022 21:16:22 +0000 Subject: [python-uk] Fwd: It Will Never Work in Theory: live! In-Reply-To: References: <67cc82b7-a223-74d8-b414-bd22aa6e2629@third-bit.com> Message-ID: On 2022-03-04 20:48, BELAHCENE Abdelkader wrote: > hi, > I receive a lot of email from the python-list, I want to disable it, when I > want to read the email, I want to go to he List. > Please How to disable it. > Regards > This page explains how to unsubscribe: https://mail.python.org/mailman/listinfo/python-list > Le jeu. 3 mars 2022 ? 18:05, Steve Holden a ?crit : > >> A communication from my good friend Greg Wilsin (instigator of the >> Software Carpentry workshops) which may be of use to some. It certainly >> looks like great value for money. >> >> Kind regards, >> Steve >> >> >> ---------- Forwarded message --------- >> From: Greg Wilson >> Date: Tue, Mar 1, 2022 at 5:11 PM >> Subject: It Will Never Work in Theory: live! >> To: Steve Holden >> >> >> Hi Steve, >> >> On April 27, It Will Never Work in Theory is running two sets of online >> lightning talks from leading software engineering researchers in which >> they?ll summarize actionable findings from their work for practitioners. >> Tickets are now on sale at https://neverworkintheory.org/, and all money >> raised will to go Books for Africa. I hope to see you there, and if you >> could help spread the word or help sponsor it by matching money raised >> from ticket sales, we'd be very grateful. >> >> Cheers, >> >> Greg >> >> _______________________________________________ >> python-uk mailing list >> python-uk at python.org >> https://mail.python.org/mailman/listinfo/python-uk >> From hjp-python at hjp.at Fri Mar 4 16:23:07 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Fri, 4 Mar 2022 22:23:07 +0100 Subject: Getting Syslog working on OSX Monterey In-Reply-To: <4EAF5AC0-446D-4493-AFF0-2FD8A8889439@barrys-emacs.org> References: <0D40B574-2975-49BD-9E97-9055317169E5@barrys-emacs.org> <20220228214109.ntoihdobzf5naqrx@hjp.at> <4EAF5AC0-446D-4493-AFF0-2FD8A8889439@barrys-emacs.org> Message-ID: <20220304212307.bd3rj67m6gaej37a@hjp.at> On 2022-02-28 22:05:05 +0000, Barry Scott wrote: > On 28 Feb 2022, at 21:41, Peter J. Holzer wrote: > > On 2022-02-27 22:16:54 +0000, Barry wrote: > >> I have always assumed that if I want a logger syslog handler that I would have > >> to implement it myself. So far I have code that uses syslog directly and have > >> not written that code yet. > > > > What do you mean by using syslog directly? The syslog(3) library > > function also just sends messages to a "syslog listener" (more commonly > > called a syslog daemon) - at least on any unix-like system I'm familiar > > with (which doesn't include MacOS). It will, however, always use the > > *local* syslog daemon - AFAIK there is no standard way to open a remote > > connection (many syslog daemons can be configured to forward messages to > > a remote server, however). > > I'm re-reading the code to check on what I'm seeing. (Its been a long > time since I last look deeply at this code). > > You can write to /dev/log if you pass that to > SysLogHandler(address='/dev/log'), but the default is to use a socket > to talk to a network listener on localhost:514. There are no deamons > listening on port 514 on my Fedora systems or mac OS. If you are saying that SysLogHandler should use a system specific default (e.g. "/dev/log" on Linux) instead of UDP port 514 everywhere, I agree 99 % (the remaining 1 % is my contrarian alter ego arguing that that's really the distribution maintainer's job since a Linux distribution might use some other socket). If you are saying it should use the libc syslog routines, I disagree for at least two reasons: a) they are OS specific, b) you can't configure the destination. So that would remove useful functionality. In any case it seems strange to me that you want to rewrite it just to avoid passing a single parameter to the constructor (or - more likely - adding a single line to a config file). > What you do not see used in the SyslogHandler() is the import syslog > and hence its nor using openlog() etc from syslog API. For good reasons. The C syslog API is missing important functionality. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From avigross at verizon.net Fri Mar 4 16:24:05 2022 From: avigross at verizon.net (Avi Gross) Date: Fri, 4 Mar 2022 21:24:05 +0000 (UTC) Subject: Behavior of the for-else construct In-Reply-To: <17f5688eb9e.1035c37393512819.582614813492286794@omajoshi.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304083852.28142cce@bigbox.attlocal.net> <998429825.641242.1646423036979@mail.yahoo.com> <1017363129.639920.1646423121331@mail.yahoo.com> <17f5688eb9e.1035c37393512819.582614813492286794@omajoshi.com> Message-ID: <1992827181.17731.1646429045301@mail.yahoo.com> Om (unless your first name is Joshi), Yes, your example captures some of my intent. I have not studied Django but your example suggests it uses special notation using a sort of set of braces like I have talked about "{%" and "%}" that allows freedom in using what might otherwise need keywords. Anything between braces can be an extension of the language in a context where users do not put their own variable names. But again, any language defined can be set up to do things their own way. In Python, I believe the contents of a loop are not examined at all if skipped so searching in it for some clause of what to do if empty is not doable. The ELSE clause is a way to get the attention if there has been no break. I would not be shocked if some language has a FOR command return a value such as the number of iterations if run and something like None if not and allows something like: result = for ... That might return 0 or None if it was part of the language but it is not. Avi (my current first name) -----Original Message----- From: Om Joshi To: Avi Gross Cc: python-list Sent: Fri, Mar 4, 2022 3:04 pm Subject: Re: Behavior of the for-else construct I'm not sure if anyone has mentioned it on this thread, but with respect to your comment about adding either on.empty or a decorator, the Django template syntax uses {% for x in iterator %}

{{ x }}

{% empty %}

Empty

{% endfor %} and this seems to work quite well and be incredibly intuitive, at least for Django html templates. I took a look and it has other enhancements like a way to know when you are in the last iteration of the loop. {% if forloop.last %} ... {% endif %} And also for the first item using forloop.first and a generalized counter in forloop.counter .. ---- On Fri, 04 Mar 2022 13:45:21 -0600 Avi Gross via Python-list wrote ---- > {NOTE, after some diversion, this long message does revert a bit to the topic.} > > Ah, Chris, the games we played when we were young and relatively immature! > > Has anyone else played with embedding "escape sequences" or other gimmicks in unexpected places like filenames so that on the right terminals, such as the VT100, it displayed oddly or was hard to open or delete as what showed was not quite the underlying representation? > > The main reason for the restrictions in olden days was cost. Everything was so costly including storage/memory and CPU time. Clearly it is a lot easier to have fixed-length filenames that fit into say 16 bytes, or storing multiple flags about file permissions as single bits, even if it meant lots of bit-twiddling or using masks to retrieve their values. We think nothing of creating structures that have many others embedded in them as attributes or function calls that allow a hundred optional arguments so that the function spends much of the time used just figuring out what was set before doing whatever calculation is required to fulfill the request. > > I was reading a novel recently (Jack Reacher Series) where the main character is noticing how much technology has changed as they have been ignoring it for a decade or so. Everything seems to be coming up faster. My view was that if something seems ten times as fast as it was, it also probably is doing a hundred or ten thousand times as much to get that result.? The real speed changes are often counterbalanced by expecting to do more. A web page served may display a screen of text but to transmit it may include not just lots of padding in the HTML, but have all kinds of code such as in Java or JavaScript or lots of back and forth with the server to keep something like a graph displayed being refreshed ... > > So back to filenames, the concept of having to search for long filenames that may not even be stored sequentially in large blocks that can be read (ahead) efficiently, may have seemed to be so illogical as not to be considered. So given that the shorter ones were not allowed to have embedded spaces, it made sense to treat them like tokens that could be broken up at whitespace. As mentioned, languages (or other programs) would often parse a command line and create something like this for the main program in C with suitable version in Python and other languages: > >? ? main(int argc, char *argv[]) > > The code variations on the above do suppose that something has already parsed the command line that invoked them and partitioned it properly into individual strings placed in an array of such strings and also returned how many arguments it saw. Users invoking the program needed to be careful such as using double quotes around anything with embedded spaces, where allowed. > > But like many paradigms, there can be a shift. Consider the fact that languages like Python are constantly parsing things like names. Can you create a variable name like "me first" with an embedded space or even other symbols normally reserved such as parentheses? Most languages do not like such things. It makes it hard to parse if not quoted in some unique way. Yet languages like R happily allow such constructs if placed in back quotes (grave accents?) as in `me & you` as a variable name or the name of a function. Of course, you then never use the darn name without the extra quotes. > > Similarly, when you make an object like a DataFrame, can you include spaces and other things in the names of columns (or sometimes rows)? If so, is there only access some ways and not others? > > The answer often is not simple. As Chris repeatedly highlights, making a language consistent as you play with features can be VERY hard and sometimes not quite possible without relaxing some rules or making exceptions. Sometimes the answer varies. In base R a data.frame can be given a column name like "me + you" which it then stores as "me...you" leading to odd results. But it happily returns that result if you ask for mydf$me using auto-completion. Spell it out fully and it won't find it! A later package added on makes modified data.frame objects called tibbles which do not autocomplete but do completely store and let you access the name so mydf$me fails and mydf$"me + you" or mydf ---- On Fri, 04 Mar 2022 13:45:21 -0600 Avi Gross via Python-list wrote ---- me + you` works but oddly an alternative format like mydf[, "me + you"] works while the similar mydf[, `me + you`] fails! > > My point is not about R but a more general one. I can rant about many other languages, LOL! Allowing spaces or other characters in what used to be a more easily viewable name that can be parsed easier, can lead to having to find every place such things are used and seeing if they can be made to work consistently. I show an example above where it is not consistent, in my view. > > But when humans view things and their perceptions differ, you are inviting disagreements about whatever you implement. You may end up having to make people do more than they would prefer such as quoting all variable names even if they do not "need" it. Wouldn't it be nice to be able to name a variable 1time at least if you felt like it? Many people have passwords like that. I think the answer is NO, not if it meant quoting every variable because there was no longer any reasonable way to parse programs. > > The issue that started the discussion was different but in a sense similar. If you want to extend the functionality of a "for" loop in one of many possible ways, how do you design a way to specify it so it can both be unambiguously parsed and implemented while at the same time making sense to humans reading it and interpreting it using their human language skills. > > I happen to think this is even harder for some who speak in languages other than English and have to program in languages loosely based on English. I am happy that I seem to think in English now but I was seven when I first encountered it after thinking in others. People who program do not all speak English or are more fluent in other languages. T may may be used to other word orders for example. They may move verbs to the end of a sentence? or place adjectives or other modifiers after versus before a word and forget about all the other games played where the same word means something completely different. To them ELSE may either mean nothing or the phrase IF ... ELSE may be said differently or adding a clause after the construct is not seen as natural. > > So was this way of doing FOR ... ELSE the only or even best way, is what some of this debate is about. > > I am thinking of a function in some languages that lets you specify what should happen in a later circumstance. In a language like R, you can specify one or more instances of on.exit(...) that are not run immediately. Each one can replace the commands in the previous one or add on to it. When the function they are defined in exits for any reason, it pauses to run any uncleared such commands. Clearly this works better if a language has a way to defer evaluation of code so there are no side effects. > > So consider the suggestion of code that should be run if you have a loop and you break out of it. Could you design an alternate way to handle that other than adding an ELSE clause after the loop? > > Clearly you could simply add a function called on.break() that can be used as described but only within the body of that loop. It might be something that can be set and unset as needed and when the loop is exited, the program implicitly checks to see if any code has been dynamically set and executes it. This clearly is not necessarily a good way or better way, but is an example of how you can implement something without using any key words. No need to think about forcing the use of ELSE versus a new keyword that may conflict with existing code. Yes, the name on.break may conflict but that is trivially handled in Python by invoking it with a full name that includes what module it is in or by creating an alias. >? > So what about considering an alternate approach that does handle a for loop that does nothing? Would it create huge incompatibilities for something like: > > for eye in range(0), on.empty=... : >? ? pass > > In some languages, arbitrary additional arguments are allowed, and if not understood, are ignored. Python does not allow anything like the above. And in this case, the entire body of the for loop is never evaluated so no gimmicks inside the body are possible. A gimmick before it might work and I even wonder if there is room here for a decorator concept like: > > @on.empty(...) > for eye in range(0): >? ? pass > > I am ending with a reminder. NOTHING I am writing here is meant to be taken seriously but merely as part of a well-intentioned debate to share ideas and not to win or lose but learn. Python is more than a language but also has aspects of a culture and we sometimes talk about whether something has a pythonic flavor or is pythonic versus translating it literally from a language like C rather than using the ideas common in python. The method chosen to implement the ELSE clause here may well be Pythonic and some of my attempts to show other ways may well not be. I am not one of those that find the current implementation to be the wrong one and will happily use it when I have code that can be done well that way. I am just discussing the issue and wider ones. Languages have an amazing variety of designs that fascinate me. > > > > -----Original Message----- > From: Chris Angelico > To: python-list at python.org > Sent: Fri, Mar 4, 2022 12:46 pm > Subject: Re: Behavior of the for-else construct > > >? On Sat, 5 Mar 2022 at 02:02, Tim Chase wrote: > > > > On 2022-03-04 11:55, Chris Angelico wrote: > > > In MS-DOS, it was perfectly possible to have spaces in file names > > > > DOS didn't allow space (0x20) in filenames unless you hacked it by > > hex-editing your filesystem (which I may have done a couple times). > > However it did allow you to use 0xFF in filenames which *appeared* as > > a space in most character-sets. > > Hmm, I'm not sure which APIs worked which way, but I do believe that I > messed something up at one point and made a file with an included > space (not FF, an actual 20) in it. Maybe it's something to do with > the (ancient) FCB-based calls. It was tricky to get rid of that file, > though I think it turned out that it could be removed by globbing, > putting a question mark where the space was. > > (Of course, internally, MS-DOS considered that the base name was > padded to eight with spaces, and the extension padded to three with > spaces, so "READ.ME" would be "READ\x20\x20\x20\x20ME\x20", but that > doesn't count, since anything that enumerates the contents of a > directory would translate that into the way humans think of it.) > > > I may have caused a mild bit of consternation in school computer labs > > doing this. ;-) > > Nice :) > > > > Windows forbade a bunch of characters in file names > > > > Both DOS and Windows also had certain reserved filenames > > > > https://www.howtogeek.com/fyi/windows-10-still-wont-let-you-use-these-file-names-reserved-in-1974/ > > > > that could cause issues if passed to programs. > > Yup. All because, way back in the day, they didn't want to demand the > colon. If you actually *want* to use the printer device, for instance, > you could get a hard copy of a directory listing like this: > > DIR >LPT1: > > and it's perfectly clear that you don't want to create a file called > "LPT1", you want to send it to the printer. But noooooo it had to be > that you could just write "LPT1" and it would go to the printer. > > > To this day, if you poke around on microsoft.com and change random > > bits of URLs to include one of those reserved filenames in the GET > > path, you'll often trigger a 5xx error rather than a 404 that you > > receive with random jibberish in the same place. > > > >? https://microsoft.com/?/asdfjkl ? 404 > >? https://microsoft.com/?/lpt1 ? 5xx > >? https://microsoft.com/?/asdfjkl/some/path ? 404 > >? https://microsoft.com/?/lpt1/some/path ? 5xx > > > > Just in case you aspire to stir up some trouble. > > > > In theory, file system based URLs could be parsed such that, if you > ever hit one of those, it returns "Directory not found". In > practice... apparently they didn't do that. > > As a side point, I've been increasingly avoiding any sort of system > whereby I take anything from the user and hand it to the file system. > The logic is usually more like: > > If path matches "/static/%s": > 1) Get a full directory listing of the declared static-files directory > 2) Search that for the token given > 3) If not found, return 404 > 4) Return the contents of the file, with cache markers > > Since Windows will never return "lpt1" in that directory listing, I > would simply never find it, never even try to open it. This MIGHT be > an issue with something that accepts file *uploads*, but I've been > getting paranoid about those too, so, uhh... my file upload system now > creates URLs that look like this: > > https://sikorsky.rosuav.com/static/upload-49497888-6bede802d13c8d2f7b92ca9fac7c > > That was uploaded as "pie.gif" but stored on the file system as > ~/stillebot/httpstatic/uploads/49497888-6bede802d13c8d2f7b92ca9fac7c > with some metadata stored elsewhere about the user-specified file > name. So hey, if you were to try to upload a file that had an NTFS > invalid character in it, I wouldn't even notice. > > Maybe I'm *too* paranoid, but at least I don't have to worry about > file system attacks. > > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > > > -- > https://mail.python.org/mailman/listinfo/python-list > From rob.cliffe at btinternet.com Fri Mar 4 17:22:30 2022 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Fri, 4 Mar 2022 22:22:30 +0000 Subject: Behavior of the for-else construct In-Reply-To: <1957563644.11129.1646427163819@mail.yahoo.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <25121.48093.26762.592197@ixdm.fritz.box> <1277767760.379444.1646412189311@mail.yahoo.com> <1957563644.11129.1646427163819@mail.yahoo.com> Message-ID: On 04/03/2022 20:52, Avi Gross via Python-list wrote: > > I have an observation about exception handling in general. Some people use exceptions, including ones they create and throw, for a similar idea. You might for example try to use an exception if your first attempt fails that specifies trying the next attempt. In my example of the PATH variable, you might use something like calling a function and giving it what you are looking for as well as a local copy of the PATH variable content and the exception would be to call the function again with the first component of PATH removed until you fail with an empty PATH. Yes, this is similar to just having a recursive function. That sounds neither readable nor efficient compared with using split() plus a loop.? Maybe you mean this to be a toy, unrealistic example? > So the example tossed at us looks a bit more like this and it does run the ELSE not because the loop is not done but because the loop never calls a break: > > for x in range(0): > print(x) > else: > print("Finally finished!") This would be more readable with a `zeroiterations` keyword, which accurately describes both the semantics and the intent. > Which leads me right back to wondering why the sentinel approach is so bad! > > It's not that bad, but it's more convenient and readable if it can be avoided. Best wishes Rob Cliffe From hjp-python at hjp.at Fri Mar 4 17:57:46 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Fri, 4 Mar 2022 23:57:46 +0100 Subject: Behavior of the for-else construct In-Reply-To: <621325684.471007.1646354302946@mail.yahoo.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> Message-ID: <20220304225746.mmebv3myg5wbpc4f@hjp.at> On 2022-03-04 00:38:22 +0000, Avi Gross via Python-list wrote: > I have seen major struggles to get other character sets into > languages. Any new language typically should have this built in from > scratch and should consider adding non-ASCII characters into the mix. > Mathematicians often use lots of weird braces/brackets as an example > while normal programs are limited to [{( and maybe < and their > counterparts. This leads to odd Python behavior (other languages too) > where symbols are re-used ad nauseam. { can mean set or dictionary or > simply some other way to group code. I think the main reason for this is that people have to be able to type the code. I do remember the days when German keyboards didn't have brackets and braces and you had to type alt-123 to get a {. Made it real fun to program in C ... If you design a language and in IDE together (the way Smalltalk was designed back in the 1970s or maybe Visual Basic) you can use all sorts of funky characters because you can also provide a standard way to enter them. But if you expect your users to type programs in a standard text editor (even a fancy one like Emacs or Vim or VS Code), you have to restrict yourself to a character set that most people can comfortably type on their keyboards with the key mapping provided by their OS. Which for historical reasons means US-ASCII. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From hjp-python at hjp.at Fri Mar 4 18:03:52 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sat, 5 Mar 2022 00:03:52 +0100 Subject: Behavior of the for-else construct In-Reply-To: <20220304083852.28142cce@bigbox.attlocal.net> References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304083852.28142cce@bigbox.attlocal.net> Message-ID: <20220304230352.vljx2rf7lcyhup25@hjp.at> On 2022-03-04 08:38:52 -0600, Tim Chase wrote: > On 2022-03-04 11:55, Chris Angelico wrote: > > In MS-DOS, it was perfectly possible to have spaces in file names > > DOS didn't allow space (0x20) in filenames unless you hacked it by > hex-editing your filesystem (which I may have done a couple times). > However it did allow you to use 0xFF in filenames which *appeared* as > a space in most character-sets. I may be misremembering (it's been 30+ years), but I seem to remember that a simple fopen("with space.txt", "w") in Turbo C would create a file with a space (actually called "WITH SPA.TXT", because upper case only and 8+3). It was a bad idea to do this, though, because there was no way to manipulate such a file from command.com (You'd have to write another C program). hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From hjp-python at hjp.at Fri Mar 4 18:11:50 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sat, 5 Mar 2022 00:11:50 +0100 Subject: Behavior of the for-else construct In-Reply-To: <17f5688eb9e.1035c37393512819.582614813492286794@omajoshi.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304083852.28142cce@bigbox.attlocal.net> <998429825.641242.1646423036979@mail.yahoo.com> <1017363129.639920.1646423121331@mail.yahoo.com> <17f5688eb9e.1035c37393512819.582614813492286794@omajoshi.com> Message-ID: <20220304231150.mencyhgmfpzujqdi@hjp.at> On 2022-03-04 14:04:48 -0600, Om Joshi wrote: > I'm not sure if anyone has mentioned it on this thread, but with > respect to your comment about adding either on.empty or a decorator, > the Django template syntax uses > > {% for x in iterator %} >

{{ x }}

> {% empty %} >

Empty

> {% endfor %} > > and this seems to work quite well and be incredibly intuitive, at > least for Django html templates. OTOH it is frequently not what you want. Take this example from the Django docs:
    {% for athlete in athlete_list %}
  • {{ athlete.name }}
    • {% empty %}
  • Sorry, no athletes in this list.
    • {% endfor %}
If athlete_list is empty, it will produce:
  • Sorry, no athletes in this list.
which is awful typography. You don't want a list with a single item, you want that text *instead of* the list. So you would have to write {% if athlete_list %}
    {% for athlete in athlete_list %}
  • {{ athlete.name }}
    • {% empty %}
{% else %}
Sorry, no athletes in this list.
{%endif %} anyway. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From avigross at verizon.net Fri Mar 4 18:23:54 2022 From: avigross at verizon.net (Avi Gross) Date: Fri, 4 Mar 2022 23:23:54 +0000 (UTC) Subject: Behavior of the for-else construct In-Reply-To: References: <21739669.459456.1646348879560@mail.yahoo.com> <25121.48093.26762.592197@ixdm.fritz.box> <1277767760.379444.1646412189311@mail.yahoo.com> <1957563644.11129.1646427163819@mail.yahoo.com> Message-ID: <1359166284.39320.1646436234935@mail.yahoo.com> Yes, Rob, I keep wondering how many decades it will take before people here understand how I make a point with an example that is not intended in any way to be complete or the way I would do it. No, I would not normally use the exception method, let alone in this case. This one practically screams for the method you suggest and iterate on a list of directories made from splitting the PATH variable. And, yes that method would likely use a for loop with a break if it is found and is a candidate for using the ELSE clause to note it has not been found. But it is not an unusual idiom these days to use exceptions in places I never dreamed of in my earlier programming languages. A common example is always testing before accidentally dividing by zero and now code that depends on throwing the right exception and dealing with it, especially if the exception happens rarely. So in situations where almost all programs are found in the first component of PATH, perhaps the exception approach is not horrible. It may even at times be done with less coding effort than other alternatives. Not my preferred way, of course. -----Original Message----- From: Rob Cliffe via Python-list To: python-list at python.org Sent: Fri, Mar 4, 2022 5:22 pm Subject: Re: Behavior of the for-else construct On 04/03/2022 20:52, Avi Gross via Python-list wrote: > > I have an observation about exception handling in general. Some people use exceptions, including ones they create and throw, for a similar idea. You might for example try to use an exception if your first attempt fails that specifies trying the next attempt. In my example of the PATH variable, you might use something like calling a function and giving it what you are looking for as well as a local copy of the PATH variable content and the exception would be to call the function again with the first component of PATH removed until you fail with an empty PATH. Yes, this is similar to just having a recursive function. That sounds neither readable nor efficient compared with using split() plus a loop.? Maybe you mean this to be a toy, unrealistic example? > So the example tossed at us looks a bit more like this and it does run the ELSE not because the loop is not done but because? the loop never calls a break: > > for x in range(0): >? ? print(x) > else: >? ? print("Finally finished!") This would be more readable with a `zeroiterations` keyword, which accurately describes both the semantics and the intent. > Which leads me right back to wondering why the sentinel approach is so bad! > > It's not that bad, but it's more convenient and readable if it can be avoided. Best wishes Rob Cliffe -- https://mail.python.org/mailman/listinfo/python-list From hjp-python at hjp.at Fri Mar 4 18:25:44 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sat, 5 Mar 2022 00:25:44 +0100 Subject: Behavior of the for-else construct In-Reply-To: References: <21739669.459456.1646348879560@mail.yahoo.com> Message-ID: <20220304232544.mcbhet7wn2wa2k2r@hjp.at> On 2022-03-04 11:34:07 +1100, Chris Angelico wrote: > On Fri, 4 Mar 2022 at 10:09, Avi Gross via Python-list > wrote: > > The drumbeat I keep hearing is that some people hear/see the same > > word as implying something else. ELSE is ambiguous in the context it > > is used. > > What I'm hearing is that there are, broadly speaking, two types of > programmers [1]: > > 1) Those who think about "for-else" as a search tool and perfectly > understand how it behaves > 2) Those who have an incorrect idea about what for-else is supposed to > do, don't understand it, and don't like it. 3) Those who understand what it does und don't even find it*s syntax very confusing, yet only very rarely find it useful. I have probably used it a handful of times in 8 years of Python programming. Coincidentally, I think I used it just this week - but I can't find it any more which probably means that it was either in a throwaway script or I have since rewritten the code. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From rosuav at gmail.com Fri Mar 4 18:31:20 2022 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 5 Mar 2022 10:31:20 +1100 Subject: Behavior of the for-else construct In-Reply-To: <20220304232544.mcbhet7wn2wa2k2r@hjp.at> References: <21739669.459456.1646348879560@mail.yahoo.com> <20220304232544.mcbhet7wn2wa2k2r@hjp.at> Message-ID: On Sat, 5 Mar 2022 at 10:28, Peter J. Holzer wrote: > > On 2022-03-04 11:34:07 +1100, Chris Angelico wrote: > > On Fri, 4 Mar 2022 at 10:09, Avi Gross via Python-list > > wrote: > > > The drumbeat I keep hearing is that some people hear/see the same > > > word as implying something else. ELSE is ambiguous in the context it > > > is used. > > > > What I'm hearing is that there are, broadly speaking, two types of > > programmers [1]: > > > > 1) Those who think about "for-else" as a search tool and perfectly > > understand how it behaves > > 2) Those who have an incorrect idea about what for-else is supposed to > > do, don't understand it, and don't like it. > > 3) Those who understand what it does und don't even find it*s syntax > very confusing, yet only very rarely find it useful. > > I have probably used it a handful of times in 8 years of Python > programming. Coincidentally, I think I used it just this week - but I > can't find it any more which probably means that it was either in a > throwaway script or I have since rewritten the code. > Part of group 1. It doesn't matter how often or rarely you use the feature; there are a LOT of features of Python that I use only occasionally, if ever. But as long as you know how to use it, it's there when you need it. ChrisA From avigross at verizon.net Fri Mar 4 18:47:09 2022 From: avigross at verizon.net (Avi Gross) Date: Fri, 4 Mar 2022 23:47:09 +0000 (UTC) Subject: Behavior of the for-else construct In-Reply-To: <20220304225746.mmebv3myg5wbpc4f@hjp.at> References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304225746.mmebv3myg5wbpc4f@hjp.at> Message-ID: <657845041.42944.1646437629325@mail.yahoo.com> I am not sure a reply is needed, Peter, and what you say is true. But as you point out, when using a German keyboard, I would already have a way to enter symbols like ?, ?, ? and ? and no reason not to include them in variable names and so on if UNICODE is being used properly. I can use my last name in German notation as a variable in Python now: Gro? = 144 Gro? / 12 12.0 But even if I was German (and despite living in Austria for a year, I am not) I might want to write some things in French which might not be on that Keyboard, unless it includes everything in ISO 8859-1, albeit that standard is missing letters used rarely in some languages. I have also used characters from Spanish and French and Hungarian and others as names in programs with no problems, albeit getting them in is not trivial on my very solid American English keyboard but can be done many ways. A fairly simple way is to use something like this using Google translate: https://translate.google.com/?sl=de&tl=en&op=translate Pick the language you want and pop up a keyboard and type a bit and then copy/paste like this: ???? If I had to do this all the time, and occasionally I do, you can get other pop-up keyboards or helpers that take several keystrokes you memorize and convert it, ... But it rarely is worth doing this if others will be using my code and complaining or trying to edit it. Chris has already pointed out the dangers of wandering too far from standard practice and making code hard for anyone else to deal with so yes, I would not rewrite key components or do something like make new function names that point to existing standard functions and only use the new (often not pronounceable) names. -----Original Message----- From: Peter J. Holzer To: python-list at python.org Sent: Fri, Mar 4, 2022 5:57 pm Subject: Re: Behavior of the for-else construct On 2022-03-04 00:38:22 +0000, Avi Gross via Python-list wrote: > I have seen major struggles to get other character sets into > languages. Any new language typically should have this built in from > scratch and should consider adding non-ASCII characters into the mix. > Mathematicians often use lots of weird braces/brackets as an example > while normal programs are limited to [{( and maybe < and their > counterparts. This leads to odd Python behavior (other languages too) > where symbols are re-used ad nauseam. { can mean set or dictionary or > simply some other way to group code. I think the main reason for this is that people have to be able to type the code. I do remember the days when German keyboards didn't have brackets and braces and you had to type alt-123 to get a {. Made it real fun to program in C ... If you design a language and in IDE together (the way Smalltalk was designed back in the 1970s or maybe Visual Basic) you can use all sorts of funky characters because you can also provide a standard way to enter them. But if you expect your users to type programs in a standard text editor (even a fancy one like Emacs or Vim or VS Code), you have to restrict yourself to a character set that most people can comfortably type on their keyboards with the key mapping provided by their OS. Which for historical reasons means US-ASCII. ? ? ? ? hp -- ? _? | Peter J. Holzer? ? | Story must make more sense than reality. |_|_) |? ? ? ? ? ? ? ? ? ? | | |? | hjp at hjp.at? ? ? ? |? ? -- Charles Stross, "Creative writing __/? | http://www.hjp.at/ |? ? ? challenge!" -- https://mail.python.org/mailman/listinfo/python-list From hjp-python at hjp.at Fri Mar 4 19:11:58 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sat, 5 Mar 2022 01:11:58 +0100 Subject: Behavior of the for-else construct In-Reply-To: <657845041.42944.1646437629325@mail.yahoo.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304225746.mmebv3myg5wbpc4f@hjp.at> <657845041.42944.1646437629325@mail.yahoo.com> Message-ID: <20220305001158.g7rmlyoxxtxufpfp@hjp.at> On 2022-03-04 23:47:09 +0000, Avi Gross via Python-list wrote: > I am not sure a reply is needed, Peter, and what you say is true. But > as you point out, when using a German keyboard, I would already have > a way to enter symbols like ?, ?, ? and ? and no reason not to include > them in variable names and so on if UNICODE is being used properly. I > can use my last name in German notation as a variable in Python now: > > Gro? = 144 > Gro? / 12 > 12.0 Yes, I'm using umlauts occasionally in variable names in Python, and I've also used Greek characters and others. But in Python I CAN use them. I DON'T HAVE to. That's a big difference. Characters like [] or {} are a part of Python's syntax. You can't avoid using them. If you can't type them, you can't write Python. If it is awkward to enter them (like having to type Alt-91 or pasting them from a character table) it is painful to write programs. German keyboards aquired an AltGr key and the ability to type these characters in the mid to late 1980's. Presumably because those characters were common in C and other programming languages and programmers were complaining. I assume the same happened with keyboards for other languages. These days you can assume that everybody can type all ASCII characters (and knows how to do it). But if you add arbitrary unicode characters to the syntax of your language, for example using ?? to delimit code blocks and ? ? for sets and ? for None, then every programmer will have to figure out how to enter those characters. And 90 % will probably say "Fuggedaboutit, I'm not going to learn a new programming language I can't even type!" hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From cs at cskk.id.au Fri Mar 4 20:15:28 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Sat, 5 Mar 2022 12:15:28 +1100 Subject: Behavior of the for-else construct In-Reply-To: References: Message-ID: On 03Mar2022 14:24, computermaster360 wrote: >I want to make a little survey here. > >Do you find the for-else construct useful? Have you used it in >practice? Do you even know how it works, or that there is such a thing >in Python? I used Python for years before understanding the for-else (and therefore, without seeing a use for it). Finally made the time to wrap my head around it quite recently and have used it a few times since. It is a little niche, but when you want it, it is very nice indeed. Certainly avoids managing a special flag variable (which can be fiddly and therefore fragile) for "did my loop not run to completion"? I sort of wish it had both "used break" and "did not use break" branches, a bit like try/except/else. But not very much. Cheers, Cameron Simpson From wlfraed at ix.netcom.com Fri Mar 4 16:44:57 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Fri, 04 Mar 2022 16:44:57 -0500 Subject: How to solve the given problem? References: <1323924283.239248.1644533825969@mail.yahoo.com> <028b56df-5ff5-4fd7-9205-eea6bdc569d7n@googlegroups.com> <1d8c2edf-6d36-4af5-81d5-b60d74a951a3n@googlegroups.com> Message-ID: On Fri, 4 Mar 2022 10:02:30 -0800 (PST), NArshad declaimed the following: >You have made everything too complicated. Just adjust the units left in the second feed into the third and fourth feed because the fish is special and that's it. The next day the second feed will be 100 units the way it used to be. > My solution is generalized for 1) any number of periods in the day, 2) mis-feed discovered at any point during the day, 3) works with any shape of feeding schedule. >What's wrong with the solution which I have written? Nothing IF you can justify that it meets the documented requirements. >From your original post: """ Implement some methods to distribute the remaining 40 unit in the rest of the day and propose the new patterns. Try to keep the distribution similar to the current feeding pattern. """ Rephrased some, with emphasis, into what I see as the two major requirements to be met: * distribute the remaining 40 unit IN THE REST OF THE DAY * TRY TO KEEP THE DISTRIBUTION SIMILAR TO THE CURRENT FEEDING PATTERN "Remaining 40 unit" is really a noise clause resulting only from the /example/ input schedule and feedings. Any programmatic solution should not know, ahead of time, what the schedule was, nor what the feedings actually were. There is no sense to writing a program in which all the input data is hard-coded -- you might as well hard-code the output too, while you are at it. I consider "..some methods" to also be noise clause -- the alternative interpretation is that you are supposed to create multiple programs doing different types of solutions to the one problem. Assume, for example, that a fish#2 has a feeding schedule that looks like: 150 100 20 20 20 20 20 20 100 150 (A big feed at start of day, an almost as large in second hour, six hours at a constant 20, and then a large feed near the end, followed by a really big feed to keep it happy overnight -- it seems to make more sense than slowly starving the fish during the afternoon ). That planned distribution could be called a bathtub curve -- high start and ends with a flat middle. (as a sideways histogram) *************** ********** ** ** ** ** ** ** ********** *************** Again, assume error in first two feeds as: 150 60 HOW DOES DUMPING THE 40 UNITS INTO HOUR 3 AND 4 SATISFY EITHER OF THE REQUIREMENTS? (... rest of day, and similar distribution). -=-=-=- C:\Users\Wulfraed\Documents\_Hg-Repositories\REXX>rexx feedme.rex Enter planned feeding schedule (feed units for each time period, space separated) 150 100 20 20 20 20 20 20 100 150 Enter actual feeding schedule up to current feed period 150 60 PLAN : 10 time periods 620 total feed units ACTUAL : 2 time periods 210 total feed units dispensed MIS-FEED: 40 underfed SCHEDULE: 150 60 |> 23 22 22 22 22 22 111 166 C:\Users\Wulfraed\Documents\_Hg-Repositories\REXX> -=-=-=- (I modified the script to put |> at the point where the modified feedings begin) The result, as a histogram *************** ********** **- **. **. **. **. **. ***********. ****************+ (Not very visible as each * is 10 units, and using . for 2.5, - for 5, + for 7.5) -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From hjp-python at hjp.at Sat Mar 5 06:51:00 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sat, 5 Mar 2022 12:51:00 +0100 Subject: Behavior of the for-else construct In-Reply-To: <20220304232544.mcbhet7wn2wa2k2r@hjp.at> References: <21739669.459456.1646348879560@mail.yahoo.com> <20220304232544.mcbhet7wn2wa2k2r@hjp.at> Message-ID: <20220305115100.s6qz6nf743r6ipw6@hjp.at> On 2022-03-05 00:25:44 +0100, Peter J. Holzer wrote: > On 2022-03-04 11:34:07 +1100, Chris Angelico wrote: > > What I'm hearing is that there are, broadly speaking, two types of > > programmers [1]: > > > > 1) Those who think about "for-else" as a search tool and perfectly > > understand how it behaves > > 2) Those who have an incorrect idea about what for-else is supposed to > > do, don't understand it, and don't like it. > > 3) Those who understand what it does und don't even find it*s syntax > very confusing, yet only very rarely find it useful. > > I have probably used it a handful of times in 8 years of Python > programming. Coincidentally, I think I used it just this week - but I > can't find it any more which probably means that it was either in a > throwaway script or I have since rewritten the code. Just found it again. Not yet committed, and I may still end up rewriting it again before turning in the pull request, but for now it's the simplest solution for the problem. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From rob.cliffe at btinternet.com Sat Mar 5 07:15:59 2022 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sat, 5 Mar 2022 12:15:59 +0000 Subject: Behavior of the for-else construct In-Reply-To: References: Message-ID: On 05/03/2022 01:15, Cameron Simpson wrote: > > I sort of wish it had both "used break" and "did not use break" > branches, a bit like try/except/else. And "zero iterations". Rob Cliffe From Marco.Sulla.Python at gmail.com Sat Mar 5 09:20:51 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Sat, 5 Mar 2022 15:20:51 +0100 Subject: Cpython: when to incref before insertdict Message-ID: I noticed that some functions inside dictobject.c that call insertdict or PyDict_SetItem do an incref of key and value before the call, and a decref after it. An example is dict_merge. Other functions, such as _PyDict_FromKeys, don't do an incref before. When an incref of key and value is needed before insertdict and when is not? And why is an incref needed? From barry at barrys-emacs.org Sat Mar 5 11:25:38 2022 From: barry at barrys-emacs.org (Barry Scott) Date: Sat, 5 Mar 2022 16:25:38 +0000 Subject: Getting Syslog working on OSX Monterey In-Reply-To: <20220304212307.bd3rj67m6gaej37a@hjp.at> References: <0D40B574-2975-49BD-9E97-9055317169E5@barrys-emacs.org> <20220228214109.ntoihdobzf5naqrx@hjp.at> <4EAF5AC0-446D-4493-AFF0-2FD8A8889439@barrys-emacs.org> <20220304212307.bd3rj67m6gaej37a@hjp.at> Message-ID: <64FFB4FB-F4BD-4E1A-AAC7-792A26D65EB6@barrys-emacs.org> > On 4 Mar 2022, at 21:23, Peter J. Holzer wrote: > > On 2022-02-28 22:05:05 +0000, Barry Scott wrote: >> On 28 Feb 2022, at 21:41, Peter J. Holzer wrote: >>> On 2022-02-27 22:16:54 +0000, Barry wrote: >>>> I have always assumed that if I want a logger syslog handler that I would have >>>> to implement it myself. So far I have code that uses syslog directly and have >>>> not written that code yet. >>> >>> What do you mean by using syslog directly? The syslog(3) library >>> function also just sends messages to a "syslog listener" (more commonly >>> called a syslog daemon) - at least on any unix-like system I'm familiar >>> with (which doesn't include MacOS). It will, however, always use the >>> *local* syslog daemon - AFAIK there is no standard way to open a remote >>> connection (many syslog daemons can be configured to forward messages to >>> a remote server, however). >> >> I'm re-reading the code to check on what I'm seeing. (Its been a long >> time since I last look deeply at this code). >> >> You can write to /dev/log if you pass that to >> SysLogHandler(address='/dev/log'), but the default is to use a socket >> to talk to a network listener on localhost:514. There are no deamons >> listening on port 514 on my Fedora systems or mac OS. > > If you are saying that SysLogHandler should use a system specific > default (e.g. "/dev/log" on Linux) instead of UDP port 514 everywhere, I > agree 99 % (the remaining 1 % is my contrarian alter ego arguing that > that's really the distribution maintainer's job since a Linux > distribution might use some other socket). > > If you are saying it should use the libc syslog routines, I disagree for > at least two reasons: a) they are OS specific, b) you can't configure the > destination. So that would remove useful functionality. > > In any case it seems strange to me that you want to rewrite it just to > avoid passing a single parameter to the constructor (or - more likely - > adding a single line to a config file). What I am used to is a setup where programs use syslog() to log and a dameon like rsyslog is responsible for routing the logs to local files and/or remote systems. On a modern linux I think the route is syslog() -> journald. And you can have syslog() -> journald -> rsyslogd. Using the syslog() function means that any platform/distro details are hidden from the user of syslog() and as is the case of macOS it "just works". (I assume, not checked, that the write to the socket does not work because Apple is not implementing the syslog protocol from the RFC). > >> What you do not see used in the SyslogHandler() is the import syslog >> and hence its nor using openlog() etc from syslog API. > > For good reasons. The C syslog API is missing important functionality. What are you think about being missing? Just curious. As an aside if I had the need to log into a system log mechanism I'd be looking to use the jounald API so that I can use structured logging on linux systems. Barry > > hp > > -- > _ | Peter J. Holzer | Story must make more sense than reality. > |_|_) | | > | | | hjp at hjp.at | -- Charles Stross, "Creative writing > __/ | http://www.hjp.at/ | challenge!" > -- > https://mail.python.org/mailman/listinfo/python-list From barry at barrys-emacs.org Sat Mar 5 11:34:41 2022 From: barry at barrys-emacs.org (Barry Scott) Date: Sat, 5 Mar 2022 16:34:41 +0000 Subject: virtualenv and make DESTDIR= In-Reply-To: <05a5b570-d917-2b25-33ea-560fcbd76ce8@goebel-consult.de> References: <05a5b570-d917-2b25-33ea-560fcbd76ce8@goebel-consult.de> Message-ID: > On 4 Mar 2022, at 13:03, Hartmut Goebel wrote: > > Hi, > > How can I make installing a virtual environment honor DESTDIR? How can I install a virtual environment in $(DESTDIR)$(PREFIX), which behaves as being set-up in $(PREFIX)? (Of course, this virtual environment can not be used. My aim is to ship it as part of a rpm package) > > In Makefiles is good practice to honor DESTDIR in the "install" target, like this > > install: > install -t $(DESTDIR)$(PREFIX)/bin build/bin/my-tool > > Now when running > > python3 -m venv $(DESTDIR)$(PREFIX) > > all paths in this virtual environment refer to $(DESTDIR)$(PREFIX) instead of just $$(PREFIX) > > Any ideas? > > > Background: > > More about DESTDIR: https://www.gnu.org/prep/standards/html_node/DESTDIR.html > > Following Redhat's commendations, I want to install my (somewhat complex) software into /opt/my-software. To make it easier for users to use the software, my idea was to setup a virtual environment in /opt/my-software. Thus users can easily use /opt/my-software/bin/python and have the library provided by my-software available. My Software also includes some scripts, which will also reside in /opt/my-software/bin and refer to /opt/my-software/bin/python. This will avoid to require users to set up PYTHONPATH when thy want to use MY Software. If you are packaging the code then I do not see the need to use a venv at all. Have the RPM install all the pythone code and dependencies and also install a short script that sets up PYTHONPATH, LD_LIBRARY_PATH, etc and execs the python3
.py. Note: you usually cannot use pip when building an RPM with mock as the network is disabled inside the build for security reasons. I package two of my projects this was for Fedora as RPMs. Barry > > > -- > Sch?nen Gru? > Hartmut Goebel > Dipl.-Informatiker (univ), CISSP, CSSLP, ISO 27001 Lead Implementer > Information Security Management, Security Governance, Secure Software Development > > Goebel Consult, Landshut > http://www.goebel-consult.de > > Blog: https://www.goe-con.de/blog/why-a-pki-is-barely-trustworthy > Kolumne: https://www.goe-con.de/hartmut-goebel/cissp-gefluester/2011-11-in-troja-nichts-neues > -- > https://mail.python.org/mailman/listinfo/python-list From Marco.Sulla.Python at gmail.com Sat Mar 5 11:59:48 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Sat, 5 Mar 2022 17:59:48 +0100 Subject: virtualenv and make DESTDIR= In-Reply-To: References: <05a5b570-d917-2b25-33ea-560fcbd76ce8@goebel-consult.de> Message-ID: On Sat, 5 Mar 2022 at 17:36, Barry Scott wrote: > Note: you usually cannot use pip when building an RPM with mock as the network is disabled inside the build for > security reasons. Can't he previously download the packages and run pip on the local packages? From barry at barrys-emacs.org Sat Mar 5 12:19:40 2022 From: barry at barrys-emacs.org (Barry Scott) Date: Sat, 5 Mar 2022 17:19:40 +0000 Subject: virtualenv and make DESTDIR= In-Reply-To: References: <05a5b570-d917-2b25-33ea-560fcbd76ce8@goebel-consult.de> Message-ID: <98D576F9-6D84-49B1-B172-EE4B619730AD@barrys-emacs.org> > On 5 Mar 2022, at 16:59, Marco Sulla wrote: > > On Sat, 5 Mar 2022 at 17:36, Barry Scott wrote: >> Note: you usually cannot use pip when building an RPM with mock as the network is disabled inside the build for >> security reasons. > > Can't he previously download the packages and run pip on the local packages? > I guess that can work. In this case I would install with pip install --user and move the files out of .local into an appropriate place in /opt/. At work I package a lot of packages for python and what we do is get the sources, review them then turn them into an RPM using the setup.py to do the file placement. We do not use the wheels from PyPI ever. They may not match the claimed sources. Which I have seen a few times; the version on PyPI can contain helpful patches that are not in the sources. Barry From rwood at therandymon.com.invalid Fri Mar 4 21:28:32 2022 From: rwood at therandymon.com.invalid (RS Wood) Date: Fri, 04 Mar 2022 21:28:32 -0500 Subject: looking for a freelancer to update a python NNTP script Message-ID: <87v8wtp1yn.fsf@amongus.com.invalid> I'm looking for a mid-level python programmer to help me with a python script that accesses Usenet. I've got a non-working 2.7 script that needs a little more attention than just running 2to3 on it. Estimated level of effort: about 2 hours or less. If interested please send me an email at moc.liamg at 71orifaz (spell that backwards - rudimentary anti-spam measure). Tell me your hourly rate or fixed price for the work and explain your qualifications. (Use email, don't post here). Payment would be through Paypal - half up front, half on completion. Selection will be made on the basis of both qualifications and cost - I'm not just looking for the lowest bidder. I'm posting the offer here because to test the script you need to have some understanding of Usenet and working Usenet access, obviously. I'll respond to this post when the job is filled. Thanks. From kvratkin at yandex.ru Sat Mar 5 12:18:36 2022 From: kvratkin at yandex.ru (Kirill Ratkin) Date: Sat, 5 Mar 2022 20:18:36 +0300 Subject: virtualenv and make DESTDIR= In-Reply-To: <05a5b570-d917-2b25-33ea-560fcbd76ce8@goebel-consult.de> References: <05a5b570-d917-2b25-33ea-560fcbd76ce8@goebel-consult.de> Message-ID: Hi, As far I know there is tool rpmvenv (https://github.com/kevinconway/rpmvenv). Try it, maybe ot helps. 04.03.2022 16:03, Hartmut Goebel wrote: > Hi, > > How can I make installing a virtual environment honor DESTDIR? How can > I install a virtual environment in $(DESTDIR)$(PREFIX), which behaves > as being set-up in $(PREFIX)? (Of course, this virtual environment can > not be used. My aim is to ship it as part of a rpm package) > > In Makefiles is good practice to honor DESTDIR in the "install" > target, like this > > install: > ??? install -t $(DESTDIR)$(PREFIX)/bin build/bin/my-tool > > Now when running > > ??? python3 -m venv $(DESTDIR)$(PREFIX) > > all paths in this virtual environment refer to $(DESTDIR)$(PREFIX) > instead of just $$(PREFIX) > > Any ideas? > > > Background: > > More about DESTDIR: > https://www.gnu.org/prep/standards/html_node/DESTDIR.html > > Following Redhat's commendations, I want to install my (somewhat > complex) software into /opt/my-software. To make it easier for users > to use the software, my idea was to setup a virtual environment in > /opt/my-software. Thus users can easily use > /opt/my-software/bin/python and have the library provided by > my-software available. My Software also includes some scripts, which > will also reside in /opt/my-software/bin and refer to > /opt/my-software/bin/python. This will avoid to require users to set > up PYTHONPATH when thy want to use MY Software. > > From avigross at verizon.net Sat Mar 5 12:48:40 2022 From: avigross at verizon.net (Avi Gross) Date: Sat, 5 Mar 2022 17:48:40 +0000 (UTC) Subject: Behavior of the for-else construct In-Reply-To: References: Message-ID: <1659671269.122157.1646502520833@mail.yahoo.com> Since we still seem to be dreaming, I wonder when someone will suggest using some variation of the new match statement. All you need is for the for loop (and possibly other such constructs) to return some kind of status that can be matched match (for var in something: ... ) : case break(): case skipped(): case did(1): case didall(): case crash(): case _ : As always, the above is not even pseudocode. But my point is there are languages I have seen that are built on such pattern matching and provide abilities to do a multi-way branching as long as you are very careful about the order you place the things to match. If not understood, the above function calls are meant to suggest something. For argument's sake, the for loop could return a non-negative number that shows the number of iterations done completely or even partially. It could also return various out-of-band (perhaps negative) values. So if it is skipped you get a zero or a value matching what my make-believe function skipped() returns. If the loop just ran once, it matched did(1). If it completed without breaks, it didall() and if it broke out, or crashed it matches something. I hope the idea comes across, even if you disagree with my indenting or exact method. Constructs like the above do in theory allow some really complex matching and often in a way easier to understand than having endless clauses added below the for group. But as repeatedly stated, Python currently has no concept of returning a value directly from a for loop or lots of other things. A brand new language that superficially resembled python could be designed but that is a rather formidable task. And where does it end? I mean you could set it up so the language allows you to return an invisible (meaning it does not print unless you ask for it) object among many possible objects. The simplest object may be an integer or perhaps an object of a class with a name like counter so you matched against counter(1) or against counter(n) if you want to capture the exact number for further processing. Various other object types might be returned including one that tells you about a break and also on which iteration it broke out of or even the value of the item it was looping on at the moment. If this starts looking a bit like how errors can be created and propagated, that is no coincidence. But the above also might come with serious overhead. And for many, it would not be seen as necessary. I will say that by allowing the match statement in the current form, the door has been opened wide for all kinds of potential enhancements. Older code and features may be left alone, but newer code is now free to do esoteric things. If I rewrote my call to the loop into a function that takes arguments including the data I want to have used and internally does the for loop while setting various variables that capture all kinds of info needed, then it potentially could return one of a series of objects you could pattern match against to your heart's content. A call to the function could be used in a match statement vaguely as in my hand-waving above. And of course you can use unpacking to make all kinds of compound patterns if done carefully. I think my part of this endless conversation may have gone a bit beyond far enough and I await some new topics. -----Original Message----- From: Rob Cliffe via Python-list To: python-list at python.org Sent: Sat, Mar 5, 2022 7:15 am Subject: Re: Behavior of the for-else construct On 05/03/2022 01:15, Cameron Simpson wrote: > > I sort of wish it had both "used break" and "did not use break" > branches, a bit like try/except/else. And "zero iterations". Rob Cliffe -- https://mail.python.org/mailman/listinfo/python-list From h.goebel at goebel-consult.de Sat Mar 5 14:56:23 2022 From: h.goebel at goebel-consult.de (Hartmut Goebel) Date: Sat, 5 Mar 2022 20:56:23 +0100 Subject: virtualenv and make DESTDIR= In-Reply-To: References: <05a5b570-d917-2b25-33ea-560fcbd76ce8@goebel-consult.de> Message-ID: <7d513446-ca0e-cbc4-36dd-09007af51a83@goebel-consult.de> Am 05.03.22 um 17:34 schrieb Barry Scott: > Have the RPM install all the pythone code and dependencies and also install a short script that > sets up PYTHONPATH, LD_LIBRARY_PATH, etc and execs the python3
.py. The scripts are already created by entry-points. So basically this means to reinvent the wheel. Or did I miss something? -- Sch?nen Gru? Hartmut Goebel Dipl.-Informatiker (univ), CISSP, CSSLP, ISO 27001 Lead Implementer Information Security Management, Security Governance, Secure Software Development Goebel Consult, Landshut http://www.goebel-consult.de Blog: https://www.goe-con.de/blog/eilt-petition-auf-weact-gegen-vds-jetzt-unterschreiben Kolumne: https://www.goe-con.de/hartmut-goebel/cissp-gefluester/2011-11-in-troja-nichts-neues From barry at barrys-emacs.org Sat Mar 5 15:15:35 2022 From: barry at barrys-emacs.org (Barry Scott) Date: Sat, 5 Mar 2022 20:15:35 +0000 Subject: virtualenv and make DESTDIR= In-Reply-To: <7d513446-ca0e-cbc4-36dd-09007af51a83@goebel-consult.de> References: <05a5b570-d917-2b25-33ea-560fcbd76ce8@goebel-consult.de> <7d513446-ca0e-cbc4-36dd-09007af51a83@goebel-consult.de> Message-ID: <9D8D4AF8-70F2-44D8-B9B7-FCAA6AA0A838@barrys-emacs.org> > On 5 Mar 2022, at 19:56, Hartmut Goebel wrote: > > Am 05.03.22 um 17:34 schrieb Barry Scott: >> Have the RPM install all the pythone code and dependencies and also install a short script that >> sets up PYTHONPATH, LD_LIBRARY_PATH, etc and execs the python3
.py. > The scripts are already created by entry-points. So basically this means to reinvent the wheel. Or did I miss something? > Are you saying that the only thing you want from the venv is a script that sets up PYTHONPATH? That a trivia thing to write and far easier then battling with packaging a venv I'd expect. Did I miss something? Barry > -- > Sch?nen Gru? > Hartmut Goebel > Dipl.-Informatiker (univ), CISSP, CSSLP, ISO 27001 Lead Implementer > Information Security Management, Security Governance, Secure Software Development > Goebel Consult, Landshut > http://www.goebel-consult.de > Blog: https://www.goe-con.de/blog/eilt-petition-auf-weact-gegen-vds-jetzt-unterschreiben > Kolumne: https://www.goe-con.de/hartmut-goebel/cissp-gefluester/2011-11-in-troja-nichts-neues From michael.stemper at gmail.com Sat Mar 5 13:39:36 2022 From: michael.stemper at gmail.com (Michael F. Stemper) Date: Sat, 5 Mar 2022 12:39:36 -0600 Subject: Behavior of the for-else construct In-Reply-To: References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304225746.mmebv3myg5wbpc4f@hjp.at> <657845041.42944.1646437629325@mail.yahoo.com> <20220305001158.g7rmlyoxxtxufpfp@hjp.at> Message-ID: On 04/03/2022 18.11, Peter J. Holzer wrote: > On 2022-03-04 23:47:09 +0000, Avi Gross via Python-list wrote: >> I am not sure a reply is needed, Peter, and what you say is true. But >> as you point out, when using a German keyboard, I would already have >> a way to enter symbols like ?, ?, ? and ? and no reason not to include >> them in variable names and so on if UNICODE is being used properly. I >> can use my last name in German notation as a variable in Python now: >> >> Gro? = 144 >> Gro? / 12 >> 12.0 > > Yes, I'm using umlauts occasionally in variable names in Python, and > I've also used Greek characters and others. > > But in Python I CAN use them. I DON'T HAVE to. > > That's a big difference. > > Characters like [] or {} are a part of Python's syntax. You can't avoid > using them. If you can't type them, you can't write Python. If it is > awkward to enter them (like having to type Alt-91 or pasting them from a > character table) it is painful to write programs. > > German keyboards aquired an AltGr key and the ability to type these > characters in the mid to late 1980's. Presumably because those > characters were common in C and other programming languages ... especially Pascal, which was probably bigger in Germany and Austria in the 1980s than was C. -- Michael F. Stemper Psalm 94:3-6 From wlfraed at ix.netcom.com Sat Mar 5 14:25:35 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Sat, 05 Mar 2022 14:25:35 -0500 Subject: Behavior of the for-else construct References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304225746.mmebv3myg5wbpc4f@hjp.at> <657845041.42944.1646437629325@mail.yahoo.com> <20220305001158.g7rmlyoxxtxufpfp@hjp.at> Message-ID: <1mc72hll06itd6jnbgdherqb3thf1fk1pr@4ax.com> On Sat, 5 Mar 2022 12:39:36 -0600, "Michael F. Stemper" declaimed the following: >... especially Pascal, which was probably bigger in Germany and Austria >in the 1980s than was C. Pascal also defined alternate representations (per Jensen&Wirth) for some of those (and I don't recall ever seeing a system that actually had an up-arrow character -- and selecting one in character map doesn't help, my client doesn't render it). direct alternate ? ^ or @ [ (. ] .) { (* } *) I'll admit... (. is a clumsy sequence to type in place of [ (RH ring finger to top-row on modern keyboards followed by RH ring finger to bottom-row .), but it is a sequence any terminal mapping to a common /typewriter/ keyboard should have available. (* isn't quite as clumsy (RH ring finger to top-row followed by RH middle finger to top-row ). More fun is had when doing APL without a dedicated APL keyboard (Though Xerox Sigma APL also had plain text alternatives: $RHO for example) -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From barry at barrys-emacs.org Sat Mar 5 15:39:13 2022 From: barry at barrys-emacs.org (Barry Scott) Date: Sat, 5 Mar 2022 20:39:13 +0000 Subject: virtualenv and make DESTDIR= In-Reply-To: <7d513446-ca0e-cbc4-36dd-09007af51a83@goebel-consult.de> References: <05a5b570-d917-2b25-33ea-560fcbd76ce8@goebel-consult.de> <7d513446-ca0e-cbc4-36dd-09007af51a83@goebel-consult.de> Message-ID: <18D07A07-FB74-4046-95AA-F297C93385C7@barrys-emacs.org> > On 5 Mar 2022, at 19:56, Hartmut Goebel wrote: > > Am 05.03.22 um 17:34 schrieb Barry Scott: >> Have the RPM install all the pythone code and dependencies and also install a short script that >> sets up PYTHONPATH, LD_LIBRARY_PATH, etc and execs the python3
.py. > The scripts are already created by entry-points. So basically this means to reinvent the wheel. Or did I miss something? > For example I assume you need something like this: #!/bin/bash export PYTHONPATH=/opt//lib exec /usr/bin/python3 /opt//lib/main.py "$@" I'm assuming you put all the python code into /opt//lib and any runnable command into /opt//bin Barry > -- > Sch?nen Gru? > Hartmut Goebel > Dipl.-Informatiker (univ), CISSP, CSSLP, ISO 27001 Lead Implementer > Information Security Management, Security Governance, Secure Software Development > Goebel Consult, Landshut > http://www.goebel-consult.de > Blog: https://www.goe-con.de/blog/eilt-petition-auf-weact-gegen-vds-jetzt-unterschreiben > Kolumne: https://www.goe-con.de/hartmut-goebel/cissp-gefluester/2011-11-in-troja-nichts-neues From avigross at verizon.net Sat Mar 5 16:40:08 2022 From: avigross at verizon.net (Avi Gross) Date: Sat, 5 Mar 2022 21:40:08 +0000 (UTC) Subject: Behavior of the for-else construct In-Reply-To: References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304225746.mmebv3myg5wbpc4f@hjp.at> <657845041.42944.1646437629325@mail.yahoo.com> <20220305001158.g7rmlyoxxtxufpfp@hjp.at> Message-ID: <1137940845.144932.1646516408119@mail.yahoo.com> I am not sure how we end up conversing about PASCAL on a Python forum. But it is worth considering how people educated in aspects of Computer Science often come from somewhat different background and how it flavors what they do now. I paid no attention to where PASCAL was being used other than I did much of my grad school work in PASCAL in the early 80's including my thesis being a document that could be typeset or run from the same file ;-) And my first job in the field also was doing programming in PASCAL. At the time it seemed to be a new and upcoming language with lots of nifty features. It seemed to have so many ways it was an improvement on languages I had been using including BASIC and Fortran and versions of LISP. So it was a tad surprising when my next job at Bell Labs focused on C (and later C++) as well as a large array of utilities and small languages in the UNIX world. PASCAL was nowhere to be seen nor others that arrived to take over the world like Modula and Ada, and yet didn't. It depends where you are and on your perspective, and perhaps it survived in places like Europe and often has been enhanced. We have mentioned how things like Fortran keep evolving and, in any case, tons of the code we use in aspects of Python and especially some add-in modules, is from libraries of well-honed libraries of functions written in Fortran or C or I am sure in some cases even assembler languages. I wonder if some languages of the future may supersede languages like Python but retain aspects of them in a similar way? As a hobby, I study lots of languages to see if they add anything or are likely to take over. Some seem to really be focused on a company like Microsoft or Google using it for their own purposes and others who wish to play in their arena may have to go along to operate there. Others seem to be the kind of things academics invent to play with various ideas. So will SCALA or GO or RUST become major players or will they wither away? Will languages that make major changes that make older software not compatible, think Python or PERL as examples, end up stronger or ... motivate some to abandon their projects and switch to a new language/system/paradigm? I do a lot of data manipulation in an assortment of languages including multiple methods within a language. I can sometimes see how a person asking for features or writing programs has been influenced by earlier experiences. Database people who started with a relational database and mainly used some dialect of SQL, may face some problems in their own way and think making lots of smaller tables in third normal form and constantly doing all kinds of table merges is the natural way to do things. Those who start doing things using Python or R may view things quite differently and often see ways to do things gradually and in a pipelined method and not do what is seen as expensive operations like some kinds of merge. I am not saying everyone does things in a stereotypical way, just that people learn tools and methods and it may show. Given how much faster some things have become, even hybrid programmers may play games. Yes, you can issue SQL to load data from a database into your Python or R programs but some issue a very basic command and then massage it locally while others do most of the processing in the SQL component so that less is dragged in. Some habits persist even after circumstances change. I have little against PASCAL and have not looked at how it has evolved but found it a very limiting language after a while. But I find most languages somewhat limited and now think they should be. What makes a language great for me is if it does not try to be too complete but provides a way to extend it so people can come up with their own ways to do more complex things, including various ways to do graphics or statistical analyses and so on, as sort of add-ons which become little languages or worlds grafted on when needed and ignored when not. Python qualifies albeit it is already too bloated! LOL! Avi Palm: 3rd digit from the right, vertically. -----Original Message----- From: Michael F. Stemper To: python-list at python.org Sent: Sat, Mar 5, 2022 1:39 pm Subject: Re: Behavior of the for-else construct On 04/03/2022 18.11, Peter J. Holzer wrote: > On 2022-03-04 23:47:09 +0000, Avi Gross via Python-list wrote: >> I am not sure a reply is needed, Peter, and what you say is true. But >> as you point out, when using a German keyboard, I would? already have >> a way to enter symbols like ?, ?, ? and ? and no reason not to include >> them in variable names and so on if UNICODE is being used properly. I >> can use my last name in German notation as a variable in Python now: >> >> Gro? = 144 >> Gro? / 12 >> 12.0 > > Yes, I'm using umlauts occasionally in variable names in Python, and > I've also used Greek characters and others. > > But in Python I CAN use them. I DON'T HAVE to. > > That's a big difference. > > Characters like [] or {} are a part of Python's syntax. You can't avoid > using them. If you can't type them, you can't write Python. If it is > awkward to enter them (like having to type Alt-91 or pasting them from a > character table) it is painful to write programs. > > German keyboards aquired an AltGr key and the ability to type these > characters in the mid to late 1980's. Presumably because those > characters were common in C and other programming languages ... especially Pascal, which was probably bigger in Germany and Austria in the 1980s than was C. -- Michael F. Stemper Psalm 94:3-6 -- https://mail.python.org/mailman/listinfo/python-list From cs at cskk.id.au Sat Mar 5 17:18:23 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Sun, 6 Mar 2022 09:18:23 +1100 Subject: Behavior of the for-else construct In-Reply-To: <1659671269.122157.1646502520833@mail.yahoo.com> References: <1659671269.122157.1646502520833@mail.yahoo.com> Message-ID: On 05Mar2022 17:48, Avi Gross wrote: >Since we still seem to be dreaming, I wonder when someone will suggest using some variation of the new match statement. Ugh :-) But... What if break were implemented with an exception, like StopIteration but for interruption instead of stop? And for-loops were an implied try/except: for ... except BrokenIteration: # <-- implied pass and you could go: for ... except BrokenIteration: ... a break occurred else: ... a break did not occur and, indeed, for ... except BrokenIteration: ... a break occurred except StopIteration as stopped: print("I did", stopped.count, "iterations") else: # not reached because of the "except StopIteration" above ... a break did not occur if we wanted to cover off the count iterations wish item. Cheers, Cameron Simpson From oladejiolofinboba at yahoo.com Sat Mar 5 15:36:06 2022 From: oladejiolofinboba at yahoo.com (Deji Olofinboba) Date: Sat, 5 Mar 2022 20:36:06 +0000 (UTC) Subject: Help: Unable to find IDLE folder inside the Python Lib folder References: <1875418232.148964.1646512566786.ref@mail.yahoo.com> Message-ID: <1875418232.148964.1646512566786@mail.yahoo.com> Dear Python officer, Please I am new to programming. I have justinstalled the python 3.10.2. After the installation, I was able to locate thePython Shell but unable to locate IDLE despite checking it before downloading in the python installation folder. I also reinstalled Python and checked IDLE; still the IDLE is still missing. Please. explain to me how I can retrieve the python IDLE. Thank You, ? Deji Olofinboba? From python at mrabarnett.plus.com Sat Mar 5 19:42:13 2022 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 6 Mar 2022 00:42:13 +0000 Subject: Help: Unable to find IDLE folder inside the Python Lib folder In-Reply-To: <1875418232.148964.1646512566786@mail.yahoo.com> References: <1875418232.148964.1646512566786.ref@mail.yahoo.com> <1875418232.148964.1646512566786@mail.yahoo.com> Message-ID: On 2022-03-05 20:36, Deji Olofinboba via Python-list wrote: > > Dear Python officer, > Please I am new to programming. I have justinstalled the python 3.10.2. After the installation, I was able to locate thePython Shell but unable to locate IDLE despite checking it before downloading in the python installation folder. I also reinstalled Python and checked IDLE; still the IDLE is still missing. Please. explain to me how I can retrieve the python IDLE. > Thank You, > If you're on Windows 10, type "idle" into the search box on the taskbar. Alternatively, look in python_folder\Lib\idlelib for idle.bat if you're on an older version of Windows. From wlfraed at ix.netcom.com Sat Mar 5 19:00:49 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Sat, 05 Mar 2022 19:00:49 -0500 Subject: Behavior of the for-else construct References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304225746.mmebv3myg5wbpc4f@hjp.at> <657845041.42944.1646437629325@mail.yahoo.com> <20220305001158.g7rmlyoxxtxufpfp@hjp.at> <1137940845.144932.1646516408119@mail.yahoo.com> Message-ID: <45r72h5lcqemt43u107t11kt2oiu9shvc7@4ax.com> On Sat, 5 Mar 2022 21:40:08 +0000 (UTC), Avi Gross declaimed the following: >I am not sure how we end up conversing about PASCAL on a Python forum. But it is worth considering how people educated in aspects of Computer Science often come from somewhat different background and how it flavors what they do now. > You'd prefer REBOL, perhaps? REXX at least has some structure to it NB: Pascal has, like Ada, always been a name -- not like the origins of COBOL, FORTRAN, BASIC, et al. >I paid no attention to where PASCAL was being used other than I did much of my grad school work in PASCAL in the early 80's including my thesis being a document that could be typeset or run from the same file ;-) > Very early? The common versions were probably UCSD (running on UCSD P-System); or a port of the P-4 compiler (which had been published in book form) -- maybe a "tinyPascal" (integer only I suspect). Radio Shack did provide Alcor Pascal for the TRS-80. Later, you might have encountered TurboPascal -- which bore little resemblance to a Jensen&Wirth Pascal. J&W was one-program<>one-file (no link libraries, no "include" files as I recall); very limited math functions if one is trying for scientific applications (sin, cos, arctan were the trig functions I recall), and that very unfriendly I/O system (console I/O required special handling from file I/O as Pascal does a one-element pre-read when an I/O channel is opened -- which occurs on program load for stdin, much before a program could output a prompt to the user). Even my first exposure to VAX/VMS Pascal, which did allow for separate compilation and linking, required me to declare interfaces to the FORTRAN run-time library to get advanced math functions (I believe later versions incorporated the FORTRAN math natively). -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From songofacandy at gmail.com Sat Mar 5 21:19:55 2022 From: songofacandy at gmail.com (Inada Naoki) Date: Sun, 6 Mar 2022 11:19:55 +0900 Subject: Cpython: when to incref before insertdict In-Reply-To: References: Message-ID: On Sat, Mar 5, 2022 at 11:22 PM Marco Sulla wrote: > > I noticed that some functions inside dictobject.c that call insertdict > or PyDict_SetItem do an incref of key and value before the call, and a > decref after it. An example is dict_merge. First of all, insertdict and PyDict is totally different about reference ownership handling. * PyDict_SetItem borrows reference of key and value from the caller as usual Python/C APIs. And it INCREF them before calling the insertdict(). * insertdict() takes the reference from its caller. In other words, insertdict() moves the owner of reference from its caller to the dict. merge_dict is very special and complex case. I assume you are talking about this part. https://github.com/python/cpython/blob/6927632492cbad86a250aa006c1847e03b03e70b/Objects/dictobject.c#L2885-L2912 In general, when reference is borrowed from a caller, the reference is available during the API. But merge_dict borrows reference of key/value from other dict, not caller. So dict_merge must have strong reference of key/value by INCREF before calling any APIs (e.g. _PyDict_Contains_KnownHash). That's why dict_merge calls INCREF key/value **twice** before calling insertdict, and DECREF key/value **once** after it. > Other functions, such as > _PyDict_FromKeys, don't do an incref before. > Again, insertdict takes the reference. So _PyDict_FromKeys() **does** INCREF before calling insertdict, when key/value is borrowed reference. https://github.com/python/cpython/blob/6927632492cbad86a250aa006c1847e03b03e70b/Objects/dictobject.c#L2287-L2290 https://github.com/python/cpython/blob/6927632492cbad86a250aa006c1847e03b03e70b/Objects/dictobject.c#L2309-L2311 On the other hand, slow path uses PyIter_Next() which returns strong reference. So no need to INCREF it. Additionally, the slow path uses PyDict_SetItem(), not insertdict(). PyDict_SetItem() does INCREF key/value for insertdict. So the slow path need to DECREF(key). https://github.com/python/cpython/blob/6927632492cbad86a250aa006c1847e03b03e70b/Objects/dictobject.c#L2327-L2329 This is complete guide why/when INCREF/DECREF key/value. -- Inada Naoki From avigross at verizon.net Sat Mar 5 23:22:01 2022 From: avigross at verizon.net (Avi Gross) Date: Sun, 6 Mar 2022 04:22:01 +0000 (UTC) Subject: Behavior of the for-else construct In-Reply-To: <45r72h5lcqemt43u107t11kt2oiu9shvc7@4ax.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304225746.mmebv3myg5wbpc4f@hjp.at> <657845041.42944.1646437629325@mail.yahoo.com> <20220305001158.g7rmlyoxxtxufpfp@hjp.at> <1137940845.144932.1646516408119@mail.yahoo.com> <45r72h5lcqemt43u107t11kt2oiu9shvc7@4ax.com> Message-ID: <1549659151.175085.1646540521369@mail.yahoo.com> I am not clear on what you are commenting, Dennis. You are responding to what I do not believe I wrote. You did not quote the part of my message where I wrote what "I" did in the early 80's and did not say when PASCAL was available elsewhere. "I paid no attention to where PASCAL was being used other than I did much of my grad school work in PASCAL in the early 80's including my thesis being a document that could be typeset or run from the same file ;-)" I might have encountered it in the early 70's had I had any room in my college course schedule where I ended up with a triple major in other subjects or while I was in Medical school. But no, I only went back to study C.S. afterwards. So, I apologize for being late. Heck I was also not there when ALGOL was created or FORTRAN, let alone COBOL! But if it helps, I taught FORTRAN and other languages to undergrads ;-) -----Original Message----- From: Dennis Lee Bieber To: python-list at python.org Sent: Sat, Mar 5, 2022 7:00 pm Subject: Re: Behavior of the for-else construct On Sat, 5 Mar 2022 21:40:08 +0000 (UTC), Avi Gross declaimed the following: >I am not sure how we end up conversing about PASCAL on a Python forum. But it is worth considering how people educated in aspects of Computer Science often come from somewhat different background and how it flavors what they do now. > ??? You'd prefer REBOL, perhaps? ??? REXX at least has some structure to it ??? NB: Pascal has, like Ada, always been a name -- not like the origins of COBOL, FORTRAN, BASIC, et al. >I paid no attention to where PASCAL was being used other than I did much of my grad school work in PASCAL in the early 80's including my thesis being a document that could be typeset or run from the same file ;-) > ??? Very early? The common versions were probably UCSD (running on UCSD P-System); or a port of the P-4 compiler (which had been published in book form) -- maybe a "tinyPascal" (integer only I suspect). Radio Shack did provide Alcor Pascal for the TRS-80. ??? Later, you might have encountered TurboPascal -- which bore little resemblance to a Jensen&Wirth Pascal. ??? J&W was one-program<>one-file (no link libraries, no "include" files as I recall); very limited math functions if one is trying for scientific applications (sin, cos, arctan were the trig functions I recall), and that very unfriendly I/O system (console I/O required special handling from file I/O as Pascal does a one-element pre-read when an I/O channel is opened -- which occurs on program load for stdin, much before a program could output a prompt to the user). ??? Even my first exposure to VAX/VMS Pascal, which did allow for separate compilation and linking, required me to declare interfaces to the FORTRAN run-time library to get advanced math functions (I believe later versions incorporated the FORTRAN math natively). -- ??? Wulfraed? ? ? ? ? ? ? ? Dennis Lee Bieber? ? ? ? AF6VN ??? wlfraed at ix.netcom.com? ? http://wlfraed.microdiversity.freeddns.org/ -- https://mail.python.org/mailman/listinfo/python-list From barry at barrys-emacs.org Sun Mar 6 05:19:41 2022 From: barry at barrys-emacs.org (Barry Scott) Date: Sun, 6 Mar 2022 10:19:41 +0000 Subject: virtualenv and make DESTDIR= In-Reply-To: <7d513446-ca0e-cbc4-36dd-09007af51a83@goebel-consult.de> References: <05a5b570-d917-2b25-33ea-560fcbd76ce8@goebel-consult.de> <7d513446-ca0e-cbc4-36dd-09007af51a83@goebel-consult.de> Message-ID: > On 5 Mar 2022, at 19:56, Hartmut Goebel wrote: > > Am 05.03.22 um 17:34 schrieb Barry Scott: >> Have the RPM install all the pythone code and dependencies and also install a short script that >> sets up PYTHONPATH, LD_LIBRARY_PATH, etc and execs the python3
.py. > The scripts are already created by entry-points. So basically this means to reinvent the wheel. Or did I miss something? > I see what are getting at - I did miss the something. It is the setup.py that will create your entry point! For my own understanding I turned one of my PyPI projects into an RPM. The project source is on https://github.com/barry-scott/CLI-tools.git First I turned the this into a tarball: $ git archive --format=tar --prefix=cli-tools-3.1.0/ master >cli-tools-3.1.0.tar $ gzip cli-tools-3.1.0.tar Then I created a RPM spec file: Summary: cli-tools example packaging for python Name: cli-tools Version: 3.1.0 Release: 1 BuildArch: noarch Source: %{name}-%{version}.tar.gz License: APL2.0 Prefix: /opt/barry-example BuildRequires: python3 BuildRequires: python3-setuptools Requires: python3 %description cli-tools example packaging for python %prep # unpack Source tarball %setup %build cd Source python3 setup_ssh_wait.py build %install cd Source python3 setup_ssh_wait.py install --prefix=%{prefix} --root=%{buildroot} # need to add the site-packages folder to the sys.path as # setup.py assumes it is installing into the systems python library cat <fix_bin.py python3 fix_bin.py %{buildroot}%{prefix}/bin/ssh-wait %{buildroot}%{prefix}/opt/barry-example/lib %files /opt/barry-example/* %changelog Then only thing I needed to fix up was that the entry point code needs to have the lib folder in /opt/barry-example added to sys.path. That is what the fix_bin.py is doing. I then use rpmbuild to create the RPM. $ rpmbuild -bb cli-tools.spec To use mock the steps are: $ rpmbuild -bs cli-tools.spec $ mock --rebuild --root fedora-35-x86_64 ../SRPMS/cli-tools-3.1.0.arc.rpm Then I install the built file: $ dnf install /var/lib/mock/fedora-35-x86_64/result/cli-tools-3.1.0-1.noarch.rpm And then I can run the command: $ /opt/barry-example/bin/ssh-wait Usage: ssh-wait ssh-wait will wait for the service to be available on . --verbose - print progress messages --nowait - check one and exit 0 if can connect 1 if cannot --wait-limit= - exit 1 if service is not avaiable within seconds. defaults to 600. --service= - check the service . Can be a service name of a port number. Default is ssh. Barry From martinp.dipaola at gmail.com Sun Mar 6 07:42:08 2022 From: martinp.dipaola at gmail.com (Martin Di Paola) Date: Sun, 6 Mar 2022 12:42:08 +0000 Subject: Execute in a multiprocessing child dynamic code loaded by the parent process Message-ID: <20220306124208.orqbcezhqqeysdpe@gmail.com> Hi everyone. I implemented time ago a small plugin engine to load code dynamically. So far it worked well but a few days ago an user told me that he wasn't able to run in parallel a piece of code in MacOS. He was using multiprocessing.Process to run the code and in MacOS, the default start method for such process is using "spawn". My understanding is that Python spawns an independent Python server (the child) which receives what to execute (the target function) from the parent process. In pseudo code this would be like: modules = loader() # load the plugins (Python modules at the end) objs = init(modules) # initialize the plugins # One of the plugins wants to execute part of its code in parallel # In MacOS this fails ch = multiprocessing.Process(target=objs[0].sayhi) ch.start() The code fails with "ModuleNotFoundError: No module named 'foo'" (where 'foo' is the name of the loaded plugin). This is because the parent program sends to the serve (the child) what needs to execute (objs[0].sayhi) using pickle as the serialization mechanism. Because Python does not really serialize code but only enough information to reload it, the serialization of "objs[0].sayhi" just points to its module, "foo". Module which it cannot be imported by the child process. So the question is, what would be the alternatives and workarounds? I came with a hack: use a trampoline() function to load the plugins in the child before executing the target function. In pseudo code it is: modules = loader() # load the plugins (Python modules at the end) objs = init(modules) # initialize the plugins def trampoline(target_str): loader() # load the plugins now that we are in the child process # deserialize the target and call it target = reduction.loads(target_str) target() # Serialize the real target function, but call in the child # trampoline(). Because it can be accessed by the child it will # not fail target_str = reduction.dumps(objs[0].sayhi) ch = multiprocessing.Process(target=trampoline, args=(target_str,)) ch.start() The hack works but is this the correct way to do it? The following gist has the minimal example code that triggers the issue and its workaround: https://gist.github.com/eldipa/d9b02875a13537e72fbce4cdb8e3f282 Thanks! Martin. From rosuav at gmail.com Sun Mar 6 07:52:51 2022 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 6 Mar 2022 23:52:51 +1100 Subject: Execute in a multiprocessing child dynamic code loaded by the parent process In-Reply-To: <20220306124208.orqbcezhqqeysdpe@gmail.com> References: <20220306124208.orqbcezhqqeysdpe@gmail.com> Message-ID: On Sun, 6 Mar 2022 at 23:43, Martin Di Paola wrote: > > Hi everyone. I implemented time ago a small plugin engine to load code > dynamically. > > So far it worked well but a few days ago an user told me that he wasn't > able to run in parallel a piece of code in MacOS. > > He was using multiprocessing.Process to run the code and in MacOS, the > default start method for such process is using "spawn". My understanding > is that Python spawns an independent Python server (the child) which > receives what to execute (the target function) from the parent process. > Because Python does not really serialize code but only enough > information to reload it, the serialization of "objs[0].sayhi" just > points to its module, "foo". > Hmm. This is a route that has some tricky hazards on it. Generally, in Python code, we can assume that a module is itself, no matter what; it won't be a perfect clone of itself, it will actually be the same module. If you want to support multiprocessing, I would recommend disconnecting yourself from the concept of loaded modules, and instead identify the target by its module name. > I came with a hack: use a trampoline() function to load the plugins > in the child before executing the target function. > > In pseudo code it is: > > modules = loader() # load the plugins (Python modules at the end) > objs = init(modules) # initialize the plugins > > def trampoline(target_str): > loader() # load the plugins now that we are in the child process > > # deserialize the target and call it > target = reduction.loads(target_str) > target() > > # Serialize the real target function, but call in the child > # trampoline(). Because it can be accessed by the child it will > # not fail > target_str = reduction.dumps(objs[0].sayhi) > ch = multiprocessing.Process(target=trampoline, args=(target_str,)) > ch.start() > > The hack works but is this the correct way to do it? > The way you've described it, it's a hack. Allow me to slightly redescribe it. modules = loader() objs = init(modules) def invoke(mod, func): # I'm assuming that the loader is smart enough to not load # a module that's already loaded. Alternatively, load just the # module you need, if that's a possibility. loader() target = getattr(modules[mod], func) target() ch = multiprocessing.Process(target=invoke, args=("some_module", "sayhi")) ch.start() Written like this, it achieves the same goal, but looks a lot less hacky, and as such, I would say that yes, this absolutely IS a correct way to do it. (I won't say "the" correct way, as there are other valid ways, but there's certainly nothing wrong with this idea.) ChrisA From shishaozhong at gmail.com Sun Mar 6 09:24:09 2022 From: shishaozhong at gmail.com (Shaozhong SHI) Date: Sun, 6 Mar 2022 14:24:09 +0000 Subject: Issues of pip install gdal and fiona Message-ID: I downloaded .whl files for fiona and gdal to go with Python3.6.5. However, I am having trouble with red error messages. Though Gdal is now working, there is a warning message - Missing global ~ gdal: DRIVER_NAME declaration gdal_array,py Can anyone advise on how to resolve the issues? Regards, David From Marco.Sulla.Python at gmail.com Sun Mar 6 11:07:33 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Sun, 6 Mar 2022 17:07:33 +0100 Subject: Cpython: when to incref before insertdict In-Reply-To: References: Message-ID: On Sun, 6 Mar 2022 at 03:20, Inada Naoki wrote: > In general, when reference is borrowed from a caller, the reference is > available during the API. > But merge_dict borrows reference of key/value from other dict, not caller. > [...] > Again, insertdict takes the reference. So _PyDict_FromKeys() **does** > INCREF before calling insertdict, when key/value is borrowed > reference. > https://github.com/python/cpython/blob/6927632492cbad86a250aa006c1847e03b03e70b/Objects/dictobject.c#L2287-L2290 > https://github.com/python/cpython/blob/6927632492cbad86a250aa006c1847e03b03e70b/Objects/dictobject.c#L2309-L2311 > > On the other hand, slow path uses PyIter_Next() which returns strong > reference. So no need to INCREF it. Thank you Inada, these points make me things clear now. (PS: dictobject will change a lot in 3.11... sigh :D) From hjp-python at hjp.at Sun Mar 6 11:39:51 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 6 Mar 2022 17:39:51 +0100 Subject: Behavior of the for-else construct In-Reply-To: <1mc72hll06itd6jnbgdherqb3thf1fk1pr@4ax.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304225746.mmebv3myg5wbpc4f@hjp.at> <657845041.42944.1646437629325@mail.yahoo.com> <20220305001158.g7rmlyoxxtxufpfp@hjp.at> <1mc72hll06itd6jnbgdherqb3thf1fk1pr@4ax.com> Message-ID: <20220306163951.2ozmrhfbtsktbbt7@hjp.at> On 2022-03-05 14:25:35 -0500, Dennis Lee Bieber wrote: > On Sat, 5 Mar 2022 12:39:36 -0600, "Michael F. Stemper" > declaimed the following: > >... especially Pascal, which was probably bigger in Germany and Austria > >in the 1980s than was C. > > Pascal also defined alternate representations (per Jensen&Wirth) for > some of those (and I don't recall ever seeing a system that actually had an > up-arrow character -- and selecting one in character map doesn't help, my > client doesn't render it). > > direct alternate > ? ^ or @ > [ (. > ] .) > { (* > } *) (* *) for comments was actually pretty commonly used - maybe because it stands out more than { }. I don't know if I've ever seen (. .) instead of [ ]. C also has alternative rerpresentations for characters not in the common subset of ISO-646 and EBCDIC. However, the trigraphs are extremely ugly (e.g ??< ??> instead of { }). I have seen them used (on an IBM/390 system with an EBCDIC variant without curly braces) and it's really no fun to read that. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From hjp-python at hjp.at Sun Mar 6 11:50:55 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 6 Mar 2022 17:50:55 +0100 Subject: virtualenv and make DESTDIR= In-Reply-To: References: <05a5b570-d917-2b25-33ea-560fcbd76ce8@goebel-consult.de> Message-ID: <20220306165055.xaflsr7lczdzabfa@hjp.at> On 2022-03-05 17:59:48 +0100, Marco Sulla wrote: > On Sat, 5 Mar 2022 at 17:36, Barry Scott wrote: > > Note: you usually cannot use pip when building an RPM with mock as the network is disabled inside the build for > > security reasons. > > Can't he previously download the packages and run pip on the local packages? That shouldn't be necessary. Any packages needed by that new package should be dependencies of that package, so they must also already exist as RPMs. The rpm build process will automatically install them if they aren't already. (That may not be true for build dependencies, though.) hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From morphex at gmail.com Sun Mar 6 07:46:39 2022 From: morphex at gmail.com (Morten W. Petersen) Date: Sun, 6 Mar 2022 13:46:39 +0100 Subject: Timezone for datetime.date objects In-Reply-To: <20220302171754.a2w3v4b2uyryj4sz@hjp.at> References: <20220302171754.a2w3v4b2uyryj4sz@hjp.at> Message-ID: On Wed, Mar 2, 2022 at 6:20 PM Peter J. Holzer wrote: > On 2022-02-28 23:28:23 +0100, Morten W. Petersen wrote: > > Well, let's say I specify the datetime 2022-02-22 02:02 (AM). I think > > everyone could agree that it also means 2022-02-22 02:02:00:00, to > > 2022-02-22 02:02:59:59. > > I disagree. The datetime 2022-02-22 02:02 specifies a point in time, not > a time period. It means 2022-02-22 02:02:00.000000000..... > > In reality, a point in time may be fuzzy. "The train departs at 02:02" > definitely doesn't mean that the train will depart exactly at 02:02:00, > but it also doesn't mean that it will depart between 02:02 and 02:03. > Rather it's a smooth probability distribution starting a bit before > 02:02:00 (a train should never leave early, but sometimes clocks are > wrong or somebody doesn't pay attention) a peak shortly after 02:02:00 > and a very long tail. > Well, the precision we have is probably nowhere as close as it will be possible to specify in the future, so it depends on context. But I agree on the train departing time, it should never be before the scheduled time, and yes, probably peaks right after the departure time. If I say I'll meet someone at 2 o'clock, that means 14:00, coming 5 minutes early or late is acceptable. So saying that implies a range in the given context. If I have a bill with a due date, it means I have to pay it on that date or earlier to do it correctly. Or even a couple of days later, if the due date is a holiday and it's not possible to make a bank transfer that day. But with crypto currencies it is. :) If I say I'll do something on the 15th of March, it would be reasonable to think that means between 06:00 and 21:00 on that day, but having it done at any time that day would be correct. So I think the correct interpretation depends on context, and if I say 14:02, in a computer and programming context, that means 14:02:00 to 14:02:59, because the seconds aren't specified, and 14:01 would not be a match, and 14:03 would also not be a match. If I had specified in terms of seconds into that day >>> (14*60*60)+(2*60) 50520 the implied and correct range would have been 50520.0 to 50520.99999. > > > And I think the same applies for a date. > > Depends on the context, but without s specific context (like business > days) I would agree. A day *is* a time period with a beginning and an > end. > Yeah, more or less what I'm saying above. Regards, Morten -- I am https://leavingnorway.info Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ From hjp-python at hjp.at Sun Mar 6 12:28:59 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 6 Mar 2022 18:28:59 +0100 Subject: Getting Syslog working on OSX Monterey In-Reply-To: <64FFB4FB-F4BD-4E1A-AAC7-792A26D65EB6@barrys-emacs.org> References: <0D40B574-2975-49BD-9E97-9055317169E5@barrys-emacs.org> <20220228214109.ntoihdobzf5naqrx@hjp.at> <4EAF5AC0-446D-4493-AFF0-2FD8A8889439@barrys-emacs.org> <20220304212307.bd3rj67m6gaej37a@hjp.at> <64FFB4FB-F4BD-4E1A-AAC7-792A26D65EB6@barrys-emacs.org> Message-ID: <20220306172859.kirgy6zgpagbkkon@hjp.at> On 2022-03-05 16:25:38 +0000, Barry Scott wrote: > On 4 Mar 2022, at 21:23, Peter J. Holzer wrote: > > If you are saying that SysLogHandler should use a system specific > > default (e.g. "/dev/log" on Linux) instead of UDP port 514 everywhere, I > > agree 99 % (the remaining 1 % is my contrarian alter ego arguing that > > that's really the distribution maintainer's job since a Linux > > distribution might use some other socket). > > > > If you are saying it should use the libc syslog routines, I disagree for > > at least two reasons: a) they are OS specific, b) you can't configure the > > destination. So that would remove useful functionality. > > > > In any case it seems strange to me that you want to rewrite it just to > > avoid passing a single parameter to the constructor (or - more likely - > > adding a single line to a config file). > > What I am used to is a setup where programs use syslog() to log and > a dameon like rsyslog is responsible for routing the logs to local files > and/or remote systems. Yes, that's a pretty common setup. > On a modern linux I think the route is syslog() -> journald. And you > can have syslog() -> journald -> rsyslogd. Yes. Journald accepts syslog (RFC 5424) protocol on /dev/log and can optionally forward messages to another syslog daemon (but only via Unix sockets, AFAICS, not via UDP or TCP). rsyslog is then configured to accept those forwarded messages (and may also be configured to accept messages on port 514, but this isn't the case by default on Debian or Ubuntu). > Using the syslog() function means that any platform/distro details are > hidden from the user of syslog() and as is the case of macOS it > "just works". That doesn't seem to be case. Philip Bloom reported in , that syslog in Python 3.6 broke in OS X Monterey. So even using syslog(3) doesn't seem to be safe across upgrades (no, I don't know how Apple managed to break this - as I wrote, I don't use OS X). > (I assume, not checked, that the write to the socket does not work > because Apple is not implementing the syslog protocol from the RFC). > > > > >> What you do not see used in the SyslogHandler() is the import syslog > >> and hence its nor using openlog() etc from syslog API. > > > > For good reasons. The C syslog API is missing important functionality. > > What are you think about being missing? Just curious. As I already wrote: * System independence. The libc syslog(3) function (and the Python syslog package using it) exists only on Unix-like OSs. Not Windows. Probably not some embedded platforms. SyslogHandler works on all platforms with networking. * Configurable targets. The SyslogHandler allows you to specify where to send the messages. syslog(3) does not. So with syslog(3) you always need a locally running daemon. In containers or embedded systems this might add extra unnecessary complexity. > As an aside if I had the need to log into a system log mechanism I'd be > looking to use the jounald API so that I can use structured logging on linux > systems. That might be useful. Journald already adds a bit of extra information to messages received over the syslog, but using the API directly would probably give the user more control. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From grant.b.edwards at gmail.com Sun Mar 6 12:29:19 2022 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Sun, 06 Mar 2022 09:29:19 -0800 (PST) Subject: Behavior of the for-else construct References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304225746.mmebv3myg5wbpc4f@hjp.at> <657845041.42944.1646437629325@mail.yahoo.com> <20220305001158.g7rmlyoxxtxufpfp@hjp.at> <1137940845.144932.1646516408119@mail.yahoo.com> Message-ID: <6224ef6f.1c69fb81.6228c.7238@mx.google.com> On 2022-03-05, Avi Gross via Python-list wrote: > I am not sure how we end up conversing about PASCAL on a Python > forum. > [...] > I paid no attention to where PASCAL was being used other than I did > much of my grad school work in PASCAL [...] It's "Pascal". It's not an acronym. It's a guy's name: https://en.wikipedia.org/wiki/Blaise_Pascal -- Grant From hjp-python at hjp.at Sun Mar 6 12:48:18 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 6 Mar 2022 18:48:18 +0100 Subject: Behavior of the for-else construct In-Reply-To: <6224ef6f.1c69fb81.6228c.7238@mx.google.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304225746.mmebv3myg5wbpc4f@hjp.at> <657845041.42944.1646437629325@mail.yahoo.com> <20220305001158.g7rmlyoxxtxufpfp@hjp.at> <1137940845.144932.1646516408119@mail.yahoo.com> <6224ef6f.1c69fb81.6228c.7238@mx.google.com> Message-ID: <20220306174818.alasogxuqxhgaprr@hjp.at> On 2022-03-06 09:29:19 -0800, Grant Edwards wrote: > On 2022-03-05, Avi Gross via Python-list wrote: > > I am not sure how we end up conversing about PASCAL on a Python > > forum. > > [...] > > I paid no attention to where PASCAL was being used other than I did > > much of my grad school work in PASCAL [...] > > It's "Pascal". It's not an acronym. It's a guy's name: > > https://en.wikipedia.org/wiki/Blaise_Pascal And similarly, it's "Perl", not "PERL" (also misspelled in this thread). hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From dieter at handshake.de Sun Mar 6 13:12:25 2022 From: dieter at handshake.de (Dieter Maurer) Date: Sun, 6 Mar 2022 19:12:25 +0100 Subject: Execute in a multiprocessing child dynamic code loaded by the parent process In-Reply-To: <20220306124208.orqbcezhqqeysdpe@gmail.com> References: <20220306124208.orqbcezhqqeysdpe@gmail.com> Message-ID: <25124.63881.205740.249595@ixdm.fritz.box> Martin Di Paola wrote at 2022-3-6 12:42 +0000: >Hi everyone. I implemented time ago a small plugin engine to load code >dynamically. > >So far it worked well but a few days ago an user told me that he wasn't >able to run in parallel a piece of code in MacOS. > >He was using multiprocessing.Process to run the code and in MacOS, the >default start method for such process is using "spawn". My understanding >is that Python spawns an independent Python server (the child) which >receives what to execute (the target function) from the parent process. > >In pseudo code this would be like: > >modules = loader() # load the plugins (Python modules at the end) >objs = init(modules) # initialize the plugins > ># One of the plugins wants to execute part of its code in parallel ># In MacOS this fails >ch = multiprocessing.Process(target=objs[0].sayhi) >ch.start() > >The code fails with "ModuleNotFoundError: No module named 'foo'" (where >'foo' is the name of the loaded plugin). > >This is because the parent program sends to the serve (the child) what >needs to execute (objs[0].sayhi) using pickle as the serialization >mechanism. > >Because Python does not really serialize code but only enough >information to reload it, the serialization of "objs[0].sayhi" just >points to its module, "foo". > >Module which it cannot be imported by the child process. > >So the question is, what would be the alternatives and workarounds? Try to use `fork` as "start method" (instead of "spawn"). From hjp-python at hjp.at Sun Mar 6 14:35:44 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 6 Mar 2022 20:35:44 +0100 Subject: Getting Syslog working on OSX Monterey In-Reply-To: <20220306172859.kirgy6zgpagbkkon@hjp.at> References: <0D40B574-2975-49BD-9E97-9055317169E5@barrys-emacs.org> <20220228214109.ntoihdobzf5naqrx@hjp.at> <4EAF5AC0-446D-4493-AFF0-2FD8A8889439@barrys-emacs.org> <20220304212307.bd3rj67m6gaej37a@hjp.at> <64FFB4FB-F4BD-4E1A-AAC7-792A26D65EB6@barrys-emacs.org> <20220306172859.kirgy6zgpagbkkon@hjp.at> Message-ID: <20220306193544.6m5s3srx5norodlt@hjp.at> On 2022-03-06 18:28:59 +0100, Peter J. Holzer wrote: > On 2022-03-05 16:25:38 +0000, Barry Scott wrote: > > Using the syslog() function means that any platform/distro details are > > hidden from the user of syslog() and as is the case of macOS it > > "just works". > > That doesn't seem to be case. Philip Bloom reported in > , > that syslog in Python 3.6 broke in OS X Monterey. So even using > syslog(3) doesn't seem to be safe across upgrades (no, I don't know > how Apple managed to break this - as I wrote, I don't use OS X). > > > > (I assume, not checked, that the write to the socket does not work > > because Apple is not implementing the syslog protocol from the RFC). I noticed that RFC 5424 describes a very different protocol from RFC 3164 and was curious which one SysLogHandler implements. Surprise: Neither. It just sends the PRI part and the MSG part. While that neatly sidesteps the question of which of the two incompatible header formats to use, it's not a a valid syslog message in either format. Obviously Linux accepts this format (the timestamp and the hostname are kind of redundant anyway when you're using a unix socket). I strongly suspect that the OS X syslog daemon got stricter and now insists on properly formatted messages. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From martinp.dipaola at gmail.com Sun Mar 6 15:36:01 2022 From: martinp.dipaola at gmail.com (Martin Di Paola) Date: Sun, 6 Mar 2022 20:36:01 +0000 Subject: Execute in a multiprocessing child dynamic code loaded by the parent process In-Reply-To: References: <20220306124208.orqbcezhqqeysdpe@gmail.com> Message-ID: <20220306203601.eqv7geya3l7joxrj@gmail.com> > >The way you've described it, it's a hack. Allow me to slightly redescribe it. > >modules = loader() >objs = init(modules) > >def invoke(mod, func): > # I'm assuming that the loader is smart enough to not load > # a module that's already loaded. Alternatively, load just the > # module you need, if that's a possibility. > loader() > target = getattr(modules[mod], func) > target() > >ch = multiprocessing.Process(target=invoke, args=("some_module", "sayhi")) >ch.start() > Yeup, that would be my first choice but the catch is that "sayhi" may not be a function of the given module. It could be a static method of some class or any other callable. And doing the lookup by hand sounds complex. The thing is that the use of multiprocessing is not something required by me (by my plugin-engine), it was a decision of the developer of a particular plugin so I don't have any control on that. Using multiprocessing.reduction was a practical decision: if the user wants to call something non-pickleable, it is not my fault, it is multiprocessing's fault. It *would* be my fault if multiprocessing.Process fails only because I'm loading the code dynamically. > [...] I won't say "the" correct way, as there are other valid >ways, but there's certainly nothing wrong with this idea. Do you have some in mind? Or may be a project that I could read? Thanks! Martin From martinp.dipaola at gmail.com Sun Mar 6 15:42:28 2022 From: martinp.dipaola at gmail.com (Martin Di Paola) Date: Sun, 6 Mar 2022 20:42:28 +0000 Subject: Execute in a multiprocessing child dynamic code loaded by the parent process In-Reply-To: <25124.63881.205740.249595@ixdm.fritz.box> References: <20220306124208.orqbcezhqqeysdpe@gmail.com> <25124.63881.205740.249595@ixdm.fritz.box> Message-ID: <20220306204228.l5pzjzc6peytbshp@gmail.com> >Try to use `fork` as "start method" (instead of "spawn"). Yes but no. Indeed with `fork` there is no need to pickle anything. In particular the child process will be a copy of the parent so it will have all the modules loaded, including the dynamic ones. Perfect. The problem is that `fork` is the default only in Linux. It works in MacOS but it may lead to crashes if the parent process is multithreaded (and the my is!) and `fork` does not work in Windows. From rosuav at gmail.com Sun Mar 6 16:13:05 2022 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 7 Mar 2022 08:13:05 +1100 Subject: Execute in a multiprocessing child dynamic code loaded by the parent process In-Reply-To: <20220306203601.eqv7geya3l7joxrj@gmail.com> References: <20220306124208.orqbcezhqqeysdpe@gmail.com> <20220306203601.eqv7geya3l7joxrj@gmail.com> Message-ID: On Mon, 7 Mar 2022 at 07:37, Martin Di Paola wrote: > > > > > > >The way you've described it, it's a hack. Allow me to slightly redescribe it. > > > >modules = loader() > >objs = init(modules) > > > >def invoke(mod, func): > > # I'm assuming that the loader is smart enough to not load > > # a module that's already loaded. Alternatively, load just the > > # module you need, if that's a possibility. > > loader() > > target = getattr(modules[mod], func) > > target() > > > >ch = multiprocessing.Process(target=invoke, args=("some_module", "sayhi")) > >ch.start() > > > > Yeup, that would be my first choice but the catch is that "sayhi" may > not be a function of the given module. It could be a static method of > some class or any other callable. Ah, fair. Are you able to define it by a "path", where each step in the path is a getattr() call? The trouble is, arbitrary callables might not be available in a reconstructed version of the module. > Using multiprocessing.reduction was a practical decision: if the user > wants to call something non-pickleable, it is not my fault, it is > multiprocessing's fault. > > It *would* be my fault if multiprocessing.Process fails only because I'm > loading the code dynamically. Fair. I guess, then, that the best thing to do is to preload the modules, then unpickle. So, basically what you already have, but with more caveats. > Do you have some in mind? Or may be a project that I could read? Not handy, but there are always many different ways to do things. For instance, instead of saying "spawn a subprocess and call this function", you could invert it, and have the function register itself as the target. Then it's just "spawn a subprocess and load this module", and that calls the registered invocation. It all depends on what the rest of your project is doing. Mainly, though, I'm just not ruling out the possibility of other options :) ChrisA From barry at barrys-emacs.org Sun Mar 6 17:04:43 2022 From: barry at barrys-emacs.org (Barry) Date: Sun, 6 Mar 2022 22:04:43 +0000 Subject: virtualenv and make DESTDIR= In-Reply-To: <20220306165055.xaflsr7lczdzabfa@hjp.at> References: <20220306165055.xaflsr7lczdzabfa@hjp.at> Message-ID: <30C773FD-B546-44F9-93A7-AC221063B146@barrys-emacs.org> > On 6 Mar 2022, at 16:53, Peter J. Holzer wrote: > > ?On 2022-03-05 17:59:48 +0100, Marco Sulla wrote: >>> On Sat, 5 Mar 2022 at 17:36, Barry Scott wrote: >>> Note: you usually cannot use pip when building an RPM with mock as the network is disabled inside the build for >>> security reasons. >> >> Can't he previously download the packages and run pip on the local packages? > > That shouldn't be necessary. Any packages needed by that new package > should be dependencies of that package, so they must also already exist > as RPMs. The rpm build process will automatically install them if they > aren't already. > > (That may not be true for build dependencies, though.) In the spec file use BuildRequires: to pull in an rpm package that is needed to do the build. For example python3-setuptools. Barry > > hp > > -- > _ | Peter J. Holzer | Story must make more sense than reality. > |_|_) | | > | | | hjp at hjp.at | -- Charles Stross, "Creative writing > __/ | http://www.hjp.at/ | challenge!" > -- > https://mail.python.org/mailman/listinfo/python-list From barry at barrys-emacs.org Sun Mar 6 17:11:30 2022 From: barry at barrys-emacs.org (Barry) Date: Sun, 6 Mar 2022 22:11:30 +0000 Subject: Getting Syslog working on OSX Monterey In-Reply-To: <20220306193544.6m5s3srx5norodlt@hjp.at> References: <20220306193544.6m5s3srx5norodlt@hjp.at> Message-ID: <7D7A977C-9D9B-475B-8DB2-004548C2FECA@barrys-emacs.org> > On 6 Mar 2022, at 19:38, Peter J. Holzer wrote: > > ?On 2022-03-06 18:28:59 +0100, Peter J. Holzer wrote: >>> On 2022-03-05 16:25:38 +0000, Barry Scott wrote: >>> Using the syslog() function means that any platform/distro details are >>> hidden from the user of syslog() and as is the case of macOS it >>> "just works". >> >> That doesn't seem to be case. Philip Bloom reported in >> , >> that syslog in Python 3.6 broke in OS X Monterey. So even using >> syslog(3) doesn't seem to be safe across upgrades (no, I don't know >> how Apple managed to break this - as I wrote, I don't use OS X). >> >> >>> (I assume, not checked, that the write to the socket does not work >>> because Apple is not implementing the syslog protocol from the RFC). > > I noticed that RFC 5424 describes a very different protocol from RFC > 3164 and was curious which one SysLogHandler implements. > > Surprise: Neither. So the logging code works by accident? > > It just sends the PRI part and the MSG part. While that neatly sidesteps > the question of which of the two incompatible header formats to use, > it's not a a valid syslog message in either format. > > Obviously Linux accepts this format (the timestamp and the hostname are > kind of redundant anyway when you're using a unix socket). > > I strongly suspect that the OS X syslog daemon got stricter and now > insists on properly formatted messages. Interesting. If I can find a spare hour I can play with this. Barry > > hp > > > -- > _ | Peter J. Holzer | Story must make more sense than reality. > |_|_) | | > | | | hjp at hjp.at | -- Charles Stross, "Creative writing > __/ | http://www.hjp.at/ | challenge!" > -- > https://mail.python.org/mailman/listinfo/python-list From avigross at verizon.net Sun Mar 6 17:50:01 2022 From: avigross at verizon.net (Avi Gross) Date: Sun, 6 Mar 2022 22:50:01 +0000 (UTC) Subject: Behavior of the for-else construct In-Reply-To: <20220306174818.alasogxuqxhgaprr@hjp.at> References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304225746.mmebv3myg5wbpc4f@hjp.at> <657845041.42944.1646437629325@mail.yahoo.com> <20220305001158.g7rmlyoxxtxufpfp@hjp.at> <1137940845.144932.1646516408119@mail.yahoo.com> <6224ef6f.1c69fb81.6228c.7238@mx.google.com> <20220306174818.alasogxuqxhgaprr@hjp.at> Message-ID: <222574152.213889.1646607001131@mail.yahoo.com> >>> Pascal versus PASCAL versus pascal (not versus paschal) and Perl versus PERL versus perl (not versus pearl) seems to be a topic. <<< The nitpickers here are irrelevant. I happen to know how things are formally spelled and if I was publishing a book, might carefully proofread it. I sometimes use upper case for Emphasis and do not give a dAMN. But feel free to correct my spelling in case you suspect someone here truly misunderstood a message because it was written with the wrong case. I do learn, but when it is something I already know and am not particularly interested in proofreading, the result tends to be that I simply get annoyed and the one pointing it out is taken as wandering off topic even more than I do. Python is named after a snake right? So is it upper case or lower case or mixed case? No, wait, it is named after Monty Python. But what made them choose that name and why would anyone then name a programming language after them? I can see some rudimentary reasoning in naming Ada and maybe for Pascal, albeit I can think of many others who deserve it as much or more that nobody has chosen to honor. Did Turing catch on, I mean as a programming language rather than a machine? I don't care if someone says PYTHON or Python or python when informally discussing things. They all mean the same thing to me. On the other hand, names like "R" and "S" and "C" are rarely presented as lower-case while I sometimes see c++ or c# for some reason. Obviously the right version is upper case, albeit the plus sign and sharp sign have no upper case version. FORTRAN supposedly stands for FORmula TRANslator or something. So why does it have to be all uppercase? Was COBOL named for COmmon Business Oriented Language. Should it be CoBOL? I have lots of awareness that Perl is generally written in mixed case, except when it isn't. I have seen people justify the name by saying it was short for Practical Extraction and Report Language but also for Pathologically Eclectic Rubbish Lister! If I felt it was important, I would write it whatever way is formally required. What I think often happens, from a human psychology perspective, is we see Perl in middle of a sentence as a spelling mistake while PERL is obviously something like an acronym. Pascal may be named for a French person I admire but may still look like a nonsense word to those who do not know or care. Not a great excuse, of course. But if this was being graded in these ways, and I see lots of such nitpicking in other ways when an example focusing on something does not carefully do lots of other things, I probably would focus on the many other things I don't get around to doing in my life because I waste the time here. As a voluntary effort, I can opt out. The argument that a programming language is named after a person and thus must be spelled some way is not impressing me. Yes, to be strictly correct, it has a proper spelling. But following that reasoning, why does anyone give an email address of john.smith at gmail.com or JANEDOE at yahoo.com instead of ...? -----Original Message----- From: Peter J. Holzer To: python-list at python.org Sent: Sun, Mar 6, 2022 12:48 pm Subject: Re: Behavior of the for-else construct On 2022-03-06 09:29:19 -0800, Grant Edwards wrote: > On 2022-03-05, Avi Gross via Python-list wrote: > > I am not sure how we end up conversing about PASCAL on a Python > > forum. > > [...] > > I paid no attention to where PASCAL was being used other than I did > > much of my grad school work in PASCAL [...] > > It's "Pascal". It's not an acronym. It's a guy's name: > >? ? https://en.wikipedia.org/wiki/Blaise_Pascal And similarly, it's "Perl", not "PERL" (also misspelled in this thread). ? ? ? ? hp -- ? _? | Peter J. Holzer? ? | Story must make more sense than reality. |_|_) |? ? ? ? ? ? ? ? ? ? | | |? | hjp at hjp.at? ? ? ? |? ? -- Charles Stross, "Creative writing __/? | http://www.hjp.at/ |? ? ? challenge!" -- https://mail.python.org/mailman/listinfo/python-list From rosuav at gmail.com Sun Mar 6 18:04:45 2022 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 7 Mar 2022 10:04:45 +1100 Subject: Behavior of the for-else construct In-Reply-To: <222574152.213889.1646607001131@mail.yahoo.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304225746.mmebv3myg5wbpc4f@hjp.at> <657845041.42944.1646437629325@mail.yahoo.com> <20220305001158.g7rmlyoxxtxufpfp@hjp.at> <1137940845.144932.1646516408119@mail.yahoo.com> <6224ef6f.1c69fb81.6228c.7238@mx.google.com> <20220306174818.alasogxuqxhgaprr@hjp.at> <222574152.213889.1646607001131@mail.yahoo.com> Message-ID: On Mon, 7 Mar 2022 at 09:51, Avi Gross via Python-list wrote: > > >>> > Pascal versus PASCAL versus pascal (not versus paschal) and > Perl versus PERL versus perl (not versus pearl) > > seems to be a topic. > <<< > > The nitpickers here are irrelevant. I happen to know how things are formally spelled and if I was publishing a book, might carefully proofread it. So what you're saying is that you don't really care unless it's in a book. Good to know. I make errors now and then, but I consider them to be errors, and I would like to reduce their number. > Python is named after a snake right? So is it upper case or lower case or mixed case? No, wait, it is named after Monty Python. But what made them choose that name and why would anyone then name a programming language after them? I can see some rudimentary reasoning in naming Ada and maybe for Pascal, albeit I can think of many others who deserve it as much or more that nobody has chosen to honor. Did Turing catch on, I mean as a programming language rather than a machine? > If you want strange naming conventions, apparently there's "Apache Pig Latin", "Ballerina", "Hopscotch", "Mercury", and even this bizarre thing called "ECMAScript", can't possibly imagine why anyone would use a thing like that, right? Things get named, usually by their creators. Deliberately misnaming something is insulting it. > I don't care if someone says PYTHON or Python or python when informally discussing things. They all mean the same thing to me. On the other hand, names like "R" and "S" and "C" are rarely presented as lower-case while I sometimes see c++ or c# for some reason. Obviously the right version is upper case, albeit the plus sign and sharp sign have no upper case version. > > FORTRAN supposedly stands for FORmula TRANslator or something. So why does it have to be all uppercase? Was COBOL named for COmmon Business Oriented Language. Should it be CoBOL? > Because that's what their creators decided. Or are you in a more authoritative position? I think you should have been named AVI144, not Avi Gross. We shouldn't bother to try to call you by your name, it's much more reasonable to call you something else. > But if this was being graded in these ways, and I see lots of such nitpicking in other ways when an example focusing on something does not carefully do lots of other things, I probably would focus on the many other things I don't get around to doing in my life because I waste the time here. As a voluntary effort, I can opt out. > > The argument that a programming language is named after a person and thus must be spelled some way is not impressing me. Yes, to be strictly correct, it has a proper spelling. But following that reasoning, why does anyone give an email address of john.smith at gmail.com or JANEDOE at yahoo.com instead of ...? > Email addresses consist of a mailbox part, interpreted entirely by the server, and a domain, which is defined as case insensitive and conventionally written in lowercase. There's no difference between john.smith at gmail.com and john.smith at GMAIL.COM, but it's up to Google's servers to decide whether john.smith is the same as JOHN.SMITH (and, in the case of Google, they've decided that it's the same as johnsmith and johns.mith too). ChrisA From drsalists at gmail.com Sun Mar 6 18:29:42 2022 From: drsalists at gmail.com (Dan Stromberg) Date: Sun, 6 Mar 2022 15:29:42 -0800 Subject: mac app from a python script? In-Reply-To: References: Message-ID: On Sun, Jan 23, 2022 at 9:59 AM Dan Stromberg wrote: > > Hi folks. > > I have a Python 3 script (built on top of gi.respository.Gtk) that runs on > Linux and macOS 11.5. It's at > https://stromberg.dnsalias.org/~strombrg/hcm/ if you're curious. > > It works the way I want on Linux, but on macOS I seem to have to start it > from the command line, like: > hcm --gui > ...because I don't know how to create a macOS "app" that goes under > /Applications. > > I don't really care about having a single executable on macOS, and I don't > really care about creating a .dmg or .pkg file. I'd be perfectly happy just > running "make install" and putting a #!'d script under /Applications with > appropriate metadata - but if it's easier to do a single executable, .dmg > or .pkg, I'd be fine with that. > It turns out that using something called "appify" does this well. I found it on a gist I believe. I liked it so much that I adopted it and put it at https://stromberg.dnsalias.org/~strombrg/mactools/ , with a few style changes and small bug fixes. From wlfraed at ix.netcom.com Sun Mar 6 13:29:12 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Sun, 06 Mar 2022 13:29:12 -0500 Subject: Behavior of the for-else construct References: <621325684.471007.1646354302946@mail.yahoo.com> <20220304225746.mmebv3myg5wbpc4f@hjp.at> <657845041.42944.1646437629325@mail.yahoo.com> <20220305001158.g7rmlyoxxtxufpfp@hjp.at> <1mc72hll06itd6jnbgdherqb3thf1fk1pr@4ax.com> <20220306163951.2ozmrhfbtsktbbt7@hjp.at> Message-ID: On Sun, 6 Mar 2022 17:39:51 +0100, "Peter J. Holzer" declaimed the following: > >(* *) for comments was actually pretty commonly used - maybe because it >stands out more than { }. I don't know if I've ever seen (. .) instead >of [ ]. > Or some terminals provided [ ] but not { } Modula-2 appears to have fixed on (* *) for comments, and only [ ] for indexing. Consider the potential mayhem going from a language where { } are comment delimiters to one where they are block delimiters >C also has alternative rerpresentations for characters not in the common >subset of ISO-646 and EBCDIC. However, the trigraphs are extremely ugly >(e.g ??< ??> instead of { }). I have seen them used (on an IBM/390 >system with an EBCDIC variant without curly braces) and it's really no >fun to read that. > My college mainframe used EBCDIC, but the available languages did not include C or Pascal. We had APL, FORTRAN-IV (in full separate compilation form, and FLAG [FORTRAN Load and Go] which was a "all in one file, compile & run" used by first year students), COBOL (74?), BASIC, SNOBOL, Meta-Symbol and AP (both assemblers, though Meta-Symbol could, provided the proper definition file, generate absolute binary code for pretty much any processor), and something called SL-1 (Simulation Language-1, which produced FORTRAN output for discrete event models). UCSD Pascal, and PDP-11 assembly were run on a pair of LSI-11 systems. Assembly used for the operating system principles course. I didn't encounter "real" C until getting a TRS-80 (first as integer LC, then Pro-MC), along with Supersoft LISP (on cassette tape!). (I had books for C and Ada before encountering compilers for them) -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From greg.ewing at canterbury.ac.nz Sun Mar 6 17:32:27 2022 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 7 Mar 2022 11:32:27 +1300 Subject: Execute in a multiprocessing child dynamic code loaded by the parent process In-Reply-To: References: <20220306124208.orqbcezhqqeysdpe@gmail.com> <20220306203601.eqv7geya3l7joxrj@gmail.com> Message-ID: On 7/03/22 9:36 am, Martin Di Paola wrote: > It *would* be my fault if multiprocessing.Process fails only because I'm > loading the code dynamically. I'm not so sure about that. The author of the plugin knows they're writing code that will be dynamically loaded, and can therefore expect the kind of problem they're having. It could be argued that it's their responsibility to ensure that all the needed code is loaded into the subprocess. -- Greg From jenkris at tutanota.com Sun Mar 6 19:32:11 2022 From: jenkris at tutanota.com (Jen Kris) Date: Mon, 7 Mar 2022 01:32:11 +0100 (CET) Subject: C API PyObject_CallFunctionObjArgs returns incorrect result Message-ID: I am using the C API in Python 3.8 with the nltk library, and I have a problem with the return from a library call implemented with PyObject_CallFunctionObjArgs.? This is the relevant Python code: import nltk from nltk.corpus import gutenberg fileids = gutenberg.fileids() sentences = gutenberg.sents(fileids[0]) sentence = sentences[0] sentence = " ".join(sentence) pt = nltk.word_tokenize(sentence) I run this at the Python command prompt to show how it works: >>> sentence = " ".join(sentence) >>> pt = nltk.word_tokenize(sentence) >>> print(pt) ['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']'] >>> type(pt) This is the relevant part of the C API code: PyObject* str_sentence = PyObject_Str(pSentence);?? // nltk.word_tokenize(sentence)?? PyObject* pNltk_WTok = PyObject_GetAttrString(pModule_mstr, "word_tokenize"); PyObject* pWTok = PyObject_CallFunctionObjArgs(pNltk_WTok, str_sentence, 0); (where pModule_mstr is the nltk library).? That should produce a list with a length of 7 that looks like it does on the command line version shown above: ['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']'] But instead the C API produces a list with a length of 24, and the REPR looks like this: '[\'[\', "\'", \'[\', "\'", \',\', "\'Emma", "\'", \',\', "\'by", "\'", \',\', "\'Jane", "\'", \',\', "\'Austen", "\'", \',\', "\'1816", "\'", \',\', "\'", \']\', "\'", \']\']' I also tried this with PyObject_CallMethodObjArgs and PyObject_Call without success.? Thanks for any help on this.? Jen From python at mrabarnett.plus.com Sun Mar 6 20:42:05 2022 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 7 Mar 2022 01:42:05 +0000 Subject: C API PyObject_CallFunctionObjArgs returns incorrect result In-Reply-To: References: Message-ID: <5ad962fc-1257-dd8d-96ab-541ae5bae2fa@mrabarnett.plus.com> On 2022-03-07 00:32, Jen Kris via Python-list wrote: > I am using the C API in Python 3.8 with the nltk library, and I have a problem with the return from a library call implemented with PyObject_CallFunctionObjArgs. > > This is the relevant Python code: > > import nltk > from nltk.corpus import gutenberg > fileids = gutenberg.fileids() > sentences = gutenberg.sents(fileids[0]) > sentence = sentences[0] > sentence = " ".join(sentence) > pt = nltk.word_tokenize(sentence) > > I run this at the Python command prompt to show how it works: >>>> sentence = " ".join(sentence) >>>> pt = nltk.word_tokenize(sentence) >>>> print(pt) > ['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']'] >>>> type(pt) > > > This is the relevant part of the C API code: > > PyObject* str_sentence = PyObject_Str(pSentence); > // nltk.word_tokenize(sentence) > PyObject* pNltk_WTok = PyObject_GetAttrString(pModule_mstr, "word_tokenize"); > PyObject* pWTok = PyObject_CallFunctionObjArgs(pNltk_WTok, str_sentence, 0); > > (where pModule_mstr is the nltk library). > > That should produce a list with a length of 7 that looks like it does on the command line version shown above: > > ['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']'] > > But instead the C API produces a list with a length of 24, and the REPR looks like this: > > '[\'[\', "\'", \'[\', "\'", \',\', "\'Emma", "\'", \',\', "\'by", "\'", \',\', "\'Jane", "\'", \',\', "\'Austen", "\'", \',\', "\'1816", "\'", \',\', "\'", \']\', "\'", \']\']' > > I also tried this with PyObject_CallMethodObjArgs and PyObject_Call without success. > > Thanks for any help on this. > What is pSentence? Is it what you think it is? To me it looks like it's either the list: ['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']'] or that list as a string: "['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']']" and that what you're tokenising. From martinp.dipaola at gmail.com Sun Mar 6 21:13:17 2022 From: martinp.dipaola at gmail.com (Martin Di Paola) Date: Mon, 7 Mar 2022 02:13:17 +0000 Subject: Execute in a multiprocessing child dynamic code loaded by the parent process In-Reply-To: References: <20220306124208.orqbcezhqqeysdpe@gmail.com> <20220306203601.eqv7geya3l7joxrj@gmail.com> Message-ID: <20220307021317.yncapixpaxlxbzb5@gmail.com> >I'm not so sure about that. The author of the plugin knows they're >writing code that will be dynamically loaded, and can therefore >expect the kind of problem they're having. It could be argued that >it's their responsibility to ensure that all the needed code is loaded >into the subprocess. Yes but I try to always make my libs/programs as much as usable as possible. "Ergonomic" would be the word. In the case of the plugin-engine I'm trying to hide any side-effect or unexpected behaviour of the engine so the developer of the plugin does not have take that into account. I agree that if the developer uses multiprocessing he/she needs to know its implications. But if I can "smooth" any rough corner, I will try to do it. For example, the main project (developed by me) uses threads for concurrency. It would be simpler to load the plugins and instantiate them *once* and ask the plugins developers to take care of any race condition (RC) within their implementation. Because the plugins were instantiated *once*, it is almost guaranteed that the plugins will suffer from race conditions and they will require some sort of locking. This is quite risky: you may forget to protect something and you will end up with a RC and/or you may put the lock in the wrong place and the whole thing will not work concurrently. My decision back then was to instantiate each plugin N+1 times: once in the main thread and then once per worker thread. With this, no single plugin instance will be shared so there is no risk of RC and no need for locking. (Yes, I know, the developer just needs to use a module variable or a class attribute and it will get a RC and these are shared but it is definitely not the default scenario). If sharing is required I provide an object that minimizes the locking needed. It was much complex for me at the design and at the implementation level but I think that it is safer and requires less from the plugin developer. Reference: https://byexamples.github.io/byexample/contrib/concurrency-model From martinp.dipaola at gmail.com Sun Mar 6 21:30:17 2022 From: martinp.dipaola at gmail.com (Martin Di Paola) Date: Mon, 7 Mar 2022 02:30:17 +0000 Subject: Execute in a multiprocessing child dynamic code loaded by the parent process In-Reply-To: References: <20220306124208.orqbcezhqqeysdpe@gmail.com> <20220306203601.eqv7geya3l7joxrj@gmail.com> Message-ID: <20220307023017.qjcqbn7znoour6rm@gmail.com> >> >> Yeup, that would be my first choice but the catch is that "sayhi" may >> not be a function of the given module. It could be a static method of >> some class or any other callable. > >Ah, fair. Are you able to define it by a "path", where each step in >the path is a getattr() call? Yes but I think that unpickle (pickle.loads()) does that plus importing any module needed in the path which it is handy because I can preload the plugins (modules) before the unpickle but the path may contain others more-standard modules as well. Something like "myplugin.re.match". unpickle should import 're' module automatically will it is loading the function "match". >Fair. I guess, then, that the best thing to do is to preload the >modules, then unpickle. So, basically what you already have, but with >more caveats. Yes, this will not be transparent for the user, just trying to minimize the changes needed. And it will require some documentation for those caveats. And tests. Thanks for the brainstorming! Martin. From grant.b.edwards at gmail.com Sun Mar 6 21:34:39 2022 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Sun, 06 Mar 2022 18:34:39 -0800 (PST) Subject: Behavior of the for-else construct References: <21739669.459456.1646348879560@mail.yahoo.com> <621325684.471007.1646354302946@mail.yahoo.com> <20220304225746.mmebv3myg5wbpc4f@hjp.at> <657845041.42944.1646437629325@mail.yahoo.com> <20220305001158.g7rmlyoxxtxufpfp@hjp.at> <1137940845.144932.1646516408119@mail.yahoo.com> <6224ef6f.1c69fb81.6228c.7238@mx.google.com> <20220306174818.alasogxuqxhgaprr@hjp.at> <222574152.213889.1646607001131@mail.yahoo.com> Message-ID: <62256f3f.1c69fb81.77d74.2adb@mx.google.com> On 2022-03-06, Avi Gross via Python-list wrote: > Python is named after a snake right? No. It's named after a comedy troupe. -- Grant From hjp-python at hjp.at Mon Mar 7 02:14:41 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Mon, 7 Mar 2022 08:14:41 +0100 Subject: Behavior of the for-else construct In-Reply-To: <62256f3f.1c69fb81.77d74.2adb@mx.google.com> References: <20220304225746.mmebv3myg5wbpc4f@hjp.at> <657845041.42944.1646437629325@mail.yahoo.com> <20220305001158.g7rmlyoxxtxufpfp@hjp.at> <1137940845.144932.1646516408119@mail.yahoo.com> <6224ef6f.1c69fb81.6228c.7238@mx.google.com> <20220306174818.alasogxuqxhgaprr@hjp.at> <222574152.213889.1646607001131@mail.yahoo.com> <62256f3f.1c69fb81.77d74.2adb@mx.google.com> Message-ID: <20220307071441.5w2dpyew62elyhqe@hjp.at> On 2022-03-06 18:34:39 -0800, Grant Edwards wrote: > On 2022-03-06, Avi Gross via Python-list wrote: > > Python is named after a snake right? > > No. It's named after a comedy troupe. He actually wrote that two sentences later. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From barry at barrys-emacs.org Mon Mar 7 03:28:15 2022 From: barry at barrys-emacs.org (Barry) Date: Mon, 7 Mar 2022 08:28:15 +0000 Subject: Execute in a multiprocessing child dynamic code loaded by the parent process In-Reply-To: <20220307023017.qjcqbn7znoour6rm@gmail.com> References: <20220307023017.qjcqbn7znoour6rm@gmail.com> Message-ID: <7E685F22-DEA2-431B-ACDC-7FB0AC933B01@barrys-emacs.org> > On 7 Mar 2022, at 02:33, Martin Di Paola wrote: > > Yes but I think that unpickle (pickle.loads()) does that plus > importing any module needed Are you sure that unpickle will import code? I thought it did not do that. Barry From martinp.dipaola at gmail.com Mon Mar 7 08:32:44 2022 From: martinp.dipaola at gmail.com (Martin Di Paola) Date: Mon, 7 Mar 2022 13:32:44 +0000 Subject: Execute in a multiprocessing child dynamic code loaded by the parent process In-Reply-To: <7E685F22-DEA2-431B-ACDC-7FB0AC933B01@barrys-emacs.org> References: <20220307023017.qjcqbn7znoour6rm@gmail.com> <7E685F22-DEA2-431B-ACDC-7FB0AC933B01@barrys-emacs.org> Message-ID: <20220307133244.l46ficx6ebtytpdj@gmail.com> I understand that yes, pickle.loads() imports any necessary module but only if they can be find in sys.path (like in any "import" statement). Dynamic code loaded from a plugin (which we presume it is *not* in sys.path) will not be loaded. Quick check. Run in one console the following: import multiprocessing import multiprocessing.reduction import pickle pickle.dumps(multiprocessing.reduction.ForkingPickler) In a separated Python console run the following: import pickle import sys 'multiprocessing' in sys.modules False pickle.loads() 'multiprocessing' in sys.modules True So the last check proves that pickle.loads imports any necessary module. Martin. On Mon, Mar 07, 2022 at 08:28:15AM +0000, Barry wrote: > > >> On 7 Mar 2022, at 02:33, Martin Di Paola wrote: >> >> Yes but I think that unpickle (pickle.loads()) does that plus >> importing any module needed > >Are you sure that unpickle will import code? I thought it did not do that. > >Barry From grant.b.edwards at gmail.com Mon Mar 7 09:00:57 2022 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Mon, 07 Mar 2022 06:00:57 -0800 (PST) Subject: Behavior of the for-else construct References: <20220304225746.mmebv3myg5wbpc4f@hjp.at> <657845041.42944.1646437629325@mail.yahoo.com> <20220305001158.g7rmlyoxxtxufpfp@hjp.at> <1137940845.144932.1646516408119@mail.yahoo.com> <6224ef6f.1c69fb81.6228c.7238@mx.google.com> <20220306174818.alasogxuqxhgaprr@hjp.at> <222574152.213889.1646607001131@mail.yahoo.com> <62256f3f.1c69fb81.77d74.2adb@mx.google.com> <20220307071441.5w2dpyew62elyhqe@hjp.at> Message-ID: <62261019.1c69fb81.42653.45e0@mx.google.com> On 2022-03-07, Peter J. Holzer wrote: > On 2022-03-06 18:34:39 -0800, Grant Edwards wrote: >> On 2022-03-06, Avi Gross via Python-list wrote: >> > Python is named after a snake right? >> >> No. It's named after a comedy troupe. > > He actually wrote that two sentences later. Yes, I missed that. His messages wrap very strangely in my newsreader. -- Grant From gvm2121 at gmail.com Mon Mar 7 09:08:59 2022 From: gvm2121 at gmail.com (Gonzalo V) Date: Mon, 7 Mar 2022 11:08:59 -0300 Subject: always return the same pdf Message-ID: Hello everyone. i had upload a Django app to an ubuntu 18.04 server and it gives me the same pdf everytime the view is called. To generate the pdf it receipts differents string buy it gives me the same pdf. Could you give some idea what is happening? thanks everyone @never_cached def generar_pdf(request): prueba = request.session.get('contenedor') cantidad_preguntas=prueba['cantidad_preguntas'] archivo_salida = open("prueba.tex","w") archivo_salida.write("\\documentclass[10pt,oneside,letterpaper]{article}") archivo_salida.write("\\usepackage[utf8x]{inputenc}") ##hace mas y mas cosas sin importancia con latex que funcionan bien archivo_a_descargar = open("prueba.pdf","rb") # respuesta = HttpResponse(archivo_a_descargar,content_type='application/pdf') respuesta['Content-Disposition'] = 'attachment; filename="{0}"'.format( archivo_a_descargar.name) return respuesta Saludos, Gonzalo From robin at reportlab.com Mon Mar 7 11:22:10 2022 From: robin at reportlab.com (Robin Becker) Date: Mon, 7 Mar 2022 16:22:10 +0000 Subject: strange problem building non-pure wheel for apple M1 arm64 Message-ID: <589e22ac-28cb-b9f3-8be9-fb4988228929@everest.reportlab.co.uk> I use cibuildwheel to build extensions with a github action. For the macos 11.0 arm64 build I get a strange message from the load command. So I am looking for assistance to try and figure out what is going wrong. The cibuild action uses the latest pip 21.2.4 and latest setuptools etc. I use brew to install freetype version 2.11.1. The compilations look like this gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -g -arch arm64 -DRENDERPM_FT -DLIBART_COMPILATION -DLIBART_VERSION=2.3.21 -Isrc/rl_addons/renderPM -Isrc/rl_addons/renderPM/libart_lgpl -Isrc/rl_addons/renderPM/gt1 -I/usr/local/include/freetype2 -I/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/rl_addons/renderPM/_renderPM.c -o build/temp.macosx-11.0-arm64-3.9/src/rl_addons/renderPM/_renderPM.o this is the load command on multiple lines for readability the strange error is gcc -bundle -undefined dynamic_lookup -g -arch arm64 build/temp.macosx-11.0-arm64-3.9/src/rl_addons/renderPM/_renderPM.o build/temp.macosx-11.0-arm64-3.9/src/rl_addons/renderPM/gt1/gt1-dict.o build/temp.macosx-11.0-arm64-3.9/src/rl_addons/renderPM/gt1/gt1-namecontext.o '''''''other compiled code build/temp.macosx-11.0-arm64-3.9/src/rl_addons/renderPM/libart_lgpl/art_vpath_dash.o -L/usr/local/lib -L/usr/lib -L/Library/Frameworks/Python.framework/Versions/3.9/lib -lfreetype -o build/lib.macosx-11.0-arm64-3.9/reportlab/graphics/_renderPM.cpython-39-darwin.so ld: warning: ignoring file /usr/local/lib/libfreetype.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64 The above message seems bizarre; everything is compiled for arm64, but gcc doesn't want to use an arm64 dylib. Can macos experts assist? -- Robin Becker From jenkris at tutanota.com Mon Mar 7 12:05:30 2022 From: jenkris at tutanota.com (Jen Kris) Date: Mon, 7 Mar 2022 18:05:30 +0100 (CET) Subject: C API PyObject_CallFunctionObjArgs returns incorrect result In-Reply-To: <5ad962fc-1257-dd8d-96ab-541ae5bae2fa@mrabarnett.plus.com> References: <5ad962fc-1257-dd8d-96ab-541ae5bae2fa@mrabarnett.plus.com> Message-ID: Thank you MRAB for your reply. Regarding your first question, pSentence is a list.? In the nltk library, nltk.word_tokenize takes a string, so we convert sentence to string before we call nltk.word_tokenize: >>> sentence = " ".join(sentence) >>> pt = nltk.word_tokenize(sentence) >>> print(sentence) [ Emma by Jane Austen 1816 ] But with the C API it looks like this: PyObject *pSentence = PySequence_GetItem(pSents, sent_count); PyObject* str_sentence = PyObject_Str(pSentence);? // Convert to string ; See what str_sentence looks like: PyObject* repr_str = PyObject_Repr(str_sentence);?? PyObject* str_str = PyUnicode_AsEncodedString(repr_str, "utf-8", "~E~");?? const char *bytes_str = PyBytes_AS_STRING(str_str); printf("REPR_String: %s\n", bytes_str);? REPR_String: "['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']']" So the two string representations are not the same ? or at least the?? PyUnicode_AsEncodedString is not the same, as each item is surrounded by single quotes.? Assuming that the conversion to bytes object for the REPR is an accurate representation of str_sentence, it looks like I need to strip the quotes from str_sentence before ?PyObject* pWTok = PyObject_CallFunctionObjArgs(pNltk_WTok, str_sentence, 0).???? So my questions now are (1) is there a C API function that will convert a list to a string exactly the same way as ??.join, and if not then (2) how can I strip characters from a string object in the C API?? Thanks. Mar 6, 2022, 17:42 by python at mrabarnett.plus.com: > On 2022-03-07 00:32, Jen Kris via Python-list wrote: > >> I am using the C API in Python 3.8 with the nltk library, and I have a problem with the return from a library call implemented with PyObject_CallFunctionObjArgs. >> >> This is the relevant Python code: >> >> import nltk >> from nltk.corpus import gutenberg >> fileids = gutenberg.fileids() >> sentences = gutenberg.sents(fileids[0]) >> sentence = sentences[0] >> sentence = " ".join(sentence) >> pt = nltk.word_tokenize(sentence) >> >> I run this at the Python command prompt to show how it works: >> >>>>> sentence = " ".join(sentence) >>>>> pt = nltk.word_tokenize(sentence) >>>>> print(pt) >>>>> >> ['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']'] >> >>>>> type(pt) >>>>> >> >> >> This is the relevant part of the C API code: >> >> PyObject* str_sentence = PyObject_Str(pSentence); >> // nltk.word_tokenize(sentence) >> PyObject* pNltk_WTok = PyObject_GetAttrString(pModule_mstr, "word_tokenize"); >> PyObject* pWTok = PyObject_CallFunctionObjArgs(pNltk_WTok, str_sentence, 0); >> >> (where pModule_mstr is the nltk library). >> >> That should produce a list with a length of 7 that looks like it does on the command line version shown above: >> >> ['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']'] >> >> But instead the C API produces a list with a length of 24, and the REPR looks like this: >> >> '[\'[\', "\'", \'[\', "\'", \',\', "\'Emma", "\'", \',\', "\'by", "\'", \',\', "\'Jane", "\'", \',\', "\'Austen", "\'", \',\', "\'1816", "\'", \',\', "\'", \']\', "\'", \']\']' >> >> I also tried this with PyObject_CallMethodObjArgs and PyObject_Call without success. >> >> Thanks for any help on this. >> > What is pSentence? Is it what you think it is? > To me it looks like it's either the list: > > ['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']'] > > or that list as a string: > > "['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']']" > > and that what you're tokenising. > -- > https://mail.python.org/mailman/listinfo/python-list > From rosuav at gmail.com Mon Mar 7 12:09:37 2022 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 8 Mar 2022 04:09:37 +1100 Subject: C API PyObject_CallFunctionObjArgs returns incorrect result In-Reply-To: References: <5ad962fc-1257-dd8d-96ab-541ae5bae2fa@mrabarnett.plus.com> Message-ID: On Tue, 8 Mar 2022 at 04:06, Jen Kris via Python-list wrote: > But with the C API it looks like this: > > PyObject *pSentence = PySequence_GetItem(pSents, sent_count); > PyObject* str_sentence = PyObject_Str(pSentence); // Convert to string > > PyObject* repr_str = PyObject_Repr(str_sentence); You convert it to a string, then take the representation of that. Is that what you intended? ChrisA From jenkris at tutanota.com Mon Mar 7 12:13:22 2022 From: jenkris at tutanota.com (Jen Kris) Date: Mon, 7 Mar 2022 18:13:22 +0100 (CET) Subject: C API PyObject_CallFunctionObjArgs returns incorrect result In-Reply-To: References: <5ad962fc-1257-dd8d-96ab-541ae5bae2fa@mrabarnett.plus.com> Message-ID: The PyObject str_sentence is a string representation of a list.? I need to convert the list to a string like "".join because that's what the library call takes.? Mar 7, 2022, 09:09 by rosuav at gmail.com: > On Tue, 8 Mar 2022 at 04:06, Jen Kris via Python-list > wrote: > >> But with the C API it looks like this: >> >> PyObject *pSentence = PySequence_GetItem(pSents, sent_count); >> PyObject* str_sentence = PyObject_Str(pSentence); // Convert to string >> >> PyObject* repr_str = PyObject_Repr(str_sentence); >> > > You convert it to a string, then take the representation of that. Is > that what you intended? > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > From rosuav at gmail.com Mon Mar 7 12:33:07 2022 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 8 Mar 2022 04:33:07 +1100 Subject: C API PyObject_CallFunctionObjArgs returns incorrect result In-Reply-To: References: <5ad962fc-1257-dd8d-96ab-541ae5bae2fa@mrabarnett.plus.com> Message-ID: On Tue, 8 Mar 2022 at 04:13, Jen Kris wrote: > > > The PyObject str_sentence is a string representation of a list. I need to convert the list to a string like "".join because that's what the library call takes. > What you're doing is the equivalent of str(sentence), not "".join(sentence). Since the join method is part of the string protocol, you'll find it here: https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_Join ChrisA From Joseph.Schachner at Teledyne.com Mon Mar 7 13:07:42 2022 From: Joseph.Schachner at Teledyne.com (Schachner, Joseph) Date: Mon, 7 Mar 2022 18:07:42 +0000 Subject: Behavior of the for-else construct In-Reply-To: References: <621325684.471007.1646354302946@mail.yahoo.com> <20220304225746.mmebv3myg5wbpc4f@hjp.at> <657845041.42944.1646437629325@mail.yahoo.com> <20220305001158.g7rmlyoxxtxufpfp@hjp.at> <1mc72hll06itd6jnbgdherqb3thf1fk1pr@4ax.com> <20220306163951.2ozmrhfbtsktbbt7@hjp.at> Message-ID: Can someone please change the topic of this thread? No longer about for-else. Teledyne Confidential; Commercially Sensitive Business Data -----Original Message----- From: Dennis Lee Bieber Sent: Sunday, March 6, 2022 1:29 PM To: python-list at python.org Subject: Re: Behavior of the for-else construct On Sun, 6 Mar 2022 17:39:51 +0100, "Peter J. Holzer" declaimed the following: > >(* *) for comments was actually pretty commonly used - maybe because it >stands out more than { }. I don't know if I've ever seen (. .) instead >of [ ]. > Or some terminals provided [ ] but not { } Modula-2 appears to have fixed on (* *) for comments, and only [ ] for indexing. Consider the potential mayhem going from a language where { } are comment delimiters to one where they are block delimiters >C also has alternative rerpresentations for characters not in the >common subset of ISO-646 and EBCDIC. However, the trigraphs are >extremely ugly (e.g ??< ??> instead of { }). I have seen them used (on >an IBM/390 system with an EBCDIC variant without curly braces) and it's >really no fun to read that. > My college mainframe used EBCDIC, but the available languages did not include C or Pascal. We had APL, FORTRAN-IV (in full separate compilation form, and FLAG [FORTRAN Load and Go] which was a "all in one file, compile & run" used by first year students), COBOL (74?), BASIC, SNOBOL, Meta-Symbol and AP (both assemblers, though Meta-Symbol could, provided the proper definition file, generate absolute binary code for pretty much any processor), and something called SL-1 (Simulation Language-1, which produced FORTRAN output for discrete event models). UCSD Pascal, and PDP-11 assembly were run on a pair of LSI-11 systems. Assembly used for the operating system principles course. I didn't encounter "real" C until getting a TRS-80 (first as integer LC, then Pro-MC), along with Supersoft LISP (on cassette tape!). (I had books for C and Ada before encountering compilers for them) -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From dieter at handshake.de Mon Mar 7 13:16:11 2022 From: dieter at handshake.de (Dieter Maurer) Date: Mon, 7 Mar 2022 19:16:11 +0100 Subject: Execute in a multiprocessing child dynamic code loaded by the parent process In-Reply-To: <20220306204228.l5pzjzc6peytbshp@gmail.com> References: <20220306124208.orqbcezhqqeysdpe@gmail.com> <25124.63881.205740.249595@ixdm.fritz.box> <20220306204228.l5pzjzc6peytbshp@gmail.com> Message-ID: <25126.19435.559817.664736@ixdm.fritz.box> Martin Di Paola wrote at 2022-3-6 20:42 +0000: >>Try to use `fork` as "start method" (instead of "spawn"). > >Yes but no. Indeed with `fork` there is no need to pickle anything. In >particular the child process will be a copy of the parent so it will >have all the modules loaded, including the dynamic ones. Perfect. > >The problem is that `fork` is the default only in Linux. It works in >MacOS but it may lead to crashes if the parent process is multithreaded >(and the my is!) and `fork` does not work in Windows. Then, you must put the initialization (dynamically loading the modules) into the function executed in the foreign process. You could wrap the payload function into a class instances to achieve this. In the foreign process, you call the instance which first performs the initialization and then executes the payload. From wlfraed at ix.netcom.com Mon Mar 7 13:36:18 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Mon, 07 Mar 2022 13:36:18 -0500 Subject: Non sequitur: Changing subject line... WAS: Behavior of the for-else construct References: <20220305001158.g7rmlyoxxtxufpfp@hjp.at> <1mc72hll06itd6jnbgdherqb3thf1fk1pr@4ax.com> <20220306163951.2ozmrhfbtsktbbt7@hjp.at> Message-ID: On Mon, 7 Mar 2022 18:07:42 +0000, "Schachner, Joseph" declaimed the following: >Can someone please change the topic of this thread? No longer about for-else. > Pretty much anyone can change the subject of the message when replying. But if one is using a threaded client, that thread on message references Message-ID: References: <621325684.471007.1646354302946 at mail.yahoo.com> <20220304225746.mmebv3myg5wbpc4f at hjp.at> <657845041.42944.1646437629325 at mail.yahoo.com> <20220305001158.g7rmlyoxxtxufpfp at hjp.at> <1mc72hll06itd6jnbgdherqb3thf1fk1pr at 4ax.com> <20220306163951.2ozmrhfbtsktbbt7 at hjp.at> it will still appear under the parent message; it will only thread differently if one's client merely sorts on subject and date/time. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From python at mrabarnett.plus.com Mon Mar 7 14:03:24 2022 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 7 Mar 2022 19:03:24 +0000 Subject: C API PyObject_CallFunctionObjArgs returns incorrect result In-Reply-To: References: <5ad962fc-1257-dd8d-96ab-541ae5bae2fa@mrabarnett.plus.com> Message-ID: <411854b3-e73d-0908-72f6-4049b87145c2@mrabarnett.plus.com> On 2022-03-07 17:05, Jen Kris wrote: > Thank you MRAB for your reply. > > Regarding your first question, pSentence is a list.? In the nltk > library, nltk.word_tokenize takes a string, so we convert sentence to > string before we call nltk.word_tokenize: > > >>> sentence = " ".join(sentence) > >>> pt = nltk.word_tokenize(sentence) > >>> print(sentence) > [ Emma by Jane Austen 1816 ] > > But with the C API it looks like this: > > PyObject *pSentence = PySequence_GetItem(pSents, sent_count); > PyObject* str_sentence = PyObject_Str(pSentence); // Convert to string > > ; See what str_sentence looks like: > PyObject* repr_str = PyObject_Repr(str_sentence); > PyObject* str_str = PyUnicode_AsEncodedString(repr_str, "utf-8", "~E~"); > const char *bytes_str = PyBytes_AS_STRING(str_str); > printf("REPR_String: %s\n", bytes_str); > > REPR_String: "['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']']" > > So the two string representations are not the same ? or at least the?? > PyUnicode_AsEncodedString is not the same, as each item is surrounded > by single quotes. > > Assuming that the conversion to bytes object for the REPR is an > accurate representation of str_sentence, it looks like I need to strip > the quotes from str_sentence before ?PyObject* pWTok = > PyObject_CallFunctionObjArgs(pNltk_WTok, str_sentence, 0).? > > So my questions now are (1) is there a C API function that will > convert a list to a string exactly the same way as ??.join, and if not > then (2) how can I strip characters from a string object in the C API? > Your Python code is joining the list with a space as the separator. The equivalent using the C API is: ??? PyObject* separator; ??? PyObject* joined; ??? separator = PyUnicode_FromString(" "); ??? joined = PyUnicode_Join(separator, pSentence); ??? Py_DECREF(sep); > > Mar 6, 2022, 17:42 by python at mrabarnett.plus.com: > > On 2022-03-07 00:32, Jen Kris via Python-list wrote: > > I am using the C API in Python 3.8 with the nltk library, and > I have a problem with the return from a library call > implemented with PyObject_CallFunctionObjArgs. > > This is the relevant Python code: > > import nltk > from nltk.corpus import gutenberg > fileids = gutenberg.fileids() > sentences = gutenberg.sents(fileids[0]) > sentence = sentences[0] > sentence = " ".join(sentence) > pt = nltk.word_tokenize(sentence) > > I run this at the Python command prompt to show how it works: > > sentence = " ".join(sentence) > pt = nltk.word_tokenize(sentence) > print(pt) > > ['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']'] > > type(pt) > > > > This is the relevant part of the C API code: > > PyObject* str_sentence = PyObject_Str(pSentence); > // nltk.word_tokenize(sentence) > PyObject* pNltk_WTok = PyObject_GetAttrString(pModule_mstr, > "word_tokenize"); > PyObject* pWTok = PyObject_CallFunctionObjArgs(pNltk_WTok, > str_sentence, 0); > > (where pModule_mstr is the nltk library). > > That should produce a list with a length of 7 that looks like > it does on the command line version shown above: > > ['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']'] > > But instead the C API produces a list with a length of 24, and > the REPR looks like this: > > '[\'[\', "\'", \'[\', "\'", \',\', "\'Emma", "\'", \',\', > "\'by", "\'", \',\', "\'Jane", "\'", \',\', "\'Austen", "\'", > \',\', "\'1816", "\'", \',\', "\'", \']\', "\'", \']\']' > > I also tried this with PyObject_CallMethodObjArgs and > PyObject_Call without success. > > Thanks for any help on this. > > What is pSentence? Is it what you think it is? > To me it looks like it's either the list: > > ['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']'] > > or that list as a string: > > "['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']']" > > and that what you're tokenising. > -- > https://mail.python.org/mailman/listinfo/python-list > > From python at mrabarnett.plus.com Mon Mar 7 14:10:49 2022 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 7 Mar 2022 19:10:49 +0000 Subject: always return the same pdf In-Reply-To: References: Message-ID: <313427bd-3ded-53ca-8a2c-631f3adee907@mrabarnett.plus.com> On 2022-03-07 14:08, Gonzalo V wrote: > Hello everyone. > i had upload a Django app to an ubuntu 18.04 server and it gives me the > same pdf everytime the view is called. To generate the pdf it receipts > differents string buy it gives me the same pdf. Could you give some idea > what is happening? > > thanks everyone > @never_cached > def generar_pdf(request): > prueba = request.session.get('contenedor') > cantidad_preguntas=prueba['cantidad_preguntas'] > archivo_salida = open("prueba.tex","w") > > archivo_salida.write("\\documentclass[10pt,oneside,letterpaper]{article}") > archivo_salida.write("\\usepackage[utf8x]{inputenc}") > > ##hace mas y mas cosas sin importancia con latex que funcionan bien > > archivo_a_descargar = open("prueba.pdf","rb") # > respuesta = > HttpResponse(archivo_a_descargar,content_type='application/pdf') > respuesta['Content-Disposition'] = 'attachment; filename="{0}"'.format( > archivo_a_descargar.name) > > return respuesta > Saludos, > Gonzalo You're using relative paths. Are you sure that they are pointing to the correct files? Is it actually generating the PDF? You might think that when it generates the PDF it overwrites any existing file of that name but is it? Is it simply giving you the PDF that's already there? From antoon.pardon at vub.be Mon Mar 7 14:20:40 2022 From: antoon.pardon at vub.be (Antoon Pardon) Date: Mon, 7 Mar 2022 20:20:40 +0100 Subject: Behavior of the for-else construct In-Reply-To: <1277831092.475364.1646356088655@mail.yahoo.com> References: <21739669.459456.1646348879560@mail.yahoo.com> <1277831092.475364.1646356088655@mail.yahoo.com> Message-ID: Op 4/03/2022 om 02:08 schreef Avi Gross via Python-list: > If Python was being designed TODAY, I wonder if a larger set of key words would be marked as RESERVED for future expansion including ORELSE and even NEVERTHELESS. I think a better solution would be to have reserved words written letters form the mathematical lettter block. Something like: ??? foo(bar): in = file(...) ??? line ?? in: ... From jenkris at tutanota.com Mon Mar 7 16:08:26 2022 From: jenkris at tutanota.com (Jen Kris) Date: Mon, 7 Mar 2022 22:08:26 +0100 (CET) Subject: C API PyObject_CallFunctionObjArgs returns incorrect result In-Reply-To: <411854b3-e73d-0908-72f6-4049b87145c2@mrabarnett.plus.com> References: <5ad962fc-1257-dd8d-96ab-541ae5bae2fa@mrabarnett.plus.com> <411854b3-e73d-0908-72f6-4049b87145c2@mrabarnett.plus.com> Message-ID: Thanks to MRAB and Chris Angelico for your help.? Here is how I implemented the string conversion, and it works correctly now for a library call that needs a list converted to a string (error handling not shown): PyObject* str_sentence = PyObject_Str(pSentence);?? PyObject* separator = PyUnicode_FromString(" "); PyObject* str_join = PyUnicode_Join(separator, pSentence); Py_DECREF(separator); PyObject* pNltk_WTok = PyObject_GetAttrString(pModule_mstr, "word_tokenize"); PyObject* pWTok = PyObject_CallFunctionObjArgs(pNltk_WTok, str_join, 0); That produces what I need (this is the REPR of pWTok): "['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']']" Thanks again to both of you.? Jen Mar 7, 2022, 11:03 by python at mrabarnett.plus.com: > On 2022-03-07 17:05, Jen Kris wrote: > >> Thank you MRAB for your reply. >> >> Regarding your first question, pSentence is a list.? In the nltk library, nltk.word_tokenize takes a string, so we convert sentence to string before we call nltk.word_tokenize: >> >> >>> sentence = " ".join(sentence) >> >>> pt = nltk.word_tokenize(sentence) >> >>> print(sentence) >> [ Emma by Jane Austen 1816 ] >> >> But with the C API it looks like this: >> >> PyObject *pSentence = PySequence_GetItem(pSents, sent_count); >> PyObject* str_sentence = PyObject_Str(pSentence); // Convert to string >> >> ; See what str_sentence looks like: >> PyObject* repr_str = PyObject_Repr(str_sentence); >> PyObject* str_str = PyUnicode_AsEncodedString(repr_str, "utf-8", "~E~"); >> const char *bytes_str = PyBytes_AS_STRING(str_str); >> printf("REPR_String: %s\n", bytes_str); >> >> REPR_String: "['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']']" >> >> So the two string representations are not the same ? or at least the?? PyUnicode_AsEncodedString is not the same, as each item is surrounded by single quotes. >> >> Assuming that the conversion to bytes object for the REPR is an accurate representation of str_sentence, it looks like I need to strip the quotes from str_sentence before ?PyObject* pWTok = PyObject_CallFunctionObjArgs(pNltk_WTok, str_sentence, 0).? >> >> So my questions now are (1) is there a C API function that will convert a list to a string exactly the same way as ??.join, and if not then (2) how can I strip characters from a string object in the C API? >> > Your Python code is joining the list with a space as the separator. > > The equivalent using the C API is: > > ??? PyObject* separator; > ??? PyObject* joined; > > ??? separator = PyUnicode_FromString(" "); > ??? joined = PyUnicode_Join(separator, pSentence); > ??? Py_DECREF(sep); > >> >> Mar 6, 2022, 17:42 by python at mrabarnett.plus.com: >> >> On 2022-03-07 00:32, Jen Kris via Python-list wrote: >> >> I am using the C API in Python 3.8 with the nltk library, and >> I have a problem with the return from a library call >> implemented with PyObject_CallFunctionObjArgs. >> >> This is the relevant Python code: >> >> import nltk >> from nltk.corpus import gutenberg >> fileids = gutenberg.fileids() >> sentences = gutenberg.sents(fileids[0]) >> sentence = sentences[0] >> sentence = " ".join(sentence) >> pt = nltk.word_tokenize(sentence) >> >> I run this at the Python command prompt to show how it works: >> >> sentence = " ".join(sentence) >> pt = nltk.word_tokenize(sentence) >> print(pt) >> >> ['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']'] >> >> type(pt) >> >> >> >> This is the relevant part of the C API code: >> >> PyObject* str_sentence = PyObject_Str(pSentence); >> // nltk.word_tokenize(sentence) >> PyObject* pNltk_WTok = PyObject_GetAttrString(pModule_mstr, >> "word_tokenize"); >> PyObject* pWTok = PyObject_CallFunctionObjArgs(pNltk_WTok, >> str_sentence, 0); >> >> (where pModule_mstr is the nltk library). >> >> That should produce a list with a length of 7 that looks like >> it does on the command line version shown above: >> >> ['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']'] >> >> But instead the C API produces a list with a length of 24, and >> the REPR looks like this: >> >> '[\'[\', "\'", \'[\', "\'", \',\', "\'Emma", "\'", \',\', >> "\'by", "\'", \',\', "\'Jane", "\'", \',\', "\'Austen", "\'", >> \',\', "\'1816", "\'", \',\', "\'", \']\', "\'", \']\']' >> >> I also tried this with PyObject_CallMethodObjArgs and >> PyObject_Call without success. >> >> Thanks for any help on this. >> >> What is pSentence? Is it what you think it is? >> To me it looks like it's either the list: >> >> ['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']'] >> >> or that list as a string: >> >> "['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']']" >> >> and that what you're tokenising. >> -- https://mail.python.org/mailman/listinfo/python-list >> > -- > https://mail.python.org/mailman/listinfo/python-list > From martinp.dipaola at gmail.com Tue Mar 8 09:50:23 2022 From: martinp.dipaola at gmail.com (Martin Di Paola) Date: Tue, 8 Mar 2022 14:50:23 +0000 Subject: Execute in a multiprocessing child dynamic code loaded by the parent process In-Reply-To: <25126.19435.559817.664736@ixdm.fritz.box> References: <20220306124208.orqbcezhqqeysdpe@gmail.com> <25124.63881.205740.249595@ixdm.fritz.box> <20220306204228.l5pzjzc6peytbshp@gmail.com> <25126.19435.559817.664736@ixdm.fritz.box> Message-ID: <20220308145023.vdisqxa3eb5a5jgy@gmail.com> >Then, you must put the initialization (dynamically loading the modules) >into the function executed in the foreign process. > >You could wrap the payload function into a class instances to achieve this. >In the foreign process, you call the instance which first performs >the initialization and then executes the payload. That's what I have in mind: loading the modules first, and then unpickle and call the real target function. From pkpearson at nowhere.invalid Tue Mar 8 11:00:59 2022 From: pkpearson at nowhere.invalid (Peter Pearson) Date: 8 Mar 2022 16:00:59 GMT Subject: strange problem building non-pure wheel for apple M1 arm64 References: <589e22ac-28cb-b9f3-8be9-fb4988228929@everest.reportlab.co.uk> Message-ID: On Mon, 7 Mar 2022 16:22:10 +0000, Robin Becker wrote: [snip] > > gcc -bundle -undefined dynamic_lookup -g -arch arm64 [snip] > -L/usr/local/lib > -L/usr/lib > -L/Library/Frameworks/Python.framework/Versions/3.9/lib > -lfreetype [snip] > > ld: warning: ignoring file /usr/local/lib/libfreetype.dylib, building > for macOS-arm64 but attempting to link with file > built for macOS-x86_64 > > The above message seems bizarre; everything is compiled for arm64, but > gcc doesn't want to use an arm64 dylib. > > Can macos experts assist? I am not at all expert in MacOS or any of the specifics of this problem, but I think you want to try to prove that the libfreetype invoked by "-lfreetype" is built for macOS-arm64 rather than for macOS-x86-64. -- To email me, substitute nowhere->runbox, invalid->com. From auriocus at gmx.de Tue Mar 8 11:08:19 2022 From: auriocus at gmx.de (Christian Gollwitzer) Date: Tue, 8 Mar 2022 17:08:19 +0100 Subject: strange problem building non-pure wheel for apple M1 arm64 In-Reply-To: References: <589e22ac-28cb-b9f3-8be9-fb4988228929@everest.reportlab.co.uk> Message-ID: Am 07.03.22 um 17:22 schrieb Robin Becker: > > I use brew to install freetype version 2.11.1. > gcc -bundle -undefined dynamic_lookup -g -arch arm64 > ????build/temp.macosx-11.0-arm64-3.9/src/rl_addons/renderPM/_renderPM.o > ????build/temp.macosx-11.0-arm64-3.9/src/rl_addons/renderPM/gt1/gt1-dict.o > ????build/temp.macosx-11.0-arm64-3.9/src/rl_addons/renderPM/gt1/gt1-namecontext.o > ????'''''''other compiled code > ????build/temp.macosx-11.0-arm64-3.9/src/rl_addons/renderPM/libart_lgpl/art_vpath_dash.o > ????-L/usr/local/lib > ????-L/usr/lib > ????-L/Library/Frameworks/Python.framework/Versions/3.9/lib > ????-lfreetype -o > build/lib.macosx-11.0-arm64-3.9/reportlab/graphics/_renderPM.cpython-39-darwin.so > > > ld: warning: ignoring file /usr/local/lib/libfreetype.dylib, building > for macOS-arm64 but attempting to link with file built for macOS-x86_64 > > The above message seems bizarre; everything is compiled for arm64, but > gcc doesn't want to use an arm64 dylib. I would interpret this as: the host is runnig in intel, by doing the "brew install" it installs the intel version of freetype into /usr/local/lib/ ; then you cross-compile the python extension for ARM, but -lfreetype picks up the host version. On macOS it is not mandatory that all symbols in a .dylib are resolved. I would guess that the resulting file will not work on M1 macs. YOu can check by otool -L /path/toyour.dylib which libraries are referenced. If you use freetype functions in your C code and do not see it referenced, then the lib will not work. Maybe you can install an ARM-version of freetype, or compile it from source during your build process? Christian From robin at reportlab.com Tue Mar 8 12:55:31 2022 From: robin at reportlab.com (Robin Becker) Date: Tue, 8 Mar 2022 17:55:31 +0000 Subject: strange problem building non-pure wheel for apple M1 arm64 In-Reply-To: References: <589e22ac-28cb-b9f3-8be9-fb4988228929@everest.reportlab.co.uk> Message-ID: On 08/03/2022 16:08, Christian Gollwitzer wrote: > Am 07.03.22 um 17:22 schrieb Robin Becker: >> >> I use brew to install freetype version 2.11.1. I find via google that homebrew/apple have split the installation of intel and arm64 into /usr/local and /opt/homebrew so I must modify the include_dirs & library_dirs in setup.py -- Robin Becker From shishaozhong at gmail.com Tue Mar 8 13:29:57 2022 From: shishaozhong at gmail.com (Shaozhong SHI) Date: Tue, 8 Mar 2022 18:29:57 +0000 Subject: Missing global # gdal DRIVER_NAME declaration in gdal_array.py Message-ID: The following warning kept coming up when running ogr2ogr. Warning 1: Missing global # gdal: DRIVER_NAME declaration in C:\Users\AppData\Local\Programs\Python\Python36\Lib\site-packages\osgeo\gdal_array.py What steps to be take to resolve this issue? Regards, David From ethan at stoneleaf.us Tue Mar 8 13:52:52 2022 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 8 Mar 2022 10:52:52 -0800 Subject: PSA: Linux vulnerability Message-ID: https://arstechnica.com/information-technology/2022/03/linux-has-been-bitten-by-its-most-high-severity-vulnerability-in-years/ -- ~Ethan~ From python at boiteameuh.org Wed Mar 9 10:53:18 2022 From: python at boiteameuh.org (Nicolas Haller) Date: Wed, 9 Mar 2022 10:53:18 -0500 Subject: Typing on child class' methods of a Generic base class Message-ID: <3fccbe66-4d1c-f40c-7282-026ebe7360e3@boiteameuh.org> Hello, I am wondering about a situation involving typing, Generic and a child class. The documentation about "user-defined generic types"[1] says that I can fix some types on a child class (class MyDict(Mapping[str, T]):) but doesn't say much about the signature of the methods I need to implement/override on that child class. Should I keep the TypeVar and remember that I fixed the type(alternative 1) or change the signature of the method to replace the TypeVar by the type I set (alternative 2)? Mypy seems to prefer the second alternative but changing the signature feels wrong to me for some reason. That's why I'm asking :-) Below is a small example to illustrate: ---- from abc import ABCMeta, abstractmethod from typing import TypeVar, Generic T = TypeVar("T") class Base(Generic[T], metaclass=ABCMeta): """A base class.""" @abstractmethod def _private_method(self, an_arg: T) -> T: ... def public_method(self, an_arg: T) -> T: from_private_method = self._private_method(an_arg) return from_private_method class Alternative1(Base[int]): def _private_method(self, an_arg: T) -> T: # I keep T return 42 class Alternative2(Base[int]): def _private_method(self, an_arg: int) -> int: # I replace T return 42 ---- Thanks, -- Nicolas Haller [1]: https://docs.python.org/3/library/typing.html#user-defined-generic-types From Marco.Sulla.Python at gmail.com Wed Mar 9 15:05:43 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Wed, 9 Mar 2022 21:05:43 +0100 Subject: PSA: Linux vulnerability In-Reply-To: References: Message-ID: So my laziness pays. I use only LTS distros, and I update only when there are security updates. PS: any suggestions for a new LTS distro? My Lubuntu is reaching its end-of-life. I prefer lightweight debian-like distros. On Tue, 8 Mar 2022 at 19:56, Ethan Furman wrote: > > https://arstechnica.com/information-technology/2022/03/linux-has-been-bitten-by-its-most-high-severity-vulnerability-in-years/ > > -- > ~Ethan~ > -- > https://mail.python.org/mailman/listinfo/python-list From aaturnerpython at outlook.com Wed Mar 9 13:42:47 2022 From: aaturnerpython at outlook.com (Adam Turner) Date: Wed, 9 Mar 2022 18:42:47 +0000 Subject: PEPs will move to peps.python.org Message-ID: With the acceptance of PEP 676, the canonical home of the Python Enhancement Proposal series will shortly move to peps.python.org. All existing links will redirect when the change is made, this announcement is to promote awareness of the new domain as canonical. Thanks, Adam Turner PEP Editor and author of PEP 676 From Marco.Sulla.Python at gmail.com Wed Mar 9 15:16:00 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Wed, 9 Mar 2022 21:16:00 +0100 Subject: Could frozendict or frozenmap be of some use for PEP 683 (Immortal objects)? Message-ID: As title. dict can't be an immortal object, but hashable frozendict and frozenmap can. I think this can increase their usefulness. Another advantage: frozen dataclass will be really immutable if they could use a frozen(dict|map) instead of a dict as __dict__ From hjp-python at hjp.at Wed Mar 9 16:32:17 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Wed, 9 Mar 2022 22:32:17 +0100 Subject: Behavior of the for-else construct In-Reply-To: <62261019.1c69fb81.42653.45e0@mx.google.com> References: <20220305001158.g7rmlyoxxtxufpfp@hjp.at> <1137940845.144932.1646516408119@mail.yahoo.com> <6224ef6f.1c69fb81.6228c.7238@mx.google.com> <20220306174818.alasogxuqxhgaprr@hjp.at> <222574152.213889.1646607001131@mail.yahoo.com> <62256f3f.1c69fb81.77d74.2adb@mx.google.com> <20220307071441.5w2dpyew62elyhqe@hjp.at> <62261019.1c69fb81.42653.45e0@mx.google.com> Message-ID: <20220309213217.lqkbocn4engs6m4y@hjp.at> On 2022-03-07 06:00:57 -0800, Grant Edwards wrote: > On 2022-03-07, Peter J. Holzer wrote: > > On 2022-03-06 18:34:39 -0800, Grant Edwards wrote: > >> On 2022-03-06, Avi Gross via Python-list wrote: > >> > Python is named after a snake right? > >> > >> No. It's named after a comedy troupe. > > > > He actually wrote that two sentences later. > > Yes, I missed that. His messages wrap very strangely in my newsreader. Yes, he writes every paragraph as a single long line which makes it hard to read. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From martinp.dipaola at gmail.com Wed Mar 9 17:27:10 2022 From: martinp.dipaola at gmail.com (Martin Di Paola) Date: Wed, 9 Mar 2022 22:27:10 +0000 Subject: Could frozendict or frozenmap be of some use for PEP 683 (Immortal objects)? In-Reply-To: References: Message-ID: <20220309222710.qzg25vclyeqmgcsd@gmail.com> I perhaps didn't understand the PEP completely but I think that the goal of marking some objects as immortal is to remove the refcount from they. For immutable objects that would make them truly immutable. However I don't think that the immortality could be applied to any immutable object by default. Think in the immutable strings (str). What would happen with a program that does heavy parsing? I imagine that it will generate thousands of little strings. If those are immortal, the program will fill its memory very quickly as the GC will not reclaim their memory. The same could happen with any frozenfoo object. Leaving immortality to only a few objects, like True and None makes more sense as they are few (bound if you want). On Wed, Mar 09, 2022 at 09:16:00PM +0100, Marco Sulla wrote: >As title. dict can't be an immortal object, but hashable frozendict >and frozenmap can. I think this can increase their usefulness. > >Another advantage: frozen dataclass will be really immutable if they >could use a frozen(dict|map) instead of a dict as __dict__ >-- >https://mail.python.org/mailman/listinfo/python-list From torriem at gmail.com Wed Mar 9 22:48:36 2022 From: torriem at gmail.com (Michael Torrie) Date: Wed, 9 Mar 2022 20:48:36 -0700 Subject: PSA: Linux vulnerability In-Reply-To: References: Message-ID: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> On 3/9/22 13:05, Marco Sulla wrote: > So my laziness pays. I use only LTS distros, and I update only when > there are security updates. > PS: any suggestions for a new LTS distro? My Lubuntu is reaching its > end-of-life. I prefer lightweight debian-like distros. Maybe Debian itself? From Marco.Sulla.Python at gmail.com Thu Mar 10 07:58:27 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Thu, 10 Mar 2022 13:58:27 +0100 Subject: Could frozendict or frozenmap be of some use for PEP 683 (Immortal objects)? In-Reply-To: <20220309222710.qzg25vclyeqmgcsd@gmail.com> References: <20220309222710.qzg25vclyeqmgcsd@gmail.com> Message-ID: On Wed, 9 Mar 2022 at 23:28, Martin Di Paola wrote: > Think in the immutable strings (str). What would happen with a program > that does heavy parsing? I imagine that it will generate thousands of > little strings. If those are immortal, the program will fill its memory > very quickly as the GC will not reclaim their memory. Well, as far as I know immortality was also suggested for interned strings. If I understood well, the problem with "normal" strings is that they are not really immutable in CPython. They have cache etc. Also frozendict caches hash, but that cache can be easily removed. From Marco.Sulla.Python at gmail.com Thu Mar 10 08:03:16 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Thu, 10 Mar 2022 14:03:16 +0100 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> Message-ID: On Thu, 10 Mar 2022 at 04:50, Michael Torrie wrote: > > On 3/9/22 13:05, Marco Sulla wrote: > > So my laziness pays. I use only LTS distros, and I update only when > > there are security updates. > > PS: any suggestions for a new LTS distro? My Lubuntu is reaching its > > end-of-life. I prefer lightweight debian-like distros. > > Maybe Debian itself? I tried Debian on a VM, but I found it too much basical. A little example: it does not have the shortcut ctrl+alt+t to open a terminal that Ubuntu has. I'm quite sure it's simple to add, but I'm starting to be old and lazy... From tdldev at gmail.com Thu Mar 10 08:12:13 2022 From: tdldev at gmail.com (Jack Dangler) Date: Thu, 10 Mar 2022 08:12:13 -0500 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> Message-ID: On 3/10/22 08:03, Marco Sulla wrote: > On Thu, 10 Mar 2022 at 04:50, Michael Torrie wrote: >> On 3/9/22 13:05, Marco Sulla wrote: >>> So my laziness pays. I use only LTS distros, and I update only when >>> there are security updates. >>> PS: any suggestions for a new LTS distro? My Lubuntu is reaching its >>> end-of-life. I prefer lightweight debian-like distros. >> Maybe Debian itself? > I tried Debian on a VM, but I found it too much basical. A little > example: it does not have the shortcut ctrl+alt+t to open a terminal > that Ubuntu has. I'm quite sure it's simple to add, but I'm starting > to be old and lazy... or why not get a cloud desktop running whatever distro you want and you don't have to do anything From rosuav at gmail.com Thu Mar 10 09:18:30 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 11 Mar 2022 01:18:30 +1100 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> Message-ID: On Fri, 11 Mar 2022 at 00:05, Marco Sulla wrote: > > On Thu, 10 Mar 2022 at 04:50, Michael Torrie wrote: > > > > On 3/9/22 13:05, Marco Sulla wrote: > > > So my laziness pays. I use only LTS distros, and I update only when > > > there are security updates. > > > PS: any suggestions for a new LTS distro? My Lubuntu is reaching its > > > end-of-life. I prefer lightweight debian-like distros. > > > > Maybe Debian itself? > > I tried Debian on a VM, but I found it too much basical. A little > example: it does not have the shortcut ctrl+alt+t to open a terminal > that Ubuntu has. I'm quite sure it's simple to add, but I'm starting > to be old and lazy... Very easy. I use Debian with Xfce, and it's an easy thing to add shortcuts - even dynamically (I have Alt+D to dictate notes, but only while I'm playing Counter-Strike competitively). Debian + Xfce is a combo that's served me very well for years now. I didn't much like the latest Adwaita colour scheme, but a small tweak later, it's fine again. ChrisA From loris.bennett at fu-berlin.de Thu Mar 10 07:16:11 2022 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Thu, 10 Mar 2022 13:16:11 +0100 Subject: How to test input via subprocess.Popen with data from file Message-ID: <878rtiq9ys.fsf@hornfels.zedat.fu-berlin.de> Hi, I have a command which produces output like the following: Job ID: 9431211 Cluster: curta User/Group: build/staff State: COMPLETED (exit code 0) Nodes: 1 Cores per node: 8 CPU Utilized: 01:30:53 CPU Efficiency: 83.63% of 01:48:40 core-walltime Job Wall-clock time: 00:13:35 Memory Utilized: 6.45 GB Memory Efficiency: 80.68% of 8.00 GB I want to parse this and am using subprocess.Popen and accessing the contents via Popen.stdout. However, for testing purposes I want to save various possible outputs of the command as text files and use those as inputs. What format should I use to pass data to the actual parsing function? I could in both production and test convert the entire input to a string and pass the string to the parsing method. However, I could use something like test_input_01 = subprocess.Popen( ["cat test_input_01.txt"], stdout=subprocess.PIPE, ) for the test input and then pass a Popen object to the parsing function. Any comments on these alternative or suggestions for doing something completely different? Cheers, Loris -- This signature is currently under construction. From loris.bennett at fu-berlin.de Thu Mar 10 08:17:44 2022 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Thu, 10 Mar 2022 14:17:44 +0100 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> Message-ID: <874k46q747.fsf@hornfels.zedat.fu-berlin.de> Marco Sulla writes: > On Thu, 10 Mar 2022 at 04:50, Michael Torrie wrote: >> >> On 3/9/22 13:05, Marco Sulla wrote: >> > So my laziness pays. I use only LTS distros, and I update only when >> > there are security updates. >> > PS: any suggestions for a new LTS distro? My Lubuntu is reaching its >> > end-of-life. I prefer lightweight debian-like distros. >> >> Maybe Debian itself? > > I tried Debian on a VM, but I found it too much basical. A little > example: it does not have the shortcut ctrl+alt+t to open a terminal > that Ubuntu has. I'm quite sure it's simple to add, but I'm starting > to be old and lazy... The shortcuts are properties of the desktop environment. You could just install LXDE/LXQt on Debian if that's what you're used to from Lubuntu. Of course, if you're too old and lazy to set up a shortcut, you might also be too old and lazy to install a different desktop environment ;-) Cheers, Loris -- This signature is currently under construction. From torriem at gmail.com Thu Mar 10 10:55:22 2022 From: torriem at gmail.com (Michael Torrie) Date: Thu, 10 Mar 2022 08:55:22 -0700 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> Message-ID: On 3/10/22 06:03, Marco Sulla wrote: > I tried Debian on a VM, but I found it too much basical. A little > example: it does not have the shortcut ctrl+alt+t to open a terminal > that Ubuntu has. I'm quite sure it's simple to add, but I'm starting > to be old and lazy... Debian has the same desktop environments available for install as the rest of the distros. Gnome 3, Mate, LXDE, XFCE, KDE, etc. Whatever works for you on Ubuntu should work on Debian. From dieter at handshake.de Thu Mar 10 12:24:59 2022 From: dieter at handshake.de (Dieter Maurer) Date: Thu, 10 Mar 2022 18:24:59 +0100 Subject: How to test input via subprocess.Popen with data from file In-Reply-To: <878rtiq9ys.fsf@hornfels.zedat.fu-berlin.de> References: <878rtiq9ys.fsf@hornfels.zedat.fu-berlin.de> Message-ID: <25130.13419.321314.959918@ixdm.fritz.box> Loris Bennett wrote at 2022-3-10 13:16 +0100: >I have a command which produces output like the >following: > > Job ID: 9431211 > Cluster: curta > User/Group: build/staff > State: COMPLETED (exit code 0) > Nodes: 1 > Cores per node: 8 > CPU Utilized: 01:30:53 > CPU Efficiency: 83.63% of 01:48:40 core-walltime > Job Wall-clock time: 00:13:35 > Memory Utilized: 6.45 GB > Memory Efficiency: 80.68% of 8.00 GB > >I want to parse this and am using subprocess.Popen and accessing the >contents via Popen.stdout. However, for testing purposes I want to save >various possible outputs of the command as text files and use those as >inputs. What do you want to test? the parsing? the "popen" interaction? You can separately test both tasks (I, at your place, would do this). For the parsing test, it is not relevant that the actual text comes from an external process. You can directly read it from a file or have it in your text. In my view, you do not need a test for the `Popen` interaction: if it works once, it will work always. From dieter at handshake.de Thu Mar 10 12:31:11 2022 From: dieter at handshake.de (Dieter Maurer) Date: Thu, 10 Mar 2022 18:31:11 +0100 Subject: Typing on child class' methods of a Generic base class In-Reply-To: <3fccbe66-4d1c-f40c-7282-026ebe7360e3@boiteameuh.org> References: <3fccbe66-4d1c-f40c-7282-026ebe7360e3@boiteameuh.org> Message-ID: <25130.13791.689201.429804@ixdm.fritz.box> Nicolas Haller wrote at 2022-3-9 10:53 -0500: > ... >The documentation about "user-defined generic types"[1] says that I can >fix some types on a child class (class MyDict(Mapping[str, T]):) but >doesn't say much about the signature of the methods I need to >implement/override on that child class. I have the fealing that this is a case of (generic type) "specialization". In this setup, `Mapping` would be a generic type with two type variables `K` (the key type) and `V` (the value type). The signatures of its methods would use those type variables. In your example, you specialize the key type to `str` (and leave the value type generic). The signatures of the methods would automatically follow this specialization -- without the need to do anything. From Marco.Sulla.Python at gmail.com Thu Mar 10 14:42:24 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Thu, 10 Mar 2022 20:42:24 +0100 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> Message-ID: On Thu, 10 Mar 2022 at 14:13, Jack Dangler wrote: > or why not get a cloud desktop running whatever distro you want and you > don't have to do anything Three reasons: privacy, speed, price. Not in this order. On Thu, 10 Mar 2022 at 15:20, Chris Angelico wrote: > Very easy. I use Debian with Xfce, and it's an easy thing to add > shortcuts - even dynamically I used Xubuntu for a long time. I like Xfce. On Thu, 10 Mar 2022 at 16:35, Loris Bennett wrote: > The shortcuts are properties of the desktop environment. You could just > install LXDE/LXQt on Debian if that's what you're used to from Lubuntu. I tried LXQt on my desktop. Very disappointed. The OS Update interface is just an "alert". LXDE unluckily is no longer developed. > Of course, if you're too old and lazy to set up a shortcut, you might > also be too old and lazy to install a different desktop environment ;-) Okay, I'm lazy for boring things :D PS: Is it just my impression or is there a plebiscite for Debian? From torriem at gmail.com Thu Mar 10 15:13:42 2022 From: torriem at gmail.com (Michael Torrie) Date: Thu, 10 Mar 2022 13:13:42 -0700 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> Message-ID: <42c22185-028d-9d61-4399-da111b0e2c3e@gmail.com> On 3/10/22 12:42, Marco Sulla wrote: > PS: Is it just my impression or is there a plebiscite for Debian? A vote? No I don't think so. Not sure what you mean. The reason we're all suggesting Debian is because you specifically said you want a LTS Debian-like distro. Can't get any more Debian-like than Debian! Debian with XFCE should give you the same experience as Xubuntu, and is always supported for a very long time. Personally I run Fedora with Mate or KDE and I upgrade the OS every 12-18 months, usually skipping a version or two. I did consider Centos 8 stream, but I needed something a little newer for various reasons. From cousinstanley at gmail.com Thu Mar 10 16:20:36 2022 From: cousinstanley at gmail.com (Cousin Stanley) Date: Thu, 10 Mar 2022 14:20:36 -0700 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> Message-ID: Marco Sulla wrote: >> >> Maybe Debian itself? > > I tried Debian on a VM, but I found it too much basical. A little > example: it does not have the shortcut ctrl+alt+t to open a terminal > that Ubuntu has. I'm quite sure it's simple to add, but I'm starting > to be old and lazy... > I use the current stable release of Debian 11.2 Bullseye with the lxqt desktop environment .... [Ctrl]-[Alt]-[T] will open a new qterminal window. The following will display a list of lxqt packages that are in the repository and available to install .... apt-cache search lxqt | grep ^lxqt -- Stanley C. Kitching Human Being Phoenix, Arizona From rosuav at gmail.com Thu Mar 10 21:50:10 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 11 Mar 2022 13:50:10 +1100 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> Message-ID: On Fri, 11 Mar 2022 at 09:51, Cousin Stanley wrote: > > Marco Sulla wrote: > > >> > >> Maybe Debian itself? > > > > I tried Debian on a VM, but I found it too much basical. A little > > example: it does not have the shortcut ctrl+alt+t to open a terminal > > that Ubuntu has. I'm quite sure it's simple to add, but I'm starting > > to be old and lazy... > > > > I use the current stable release of Debian 11.2 Bullseye > with the lxqt desktop environment .... > > [Ctrl]-[Alt]-[T] will open a new qterminal window. > > > The following will display a list of lxqt packages > that are in the repository and available to install .... > > apt-cache search lxqt | grep ^lxqt > Much faster: apt-cache pkgnames lxqt (apt-cache search will look for "lxqt" in descriptions too, hence the need to filter those out - apt-cache pkgnames is used by tab completion) ChrisA From drsalists at gmail.com Fri Mar 11 00:37:54 2022 From: drsalists at gmail.com (Dan Stromberg) Date: Thu, 10 Mar 2022 21:37:54 -0800 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> Message-ID: On Thu, Mar 10, 2022 at 5:04 AM Marco Sulla wrote: > On Thu, 10 Mar 2022 at 04:50, Michael Torrie wrote: > > > > On 3/9/22 13:05, Marco Sulla wrote: > > > So my laziness pays. I use only LTS distros, and I update only when > > > there are security updates. > > > PS: any suggestions for a new LTS distro? My Lubuntu is reaching its > > > end-of-life. I prefer lightweight debian-like distros. > > > > Maybe Debian itself? > > I tried Debian on a VM, but I found it too much basical. A little > example: it does not have the shortcut ctrl+alt+t to open a terminal > that Ubuntu has. I'm quite sure it's simple to add, but I'm starting > to be old and lazy... > That's an attribute of your desktop environment, not the Linux distribution. EG: I'm using Debian with Cinnamon, which does support ctrl-alt-t. Some folks say the desktop environment matters more than the distribution, when choosing what OS to install. From rosuav at gmail.com Fri Mar 11 00:49:39 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 11 Mar 2022 16:49:39 +1100 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> Message-ID: On Fri, 11 Mar 2022 at 16:39, Dan Stromberg wrote: > Some folks say the desktop environment matters more than the distribution, > when choosing what OS to install. Matters more to the choice? Impossible to say. Matters more to the UI? Without a doubt. ChrisA From loris.bennett at fu-berlin.de Fri Mar 11 01:40:52 2022 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Fri, 11 Mar 2022 07:40:52 +0100 Subject: How to test input via subprocess.Popen with data from file In-Reply-To: <25130.13419.321314.959918@ixdm.fritz.box> (Dieter Maurer's message of "Thu, 10 Mar 2022 18:24:59 +0100") References: <878rtiq9ys.fsf@hornfels.zedat.fu-berlin.de> <25130.13419.321314.959918@ixdm.fritz.box> Message-ID: <8735jpf0uj.fsf@hornfels.zedat.fu-berlin.de> Dieter Maurer writes: > Loris Bennett wrote at 2022-3-10 13:16 +0100: >>I have a command which produces output like the >>following: >> >> Job ID: 9431211 >> Cluster: curta >> User/Group: build/staff >> State: COMPLETED (exit code 0) >> Nodes: 1 >> Cores per node: 8 >> CPU Utilized: 01:30:53 >> CPU Efficiency: 83.63% of 01:48:40 core-walltime >> Job Wall-clock time: 00:13:35 >> Memory Utilized: 6.45 GB >> Memory Efficiency: 80.68% of 8.00 GB >> >>I want to parse this and am using subprocess.Popen and accessing the >>contents via Popen.stdout. However, for testing purposes I want to save >>various possible outputs of the command as text files and use those as >>inputs. > > What do you want to test? the parsing? the "popen" interaction? > You can separately test both tasks (I, at your place, would do this). I just want to test the parsing. > For the parsing test, it is not relevant that the actual text > comes from an external process. You can directly read it from a file > or have it in your text. As mentioned in the original post, for the tests I indeed want to read the input from files. > In my view, you do not need a test for the `Popen` interaction: > if it works once, it will work always. Sorry if I was unclear but my question is: Given that the return value from Popen is a Popen object and given that the return value from reading a file is a single string or maybe a list of strings, what should the common format for the argument which is passed to the actual parsing function be? Cheers, Loris -- This signature is currently under construction. From dieter at handshake.de Fri Mar 11 01:56:39 2022 From: dieter at handshake.de (Dieter Maurer) Date: Fri, 11 Mar 2022 07:56:39 +0100 Subject: How to test input via subprocess.Popen with data from file In-Reply-To: <8735jpf0uj.fsf@hornfels.zedat.fu-berlin.de> References: <878rtiq9ys.fsf@hornfels.zedat.fu-berlin.de> <25130.13419.321314.959918@ixdm.fritz.box> <8735jpf0uj.fsf@hornfels.zedat.fu-berlin.de> Message-ID: <25130.62119.387483.126247@ixdm.fritz.box> Loris Bennett wrote at 2022-3-11 07:40 +0100: > ... I want to test the parsing ... >Sorry if I was unclear but my question is: > >Given that the return value from Popen is a Popen object and given that >the return value from reading a file is a single string or maybe a list >of strings, what should the common format for the argument which is >passed to the actual parsing function be? What methods (of its input argument) does the parsing use? If it uses `Popen` methods, then you pass a `POpen` object; if it uses only (typical) file methods, then you pass a file object; if it assumes its input to be a (line) interator, you pass a (line) iterator (such as a "file" object). I would design the parsing that it makes as few assumptions about its input as possible -- to ease testing and increase the chance for reuse. That said, I would not design it to work with `Popen` objects but likely to have a line iterator as input. From roel at roelschroeven.net Fri Mar 11 03:56:05 2022 From: roel at roelschroeven.net (Roel Schroeven) Date: Fri, 11 Mar 2022 09:56:05 +0100 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> Message-ID: <001fde04-bacd-3777-7f7d-bfd5500f8f5a@roelschroeven.net> Op 11/03/2022 om 3:50 schreef Chris Angelico: > On Fri, 11 Mar 2022 at 09:51, Cousin Stanley wrote: > > The following will display a list of lxqt packages > > that are in the repository and available to install .... > > > > apt-cache search lxqt | grep ^lxqt > > > Much faster: > > apt-cache pkgnames lxqt > > (apt-cache search will look for "lxqt" in descriptions too, hence the > need to filter those out - apt-cache pkgnames is used by tab > completion) > Cousing Stanley's suggestion has the advantage that it also prints the short descriptions instead of just the package names. The packages names are often a bit too cryptic, I think. To search only in package names but still show the short description you could also --names-only instead of grep: ??? apt-cache search --names-only lxqt or ??? apt-cache search --names-only ^lxqt Chris, when you say "Much faster", do you mean faster to type or faster to execute? Your suggestion is certainly faster and easier to type. But as for execution speed: on my systems apt-cache search is fast enough that I don't really care about its execution time. When listing packages that is; tab completion is a different matter, where every delay can be quite annoying. -- "Don't Panic." -- Douglas Adams, The Hitchhiker's Guide to the Galaxy From rosuav at gmail.com Fri Mar 11 04:01:54 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 11 Mar 2022 20:01:54 +1100 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: <001fde04-bacd-3777-7f7d-bfd5500f8f5a@roelschroeven.net> References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> <001fde04-bacd-3777-7f7d-bfd5500f8f5a@roelschroeven.net> Message-ID: On Fri, 11 Mar 2022 at 19:57, Roel Schroeven wrote: > > Op 11/03/2022 om 3:50 schreef Chris Angelico: > > On Fri, 11 Mar 2022 at 09:51, Cousin Stanley wrote: > > > The following will display a list of lxqt packages > > > that are in the repository and available to install .... > > > > > > apt-cache search lxqt | grep ^lxqt > > > > > Much faster: > > > > apt-cache pkgnames lxqt > > > > (apt-cache search will look for "lxqt" in descriptions too, hence the > > need to filter those out - apt-cache pkgnames is used by tab > > completion) > > > Cousing Stanley's suggestion has the advantage that it also prints the > short descriptions instead of just the package names. The packages names > are often a bit too cryptic, I think. > > To search only in package names but still show the short description you > could also --names-only instead of grep: > > apt-cache search --names-only lxqt > > or > > apt-cache search --names-only ^lxqt > > Chris, when you say "Much faster", do you mean faster to type or faster > to execute? Your suggestion is certainly faster and easier to type. But > as for execution speed: on my systems apt-cache search is fast enough > that I don't really care about its execution time. When listing packages > that is; tab completion is a different matter, where every delay can be > quite annoying. Notably faster to execute on my system. Maybe the difference is insignificant on yours, but it's highly unlikely to be *slower*. The speed difference probably depends on how many package repositories you have, whether you have source packages, etc, etc, etc. (And of course, whether it's on a hard drive or SSD.) And what I would *actually* type is "apt i lxqt", so it's quite a lot faster. (I actually had to look up the precise command.) ChrisA From roel at roelschroeven.net Fri Mar 11 04:11:26 2022 From: roel at roelschroeven.net (Roel Schroeven) Date: Fri, 11 Mar 2022 10:11:26 +0100 Subject: How to test input via subprocess.Popen with data from file In-Reply-To: <878rtiq9ys.fsf@hornfels.zedat.fu-berlin.de> References: <878rtiq9ys.fsf@hornfels.zedat.fu-berlin.de> Message-ID: <33e51b22-3d1e-7c17-7aac-385294712120@roelschroeven.net> Op 10/03/2022 om 13:16 schreef Loris Bennett: > Hi, > > I have a command which produces output like the > following: > > Job ID: 9431211 > Cluster: curta > User/Group: build/staff > State: COMPLETED (exit code 0) > Nodes: 1 > Cores per node: 8 > CPU Utilized: 01:30:53 > CPU Efficiency: 83.63% of 01:48:40 core-walltime > Job Wall-clock time: 00:13:35 > Memory Utilized: 6.45 GB > Memory Efficiency: 80.68% of 8.00 GB > > I want to parse this and am using subprocess.Popen and accessing the > contents via Popen.stdout. However, for testing purposes I want to save > various possible outputs of the command as text files and use those as > inputs. > > What format should I use to pass data to the actual parsing function? > Is this a command you run, produces that output, and then stops (as opposed to a long-running program that from time to time generates a bunch of output)? Because in that case I would use subprocess.run() with capture_output=True instead of subprocess.Popen(). subprocess.run() returns a CompletedProcess instance wich has stdout and stderr members that contain the captured output as byte sequences or strings, depending on the parameters you passed. So in your case I would simply read the content of each text file as a whole into a string, and use subprocess.run() to get the command's output also as a string. Then you can have a parse function that accepts such strings, and works exactly the same for the text files as for the command output. Your parse function can then use splitlines() to access the lines individually. The data size is very small so it's not a problem to have it all in memory at the same time (i.e. no need to worry about trying to stream it instead). -- "Don't Panic." -- Douglas Adams, The Hitchhiker's Guide to the Galaxy From roel at roelschroeven.net Fri Mar 11 04:30:44 2022 From: roel at roelschroeven.net (Roel Schroeven) Date: Fri, 11 Mar 2022 10:30:44 +0100 Subject: How to test input via subprocess.Popen with data from file In-Reply-To: <33e51b22-3d1e-7c17-7aac-385294712120@roelschroeven.net> References: <878rtiq9ys.fsf@hornfels.zedat.fu-berlin.de> <33e51b22-3d1e-7c17-7aac-385294712120@roelschroeven.net> Message-ID: <43e8e4dd-aa45-94db-9307-7d6219de04d2@roelschroeven.net> Op 11/03/2022 om 10:11 schreef Roel Schroeven: > Op 10/03/2022 om 13:16 schreef Loris Bennett: >> Hi, >> >> I have a command which produces output like the >> following: >> >> ?? Job ID: 9431211 >> ?? Cluster: curta >> ?? User/Group: build/staff >> ?? State: COMPLETED (exit code 0) >> ?? Nodes: 1 >> ?? Cores per node: 8 >> ?? CPU Utilized: 01:30:53 >> ?? CPU Efficiency: 83.63% of 01:48:40 core-walltime >> ?? Job Wall-clock time: 00:13:35 >> ?? Memory Utilized: 6.45 GB >> ?? Memory Efficiency: 80.68% of 8.00 GB >> >> I want to parse this and am using subprocess.Popen and accessing the >> contents via Popen.stdout.? However, for testing purposes I want to save >> various possible outputs of the command as text files and use those as >> inputs. >> >> What format should I use to pass data to the actual parsing function? >> > Is this a command you run, produces that output, and then stops (as > opposed to a long-running program that from time to time generates a > bunch of output)? > Because in that case I would use subprocess.run() with > capture_output=True instead of subprocess.Popen(). subprocess.run() > returns a CompletedProcess instance wich has stdout and stderr members > that contain the captured output as byte sequences or strings, > depending on the parameters you passed. > > So in your case I would simply read the content of each text file as a > whole into a string, and use subprocess.run() to get the command's > output also as a string. Then you can have a parse function that > accepts such strings, and works exactly the same for the text files as > for the command output. Your parse function can then use splitlines() > to access the lines individually. The data size is very small so it's > not a problem to have it all in memory at the same time (i.e. no need > to worry about trying to stream it instead). > Very simple example: ??? import subprocess ??? from pprint import pprint ??? def parse(state_data): ??????? lines = state_data.splitlines(keepends=False) ??????? state_dict = {} ??????? for line in lines: ??????????? key, value = line.split(': ') ??????????? state_dict[key] = value ??????? return state_dict ??? def read_from_command(): ??????? return subprocess.run(['./jobstate'], capture_output=True, check=True, encoding='UTF-8').stdout ??? def read_from_file(fn): ??????? with open(fn, 'rt', encoding='UTF-8') as f: ??????????? return f.read() ??? pprint(parse(read_from_command())) ??? pprint(parse(read_from_file('jobfile'))) -- "Iceland is the place you go to remind yourself that planet Earth is a machine... and that all organic life that has ever existed amounts to a greasy film that has survived on the exterior of that machine thanks to furious improvisation." -- Sam Hughes, Ra From PythonList at DancesWithMice.info Fri Mar 11 04:53:28 2022 From: PythonList at DancesWithMice.info (dn) Date: Fri, 11 Mar 2022 22:53:28 +1300 Subject: Invitation to join AuckPUG's virtual meeting on Testing Message-ID: <96570017-6784-ee9c-c6df-f07fd0fdbb34@DancesWithMice.info> "The Alex to Zac of Testing" 0530~0730 UTC (1830~2030 NZDT), Wed 16Mar2022 Alex Znamensky (AuckPUG member) will introduce testing as a practice (what your training course didn't tell you about the real-world!). Zac Hatfield-Dodds will aim at more advanced users, talking about Property-Based testing and the Hypothesis library (https://hypothesis.readthedocs.io/en/latest/). Originally based at the Australian National University, he has recently moved and will come to us from San Francisco. We use a BigBlueButton-instance for web-conferencing. A recent web-browser and head-set should enable you to join us. No downloads/plug-ins etc. Further details including access instructions: https://www.meetup.com/NZPUG-Auckland/ or email me (off-list)... -- Regards, =dn From Marco.Sulla.Python at gmail.com Fri Mar 11 13:03:21 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Fri, 11 Mar 2022 19:03:21 +0100 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> Message-ID: On Fri, 11 Mar 2022 at 06:38, Dan Stromberg wrote: > That's an attribute of your desktop environment, not the Linux distribution. > > EG: I'm using Debian with Cinnamon, which does support ctrl-alt-t. Never used Cinnamon. It comes from Mint, right? > Some folks say the desktop environment matters more than the distribution, when choosing what OS to install. Yes, it's important. I switched from Ubuntu to Xubuntu (then Lubuntu) when Ubuntu started using Unity. I liked GNOME 2 and KDE prior to Plasma. They were simple, lightweight and effective. I found these qualities in XFCE and LXDE. Anyway I think I'll not install Debian, because it's LTS releases are not long enough for me. I don't know if there's a distro based on Debian that has a long LTS support, Ubuntu apart. From torriem at gmail.com Fri Mar 11 13:08:19 2022 From: torriem at gmail.com (Michael Torrie) Date: Fri, 11 Mar 2022 11:08:19 -0700 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> Message-ID: On 3/11/22 11:03, Marco Sulla wrote: > Anyway I think I'll not install Debian, because it's LTS releases are > not long enough for me. I don't know if there's a distro based on > Debian that has a long LTS support, Ubuntu apart. Both Debian stable and Ubuntu LTS state they have a five year support life cycle. Ubuntu will support longer if you pay for it. Do you require more than five years? Anyway, use whatever works for you. From Marco.Sulla.Python at gmail.com Fri Mar 11 15:50:06 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Fri, 11 Mar 2022 21:50:06 +0100 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> Message-ID: On Fri, 11 Mar 2022 at 19:10, Michael Torrie wrote: > Both Debian stable and Ubuntu LTS state they have a five year support > life cycle. Yes, but it seems that official security support in Debian ends after three years: "Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success" https://wiki.debian.org/LTS This is the only problem for me. From toby at tobiah.org Fri Mar 11 12:38:07 2022 From: toby at tobiah.org (Tobiah) Date: Fri, 11 Mar 2022 09:38:07 -0800 Subject: How to test input via subprocess.Popen with data from file References: <878rtiq9ys.fsf@hornfels.zedat.fu-berlin.de> Message-ID: Why not just have scripts that echo out the various sets of test data you are interested in? That way, Popen would always be your interface and you wouldn't have to make two cases in the consumer script. In other words, make program that outputs test data just like your main data source program. Then the consumer would only have to work in one way. On 3/10/22 04:16, Loris Bennett wrote: > Hi, > > I have a command which produces output like the > following: > > Job ID: 9431211 > Cluster: curta > User/Group: build/staff > State: COMPLETED (exit code 0) > Nodes: 1 > Cores per node: 8 > CPU Utilized: 01:30:53 > CPU Efficiency: 83.63% of 01:48:40 core-walltime > Job Wall-clock time: 00:13:35 > Memory Utilized: 6.45 GB > Memory Efficiency: 80.68% of 8.00 GB > > I want to parse this and am using subprocess.Popen and accessing the > contents via Popen.stdout. However, for testing purposes I want to save > various possible outputs of the command as text files and use those as > inputs. > > What format should I use to pass data to the actual parsing function? > > I could in both production and test convert the entire input to a string > and pass the string to the parsing method. > > However, I could use something like > > test_input_01 = subprocess.Popen( > ["cat test_input_01.txt"], > stdout=subprocess.PIPE, > ) > > for the test input and then pass a Popen object to the parsing function. > > Any comments on these alternative or suggestions for doing something > completely different? > > Cheers, > > Loris > From cousinstanley at gmail.com Fri Mar 11 14:03:02 2022 From: cousinstanley at gmail.com (Cousin Stanley) Date: Fri, 11 Mar 2022 12:03:02 -0700 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> Message-ID: Cousin Stanley wrote: >> apt-cache search lxqt | grep ^lxqt Chris Angelico wrote: > Much faster: > > apt-cache pkgnames lxqt > > apt-cache search will look for "lxqt" in descriptions too, > hence the need to filter those out > > apt-cache pkgnames is used by tab completion) > Thanks .... I didn't know about using pkgnames with apt-cache. -- Stanley C. Kitching Human Being Phoenix, Arizona From python at boiteameuh.org Sat Mar 12 12:05:43 2022 From: python at boiteameuh.org (Nicolas Haller) Date: Sat, 12 Mar 2022 12:05:43 -0500 Subject: Typing on child class' methods of a Generic base class In-Reply-To: <25130.13791.689201.429804@ixdm.fritz.box> References: <3fccbe66-4d1c-f40c-7282-026ebe7360e3@boiteameuh.org> <25130.13791.689201.429804@ixdm.fritz.box> Message-ID: <01e56985-280e-f5c3-6bf4-1d7c1a101284@boiteameuh.org> On 2022-03-10 12:31, Dieter Maurer wrote: > Nicolas Haller wrote at 2022-3-9 10:53 -0500: >> ... >> The documentation about "user-defined generic types"[1] says that I can >> fix some types on a child class (class MyDict(Mapping[str, T]):) but >> doesn't say much about the signature of the methods I need to >> implement/override on that child class. > > I have the fealing that this is a case of (generic type) "specialization". > In this setup, `Mapping` would be a generic type with two type > variables `K` (the key type) and `V` (the value type). > The signatures of its methods would use those type variables. > In your example, you specialize the key type to `str` (and > leave the value type generic). The signatures of the methods > would automatically follow this specialization -- without the need > to do anything. If I understand correctly, you're saying I should use the first alternative by keeping the signature as it is in the base class: --- T = TypeVar("T") class Base(Generic[T], metaclass=ABCMeta): """A base class.""" @abstractmethod def _private_method(self, an_arg: T) -> T: ... def public_method(self, an_arg: T) -> T: from_private_method = self._private_method(an_arg) return from_private_method class Alternative1(Base[int]): def _private_method(self, an_arg: T) -> T: # I keep T return 42 --- The problem with it is that mypy doesn?t seem quite happy with it: ./scratch.py:22: error: Incompatible return value type (got "int", expected "T") Do you think this is error is incorrect? If so, I can open an issue with mypy. Thanks, -- Nicolas From dieter at handshake.de Sat Mar 12 13:24:59 2022 From: dieter at handshake.de (Dieter Maurer) Date: Sat, 12 Mar 2022 19:24:59 +0100 Subject: Typing on child class' methods of a Generic base class In-Reply-To: <01e56985-280e-f5c3-6bf4-1d7c1a101284@boiteameuh.org> References: <3fccbe66-4d1c-f40c-7282-026ebe7360e3@boiteameuh.org> <25130.13791.689201.429804@ixdm.fritz.box> <01e56985-280e-f5c3-6bf4-1d7c1a101284@boiteameuh.org> Message-ID: <25132.58747.285988.346583@ixdm.fritz.box> Nicolas Haller wrote at 2022-3-12 12:05 -0500: >On 2022-03-10 12:31, Dieter Maurer wrote: >> Nicolas Haller wrote at 2022-3-9 10:53 -0500: >>> ... >>> The documentation about "user-defined generic types"[1] says that I can >>> fix some types on a child class (class MyDict(Mapping[str, T]):) but >>> doesn't say much about the signature of the methods I need to >>> implement/override on that child class. >> >> I have the fealing that this is a case of (generic type) "specialization". >> In this setup, `Mapping` would be a generic type with two type >> variables `K` (the key type) and `V` (the value type). >> The signatures of its methods would use those type variables. >> In your example, you specialize the key type to `str` (and >> leave the value type generic). The signatures of the methods >> would automatically follow this specialization -- without the need >> to do anything. > >If I understand correctly, you're saying I should use the first >alternative by keeping the signature as it is in the base class: >--- >T = TypeVar("T") > > >class Base(Generic[T], metaclass=ABCMeta): > """A base class.""" > > @abstractmethod > def _private_method(self, an_arg: T) -> T: > ... > > def public_method(self, an_arg: T) -> T: > from_private_method = self._private_method(an_arg) > return from_private_method > >class Alternative1(Base[int]): > def _private_method(self, an_arg: T) -> T: # I keep T > return 42 >--- > >The problem with it is that mypy doesn?t seem quite happy with it: >./scratch.py:22: error: Incompatible return value type (got "int", >expected "T") > >Do you think this is error is incorrect? No, I do not think so. The `Base[int]` means that the generic type variable `T` was specialized to `int`. This means that in this context `_private_method` returns `int` (not a generic `T`). You either do not type annotate the overriding methods or you use the specialization (if any). From Marco.Sulla.Python at gmail.com Sat Mar 12 15:45:56 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Sat, 12 Mar 2022 21:45:56 +0100 Subject: Best practice for caching hash Message-ID: I have a custom immutable object, and I added a cache for its hash value. The problem is the object can be composed of mutable or immutable objects, so the hash can raise TypeError. In this case I currently cache the value -1. The subsequent calls to __hash__() will check if the value is -1. If so, a TypeError is immediately raised. The problem is the first time I get an error with details, for example: TypeError: unhashable type: 'list' The subsequent times I simply raise a generic error: TypeError Ok, I can improve it by raising, for example, TypeError: not all values are hashable. But do you think this is acceptable? Now I'm thinking about it and it seems a little hacky to me. Furthermore, in the C extension I have to define another property in the struct, ma_hash_calculated, to track if the hash value is cached or not, since there's no bogus value I can use in cache property, ma_hash, to signal this. If I don't cache unhashable values, -1 can be used to signal that ma_hash contains no cached value. So if I do not cache if the object is unhashable, I save a little memory per object (1 int) and I get a better error message every time. On the other hand, if I leave the things as they are, testing the unhashability of the object multiple times is faster. The code: try: hash(o) except TypeError: pass execute in nanoseconds, if called more than 1 time, even if o is not hashable. Not sure if this is a big advantage. What do you think about? Here is the python code: https://github.com/Marco-Sulla/python-frozendict/blob/35611f4cd869383678104dc94f82aa636c20eb24/frozendict/src/3_10/frozendictobject.c#L652-L697 From jenkris at tutanota.com Sat Mar 12 16:24:09 2022 From: jenkris at tutanota.com (Jen Kris) Date: Sat, 12 Mar 2022 22:24:09 +0100 (CET) Subject: Problem slicing a list with the C API Message-ID: I have a C API project where I have to slice a list into two parts.?? Unfortunately the documentation on the slice objects is not clear enough for me to understand how to do this, and I haven?t found enough useful info through research.? The list contains tuple records where each tuple consists of a dictionary object and a string.? The relevant part of the Python code is: half_slice = int(len(dictdata) * 0.5) subdata_a = dictdata[half_slice:] subdata_b = dictdata[:half_slice] This is what I?ve done so far with the C API: int64_t Calc_Slices(PyObject* pDictdata, int64_t records_count) { long round_half = records_count * 0.5; PyObject* half_slice = PyLong_FromLong(round_half); PyObject* slice = PySlice_New(PyLong_FromLong(0), half_slice, 0); PyObject* subdata_a = PyObject_GetItem(pDictddata, slice); return 0; } On the final line (subdata_a) I get a segfault.? I know that the second parameter of? PyObject_GetItem is a ?key? and I suspect that?s where the problem comes from, but I don?t understand what a key is in this context.? The code shown above omits error handling but none of the objects leading up to the final line is null, they all succeed.? Thanks for any ideas. Jen From 2QdxY4RzWzUUiLuE at potatochowder.com Sat Mar 12 16:16:03 2022 From: 2QdxY4RzWzUUiLuE at potatochowder.com (2QdxY4RzWzUUiLuE at potatochowder.com) Date: Sat, 12 Mar 2022 16:16:03 -0500 Subject: Best practice for caching hash In-Reply-To: References: Message-ID: On 2022-03-12 at 21:45:56 +0100, Marco Sulla wrote: [ ... ] > So if I do not cache if the object is unhashable, I save a little > memory per object (1 int) and I get a better error message every time. > On the other hand, if I leave the things as they are, testing the > unhashability of the object multiple times is faster. The code: > > try: > hash(o) > except TypeError: > pass > > execute in nanoseconds, if called more than 1 time, even if o is not > hashable. Not sure if this is a big advantage. Once hashing an object fails, why would an application try again? I can see an application using a hashable value in a hashable situation again and again and again (i.e., taking advantage of the cache), but what's the use case for *repeatedly* trying to use an unhashable value again and again and again (i.e., taking advantage of a cached failure)? So I think that caching the failure is a lot of extra work for no benefit. From python at mrabarnett.plus.com Sat Mar 12 16:40:38 2022 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 12 Mar 2022 21:40:38 +0000 Subject: Problem slicing a list with the C API In-Reply-To: References: Message-ID: On 2022-03-12 21:24, Jen Kris via Python-list wrote: > I have a C API project where I have to slice a list into two parts.?? Unfortunately the documentation on the slice objects is not clear enough for me to understand how to do this, and I haven?t found enough useful info through research.? The list contains tuple records where each tuple consists of a dictionary object and a string. > > The relevant part of the Python code is: > > half_slice = int(len(dictdata) * 0.5) > subdata_a = dictdata[half_slice:] > subdata_b = dictdata[:half_slice] > > This is what I?ve done so far with the C API: > > int64_t Calc_Slices(PyObject* pDictdata, int64_t records_count) > { > long round_half = records_count * 0.5; > PyObject* half_slice = PyLong_FromLong(round_half); > > PyObject* slice = PySlice_New(PyLong_FromLong(0), half_slice, 0); > PyObject* subdata_a = PyObject_GetItem(pDictddata, slice); > > return 0; > } > > On the final line (subdata_a) I get a segfault.? I know that the second parameter of? PyObject_GetItem is a ?key? and I suspect that?s where the problem comes from, but I don?t understand what a key is in this context. > > The code shown above omits error handling but none of the objects leading up to the final line is null, they all succeed. > > Thanks for any ideas. > Use PySequence_GetSlice to slice the list. Also, why use floats when you can use integers? long round_half = records_count / 2; (In Python that would be half_slice = len(dictdata) // 2.) From rosuav at gmail.com Sat Mar 12 16:41:12 2022 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 13 Mar 2022 08:41:12 +1100 Subject: Problem slicing a list with the C API In-Reply-To: References: Message-ID: On Sun, 13 Mar 2022 at 08:25, Jen Kris via Python-list wrote: > PyObject* slice = PySlice_New(PyLong_FromLong(0), half_slice, 0); > PyObject* subdata_a = PyObject_GetItem(pDictddata, slice); > > On the final line (subdata_a) I get a segfault. I know that the second parameter of PyObject_GetItem is a ?key? and I suspect that?s where the problem comes from, but I don?t understand what a key is in this context. > The key is simply whatever would be in the square brackets in Python code, so that part looks fine. But dictionaries aren't usually subscripted with slices, so I'm a bit confused as to what's going on here. What exactly is dictdata/pDictdata? Have you confirmed that pDictdata (a) isn't NULL, (b) is the object you intend it to be, and (c) contains the objects you expect it to? The segfault might not be from the slice object itself, it might be from actually iterating over the thing being sliced and touching all its elements. For instance, if dictdata is actually a list, that call will be constructing a new list with references to the same elements, so if one of them is broken (maybe NULL), it'll break badly. ChrisA From jenkris at tutanota.com Sat Mar 12 16:52:48 2022 From: jenkris at tutanota.com (Jen Kris) Date: Sat, 12 Mar 2022 22:52:48 +0100 (CET) Subject: Problem slicing a list with the C API In-Reply-To: References: Message-ID: Thanks to you both.? I am going to implement PySequence_Get_Slice now.? If I have trouble then, per comments from Chris Angelico, I will iterate through pDictData to verify it because I haven't done that.? It is not null, however.? ?Jen Mar 12, 2022, 13:40 by python at mrabarnett.plus.com: > On 2022-03-12 21:24, Jen Kris via Python-list wrote: > >> I have a C API project where I have to slice a list into two parts.?? Unfortunately the documentation on the slice objects is not clear enough for me to understand how to do this, and I haven?t found enough useful info through research.? The list contains tuple records where each tuple consists of a dictionary object and a string. >> >> The relevant part of the Python code is: >> >> half_slice = int(len(dictdata) * 0.5) >> subdata_a = dictdata[half_slice:] >> subdata_b = dictdata[:half_slice] >> >> This is what I?ve done so far with the C API: >> >> int64_t Calc_Slices(PyObject* pDictdata, int64_t records_count) >> { >> long round_half = records_count * 0.5; >> PyObject* half_slice = PyLong_FromLong(round_half); >> >> PyObject* slice = PySlice_New(PyLong_FromLong(0), half_slice, 0); >> PyObject* subdata_a = PyObject_GetItem(pDictddata, slice); >> >> return 0; >> } >> >> On the final line (subdata_a) I get a segfault.? I know that the second parameter of? PyObject_GetItem is a ?key? and I suspect that?s where the problem comes from, but I don?t understand what a key is in this context. >> >> The code shown above omits error handling but none of the objects leading up to the final line is null, they all succeed. >> >> Thanks for any ideas. >> > Use PySequence_GetSlice to slice the list. > > Also, why use floats when you can use integers? > > long round_half = records_count / 2; > > (In Python that would be half_slice = len(dictdata) // 2.) > -- > https://mail.python.org/mailman/listinfo/python-list > From jenkris at tutanota.com Sat Mar 12 16:54:50 2022 From: jenkris at tutanota.com (Jen Kris) Date: Sat, 12 Mar 2022 22:54:50 +0100 (CET) Subject: Problem slicing a list with the C API In-Reply-To: References: Message-ID: pDictData, despite the name, is a list of 2-tuples where each 2-tuple is a dictionary object and a string.? Mar 12, 2022, 13:41 by rosuav at gmail.com: > On Sun, 13 Mar 2022 at 08:25, Jen Kris via Python-list > wrote: > >> PyObject* slice = PySlice_New(PyLong_FromLong(0), half_slice, 0); >> PyObject* subdata_a = PyObject_GetItem(pDictddata, slice); >> >> On the final line (subdata_a) I get a segfault. I know that the second parameter of PyObject_GetItem is a ?key? and I suspect that?s where the problem comes from, but I don?t understand what a key is in this context. >> > > The key is simply whatever would be in the square brackets in Python > code, so that part looks fine. > > But dictionaries aren't usually subscripted with slices, so I'm a bit > confused as to what's going on here. What exactly is > dictdata/pDictdata? > > Have you confirmed that pDictdata (a) isn't NULL, (b) is the object > you intend it to be, and (c) contains the objects you expect it to? > The segfault might not be from the slice object itself, it might be > from actually iterating over the thing being sliced and touching all > its elements. For instance, if dictdata is actually a list, that call > will be constructing a new list with references to the same elements, > so if one of them is broken (maybe NULL), it'll break badly. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > From rosuav at gmail.com Sat Mar 12 16:57:27 2022 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 13 Mar 2022 08:57:27 +1100 Subject: Problem slicing a list with the C API In-Reply-To: References: Message-ID: On Sun, 13 Mar 2022 at 08:54, Jen Kris wrote: > > > pDictData, despite the name, is a list of 2-tuples where each 2-tuple is a dictionary object and a string. > Ah, gotcha. In that case, yeah, slicing it will involve referencing the tuples all the way down the line (adding to their refcounts, so if there's a borked one, kaboom). ChrisA From jenkris at tutanota.com Sat Mar 12 18:30:55 2022 From: jenkris at tutanota.com (Jen Kris) Date: Sun, 13 Mar 2022 00:30:55 +0100 (CET) Subject: Problem slicing a list with the C API In-Reply-To: References: Message-ID: Chris, you were right to focus on the list pDictData itself.?? As I said, that is a list of 2-tuples, but I added each of the 2-tuples with PyList_Append, but you can only append a tuple to a list with the extend method.? However, there is no append method in the C API as far as I can tell -- hence pDictData is empty.? I tried with PyList_SetItem but that doesn't work.? Do you know of way to "extend" a list in the C API.? Thanks very much.? Jen Mar 12, 2022, 13:57 by rosuav at gmail.com: > On Sun, 13 Mar 2022 at 08:54, Jen Kris wrote: > >> >> >> pDictData, despite the name, is a list of 2-tuples where each 2-tuple is a dictionary object and a string. >> > > Ah, gotcha. In that case, yeah, slicing it will involve referencing > the tuples all the way down the line (adding to their refcounts, so if > there's a borked one, kaboom). > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > From rosuav at gmail.com Sat Mar 12 18:36:02 2022 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 13 Mar 2022 10:36:02 +1100 Subject: Problem slicing a list with the C API In-Reply-To: References: Message-ID: On Sun, 13 Mar 2022 at 10:30, Jen Kris wrote: > > > Chris, you were right to focus on the list pDictData itself. As I said, that is a list of 2-tuples, but I added each of the 2-tuples with PyList_Append, but you can only append a tuple to a list with the extend method. However, there is no append method in the C API as far as I can tell -- hence pDictData is empty. I tried with PyList_SetItem but that doesn't work. Do you know of way to "extend" a list in the C API. > Hmm. Not entirely sure I understand the question. In Python, a list has an append method, which takes any object (which may be a tuple) and adds that object to the end of the list: >>> x = ["spam", "ham"] >>> x.append((1,2)) >>> x ['spam', 'ham', (1, 2)] A list also has an extend method, which takes any sequence (that also includes tuples), and adds *the elements from it* to the end of the list: >>> x = ["spam", "ham"] >>> x.extend((1,2)) >>> x ['spam', 'ham', 1, 2] The append method corresponds to PyList_Append, as you mentioned. It should be quite happy to append a tuple, and will add the tuple itself, not the contents of it. So when you iterate over the list, you'll get tuples. Extending a list can be done with the sequence API. In Python, you can write extend() as +=, indicating that you're adding something onto the end: >>> x = ["spam", "ham"] >>> x += (1, 2) >>> x ['spam', 'ham', 1, 2] This corresponds to PySequence_InPlaceConcat, so if that's the behaviour you want, that would be the easiest way to do it. Based on your other comments, I would suspect that appending the tuples is probably what you want here? ChrisA From jenkris at tutanota.com Sat Mar 12 18:41:31 2022 From: jenkris at tutanota.com (Jen Kris) Date: Sun, 13 Mar 2022 00:41:31 +0100 (CET) Subject: Problem slicing a list with the C API In-Reply-To: References: Message-ID: Thanks for PySequence_InPlaceConcat, so when I need to extend I'll know what to use.? But my previous email was based on incorrect information from several SO posts that claimed only the extend method will work to add tuples to a list.? I found that's wrong -- even my own Python code uses the append method.? But my PyList_Append is not doing the job so that's where I'm looking now.? Thanks very much for your reply. Mar 12, 2022, 15:36 by rosuav at gmail.com: > On Sun, 13 Mar 2022 at 10:30, Jen Kris wrote: > >> >> >> Chris, you were right to focus on the list pDictData itself. As I said, that is a list of 2-tuples, but I added each of the 2-tuples with PyList_Append, but you can only append a tuple to a list with the extend method. However, there is no append method in the C API as far as I can tell -- hence pDictData is empty. I tried with PyList_SetItem but that doesn't work. Do you know of way to "extend" a list in the C API. >> > > Hmm. Not entirely sure I understand the question. > > In Python, a list has an append method, which takes any object (which > may be a tuple) and adds that object to the end of the list: > >>>> x = ["spam", "ham"] >>>> x.append((1,2)) >>>> x >>>> > ['spam', 'ham', (1, 2)] > > A list also has an extend method, which takes any sequence (that also > includes tuples), and adds *the elements from it* to the end of the > list: > >>>> x = ["spam", "ham"] >>>> x.extend((1,2)) >>>> x >>>> > ['spam', 'ham', 1, 2] > > The append method corresponds to PyList_Append, as you mentioned. It > should be quite happy to append a tuple, and will add the tuple > itself, not the contents of it. So when you iterate over the list, > you'll get tuples. > > Extending a list can be done with the sequence API. In Python, you can > write extend() as +=, indicating that you're adding something onto the > end: > >>>> x = ["spam", "ham"] >>>> x += (1, 2) >>>> x >>>> > ['spam', 'ham', 1, 2] > > This corresponds to PySequence_InPlaceConcat, so if that's the > behaviour you want, that would be the easiest way to do it. > > Based on your other comments, I would suspect that appending the > tuples is probably what you want here? > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > From rosuav at gmail.com Sat Mar 12 18:50:26 2022 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 13 Mar 2022 10:50:26 +1100 Subject: Problem slicing a list with the C API In-Reply-To: References: Message-ID: On Sun, 13 Mar 2022 at 10:41, Jen Kris wrote: > > > Thanks for PySequence_InPlaceConcat, so when I need to extend I'll know what to use. But my previous email was based on incorrect information from several SO posts that claimed only the extend method will work to add tuples to a list. I found that's wrong -- even my own Python code uses the append method. But my PyList_Append is not doing the job so that's where I'm looking now. > Ah. I guess that's one of the fundamental vulnerabilities of Stack Overflow: people answer the question in front of them, which isn't necessarily the question that YOU are trying to answer. Sometimes, the solutions can be misleading, because your goal is actually different. Fortunately, tuples in Python are objects, like every other value. It's extremely convenient - to word that another way, it's extremely INconvenient to work with a language where that isn't the case, and you need to work differently depending on whether you're using integers or strings. This is how you work with a trie (actually an artifact name now, as it's implemented with a hashtable) in SourceMod - this is the equivalent of dictionary subscript assignment: https://sm.alliedmods.net/new-api/adt_trie/SetTrieValue - works for integers, floats, and handles (which are integers) https://sm.alliedmods.net/new-api/adt_trie/SetTrieString - works for strings Yup, it's annoying :) ChrisA From renesd at gmail.com Sun Mar 13 09:10:47 2022 From: renesd at gmail.com (=?UTF-8?Q?Ren=C3=A9_Dudfield?=) Date: Sun, 13 Mar 2022 14:10:47 +0100 Subject: Do you support The #Python Software Foundation making a statement in support of peace? Message-ID: hey hey, Do you support The Python Software Foundation making a statement in support of peace? Vote here: https://twitter.com/pygame_org/status/1502989885296238593 cheerio, From nagylzs at gmail.com Sun Mar 13 16:55:36 2022 From: nagylzs at gmail.com (Les) Date: Sun, 13 Mar 2022 21:55:36 +0100 Subject: Reportlab / platypus bug? Message-ID: Hello, I have found an error, and I created a minimal working example. The minimal working example starts with the very first example from Platypus user guide: from reportlab.platypus import SimpleDocTemplate, Paragraph, Spacer from reportlab.lib.styles import getSampleStyleSheet from reportlab.lib.pagesizes import A4 from reportlab.lib.units import inch PAGE_HEIGHT = A4[1] PAGE_WIDTH = A4[0] styles = getSampleStyleSheet() Title = "Hello world" pageinfo = "platypus example" def myFirstPage(canvas, doc): canvas.saveState() canvas.setFont('Times-Bold', 16) canvas.drawCentredString(PAGE_WIDTH / 2.0, PAGE_HEIGHT - 108, Title) canvas.setFont('Times-Roman', 9) canvas.drawString(inch, 0.75 * inch, "First Page / %s" % pageinfo) canvas.restoreState() def myLaterPages(canvas, doc): canvas.saveState() canvas.setFont('Times-Roman', 9) canvas.drawString(inch, 0.75 * inch, "Page %d %s" % (doc.page, pageinfo)) canvas.restoreState() def go(): Story = [Spacer(1, 2 * inch)] style = styles["Normal"] for i in range(100): bogustext = ("This is Paragraph number %s. " % i) * 20 p = Paragraph(bogustext, style) Story.append(p) Story.append(Spacer(1, 0.2 * inch)) doc = SimpleDocTemplate("phello.pdf") doc.build(Story, onFirstPage=myFirstPage, onLaterPages=myLaterPages) go() If I change it to this (e.g. generate two identical files): doc = SimpleDocTemplate("phello.pdf") doc.build(Story, onFirstPage=myFirstPage, onLaterPages=myLaterPages) doc = SimpleDocTemplate("phello2.pdf") doc.build(Story, onFirstPage=myFirstPage, onLaterPages=myLaterPages) then it builds phello.pdf correctly, but builds a totally empty phello2.pdf (960 bytes, a single white empty page). It is hard to explain as it is, but something even more interesting happens if you try to make them totally independent, and create a copy of the story as well: import copy doc = SimpleDocTemplate("phello.pdf") doc.build(copy.copy(Story), onFirstPage=myFirstPage, onLaterPages=myLaterPages) doc = SimpleDocTemplate("phello2.pdf") doc.build(copy.copy(Story), onFirstPage=myFirstPage, onLaterPages=myLaterPages) This will render phello.pdf correctly, and it will throw this error when rendering phello2.pdf: Traceback (most recent call last): File "C:\Projects\test\test2.py", line 48, in go() File "C:\Projects\test\test2.py", line 45, in go doc.build(copy.copy(Story), onFirstPage=myFirstPage, onLaterPages=myLaterPages) File "C:\Users\nagyl\.virtualenvs\test-NC9-O-tN\lib\site-packages\reportlab\platypus\doctemplate.py", line 1314, in build BaseDocTemplate.build(self,flowables, canvasmaker=canvasmaker) File "C:\Users\nagyl\.virtualenvs\ test-NC9-O-tN\lib\site-packages\reportlab\platypus\doctemplate.py", line 1079, in build self.handle_flowable(flowables) File "C:\Users\nagyl\.virtualenvs\ test-NC9-O-tN\lib\site-packages\reportlab\platypus\doctemplate.py", line 958, in handle_flowable raise LayoutError(ident) reportlab.platypus.doctemplate.LayoutError: Flowable This is Paragraph number 6. This is Paragraph number 6. This(439.27559055118115 x 72) too large on page 1 in frame 'normal'(439.27559055118115 x 685.8897637795277) of template 'First' And finally, here is the "solution" that solves all problems: def go(): def create_story(): Story = [Spacer(1, 2 * inch)] style = styles["Normal"] for i in range(100): bogustext = ("This is Paragraph number %s. " % i) * 20 p = Paragraph(bogustext, style) Story.append(p) Story.append(Spacer(1, 0.2 * inch)) return Story doc = SimpleDocTemplate("phello.pdf") doc.build(create_story(), onFirstPage=myFirstPage, onLaterPages=myLaterPages) doc = SimpleDocTemplate("phello2.pdf") doc.build(create_story(), onFirstPage=myFirstPage, onLaterPages=myLaterPages) This does not throw an error, and it renders both files correctly. But I do not see why this last version works, and the previous one (that uses copy.copy) does not. I was looking for an explanation in the user guide. I was looking for a note, telling me that a story can be used for document generation only once. But there is no such thing in the docs. Or maybe I just did not find it. Can somebody please explain what is happening here? Environment details: Python 3.10.1 amd64 on Windows, reportlab 3.6.6 Thanks, Laszlo From barry at barrys-emacs.org Sun Mar 13 18:29:30 2022 From: barry at barrys-emacs.org (Barry) Date: Sun, 13 Mar 2022 22:29:30 +0000 Subject: Reportlab / platypus bug? In-Reply-To: References: Message-ID: > On 13 Mar 2022, at 21:41, Les wrote: > > I have found an error, and I created a minimal working example. The minimal > working example starts with the very first example from Platypus user guide: I would suggest that you report to reportlab.com directly, any fix will come from them. Barry From nagylzs at gmail.com Sun Mar 13 18:34:07 2022 From: nagylzs at gmail.com (Les) Date: Sun, 13 Mar 2022 23:34:07 +0100 Subject: Reportlab / platypus bug? In-Reply-To: References: Message-ID: I will, thanks. I just wanted to make sure that this is a real bug, and not a mistake on my side. Barry ezt ?rta (id?pont: 2022. m?rc. 13., V 23:29): > > > > On 13 Mar 2022, at 21:41, Les wrote: > > > > I have found an error, and I created a minimal working example. The > minimal > > working example starts with the very first example from Platypus user > guide: > > I would suggest that you report to reportlab.com directly, any fix will > come from them. > > Barry > From barry at barrys-emacs.org Sun Mar 13 18:37:42 2022 From: barry at barrys-emacs.org (Barry) Date: Sun, 13 Mar 2022 22:37:42 +0000 Subject: Reportlab / platypus bug? In-Reply-To: References: Message-ID: <3FDEE8EA-C3F2-45AE-848B-6B8DF0A8D0DB@barrys-emacs.org> On 13 Mar 2022, at 22:34, Les wrote: > > ? > I will, thanks. I just wanted to make sure that this is a real bug, and not a mistake on my side. You followed the docs and got an error. Doc error? Docs out of date? Barry > > Barry ezt ?rta (id?pont: 2022. m?rc. 13., V 23:29): >> >> >> > On 13 Mar 2022, at 21:41, Les wrote: >> > >> > I have found an error, and I created a minimal working example. The minimal >> > working example starts with the very first example from Platypus user guide: >> >> I would suggest that you report to reportlab.com directly, any fix will come from them. >> >> Barry From wlfraed at ix.netcom.com Sun Mar 13 21:26:37 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Sun, 13 Mar 2022 21:26:37 -0400 Subject: Reportlab / platypus bug? References: Message-ID: On Sun, 13 Mar 2022 21:55:36 +0100, Les declaimed the following: >It is hard to explain as it is, but something even more interesting happens >if you try to make them totally independent, and create a copy of the story >as well: > >import copy >doc = SimpleDocTemplate("phello.pdf") >doc.build(copy.copy(Story), onFirstPage=myFirstPage, onLaterPages=myLaterPages) >doc = SimpleDocTemplate("phello2.pdf") >doc.build(copy.copy(Story), onFirstPage=myFirstPage, onLaterPages=myLaterPages) > Note that copy.copy() is a SHALLOW copy (ie; only the top level objects are copied, anything those objects contain are still shared). Try copy.deepcopy() https://docs.python.org/3/library/copy.html """ The difference between shallow and deep copying is only relevant for compound objects (objects that contain other objects, like lists or class instances): A shallow copy constructs a new compound object and then (to the extent possible) inserts references into it to the objects found in the original. A deep copy constructs a new compound object and then, recursively, inserts copies into it of the objects found in the original. """ My hypothesis is that rendering the "story" results in changes to the contained objects (possibly they are even "consumed" as they are rendered -- so you first attempt ends up passing an empty "story" for the second PDF). -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From nagylzs at gmail.com Mon Mar 14 03:53:45 2022 From: nagylzs at gmail.com (Les) Date: Mon, 14 Mar 2022 08:53:45 +0100 Subject: Reportlab / platypus bug? In-Reply-To: References: Message-ID: Good point. I can confirm, that it works with copy.deepcopy. Probably you are right, the story is somehow consumed by doc.build. But it is not documented anywhere. I'm going to submit a bug report, thanks. From loris.bennett at fu-berlin.de Mon Mar 14 04:07:06 2022 From: loris.bennett at fu-berlin.de (Loris Bennett) Date: Mon, 14 Mar 2022 09:07:06 +0100 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> Message-ID: <877d8xnej9.fsf@hornfels.zedat.fu-berlin.de> Marco Sulla writes: > On Fri, 11 Mar 2022 at 19:10, Michael Torrie wrote: >> Both Debian stable and Ubuntu LTS state they have a five year support >> life cycle. > > Yes, but it seems that official security support in Debian ends after > three years: > > "Debian LTS is not handled by the Debian security team, but by a > separate group of volunteers and companies interested in making it a > success" > https://wiki.debian.org/LTS > > This is the only problem for me. I am not sure how different the two situations are. Ubuntu is presumably relying on the Debian security team as well as other volunteers and at least one company, namely Canonical. The sysadmins I know who are interested in long-term stability and avoiding unnecessary OS updates use Debian rather than Ubuntu, but that's maybe just my bubble. Cheers, Loris -- This signature is currently under construction. From Joseph.Schachner at Teledyne.com Mon Mar 14 14:09:33 2022 From: Joseph.Schachner at Teledyne.com (Schachner, Joseph) Date: Mon, 14 Mar 2022 18:09:33 +0000 Subject: Reportlab / platypus bug? In-Reply-To: References: Message-ID: I realize this is Python code, but I doubt that the question is a Python question. I have used Python +numpy, scipy, matplotlib for years. I have not used reportlab and have no idea about the reported problem except that I will be very surprised if it turns out to be a Python language issue. Is there possibly a better place to ask this question? Teledyne Confidential; Commercially Sensitive Business Data -----Original Message----- From: Les Sent: Sunday, March 13, 2022 4:56 PM To: python-list at python.org Subject: Reportlab / platypus bug? Hello, I have found an error, and I created a minimal working example. The minimal working example starts with the very first example from Platypus user guide: from reportlab.platypus import SimpleDocTemplate, Paragraph, Spacer from reportlab.lib.styles import getSampleStyleSheet from reportlab.lib.pagesizes import A4 from reportlab.lib.units import inch PAGE_HEIGHT = A4[1] PAGE_WIDTH = A4[0] styles = getSampleStyleSheet() Title = "Hello world" pageinfo = "platypus example" def myFirstPage(canvas, doc): canvas.saveState() canvas.setFont('Times-Bold', 16) canvas.drawCentredString(PAGE_WIDTH / 2.0, PAGE_HEIGHT - 108, Title) canvas.setFont('Times-Roman', 9) canvas.drawString(inch, 0.75 * inch, "First Page / %s" % pageinfo) canvas.restoreState() def myLaterPages(canvas, doc): canvas.saveState() canvas.setFont('Times-Roman', 9) canvas.drawString(inch, 0.75 * inch, "Page %d %s" % (doc.page, pageinfo)) canvas.restoreState() def go(): Story = [Spacer(1, 2 * inch)] style = styles["Normal"] for i in range(100): bogustext = ("This is Paragraph number %s. " % i) * 20 p = Paragraph(bogustext, style) Story.append(p) Story.append(Spacer(1, 0.2 * inch)) doc = SimpleDocTemplate("phello.pdf") doc.build(Story, onFirstPage=myFirstPage, onLaterPages=myLaterPages) go() If I change it to this (e.g. generate two identical files): doc = SimpleDocTemplate("phello.pdf") doc.build(Story, onFirstPage=myFirstPage, onLaterPages=myLaterPages) doc = SimpleDocTemplate("phello2.pdf") doc.build(Story, onFirstPage=myFirstPage, onLaterPages=myLaterPages) then it builds phello.pdf correctly, but builds a totally empty phello2.pdf (960 bytes, a single white empty page). It is hard to explain as it is, but something even more interesting happens if you try to make them totally independent, and create a copy of the story as well: import copy doc = SimpleDocTemplate("phello.pdf") doc.build(copy.copy(Story), onFirstPage=myFirstPage, onLaterPages=myLaterPages) doc = SimpleDocTemplate("phello2.pdf") doc.build(copy.copy(Story), onFirstPage=myFirstPage, onLaterPages=myLaterPages) This will render phello.pdf correctly, and it will throw this error when rendering phello2.pdf: Traceback (most recent call last): File "C:\Projects\test\test2.py", line 48, in go() File "C:\Projects\test\test2.py", line 45, in go doc.build(copy.copy(Story), onFirstPage=myFirstPage, onLaterPages=myLaterPages) File "C:\Users\nagyl\.virtualenvs\test-NC9-O-tN\lib\site-packages\reportlab\platypus\doctemplate.py", line 1314, in build BaseDocTemplate.build(self,flowables, canvasmaker=canvasmaker) File "C:\Users\nagyl\.virtualenvs\ test-NC9-O-tN\lib\site-packages\reportlab\platypus\doctemplate.py", line 1079, in build self.handle_flowable(flowables) File "C:\Users\nagyl\.virtualenvs\ test-NC9-O-tN\lib\site-packages\reportlab\platypus\doctemplate.py", line 958, in handle_flowable raise LayoutError(ident) reportlab.platypus.doctemplate.LayoutError: Flowable This is Paragraph number 6. This is Paragraph number 6. This(439.27559055118115 x 72) too large on page 1 in frame 'normal'(439.27559055118115 x 685.8897637795277) of template 'First' And finally, here is the "solution" that solves all problems: def go(): def create_story(): Story = [Spacer(1, 2 * inch)] style = styles["Normal"] for i in range(100): bogustext = ("This is Paragraph number %s. " % i) * 20 p = Paragraph(bogustext, style) Story.append(p) Story.append(Spacer(1, 0.2 * inch)) return Story doc = SimpleDocTemplate("phello.pdf") doc.build(create_story(), onFirstPage=myFirstPage, onLaterPages=myLaterPages) doc = SimpleDocTemplate("phello2.pdf") doc.build(create_story(), onFirstPage=myFirstPage, onLaterPages=myLaterPages) This does not throw an error, and it renders both files correctly. But I do not see why this last version works, and the previous one (that uses copy.copy) does not. I was looking for an explanation in the user guide. I was looking for a note, telling me that a story can be used for document generation only once. But there is no such thing in the docs. Or maybe I just did not find it. Can somebody please explain what is happening here? Environment details: Python 3.10.1 amd64 on Windows, reportlab 3.6.6 Thanks, Laszlo From nagylzs at gmail.com Mon Mar 14 14:17:31 2022 From: nagylzs at gmail.com (Les) Date: Mon, 14 Mar 2022 19:17:31 +0100 Subject: Reportlab / platypus bug? In-Reply-To: References: Message-ID: Unfortunately, the reportlab-users mailing list is unavailable (I cannot subscribe). There is paid support but since I already have a workaround, I won't pay for this. I think this is a documentation error of the reportlab package. (They do not mention that stories cannot be reused.) I think we can say that my original problem is solved, because I have a workaround that always works. Schachner, Joseph ezt ?rta (id?pont: 2022. m?rc. 14., H 19:09): > I realize this is Python code, but I doubt that the question is a Python > question. I have used Python +numpy, scipy, matplotlib for years. I have > not used reportlab and have no idea about the reported problem except that > I will be very surprised if it turns out to be a Python language issue. > Is there possibly a better place to ask this question? > > > Teledyne Confidential; Commercially Sensitive Business Data > > -----Original Message----- > From: Les > Sent: Sunday, March 13, 2022 4:56 PM > To: python-list at python.org > Subject: Reportlab / platypus bug? > > Hello, > > I have found an error, and I created a minimal working example. The > minimal working example starts with the very first example from Platypus > user guide: > > from reportlab.platypus import SimpleDocTemplate, Paragraph, Spacer from > reportlab.lib.styles import getSampleStyleSheet from > reportlab.lib.pagesizes import A4 from reportlab.lib.units import inch > > PAGE_HEIGHT = A4[1] > PAGE_WIDTH = A4[0] > styles = getSampleStyleSheet() > > Title = "Hello world" > pageinfo = "platypus example" > > > def myFirstPage(canvas, doc): > canvas.saveState() > canvas.setFont('Times-Bold', 16) > canvas.drawCentredString(PAGE_WIDTH / 2.0, PAGE_HEIGHT - 108, Title) > canvas.setFont('Times-Roman', 9) > canvas.drawString(inch, 0.75 * inch, "First Page / %s" % pageinfo) > canvas.restoreState() > > > def myLaterPages(canvas, doc): > canvas.saveState() > canvas.setFont('Times-Roman', 9) > canvas.drawString(inch, 0.75 * inch, "Page %d %s" % (doc.page, > pageinfo)) > canvas.restoreState() > > > def go(): > Story = [Spacer(1, 2 * inch)] > style = styles["Normal"] > for i in range(100): > bogustext = ("This is Paragraph number %s. " % i) * 20 > p = Paragraph(bogustext, style) > Story.append(p) > Story.append(Spacer(1, 0.2 * inch)) > doc = SimpleDocTemplate("phello.pdf") > doc.build(Story, onFirstPage=myFirstPage, onLaterPages=myLaterPages) > > > go() > > > If I change it to this (e.g. generate two identical files): > > doc = SimpleDocTemplate("phello.pdf") > doc.build(Story, onFirstPage=myFirstPage, onLaterPages=myLaterPages) doc = > SimpleDocTemplate("phello2.pdf") doc.build(Story, onFirstPage=myFirstPage, > onLaterPages=myLaterPages) > > > then it builds phello.pdf correctly, but builds a totally empty phello2.pdf > (960 bytes, a single white empty page). > > It is hard to explain as it is, but something even more interesting > happens if you try to make them totally independent, and create a copy of > the story as well: > > import copy > doc = SimpleDocTemplate("phello.pdf") > doc.build(copy.copy(Story), onFirstPage=myFirstPage, > onLaterPages=myLaterPages) doc = SimpleDocTemplate("phello2.pdf") > doc.build(copy.copy(Story), onFirstPage=myFirstPage, > onLaterPages=myLaterPages) > > > This will render phello.pdf correctly, and it will throw this error when > rendering phello2.pdf: > > Traceback (most recent call last): > File "C:\Projects\test\test2.py", line 48, in > go() > File "C:\Projects\test\test2.py", line 45, in go > doc.build(copy.copy(Story), onFirstPage=myFirstPage, > onLaterPages=myLaterPages) > File > > "C:\Users\nagyl\.virtualenvs\test-NC9-O-tN\lib\site-packages\reportlab\platypus\doctemplate.py", > line 1314, in build > BaseDocTemplate.build(self,flowables, canvasmaker=canvasmaker) > File "C:\Users\nagyl\.virtualenvs\ > test-NC9-O-tN\lib\site-packages\reportlab\platypus\doctemplate.py", line > 1079, in build > self.handle_flowable(flowables) > File "C:\Users\nagyl\.virtualenvs\ > test-NC9-O-tN\lib\site-packages\reportlab\platypus\doctemplate.py", line > 958, in handle_flowable > raise LayoutError(ident) > reportlab.platypus.doctemplate.LayoutError: Flowable 0x148e102cb80 frame=normal>This is Paragraph number 6. This is Paragraph > number 6. This(439.27559055118115 x 72) too large on page 1 in frame > 'normal'(439.27559055118115 x 685.8897637795277) of template 'First' > > And finally, here is the "solution" that solves all problems: > > > def go(): > def create_story(): > Story = [Spacer(1, 2 * inch)] > style = styles["Normal"] > for i in range(100): > bogustext = ("This is Paragraph number %s. " % i) * 20 > p = Paragraph(bogustext, style) > Story.append(p) > Story.append(Spacer(1, 0.2 * inch)) > return Story > > doc = SimpleDocTemplate("phello.pdf") > doc.build(create_story(), onFirstPage=myFirstPage, > onLaterPages=myLaterPages) > doc = SimpleDocTemplate("phello2.pdf") > doc.build(create_story(), onFirstPage=myFirstPage, > onLaterPages=myLaterPages) > > > This does not throw an error, and it renders both files correctly. But I > do not see why this last version works, and the previous one (that uses > copy.copy) does not. > > I was looking for an explanation in the user guide. I was looking for a > note, telling me that a story can be used for document generation only > once. But there is no such thing in the docs. Or maybe I just did not find > it. Can somebody please explain what is happening here? > > Environment details: Python 3.10.1 amd64 on Windows, reportlab 3.6.6 > > Thanks, > > Laszlo > -- > https://mail.python.org/mailman/listinfo/python-list > From wlfraed at ix.netcom.com Mon Mar 14 14:59:16 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Mon, 14 Mar 2022 14:59:16 -0400 Subject: Reportlab / platypus bug? References: Message-ID: <2e3v2h1qblv12ipkqtj55rkv2v2oobgu5v@4ax.com> On Mon, 14 Mar 2022 19:17:31 +0100, Les declaimed the following: >Unfortunately, the reportlab-users mailing list is unavailable (I cannot >subscribe). There is paid support but since I already have a workaround, I >won't pay for this. I think this is a documentation error of the reportlab >package. (They do not mention that stories cannot be reused.) > https://github.com/eduardocereto/reportlab/blob/master/src/reportlab/platypus/doctemplate.py """ A document is built when a DocumentTemplate is fed a sequence of Flowables. The action of the build consumes the flowables in order and places them onto frames on pages as space allows. When a frame runs out of space the next frame of the page is used. If no frame remains a new page is created. A new page can also be created if a page break is forced. """ Well, the code does use the term "consumes" And down near the bottom (line 980 or so; this is a section that does multiple passes for special cases) """ # work with a copy of the story, since it is consumed tempStory = story[:] self.build(tempStory, **buildKwds) #self.notify('debug',None) """ -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From Marco.Sulla.Python at gmail.com Mon Mar 14 15:10:48 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Mon, 14 Mar 2022 20:10:48 +0100 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: <877d8xnej9.fsf@hornfels.zedat.fu-berlin.de> References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> <877d8xnej9.fsf@hornfels.zedat.fu-berlin.de> Message-ID: On Mon, 14 Mar 2022 at 18:33, Loris Bennett wrote: > I am not sure how different the two situations are. Ubuntu is > presumably relying on the Debian security team as well as other > volunteers and at least one company, namely Canonical. So do you think that Canonical contributes to the LTS security team of Debian? It could be. In this perspective, there should be little difference between Debian and Ubuntu. Debian 11 with XFCE is really tempting... From drsalists at gmail.com Mon Mar 14 17:26:51 2022 From: drsalists at gmail.com (Dan Stromberg) Date: Mon, 14 Mar 2022 14:26:51 -0700 Subject: Weird strace of #! python script Message-ID: Hi folks. First off, I know, python2 is ancient history. Porting to 3.x is on my list of things to do (though I'm afraid it's not yet at the top of the list), and the same thing happens with python3. So anyway, I'm strace'ing a #!/usr/bin/python2 script. I expected to see an exec of /usr/bin/python2, but I don't. I just see an exec of /tmp/t. As follows: tact at celery_worker:/app$ strace -f -s 1024 -o /tmp/t.strace /tmp/t ^Z [1]+ Stopped strace -f -s 1024 -o /tmp/t.strace /tmp/t tact at celery_worker:/app$ bg [1]+ strace -f -s 1024 -o /tmp/t.strace /tmp/t & tact at celery_worker:/app$ ps axf PID TTY STAT TIME COMMAND 1163 pts/0 Ss 0:00 bash 1363 pts/0 S 0:00 \_ strace -f -s 1024 -o /tmp/t.strace /tmp/t 1366 pts/0 S 0:00 | \_ /usr/bin/python2 /tmp/t 1367 pts/0 R+ 0:00 \_ ps axf tact at celery_worker:/app$ fg bash: fg: job has terminated [1]+ Done strace -f -s 1024 -o /tmp/t.strace /tmp/t tact at celery_worker:/app$ grep execve /tmp/t.strace 1366 execve("/tmp/t", ["/tmp/t"], 0x7ffd89f9c3b8 /* 49 vars */) = 0 tact at celery_worker:/app$ I've deleted some irrelevant processes from the 'ps axf'. /tmp/t is actually just: tact at celery_worker:/app$ cat /tmp/t #!/usr/bin/python2 import time time.sleep(10) Was this some sort of security feature I never heard about? I'm tracing a very simple time.sleep(10) here, but the same thing is (not) happening in a larger script that I need to track down a bug in. Is there a way I can coax Linux and/or strace to show all the exec's, like they used to? Not having them makes me wonder what else is missing from the strace report. I'm on a Debian 11.2 system with strace 5.10 and Python 2.7.18. Thanks! From rosuav at gmail.com Mon Mar 14 17:39:24 2022 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 15 Mar 2022 08:39:24 +1100 Subject: Weird strace of #! python script In-Reply-To: References: Message-ID: On Tue, 15 Mar 2022 at 08:28, Dan Stromberg wrote: > > Hi folks. > > First off, I know, python2 is ancient history. Porting to 3.x is on my > list of things to do (though I'm afraid it's not yet at the top of the > list), and the same thing happens with python3. > > So anyway, I'm strace'ing a #!/usr/bin/python2 script. > > I expected to see an exec of /usr/bin/python2, but I don't. I just see an > exec of /tmp/t. > > As follows: > tact at celery_worker:/app$ strace -f -s 1024 -o /tmp/t.strace /tmp/t > ^Z > [1]+ Stopped strace -f -s 1024 -o /tmp/t.strace /tmp/t > tact at celery_worker:/app$ bg > [1]+ strace -f -s 1024 -o /tmp/t.strace /tmp/t & > tact at celery_worker:/app$ ps axf > PID TTY STAT TIME COMMAND > 1163 pts/0 Ss 0:00 bash > 1363 pts/0 S 0:00 \_ strace -f -s 1024 -o /tmp/t.strace /tmp/t > 1366 pts/0 S 0:00 | \_ /usr/bin/python2 /tmp/t > 1367 pts/0 R+ 0:00 \_ ps axf > tact at celery_worker:/app$ fg > bash: fg: job has terminated > [1]+ Done strace -f -s 1024 -o /tmp/t.strace /tmp/t > tact at celery_worker:/app$ grep execve /tmp/t.strace > 1366 execve("/tmp/t", ["/tmp/t"], 0x7ffd89f9c3b8 /* 49 vars */) = 0 > tact at celery_worker:/app$ > > I've deleted some irrelevant processes from the 'ps axf'. > > /tmp/t is actually just: > tact at celery_worker:/app$ cat /tmp/t > #!/usr/bin/python2 > > import time > > time.sleep(10) > > > Was this some sort of security feature I never heard about? I'm tracing a > very simple time.sleep(10) here, but the same thing is (not) happening in a > larger script that I need to track down a bug in. > > Is there a way I can coax Linux and/or strace to show all the exec's, like > they used to? Not having them makes me wonder what else is missing from > the strace report. > > I'm on a Debian 11.2 system with strace 5.10 and Python 2.7.18. > > Thanks! I'm not sure, but I suspect that strace is skipping the exec into the process itself. When I try this myself, I can see the trace of the sleep call itself (it's implemented on top of select()), so it does seem to be doing the proper trace. My guess is that the shebang is being handled inside the exec of /tmp/t itself, rather than being a separate syscall. In any case, it doesn't seem to be a Python thing; I can trigger the same phenomenon with other interpreters. ChrisA From barry at barrys-emacs.org Mon Mar 14 18:20:47 2022 From: barry at barrys-emacs.org (Barry) Date: Mon, 14 Mar 2022 22:20:47 +0000 Subject: Weird strace of #! python script In-Reply-To: References: Message-ID: > On 14 Mar 2022, at 21:29, Dan Stromberg wrote: > > I expected to see an exec of /usr/bin/python2, but I don't. I just see an > exec of /tmp/t. I understand that the kernel handles the #! Line itself, which is why you do not see it in strace. Barry From nagylzs at gmail.com Tue Mar 15 08:11:28 2022 From: nagylzs at gmail.com (Les) Date: Tue, 15 Mar 2022 13:11:28 +0100 Subject: Reportlab / platypus bug? In-Reply-To: <2e3v2h1qblv12ipkqtj55rkv2v2oobgu5v@4ax.com> References: <2e3v2h1qblv12ipkqtj55rkv2v2oobgu5v@4ax.com> Message-ID: Dennis Lee Bieber ezt ?rta (id?pont: 2022. m?rc. 14., H 20:03): > On Mon, 14 Mar 2022 19:17:31 +0100, Les declaimed the > following: > > >Unfortunately, the reportlab-users mailing list is unavailable (I cannot > >subscribe). There is paid support but since I already have a workaround, I > >won't pay for this. I think this is a documentation error of the reportlab > >package. (They do not mention that stories cannot be reused.) > > > > > https://github.com/eduardocereto/reportlab/blob/master/src/reportlab/platypus/doctemplate.py > """ > A document is built when a DocumentTemplate is fed a sequence of Flowables. > The action of the build consumes the flowables in order and places them > onto frames on pages as space allows. When a frame runs out of space the > next frame of the page is used. If no frame remains a new page is created. > A new page can also be created if a page break is forced. > """ > > Well, the code does use the term "consumes" > > And down near the bottom (line 980 or so; this is a section that > does > multiple passes for special cases) > """ > # work with a copy of the story, since it is consumed > tempStory = story[:] > self.build(tempStory, **buildKwds) > #self.notify('debug',None) > """ > Well, the official user guide ( https://www.reportlab.com/docs/reportlab-userguide.pdf ) only uses the word "consume" in section "5.5 Frames", but it does not imply that a story cannot be reused for generating multiple documents. It could also mean that each document builder consumes each Flowable only once. So I still think that this behaviour is undocumented. It is very interesting that they use story[:] to make a swallow copy for multiple passes. It must work in that particular case (e.g. when you build the same document multiple times), but it surely does not work when you build different documents using the same story (throws a LayoutError). From robin at reportlab.com Tue Mar 15 09:06:57 2022 From: robin at reportlab.com (Robin Becker) Date: Tue, 15 Mar 2022 13:06:57 +0000 Subject: Reportlab / platypus bug? In-Reply-To: References: Message-ID: <22e928db-8330-dfdd-fbd5-403d376ae50e@everest.reportlab.co.uk> On 14/03/2022 18:17, Les wrote: > Unfortunately, the reportlab-users mailing list is unavailable (I cannot > subscribe). There is paid support but since I already have a workaround, I > won't pay for this. I think this is a documentation error of the reportlab > package. (They do not mention that stories cannot be reused.) > > I think we can say that my original problem is solved, because I have a > workaround that always works. > > Schachner, Joseph ezt ?rta (id?pont: 2022. > m?rc. 14., H 19:09): Hi Les, so far as I know the reportlab-users list is still running it is hosted (nad has been for many years) at https://pairlist2.pair.net/mailman/listinfo/reportlab-users is that the address you used? I see messages in the archives so some people can use it.-- Robin Becker From nagylzs at gmail.com Tue Mar 15 09:20:32 2022 From: nagylzs at gmail.com (Les) Date: Tue, 15 Mar 2022 14:20:32 +0100 Subject: Reportlab / platypus bug? In-Reply-To: <22e928db-8330-dfdd-fbd5-403d376ae50e@everest.reportlab.co.uk> References: <22e928db-8330-dfdd-fbd5-403d376ae50e@everest.reportlab.co.uk> Message-ID: Robin Becker ezt ?rta (id?pont: 2022. m?rc. 15., K, 14:06): > > > Hi Les, so far as I know the reportlab-users list is still running it is > hosted (nad has been for many years) at > > > https://pairlist2.pair.net/mailman/listinfo/reportlab-users > > is that the address you used? I see messages in the archives so some > people can use it.-- > Yes, it is. I tried to subscribe (twice), but never got the confirmation email. Checked in the spam folder too, but it is nowhere to be found. From robin at reportlab.com Tue Mar 15 09:21:59 2022 From: robin at reportlab.com (Robin Becker) Date: Tue, 15 Mar 2022 13:21:59 +0000 Subject: Reportlab / platypus bug? In-Reply-To: <22e928db-8330-dfdd-fbd5-403d376ae50e@everest.reportlab.co.uk> References: <22e928db-8330-dfdd-fbd5-403d376ae50e@everest.reportlab.co.uk> Message-ID: <1f6111cf-b748-1fda-8b55-a9fc8cc5ab7d@everest.reportlab.co.uk> .......... > > Hi Les, so far as I know the reportlab-users list is still running it is hosted (nad has been for many years) at > > > https://pairlist2.pair.net/mailman/listinfo/reportlab-users > > is that the address you used? I see messages in the archives so some people can use it.-- > Robin Becker as a test I subscribed under my private email address and the list responded pretty quickly; the request confirmation email ended up in spam though. I believe the list is a fairly old version of mailman, but I don't have any access to the server. -- Robin Becker From nagylzs at gmail.com Tue Mar 15 09:25:32 2022 From: nagylzs at gmail.com (Les) Date: Tue, 15 Mar 2022 14:25:32 +0100 Subject: Reportlab / platypus bug? In-Reply-To: <1f6111cf-b748-1fda-8b55-a9fc8cc5ab7d@everest.reportlab.co.uk> References: <22e928db-8330-dfdd-fbd5-403d376ae50e@everest.reportlab.co.uk> <1f6111cf-b748-1fda-8b55-a9fc8cc5ab7d@everest.reportlab.co.uk> Message-ID: > > > > > as a test I subscribed under my private email address and the list > responded pretty quickly; the request confirmation > email ended up in spam though. I believe the list is a fairly old version > of mailman, but I don't have any access to the > server. > I tried again, and now I got the confirmation e-mail (in Spam). Thank you! > > From robin at reportlab.com Tue Mar 15 09:31:03 2022 From: robin at reportlab.com (Robin Becker) Date: Tue, 15 Mar 2022 13:31:03 +0000 Subject: Reportlab / platypus bug? In-Reply-To: References: <22e928db-8330-dfdd-fbd5-403d376ae50e@everest.reportlab.co.uk> Message-ID: <1e371260-5f81-3a9c-f35b-12a2404562c2@everest.reportlab.co.uk> On 15/03/2022 13:20, Les wrote: > Robin Becker ezt ?rta (id?pont: 2022. m?rc. 15., K, > 14:06): > >> >> >> Hi Les, so far as I know the reportlab-users list is still running it is >> hosted (nad has been for many years) at >> >> >> https://pairlist2.pair.net/mailman/listinfo/reportlab-users >> >> is that the address you used? I see messages in the archives so some >> people can use it.-- >> > > Yes, it is. I tried to subscribe (twice), but never got the confirmation > email. Checked in the spam folder too, but it is nowhere to be found. Hi again, Les. I just looked at the list using the admin login and find you are already a member nagylzs at gmail.com Laszlo Nagy in fact you are number 1 in the N's. Perhaps you could just try sending a test email. -- Robin Becker From Marco.Sulla.Python at gmail.com Tue Mar 15 15:54:20 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Tue, 15 Mar 2022 20:54:20 +0100 Subject: Best practice for caching hash In-Reply-To: References: Message-ID: On Sat, 12 Mar 2022 at 22:37, <2QdxY4RzWzUUiLuE at potatochowder.com> wrote: > Once hashing an object fails, why would an application try again? I can > see an application using a hashable value in a hashable situation again > and again and again (i.e., taking advantage of the cache), but what's > the use case for *repeatedly* trying to use an unhashable value again > and again and again (i.e., taking advantage of a cached failure)? Honestly? Don't know. Maybe because the object is passed to different functions and all of them independently test the hashability? I'm clutching at straws. From cs at cskk.id.au Tue Mar 15 17:32:37 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Wed, 16 Mar 2022 08:32:37 +1100 Subject: Best practice for caching hash In-Reply-To: References: Message-ID: On 12Mar2022 21:45, Marco Sulla wrote: >I have a custom immutable object, and I added a cache for its hash >value. The problem is the object can be composed of mutable or >immutable objects, so the hash can raise TypeError. Is it sensible to compute the hash only from the immutable parts? Bearing in mind that usually you need an equality function as well and it may have the same stability issues. >In this case I currently cache the value -1. The subsequent calls to >__hash__() will check if the value is -1. If so, a TypeError is >immediately raised. This will also make these values behave badly in dicts/sets, as they all hash to the same bucket. The performance of a hash index is pretty dependent on having roughly even distribution of objects amongst the hash slots. >The problem is the first time I get an error with details, for example: > >TypeError: unhashable type: 'list' > >The subsequent times I simply raise a generic error: > >TypeError You could, you know, cache the original exception. That does keep links to the traceback and therefore all the associates stack frames, so that isn't cheap (it is peerfectly fast - just the reference, t just prevents those things from being reclaimed). >Ok, I can improve it by raising, for example, TypeError: not all >values are hashable. But do you think this is acceptable? Now I'm >thinking about it and it seems a little hacky to me. That's a policy issue. "Acceptable" depends on the imagined use cases. Im' just having a read of your intro: https://github.com/Marco-Sulla/python-frozendict/blob/35611f4cd869383678104dc94f82aa636c20eb24/README.md So your objective is that a frozendict can itself be hashable, allowing, say, sets of frozendicts? In that case I would be inclined to never raise TypeError at all. I'd compute the hash entirely from the keys of the dict and compute equality in the normal fashion: identical keys and then equal corresponding values. That removes the requirement that values be immutable and/or hashable. It that workable? Cheers, Cameron Simpson From rosuav at gmail.com Tue Mar 15 19:57:48 2022 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 16 Mar 2022 10:57:48 +1100 Subject: Best practice for caching hash In-Reply-To: References: Message-ID: On Wed, 16 Mar 2022 at 10:42, Cameron Simpson wrote: > > On 12Mar2022 21:45, Marco Sulla wrote: > >I have a custom immutable object, and I added a cache for its hash > >value. The problem is the object can be composed of mutable or > >immutable objects, so the hash can raise TypeError. > > Is it sensible to compute the hash only from the immutable parts? > Bearing in mind that usually you need an equality function as well and > it may have the same stability issues. My understanding - and I'm sure Marco will correct me if I'm wrong here - is that this behaves like a tuple: if it contains nothing but hashable objects, it is itself hashable, but if it contains anything unhashable, the entire tuple isn't hashable. (Though it's a little confusing; a frozendict has to have nothing but immutable objects, yet it permits them to be unhashable? I know of hashable mutable objects, but can't think of any unhashable immutable objects. And I'm not sure whether a hashable-mutable would be permitted in a frozendict, or whether it'd even know.) As such, any valid hash value will be stable, and "asking for a hash will raise TypeError" is also stable. > >The problem is the first time I get an error with details, for example: > > > >TypeError: unhashable type: 'list' > > > >The subsequent times I simply raise a generic error: > > > >TypeError > > You could, you know, cache the original exception. That does keep links > to the traceback and therefore all the associates stack frames, so that > isn't cheap (it is peerfectly fast - just the reference, t just prevents > those things from being reclaimed). I don't like that idea myself, for that exact reason - it'll keep arbitrary numbers of objects alive. But caching the stringified form would be more reasonable here, and have similar effect. ChrisA From greg.ewing at canterbury.ac.nz Tue Mar 15 19:16:29 2022 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 16 Mar 2022 12:16:29 +1300 Subject: Reportlab / platypus bug? In-Reply-To: References: <22e928db-8330-dfdd-fbd5-403d376ae50e@everest.reportlab.co.uk> Message-ID: On 16/03/22 2:20 am, Les wrote: > I tried to subscribe (twice), but never got the confirmation > email. Checked in the spam folder too, but it is nowhere to be found. Is there any chance your email provider has some kind of quarantine system separate from your spam folder? The University of Canterbury recently started using a Microsoft thing for all its email, and it thinks a large proportion of messages in the Python mailing lists are Very Dangerous Phishing Emails and helpfully quarantines them for me behind a clunky web interface. :-( -- Greg From cs at cskk.id.au Tue Mar 15 22:58:53 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Wed, 16 Mar 2022 13:58:53 +1100 Subject: Best practice for caching hash In-Reply-To: References: Message-ID: On 16Mar2022 10:57, Chris Angelico wrote: >> Is it sensible to compute the hash only from the immutable parts? >> Bearing in mind that usually you need an equality function as well and >> it may have the same stability issues. > >My understanding - and I'm sure Marco will correct me if I'm wrong >here - is that this behaves like a tuple: if it contains nothing but >hashable objects, it is itself hashable, but if it contains anything >unhashable, the entire tuple isn't hashable. A significant difference is that tuples have no keys, unlike a dict. A hash does not have to hash all the internal state, ony the relevant state, and not even all of that. The objective of the hash is twofold to my mind: - that "equal" objects (in the `==` sense) have the same hash, so that they hash to the same backet in dicts and can therefore be found - that hash values are widely distributed, to maximise the evenness of the object distribution in buckets For dicts to work, the former has to hold. The latter has more flexibility. A dict has keys. If the dicts are quite varied (sets of tags for example), it may be effective to just hash the keys. But if the dict keys are similar (labelled CSV-rows-as-dicts, or likewise with database rows) this will go badly because the hashes will all (or maybe mostly) collide. >As such, any valid hash value will be stable, and "asking for a hash >will raise TypeError" is also stable. I would seek to avoid a TypeError for a frozendict, but as you can see above I have not thought of a way to do that which would also have desireable hash characteristics in almost all circumstances. (I think we can accept that almost anything will have pathological cases, but the bad cases in my hash-the-keys notion are not, to my mind, rare.) >> >The problem is the first time I get an error with details, for example: >> >TypeError: unhashable type: 'list' >> >The subsequent times I simply raise a generic error: >> >TypeError >> >> You could, you know, cache the original exception. That does keep links >> to the traceback and therefore all the associates stack frames, so that >> isn't cheap (it is peerfectly fast - just the reference, t just prevents >> those things from being reclaimed). > >I don't like that idea myself, for that exact reason - it'll keep >arbitrary numbers of objects alive. I don't like it either, for that exact reason. That reason is the same reason which has Python 3 exception variables get unset as you leave an `except` clause. I'm sure it irks everyone, but the memory penalty of not doing so is high. >But caching the stringified form >would be more reasonable here, and have similar effect. Mmm, yes. Cheers, Cameron Simpson From rosuav at gmail.com Wed Mar 16 01:58:22 2022 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 16 Mar 2022 16:58:22 +1100 Subject: Best practice for caching hash In-Reply-To: References: Message-ID: On Wed, 16 Mar 2022 at 14:00, Cameron Simpson wrote: > > On 16Mar2022 10:57, Chris Angelico wrote: > >> Is it sensible to compute the hash only from the immutable parts? > >> Bearing in mind that usually you need an equality function as well and > >> it may have the same stability issues. > > > >My understanding - and I'm sure Marco will correct me if I'm wrong > >here - is that this behaves like a tuple: if it contains nothing but > >hashable objects, it is itself hashable, but if it contains anything > >unhashable, the entire tuple isn't hashable. > > A significant difference is that tuples have no keys, unlike a dict. > > A hash does not have to hash all the internal state, ony the relevant > state, and not even all of that. The objective of the hash is twofold to > my mind: > - that "equal" objects (in the `==` sense) have the same hash, so that > they hash to the same backet in dicts and can therefore be found > - that hash values are widely distributed, to maximise the evenness of > the object distribution in buckets > > For dicts to work, the former has to hold. Python only demands the first one. And Python's own integers hash to themselves, which isn't what I'd call "widely distributed", but which works well in practice. > The latter has more flexibility. A dict has keys. If the dicts are quite > varied (sets of tags for example), it may be effective to just hash the > keys. But if the dict keys are similar (labelled CSV-rows-as-dicts, or > likewise with database rows) this will go badly because the hashes will > all (or maybe mostly) collide. The general recommendation is to use all the same data for hashing as you use for equality checking. What's the point of just hashing the keys, if the values also contribute to equality? I'd just keep it simple, and hash both. ChrisA From cs at cskk.id.au Wed Mar 16 03:47:05 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Wed, 16 Mar 2022 18:47:05 +1100 Subject: Best practice for caching hash In-Reply-To: References: Message-ID: On 16Mar2022 16:58, Chris Angelico wrote: >On Wed, 16 Mar 2022 at 14:00, Cameron Simpson wrote: >> On 16Mar2022 10:57, Chris Angelico wrote: >> A significant difference is that tuples have no keys, unlike a dict. >> >> A hash does not have to hash all the internal state, ony the relevant >> state, and not even all of that. The objective of the hash is twofold to >> my mind: >> - that "equal" objects (in the `==` sense) have the same hash, so that >> they hash to the same backet in dicts and can therefore be found >> - that hash values are widely distributed, to maximise the evenness of >> the object distribution in buckets >> >> For dicts to work, the former has to hold. > >Python only demands the first one. I think we're in violent agreement here. >And Python's own integers hash to >themselves, which isn't what I'd call "widely distributed", but which >works well in practice. This may be because the "raw" hash (in this case the int value) is itself further hashed (giving more evenness) and then moduloed into the finite number of slots in the dict. >> The latter has more flexibility. A dict has keys. If the dicts are quite >> varied (sets of tags for example), it may be effective to just hash the >> keys. But if the dict keys are similar (labelled CSV-rows-as-dicts, or >> likewise with database rows) this will go badly because the hashes will >> all (or maybe mostly) collide. > >The general recommendation is to use all the same data for hashing as >you use for equality checking. What's the point of just hashing the >keys, if the values also contribute to equality? I'd just keep it >simple, and hash both. Well, hash functions want to be fast. The case where you look up a key in a dict (which needs both the hash and a comparison) is not the only situation. When a dict's bucket array is resized, the hash functions of all stored objects require recomputation, but the equality test is _not_ required. Is the hash is very cheap, that is a large benefit here. I just took a look at the CPython hashtable.c, you can see the rehashing process here: https://github.com/python/cpython/blob/7c776521418676c074a483266339d31c950f516e/Python/hashtable.c#L280 Just an aside, that's got a bunch of interesting opimitsations in it, people has put serious work into making this fast and it remains very readable! Suppose I have a nested tree of (hashable) frozen dicts of whatever flavour. An equality test requires a deep comparison all the way down including the values. You could make a much cheaper hash function which hashed just keys, yea, even only unto the upper layer or 2, and still meet the primary criterion: all equal items have the same hash value. My recommendation is not inherently to hash everything involved in the equality test. If that's cheap, then sure. But it isn't necessary and (given a suitable use domain) it may even be very detrimental to hash everything involved in equality, as in the example above. Cheers, Cameron Simpson From rosuav at gmail.com Wed Mar 16 04:09:19 2022 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 16 Mar 2022 19:09:19 +1100 Subject: Best practice for caching hash In-Reply-To: References: Message-ID: On Wed, 16 Mar 2022 at 18:48, Cameron Simpson wrote: > > On 16Mar2022 16:58, Chris Angelico wrote: > >On Wed, 16 Mar 2022 at 14:00, Cameron Simpson wrote: > >> On 16Mar2022 10:57, Chris Angelico wrote: > >> A significant difference is that tuples have no keys, unlike a dict. > >> > >> A hash does not have to hash all the internal state, ony the relevant > >> state, and not even all of that. The objective of the hash is twofold to > >> my mind: > >> - that "equal" objects (in the `==` sense) have the same hash, so that > >> they hash to the same backet in dicts and can therefore be found > >> - that hash values are widely distributed, to maximise the evenness of > >> the object distribution in buckets > >> > >> For dicts to work, the former has to hold. > > > >Python only demands the first one. > > I think we're in violent agreement here. > > >And Python's own integers hash to > >themselves, which isn't what I'd call "widely distributed", but which > >works well in practice. > > This may be because the "raw" hash (in this case the int value) is > itself further hashed (giving more evenness) and then moduloed into the > finite number of slots in the dict. Not in a CPython dict, no. (There is some collision management that uses upper bits, but the initial selection of bucket simply uses the raw hash.) See comments at the top of dictobject.c. """ This isn't necessarily bad! To the contrary, in a table of size 2**i, taking the low-order i bits as the initial table index is extremely fast, and there are no collisions at all for dicts indexed by a contiguous range of ints. So this gives better-than-random behavior in common cases, and that's very desirable. """ > >> The latter has more flexibility. A dict has keys. If the dicts are quite > >> varied (sets of tags for example), it may be effective to just hash the > >> keys. But if the dict keys are similar (labelled CSV-rows-as-dicts, or > >> likewise with database rows) this will go badly because the hashes will > >> all (or maybe mostly) collide. > > > >The general recommendation is to use all the same data for hashing as > >you use for equality checking. What's the point of just hashing the > >keys, if the values also contribute to equality? I'd just keep it > >simple, and hash both. > > Well, hash functions want to be fast. The case where you look up a key > in a dict (which needs both the hash and a comparison) is not the only > situation. When a dict's bucket array is resized, the hash functions of > all stored objects require recomputation, but the equality test is _not_ > required. Is the hash is very cheap, that is a large benefit here. Fast is less important than accurate. Hashing functions are usually *fast enough*, and since the vast majority of the work of hashing is usually going to be strings, every other data type can just recalculate its hash based on its children. Caching the hash of a string is very useful; caching the hash of a tuple, not so much; again quoting from the CPython source code: /* Tests have shown that it's not worth to cache the hash value, see https://bugs.python.org/issue9685 */ > I just took a look at the CPython hashtable.c, you can see the rehashing > process here: > > https://github.com/python/cpython/blob/7c776521418676c074a483266339d31c950f516e/Python/hashtable.c#L280 Not sure where that file is used; it's purely internal to CPython and seems to be used for tracemalloc. It's not the dictionary type - that's in Objects/dictobject.c. In any case, the rehash operation is basically for after a resize on the hashtable, and it doesn't rely on the objects caching their hashes - it retains them in the hashtable. > Just an aside, that's got a bunch of interesting opimitsations in it, > people has put serious work into making this fast and it remains very > readable! > > Suppose I have a nested tree of (hashable) frozen dicts of whatever > flavour. An equality test requires a deep comparison all the way down > including the values. You could make a much cheaper hash function which > hashed just keys, yea, even only unto the upper layer or 2, and still > meet the primary criterion: all equal items have the same hash value. Yes, but "def __hash__(self): return 42" also meets the primary criterion. I don't know what use-cases frozendicts have, but I would suspect that if they are used at all, they'll often be used in cases where their keys are identical (for instance, the __dict__ of an immutable object type, where the keys are extremely stable across objects of the same type). > My recommendation is not inherently to hash everything involved in the > equality test. If that's cheap, then sure. But it isn't necessary and > (given a suitable use domain) it may even be very detrimental to hash > everything involved in equality, as in the example above. > My recommendation is to hash everything involved in the equality test, unless there's demonstrable benefit from not doing so. ChrisA From cs at cskk.id.au Wed Mar 16 08:04:55 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Wed, 16 Mar 2022 23:04:55 +1100 Subject: Best practice for caching hash In-Reply-To: References: Message-ID: On 16Mar2022 19:09, Chris Angelico wrote: >On Wed, 16 Mar 2022 at 18:48, Cameron Simpson wrote: >> This may be because the "raw" hash (in this case the int value) is >> itself further hashed (giving more evenness) and then moduloed into the >> finite number of slots in the dict. > >Not in a CPython dict, no. (There is some collision management that >uses upper bits, but the initial selection of bucket simply uses the >raw hash.) See comments at the top of dictobject.c. Thank you, I stand corrected. That's a very interesting read. Cheers, Cameron Simpson From Marco.Sulla.Python at gmail.com Wed Mar 16 09:00:13 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Wed, 16 Mar 2022 14:00:13 +0100 Subject: Best practice for caching hash In-Reply-To: References: Message-ID: On Wed, 16 Mar 2022 at 00:42, Cameron Simpson wrote: > > Is it sensible to compute the hash only from the immutable parts? > Bearing in mind that usually you need an equality function as well and > it may have the same stability issues. [...] > In that case I would be inclined to never raise TypeError at all. I'd > compute the hash entirely from the keys of the dict and compute equality > in the normal fashion: identical keys and then equal corresponding > values. That removes the requirement that values be immutable and/or > hashable. Well, I followed PEP 416, so I allowed immutable types for values, as tuple does. Also tuple is hashable only if all its values are hashable. The equality function is the same as dict, with a little modification. I do not check for hash in equality. I could add that, if both the hash are calculated and different from -1 and they differ, False is returned. > >In this case I currently cache the value -1. The subsequent calls to > >__hash__() will check if the value is -1. If so, a TypeError is > >immediately raised. > > This will also make these values behave badly in dicts/sets, as they all > hash to the same bucket. Not sure to understand. If the hash is -1, it's not hashable, so it can't be a member of a dict or set. > You could, you know, cache the original exception. I thought about it :) What prevented me is that is another PySsize_t to store in memory From nagylzs at gmail.com Wed Mar 16 10:44:16 2022 From: nagylzs at gmail.com (Les) Date: Wed, 16 Mar 2022 15:44:16 +0100 Subject: Reportlab / platypus bug? In-Reply-To: References: <22e928db-8330-dfdd-fbd5-403d376ae50e@everest.reportlab.co.uk> Message-ID: Greg Ewing ezt ?rta (id?pont: 2022. m?rc. 16., Sze, 1:01): > On 16/03/22 2:20 am, Les wrote: > > I tried to subscribe (twice), but never got the confirmation > > email. Checked in the spam folder too, but it is nowhere to be found. > > Is there any chance your email provider has some kind of quarantine > system separate from your spam folder? > Probably not, I was trying to register with gmail. It is interesting, I did not get the confirmation email (tried twice). Next day, I got it instantly. > > From lukasz at langa.pl Wed Mar 16 12:11:58 2022 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Wed, 16 Mar 2022 17:11:58 +0100 Subject: [RELEASE] Python 3.10.3, 3.9.11, 3.8.13, and 3.7.13 are now available with security content Message-ID: <32B640CA-F395-4933-ACE7-349DF28BE1EE@langa.pl> Welcome again to the exciting world of releasing new Python versions! Last time around I was complaining about cursed releases . This time around I could complain about security content galore and how one of them ruined my ingenious idea to release Python on Pi Day and call it Py Day . Well, you can?t have everything in life. Or at least not everything at once. And yet it seems this time around we?ve got a lot of security fixes all at once. Just look at this list: 15 (sic!) CVEs: libexpat upgraded from 2.4.1 to 2.4.7 (BPO-46794 , BPO-46932 , BPO-46811 , BPO-46784 , BPO-46400 ) CVE-2022-0778: OpenSSL upgraded from 1.1.1l to 1.1.1n in macOS and Windows installers (BPO-47024 ) CVE-2016-3189, CVE-2019-12900: bzip2 upgraded from 1.0.6 to 1.0.8 in Windows installers (BPO-44549 ) CVE-2022-26488: Windows installer now ensures the correct path is being repaired when ?Add to PATH? is used (BPO-46948 ) CVE-2021-28363: bundled pip upgraded from 21.2.4 to 22.0.4 (BPO-46985 ) authorization bypass fixed in urllib.request (BPO-46756 ) REDoS avoided in importlib.metadata (BPO-46474 ) SQLite upgraded from 3.36.0 to 3.37.2 in macOS and Windows installers (BPO-45925 ) Python 3.10.3 Get it here: https://www.python.org/downloads/release/python-3103/ Python 3.10.3 is the third maintenance release of the newest version of the Python programming language, which contains many new features and optimizations. We recommend it over the other releases listed below. This is a large bugfix release with 220 commits since 3.10.2. Just look at the change log ! The next maintenance release of Python 3.10 is planned for early June. Python 3.9.11 Get it here: https://www.python.org/downloads/release/python-3911/ This is the penultimate planned full bugfix release of Python 3.9. In mid-May, we?ll be releasing the last one, after which the 3.9 series will enter its security-only fixes period. More details in PEP 596 . Python 3.9 is the first Python version since 2.7 to have a regular bugfix release larger than ?.10?. It?s also still a significant release at 163 commits since 3.9.10. That?s in fact 30+ commits more than between 3.9.9 and 3.9.10. The change log isn?t as long as the 3.10.3 one but it?s still pretty extensive! As a reminder, on macOS, the default installer is now the new universal2 variant. It?s compatible with Mac OS X 10.9 and newer, including macOS 11 Big Sur and macOS 12 Monterey. Python installed with this variant will work natively on Apple Silicon processors. Python 3.8.13 Get it here: https://www.python.org/downloads/release/python-3813/ Changes here are almost exclusively security-only as the life cycle of Python versions prescribes. You might have noticed there is a small number of regular bug fixes nonetheless. This is because without those we wouldn?t be able to continue running the full test suite for the 3.8 branch. This in turn could hide regressions in future security fixes. Python 3.7.13 Get it here: https://www.python.org/downloads/release/python-3713/ Just like 3.8, Python 3.7 is in its security-only fixes period. In turn, the changes in 3.7.13 look almost identical to the ones in 3.8.13. Python 3.7 will continue to receive source-only releases until June 2023. We hope you enjoy the new releases Your friendly release team, ?ukasz Langa @ambv Pablo Galindo Salgado @pablogsal Ned Deily @nad Steve Dower @steve.dower -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From Marco.Sulla.Python at gmail.com Wed Mar 16 14:21:07 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Wed, 16 Mar 2022 19:21:07 +0100 Subject: Best practice for caching hash In-Reply-To: References: Message-ID: On Wed, 16 Mar 2022 at 00:59, Chris Angelico wrote: > > (Though it's a little confusing; a frozendict has to have nothing but > immutable objects, yet it permits them to be unhashable? It can have mutable objects. For example, a key k can have a list v as value. You can modify v, but you can't assign to the key k another value w. It's the same with the tuples, as you said. An index i can contain a list l. Since it's a tuple, you can't set another object at the index i, but you can modify the list l. From greg.ewing at canterbury.ac.nz Wed Mar 16 02:36:16 2022 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 16 Mar 2022 19:36:16 +1300 Subject: Best practice for caching hash In-Reply-To: References: Message-ID: On 16/03/22 6:58 pm, Chris Angelico wrote: > And Python's own integers hash to themselves, > which isn't what I'd call "widely distributed", but which > works well in practice. Not exactly, they seem to be limited to 60 bits: >>> hex(hash(0xfffffffffffffff)) '0xfffffffffffffff' >>> hex(hash(0x1fffffffffffffff)) '0x0' And up to that limit they're as widely distributed as you can get -- each integer hashes to a unique value! But keep in mind that the hash value itself is not directly used to locate a dict slot -- there is extra scrambling that goes on in the dict code. -- Greg From esxi1979 at gmail.com Tue Mar 15 21:09:47 2022 From: esxi1979 at gmail.com (Kiran Kumar) Date: Tue, 15 Mar 2022 18:09:47 -0700 (PDT) Subject: basic question on loop & type casting / list/str/array Message-ID: <905c5b05-b97d-40cf-abec-d008d65e161en@googlegroups.com> Hi. Pls check on below poython 3.9.x code & suggest how can i keep the string intactst in 2nd loop... ? these are aws ec2 ids >>> INSTANCE_ID = ['i-0dccf1ede229ce1','i-0285506fee62051'] >>> for i in INSTANCE_ID: ... print (i) ... i-0dccf1ede229ce1 i-0285506fee62051 >>> >>> >>> >>> for i in INSTANCE_ID: ... for j in i: ... print (j) ... i - 0 d c c f 1 e d e 2 2 9 c e 1 i - 0 2 8 5 5 0 6 f e e 6 2 0 5 1 >>> Thanks Kiran From python at example.invalid Tue Mar 15 22:36:17 2022 From: python at example.invalid (Python) Date: Wed, 16 Mar 2022 03:36:17 +0100 Subject: basic question on loop & type casting / list/str/array References: <905c5b05-b97d-40cf-abec-d008d65e161en@googlegroups.com> Message-ID: Kiran Kumar wrote: > Hi. > > Pls check on below poython 3.9.x code & suggest how can i keep the string intactst in 2nd loop... ? these are aws ec2 ids Don't loop through it then. From Marco.Sulla.Python at gmail.com Wed Mar 16 15:02:27 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Wed, 16 Mar 2022 20:02:27 +0100 Subject: Best practice for caching hash In-Reply-To: References: Message-ID: On Wed, 16 Mar 2022 at 09:11, Chris Angelico wrote: > Caching the hash of a > string is very useful; caching the hash of a tuple, not so much; again > quoting from the CPython source code: > > /* Tests have shown that it's not worth to cache the hash value, see > https://bugs.python.org/issue9685 */ This is really interesting. Unluckily I can't use the pyperformance benchmarks. I should use the code that uses frozendict, but I suppose it's really hard... Anyway this discourages me to continue to store unashable value, since I should also store the error message. Storing only the hash when the object is hashable is much cheaper, and maybe the extra field is not so much a problem, since dict consumes more space than a tuple: >>> sys.getsizeof({}) 64 >>> sys.getsizeof(()) 40 >>> sys.getsizeof({1:1}) 232 >>> sys.getsizeof((1,)) 48 > I don't know what use-cases frozendicts have, but I would > suspect that if they are used at all, they'll often be used in cases > where their keys are identical (for instance, the __dict__ of an > immutable object type, where the keys are extremely stable across > objects of the same type). Well, I tried to implement them as dicts with shared keys, but I abandoned it when Inada optimized dict(another_dict), where another_dict is a compact dict. Since it's compact, you have "only" to memcopy the entries (oversimplification). I tried to do the same trick for the sparse dict structure, but memcopy the keys and the values was not enough. I had to incref all value *two* times and this slowed down the creation a lot. So I decided to move to compact structure. From cs at cskk.id.au Wed Mar 16 18:32:50 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Thu, 17 Mar 2022 09:32:50 +1100 Subject: Best practice for caching hash In-Reply-To: References: Message-ID: On 16Mar2022 14:00, Marco Sulla wrote: >On Wed, 16 Mar 2022 at 00:42, Cameron Simpson wrote: >> >In this case I currently cache the value -1. The subsequent calls to >> >__hash__() will check if the value is -1. If so, a TypeError is >> >immediately raised. >> >> This will also make these values behave badly in dicts/sets, as they all >> hash to the same bucket. > >Not sure to understand. If the hash is -1, it's not hashable, so it >can't be a member of a dict or set. Sorry, I misread you and took -1 to be a valid hash code, not a signal to raise TypeError. Cheers, Cameron Simpson From soyeomul at doraji.xyz Thu Mar 17 02:21:43 2022 From: soyeomul at doraji.xyz (=?utf-8?B?7Zmp67OR7Z2s?=) Date: Thu, 17 Mar 2022 15:21:43 +0900 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: <877d8xnej9.fsf@hornfels.zedat.fu-berlin.de> (Loris Bennett's message of "Mon, 14 Mar 2022 09:07:06 +0100") References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> <877d8xnej9.fsf@hornfels.zedat.fu-berlin.de> Message-ID: <87zglpayko.fsf@penguin> Dear Loris, "Loris Bennett" writes: > (...thanks...) > The sysadmins I know who are interested in long-term stability and > avoiding unnecessary OS updates use Debian rather than Ubuntu, +1; Reasonable! Sincerely, Linux fan Byung-Hee -- ^????? _????_ ?????_^))// From rathmann.os at gmail.com Thu Mar 17 23:14:49 2022 From: rathmann.os at gmail.com (Rathmann) Date: Thu, 17 Mar 2022 20:14:49 -0700 (PDT) Subject: Reducing "yield from" overhead in recursive generators Message-ID: <5ba8e557-1545-4547-b15a-9a45547c49d5n@googlegroups.com> Summary: I am looking for feedback on a potential technique for coding recursive generators. Often times the most convenient algorithms for creating a sequence of results are recursive, while the most convenient interface for the consumption of those results is an iterator. PEP 380 introduced the "yield from" syntax which provides a technique for generators to delegate to other generators, including recursively. The trouble is that recursion and generators don't necessarily play all that well together from a performance point of view. As Greg Ewing noted in PEP 380: The overhead of passing __next__() calls and yielded values down and up the chain can cause what ought to be an O(n) operation to become, in the worst case, O(n**2). Ewing goes on to describe some possible optimizations possible in Python implementations. However, in the Python versions I have checked, there is still some O(depth) overhead to a chain of "yield from" calls. To make this concrete, let's consider a problem with a canonical recursive implementation -- an inorder tree traversal. (My actual motivation for thinking about this issue comes from implementing generators for various kinds of combinatorial objects. But tree traversal provides a good familiar example.) For discussion purposes, let's represent a tree node as a tuple of the form (key left right). Then, a simple tree might be ```Python tree1 = ('b', ('a', None, None), ('c', None, None)) ``` Or, a badly unbalanced example could be: ```python tree2 = ('a', None, ('b', None, ('c', None, ('d', None, ('e', None, ('f', None, ('g', None, ('h', None, ('i', None, None))))))))) ``` Then, we can code inorder traversal as: ```python def inorder_traverse(node): k, l, r = node if l: yield from inorder_traverse(l) yield k if r: yield from inorder_traverse(r) ``` So far, all very familiar. The trouble is that if you are at a depth of d, every value that is yielded by the algorithm gets passed through d stack frames before it reaches the actual consumer of the data. This issue, specifically the case of tree traversal, was the subject of a 2016 thread in comp.lang.python "Linear Time Tree Traversal Generator". I am not reviving that 5+-year-old thread since my objectives are a little different. In that thread, Ned Batchelder suggested (and gave a nice implementation for) reimplementing the traversal to use an explicit stack and a loop to traverse the tree, so that "each yield directly produces a value to the caller." That, to my mind, basically solved the original poster's problem -- the code for the iterative version is reasonably concise and understandable, and is faster than the recursive version both in the O(n) sense and also for small problems. But, when the candidate recursive algorithm is more complicated than that for tree traversal, translating to an iterative version can be difficult. Maintaining a stack of local variables is not too hard, but converting a recursive function to iterative completely changes the flow, such that familiar control structures become a tangled mass of GOTOs. (Which is even harder in a language without a GOTO.) The result (at least when I have tried) can be hard to write and (worse) hard to read. I was looking at the problem and realized that there might be a middle ground -- avoiding the O(depth) cost of a chain of "yield from" clauses, but still keeping the recursive code looking reasonably close to the original. The idea is to modify the recursive function by replacing the "yield from" forms with "yield". The transformed function needs to run from a driver, which simulates the normal call stack. The driver checks each result yielded from the function -- if it is a regular value, it is returned (yielded) to the caller immediately, but if the result is a generator, it is pushed onto that stack. The Python code for such a driver is easier to understand than any english description I have been able to create, so here it is: ```python def recursive_generator_driver(g): """g is a generator object, which is likely to call other generators""" stack = [g] while True: g = stack[-1] try: val = next(g) if isinstance(val, types.GeneratorType): # Yielded value represented a recursive call, so push stack.append(val) else: # Regular value for iterator, give to caller yield val except StopIteration: stack.pop() if len(stack) == 0: return ``` and the modified tree traverser is just ```python def inorder_traverse_mod(node): """ONLY for use with recursive_generator_driver""" k, l, r = node if l: yield inorder_traverse_mod(l) # NOT yield from yield k if r: yield inorder_traverse_mod(r) # NOT yield from ``` This seems to work correctly (at least in my testing). But does saving the "yield from" cost overcome the cost of the driver function with its explicit stack and isinstance() calls? The tradeoffs depend on the specific Python implementation and the algorithm that is to be modified. For the tree traversal example, with PyPy (Ubuntu, 7.3.1) the driver version becomes faster with a complete binary tree of height 12 (8191 nodes) while for CPython 3.8.10 it is not until a height of 22 that the driver version dominates. A perfectly balanced tree is the a best case for the "yield from" code. With a sufficiently badly balanced tree, the driver version can be many times faster. So what do people think of this hack/technique? Is it A) A terrible idea? (Please be specific.) B) Already well-known (and I just missed it.) C) Potentially useful for the niche case of deeply recursive generators? Some additional notes (and thanks if you are still reading this long posting): 1) The same driver can be used with different recursive functions, so this is easy to try out. 2) If a function can be converted to be purely iterative, (as with Batchelder's tree traversal) it will often be faster than either the driver or yield from versions. Hence the driver version would be used for those functions where conversion to an iterative version would be more difficult. 3) If you don't like the isinstance() calls, the type can be signaled some other way. For example the yield statements can be written to pass back a pair, the first element of which is a flag indicating whether the second element is a result value or a generator. 4) This approach means that recursion depth is not restricted by the size of the Python stack. That might be a benefit, but if not, the driver could be modified to enforce a maximum recursion depth. From antoon.pardon at vub.be Fri Mar 18 05:07:19 2022 From: antoon.pardon at vub.be (Antoon Pardon) Date: Fri, 18 Mar 2022 10:07:19 +0100 Subject: Reducing "yield from" overhead in recursive generators In-Reply-To: <5ba8e557-1545-4547-b15a-9a45547c49d5n@googlegroups.com> References: <5ba8e557-1545-4547-b15a-9a45547c49d5n@googlegroups.com> Message-ID: <47fdf8ed-be7c-ae31-e8ba-9628414eea74@vub.be> Op 18/03/2022 om 04:14 schreef Rathmann: > ... > > ```python > def recursive_generator_driver(g): > """g is a generator object, which is likely to call other generators""" > stack = [g] > while True: > g = stack[-1] > try: > val = next(g) > if isinstance(val, types.GeneratorType): > # Yielded value represented a recursive call, so push > stack.append(val) > else: > # Regular value for iterator, give to caller > yield val > except StopIteration: > stack.pop() > if len(stack) == 0: > return > ``` > > and the modified tree traverser is just > > ```python > def inorder_traverse_mod(node): > """ONLY for use with recursive_generator_driver""" > k, l, r = node > if l: > yield inorder_traverse_mod(l) # NOT yield from > yield k > if r: > yield inorder_traverse_mod(r) # NOT yield from > ``` > > ... > > So what do people think of this hack/technique? Is it > > A) A terrible idea? (Please be specific.) > B) Already well-known (and I just missed it.) > C) Potentially useful for the niche case of deeply recursive > generators? I would go with B' + C. It seems there is a resemblance with the trampoline technique in order to eliminate tail recursion. I know the people of macropy have written a decorating macro that would eliminate tail recursion from such decorated functions. Maybe if you contact them they can be interested in making a similar decorating macro for use with such recursive decorators. -- Antoon Pardon. From rathmann.os at gmail.com Fri Mar 18 16:40:59 2022 From: rathmann.os at gmail.com (Rathmann) Date: Fri, 18 Mar 2022 13:40:59 -0700 (PDT) Subject: Reducing "yield from" overhead in recursive generators In-Reply-To: References: <47fdf8ed-be7c-ae31-e8ba-9628414eea74@vub.be> <5ba8e557-1545-4547-b15a-9a45547c49d5n@googlegroups.com> Message-ID: On Friday, March 18, 2022 at 2:07:45 AM UTC-7, Antoon Pardon wrote: > Op 18/03/2022 om 04:14 schreef Rathmann: > > > > So what do people think of this hack/technique? Is it > > > > A) A terrible idea? (Please be specific.) > > B) Already well-known (and I just missed it.) > > C) Potentially useful for the niche case of deeply recursive > > generators? > I would go with B' + C. It seems there is a resemblance with the trampoline technique > in order to eliminate tail recursion. I know the people of macropy have written a > decorating macro that would eliminate tail recursion from such decorated functions. > Maybe if you contact them they can be interested in making a similar decorating macro > for use with such recursive decorators. > > -- > Antoon Pardon. Thank you Antoon. I was not aware of macropy. It certainly seems like a macro could automate the process of switching the "yield from" to yield forms. That is already pretty easy to do by hand, though. The driver itself could be packaged as a decorator using just the facilities available in regular Python implementations. **Much** more ambitious would be a macro to automate the transformation of a generator from (general) recursive to iterative form, so as to avoid the need for this technique entirely. The other challenge/question would be to see if there is a way to implement this basic approach with lower overhead. So far, the savings don't kick in until the recursion hits a depth of 12 or so (even more with CPython), and many use cases have an expected recursion depth shallower than that. From ankit at applied.co Fri Mar 18 20:08:48 2022 From: ankit at applied.co (Ankit Agarwal) Date: Fri, 18 Mar 2022 17:08:48 -0700 Subject: Compiling and Linking pre-built Windows Python libraries with C++ files on Linux for Windows Message-ID: Hi, This is a very specific question. I am trying to figure out whether or not I can use pre-built python libraries and headers on Windows in a MinGW build on Linux. Essentially I have some python and C++ code which interface via cython and pybind. I want to build a self contained C++ binary for Windows with the MinGW compiler that runs on Linux. For both Cython and PyBind, they need to compile with the python headers and link against the python DLL in order to run on Windows. I know that the python DLL specifically are compiled with the MSVC compiler, however since it is in C, the ABI between the DLL should be compatible with MinGW, and I should be able to import and link against it. My question is will this work, or will there be some other problem that I might run into. I am using python 3.7 by the way for this. Let me know what you think, Thanks :) Ankit *Ankit Agarwal* Software Engineer ankit at applied.co +1 630.506.4914 Applied Intuition, Inc. blog.applied.co From greg.ewing at canterbury.ac.nz Fri Mar 18 21:45:29 2022 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 19 Mar 2022 14:45:29 +1300 Subject: Reducing "yield from" overhead in recursive generators In-Reply-To: References: <47fdf8ed-be7c-ae31-e8ba-9628414eea74@vub.be> <5ba8e557-1545-4547-b15a-9a45547c49d5n@googlegroups.com> Message-ID: On 19/03/22 9:40 am, Rathmann wrote: > The other challenge/question would be to see if there is a way to implement > this basic approach with lower overhead. My original implementation of yield-from didn't have this problem, because it didn't enter any of the intermediate frames -- it just ran down a chain of C pointers to find the currently active one. At some point this was changed, I think so that all the frames would show up in the traceback in the event of an exception. This was evidently seen as more important than having efficient resumption of nested generators. I'm not sure exactly how it works now, but I believe it involves re-executing the YIELD_FROM bytecode in each generator down the chain whenever a resumption occurs. This is nice and simple, but not very efficient. Maybe another way could be found of preserving the traceback, such as temporarily splicing the chain of frames into the call stack and then resuming the last one. -- Greg From rosuav at gmail.com Sat Mar 19 00:04:24 2022 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 19 Mar 2022 15:04:24 +1100 Subject: Reducing "yield from" overhead in recursive generators In-Reply-To: References: <47fdf8ed-be7c-ae31-e8ba-9628414eea74@vub.be> <5ba8e557-1545-4547-b15a-9a45547c49d5n@googlegroups.com> Message-ID: On Sat, 19 Mar 2022 at 14:06, Greg Ewing wrote: > > On 19/03/22 9:40 am, Rathmann wrote: > > The other challenge/question would be to see if there is a way to implement > > this basic approach with lower overhead. > > My original implementation of yield-from didn't have this problem, > because it didn't enter any of the intermediate frames -- it just > ran down a chain of C pointers to find the currently active one. > > At some point this was changed, I think so that all the frames > would show up in the traceback in the event of an exception. > This was evidently seen as more important than having efficient > resumption of nested generators. > > I'm not sure exactly how it works now, but I believe it involves > re-executing the YIELD_FROM bytecode in each generator down the > chain whenever a resumption occurs. This is nice and simple, but > not very efficient. > > Maybe another way could be found of preserving the traceback, > such as temporarily splicing the chain of frames into the call > stack and then resuming the last one. > This is a fundamentally difficult problem with async/await too, so if there's any good solution, it would be worth carrying across. ChrisA From drsalists at gmail.com Sat Mar 19 12:42:29 2022 From: drsalists at gmail.com (Dan Stromberg) Date: Sat, 19 Mar 2022 09:42:29 -0700 Subject: Compiling and Linking pre-built Windows Python libraries with C++ files on Linux for Windows In-Reply-To: References: Message-ID: On Fri, Mar 18, 2022 at 8:03 PM Ankit Agarwal wrote: > Hi, > > This is a very specific question. I am trying to figure out whether or not > I can use pre-built python libraries and headers on Windows in a MinGW > build on Linux. Essentially I have some python and C++ code which interface > via cython and pybind. I want to build a self contained C++ binary for > Windows with the MinGW compiler that runs on Linux. For both Cython and > PyBind, they need to compile with the python headers and link against the > python DLL in order to run on Windows. > > I know that the python DLL specifically are compiled with the MSVC > compiler, however since it is in C, the ABI between the DLL should be > compatible with MinGW, and I should be able to import and link against it. > My question is will this work, or will there be some other problem that I > might run into. > > I haven't tried this. However, I used to cross-compile the Linux kernel from Solaris on Sparc to Intel. I just had to: 1) Get the relevant headers and libraries on Solaris 2) Deal with the byte sex issues - Sparc is Big Endian, Intel is Little Endian 3) Natively compile a little bit of code that was needed by the build process You appear to be aware of #1. You probably won't need to worry about #2, since you're going Intel -> Intel. #3 could be an issue for you, but it's just a matter of using two different compilers for some different parts of the build process - one native, one cross. I'd try a little hello world first, then worry about your larger project. You could also put some feelers out about Cython and Pybind too, to see if they've been used for cross-compilation before. If yes, you're probably in like Flynn, otherwise it could potentially turn out to be a big project. If cross-compilation doesn't work out, you could probably set up a Windows virtual machine with an sshd, and build on that. Either way, you may find Wine useful for testing. HTH. From eryksun at gmail.com Sat Mar 19 13:09:46 2022 From: eryksun at gmail.com (Eryk Sun) Date: Sat, 19 Mar 2022 12:09:46 -0500 Subject: Compiling and Linking pre-built Windows Python libraries with C++ files on Linux for Windows In-Reply-To: References: Message-ID: On 3/18/22, Ankit Agarwal wrote: > Hi, > > This is a very specific question. I am trying to figure out whether or not > I can use pre-built python libraries and headers on Windows in a MinGW > build on Linux. Essentially I have some python and C++ code which interface > via cython and pybind. I want to build a self contained C++ binary for > Windows with the MinGW compiler that runs on Linux. For both Cython and > PyBind, they need to compile with the python headers and link against the > python DLL in order to run on Windows. > > I know that the python DLL specifically are compiled with the MSVC > compiler, however since it is in C, the ABI between the DLL should be > compatible with MinGW, and I should be able to import and link against it. > My question is will this work, or will there be some other problem that I > might run into. MinGW used to link with msvcrt (the private CRT for system components) instead of ucrt (the universal CRT). If it still does that, then you won't be able to share some of the POSIX compatibility features between the two CRTs, such as file descriptors and the locale. Their FILE stream records are also incompatible. Also, you'll have to be certain to never free() memory that was allocated by a different CRT. msvcrt uses a private heap, and ucrt uses the main process heap. For example (with a debugger attached): import ctypes ucrt = ctypes.CDLL('ucrtbase', use_errno=True) msvcrt = ctypes.CDLL('msvcrt', use_errno=True) ucrt.malloc.restype = ctypes.c_void_p msvcrt.free.argtypes = (ctypes.c_void_p,) >>> b = ucrt.malloc(4096) >>> msvcrt.free(b) HEAP[python.exe]: Invalid address specified to RtlFreeHeap( 0000024389C10000, 0000024389A58FB0 ) (1b44.1ca0): Break instruction exception - code 80000003 (first chance) ntdll!RtlpBreakPointHeap+0x16: 00007ff8`8baa511e cc int 3 0:000> From barry at barrys-emacs.org Sat Mar 19 13:31:22 2022 From: barry at barrys-emacs.org (Barry) Date: Sat, 19 Mar 2022 17:31:22 +0000 Subject: Reducing "yield from" overhead in recursive generators In-Reply-To: References: Message-ID: > On 19 Mar 2022, at 03:07, Greg Ewing wrote: > > ?On 19/03/22 9:40 am, Rathmann wrote: >> The other challenge/question would be to see if there is a way to implement >> this basic approach with lower overhead. > > My original implementation of yield-from didn't have this problem, > because it didn't enter any of the intermediate frames -- it just > ran down a chain of C pointers to find the currently active one. > > At some point this was changed, I think so that all the frames > would show up in the traceback in the event of an exception. > This was evidently seen as more important than having efficient > resumption of nested generators. I would have thought that it would be possible to do both sets of pointer/accounting. One that supports your original fast implementation and add to that the info to do the trace back. Barry > > I'm not sure exactly how it works now, but I believe it involves > re-executing the YIELD_FROM bytecode in each generator down the > chain whenever a resumption occurs. This is nice and simple, but > not very efficient. > > Maybe another way could be found of preserving the traceback, > such as temporarily splicing the chain of frames into the call > stack and then resuming the last one. > > -- > Greg > -- > https://mail.python.org/mailman/listinfo/python-list > From auriocus at gmx.de Sat Mar 19 02:04:33 2022 From: auriocus at gmx.de (Christian Gollwitzer) Date: Sat, 19 Mar 2022 07:04:33 +0100 Subject: Compiling and Linking pre-built Windows Python libraries with C++ files on Linux for Windows In-Reply-To: References: Message-ID: Am 19.03.22 um 01:08 schrieb Ankit Agarwal: > This is a very specific question. I am trying to figure out whether or not > I can use pre-built python libraries and headers on Windows in a MinGW > build on Linux. With the mingw cross-compiler on Linux that should be possible, however I guess it might be difficult to set it up. Setuptools is not good for crosscompiling. The easiest way to build Python extensions across multiple OSes is cibuildwheel: https://github.com/pypa/cibuildwheel Especially if your code lives on Github, then you can simply add cibuildwheel to your workflow and it will build for multiple OSes and Python versions on every commit. For OpenSource projects, Githubs build servers are even for free! Otherwise, the prices are moderate as well. If you happen to have your own build server farm (as in a commercial setting) then you can also use other CI tools with cibuildwheel. Christian From kevinmwilson1956 at yahoo.com Mon Mar 21 11:36:50 2022 From: kevinmwilson1956 at yahoo.com (Kevin M. Wilson) Date: Mon, 21 Mar 2022 15:36:50 +0000 (UTC) Subject: Pycharm IDE: seeking an assist! References: <844768148.1376524.1647877010498.ref@mail.yahoo.com> Message-ID: <844768148.1376524.1647877010498@mail.yahoo.com> Greetings Python coders, ????????? ? I have installed the Pycharm IDE, and upon successfully auto install of the path/environment statements. The IDE opened and displayed (bottom right corner):? The use of Java options environment variables detected. Such variables override IDE configuration files (*.vmoptions) and may cause performance and stability issues. Please consider deleting these variables: _JAVA_OPTIONS. Now I've opened the installed .bat files...append.bat, format.bat, inspect.bat, itedit.bat, and pycharm.bat! Of the Five(5) listed above, only 'pycharm.bat' contains statements setting up the IDE' run environment, beyond Seven (7) lines. Having searched the 'pycharm.bat' file for "_JAVA_OPTIONS", and the 'pycharm64.exe.vmoptions' file as well. I was able to add the line '-Djava.net.preferIPv4Stack=False', reboot, and still no JOY.? Message is still popping open, when the IDE first executes. No documentation have I found, details what this option, the setting of...will do! Any and all help, please! Kevin Good sense makes one slow to anger, and it is his glory to overlook an offense. Proverbs 19:11 From wlfraed at ix.netcom.com Mon Mar 21 12:09:17 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Mon, 21 Mar 2022 12:09:17 -0400 Subject: Pycharm IDE: seeking an assist! References: <844768148.1376524.1647877010498.ref@mail.yahoo.com> <844768148.1376524.1647877010498@mail.yahoo.com> Message-ID: <8b8h3h5o6taj5ve13u4uiv1m5ilajmf47m@4ax.com> On Mon, 21 Mar 2022 15:36:50 +0000 (UTC), "Kevin M. Wilson" declaimed the following: >The use of Java options environment variables detected. >Such variables override IDE configuration files (*.vmoptions) and may cause performance and stability issues. >Please consider deleting these variables: _JAVA_OPTIONS. > >Now I've opened the installed .bat files...append.bat, format.bat, inspect.bat, itedit.bat, and pycharm.bat! >Of the Five(5) listed above, only 'pycharm.bat' contains statements setting up the IDE' run environment, beyond >Seven (7) lines. > >Having searched the 'pycharm.bat' file for "_JAVA_OPTIONS", and the 'pycharm64.exe.vmoptions' file as well. The key words are "environment variables". The message is telling you that you have things set on your computer that take effect regardless of config files -- so will not be found IN the config files. https://docs.microsoft.com/en-us/windows/win32/procthread/environment-variables https://www.computerhope.com/issues/ch000549.htm You may need to find out what you installed before that created the environment variable -- since deleting the variable might affect how that application operates. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From email at paulstgeorge.com Mon Mar 21 17:12:03 2022 From: email at paulstgeorge.com (Paul St George) Date: Mon, 21 Mar 2022 22:12:03 +0100 Subject: for convenience Message-ID: When I am writing code, I often do things like this: context = bpy.context # convenience then whenever I need bpy.context, I only need to write context Here?s my question: When I forget to use the convenient shorter form why is bpy.context not interpreted as bpy.bpy.context? ? Paul St George From rosuav at gmail.com Mon Mar 21 17:17:32 2022 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 22 Mar 2022 08:17:32 +1100 Subject: for convenience In-Reply-To: References: Message-ID: On Tue, 22 Mar 2022 at 08:13, Paul St George wrote: > > > When I am writing code, I often do things like this: > > context = bpy.context # convenience > > then whenever I need bpy.context, I only need to write context > > > Here?s my question: > > When I forget to use the convenient shorter form > > why is bpy.context not interpreted as bpy.bpy.context? > I don't understand the question. When you do that "for convenience" assignment, what you're doing is creating a local variable named "context" which refers to the same thing that bpy.context does (or did at the time of the assignment, but presumably you only do this when bpy.context won't get reassigned). It has no effect on any other name. There's no magic happening here - it's just assignment to the name context, like anything else. What are you expecting to happen here? ChrisA From pbryan at anode.ca Mon Mar 21 17:24:36 2022 From: pbryan at anode.ca (Paul Bryan) Date: Mon, 21 Mar 2022 14:24:36 -0700 Subject: for convenience In-Reply-To: References: Message-ID: <65a0bc8835d565657442d9cde09a1ee9c7c9b61b.camel@anode.ca> Assuming `bpy` is a module, you're creating a new attribute in your module, `context`, that contains a reference to the same object that is referenced in the `context` attribute in the `bpy` module. On Mon, 2022-03-21 at 22:12 +0100, Paul St George wrote: > > When I am writing code, I often do things like this: > > context = bpy.context? # convenience > > then whenever I need bpy.context, I only need to write context > > > Here?s my question: > > When I forget to use the convenient shorter form > > why is bpy.context not interpreted as bpy.bpy.context? > > > ? > Paul St George > > > > > From email at paulstgeorge.com Mon Mar 21 17:31:24 2022 From: email at paulstgeorge.com (Paul St George) Date: Mon, 21 Mar 2022 22:31:24 +0100 Subject: for convenience In-Reply-To: <65a0bc8835d565657442d9cde09a1ee9c7c9b61b.camel@anode.ca> References: <65a0bc8835d565657442d9cde09a1ee9c7c9b61b.camel@anode.ca> Message-ID: <21057883-9FB1-4A0E-A2B4-989F88DE789D@paulstgeorge.com> Hi, I do not (knowingly) have a module called ?context'. > On 21 Mar 2022, at 22:24, Paul Bryan wrote: > > Assuming `bpy` is a module, you're creating a new attribute in your module, `context`, that contains a reference to the same object that is referenced in the `context` attribute in the `bpy` module. > > On Mon, 2022-03-21 at 22:12 +0100, Paul St George wrote: >> >> When I am writing code, I often do things like this: >> >> context = bpy.context # convenience >> >> then whenever I need bpy.context, I only need to write context >> >> >> Here?s my question: >> >> When I forget to use the convenient shorter form >> >> why is bpy.context not interpreted as bpy.bpy.context? >> >> >> ? >> Paul St George >> >> >> >> >> > From pbryan at anode.ca Mon Mar 21 17:41:34 2022 From: pbryan at anode.ca (Paul Bryan) Date: Mon, 21 Mar 2022 14:41:34 -0700 Subject: for convenience In-Reply-To: <21057883-9FB1-4A0E-A2B4-989F88DE789D@paulstgeorge.com> References: <65a0bc8835d565657442d9cde09a1ee9c7c9b61b.camel@anode.ca> <21057883-9FB1-4A0E-A2B4-989F88DE789D@paulstgeorge.com> Message-ID: No, nor did I suggest that you did. `context` is presumably an attribute in the `bpy` module, for which you are creating a `context` attribute in your module. On Mon, 2022-03-21 at 22:31 +0100, Paul St George wrote: > Hi, > I do not (knowingly) have a module called ?context'. > > > > > > On 21 Mar 2022, at 22:24, Paul Bryan wrote: > > > > Assuming `bpy` is a module, you're creating a new attribute in your > > module, `context`, that contains a reference to the same object > > that is referenced in the `context` attribute in the `bpy` module. > > > > On Mon, 2022-03-21 at 22:12 +0100, Paul St George wrote: > > > > > > When I am writing code, I often do things like this: > > > > > > context = bpy.context? # convenience > > > > > > then whenever I need bpy.context, I only need to write context > > > > > > > > > Here?s my question: > > > > > > When I forget to use the convenient shorter form > > > > > > why is bpy.context not interpreted as bpy.bpy.context? > > > > > > > > > ? > > > Paul St George > > > > > > > > > > > > > > > > > > > > From avigross at verizon.net Mon Mar 21 17:47:56 2022 From: avigross at verizon.net (Avi Gross) Date: Mon, 21 Mar 2022 21:47:56 +0000 (UTC) Subject: for convenience In-Reply-To: References: Message-ID: <651952493.752313.1647899276486@mail.yahoo.com> Chris, I think you understood the context but not the premise in a sense that wasin the way Paul was thinking. His premise is way off He seems to be thinking of something like a macro concept as iscommonly used in languages like C so: #define context bpy.context That could, in such languages, use a macro preprocessor, to parse the code?once to make various textual changes without any real understanding of thelanguage and produce output ready for a compiler or interpreter phase.?Python has nothing really like that albeit I can think of some things as well asgimmicks and tricks that can result in less typing. As an example, you canarrange for an object embedded within another to have some way to?manipulate deeper structures from a higher level meaning less typing. As you point out, in many? languages, often we are dealing with an implicit pointer withvery different results. And it is not at all the same in other ways, as you againpoint out so if you later re-assign bpy to anything else, including anotherdistinct object of the same kind, the variable called "context" keeps pointingto the old part of bpy. It is not an alias that can be used and it also is a potential problemfor various reasons such as retarding garbage collection or being used laterto make a change in the wrong place. So, I ask Paul what other language than python he has used before, just out of curiosity.Many and perhaps most of use regulars here have used quite a few others and eachhas its own quirks and we have adjusted our thinking multiple times by the time welearned Python. What Paul suggests is just a convenience is more than that. It is a variablebinding with many ramifications. -----Original Message----- From: Chris Angelico To: python-list at python.org Sent: Mon, Mar 21, 2022 5:17 pm Subject: Re: for convenience On Tue, 22 Mar 2022 at 08:13, Paul St George wrote: > > > When I am writing code, I often do things like this: > > context = bpy.context? # convenience > > then whenever I need bpy.context, I only need to write context > > > Here?s my question: > > When I forget to use the convenient shorter form > > why is bpy.context not interpreted as bpy.bpy.context? > I don't understand the question. When you do that "for convenience" assignment, what you're doing is creating a local variable named "context" which refers to the same thing that bpy.context does (or did at the time of the assignment, but presumably you only do this when bpy.context won't get reassigned). It has no effect on any other name. There's no magic happening here - it's just assignment to the name context, like anything else. What are you expecting to happen here? ChrisA -- https://mail.python.org/mailman/listinfo/python-list From cs at cskk.id.au Mon Mar 21 18:02:49 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Tue, 22 Mar 2022 09:02:49 +1100 Subject: for convenience In-Reply-To: References: Message-ID: On 21Mar2022 22:12, Paul St George wrote: >When I am writing code, I often do things like this: > >context = bpy.context # convenience > >then whenever I need bpy.context, I only need to write context > > >Here?s my question: > >When I forget to use the convenient shorter form > >why is bpy.context not interpreted as bpy.bpy.context? Because it still has its original meaning. You haven't changed the meaning of the word "context" in any position, you have simply made a module level name "context" referring to the same object to which "bpy.context" refers. So your module global namespace contains, initially, "bpy". Then you assign: context = bpy.context and now your module global namespace contains "bpy" and "context". But "bpy.context" is still what it was, because "bpy" is an object and you have done nothing to its ".context" attribute. Consider this code: class O: pass o = O() o.x = 3 x = o.x print(x) print(o.x) I expect to see "3" twice. What do you expect? "bpy" is no different to "o" - it's just a name. Cheers, Cameron Simpson From PythonList at DancesWithMice.info Mon Mar 21 18:04:53 2022 From: PythonList at DancesWithMice.info (dn) Date: Tue, 22 Mar 2022 11:04:53 +1300 Subject: for convenience In-Reply-To: References: Message-ID: On 22/03/2022 10.17, Chris Angelico wrote: > On Tue, 22 Mar 2022 at 08:13, Paul St George wrote: >> >> >> When I am writing code, I often do things like this: >> >> context = bpy.context # convenience >> >> then whenever I need bpy.context, I only need to write context >> >> >> Here?s my question: >> >> When I forget to use the convenient shorter form >> >> why is bpy.context not interpreted as bpy.bpy.context? >> > > I don't understand the question. When you do that "for convenience" > assignment, what you're doing is creating a local variable named > "context" which refers to the same thing that bpy.context does (or did > at the time of the assignment, but presumably you only do this when > bpy.context won't get reassigned). It has no effect on any other name. > There's no magic happening here - it's just assignment to the name > context, like anything else. > > What are you expecting to happen here? It's the way Python works. try: context = bpy.context # convenience print( id(context), id(bpy.context) ) Remember that the 'relationship' between the two is established at run-time and at the data/address 'level' - and not at compile-time. Thus "context" points to a memory location, and does not 'stand for' "bpy.context" anywhere other than in your mind. (which is why we often need to use a copy() when we want 'separate data' - see also the 'counters' Python uses to manage "garbage collection") -- Regards, =dn From skip.montanaro at gmail.com Mon Mar 21 19:50:12 2022 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Mon, 21 Mar 2022 18:50:12 -0500 Subject: asyncio+tkinter Message-ID: Given that both asyncio & tkinter are modules in the standard lib and both have event loops, I would have expected to find some "best practice" solution to mixing the two. I've not used asyncio, but might find it useful with the pynput module in the context of a Tk app. I see a few solutions out in the wild, but this seems like something which might best be addressed in either the asyncio or tkinter documentation, or better yet, implemented in one or the other. Skip Montanaro From rosuav at gmail.com Mon Mar 21 19:58:45 2022 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 22 Mar 2022 10:58:45 +1100 Subject: asyncio+tkinter In-Reply-To: References: Message-ID: On Tue, 22 Mar 2022 at 10:52, Skip Montanaro wrote: > > Given that both asyncio & tkinter are modules in the standard lib and both > have event loops, I would have expected to find some "best practice" > solution to mixing the two. I've not used asyncio, but might find it useful > with the pynput module in the context of a Tk app. I see a few solutions > out in the wild, but this seems like something which might best be > addressed in either the asyncio or tkinter documentation, or better yet, > implemented in one or the other. > Agreed. For GTK, you can use a dedicated loop policy like this: import asyncio_glib asyncio.set_event_loop_policy(asyncio_glib.GLibEventLoopPolicy()) It should be possible to make it this easy for asyncio + tkinter. ChrisA From avigross at verizon.net Tue Mar 22 14:00:08 2022 From: avigross at verizon.net (Avi Gross) Date: Tue, 22 Mar 2022 18:00:08 +0000 (UTC) Subject: convenience References: <1635993559.938025.1647972008415.ref@mail.yahoo.com> Message-ID: <1635993559.938025.1647972008415@mail.yahoo.com> An earlier post talked about a method they used for "convenience" in a way they apparently did not understand and many of us educated them, hopefully. That made me wonder of teh impact on our code when we use various forms of convenience. Is it convenient for us as programmers, other potential readers, or a compiler or interpreter? The example used was something like: varname = objectname.varname The above clearly requires an existing object. But there is no reason it has to be the same name. Consider the somewhat related idea used almost always in code: import numpy as np Both cases make a sort of pointer variable and are a tad shorter to write. But what impact does it have on interpreters and compilers? I assume objectname.varname makes the interpreter look up "objectname" in some stack of namespaces and find it. Then it looks within the namespace inside the object and finds varname. Finally, it does whatever you asked for such as getting or setting a value or something more complex. So what happens if you simply call "varname" after the above line of code has set it to be a partial synonym for the longer name? Clearly it no longer does any work on evaluating "objectname" and that may be a significant saving as the name may be deep in the stack of namespaces. But are there costs or even errors if you approach an inner part of an object directly? Can there be dunder methods not invoked that would be from the standard approach? What kind of inadvertent errors can creep in? I have seen lots of errors in many languages and functions designed to make some things superficially easier. Sometimes it involves adding a namespace to the stack of namespaces that contains the names of all column names of a DataFrame, for example to avoid extra typing, and this can make fairly complex and long pieces of code way shorter. But this can also make other existing names go deeper into the stack and temporarily shadowed or if a similar technique is used a bit later in the code while this is in effect, can shadow it as in opening another object of the same kind. Sometimes there is a price for convenience. And even if you are fully aware of the pitfalls, and expect to gain from the convenience to you or even the interpreter, your work may be copied or expanded on by others who do not take care. In the example above in a language like R, you can add the namespace to the environment and use it in read-only mode but writing to a resulting variable like "col1" makes a new local variable and does not make any change to the underlying filed in a data.frame. I mean (again this is not Python code) that: mydata$result <- mydata$col1 + mydata$col2 will create or update a column called "result" within the data.frame called mydata, BUT with(mydata, result <- col1 + col2) That does not work well and leaves "result" alone or non-existent. Instead, a local variable called result is created and then rapidly abandoned as shown by code that displays it: with(mydata, {result <- col1 + col2; print(result)}) The above returns 3 when the columns in mydata are 1 and 2. Just FYI, there are other packages that handle such short names properly and well, but they are done carefully. Regular users may easily make mistakes for the sake of convenience. So I wonder if some pythonic methods fit into convenience modes or can veer into dangerous modes. Spelling things out in full detail may be annoying but generally safer. From reuels.life at gmail.com Tue Mar 22 09:50:55 2022 From: reuels.life at gmail.com (Reuel Lewis) Date: Tue, 22 Mar 2022 19:20:55 +0530 Subject: Difficulty in installing Python Message-ID: Hello, I'm trying to install Python as I'd like to learn it. I'm a newbie in all things related to software programming. I have Windows 10 installed on my HP laptop. I tried to install Python but I didn't Click on the 'Add python 3.10 to path'. So I tried to uninstall it and install Python once more. But I only get options like Modify / Repair. If I click on either of those options it does the Modify / Repair thing and then I just see the 'Close' button. It does not go anywhere else from there. So what should I do to start this program so I can start learning? Thank you and stay safe always. Kind Regards, Reuel R. Lewis From email at paulstgeorge.com Tue Mar 22 16:23:10 2022 From: email at paulstgeorge.com (Paul St George) Date: Tue, 22 Mar 2022 21:23:10 +0100 Subject: for convenience Message-ID: <25C8E2FE-878A-416F-89C6-6BD33F8B49D2@paulstgeorge.com> On 21/03/2022 18.04, dn wrote: > On 22/03/2022 10.17, Chris Angelico wrote: > > On Tue, 22 Mar 2022 at 08:13, Paul St George > wrote: > >> > >> > >> When I am writing code, I often do things like this: > >> > >> context = bpy.context # convenience > >> > >> then whenever I need bpy.context, I only need to write context > >> > >> > >> Here?s my question: > >> > >> When I forget to use the convenient shorter form > >> > >> why is bpy.context not interpreted as bpy.bpy.context? > >> > > > > I don't understand the question. When you do that "for convenience" > > assignment, what you're doing is creating a local variable named > > "context" which refers to the same thing that bpy.context does (or did > > at the time of the assignment, but presumably you only do this when > > bpy.context won't get reassigned). It has no effect on any other name. > > There's no magic happening here - it's just assignment to the name > > context, like anything else. > > > > What are you expecting to happen here? > > > It's the way Python works. > > try: > > context = bpy.context # convenience > print( id(context), id(bpy.context) ) > > Remember that the 'relationship' between the two is established at > run-time and at the data/address 'level' - and not at compile-time. Thus > "context" points to a memory location, and does not 'stand for' > "bpy.context" anywhere other than in your mind. > > (which is why we often need to use a copy() when we want 'separate data' > - see also the 'counters' Python uses to manage "garbage collection") > > -- > Regards, > =dn Thanks dn, I did exactly as you suggested. import bpy context = bpy.context # convenience variable print( id(context), id(bpy.context) ) 4932508928 4932508928 The two are the same, and that makes it crystal clear. And, as a bonus you have explained a use of copy(). ? Thanks, Paul From email at paulstgeorge.com Tue Mar 22 16:39:14 2022 From: email at paulstgeorge.com (Paul St George) Date: Tue, 22 Mar 2022 21:39:14 +0100 Subject: for convenience Message-ID: On 21/03/2022 18.02, Cameron Simpson wrote: > On 21Mar2022 22:12, Paul St George wrote: > >When I am writing code, I often do things like this: > > > >context = bpy.context # convenience > > > >then whenever I need bpy.context, I only need to write context > > > > > >Here?s my question: > > > >When I forget to use the convenient shorter form > > > >why is bpy.context not interpreted as bpy.bpy.context? > > Because it still has its original meaning. You haven't changed the > meaning of the word "context" in any position, you have simply made a > module level name "context" referring to the same object to which > "bpy.context" refers. > > So your module global namespace contains, initially, "bpy". Then you > assign: > > context = bpy.context > > and now your module global namespace contains "bpy" and "context". But > "bpy.context" is still what it was, because "bpy" is an object and you > have done nothing to its ".context" attribute. > > Consider this code: > > class O: > pass > > o = O() > o.x = 3 > > x = o.x > > print(x) > print(o.x) > > I expect to see "3" twice. What do you expect? > > "bpy" is no different to "o" - it's just a name. Thanks Cameron, What did I expect? Well, I expected to see "3? twice, but I did not understand why. Now I do! ? Thanks, Paul From email at paulstgeorge.com Tue Mar 22 16:52:43 2022 From: email at paulstgeorge.com (Paul St George) Date: Tue, 22 Mar 2022 21:52:43 +0100 Subject: for convenience Message-ID: <2AC67ABA-04F3-4ED9-8F74-86D5F31D2253@paulstgeorge.com> On 21/03/2022 17.47, Avi Gross wrote: > So, I ask Paul what other language than python he has used before, just out of curiosity. The other language I have used (and often) is Processing. Before that, and a long time ago, Lingo. ? Paul From PythonList at DancesWithMice.info Tue Mar 22 18:03:54 2022 From: PythonList at DancesWithMice.info (dn) Date: Wed, 23 Mar 2022 11:03:54 +1300 Subject: for convenience In-Reply-To: <25C8E2FE-878A-416F-89C6-6BD33F8B49D2@paulstgeorge.com> References: <25C8E2FE-878A-416F-89C6-6BD33F8B49D2@paulstgeorge.com> Message-ID: On 23/03/2022 09.23, Paul St George wrote: > On 21/03/2022 18.04, dn wrote: > >> On 22/03/2022 10.17, Chris Angelico wrote: >>> On Tue, 22 Mar 2022 at 08:13, Paul St George > wrote: >>>> >>>> >>>> When I am writing code, I often do things like this: >>>> >>>> context = bpy.context # convenience >>>> >>>> then whenever I need bpy.context, I only need to write context >>>> >>>> >>>> Here?s my question: >>>> >>>> When I forget to use the convenient shorter form >>>> >>>> why is bpy.context not interpreted as bpy.bpy.context? >>>> >>> >>> I don't understand the question. When you do that "for convenience" >>> assignment, what you're doing is creating a local variable named >>> "context" which refers to the same thing that bpy.context does (or did >>> at the time of the assignment, but presumably you only do this when >>> bpy.context won't get reassigned). It has no effect on any other name. >>> There's no magic happening here - it's just assignment to the name >>> context, like anything else. >>> >>> What are you expecting to happen here? >> >> >> It's the way Python works. >> >> try: >> >> context = bpy.context # convenience >> print( id(context), id(bpy.context) ) >> >> Remember that the 'relationship' between the two is established at >> run-time and at the data/address 'level' - and not at compile-time. Thus >> "context" points to a memory location, and does not 'stand for' >> "bpy.context" anywhere other than in your mind. >> >> (which is why we often need to use a copy() when we want 'separate data' >> - see also the 'counters' Python uses to manage "garbage collection") >> >> -- >> Regards, >> =dn > > > Thanks dn, > I did exactly as you suggested. > > import bpy > > context = bpy.context # convenience variable > print( id(context), id(bpy.context) ) > > 4932508928 4932508928 > > The two are the same, and that makes it crystal clear. And, as a bonus you have explained a use of copy(). > > ? > Thanks, > Paul and thank you - it is refreshing, if not enervating, to receive feedback on efforts-expended! You will also notice, that now you understand the id() stuff, the tag-team effect between @Chris and I (which we have often played, albeit not by-design), now makes sense as an whole (if you didn't quite follow, earlier). My research-topic is Cognitive Psychology (how we learn - albeit not usually in Python). I found this conversation useful, and may well apply it as an example (with your permission, and suitably anonymised) - one doesn't need to be a 'computer person' to follow the logic and thus realise the dissonance! While learning (this part of) Python and adding to 'previous experience', you formed a "mental-model" of how things work (just as we all do). However, when it came time to implement this knowledge: - you created a 'situation' - (all) things didn't 'work' (which also required realisation) - you analysed and rationalised (but noted inconsistency) - you asked a question (which many of us quickly understood) - you've learned/corrected The 'issue' is *not* a fault on your part, nor (necessarily) a lack of learning or a lack of effort. So, no criticism from me! The (under-lying) lesson, is that we (as trainers, but with application to all helpers, pair-programmers, mentors, documentation-writers, et al - working with less-experienced colleagues) shouldn't spout a whole load of 'facts', 'rules', and formulae/s - which we expect to be committed to memory. We need to help form a 'correct' mental-model ("correct" being defined by the Python interpreter and 'the Python gods' who build it - big "thank you" to them!). Accordingly, my criticism of tests/exams which require recitation of facts ("parroting"), compared with "mastery" (can you actually DO what is being asked). More importantly, and finally getting to the point: 'tests' should be defined to reveal these (personal) 'quirks' of learning/understanding, which led to a 'faulty' mental-model! Your rationale made sense, was logical and understandable. How are you to know that Python deems it 'wrong'? (until a 'test' shows you!) The 'interest' should not be on the people who, and all the 'answers' which, were 'correct'. What is more informative, is why someone (aside from guessing, ie intelligent, reasonable, after learning the material, exerting effort...) got it 'wrong' - but thought his/her path was true! -- Regards, =dn From rathmann.os at gmail.com Tue Mar 22 18:44:04 2022 From: rathmann.os at gmail.com (Rathmann) Date: Tue, 22 Mar 2022 15:44:04 -0700 (PDT) Subject: Reducing "yield from" overhead in recursive generators In-Reply-To: References: Message-ID: <59370296-e91d-4c21-aced-613b273fceb9n@googlegroups.com> > On 19 Mar 2022, at 03:07, Greg Ewing wrote: > > ?On 19/03/22 9:40 am, Rathmann wrote: >> The other challenge/question would be to see if there is a way to implement >> this basic approach with lower overhead. > > My original implementation of yield-from didn't have this problem, > because it didn't enter any of the intermediate frames -- it just > ran down a chain of C pointers to find the currently active one. > > At some point this was changed, I think so that all the frames > would show up in the traceback in the event of an exception. > This was evidently seen as more important than having efficient > resumption of nested generators. So that sounds like the original CPython implementation had an O(depth) component, but with a lower constant factor than the current version? (Of course, since Python limits recursion depth, speaking of O(n) is not quite right.) Prioritizing stack traces over performance seems consistent with the well known blog post on tail recursion elimination. This is one of the reasons I was motivated to try to attack this at the library/application level -- when the programmer invokes a transformation like I am proposing, they know that it is going to affect the stack trace. I am still hoping to find a way to implement the transformed function/driver combination with lower overhead. This is a little tricky since the performance of the approaches is very dependent on the Python implementation. Below are some timings for 2.7 and 3.8 CPython and PyPy. For 2.7, for the untransformed version I used a loop of yield statements, and timed with time.clock(), for 3.8, it is "yield from" and time.perf_counter(). All times are for traversing a balanced tree of 8 million nodes on X86_64 Linux. 3.8.10 (default, Nov 26 2021, 20:14:08) [GCC 9.3.0] For depth= 22 nodes= 8388607 iters= 1 Elapsed time (yield from): 6.840768865999053 driver: 6.702329244999419 3.6.9 (7.3.1+dfsg-4, Apr 22 2020, 05:15:29) [PyPy 7.3.1 with GCC 9.3.0] For depth= 22 nodes= 8388607 iters= 1 Elapsed time (yield from): 4.037276787999872 driver: 2.3724582960003318 2.7.18 (default, Mar 8 2021, 13:02:45) [GCC 9.3.0] For depth= 22 nodes= 8388607 iters= 1 Elapsed time (yield loop): 8.863244 driver: 13.788312 2.7.13 (7.3.1+dfsg-2, Apr 21 2020, 05:05:41) [PyPy 7.3.1 with GCC 9.3.0] For depth= 22 nodes= 8388607 iters= 1 Elapsed time (yield loop): 8.802712562 driver: 2.034388533 The most extreme variation was 2.7 pypy, where the driver was 4 times faster. (Of course, 2.7 is less used these days.) From avigross at verizon.net Tue Mar 22 20:10:19 2022 From: avigross at verizon.net (Avi Gross) Date: Wed, 23 Mar 2022 00:10:19 +0000 (UTC) Subject: for convenience In-Reply-To: <2AC67ABA-04F3-4ED9-8F74-86D5F31D2253@paulstgeorge.com> References: <2AC67ABA-04F3-4ED9-8F74-86D5F31D2253@paulstgeorge.com> Message-ID: <617094999.1023591.1647994219661@mail.yahoo.com> I sent George a private reply as discussing other languages gets rapidly off-topic. I want to add a small addendum here about the technique he used and a Dave Neal and others are trying, a way to imagine things that is more compatible with how a language like Python works so it meets expectation. Your example was name= obj.name and you saw it as a convenience. But consider some form of linked list, perhaps a multidimensional one like a tree or generalized graph. If you are traversing the list there is a concept of following a sort of pointer to the "next" and once there following that to the "next" and so on, till you find what you want or reach a leaf node and back up and so on. So your routine may have multiple lines that look like: here = here.next As you traverse, "here" keeps moving and a sublist or subtree or whatever lies below you. It is not a convenience but a portable way to keep track of your current location. It keeps changing. And, in some cases, you retreat back and follow a nextleft or nextright pointer of sorts. In examples like that, the construct is not for convenience but simply because it makes no sense to approach an arbitrary data structure and have to say first.next.next.next.next ... Also noted is your use of a direct pointer had positive and also negative ramifications to consider. It is not a synonym just for convenience. -----Original Message----- From: Paul St George To: python-list at python.org Sent: Tue, Mar 22, 2022 4:52 pm Subject: Re: for convenience On 21/03/2022 17.47, Avi Gross wrote: > So, I ask Paul what other language than python he has used before, just out of curiosity. The other language I have used (and often) is Processing. Before that, and a long time ago, Lingo. ? Paul -- https://mail.python.org/mailman/listinfo/python-list From greg.ewing at canterbury.ac.nz Tue Mar 22 19:12:33 2022 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 23 Mar 2022 12:12:33 +1300 Subject: convenience In-Reply-To: References: <1635993559.938025.1647972008415.ref@mail.yahoo.com> <1635993559.938025.1647972008415@mail.yahoo.com> Message-ID: On 23/03/22 7:00 am, Avi Gross wrote: > But are there costs or even errors > if you approach an inner part of an object directly? Can there be dunder methods not > invoked that would be from the standard approach? What kind of inadvertent > errors can creep in? The attribute could be a property that returns different objects at different times depending on some condition. In that case you will need to think about whether you want the current value of the attribute, or the value it had when you looked it up before. Concerning dunder methods, there are a bunch of them involved in looking up an attribute of an object. Theoretically it's possible for these to have side effects and for bad things to happen if you skip them. But that would be very unusual and surprising, and we generally assume such things don't happen. -- Greg From greg.ewing at canterbury.ac.nz Tue Mar 22 19:17:37 2022 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 23 Mar 2022 12:17:37 +1300 Subject: Reducing "yield from" overhead in recursive generators In-Reply-To: <59370296-e91d-4c21-aced-613b273fceb9n@googlegroups.com> References: <59370296-e91d-4c21-aced-613b273fceb9n@googlegroups.com> Message-ID: On 23/03/22 11:44 am, Rathmann wrote: > So that sounds like the original CPython implementation had an > O(depth) component, but with a lower constant factor than the current > version? Yes, but so much lower that I would expect it to be unmeasurable. -- Greg From avigross at verizon.net Wed Mar 23 00:40:24 2022 From: avigross at verizon.net (Avi Gross) Date: Wed, 23 Mar 2022 04:40:24 +0000 (UTC) Subject: convenience In-Reply-To: References: <1635993559.938025.1647972008415.ref@mail.yahoo.com> <1635993559.938025.1647972008415@mail.yahoo.com> Message-ID: <791244324.1048326.1648010424131@mail.yahoo.com> Greg, Yes, what I describe may not be common and some code goes to serious lengths precisely to make direct connections to internals on an object hard to access. But Python indeed allows and perhaps encourages you to use what you consider side effects but perhaps more. There are many dunder methods that allow all sorts of side effects or direct effects. You can set it up so an access of a variable increments a counter but perhaps it can set it up so three tries at guessing a password disables it or refuses to accept a number out of your bounds. Or it may be that some accesses to say write a value might also update several other values. For example, if you store a distance and also a time, any change of either one resets the value of a stored item holding a velocity. Perhaps adding additional values (or removing any) results in an immediate recalculation of the mean or standard deviation or whatever. If you allow direct access then you can make changes that get around that and I see no reason to assume programs do not use such ideas. Perhaps worse is that items may be dynamic. If an object representing me and my library card checks out a book, a new object may be created that holds the book. If you copy a pointer to the reference to the book in the library card object and later the book is returned and a new book checked out, then your reference is now pointing to the old book. If your goal is to get to the CURRENT book, you failed! So I stand by my suggestion that unless you know the code well and can be sure you won't corrupt things, this kind of convenient access has possible dangers. -----Original Message----- From: Greg Ewing To: python-list at python.org Sent: Tue, Mar 22, 2022 7:12 pm Subject: Re: convenience On 23/03/22 7:00 am, Avi Gross wrote: > But are there costs or even errors > if you approach an inner part of an object directly? Can there be dunder methods not > invoked that would be from the standard approach? What kind of inadvertent > errors can creep in? The attribute could be a property that returns different objects at different times depending on some condition. In that case you will need to think about whether you want the current value of the attribute, or the value it had when you looked it up before. Concerning dunder methods, there are a bunch of them involved in looking up an attribute of an object. Theoretically it's possible for these to have side effects and for bad things to happen if you skip them. But that would be very unusual and surprising, and we generally assume such things don't happen. -- Greg -- https://mail.python.org/mailman/listinfo/python-list From barry at barrys-emacs.org Wed Mar 23 05:09:21 2022 From: barry at barrys-emacs.org (Barry Scott) Date: Wed, 23 Mar 2022 09:09:21 +0000 Subject: convenience In-Reply-To: <1635993559.938025.1647972008415@mail.yahoo.com> References: <1635993559.938025.1647972008415.ref@mail.yahoo.com> <1635993559.938025.1647972008415@mail.yahoo.com> Message-ID: <105AF938-1144-4EF1-90F5-6847CEE50C2F@barrys-emacs.org> > On 22 Mar 2022, at 18:00, Avi Gross via Python-list wrote: > > An earlier post talked about a method they used for "convenience" in a way > they apparently did not understand and many of us educated them, hopefully. > > That made me wonder of teh impact on our code when we use various forms > of convenience. Is it convenient for us as programmers, other potential readers, > or a compiler or interpreter? Using aliases can impose a cost for maintenance. It adds a burden on the reader to figure what an alias really refers to. I encountered this: from time import time as now. The use of now() was just confusing to maintainers. I avoid the convenience path as it often makes maintenance harder. > > The example used was something like: > > varname = objectname.varname > > The above clearly requires an existing object. But there is no reason it > has to be the same name. Consider the somewhat related idea used > almost always in code: > > import numpy as np > > Both cases make a sort of pointer variable and are a tad shorter to write. > > But what impact does it have on interpreters and compilers? On the positive side aliases are very useful to improve the speed of hot-path code. read = this.that.the_other.read while condition: buf = read() ... As with all optimisations only do this if you can justify it with benchmark results. Barry From kazya.ito.dream at gmail.com Wed Mar 23 04:55:37 2022 From: kazya.ito.dream at gmail.com (Kazuya Ito) Date: Wed, 23 Mar 2022 01:55:37 -0700 (PDT) Subject: Feature Request Message-ID: <2a319403-5783-4f87-be2a-c07c59b677a3n@googlegroups.com> Add "trun()" function to Python to truncate decimal part. From patatetom at gmail.com Wed Mar 23 09:14:54 2022 From: patatetom at gmail.com (lacsaP Patatetom) Date: Wed, 23 Mar 2022 06:14:54 -0700 (PDT) Subject: difficult start with asyncio async/await Message-ID: <8c8f7c56-ec7c-4bd5-a15c-3c8ef06b61b8n@googlegroups.com> hi, difficult start with asyncio async/await... I'm trying to make a mockup that queries a few sites and posts the results to a server, but the result is not what I expected. I was expecting to get the 4 "get" one after the other, followed by the "post" (eventually mixed) but I get something that looks more like procedural than anything else. some help/clarification would be welcome. here is my code : ```python import asyncio from itertools import cycle from random import randint from termcolor import colored urls = ('yellow', 'cyan', 'green', 'magenta') async def getItems(url): print(colored(f'get new items from url {url} with aiohttp...', url)) # simulate aiohttp request/response await asyncio.sleep(randint(1, 5)) # return a list of some items return [ url + str(i) for i in range(1, randint(2, 5)) ] async def postItem(item, server, url): # url is here only to color print print(colored(f'post new item {item} to server {server} with aiohttp...', url)) # simulate aiohttp request/response await asyncio.sleep(randint(1, 5)) async def runItems(url): items = await getItems(url) for item in items: await postItem(str(item), 'localhost', url) async def loopForever(): while True: print(colored('looping...', 'red')) for url in urls: await runItems(url) # sleeping 30s before next loop await asyncio.sleep(10) loop = asyncio.get_event_loop() tasks = [ loop.create_task(loopForever()) ] loop.run_until_complete(asyncio.wait(tasks)) loop.close() ``` and here is the results: ``` looping... get new items from url yellow with aiohttp... post new item yellow1 to server localhost with aiohttp... post new item yellow2 to server localhost with aiohttp... post new item yellow3 to server localhost with aiohttp... post new item yellow4 to server localhost with aiohttp... get new items from url cyan with aiohttp... post new item cyan1 to server localhost with aiohttp... post new item cyan2 to server localhost with aiohttp... post new item cyan3 to server localhost with aiohttp... post new item cyan4 to server localhost with aiohttp... get new items from url green with aiohttp... post new item green1 to server localhost with aiohttp... post new item green2 to server localhost with aiohttp... post new item green3 to server localhost with aiohttp... post new item green4 to server localhost with aiohttp... get new items from url magenta with aiohttp... post new item magenta1 to server localhost with aiohttp... post new item magenta2 to server localhost with aiohttp... post new item magenta3 to server localhost with aiohttp... post new item magenta4 to server localhost with aiohttp... looping... get new items from url yellow with aiohttp... post new item yellow1 to server localhost with aiohttp... post new item yellow2 to server localhost with aiohttp... post new item yellow3 to server localhost with aiohttp... ... ``` here is what was expected : ``` looping... get new items from url yellow with aiohttp... get new items from url cyan with aiohttp... get new items from url green with aiohttp... get new items from url magenta with aiohttp... post new item... looping... get new items from url yellow with aiohttp... get new items from url cyan with aiohttp... get new items from url green with aiohttp... get new items from url magenta with aiohttp... post new item... ... ``` regards, lacsaP. From wlfraed at ix.netcom.com Wed Mar 23 09:28:57 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Wed, 23 Mar 2022 09:28:57 -0400 Subject: Difficulty in installing Python References: Message-ID: On Tue, 22 Mar 2022 19:20:55 +0530, Reuel Lewis declaimed the following: >I'm trying to install Python as I'd like to learn it. I'm a newbie in all >things related to software programming. Nothing programming related with your problem... >I have Windows 10 installed on my HP laptop. I tried to install Python but >I didn't Click on the 'Add python 3.10 to path'. So I tried to uninstall it >and install Python once more. But I only get options like Modify / Repair. >If I click on either of those options it does the Modify / Repair thing and >then I just see the 'Close' button. It does not go anywhere else from >there. The Python interpreter (what you use to run Python programs) is NOT the Python installer (what you use to make Python available). Once you've installed Python -- hide the installer! It is only used to, well, modify/repair/uninstall Python. Python, by itself, is not an IDE (it is not something like Visual Studio, FreePascal's Lazarus, et al.). It is just a command-line interpreter. >So what should I do to start this program so I can start learning? Thank >you and stay safe always. Well, if you'd added it to the PATH, you'd open a command shell, and provide a script file to the interpreter (if you don't provide a script file, you'd end up in the interactive interpreter prompt -- to exit) -=-=- Microsoft Windows [Version 10.0.19041.1415] (c) Microsoft Corporation. All rights reserved. C:\Users\Wulfraed>python Python ActivePython 3.8.2 (ActiveState Software Inc.) based on on win32 Type "help", "copyright", "credits" or "license" for more information. >>> C:\Users\Wulfraed> -=-=- Recent versions of Python install a generic launcher on Windows named "py", which is supposed to reduce the need to track/update the PATH variable... but for me... I found too many other packages were installing Python interpreters which hijacked the version "py" defaulted to running. Most versions of Python also install a Tkinter script called IDLE which provides a rudimentary IDE capability. > >Kind Regards, >Reuel R. Lewis -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From michael.stemper at gmail.com Wed Mar 23 09:44:45 2022 From: michael.stemper at gmail.com (Michael F. Stemper) Date: Wed, 23 Mar 2022 08:44:45 -0500 Subject: Feature Request In-Reply-To: <2a319403-5783-4f87-be2a-c07c59b677a3n@googlegroups.com> References: <2a319403-5783-4f87-be2a-c07c59b677a3n@googlegroups.com> Message-ID: On 23/03/2022 03.55, Kazuya Ito wrote: > Add "trun()" function to Python to truncate decimal part. Which of these should its behavior copy? >>> from math import pi >>> int(pi) 3 >>> pi-int(pi) 0.14159265358979312 >>> -- Michael F. Stemper This post contains greater than 95% post-consumer bytes by weight. From patatetom at gmail.com Wed Mar 23 10:54:22 2022 From: patatetom at gmail.com (lacsaP Patatetom) Date: Wed, 23 Mar 2022 07:54:22 -0700 (PDT) Subject: difficult start with asyncio async/await In-Reply-To: <8c8f7c56-ec7c-4bd5-a15c-3c8ef06b61b8n@googlegroups.com> References: <8c8f7c56-ec7c-4bd5-a15c-3c8ef06b61b8n@googlegroups.com> Message-ID: some redesigns around asyncio.gather gave me what I wanted. here is the code used : ```python import asyncio from random import randint from termcolor import colored from datetime import datetime urls = ('yellow', 'cyan', 'green', 'magenta') async def getItems(url): print(colored(f'get new items from url {url} with aiohttp...', url)) # simulate aiohttp request/response await asyncio.sleep(randint(1, 3)) # return a list of some items return [ url + str(i) for i in range(1, randint(2, 5)) ] async def postItem(item, server, url): # url is here only to color print print(colored(f'post new item {item} to server {server} with aiohttp...', url)) # simulate aiohttp request/response await asyncio.sleep(randint(1, 3)) async def runItems(url): items = await getItems(url) posts = [ postItem(item, 'localhost', url) for item in items ] posts and await asyncio.gather(*posts) async def loopForever(): while True: print(colored('\n' + datetime.now().strftime('%H:%M:%S') + ' looping...', 'red')) tasks = [ runItems(url) for url in urls ] tasks and await asyncio.gather(*tasks) await asyncio.sleep(30) asyncio.run(loopForever()) ``` do you think this is the right way for what I want to do ? regards, lacsaP. Le mercredi 23 mars 2022 ? 14:15:08 UTC+1, lacsaP Patatetom a ?crit?: > hi, > > difficult start with asyncio async/await... > > I'm trying to make a mockup that queries a few sites and posts the results to a server, but the result is not what I expected. > > I was expecting to get the 4 "get" one after the other, followed by the "post" (eventually mixed) but I get something that looks more like procedural than anything else. > > some help/clarification would be welcome. > > here is my code : > ```python > import asyncio > from itertools import cycle > from random import randint > from termcolor import colored > > urls = ('yellow', 'cyan', 'green', 'magenta') > > async def getItems(url): > print(colored(f'get new items from url {url} with aiohttp...', url)) > # simulate aiohttp request/response > await asyncio.sleep(randint(1, 5)) > # return a list of some items > return [ url + str(i) for i in range(1, randint(2, 5)) ] > > async def postItem(item, server, url): > # url is here only to color print > print(colored(f'post new item {item} to server {server} with aiohttp...', url)) > # simulate aiohttp request/response > await asyncio.sleep(randint(1, 5)) > > async def runItems(url): > items = await getItems(url) > for item in items: > await postItem(str(item), 'localhost', url) > > async def loopForever(): > while True: > print(colored('looping...', 'red')) > for url in urls: > await runItems(url) > # sleeping 30s before next loop > await asyncio.sleep(10) > > loop = asyncio.get_event_loop() > tasks = [ loop.create_task(loopForever()) ] > loop.run_until_complete(asyncio.wait(tasks)) > loop.close() > ``` > > and here is the results: > ``` > looping... > get new items from url yellow with aiohttp... > post new item yellow1 to server localhost with aiohttp... > post new item yellow2 to server localhost with aiohttp... > post new item yellow3 to server localhost with aiohttp... > post new item yellow4 to server localhost with aiohttp... > get new items from url cyan with aiohttp... > post new item cyan1 to server localhost with aiohttp... > post new item cyan2 to server localhost with aiohttp... > post new item cyan3 to server localhost with aiohttp... > post new item cyan4 to server localhost with aiohttp... > get new items from url green with aiohttp... > post new item green1 to server localhost with aiohttp... > post new item green2 to server localhost with aiohttp... > post new item green3 to server localhost with aiohttp... > post new item green4 to server localhost with aiohttp... > get new items from url magenta with aiohttp... > post new item magenta1 to server localhost with aiohttp... > post new item magenta2 to server localhost with aiohttp... > post new item magenta3 to server localhost with aiohttp... > post new item magenta4 to server localhost with aiohttp... > looping... > get new items from url yellow with aiohttp... > post new item yellow1 to server localhost with aiohttp... > post new item yellow2 to server localhost with aiohttp... > post new item yellow3 to server localhost with aiohttp... > ... > ``` > > here is what was expected : > ``` > looping... > get new items from url yellow with aiohttp... > get new items from url cyan with aiohttp... > get new items from url green with aiohttp... > get new items from url magenta with aiohttp... > post new item... > looping... > get new items from url yellow with aiohttp... > get new items from url cyan with aiohttp... > get new items from url green with aiohttp... > get new items from url magenta with aiohttp... > post new item... > ... > ``` > > regards, lacsaP. From wlfraed at ix.netcom.com Wed Mar 23 11:55:06 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Wed, 23 Mar 2022 11:55:06 -0400 Subject: Feature Request References: <2a319403-5783-4f87-be2a-c07c59b677a3n@googlegroups.com> Message-ID: <5pfm3h5mmaebl0rp6gn66rtu2861ivq1oh@4ax.com> On Wed, 23 Mar 2022 01:55:37 -0700 (PDT), Kazuya Ito declaimed the following: >Add "trun()" function to Python to truncate decimal part. You'll have to define what specific behavior you think is missing from the currently available functions? >>> plusover = 2.78 >>> plusunder = 3.14 >>> minusover = -2.78 >>> minusunder = -3.14 >>> import math >>> for v in (plusover, plusunder, minusover, minusunder): ... print("%s: int %s, round %s, math.trunc %s, math.floor %s, math.ceil %s" ... % (v, int(v), round(v), math.trunc(v), math.floor(v), math.ceil(v))) ... 2.78: int 2, round 3, math.trunc 2, math.floor 2, math.ceil 3 3.14: int 3, round 3, math.trunc 3, math.floor 3, math.ceil 4 -2.78: int -2, round -3, math.trunc -2, math.floor -3, math.ceil -2 -3.14: int -3, round -3, math.trunc -3, math.floor -4, math.ceil -3 >>> int() and .trunc() move toward 0, .floor() moves to less positive, .ceil() moves to more positive. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From email at paulstgeorge.com Thu Mar 24 06:31:38 2022 From: email at paulstgeorge.com (Paul St George) Date: Thu, 24 Mar 2022 11:31:38 +0100 Subject: for convenience Message-ID: <433F5B45-E0FC-4DE7-8EA8-CF576B4A542C@paulstgeorge.com> On 22/03/2022 18.04, dn wrote: > and thank you - it is refreshing, if not enervating, to receive feedback > on efforts-expended! > > You will also notice, that now you understand the id() stuff, the > tag-team effect between @Chris and I (which we have often played, albeit > not by-design), now makes sense as an whole (if you didn't quite follow, > earlier). > > > My research-topic is Cognitive Psychology (how we learn - albeit not > usually in Python). I found this conversation useful, and may well apply > it as an example (with your permission, and suitably anonymised) - one > doesn't need to be a 'computer person' to follow the logic and thus > realise the dissonance! > > While learning (this part of) Python and adding to 'previous > experience', you formed a "mental-model" of how things work (just as we > all do). However, when it came time to implement this knowledge: > > - you created a 'situation' > - (all) things didn't 'work' (which also required realisation) > - you analysed and rationalised (but noted inconsistency) > - you asked a question (which many of us quickly understood) > - you've learned/corrected > > > The 'issue' is *not* a fault on your part, nor (necessarily) a lack of > learning or a lack of effort. So, no criticism from me! > > The (under-lying) lesson, is that we (as trainers, but with application > to all helpers, pair-programmers, mentors, documentation-writers, et al > - working with less-experienced colleagues) shouldn't spout a whole load > of 'facts', 'rules', and formulae/s - which we expect to be committed to > memory. We need to help form a 'correct' mental-model ("correct" being > defined by the Python interpreter and 'the Python gods' who build it - > big "thank you" to them!). > > Accordingly, my criticism of tests/exams which require recitation of > facts ("parroting"), compared with "mastery" (can you actually DO what > is being asked). More importantly, and finally getting to the point: > 'tests' should be defined to reveal these (personal) 'quirks' of > learning/understanding, which led to a 'faulty' mental-model! > > Your rationale made sense, was logical and understandable. How are you > to know that Python deems it 'wrong'? (until a 'test' shows you!) > > The 'interest' should not be on the people who, and all the 'answers' > which, were 'correct'. What is more informative, is why someone (aside > from guessing, ie intelligent, reasonable, after learning the material, > exerting effort...) got it 'wrong' - but thought his/her path was true! > -- > Regards, > =dn Wow, this is super interesting. You have my permission, and please feel free to contact me offline if you want to ask anything. Yes, I had noticed the tandem with @Chris. I think I needed both! I already have a folder on my Mac called ?Cameron?. Perhaps I now need an additional folder. Then I can ask my question about whether Python grows to be more like its programmers, or do programmers learn to think Pythonically? ? Paul St George From lukasz at langa.pl Thu Mar 24 08:10:46 2022 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Thu, 24 Mar 2022 13:10:46 +0100 Subject: [RELEASE] Python 3.10.4 and 3.9.12 are now available out of schedule Message-ID: Did anybody say cursed releases ? Well, it turns out that 3.10.3 and 3.9.11 both shipped a regression which caused those versions not to build on Red Hat Enterprise Linux 6. While this 11-year-old version is now out of maintenance support , it?s still used in production workloads. Some of those rely on Python 3.9 and/or 3.10. In particular, our own manylinux2010 image used to build widely compatible Linux wheels is based on CentOS 6. (Don?t worry, we do have newer manylinux* variants, see PEP 599 and PEP 600 for details.) Due to the out-of-schedule release, the respective versions released today contain a very limited set of changes. Python 3.9.12 only contains 12 other bug fixes on top of 3.9.11. Python 3.10.4 only contains 10 other bug fixes on top of 3.10.3. Get 3.10.4 here: Python Release 3.10.4 | Python.org Get 3.9.12 here: Python Release 3.9.12 | Python.org Hopefully, the third time?s a charm and we?ll return no sooner than May with the regularly scheduled bug fix releases of 3.9 and 3.10. We hope you enjoy the new releases Your friendly release team, ?ukasz Langa @ambv Pablo Galindo Salgado @pablogsal Ned Deily @nad Steve Dower @steve.dower -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From steevekerou at yahoo.fr Thu Mar 24 08:38:58 2022 From: steevekerou at yahoo.fr (Steeve Kerou) Date: Thu, 24 Mar 2022 12:38:58 +0000 (UTC) Subject: Pyto Implementation - GROWING TREND References: <739944534.3110098.1648125538279.ref@mail.yahoo.com> Message-ID: <739944534.3110098.1648125538279@mail.yahoo.com> Hi, We develop Pyto - the first python class with an animated character that helps you learn the basics concepts of Python Language like you're playing video game - and we'd like it to be implemented.? Potential is limitless and can reach unlimited number of new users who will then use your software suite. (Don't underestimate the Happy Meal Effect !!) Interested is growing, don?t let another IDE developer implement it before you ! We also developed a series of short classes to learn basic concepts, show the potential of Pyto and how it can be implemented: https://www.youtube.com/watch?v=Eoh7FVR_QcM&list=PLipSv-7x4nWdPTwpZ0pDoOnGhhg3vboh5 Have a look, Learn Python With Pytohttps://www.youtube.com/channel/UCcMXvj1uADEpa1EkqqWxz4galan.vectol at gmail.com From grant.b.edwards at gmail.com Thu Mar 24 11:48:20 2022 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Thu, 24 Mar 2022 08:48:20 -0700 (PDT) Subject: Pyto Implementation - GROWING TREND References: <739944534.3110098.1648125538279.ref@mail.yahoo.com> <739944534.3110098.1648125538279@mail.yahoo.com> Message-ID: <623c92c4.1c69fb81.4d82e.958f@mx.google.com> On 2022-03-24, Steeve Kerou via Python-list wrote: > We develop Pyto - the first python class with an animated character > that helps you learn the basics concepts of Python Language ... Let me guess, the character is named "clipPy"? -- Grant From avigross at verizon.net Thu Mar 24 13:14:10 2022 From: avigross at verizon.net (Avi Gross) Date: Thu, 24 Mar 2022 17:14:10 +0000 (UTC) Subject: for convenience In-Reply-To: <433F5B45-E0FC-4DE7-8EA8-CF576B4A542C@paulstgeorge.com> References: <433F5B45-E0FC-4DE7-8EA8-CF576B4A542C@paulstgeorge.com> Message-ID: <487077501.1160671.1648142050989@mail.yahoo.com> Hopefully, adding to what Dave said, it helps to understand there often are choices and tradeoffs in everything and in particular to language design. And choices propagate so that making choice A and B may box you in so at some point choice Z is pretty much forced unless you start over and make other choices. Python made lots of choices early on and then tried to graft on ever more features, sometimes well and sometimes not optimally. The same is true for so many other languages. A carefully designed new language built now might analyze and make changes. I would even argue that a new language might deliberately include a pre-processor that works more like Paul first assumed, perhaps as something that could be turned on and off so it only operated on designated sections or only when some command-line switch asked for it. Some might use that, especially if migrating code from a language that had used similar constructs. But preprocessors work best in compiled code where it is reasonable to make multiple passes across a changing text but not so much in an interpreted environment that tries to read and parse things once or in small defined chunks dynamically evaluated. As it happens, Paul used what is not at all the same thing as what he has seen elsewhere or that made intuitive sense to him but a Python feature that can (among many other considerations already discussed) effectively do something similar enough when used carefully. But Languages, human as well as computer, can differ widely and yet seem natural enough to most humans who grew up with them. One of my first human languages is not big on "gender" and many sentences are exactly the same no matter if it was being done by a male or female or anything else. So when I later learned German with tons of grammar that included three genders, that was annoying. Later I learned English and it seemed more civilized to me but in some ways not. Is there really a right way to say put verbs first versus last or place adjectives before versus after what they modify? Well every darn language I learn has differences in these ways including exceptions and special cases galore. And even the choice of genders for words like "ship" or "road" vary widely across languages. You either memorize or give up, or more likely, make lots of small mistakes that mark you as a non-native speaker. At some point, you accept it as it IS and work with it an realize the ones you are talking to think it works fine and gets the job done. My first approach to French was to marvel at the weird spelling and how much of it does not seem to get pronounced. It seemed worse than some other Romance languages in some regards. But over time, I got used to it. It started to feel natural, not as in THE right way but as one of many right ways that was right for the situation. I approach computer languages similarly albeit most were not designed over many centuries and influenced by invaders and so on. Features of the language cannot be assumed to be exactly the same as other languages or even assumed to be consistent in other ways. People have made many categorizations of such features such as how memory is managed, whether variables that are first used will default to zero or NULL or the empty string or just generate an error if not initialized and on and on. You can build sort of decision trees showing how several languages keep diverging from others in these abstract ways as you reach a new decision point in such trees and choose a new path. And interestingly, one reason there are so many languages out there is that people seem to have chosen quite widely among alternatives. Some older languages that stop being used may embody decisions that turned out to be too loose or restrictive or simply because a new language seemed sexier and took over. Often it may be that programmers do not like the way it tries to force them to think about what they are doing or that it makes it hard to do common things ... So, in a sense, the learning process is worth studying as many facets of a language that diverge from expectations are either not obvious or were taken after lots of deep thought by others and make sense only once the person learning it has both learned some new things and sort of let go of old things. Languages like Python are actually a bit more of a challenge. Like many modern languages that have been overhauled to support multiple programming methods/styles (like object-oriented and functional programming) and that have been extended with endless add-ons with names like modules and packages, it seems anything you learn may turn out to be "wrong" in the sense that others write code completely differently than you first expected. You may learn base Python and think you use lists for all kinds of things and maybe even lists of lists to emulate matrices, for example. Then one day someone shows you a program done almost completely using modules like numpy and pandas and scipy and so on with bits and pieces of what looks like python to glue it together. Some add-ons define entirely new mini-languages whose rules only make sense within narrow areas and do not generalize. It is very common to have half a dozen ways to do anything, like formatting stuff to print even within the base language. So for many/most people, intuition from other experiences may not be the best guide. Lots of reading helps, and especially books that spell out some hidden assumptions such as books written say for C programmers wanting to learn Python which may also tell you what stuff from C is not carried over, or what work-around is used instead, and even try to instill a concept of what is deemed Pythonic thinking. I have interacted offline with Paul (and others here) a bit. Paul is quite intelligent but also in a somewhat new arena and is learning. Often the best way to learn is by asking questions and applying the answers. I have had people who assumed a language passed values to functions by reference and others by value, and yet others who wondered if there was a way to specify which in a given circumstance. Others assumed it would get evaluated before the function is called and yet others that it would only get evaluated later. There are so many such "choices" in how a language might work and I have seen all the above and more across languages, but generally within a language, you need to ask what it DOES and then accept that and work with it. This reminds me too much of a recent debate here on whether the word "ELSE" means one thing versus another in a new context and I think we had to agree to disagree. Clearly some of us are primed to see it one way and some the other and neither of them is right but also there is NOW a right way to view the construct within Python because it is what it is so get over it! -----Original Message----- From: Paul St George To: python-list at python.org Sent: Thu, Mar 24, 2022 6:31 am Subject: Re: for convenience On 22/03/2022 18.04, dn wrote: > and thank you - it is refreshing, if not enervating, to receive feedback > on efforts-expended! > > You will also notice, that now you understand the id() stuff, the > tag-team effect between @Chris and I (which we have often played, albeit > not by-design), now makes sense as an whole (if you didn't quite follow, > earlier). > > > My research-topic is Cognitive Psychology (how we learn - albeit not > usually in Python). I found this conversation useful, and may well apply > it as an example (with your permission, and suitably anonymised) - one > doesn't need to be a 'computer person' to follow the logic and thus > realise the dissonance! > > While learning (this part of) Python and adding to 'previous > experience', you formed a "mental-model" of how things work (just as we > all do). However, when it came time to implement this knowledge: > > - you created a 'situation' > - (all) things didn't 'work' (which also required realisation) > - you analysed and rationalised (but noted inconsistency) > - you asked a question (which many of us quickly understood) > - you've learned/corrected > > > The 'issue' is *not* a fault on your part, nor (necessarily) a lack of > learning or a lack of effort. So, no criticism from me! > > The (under-lying) lesson, is that we (as trainers, but with application > to all helpers, pair-programmers, mentors, documentation-writers, et al > - working with less-experienced colleagues) shouldn't spout a whole load > of 'facts', 'rules', and formulae/s - which we expect to be committed to > memory. We need to help form a 'correct' mental-model ("correct" being > defined by the Python interpreter and 'the Python gods' who build it - > big "thank you" to them!). > > Accordingly, my criticism of tests/exams which require recitation of > facts ("parroting"), compared with "mastery" (can you actually DO what > is being asked). More importantly, and finally getting to the point: > 'tests' should be defined to reveal these (personal) 'quirks' of > learning/understanding, which led to a 'faulty' mental-model! > > Your rationale made sense, was logical and understandable. How are you > to know that Python deems it 'wrong'? (until a 'test' shows you!) > > The 'interest' should not be on the people who, and all the 'answers' > which, were 'correct'. What is more informative, is why someone (aside > from guessing, ie intelligent, reasonable, after learning the material, > exerting effort...) got it 'wrong' - but thought his/her path was true! > -- > Regards, > =dn Wow, this is super interesting. You have my permission, and please feel free to contact me offline if you want to ask anything. Yes, I had noticed the tandem with @Chris. I think I needed both! I already have a folder on my Mac called ?Cameron?. Perhaps I now need an additional folder. Then I can ask my question about whether Python grows to be more like its programmers, or do programmers learn to think Pythonically? ? Paul St George -- https://mail.python.org/mailman/listinfo/python-list From rosuav at gmail.com Thu Mar 24 13:37:57 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 25 Mar 2022 04:37:57 +1100 Subject: for convenience In-Reply-To: <487077501.1160671.1648142050989@mail.yahoo.com> References: <433F5B45-E0FC-4DE7-8EA8-CF576B4A542C@paulstgeorge.com> <487077501.1160671.1648142050989@mail.yahoo.com> Message-ID: On Fri, 25 Mar 2022 at 04:15, Avi Gross via Python-list wrote: > Python made lots of choices early on and then tried to graft on ever more > features, sometimes well and sometimes not optimally. The same is true > for so many other languages. A carefully designed new language built now > might analyze and make changes. This is technically true, but it's important to keep in mind that it's very difficult to understand the reasons behind all the choices in a language, so if you were to craft your own language now, you would run the risk of throwing away *good* decisions too. > I would even argue that a new language might deliberately include a > pre-processor that works more like Paul first assumed, perhaps as > something that could be turned on and off so it only operated on designated > sections or only when some command-line switch asked for it. Some might > use that, especially if migrating code from a language that had used similar > constructs. But preprocessors work best in compiled code where it is reasonable > to make multiple passes across a changing text but not so much in an > interpreted environment that tries to read and parse things once or in small > defined chunks dynamically evaluated. No, I would say that a preprocessor of that sort isn't necessary to a Python-like language. If you really want one, it's honestly not that hard to do; remember, "preprocessor" means that it processes the source code before the main language sees it, so you can do that even with Python as it currently is. But I suggest that a naive preprocessor like C's wouldn't be of much benefit. Much more interesting is the possibility of designing a language with a proper grammar, which you then compile to Python code and execute. (Part of the reason that a naive preprocessor wouldn't work well is that it would play very badly with dynamic lookups. Good luck making those work with a source-code transformation.) > But Languages, human as well as computer, can differ widely and yet seem > natural enough to most humans who grew up with them. One of my first human > languages is not big on "gender" and many sentences are exactly the same no > matter if it was being done by a male or female or anything else. So when I later > learned German with tons of grammar that included three genders, that was annoying. > Later I learned English and it seemed more civilized to me but in some ways not. > Is there really a right way to say put verbs first versus last or place adjectives > before versus after what they modify? Well every darn language I learn has > differences in these ways including exceptions and special cases galore. And > even the choice of genders for words like "ship" or "road" vary widely across > languages. You either memorize or give up, or more likely, make lots of small > mistakes that mark you as a non-native speaker. Yes, and despite what some people try to claim, they're not just all the same thing with different symbols :) Languages ARE different, different in real ways that make them good at different things. > Languages like Python are actually a bit more of a challenge. Like many modern > languages that have been overhauled to support multiple programming methods/styles > (like object-oriented and functional programming) and that have been extended with > endless add-ons with names like modules and packages, it seems anything you learn > may turn out to be "wrong" in the sense that others write code completely differently > than you first expected. You may learn base Python and think you use lists for all > kinds of things and maybe even lists of lists to emulate matrices, for example. > > Then one day someone shows you a program done almost completely using modules > like numpy and pandas and scipy and so on with bits and pieces of what looks like > python to glue it together. Some add-ons define entirely new mini-languages whose > rules only make sense within narrow areas and do not generalize. To a very large degree, the rules are and must be the same regardless of the library/module. > It is very common > to have half a dozen ways to do anything, like formatting stuff to print even within the > base language. That's because formatting things into strings is such an incredibly broad subject that it's best handled in multiple ways :) You could equally say that there are half a dozen ways to do arithmetic, which there are (or even more, in fact). > This reminds me too much of a recent debate here on whether the word "ELSE" means one > thing versus another in a new context and I think we had to agree to disagree. Clearly > some of us are primed to see it one way and some the other and neither of them is right > but also there is NOW a right way to view the construct within Python because it is what it > is so get over it! I think "if/else" and "try/else" are close enough that nobody should have any concerns about it, and that "for/else" is only confusing because people have an expectation that isn't supported by *any language I have ever used*. If people said "hey, I don't know what for/else should do", that would be fine, but since people say "why doesn't for/else do what I expect it to", it's very hard to explain, since it's nearly impossible to figure out people's preconceptions, since they can't even be explained by other languages' semantics. ChrisA From grant.b.edwards at gmail.com Thu Mar 24 15:04:11 2022 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Thu, 24 Mar 2022 12:04:11 -0700 (PDT) Subject: for convenience References: <433F5B45-E0FC-4DE7-8EA8-CF576B4A542C@paulstgeorge.com> <487077501.1160671.1648142050989@mail.yahoo.com> Message-ID: <623cc0ab.1c69fb81.7726c.ab38@mx.google.com> On 2022-03-24, Chris Angelico wrote: > No, I would say that a preprocessor of that sort isn't necessary to a > Python-like language. If you really want one, it's honestly not that > hard to do; remember, "preprocessor" means that it processes the > source code before the main language sees it, so you can do that even > with Python as it currently is. And they're best used to make it look like you're actually programming in a different language. You can make your C program look like BASIC, or your Python program look like Pascal! The best part is that nobody else will be able to figure out WTH is going on, even if they are fluent in _both_ of the languages! Amaze your friends! Turn complete strangers into enemies! -- Grant From rosuav at gmail.com Thu Mar 24 15:07:56 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 25 Mar 2022 06:07:56 +1100 Subject: for convenience In-Reply-To: <623cc0ab.1c69fb81.7726c.ab38@mx.google.com> References: <433F5B45-E0FC-4DE7-8EA8-CF576B4A542C@paulstgeorge.com> <487077501.1160671.1648142050989@mail.yahoo.com> <623cc0ab.1c69fb81.7726c.ab38@mx.google.com> Message-ID: On Fri, 25 Mar 2022 at 06:05, Grant Edwards wrote: > > On 2022-03-24, Chris Angelico wrote: > > > No, I would say that a preprocessor of that sort isn't necessary to a > > Python-like language. If you really want one, it's honestly not that > > hard to do; remember, "preprocessor" means that it processes the > > source code before the main language sees it, so you can do that even > > with Python as it currently is. > > And they're best used to make it look like you're actually programming > in a different language. You can make your C program look like BASIC, > or your Python program look like Pascal! > > The best part is that nobody else will be able to figure out WTH is > going on, even if they are fluent in _both_ of the languages! > > Amaze your friends! > > Turn complete strangers into enemies! > Yes! This is, in a sense, the counterpart to polyglot code, where the same script can be run with any of several interpreters. Both are extremely effective at turning people's brains to mush. ChrisA From avigross at verizon.net Thu Mar 24 19:44:06 2022 From: avigross at verizon.net (Avi Gross) Date: Thu, 24 Mar 2022 23:44:06 +0000 (UTC) Subject: for convenience In-Reply-To: References: <433F5B45-E0FC-4DE7-8EA8-CF576B4A542C@paulstgeorge.com> <487077501.1160671.1648142050989@mail.yahoo.com> Message-ID: <1439595407.54635.1648165446635@mail.yahoo.com> Chris: > No, I would say that a preprocessor of that sort isn't necessary to a > Python-like language. If you really want one, it's honestly not that > hard to do; remember, "preprocessor" means that it processes the > source code before the main language sees it, so you can do that even > with Python as it currently is. > But I suggest that a naive preprocessor like C's wouldn't be of much > benefit. Much more interesting is the possibility of designing a > language with a proper grammar, which you then compile to Python code > and execute. > (Part of the reason that a naive preprocessor wouldn't work well is > that it would play very badly with dynamic lookups. Good luck making > those work with a source-code transformation.) Again, Chris, I tend to agree and my point was not that Python needs some preprocessor stage but that if it did, then it might make transformations more on a textual level like Paul initially expected. But would it be helpful? Maybe. I am thinking back to decades ago when I did C and C++ programming and how we used it. It was way more that just: #DEFINE TIMEZONE 5 The above use is sort of to create a constant. What we often used was ways to customize the code so different code would be compiled say when testing or to work one way on machine A with operating system A' versus machines B and C, perhaps with different needs and abilities. Sometimes all kinds of features only made it into one version of the other. The compiler saw different code each time. I have also written many things that effectively are read by an early stage and selected regions are parsed and replaced with code in another language so that by the time a compiler or interpreter sees it, ... So sure, you could write code that reads fileA.py and outputs fileB.py and only run fileB through the interpreter. Yes, instead of making versions which have all kinds of text written in various languages, you could just have the python code read in various files at run time or other techniques. -----Original Message----- From: Chris Angelico To: python-list at python.org Sent: Thu, Mar 24, 2022 1:37 pm Subject: Re: for convenience On Fri, 25 Mar 2022 at 04:15, Avi Gross via Python-list wrote: > Python made lots of choices early on and then tried to graft on ever more > features, sometimes well and sometimes not optimally. The same is true > for so many other languages. A carefully designed new language built now > might analyze and make changes. This is technically true, but it's important to keep in mind that it's very difficult to understand the reasons behind all the choices in a language, so if you were to craft your own language now, you would run the risk of throwing away *good* decisions too. > I would even argue that a new language might deliberately include a > pre-processor that works more like Paul first assumed, perhaps as > something that could be turned on and off so it only operated on designated > sections or only when some command-line switch asked for it. Some might > use that, especially if migrating code from a language that had used similar > constructs. But preprocessors work best in compiled code where it is reasonable > to make multiple passes across a changing text but not so much in an > interpreted environment that tries to read and parse things once or in small > defined chunks dynamically evaluated. No, I would say that a preprocessor of that sort isn't necessary to a Python-like language. If you really want one, it's honestly not that hard to do; remember, "preprocessor" means that it processes the source code before the main language sees it, so you can do that even with Python as it currently is. But I suggest that a naive preprocessor like C's wouldn't be of much benefit. Much more interesting is the possibility of designing a language with a proper grammar, which you then compile to Python code and execute. (Part of the reason that a naive preprocessor wouldn't work well is that it would play very badly with dynamic lookups. Good luck making those work with a source-code transformation.) > But Languages, human as well as computer, can differ widely and yet seem > natural enough to most humans who grew up with them. One of my first human > languages is not big on "gender" and many sentences are exactly the same no > matter if it was being done by a male or female or anything else. So when I later > learned German with tons of grammar that included three genders, that was annoying. > Later I learned English and it seemed more civilized to me but in some ways not. > Is there really a right way to say put verbs first versus last? or place adjectives > before versus after what they modify? Well every darn language I learn has > differences in these ways including exceptions and special cases galore. And > even the choice of genders for words like "ship" or "road" vary widely across > languages. You either memorize or give up, or more likely, make lots of small > mistakes that mark you as a non-native speaker. Yes, and despite what some people try to claim, they're not just all the same thing with different symbols :) Languages ARE different, different in real ways that make them good at different things. > Languages like Python are actually a bit more of a challenge. Like many modern > languages that have been overhauled to support multiple programming methods/styles > (like object-oriented and functional programming) and that have been extended with > endless add-ons with names like modules and packages, it seems anything you learn > may turn out to be "wrong" in the sense that others write code completely differently > than you first expected. You may learn base Python and think you use lists for all > kinds of things and maybe even lists of lists to emulate matrices, for example. > > Then one day someone shows you a program done almost completely using modules > like numpy and pandas and scipy and so on with bits and pieces of what looks like > python to glue it together. Some add-ons define entirely new mini-languages whose > rules only make sense within narrow areas and do not generalize. To a very large degree, the rules are and must be the same regardless of the library/module. > It is very common > to have half a dozen ways to do anything, like formatting stuff to print even within the > base language. That's because formatting things into strings is such an incredibly broad subject that it's best handled in multiple ways :) You could equally say that there are half a dozen ways to do arithmetic, which there are (or even more, in fact). > This reminds me too much of a recent debate here on whether the word "ELSE" means one > thing versus another in a new context and I think we had to agree to disagree. Clearly > some of us are primed to see it one way and some the other and neither of them is right > but also there is NOW a right way to view the construct within Python because it is what it > is so get over it! I think "if/else" and "try/else" are close enough that nobody should have any concerns about it, and that "for/else" is only confusing because people have an expectation that isn't supported by *any language I have ever used*. If people said "hey, I don't know what for/else should do", that would be fine, but since people say "why doesn't for/else do what I expect it to", it's very hard to explain, since it's nearly impossible to figure out people's preconceptions, since they can't even be explained by other languages' semantics. ChrisA -- https://mail.python.org/mailman/listinfo/python-list From rosuav at gmail.com Thu Mar 24 19:57:02 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 25 Mar 2022 10:57:02 +1100 Subject: for convenience In-Reply-To: <1439595407.54635.1648165446635@mail.yahoo.com> References: <433F5B45-E0FC-4DE7-8EA8-CF576B4A542C@paulstgeorge.com> <487077501.1160671.1648142050989@mail.yahoo.com> <1439595407.54635.1648165446635@mail.yahoo.com> Message-ID: On Fri, 25 Mar 2022 at 10:44, Avi Gross wrote: > But would it be helpful? Maybe. I am thinking back to decades ago when I > did C and C++ programming and how we used it. It was way more that just: > > #DEFINE TIMEZONE 5 > > The above use is sort of to create a constant. What we often used was ways > to customize the code so different code would be compiled say when testing > or to work one way on machine A with operating system A' versus machines > B and C, perhaps with different needs and abilities. Sometimes all kinds > of features only made it into one version of the other. The compiler saw > different code each time. Yes, I'm aware of how it was used... but to what extent is that actually necessary in Python? When do you ever need something that you can't do with simple conditional function definition or something of the sort? Textual manipulation lets you create all manner of nightmares. A lot of them are acceptable evils because of the benefits gained, but you'd be hard-pressed to pitch those same benefits to Python programmers, where most of the differences are already handled by (eg) the functions in the os module. > I have also written many things that effectively are read by an early stage > and selected regions are parsed and replaced with code in another language > so that by the time a compiler or interpreter sees it, ... > > So sure, you could write code that reads fileA.py and outputs fileB.py and only > run fileB through the interpreter. Yes, instead of making versions which have all > kinds of text written in various languages, you could just have the python code > read in various files at run time or other techniques. And that's what I meant by having your own preprocessor. I've done various DSLs that way, having something that effectively compiles to Python code. (I've also used Python to write preprocessors for things in other languages, since Python is excellent at text manipulation and even has things like JavaScript lexers available on PyPI.) But the best of these are NOT things that the C preprocessor would be capable of. They can have arbitrarily complex levels of syntactic comprehension, making them far safer to work with. ChrisA From avigross at verizon.net Thu Mar 24 19:59:27 2022 From: avigross at verizon.net (Avi Gross) Date: Thu, 24 Mar 2022 23:59:27 +0000 (UTC) Subject: for convenience In-Reply-To: References: <433F5B45-E0FC-4DE7-8EA8-CF576B4A542C@paulstgeorge.com> <487077501.1160671.1648142050989@mail.yahoo.com> Message-ID: <1821154538.57709.1648166367897@mail.yahoo.com> Chris: > To a very large degree, the rules are and must be the same regardless > of the library/module. I may not have been clear. I am aware of all kinds of mini-languages such as various forms of regular expressions where some string contains a fairly complex program using existing symbols like [~* and so on with no real relationship to the language being employed. You can often write code using an identical regular expression and use it with functions in Python, Perl, R, C++ and lots of other languages. Similarly, the printf() family has a guiding string that uses symbols like % and : and others as a sort of mini language that is often independent of the language it is being used in. Yes, some languages contain functions that only support a subset of functionality or a superset. For example, some allow you to specify an argument that you want Perl-style matching enhancements. So, no, I do not think what I was talking about is necessarily completely the same as what the main language uses. And another example is how you can sometimes change the underlying language at run time by all kinds of techniques such as replacing built-in functions, and in languages like R, creating all kinds of new functionality as in the pipes I mentioned including brand new top-level operators or back in Python, tricks like making a new class inherit from a standard class that then over-rides and augments or even plays games with multiple inheritance. I will agree that fundamentally the overall infrastructure seems the same. But in my view, some changes step out of the original language. As I have also mentioned, there are ways to get a program to perform calculations in a shared way with two or more languages with data moving back and forth as needed. Definitely that example, which I sometimes use in my programming, literally jumps out of the initial language. -----Original Message----- From: Chris Angelico To: python-list at python.org Sent: Thu, Mar 24, 2022 1:37 pm Subject: Re: for convenience On Fri, 25 Mar 2022 at 04:15, Avi Gross via Python-list wrote: > Python made lots of choices early on and then tried to graft on ever more > features, sometimes well and sometimes not optimally. The same is true > for so many other languages. A carefully designed new language built now > might analyze and make changes. This is technically true, but it's important to keep in mind that it's very difficult to understand the reasons behind all the choices in a language, so if you were to craft your own language now, you would run the risk of throwing away *good* decisions too. > I would even argue that a new language might deliberately include a > pre-processor that works more like Paul first assumed, perhaps as > something that could be turned on and off so it only operated on designated > sections or only when some command-line switch asked for it. Some might > use that, especially if migrating code from a language that had used similar > constructs. But preprocessors work best in compiled code where it is reasonable > to make multiple passes across a changing text but not so much in an > interpreted environment that tries to read and parse things once or in small > defined chunks dynamically evaluated. No, I would say that a preprocessor of that sort isn't necessary to a Python-like language. If you really want one, it's honestly not that hard to do; remember, "preprocessor" means that it processes the source code before the main language sees it, so you can do that even with Python as it currently is. But I suggest that a naive preprocessor like C's wouldn't be of much benefit. Much more interesting is the possibility of designing a language with a proper grammar, which you then compile to Python code and execute. (Part of the reason that a naive preprocessor wouldn't work well is that it would play very badly with dynamic lookups. Good luck making those work with a source-code transformation.) > But Languages, human as well as computer, can differ widely and yet seem > natural enough to most humans who grew up with them. One of my first human > languages is not big on "gender" and many sentences are exactly the same no > matter if it was being done by a male or female or anything else. So when I later > learned German with tons of grammar that included three genders, that was annoying. > Later I learned English and it seemed more civilized to me but in some ways not. > Is there really a right way to say put verbs first versus last? or place adjectives > before versus after what they modify? Well every darn language I learn has > differences in these ways including exceptions and special cases galore. And > even the choice of genders for words like "ship" or "road" vary widely across > languages. You either memorize or give up, or more likely, make lots of small > mistakes that mark you as a non-native speaker. Yes, and despite what some people try to claim, they're not just all the same thing with different symbols :) Languages ARE different, different in real ways that make them good at different things. > Languages like Python are actually a bit more of a challenge. Like many modern > languages that have been overhauled to support multiple programming methods/styles > (like object-oriented and functional programming) and that have been extended with > endless add-ons with names like modules and packages, it seems anything you learn > may turn out to be "wrong" in the sense that others write code completely differently > than you first expected. You may learn base Python and think you use lists for all > kinds of things and maybe even lists of lists to emulate matrices, for example. > > Then one day someone shows you a program done almost completely using modules > like numpy and pandas and scipy and so on with bits and pieces of what looks like > python to glue it together. Some add-ons define entirely new mini-languages whose > rules only make sense within narrow areas and do not generalize. To a very large degree, the rules are and must be the same regardless of the library/module. > It is very common > to have half a dozen ways to do anything, like formatting stuff to print even within the > base language. That's because formatting things into strings is such an incredibly broad subject that it's best handled in multiple ways :) You could equally say that there are half a dozen ways to do arithmetic, which there are (or even more, in fact). > This reminds me too much of a recent debate here on whether the word "ELSE" means one > thing versus another in a new context and I think we had to agree to disagree. Clearly > some of us are primed to see it one way and some the other and neither of them is right > but also there is NOW a right way to view the construct within Python because it is what it > is so get over it! I think "if/else" and "try/else" are close enough that nobody should have any concerns about it, and that "for/else" is only confusing because people have an expectation that isn't supported by *any language I have ever used*. If people said "hey, I don't know what for/else should do", that would be fine, but since people say "why doesn't for/else do what I expect it to", it's very hard to explain, since it's nearly impossible to figure out people's preconceptions, since they can't even be explained by other languages' semantics. ChrisA -- https://mail.python.org/mailman/listinfo/python-list From avigross at verizon.net Thu Mar 24 21:48:35 2022 From: avigross at verizon.net (Avi Gross) Date: Fri, 25 Mar 2022 01:48:35 +0000 (UTC) Subject: for convenience In-Reply-To: References: <433F5B45-E0FC-4DE7-8EA8-CF576B4A542C@paulstgeorge.com> <487077501.1160671.1648142050989@mail.yahoo.com> <1439595407.54635.1648165446635@mail.yahoo.com> Message-ID: <233820632.66899.1648172915573@mail.yahoo.com> Yes, Chris, you can do all kinds of useful things in Python and I can not make much of a case for requiring a pre-processor. The main reason would be to make code that interprets faster or produces a smaller file of Python commands. All I was saying was that there might be a scenario where a textual replacement method could happen the way Paul sort of guessed. I note that some Python files can be partially processed once into bytecode and then run many times. That may be a level where a preprocessor phase might customize it better so what runs is already half-configured. And as noted, anyone porting code from another language where the code is already segmented with #IFDEF statements and other such fairly primitive methods might have to work harder to move that too into their Python program. I vaguely recall having to write such a preprocessor once upon a time as an exercise for a programming class and it definitely would have been much easier to do in Python. -----Original Message----- From: Chris Angelico To: python-list at python.org Sent: Thu, Mar 24, 2022 7:57 pm Subject: Re: for convenience On Fri, 25 Mar 2022 at 10:44, Avi Gross wrote: > But would it be helpful? Maybe. I am thinking back to decades ago when I > did C and C++ programming and how we used it. It was way more that just: > > #DEFINE TIMEZONE 5 > > The above use is sort of to create a constant. What we often used was ways > to customize the code so different code would be compiled say when testing > or to work one way on machine A with operating system A' versus machines > B and C, perhaps with different needs and abilities. Sometimes all kinds > of features only made it into one version of the other. The compiler saw > different code each time. Yes, I'm aware of how it was used... but to what extent is that actually necessary in Python? When do you ever need something that you can't do with simple conditional function definition or something of the sort? Textual manipulation lets you create all manner of nightmares. A lot of them are acceptable evils because of the benefits gained, but you'd be hard-pressed to pitch those same benefits to Python programmers, where most of the differences are already handled by (eg) the functions in the os module. > I have also written many things that effectively are read by an early stage > and selected regions are parsed and replaced with code in another language > so that by the time a compiler or interpreter sees it, ... > > So sure, you could write code that reads fileA.py and outputs fileB.py and only > run fileB through the interpreter. Yes, instead of making versions which have all > kinds of text written in various languages, you could just have the python code > read in various files at run time or other techniques. And that's what I meant by having your own preprocessor. I've done various DSLs that way, having something that effectively compiles to Python code. (I've also used Python to write preprocessors for things in other languages, since Python is excellent at text manipulation and even has things like JavaScript lexers available on PyPI.) But the best of these are NOT things that the C preprocessor would be capable of. They can have arbitrarily complex levels of syntactic comprehension, making them far safer to work with. ChrisA -- https://mail.python.org/mailman/listinfo/python-list From 19dmcelwaine.student at daos.uk Fri Mar 25 06:12:30 2022 From: 19dmcelwaine.student at daos.uk (jlfivn) Date: Fri, 25 Mar 2022 03:12:30 -0700 (PDT) Subject: code blocks in Python In-Reply-To: References: <8ef9bea6.0311221215.804bdb3@posting.google.com> <8ef9bea6.0311221913.427cb53e@posting.google.com> <8ef9bea6.0311231634.1ca497ac@posting.google.com> <8ef9bea6.0311232220.4e308a4c@posting.google.com> <8ef9bea6.0311241105.e53c47d@posting.google.com> <8ef9bea6.0311241527.61a27b6c@posting.google.com> Message-ID: On Monday, 24 November 2003 at 23:54:49 UTC, John Roth wrote: > "Hung Jung Lu" wrote in message > news:8ef9bea6.03112... at posting.google.com... > > Skip Montanaro wrote in message > news:... > > > > > I come back again to repeat it one more time: the compile() function > > already exists and works in Python. Before you do your posting, please > > think about the compile() function first. (Do I need to say it one > > more time? I don't mind.) > So what? > Functionality is not added to Python simply because it looks like > a logical extension of something else that already exists. First, > you need to show a compelling use case. > So far, I've seen one thing in your proposal: dynamic binding > of free variables in a function, rather than static binding. All > questions of syntax aside, please show me why this matters, > bearing in mind that I've never programmed in a language > that has this, and am not going to be convinced by references > to such languages. > While I'm not *the* person that has to be convinced (that's Guido), > I'm probably representative. If you don't manage a compelling > case for why dynamic binding is a useful option, then you're not going > to get anywhere with this proposal. > By the way - if I understand the guts of the proposal, the compile > function has nothing to do with it, and wouldn't be used to implement > it in any case. > John Roth One use case is modifying the underlying bytecode of the code block to allow features that are not implemented or should not be implemented in python but still make sense in context. For example python-goto is a module that allows you to add goto statements to python code. Currently it has to be applied as a decorator to a function which then needs to be called. Being able to create a code block and then run it instead of having to rap it in a function could be very useful is such cases. I propose the following syntax: code_block : #start a code block print("foobar") #the code block would be executed immediately #would print "foobar" code_block as my_code: #save the code to the named variable;does not execute immediately print("foobar") #no output but code is in my_code exec(my_code) #would print "foobar" @with_goto #the code block would be past to the decorator then executed code_block: print("ignore the fact that I'm using goto") goto .cheat print("goto is a bad idea") label .cheat #output :"ignore the fact that I'm using goto" From python at mrabarnett.plus.com Fri Mar 25 17:58:30 2022 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 25 Mar 2022 21:58:30 +0000 Subject: code blocks in Python In-Reply-To: References: <8ef9bea6.0311221215.804bdb3@posting.google.com> <8ef9bea6.0311221913.427cb53e@posting.google.com> <8ef9bea6.0311231634.1ca497ac@posting.google.com> <8ef9bea6.0311232220.4e308a4c@posting.google.com> <8ef9bea6.0311241105.e53c47d@posting.google.com> <8ef9bea6.0311241527.61a27b6c@posting.google.com> Message-ID: <05fd9979-1ce5-d51d-27b5-7be43ab2b5a3@mrabarnett.plus.com> On 2022-03-25 10:12, jlfivn wrote: > On Monday, 24 November 2003 at 23:54:49 UTC, John Roth wrote: >> "Hung Jung Lu" wrote in message >> news:8ef9bea6.03112... at posting.google.com... >> > Skip Montanaro wrote in message >> news:... >> > > >> > I come back again to repeat it one more time: the compile() function >> > already exists and works in Python. Before you do your posting, please >> > think about the compile() function first. (Do I need to say it one >> > more time? I don't mind.) >> So what? >> Functionality is not added to Python simply because it looks like >> a logical extension of something else that already exists. First, >> you need to show a compelling use case. >> So far, I've seen one thing in your proposal: dynamic binding >> of free variables in a function, rather than static binding. All >> questions of syntax aside, please show me why this matters, >> bearing in mind that I've never programmed in a language >> that has this, and am not going to be convinced by references >> to such languages. >> While I'm not *the* person that has to be convinced (that's Guido), >> I'm probably representative. If you don't manage a compelling >> case for why dynamic binding is a useful option, then you're not going >> to get anywhere with this proposal. >> By the way - if I understand the guts of the proposal, the compile >> function has nothing to do with it, and wouldn't be used to implement >> it in any case. >> John Roth > > One use case is modifying the underlying bytecode of the code block to allow features that are not implemented or should not be implemented in python but still make sense in context. For example python-goto is a module that allows you to add goto statements to python code. Currently it has to be applied as a decorator to a function which then needs to be called. Being able to create a code block and then run it instead of having to rap it in a function could be very useful is such cases. > I propose the following syntax: > > code_block : #start a code block > print("foobar") > > #the code block would be executed immediately > #would print "foobar" > > code_block as my_code: #save the code to the named variable;does not execute immediately > print("foobar") > #no output but code is in my_code > > exec(my_code) > #would print "foobar" > > @with_goto #the code block would be past to the decorator then executed > code_block: > print("ignore the fact that I'm using goto") > goto .cheat > print("goto is a bad idea") > label .cheat > > #output :"ignore the fact that I'm using goto" > That must be some kind of record, replying to a post from over 18 _years_ ago! :-) From timocapoeira at hotmail.com Sat Mar 26 15:29:07 2022 From: timocapoeira at hotmail.com (Timo Meijer) Date: Sat, 26 Mar 2022 19:29:07 +0000 Subject: can't use Python in Materialise Mimics Message-ID: Dear Sir/Madam, Due to uknown reasons my materialise mimics floating license does not recognize Python. I have installed it under my C drive, program files, Python. But when selecting this folder it says it does not contain the program. I have followed these steps: https://www.youtube.com/watch?v=iVBCNZykcrc&ab_channel=MaterialiseMedical But unfortunately it does not work. Could you help me out? Kind regards, Timo Meijer 0031651634920 Verzonden vanuit Mail voor Windows From PythonList at DancesWithMice.info Sat Mar 26 17:31:38 2022 From: PythonList at DancesWithMice.info (dn) Date: Sun, 27 Mar 2022 10:31:38 +1300 Subject: can't use Python in Materialise Mimics In-Reply-To: References: Message-ID: Dear Timo, On 27/03/2022 08.29, Timo Meijer wrote: > > Dear Sir/Madam, > > Due to uknown reasons my materialise mimics floating license does not recognize Python. > I have installed it under my C drive, program files, Python. But when selecting this folder it says it does not contain the program. I have followed these steps: > > https://www.youtube.com/watch?v=iVBCNZykcrc&ab_channel=MaterialiseMedical > > But unfortunately it does not work. > Could you help me out? Step 1 is to ensure that Python is working on that machine. In the Documentation, please find: https://docs.python.org/3/using/windows.html https://docs.python.org/3/faq/windows.html Step 2 (if necessary) will be to seek support under the software's license terms. -- Regards, =dn From voodoo.bender at gmail.com Sat Mar 26 18:47:53 2022 From: voodoo.bender at gmail.com (alberto) Date: Sat, 26 Mar 2022 15:47:53 -0700 (PDT) Subject: save data from multiple txt files Message-ID: <361743a0-186f-4938-95e0-7864a789accbn@googlegroups.com> Hi to everyone, I would save data from multiple files in one using pandas. below my script # Read results GCMG LAMMPS import pandas as pd import os import glob path = r"C:\Users\Documenti\Pyton\plot\results_CH4_180K\METHANE_180K_LJ_2.5-35.0_bar" os.chdir(path) results = pd.DataFrame() for counter, current_file in enumerate(glob.glob("results_*.log")): gcmcdatadf = pd.read_csv(current_file, header=2, sep=" ", usecols=[1, 2, 3]) print(gcmcdatadf) results = pd.concat([results, gcmcdatadf]) #results.to_csv('resuls_tot.log', header=None, sep=" ") In console I obtain Empty DataFrame Columns: [182.244, 10, 0.796176] Index: [] Empty DataFrame Columns: [181.126, 12.5, 0.995821] Index: [] Empty DataFrame Columns: [180.419, 15, 1.21188] Index: [] Empty DataFrame Columns: [179.8, 17.5, 1.43485] Index: [] Empty DataFrame Columns: [198.308, 2.5, 0.255925] Index: [] Empty DataFrame Columns: [179.68, 20, 1.65276] Index: [] Empty DataFrame Columns: [179.629, 22.5, 1.91916] Index: [] Empty DataFrame Columns: [179.838, 25, 2.20493] Index: [] Empty DataFrame Columns: [179.73, 27.5, 2.5408] Index: [] Empty DataFrame Columns: [179.645, 30, 2.98309] Index: [] Empty DataFrame Columns: [179.799, 32.5, 3.65617] Index: [] Empty DataFrame Columns: [180.184, 35, 22.3041] Index: [] Empty DataFrame Columns: [187.835, 5, 0.427784] Index: [] Empty DataFrame Columns: [183.896, 7.5, 0.606451] Index: [] #results.to_csv('resuls_tot.log', name=(T [K],)) How could I eliminate 'index:[]', an print results in one file as #T Pres Density 198.308, 2.5, 0.255925 . . . 180.184, 35, 22.3041 regards A. From cs at cskk.id.au Sat Mar 26 22:11:15 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Sun, 27 Mar 2022 13:11:15 +1100 Subject: save data from multiple txt files In-Reply-To: <361743a0-186f-4938-95e0-7864a789accbn@googlegroups.com> References: <361743a0-186f-4938-95e0-7864a789accbn@googlegroups.com> Message-ID: On 26Mar2022 15:47, alberto wrote: >Hi to everyone, >I would save data from multiple files in one using pandas. >below my script Well, it looks like you're doing the right thing. You've got this: results = pd.DataFrame() for counter, current_file in enumerate(glob.glob("results_*.log")): gcmcdatadf = pd.read_csv(current_file, header=2, sep=" ", usecols=[1, 2, 3]) print(gcmcdatadf) results = pd.concat([results, gcmcdatadf]) #results.to_csv('resuls_tot.log', header=None, sep=" ") You're printing `gcmcdatadf`: Empty DataFrame Columns: [182.244, 10, 0.796176] Index: [] Empty DataFrame Columns: [181.126, 12.5, 0.995821] Index: [] It looks to me like it is getting the column names from the wrong row of data. You have supplied `header=2`, which says that row 2 contains the column names. Rows count from 0 in this context; maybe you want `header=1`? Have a look at one of the log files to check. Also, are there data lines in these files? Just wondering, since it says "Empty Dataframe". The docs for `read_csv()` are here: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html#pandas.read_csv You've also commented out: #results.to_csv('resuls_tot.log', header=None, sep=" ") which means you're not printing the concatenated results to a file. Also note that your final output is `'resuls_tot.log'` (which looks misspelled). This is in the same directory as the other `"results_*.log"` files which means that if you run this again you will pick up the concatenated results along with the original files. I'd give it another name which will not match your pattern, such as `"all_results.log"`. Cheers, Cameron Simpson From ml_news at posteo.de Sun Mar 27 05:24:30 2022 From: ml_news at posteo.de (Manfred Lotz) Date: Sun, 27 Mar 2022 11:24:30 +0200 Subject: How to detect an undefined method? Message-ID: Let's say I have a Python app and have used an undefined method somewhere. Let us further assume I have not detected it thru my tests. Is there a way to detect it before deploying the app? pylint doesn't notice it. Minimal example: #!/usr/bin/env python3 import logging from logging import Logger from random import randrange def main(): """ Below logger.err gives 'Logger' object has no attribute 'err' """ logger = logging.getLogger('sample') logger.setLevel(logging.DEBUG) handler = logging.StreamHandler() logger.addHandler(handler) num = randrange(0,1000) if num == 0: logger.err("got zero") else: logger.info(f'got a positive integer: {num}') if __name__ == "__main__": main() -- Manfred From PythonList at DancesWithMice.info Sun Mar 27 10:59:55 2022 From: PythonList at DancesWithMice.info (dn) Date: Mon, 28 Mar 2022 03:59:55 +1300 Subject: How to detect an undefined method? In-Reply-To: References: Message-ID: On 27/03/2022 22.24, Manfred Lotz wrote: > Let's say I have a Python app and have used an undefined method somewhere. Let > us further assume I have not detected it thru my tests. > > Is there a way to detect it before deploying the app? pylint doesn't notice it. A competent IDE will 'find' such a fault, eg PyCharm indicates the usage of an identifier for which it can't find a definition, by displaying a wavy/colored line beneath it - but it won't complain or stop us from running such code! Similarly, the interpreter will not object to running the code, until that line, that branch of logic, is actually executed! The answer may be to improve the testing regime, to ensure that the tests cover every line of logic in the code. Much of the time this is a reasonable target. Sometimes it is too 'expensive'. The following article includes both pros and cons of using coverage.py: How to use code coverage in Python with pytest? April 11, 2021 Sebastian python, testing software Basics What is code coverage? In the simplest words, code coverage is a measure of exhaustiveness of a test suite. 100% code coverage means that a system is fully tested. ... https://breadcrumbscollector.tech/how-to-use-code-coverage-in-python-with-pytest/ -- Regards, =dn From __peter__ at web.de Sun Mar 27 12:12:35 2022 From: __peter__ at web.de (Peter Otten) Date: Sun, 27 Mar 2022 18:12:35 +0200 Subject: How to detect an undefined method? In-Reply-To: References: Message-ID: <7c7fac94-e340-3c8c-de6f-1e6e3c751942@web.de> On 27/03/2022 11:24, Manfred Lotz wrote: > Let's say I have a Python app and have used an undefined method somewhere. Let > us further assume I have not detected it thru my tests. > > Is there a way to detect it before deploying the app? pylint doesn't notice it. > > > Minimal example: > > #!/usr/bin/env python3 > > import logging > from logging import Logger > from random import randrange > > def main(): > """ > Below logger.err gives > > 'Logger' object has no attribute 'err' > """ > > logger = logging.getLogger('sample') > logger.setLevel(logging.DEBUG) > handler = logging.StreamHandler() > logger.addHandler(handler) > > num = randrange(0,1000) > if num == 0: > logger.err("got zero") > else: > logger.info(f'got a positive integer: {num}') > > if __name__ == "__main__": > main() mypy --strict will find that one. Be warned though that this is a slippery slope: you may end up fixing quite a few non-errors... From skip.montanaro at gmail.com Sun Mar 27 12:36:01 2022 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Sun, 27 Mar 2022 11:36:01 -0500 Subject: How to detect an undefined method? In-Reply-To: References: Message-ID: > Let's say I have a Python app and have used an undefined method somewhere. > Let > us further assume I have not detected it thru my tests. > > Is there a way to detect it before deploying the app? pylint doesn't > notice it. > This is maybe not exactly what you're looking for, but writing a test to exercise that function call (if possible) would be top of my list. Next on my list would be to use coverage to get a good idea of what code is not tested. Writing more test cases to reduce the number of uncovered lines will also make it easier to manually examine the rest of your (untested) code to find calls to missing functions or methods. Skip > From kirill.ratkin at devoteam.com Sun Mar 27 12:57:38 2022 From: kirill.ratkin at devoteam.com (Kirill Ratkin) Date: Sun, 27 Mar 2022 19:57:38 +0300 Subject: How to detect an undefined method? In-Reply-To: References: Message-ID: <9d5c3247-b06a-7cb5-894c-64b38218ae05@devoteam.com> Hi You can get all methods of your object and check the method you want to call is there or not. |methods = [method for method in dir() if callable(getattr(, method))] if 'method_you_need' in methods: . // BR | 27.03.2022 12:24, Manfred Lotz ?????: > Let's say I have a Python app and have used an undefined method somewhere. Let > us further assume I have not detected it thru my tests. > > Is there a way to detect it before deploying the app? pylint doesn't notice it. > > > Minimal example: > > #!/usr/bin/env python3 > > import logging > from logging import Logger > from random import randrange > > def main(): > """ > Below logger.err gives > > 'Logger' object has no attribute 'err' > """ > > logger = logging.getLogger('sample') > logger.setLevel(logging.DEBUG) > handler = logging.StreamHandler() > logger.addHandler(handler) > > num = randrange(0,1000) > if num == 0: > logger.err("got zero") > else: > logger.info(f'got a positive integer: {num}') > > if __name__ == "__main__": > main() > > > From ml_news at posteo.de Sun Mar 27 12:47:06 2022 From: ml_news at posteo.de (Manfred Lotz) Date: Sun, 27 Mar 2022 18:47:06 +0200 Subject: How to detect an undefined method? In-Reply-To: References: <7c7fac94-e340-3c8c-de6f-1e6e3c751942@web.de> Message-ID: On 3/27/22 18:12, Peter Otten wrote: > On 27/03/2022 11:24, Manfred Lotz wrote: >> Let's say I have a Python app and have used an undefined method somewhere. Let >> us further assume I have not detected it thru my tests. >> >> Is there a way to detect it before deploying the app? pylint doesn't notice it. >> >> >> Minimal example: >> >> #!/usr/bin/env python3 >> >> import logging >> from logging import Logger >> from random import randrange >> >> def main(): >> ???? """ >> ???? Below logger.err gives >> >> ???? 'Logger' object has no attribute 'err' >> ???? """ >> >> ???? logger = logging.getLogger('sample') >> ???? logger.setLevel(logging.DEBUG) >> ???? handler = logging.StreamHandler() >> ???? logger.addHandler(handler) >> >> ???? num = randrange(0,1000) >> ???? if num == 0: >> ???????? logger.err("got zero") >> ???? else: >> ???????? logger.info(f'got a positive integer: {num}') >> >> if __name__ == "__main__": >> ???? main() > > mypy --strict will find that one. Be warned though that this is a > slippery slope: you may end up fixing quite a few non-errors... > Great. I wasn't aware of --strict. Regarding 'slippery slope': I anyway would only fix what I believe is an error. Thanks. -- Manfred From ml_news at posteo.de Sun Mar 27 12:48:53 2022 From: ml_news at posteo.de (Manfred Lotz) Date: Sun, 27 Mar 2022 18:48:53 +0200 Subject: How to detect an undefined method? In-Reply-To: References: Message-ID: On 3/27/22 18:36, Skip Montanaro wrote: >> Let's say I have a Python app and have used an undefined method somewhere. >> Let >> us further assume I have not detected it thru my tests. >> >> Is there a way to detect it before deploying the app? pylint doesn't >> notice it. >> > > This is maybe not exactly what you're looking for, but writing a test to > exercise that function call (if possible) would be top of my list. Next on > my list would be to use coverage to get a good idea of what code is not > tested. Writing more test cases to reduce the number of uncovered lines > will also make it easier to manually examine the rest of your (untested) code > to find calls to missing functions or methods. > If the code base is large it might be difficult to have a test case for all things. But I agree that coverage is surely a good thing to use. -- Manfred From ml_news at posteo.de Sun Mar 27 13:07:35 2022 From: ml_news at posteo.de (Manfred Lotz) Date: Sun, 27 Mar 2022 19:07:35 +0200 Subject: How to detect an undefined method? In-Reply-To: References: <9d5c3247-b06a-7cb5-894c-64b38218ae05@devoteam.com> Message-ID: On 3/27/22 18:57, Kirill Ratkin wrote: > Hi > > You can get all methods of your object and check the method you want to call is > there or not. > > |methods = [method for method in dir() if > callable(getattr(, method))] if 'method_you_need' in methods: > . // BR | > I don't understand how this may help. Assume somebody has a codebase of 15T lines of Python code. How do you want to apply your method? But perhaps I overlook things. -- Manfred > 27.03.2022 12:24, Manfred Lotz ?????: >> Let's say I have a Python app and have used an undefined method somewhere. Let >> us further assume I have not detected it thru my tests. >> >> Is there a way to detect it before deploying the app? pylint doesn't notice it. >> >> >> Minimal example: >> >> #!/usr/bin/env python3 >> >> import logging >> from logging import Logger >> from random import randrange >> >> def main(): >> ???? """ >> ???? Below logger.err gives >> >> ???? 'Logger' object has no attribute 'err' >> ???? """ >> >> ???? logger = logging.getLogger('sample') >> ???? logger.setLevel(logging.DEBUG) >> ???? handler = logging.StreamHandler() >> ???? logger.addHandler(handler) >> >> ???? num = randrange(0,1000) >> ???? if num == 0: >> ???????? logger.err("got zero") >> ???? else: >> ???????? logger.info(f'got a positive integer: {num}') >> >> if __name__ == "__main__": >> ???? main() >> >> >> From kirill.ratkin at devoteam.com Sun Mar 27 14:29:31 2022 From: kirill.ratkin at devoteam.com (Kirill Ratkin) Date: Sun, 27 Mar 2022 21:29:31 +0300 Subject: How to detect an undefined method? In-Reply-To: References: <9d5c3247-b06a-7cb5-894c-64b38218ae05@devoteam.com> Message-ID: <903038a3-ee86-c3af-e9a9-19f4260aeb7b@devoteam.com> I just started to think from your example with method 'err' of logger object. In this particular case you can check method 'err' exists or not before call this. But if you mean general case ... . If for example I use some library which uses another library and someone just 'typo' there ... Here is example from my code: ??? calc: Calculator = Calculator() ??? ... ??? actions: Dict[str, Callable] = { ?????????????????? "s.sysinfo":????? calc.get_system_info, ?????????????????? "s.setloglevel":? calc.set_log_level, ?????????????????? ... ??????????????? } ??? ... ??? def do_action(action: str, params: Dict[str, str]) -> ActionResult: ??????? return await actions[action](params) And if I make mistake and type 'calc.get_s*i*stem_info' instead 'calc.get_s*y*stem_info. Error appears on rutime stage only. And neither 'mypy --strict' or 'pyre' can find such error. I guess there? is not warranty to detect such sitations in huge codebase. It's python dynamic nature. May be dynamic checkers can help in such situations ... 27.03.2022 20:07, Manfred Lotz ?????: > On 3/27/22 18:57, Kirill Ratkin wrote: >> Hi >> >> You can get all methods of your object and check the method you want to call is >> there or not. >> >> |methods = [method for method in dir() if >> callable(getattr(, method))] if 'method_you_need' in methods: >> . // BR | >> > I don't understand how this may help. Assume somebody has a codebase of 15T > lines of Python code. How do you want to apply your method? > > But perhaps I overlook things. > From avigross at verizon.net Sun Mar 27 16:02:21 2022 From: avigross at verizon.net (Avi Gross) Date: Sun, 27 Mar 2022 20:02:21 +0000 (UTC) Subject: How to detect an undefined method? In-Reply-To: <903038a3-ee86-c3af-e9a9-19f4260aeb7b@devoteam.com> References: <9d5c3247-b06a-7cb5-894c-64b38218ae05@devoteam.com> <903038a3-ee86-c3af-e9a9-19f4260aeb7b@devoteam.com> Message-ID: <1924483235.446772.1648411341452@mail.yahoo.com> The question seems to be how or whether you can check Python code in advance for any instances of a method for an object being called that is not instantiated. Right? As Kirill points out, Python can be quite dynamic. I can think of oodles of ways checking would not work well in a static examination of the code. Just to mention a few, I can have a collection of objects (such as a list) and I may iterate on it to take each object and call a specific method. Some objects may have the method and others may not. The dynamic code may easily include objects that conform to expectations or not. For example, you ay be reading in saved objects from a file or objects are being created as the user interacts. And is it an error if a program is written to detect and perhaps correct errors? What if I want to display an object and in a "try" block I ask to run one of several methods and on failure, try the next. Some objects may have a method for full_profile() while others just might have a name() and yet others might have no known way and will be shown as ANONYMOUS. There also seem to be ways to extend an existing object or the entire class after it has been created. Objects with multiple inheritance also may be headache. So I suspect warnings for some cases make sense but a tool that catches everything, maybe not easy or even possible. You can, of course, make your code a tad more bulletproof, there are ways you can have your code deliberately check if the object has that method before trying to invoke it, or you can handle exceptions generated if it doesn't. -----Original Message----- From: Kirill Ratkin via Python-list To: python-list at python.org Sent: Sun, Mar 27, 2022 2:29 pm Subject: Re: How to detect an undefined method? I just started to think from your example with method 'err' of logger object. In this particular case you can check method 'err' exists or not before call this. But if you mean general case ... . If for example I use some library which uses another library and someone just 'typo' there ... Here is example from my code: ??? calc: Calculator = Calculator() ??? ... ??? actions: Dict[str, Callable] = { ?????????????????? "s.sysinfo":????? calc.get_system_info, ?????????????????? "s.setloglevel":? calc.set_log_level, ?????????????????? ... ??????????????? } ??? ... ??? def do_action(action: str, params: Dict[str, str]) -> ActionResult: ??????? return await actions[action](params) And if I make mistake and type 'calc.get_s*i*stem_info' instead 'calc.get_s*y*stem_info. Error appears on rutime stage only. And neither 'mypy --strict' or 'pyre' can find such error. I guess there? is not warranty to detect such sitations in huge codebase. It's python dynamic nature. May be dynamic checkers can help in such situations ... 27.03.2022 20:07, Manfred Lotz ?????: > On 3/27/22 18:57, Kirill Ratkin wrote: >> Hi >> >> You can get all methods of your object and check the method you want to call is >> there or not. >> >> |methods = [method for method in dir() if >> callable(getattr(, method))] if 'method_you_need' in methods: >> . // BR | >> > I don't understand how this may help. Assume somebody has a codebase of 15T > lines of Python code. How do you want to apply your method? > > But perhaps I overlook things. > -- https://mail.python.org/mailman/listinfo/python-list From skip.montanaro at gmail.com Sun Mar 27 17:23:01 2022 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Sun, 27 Mar 2022 16:23:01 -0500 Subject: Set tkinter top-level window to "always on visible workspace" Message-ID: I have a tkinter app (Ubuntu/X11 env) whose main window should always be displayed on the currently visible workspace. Is there some way to set that attribute programmatically? I see that there is a tkinter.Wm class and that Toplevel widgets have a wm_attributes method, but haven't found any examples in the library doc or on the wider net which demonstrate control of this particular window manager interaction. (I don't care about Windows or Mac, at least for the time being.) Thx, Skip From cs at cskk.id.au Sun Mar 27 18:07:49 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Mon, 28 Mar 2022 09:07:49 +1100 Subject: Set tkinter top-level window to "always on visible workspace" In-Reply-To: References: Message-ID: On 27Mar2022 16:23, Skip Montanaro wrote: >I have a tkinter app (Ubuntu/X11 env) whose main window should always be >displayed on the currently visible workspace. Is there some way to set that >attribute programmatically? I see that there is a tkinter.Wm class and that >Toplevel widgets have a wm_attributes method, but haven't found any >examples in the library doc or on the wider net which demonstrate control >of this particular window manager interaction. I fear not. That is normally a window manager-side control. So you might tell your window manager to keep that window on the main workspace. I might be wrong - there might be wm_attributes which are meant to request this behaviour (from the window manager), But I'm pretty sure when I was using FVWM I did this on the window manager side, saying "these apps should be on desktop N" and stuff like that. >(I don't care about Windows >or Mac, at least for the time being.) I think this is window manager stuff on the Mac too, in that you can tell the desktop to put an app's windows on every desktop, of a specific desktop, or just wherever they get opened. Hmm... Yeah, here's part of my FVWM config: Style "sticky" Sticky Style "topsticky" UseStyle sticky, StaysOnTop, WindowListSkip Style "bottomsticky" UseStyle sticky, StaysOnBottom Style "X2x" UseStyle topsticky Style "gkrellm" UseStyle bottomsticky which uses the application names to specify how the WM treats them. The two examples are gkrellm, a load monitor, to be always on the current workspace, below/behind everything else, and x2x to be on every workspace, above everything else. (X2x runs a 1-pixel-wide full height window at the edge of the display to notice the mouse hit that edge, and to take it and drive another X11 display - thus allowing one to slide the mouse from the primary display to the adjacent display run by another computer; see also x2vnc et al). But alas, these are WM-side controls. But maybe the WM might know about and honour some wm_attributes? I have no knowledge there. Cheers, Cameron Simpson From skip.montanaro at gmail.com Sun Mar 27 18:55:02 2022 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Sun, 27 Mar 2022 17:55:02 -0500 Subject: Set tkinter top-level window to "always on visible workspace" In-Reply-To: References: Message-ID: > So you might tell your window manager to keep that window on the main workspace. Thanks. I'd forgotten about the possibility of doing this sort of thing in the window manager config. That would certainly be fine in this case. (It's been ages since I messed with this sort of thing.) Skip From auriocus at gmx.de Mon Mar 28 01:16:10 2022 From: auriocus at gmx.de (Christian Gollwitzer) Date: Mon, 28 Mar 2022 07:16:10 +0200 Subject: Set tkinter top-level window to "always on visible workspace" In-Reply-To: References: Message-ID: Am 28.03.22 um 00:55 schrieb Skip Montanaro: >> So you might tell your window manager to keep that window on the main > workspace. > > Thanks. I'd forgotten about the possibility of doing this sort of thing in > the window manager config. That would certainly be fine in this case. (It's > been ages since I messed with this sort of thing.) I might be misguided, but on modern desktops that should be possible with a few mouseclicks. E.g. in KDE, there is a little pin icon displayed in every title bar of a toplevel window on the left side. If you click it, the window is shown on every workspace. There is also a way to set properties for a window permanently, by right-clicking on the title bar and then "enhanced attributes" (or similar, don't have KDE to check it roght now here) which gives a menu with multiple options to force the geometry. I don't know for GNOME desktop, but suspect it should be similarly easy. Christian From kristinerabia89 at gmail.com Mon Mar 28 08:06:30 2022 From: kristinerabia89 at gmail.com (kristine RABIA) Date: Mon, 28 Mar 2022 14:06:30 +0200 Subject: Unable to open Python Message-ID: Dear Python team I downloaded successfully Python, however when I am trying to open it, brings the window repair, modify or uninstall, I tried to click on repair and modify after all it came with the same window. Please advise the further step. Thanks Kristine From Cecil at decebal.nl Mon Mar 28 09:35:07 2022 From: Cecil at decebal.nl (Cecil Westerhof) Date: Mon, 28 Mar 2022 15:35:07 +0200 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> <877d8xnej9.fsf@hornfels.zedat.fu-berlin.de> Message-ID: <87pmm6kxo4.fsf@munus.decebal.nl> "Loris Bennett" writes: > Marco Sulla writes: > >> On Fri, 11 Mar 2022 at 19:10, Michael Torrie wrote: >>> Both Debian stable and Ubuntu LTS state they have a five year support >>> life cycle. >> >> Yes, but it seems that official security support in Debian ends after >> three years: >> >> "Debian LTS is not handled by the Debian security team, but by a >> separate group of volunteers and companies interested in making it a >> success" >> https://wiki.debian.org/LTS >> >> This is the only problem for me. > > I am not sure how different the two situations are. Ubuntu is > presumably relying on the Debian security team as well as other > volunteers and at least one company, namely Canonical. Nope. One important reason that I really hate that people use Ubuntu for servers is that Ubuntu wants to be up to date. So Ubuntu starts very close to Debian security wise, but will shift rapidly. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From rshepard at appl-ecosys.com Mon Mar 28 11:10:45 2022 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Mon, 28 Mar 2022 08:10:45 -0700 (PDT) Subject: Unable to open Python In-Reply-To: References: Message-ID: On Mon, 28 Mar 2022, kristine RABIA wrote: > I downloaded successfully Python, however when I am trying to open it, > brings the window repair, modify or uninstall, I tried to click on repair > and modify after all it came with the same window. Please advise the > further step. Kristine, What operating system do you use? Python is a programming language, not an application. It is not 'opened' but used to do a particular job. You use the language to tell the computer what to do. What do you want to do with Python? Rich From skip.montanaro at gmail.com Mon Mar 28 11:54:17 2022 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Mon, 28 Mar 2022 10:54:17 -0500 Subject: Set tkinter top-level window to "always on visible workspace" In-Reply-To: References: Message-ID: > I might be misguided, but on modern desktops that should be possible > with a few mouseclicks. E.g. in KDE, there is a little pin icon > displayed in every title bar of a toplevel window on the left side. Correct, and that's what I'm currently doing. I'm lazy though. I want the program to start always visible on the active virtual desktop. As Cameron showed, this is possible. I have vague recollection of doing this with the same program probably 15-20 years ago in an age when window managers weren't largely configured with the mouse. Unfortunately, I no longer recall what window manager(s) I used at the time (probably twm or fvwm). Now I use fvwm4 and can't find squat online about configuration files. I do have a ~/.config/xfce4/xfwm4/ directory, but it is completely empty. Skip From grant.b.edwards at gmail.com Mon Mar 28 11:56:05 2022 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Mon, 28 Mar 2022 08:56:05 -0700 (PDT) Subject: Unable to open Python References: Message-ID: <6241da95.1c69fb81.48e81.fb63@mx.google.com> On 2022-03-28, kristine RABIA wrote: > I downloaded successfully Python, however when I am trying to open it, > brings the window repair, modify or uninstall, I tried to click on repair > and modify after all it came with the same window. I assume you're using Windows? You're re-running the installer. That's just reinstalling/repairing Python. If you want to run Python, open a command prompt (cmd.exe), and enter "python". From grant.b.edwards at gmail.com Mon Mar 28 12:02:21 2022 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Mon, 28 Mar 2022 09:02:21 -0700 (PDT) Subject: Unable to open Python References: <6241da95.1c69fb81.48e81.fb63@mx.google.com> Message-ID: <6241dc0d.1c69fb81.5ca98.9a89@mx.google.com> On 2022-03-28, Grant Edwards wrote: > On 2022-03-28, kristine RABIA wrote: > >> I downloaded successfully Python, however when I am trying to open it, >> brings the window repair, modify or uninstall, I tried to click on repair >> and modify after all it came with the same window. > > I assume you're using Windows? > > You're re-running the installer. That's just reinstalling/repairing Python. > > If you want to run Python, open a command prompt (cmd.exe), and enter "python". That might not work if you didn't check the box during installation that causes python to be added to your PATH environment variable. There should be a python entry in your start menu. From mats at wichmann.us Mon Mar 28 13:09:14 2022 From: mats at wichmann.us (Mats Wichmann) Date: Mon, 28 Mar 2022 11:09:14 -0600 Subject: Unable to open Python In-Reply-To: <6241dc0d.1c69fb81.5ca98.9a89@mx.google.com> References: <6241da95.1c69fb81.48e81.fb63@mx.google.com> <6241dc0d.1c69fb81.5ca98.9a89@mx.google.com> Message-ID: <7c1705a7-05be-8604-1c6c-320da8cfe048@wichmann.us> On 3/28/22 10:02, Grant Edwards wrote: > On 2022-03-28, Grant Edwards wrote: >> On 2022-03-28, kristine RABIA wrote: >> >>> I downloaded successfully Python, however when I am trying to open it, >>> brings the window repair, modify or uninstall, I tried to click on repair >>> and modify after all it came with the same window. >> >> I assume you're using Windows? >> >> You're re-running the installer. That's just reinstalling/repairing Python. >> >> If you want to run Python, open a command prompt (cmd.exe), and enter "python". > > That might not work if you didn't check the box during installation > that causes python to be added to your PATH environment variable. > > There should be a python entry in your start menu. Get rid of the installer after using it, it seems to be able to confuse Windows in some senses, and read this: https://docs.python.org/3/using/windows.html From rosuav at gmail.com Mon Mar 28 14:03:39 2022 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 29 Mar 2022 05:03:39 +1100 Subject: Set tkinter top-level window to "always on visible workspace" In-Reply-To: References: Message-ID: On Tue, 29 Mar 2022 at 02:56, Skip Montanaro wrote: > > > I might be misguided, but on modern desktops that should be possible > > with a few mouseclicks. E.g. in KDE, there is a little pin icon > > displayed in every title bar of a toplevel window on the left side. > > Correct, and that's what I'm currently doing. I'm lazy though. I want > the program to start always visible on the active virtual desktop. As > Cameron showed, this is possible. I have vague recollection of doing > this with the same program probably 15-20 years ago in an age when > window managers weren't largely configured with the mouse. > Unfortunately, I no longer recall what window manager(s) I used at the > time (probably twm or fvwm). Now I use fvwm4 and can't find squat > online about configuration files. I do have a ~/.config/xfce4/xfwm4/ > directory, but it is completely empty. > Would you accept a solution that involves a subprocess call? wmctrl -ir {id} -b add,sticky Now, the only problem is... figuring out your window ID. Worst case, parse wmctrl -lG to get that info, but it might be possible to get the window ID from Tkinter itself. ChrisA From auriocus at gmx.de Mon Mar 28 14:51:30 2022 From: auriocus at gmx.de (Christian Gollwitzer) Date: Mon, 28 Mar 2022 20:51:30 +0200 Subject: Set tkinter top-level window to "always on visible workspace" In-Reply-To: References: Message-ID: Am 28.03.22 um 20:03 schrieb Chris Angelico: > Would you accept a solution that involves a subprocess call? > > wmctrl -ir {id} -b add,sticky > > Now, the only problem is... figuring out your window ID. Worst case, > parse wmctrl -lG to get that info, but it might be possible to get the > window ID from Tkinter itself. Sure: Call "winfo_id()" on the toplevel. You might want to reformat in it in hex format, which is the usual way to pass these IDs around. Tk actually returns it in hex format, but Tkinter reformats it as an integer. Christian From rosuav at gmail.com Mon Mar 28 15:10:39 2022 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 29 Mar 2022 06:10:39 +1100 Subject: Set tkinter top-level window to "always on visible workspace" In-Reply-To: References: Message-ID: On Tue, 29 Mar 2022 at 06:08, Christian Gollwitzer wrote: > > Am 28.03.22 um 20:03 schrieb Chris Angelico: > > Would you accept a solution that involves a subprocess call? > > > > wmctrl -ir {id} -b add,sticky > > > > Now, the only problem is... figuring out your window ID. Worst case, > > parse wmctrl -lG to get that info, but it might be possible to get the > > window ID from Tkinter itself. > > Sure: Call "winfo_id()" on the toplevel. You might want to reformat in > it in hex format, which is the usual way to pass these IDs around. Tk > actually returns it in hex format, but Tkinter reformats it as an integer. > Ah sweet, there you go then. (As you can see, I don't use Tkinter much.) I have no idea how wmctrl does its work, but if calling on another process is viable, that's at least a fully automated way to stickify the window. ChrisA From cs at cskk.id.au Mon Mar 28 18:02:52 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Tue, 29 Mar 2022 09:02:52 +1100 Subject: Set tkinter top-level window to "always on visible workspace" In-Reply-To: References: Message-ID: On 29Mar2022 06:10, Chris Angelico wrote: >On Tue, 29 Mar 2022 at 06:08, Christian Gollwitzer wrote: >> Am 28.03.22 um 20:03 schrieb Chris Angelico: >> > Would you accept a solution that involves a subprocess call? >> > >> > wmctrl -ir {id} -b add,sticky >> > >> > Now, the only problem is... figuring out your window ID. Worst case, >> > parse wmctrl -lG to get that info, but it might be possible to get the >> > window ID from Tkinter itself. >> >> Sure: Call "winfo_id()" on the toplevel. You might want to reformat in >> it in hex format, which is the usual way to pass these IDs around. Tk >> actually returns it in hex format, but Tkinter reformats it as an integer. >> > >Ah sweet, there you go then. (As you can see, I don't use Tkinter >much.) I have no idea how wmctrl does its work, It sets properties on the window itself. A window manager can listen for such changes and honour the settings. Cheers, Cameron Simpson From rosuav at gmail.com Mon Mar 28 18:05:51 2022 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 29 Mar 2022 09:05:51 +1100 Subject: Set tkinter top-level window to "always on visible workspace" In-Reply-To: References: Message-ID: On Tue, 29 Mar 2022 at 09:04, Cameron Simpson wrote: > > On 29Mar2022 06:10, Chris Angelico wrote: > >On Tue, 29 Mar 2022 at 06:08, Christian Gollwitzer wrote: > >> Am 28.03.22 um 20:03 schrieb Chris Angelico: > >> > Would you accept a solution that involves a subprocess call? > >> > > >> > wmctrl -ir {id} -b add,sticky > >> > > >> > Now, the only problem is... figuring out your window ID. Worst case, > >> > parse wmctrl -lG to get that info, but it might be possible to get the > >> > window ID from Tkinter itself. > >> > >> Sure: Call "winfo_id()" on the toplevel. You might want to reformat in > >> it in hex format, which is the usual way to pass these IDs around. Tk > >> actually returns it in hex format, but Tkinter reformats it as an integer. > >> > > > >Ah sweet, there you go then. (As you can see, I don't use Tkinter > >much.) I have no idea how wmctrl does its work, > > It sets properties on the window itself. A window manager can listen for > such changes and honour the settings. > Yeah but what I mean is, I don't know how to replicate its behaviour. Though I could, of course, just go read the source code. Sometimes it turns out that it really isn't that hard to replicate (as I discovered when I browsed the source code for "tail -F" recently - it's just a couple of inotify calls, way simpler than I expected). ChrisA From cs at cskk.id.au Mon Mar 28 18:06:23 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Tue, 29 Mar 2022 09:06:23 +1100 Subject: Set tkinter top-level window to "always on visible workspace" In-Reply-To: References: Message-ID: On 28Mar2022 10:54, Skip Montanaro wrote: >Unfortunately, I no longer recall what window manager(s) I used at the >time (probably twm or fvwm). Now I use fvwm4 and can't find squat >online about configuration files. I do have a ~/.config/xfce4/xfwm4/ >directory, but it is completely empty. I think you must be using xfce4, not fvwm4 (there's an fvwm3 in development). See https://xfce.org/ If I were using X11 I'd still be using fvwm2 or trying out 3, but my desktops have always be specially configured. Cheers, Cameron Simpson ... you could spend *all day* customizing the title bar. Believe me. I speak from experience. - Matt Welsh From cs at cskk.id.au Mon Mar 28 18:07:49 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Tue, 29 Mar 2022 09:07:49 +1100 Subject: Set tkinter top-level window to "always on visible workspace" In-Reply-To: References: Message-ID: On 28Mar2022 10:54, Skip Montanaro wrote: >I do have a ~/.config/xfce4/xfwm4/ >directory, but it is completely empty. On reflection, maybe you're running something else. "ps axf" FTW. Cheers, Cameron Simpson From hjp-python at hjp.at Mon Mar 28 18:03:02 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Tue, 29 Mar 2022 00:03:02 +0200 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: <87pmm6kxo4.fsf@munus.decebal.nl> References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> <877d8xnej9.fsf@hornfels.zedat.fu-berlin.de> <87pmm6kxo4.fsf@munus.decebal.nl> Message-ID: <20220328220302.n2pjqqbu7qf3aslf@hjp.at> On 2022-03-28 15:35:07 +0200, Cecil Westerhof via Python-list wrote: > "Loris Bennett" writes: > > Ubuntu is presumably relying on the Debian security team as well as > > other volunteers and at least one company, namely Canonical. > > Nope. One important reason that I really hate that people use Ubuntu > for servers is that Ubuntu wants to be up to date. Not sure what you mean by that. There is an Ubuntu LTS release every 2 years. There is also a Debian release roughly every 2 years (although not on quite as strict a schedule). So that's very similar. > So Ubuntu starts very close to Debian security wise, but will shift > rapidly. They are are about a year apart, so they will usually contain different versions of most packages right from the start. So the Ubuntu and Debian security teams probably can't benefit much from each other. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From skip.montanaro at gmail.com Mon Mar 28 20:39:27 2022 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Mon, 28 Mar 2022 19:39:27 -0500 Subject: Set tkinter top-level window to "always on visible workspace" In-Reply-To: References: Message-ID: > I think you must be using xfce4, not fvwm4 (there's an fvwm3 in > development). See https://xfce.org/ Both? % pgrep -fla xfwm4 1803 xfwm4 --replace 539426 xfwm4-settings % pgrep -fla xfce 1599 xfce4-session 1755 /usr/bin/ssh-agent /usr/bin/im-launch startxfce4 1782 /usr/lib/x86_64-linux-gnu/xfce4/xfconf/xfconfd ... I kind of assume xfce4 is the session manager sort of thing, while xfwm4 is the actual window manager. Skip From cs at cskk.id.au Mon Mar 28 20:47:02 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Tue, 29 Mar 2022 11:47:02 +1100 Subject: Set tkinter top-level window to "always on visible workspace" In-Reply-To: References: Message-ID: On 28Mar2022 19:39, Skip Montanaro wrote: >> I think you must be using xfce4, not fvwm4 (there's an fvwm3 in >> development). See https://xfce.org/ > >Both? [...] >1803 xfwm4 --replace >539426 xfwm4-settings [...] >1755 /usr/bin/ssh-agent /usr/bin/im-launch startxfce4 >1782 /usr/lib/x86_64-linux-gnu/xfce4/xfconf/xfconfd > >I kind of assume xfce4 is the session manager sort of thing, while >xfwm4 is the actual window manager. Sounds reasonable. I was more looking at your earlier "Now I use fvwm4" and pointing out that there is no fvwm4. These transcription issues brought to you by the letters W and M, and by the number 4. Cheers, Cameron Simpson From skip.montanaro at gmail.com Mon Mar 28 21:47:17 2022 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Mon, 28 Mar 2022 20:47:17 -0500 Subject: Set tkinter top-level window to "always on visible workspace" In-Reply-To: References: Message-ID: > Would you accept a solution that involves a subprocess call? > > wmctrl -ir {id} -b add,sticky I'm already checking idle time with xprintidle(1), so what's one more (one-time) subprocess call? Small amount of history, I'm referring to this mouse/typing watcher: https://github.com/smontanaro/python-bits/blob/master/src/watch.py I wrote this years and years ago when I first started having RSI problems. So long ago that I just deleted a commented out piece of code which was there in case I ever ran it with Python 1.4(!). Over the years I've needed it on Linux, Linux+WIndows, Mac, etc. I'd put it away for a couple years, then suffer a flare-up and pull it back out. I never thought I'd need it after I retired, but here I am again. I've tried all sorts of things to monitor mouse and keyboard activity. At the moment I'm just using a Linux laptop, so have tossed out any sort of attempted cross-platform tracking functionality. > Now, the only problem is... figuring out your window ID. Worst case, > parse wmctrl -lG to get that info, but it might be possible to get the > window ID from Tkinter itself. I'm struggling to get the outermost window (not sure what's going on), but I will keep messing around. Skip From cs at cskk.id.au Mon Mar 28 22:02:56 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Tue, 29 Mar 2022 13:02:56 +1100 Subject: Set tkinter top-level window to "always on visible workspace" In-Reply-To: References: Message-ID: On 28Mar2022 20:47, Skip Montanaro wrote: >I'm struggling to get the outermost window (not sure what's going on), >but >I will keep messing around. See what xlsclients says to you. Cheers, Cameron Simpson From cs at cskk.id.au Mon Mar 28 22:07:26 2022 From: cs at cskk.id.au (Cameron Simpson) Date: Tue, 29 Mar 2022 13:07:26 +1100 Subject: Set tkinter top-level window to "always on visible workspace" In-Reply-To: References: Message-ID: On 29Mar2022 13:02, Cameron Simpson wrote: >On 28Mar2022 20:47, Skip Montanaro wrote: >>I'm struggling to get the outermost window (not sure what's going on), >>but >>I will keep messing around. > >See what xlsclients says to you. Also, wmctrl accepts window names instead of ids if you omit the -i option. In case winfo_id() is not returning the id you need. Cheers, Cameron Simpson From alister.ware at ntlworld.com Tue Mar 29 15:26:03 2022 From: alister.ware at ntlworld.com (alister) Date: Tue, 29 Mar 2022 19:26:03 -0000 (UTC) Subject: Python Qualification? Message-ID: I'm currently considering a career change (not much choice actually just been made redundant). I'd like to be able to turn my interest in python to my advantage, What qualifications do employers look for? -- I'm reporting for duty as a modern person. I want to do the Latin Hustle now! From arj.python at gmail.com Tue Mar 29 16:05:47 2022 From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer) Date: Wed, 30 Mar 2022 00:05:47 +0400 Subject: Python Qualification? In-Reply-To: References: Message-ID: I'd say that you need: programming fundamentals common tools Python expertise As for Python expertise, you are expected to know Python fundamentals, a fair amount of the standard library, a nice level of popular fields (data science, web scraping, networking, async, etc) To specialize in one. I'd say that contributing to OpenSource helps a lot to sharpen the sense of knowing what you need exactly! Kind Regards, Abdur-Rahmaan Janhangeer about | blog github Mauritius On Tue, Mar 29, 2022 at 11:34 PM alister via Python-list < python-list at python.org> wrote: > I'm currently considering a career change (not much choice actually just > been made redundant). > I'd like to be able to turn my interest in python to my advantage, What > qualifications do employers look for? > > > > -- > I'm reporting for duty as a modern person. I want to do the Latin Hustle > now! > -- > https://mail.python.org/mailman/listinfo/python-list > From skip.montanaro at gmail.com Tue Mar 29 17:28:24 2022 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Tue, 29 Mar 2022 16:28:24 -0500 Subject: Set tkinter top-level window to "always on visible workspace" In-Reply-To: References: Message-ID: > Also, wmctrl accepts window names instead of ids if you omit the -i > option. In case winfo_id() is not returning the id you need. Nice. That worked. FWIW, Tk never gave me the very outermost window as the parent of the tkinter.Tk instance. Here's the start of hierarchy: % xwininfo -tree xwininfo: Please select the window about which you would like information by clicking the mouse in that window. xwininfo: Window id: 0x5a0001f "Typing Watcher" Root window id: 0x7a2 (the root window) (has no name) Parent window id: 0xc73880 (has no name) 1 child: 0x5a0001e (has no name): () 178x126+0+0 +2+43 1 child: 0x5a00020 (has no name): () 178x126+0+0 +2+43 5 children: ... Tk thought 0x5a0001e was the top, but as you can see 0x5a0001f is the named window. Calling wmctrl using "Typing Watcher" as the target window did the trick. Thanks, Skip From wlfraed at ix.netcom.com Tue Mar 29 20:49:53 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Tue, 29 Mar 2022 20:49:53 -0400 Subject: Python Qualification? References: Message-ID: On Tue, 29 Mar 2022 19:26:03 -0000 (UTC), alister declaimed the following: >I'm currently considering a career change (not much choice actually just >been made redundant). >I'd like to be able to turn my interest in python to my advantage, What >qualifications do employers look for? Strangely -- knowledge of Python was never a consideration in my history... Having a familiarity with multiple languages, software engineering principles, and requirements/design analysis were larger factors. Python was something I used in support of the primary task, but was not the end-product itself (for example, an evaluation of various secure network filtering hardware, by sending serial numbered packets out one NIC, through the filter, and in through a second NIC; capturing both out&in via Wireshark; later merging the two captures into a single file of time delays, and plotting the timing of the packets intended to pass through and verifying that "classified" contents were blocked or sanitized). -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From Marco.Sulla.Python at gmail.com Wed Mar 30 02:48:36 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Wed, 30 Mar 2022 08:48:36 +0200 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: <20220328220302.n2pjqqbu7qf3aslf@hjp.at> References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> <877d8xnej9.fsf@hornfels.zedat.fu-berlin.de> <87pmm6kxo4.fsf@munus.decebal.nl> <20220328220302.n2pjqqbu7qf3aslf@hjp.at> Message-ID: On Tue, 29 Mar 2022 at 00:10, Peter J. Holzer wrote: > They are are about a year apart, so they will usually contain different > versions of most packages right from the start. So the Ubuntu and Debian > security teams probably can't benefit much from each other. Are you sure? Since LTS of Debian and Ubuntu lasts 5 years, I suppose the versions of the packages should overlap at some point in the past. From hjp-python at hjp.at Wed Mar 30 04:17:44 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Wed, 30 Mar 2022 10:17:44 +0200 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: References: <877d8xnej9.fsf@hornfels.zedat.fu-berlin.de> <87pmm6kxo4.fsf@munus.decebal.nl> <20220328220302.n2pjqqbu7qf3aslf@hjp.at> Message-ID: <20220330081744.oj5copxjtjyfmp5k@hjp.at> On 2022-03-30 08:48:36 +0200, Marco Sulla wrote: > On Tue, 29 Mar 2022 at 00:10, Peter J. Holzer wrote: > > They are are about a year apart, so they will usually contain different > > versions of most packages right from the start. So the Ubuntu and Debian > > security teams probably can't benefit much from each other. > > Are you sure? Since LTS of Debian and Ubuntu lasts 5 years, I suppose > the versions of the packages should overlap at some point in the past. Standard policy (there are exceptions) on most distros is to stay with the same version of any package for the entire lifetime. So for example, Ubuntu 20.04 was released with Apache 2.4.41 and Python 3.8.10 and Debian 11 was released with Apache 2.4.53 and Python 3.9.2 and they are still on these versions. Any security fixes and other critical bug fixes were back-ported to these versions. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From barry at barrys-emacs.org Wed Mar 30 09:52:28 2022 From: barry at barrys-emacs.org (Barry) Date: Wed, 30 Mar 2022 14:52:28 +0100 Subject: Python Qualification? In-Reply-To: References: Message-ID: <16AB7749-0978-44F6-9DFE-54A868AFCC0A@barrys-emacs.org> > On 29 Mar 2022, at 20:37, alister via Python-list wrote: > > ?I'm currently considering a career change (not much choice actually just > been made redundant). > I'd like to be able to turn my interest in python to my advantage, What > qualifications do employers look for? Where I work we use python in the product and look for expertise in object oriented design and python expertise. But will take people with no python expertise if they are willing the learn python and have experience of other OO languages, C+c, Java, etc. We do not look for formal qualifications. Barry > > > > -- > I'm reporting for duty as a modern person. I want to do the Latin Hustle > now! > -- > https://mail.python.org/mailman/listinfo/python-list > From tedpfspencer at gmail.com Wed Mar 30 04:34:22 2022 From: tedpfspencer at gmail.com (Edward Spencer) Date: Wed, 30 Mar 2022 01:34:22 -0700 (PDT) Subject: Add a method to list the current named logging levels In-Reply-To: References: <24881.20924.937699.552812@ixdm.fritz.box> Message-ID: <4b5b58c8-b575-413f-91a3-1dcb628b5ff0n@googlegroups.com> ? 2021?9?3???? UTC+1 18:50:51? ??? > > On 2 Sep 2021, at 23:38, Dieter Maurer wrote: > > > > ?Edward Spencer wrote at 2021-9-2 10:02 -0700: > >> Sometimes I like to pass the logging level up to the command line params so my user can specific what level of logging they want. However there is no easy method for pulling the named logging level names. > >> > >> Looking into the code, it would actually be incredibly easy to implement; > >> > >> in `logging.__init__.py`; > >> > >> def listLevelNames(): > >> return _nameToLevel.keys() > >> > >> You could obviously add some other features, like listing only the defaults, sorted by numerical level or alphabetically, etc. But really this basic implementation would be enough to expose the internal variables which shouldn't be accessed because they change (and in fact, between python 2 and 3, they did). > >> > >> Any thoughts? > > > > Usually, you use 5 well known log levels: "DEBUG", "INFO", "WARNING", > > "ERROR" and "CRITICAL". > > No need to provide a special function listing those levels. > I add my own levels, but then I know I did it. > > Barry > > > > > > > > > -- > > Dieter > > -- > > https://mail.python.org/mailman/listinfo/python-list > > Yes, the names are already well defined. But every software project anyone has that needs to use logging then has to define that list, which is a waste of time since they're already defined inside the logging repo. But no-one can access them unless they use protected variables. If it's a case of not wanting users to be able to modify the defaults, then just define that list of log levels as a tuple. Hiding it is unnecessary. From alister.ware at ntlworld.com Wed Mar 30 04:34:55 2022 From: alister.ware at ntlworld.com (alister) Date: Wed, 30 Mar 2022 08:34:55 -0000 (UTC) Subject: Python Qualification? References: Message-ID: On Tue, 29 Mar 2022 20:49:53 -0400, Dennis Lee Bieber wrote: > On Tue, 29 Mar 2022 19:26:03 -0000 (UTC), alister > declaimed the following: > >>I'm currently considering a career change (not much choice actually just >>been made redundant). >>I'd like to be able to turn my interest in python to my advantage, What >>qualifications do employers look for? > > Strangely -- knowledge of Python was never a consideration in my > history... Having a familiarity with multiple languages, software > engineering principles, and requirements/design analysis were larger > factors. > > Python was something I used in support of the primary task, but was not > the end-product itself (for example, an evaluation of various secure > network filtering hardware, by sending serial numbered packets out one > NIC, > through the filter, and in through a second NIC; capturing both out&in > via Wireshark; later merging the two captures into a single file of time > delays, and plotting the timing of the packets intended to pass through > and verifying that "classified" contents were blocked or sanitized). Pretty much where I am (only less so) I have used python & flask to make a reasonably sucsessfull web app for my fellow engineers at work, lots of stuff with home Raspberry pi robotics but no formal qualifications in the field. -- Earth is a beta site. From barry at barrys-emacs.org Wed Mar 30 11:28:40 2022 From: barry at barrys-emacs.org (Barry) Date: Wed, 30 Mar 2022 16:28:40 +0100 Subject: Python Qualification? In-Reply-To: <02f9f0af-f43d-976c-e523-a9a03d8fa77a@ntlworld.com> References: <02f9f0af-f43d-976c-e523-a9a03d8fa77a@ntlworld.com> Message-ID: <4211CFA8-1FF5-4605-AF98-FA0D3734EF38@barrys-emacs.org> > On 30 Mar 2022, at 15:40, alister.ware wrote: > > ?OK > > where do you work & are there any vacancies (a junior position would suffice) At the moment we are only hiring in India and for senior rolls. Barry > >> On 30/03/2022 14:52, Barry wrote: >> >>>> On 29 Mar 2022, at 20:37, alister via Python-list wrote: >>> >>> ?I'm currently considering a career change (not much choice actually just >>> been made redundant). >>> I'd like to be able to turn my interest in python to my advantage, What >>> qualifications do employers look for? >> Where I work we use python in the product and look for expertise in >> object oriented design and python expertise. But will take people with >> no python expertise if they are willing the learn python and have experience >> of other OO languages, C+c, Java, etc. >> >> We do not look for formal qualifications. >> >> Barry >> >>> >>> >>> -- >>> I'm reporting for duty as a modern person. I want to do the Latin Hustle >>> now! >>> -- >>> https://mail.python.org/mailman/listinfo/python-list >>> > From barry at barrys-emacs.org Wed Mar 30 11:37:57 2022 From: barry at barrys-emacs.org (Barry) Date: Wed, 30 Mar 2022 16:37:57 +0100 Subject: Add a method to list the current named logging levels In-Reply-To: <4b5b58c8-b575-413f-91a3-1dcb628b5ff0n@googlegroups.com> References: <4b5b58c8-b575-413f-91a3-1dcb628b5ff0n@googlegroups.com> Message-ID: <63E210B9-F696-490E-A6AC-AABCAD93F9AA@barrys-emacs.org> > On 30 Mar 2022, at 16:11, Edward Spencer wrote: > > ?? 2021?9?3???? UTC+1 18:50:51? ??? >>>> On 2 Sep 2021, at 23:38, Dieter Maurer wrote: >>> >>> ?Edward Spencer wrote at 2021-9-2 10:02 -0700: >>>> Sometimes I like to pass the logging level up to the command line params so my user can specific what level of logging they want. However there is no easy method for pulling the named logging level names. >>>> >>>> Looking into the code, it would actually be incredibly easy to implement; >>>> >>>> in `logging.__init__.py`; >>>> >>>> def listLevelNames(): >>>> return _nameToLevel.keys() >>>> >>>> You could obviously add some other features, like listing only the defaults, sorted by numerical level or alphabetically, etc. But really this basic implementation would be enough to expose the internal variables which shouldn't be accessed because they change (and in fact, between python 2 and 3, they did). >>>> >>>> Any thoughts? >>> >>> Usually, you use 5 well known log levels: "DEBUG", "INFO", "WARNING", >>> "ERROR" and "CRITICAL". >>> No need to provide a special function listing those levels. >> I add my own levels, but then I know I did it. >> >> Barry >> >>> >>> >>> >>> -- >>> Dieter >>> -- >>> https://mail.python.org/mailman/listinfo/python-list >>> > > Yes, the names are already well defined. But every software project anyone has that needs to use logging then has to define that list, which is a waste of time since they're already defined inside the logging repo. But no-one can access them unless they use protected variables. If it's a case of not wanting users to be able to modify the defaults, then just define that list of log levels as a tuple. Hiding it is unnecessary. Is logging.getLevelNamesMapping() what you are looking for? Barry > -- > https://mail.python.org/mailman/listinfo/python-list From larry.martell at gmail.com Wed Mar 30 12:10:16 2022 From: larry.martell at gmail.com (Larry Martell) Date: Wed, 30 Mar 2022 12:10:16 -0400 Subject: calling a function asynchronously Message-ID: I have a django app, and for a certain request I need to kick off a long running task. I want to do this asynchronously and immediately return a response. I tried using subprocess.Process() but the forked process does not have a django database connection. I then tried posting a request using ajax but that does not have a django session so it's not authorized. Tried using asyncio but I am finding the caller of the async function does not return until the async function returns - maybe I am doing something wrong, as it does appear to be actually asynchronous. I tried this test: import asyncio import time async def long(): for i in range(100): time.sleep(10) asyncio.run(long()) print('after asyncio.run') The final print does not come out until after long() completes. Is there any way to do this? From kirill.ratkin at devoteam.com Wed Mar 30 12:27:54 2022 From: kirill.ratkin at devoteam.com (Kirill Ratkin) Date: Wed, 30 Mar 2022 19:27:54 +0300 Subject: calling a function asynchronously In-Reply-To: References: Message-ID: Hi, You can use asyncio.create_task and gather results. See docs - https://docs.python.org/3/library/asyncio-task.html But think twice what you want to do in async task. Do you use synchronous requests to database? If yes it will blocks eventloop... If you use Django it makes sense to use something like 'django-background-tasks' It just saves your time possibly. // BR ??, 30 ???. 2022 ?., 19:14 Larry Martell : > I have a django app, and for a certain request I need to kick off a > long running task. I want to do this asynchronously and immediately > return a response. I tried using subprocess.Process() but the forked > process does not have a django database connection. I then tried > posting a request using ajax but that does not have a django session > so it's not authorized. Tried using asyncio but I am finding the > caller of the async function does not return until the async function > returns - maybe I am doing something wrong, as it does appear to be > actually asynchronous. I tried this test: > > import asyncio > import time > > async def long(): > for i in range(100): > time.sleep(10) > > asyncio.run(long()) > print('after asyncio.run') > > The final print does not come out until after long() completes. > > Is there any way to do this? > -- > https://mail.python.org/mailman/listinfo/python-list > From tedpfspencer at gmail.com Wed Mar 30 12:11:22 2022 From: tedpfspencer at gmail.com (Edward Spencer) Date: Wed, 30 Mar 2022 09:11:22 -0700 (PDT) Subject: Add a method to list the current named logging levels In-Reply-To: References: <63E210B9-F696-490E-A6AC-AABCAD93F9AA@barrys-emacs.org> <4b5b58c8-b575-413f-91a3-1dcb628b5ff0n@googlegroups.com> Message-ID: <39ecba7c-3b5a-4b00-b0d2-3ba57ee320e8n@googlegroups.com> ? 2022?3?30???? UTC+1 16:38:26? ??? > > On 30 Mar 2022, at 16:11, Edward Spencer wrote: > > > > ?? 2021?9?3???? UTC+1 18:50:51? ??? > >>>> On 2 Sep 2021, at 23:38, Dieter Maurer wrote: > >>> > >>> ?Edward Spencer wrote at 2021-9-2 10:02 -0700: > >>>> Sometimes I like to pass the logging level up to the command line params so my user can specific what level of logging they want. However there is no easy method for pulling the named logging level names. > >>>> > >>>> Looking into the code, it would actually be incredibly easy to implement; > >>>> > >>>> in `logging.__init__.py`; > >>>> > >>>> def listLevelNames(): > >>>> return _nameToLevel.keys() > >>>> > >>>> You could obviously add some other features, like listing only the defaults, sorted by numerical level or alphabetically, etc. But really this basic implementation would be enough to expose the internal variables which shouldn't be accessed because they change (and in fact, between python 2 and 3, they did). > >>>> > >>>> Any thoughts? > >>> > >>> Usually, you use 5 well known log levels: "DEBUG", "INFO", "WARNING", > >>> "ERROR" and "CRITICAL". > >>> No need to provide a special function listing those levels. > >> I add my own levels, but then I know I did it. > >> > >> Barry > >> > >>> > >>> > >>> > >>> -- > >>> Dieter > >>> -- > >>> https://mail.python.org/mailman/listinfo/python-list > >>> > > > > Yes, the names are already well defined. But every software project anyone has that needs to use logging then has to define that list, which is a waste of time since they're already defined inside the logging repo. But no-one can access them unless they use protected variables. If it's a case of not wanting users to be able to modify the defaults, then just define that list of log levels as a tuple. Hiding it is unnecessary. > Is logging.getLevelNamesMapping() what you are looking for? > > Barry > > > > -- > > https://mail.python.org/mailman/listinfo/python-list Hi Barry, What version for python / logging are you seeing that method in? I don't appear to be able to find it. I vaguely remember seeing something similar to it though, did it return a dict of {: } only or did it also include the reverse of int to str? Thanks, Ed From rosuav at gmail.com Wed Mar 30 12:39:33 2022 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 31 Mar 2022 03:39:33 +1100 Subject: Add a method to list the current named logging levels In-Reply-To: <39ecba7c-3b5a-4b00-b0d2-3ba57ee320e8n@googlegroups.com> References: <63E210B9-F696-490E-A6AC-AABCAD93F9AA@barrys-emacs.org> <4b5b58c8-b575-413f-91a3-1dcb628b5ff0n@googlegroups.com> <39ecba7c-3b5a-4b00-b0d2-3ba57ee320e8n@googlegroups.com> Message-ID: On Thu, 31 Mar 2022 at 03:33, Edward Spencer wrote: > > ? 2022?3?30???? UTC+1 16:38:26? ??? > > > On 30 Mar 2022, at 16:11, Edward Spencer wrote: > > > > > > ?? 2021?9?3???? UTC+1 18:50:51? ??? > > >>>> On 2 Sep 2021, at 23:38, Dieter Maurer wrote: > > >>> > > >>> ?Edward Spencer wrote at 2021-9-2 10:02 -0700: > > >>>> Sometimes I like to pass the logging level up to the command line params so my user can specific what level of logging they want. However there is no easy method for pulling the named logging level names. > > >>>> > > >>>> Looking into the code, it would actually be incredibly easy to implement; > > >>>> > > >>>> in `logging.__init__.py`; > > >>>> > > >>>> def listLevelNames(): > > >>>> return _nameToLevel.keys() > > >>>> > > >>>> You could obviously add some other features, like listing only the defaults, sorted by numerical level or alphabetically, etc. But really this basic implementation would be enough to expose the internal variables which shouldn't be accessed because they change (and in fact, between python 2 and 3, they did). > > >>>> > > >>>> Any thoughts? > > >>> > > >>> Usually, you use 5 well known log levels: "DEBUG", "INFO", "WARNING", > > >>> "ERROR" and "CRITICAL". > > >>> No need to provide a special function listing those levels. > > >> I add my own levels, but then I know I did it. > > >> > > >> Barry > > >> > > >>> > > >>> > > >>> > > >>> -- > > >>> Dieter > > >>> -- > > >>> https://mail.python.org/mailman/listinfo/python-list > > >>> > > > > > > Yes, the names are already well defined. But every software project anyone has that needs to use logging then has to define that list, which is a waste of time since they're already defined inside the logging repo. But no-one can access them unless they use protected variables. If it's a case of not wanting users to be able to modify the defaults, then just define that list of log levels as a tuple. Hiding it is unnecessary. > > Is logging.getLevelNamesMapping() what you are looking for? > > > > Barry > > > > > > > -- > > > https://mail.python.org/mailman/listinfo/python-list > > Hi Barry, > > What version for python / logging are you seeing that method in? I don't appear to be able to find it. > I vaguely remember seeing something similar to it though, did it return a dict of {: } only or did it also include the reverse of int to str? > New in 3.11: https://docs.python.org/3.11/library/logging.html#logging.getLevelNamesMapping I'd say it's reasonable to use this, and then to backport it to older versions by monkeypatching it in (by referencing the internal). You potentially might have issues with other Python implementations, but I'm pretty sure CPython has logging._nameToLevel with the same semantics for quite a while. ChrisA From python.list at tim.thechases.com Wed Mar 30 12:48:12 2022 From: python.list at tim.thechases.com (Tim Chase) Date: Wed, 30 Mar 2022 11:48:12 -0500 Subject: Add a method to list the current named logging levels In-Reply-To: <63E210B9-F696-490E-A6AC-AABCAD93F9AA@barrys-emacs.org> References: <4b5b58c8-b575-413f-91a3-1dcb628b5ff0n@googlegroups.com> <63E210B9-F696-490E-A6AC-AABCAD93F9AA@barrys-emacs.org> Message-ID: <20220330114812.4ca5cbaa@bigbox.attlocal.net> On 2022-03-30 16:37, Barry wrote: > Is logging.getLevelNamesMapping() what you are looking for? Is this in some version newer than the 3.8 that comes stock on my machine? $ python3 -q >>> import logging >>> logging.getLevelNamesMapping() Traceback (most recent call last): File "", line 1, in AttributeError: module 'logging' has no attribute 'getLevelNamesMapping' -tkc From mats at wichmann.us Wed Mar 30 14:34:06 2022 From: mats at wichmann.us (Mats Wichmann) Date: Wed, 30 Mar 2022 12:34:06 -0600 Subject: Add a method to list the current named logging levels In-Reply-To: References: <63E210B9-F696-490E-A6AC-AABCAD93F9AA@barrys-emacs.org> <4b5b58c8-b575-413f-91a3-1dcb628b5ff0n@googlegroups.com> <39ecba7c-3b5a-4b00-b0d2-3ba57ee320e8n@googlegroups.com> Message-ID: <9c4999d0-48da-847b-5be5-23ddd8b1b42c@wichmann.us> On 3/30/22 10:39, Chris Angelico wrote: > New in 3.11: > > https://docs.python.org/3.11/library/logging.html#logging.getLevelNamesMapping > > I'd say it's reasonable to use this, and then to backport it to older > versions by monkeypatching it in (by referencing the internal). You > potentially might have issues with other Python implementations, but > I'm pretty sure CPython has logging._nameToLevel with the same > semantics for quite a while. since 2013, it looks like. From kirill.ratkin at devoteam.com Wed Mar 30 14:36:09 2022 From: kirill.ratkin at devoteam.com (Kirill Ratkin) Date: Wed, 30 Mar 2022 21:36:09 +0300 Subject: calling a function asynchronously In-Reply-To: References: Message-ID: <7e14172c-dd9e-1e5d-65ed-05c02eeedbf8@devoteam.com> Hi again, I changed a bit your example and it works as you expected I hope. import asyncio async def long(): ??? for i in range(100): ??????? await asyncio.sleep(10) ??????? print("long is done") loop = asyncio.get_event_loop() task = loop.create_task(long()) print('after asyncio.run') loop.run_until_complete(asyncio.gather(task)) But how I wrote before ... if you are in big Django project just look at existent django libraries for long task running. One of it I sent you. There is 'celery' as well. It's more pragmatic way ... 30.03.2022 19:10, Larry Martell ?????: > import asyncio > import time > > async def long(): > for i in range(100): > time.sleep(10) > > asyncio.run(long()) > print('after asyncio.run') From larry.martell at gmail.com Wed Mar 30 14:44:40 2022 From: larry.martell at gmail.com (Larry Martell) Date: Wed, 30 Mar 2022 14:44:40 -0400 Subject: calling a function asynchronously In-Reply-To: <7e14172c-dd9e-1e5d-65ed-05c02eeedbf8@devoteam.com> References: <7e14172c-dd9e-1e5d-65ed-05c02eeedbf8@devoteam.com> Message-ID: On Wed, Mar 30, 2022 at 2:40 PM Kirill Ratkin via Python-list wrote: > > Hi again, > > I changed a bit your example and it works as you expected I hope. > > import asyncio > > > async def long(): > for i in range(100): > await asyncio.sleep(10) > print("long is done") > > > loop = asyncio.get_event_loop() > > task = loop.create_task(long()) > print('after asyncio.run') > loop.run_until_complete(asyncio.gather(task)) > > > But how I wrote before ... if you are in big Django project just look at > existent django libraries for long task running. One of it I sent you. > There is 'celery' as well. > > It's more pragmatic way ... Appreciate the reply. I did not know about django-background-tasks - thanks. I've been trying to make use of that. I do not get the errors I was getting before but it does not appear that my long running task is running at all. Still debugging. But concerting asyncio - doesn't run_until_complete block until long() completes? > > 30.03.2022 19:10, Larry Martell ?????: > > import asyncio > > import time > > > > async def long(): > > for i in range(100): > > time.sleep(10) > > > > asyncio.run(long()) > > print('after asyncio.run') > -- > https://mail.python.org/mailman/listinfo/python-list From kirill.ratkin at devoteam.com Wed Mar 30 15:32:11 2022 From: kirill.ratkin at devoteam.com (Kirill Ratkin) Date: Wed, 30 Mar 2022 22:32:11 +0300 Subject: calling a function asynchronously In-Reply-To: References: <7e14172c-dd9e-1e5d-65ed-05c02eeedbf8@devoteam.com> Message-ID: Hi 30.03.2022 21:44, Larry Martell ?????: > On Wed, Mar 30, 2022 at 2:40 PM Kirill Ratkin via Python-list > wrote: >> Hi again, >> >> I changed a bit your example and it works as you expected I hope. >> >> import asyncio >> >> >> async def long(): >> for i in range(100): >> await asyncio.sleep(10) >> print("long is done") >> >> >> loop = asyncio.get_event_loop() >> >> task = loop.create_task(long()) >> print('after asyncio.run') >> loop.run_until_complete(asyncio.gather(task)) >> >> >> But how I wrote before ... if you are in big Django project just look at >> existent django libraries for long task running. One of it I sent you. >> There is 'celery' as well. >> >> It's more pragmatic way ... > Appreciate the reply. I did not know about django-background-tasks - > thanks. I've been trying to make use of that. I do not get the errors > I was getting before but it does not appear that my long running task > is running at all. Still debugging. But concerting asyncio - doesn't > run_until_complete block until long() completes? Yes, It runs until the /future/ has completed. In example above 'gather' is not necessary because you create one async task only. You can pass 'task' to run_until_complete directly. But if you create several tasks, all of it need to be run concurently and 'gather' does it. import asyncio import random async def long(x): ??? duration = random.randint(2, 5) ??? await asyncio.sleep(duration) ??? print(f"long is done {x} slept for {duration} seconds") loop = asyncio.get_event_loop() task1 = loop.create_task(long(1)) task2 = loop.create_task(long(2)) task3 = loop.create_task(long(3)) print('after asyncio.run') loop.run_until_complete(asyncio.gather(task1, task2, task3)) So here run_until_complete will wait untill all three tasks complete? or cancel for some reasons (then exception happens. btw we don't handle it in example). In your example you use 'asyncio.run'. Internally it does things like to this def run(ft) loop = asyncio.get_event_loop() task = loop.create_task(ft()) loop.run_until_complete(asyncio.gather(task)) Of course real asyncio.run is much more complicated but model is same, i think. It creates loop and future and 'run_until_complete' for future/task in the loop. That's why you never get 'print' before asyncio.run finishes. You also can split run_until_complete and gather. s = asyncio.gather(task1, task2, task3) print('after asyncio.run') loop.run_until_complete(s) Then 'gather' starts schedule all tasks before 'print' executes. For example, if you have some network IO operations instead this 'print' ... maybe few of your async tasks are done during this time. > >> 30.03.2022 19:10, Larry Martell ?????: >>> import asyncio >>> import time >>> >>> async def long(): >>> for i in range(100): >>> time.sleep(10) >>> >>> asyncio.run(long()) >>> print('after asyncio.run') >> -- >> https://mail.python.org/mailman/listinfo/python-list From grant.b.edwards at gmail.com Thu Mar 31 10:41:23 2022 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Thu, 31 Mar 2022 07:41:23 -0700 (PDT) Subject: Exchange OWA using Python? Message-ID: <6245bd93.1c69fb81.aa4d6.39b7@mx.google.com> Is anybody aware of any Python code for the Exchange OWA protocol/API? The OWA e-mail client I've been using has stopped working. It was a commerical Python application named Hiri, wich has been abandoned by the developer. So, for now, I'm stuck with the OWA web client. It's clumsy and everything takes 3X as long to accomplish as it would with a real e-mail client, but I can survive with it if I have to. [Retirement's not that far away...] The main problem with the web client is lack of notification when new mail arrives. Even if you always have a browser running (I don't), and have a tab that has the OWA client running (I often don't), notification doesn't seem to work reliably (and when it does work, it doesn't use the standard linux desktop notification system). Back when the Exchange IMAP server was enabled (and I used mutt as my e-mail client), I wrote a e-mail notifier app in Python/GTK, so I knew when new mail was waiting. I've been wondering if it would be possible to do the same thing using the Exchange OWA API/protocol instead of IMAP? But, I haven't found any Python support for OWA. I've found libraries for the older Exchange EWS API, but AFAICT the Exchange server in question doesn't support EWS, only OWA. I've also found a bunch of info for Outlook/REST and Office365/REST, but I'm pretty sure those won't work either... -- Grant From Cecil at decebal.nl Thu Mar 31 03:42:39 2022 From: Cecil at decebal.nl (Cecil Westerhof) Date: Thu, 31 Mar 2022 09:42:39 +0200 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> <877d8xnej9.fsf@hornfels.zedat.fu-berlin.de> <87pmm6kxo4.fsf@munus.decebal.nl> <20220328220302.n2pjqqbu7qf3aslf@hjp.at> Message-ID: <875ynulg9c.fsf@munus.decebal.nl> "Peter J. Holzer" writes: > On 2022-03-28 15:35:07 +0200, Cecil Westerhof via Python-list wrote: >> "Loris Bennett" writes: >> > Ubuntu is presumably relying on the Debian security team as well as >> > other volunteers and at least one company, namely Canonical. >> >> Nope. One important reason that I really hate that people use Ubuntu >> for servers is that Ubuntu wants to be up to date. > > Not sure what you mean by that. > > There is an Ubuntu LTS release every 2 years. There is also a Debian > release roughly every 2 years (although not on quite as strict a > schedule). So that's very similar. > >> So Ubuntu starts very close to Debian security wise, but will shift >> rapidly. > > They are are about a year apart, so they will usually contain different > versions of most packages right from the start. So the Ubuntu and Debian > security teams probably can't benefit much from each other. That is is what I partly mean. Debian is very big on security and stability. Most people think that Ubuntu is that also, because it is based on Debian. But Ubuntu wants also provide the newest versions of software and this will affect the stability and security negatively. Even for a desktop I find stability and security more important as the newest versions. That is why I even for the desktop use Debian. Personally I find it strange that people choose newest versions over stability and security for a server. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From Cecil at decebal.nl Thu Mar 31 03:46:14 2022 From: Cecil at decebal.nl (Cecil Westerhof) Date: Thu, 31 Mar 2022 09:46:14 +0200 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) References: <877d8xnej9.fsf@hornfels.zedat.fu-berlin.de> <87pmm6kxo4.fsf@munus.decebal.nl> <20220328220302.n2pjqqbu7qf3aslf@hjp.at> <20220330081744.oj5copxjtjyfmp5k@hjp.at> Message-ID: <871qyilg3d.fsf@munus.decebal.nl> "Peter J. Holzer" writes: > On 2022-03-30 08:48:36 +0200, Marco Sulla wrote: >> On Tue, 29 Mar 2022 at 00:10, Peter J. Holzer wrote: >> > They are are about a year apart, so they will usually contain different >> > versions of most packages right from the start. So the Ubuntu and Debian >> > security teams probably can't benefit much from each other. >> >> Are you sure? Since LTS of Debian and Ubuntu lasts 5 years, I suppose >> the versions of the packages should overlap at some point in the past. > > Standard policy (there are exceptions) on most distros is to stay with > the same version of any package for the entire lifetime. So for example, > Ubuntu 20.04 was released with Apache 2.4.41 and Python 3.8.10 and > Debian 11 was released with Apache 2.4.53 and Python 3.9.2 and they are > still on these versions. Any security fixes and other critical bug fixes > were back-ported to these versions. Are you sure? In the past this was not the case, but it is possible that this has changed. (I do not really follow other distributions. I am quite happy with Debian.) -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From Cecil at decebal.nl Thu Mar 31 06:30:33 2022 From: Cecil at decebal.nl (Cecil Westerhof) Date: Thu, 31 Mar 2022 12:30:33 +0200 Subject: Temporally disabling buffering Message-ID: <87wngajtx2.fsf@munus.decebal.nl> In Python when the output of a script is going to a pipe stdout is buffered. When sending output to tee that is very inconvenient. We can set PYTHONUNBUFFERED, but then stdout is always unbuffered. On Linux we can do: PYTHONUNBUFFERED=T script.py | tee script.log Now the output is only unbuffered for the current run and buffered for other runs where the output goes to a pipe. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From wlfraed at ix.netcom.com Thu Mar 31 10:06:00 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Thu, 31 Mar 2022 10:06:00 -0400 Subject: =?ISO-8859-1?Q?'=E4=C4=F6=D6=FC=DC'_in_Unicode?= =?ISO-8859-1?Q?_(utf-8)?= References: <30d6b0a8-731b-41a8-aa8e-1e77c21e0e9en@googlegroups.com> Message-ID: On Thu, 31 Mar 2022 00:36:10 -0700 (PDT), moi declaimed the following: >>>> '??????'.encode('utf-8') >b'\xc3\xa4\xc3\x84\xc3\xb6\xc3\x96\xc3\xbc\xc3\x9c' >>>> len('??????'.encode('utf-8')) >12 >>>> >>>> ? Is there a question in there somewhere? Crystal ball is hazy... However... Note that once you encode the Unicode literal, you have a BYTE string. There are 12 bytes in that binary -- it is NOT considered Unicode at that point (only when you decode it with the same CODEC will it be Unicode). -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From hjp-python at hjp.at Thu Mar 31 12:53:08 2022 From: hjp-python at hjp.at (Peter J. Holzer) Date: Thu, 31 Mar 2022 18:53:08 +0200 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: <871qyilg3d.fsf@munus.decebal.nl> References: <877d8xnej9.fsf@hornfels.zedat.fu-berlin.de> <87pmm6kxo4.fsf@munus.decebal.nl> <20220328220302.n2pjqqbu7qf3aslf@hjp.at> <20220330081744.oj5copxjtjyfmp5k@hjp.at> <871qyilg3d.fsf@munus.decebal.nl> Message-ID: <20220331165308.v3eb73br5dccqr22@hjp.at> On 2022-03-31 09:46:14 +0200, Cecil Westerhof via Python-list wrote: > "Peter J. Holzer" writes: > > Standard policy (there are exceptions) on most distros is to stay with > > the same version of any package for the entire lifetime. So for example, > > Ubuntu 20.04 was released with Apache 2.4.41 and Python 3.8.10 and > > Debian 11 was released with Apache 2.4.53 and Python 3.9.2 and they are > > still on these versions. Any security fixes and other critical bug fixes > > were back-ported to these versions. > > Are you sure? In the past this was not the case, but it is possible > that this has changed. (I do not really follow other distributions. I > am quite happy with Debian.) This has always been the case with Debian (they even created a special repo for packages where this wasn't feasible, like browsers (which typically update every few weeks and are too large for the security team to backport security fixes). In my experience it's also the case for Ubuntu (see the version numbers I posted). It also was the case for Redhat, but they seem to have switched to a rolling updates model some time ago. I'm not sure how they handle that now. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From Marco.Sulla.Python at gmail.com Thu Mar 31 13:02:23 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Thu, 31 Mar 2022 19:02:23 +0200 Subject: Temporally disabling buffering In-Reply-To: <87wngajtx2.fsf@munus.decebal.nl> References: <87wngajtx2.fsf@munus.decebal.nl> Message-ID: Dirty suggestion: stderr? On Thu, 31 Mar 2022 at 18:38, Cecil Westerhof via Python-list wrote: > > In Python when the output of a script is going to a pipe stdout is > buffered. When sending output to tee that is very inconvenient. > > We can set PYTHONUNBUFFERED, but then stdout is always unbuffered. > > On Linux we can do: > PYTHONUNBUFFERED=T script.py | tee script.log > > Now the output is only unbuffered for the current run and buffered for > other runs where the output goes to a pipe. > > -- > Cecil Westerhof > Senior Software Engineer > LinkedIn: http://www.linkedin.com/in/cecilwesterhof > -- > https://mail.python.org/mailman/listinfo/python-list From barry at barrys-emacs.org Thu Mar 31 13:23:40 2022 From: barry at barrys-emacs.org (Barry) Date: Thu, 31 Mar 2022 18:23:40 +0100 Subject: Temporally disabling buffering In-Reply-To: <87wngajtx2.fsf@munus.decebal.nl> References: <87wngajtx2.fsf@munus.decebal.nl> Message-ID: <0DFC5C72-4849-44B6-9DC9-C8BD9C855943@barrys-emacs.org> > On 31 Mar 2022, at 17:39, Cecil Westerhof via Python-list wrote: > > ?In Python when the output of a script is going to a pipe stdout is > buffered. When sending output to tee that is very inconvenient. > > We can set PYTHONUNBUFFERED, but then stdout is always unbuffered. > > On Linux we can do: > PYTHONUNBUFFERED=T script.py | tee script.log > > Now the output is only unbuffered for the current run and buffered for > other runs where the output goes to a pipe. I prefer to use the -u when I run the output of python into the. python -u script.py | tee script.log And use py -u on windows. Barry > > -- > Cecil Westerhof > Senior Software Engineer > LinkedIn: http://www.linkedin.com/in/cecilwesterhof > -- > https://mail.python.org/mailman/listinfo/python-list > From dieter at handshake.de Thu Mar 31 13:12:10 2022 From: dieter at handshake.de (Dieter Maurer) Date: Thu, 31 Mar 2022 19:12:10 +0200 Subject: Exchange OWA using Python? In-Reply-To: <6245bd93.1c69fb81.aa4d6.39b7@mx.google.com> References: <6245bd93.1c69fb81.aa4d6.39b7@mx.google.com> Message-ID: <25157.57578.186995.641060@ixdm.fritz.box> Grant Edwards wrote at 2022-3-31 07:41 -0700: >Is anybody aware of any Python code for the Exchange OWA protocol/API? According to "https://en.wikipedia.org/wiki/Outlook.com#Mail_client_access" Outlook.com (the modern name for OWA) supports "pop3" and "imap", both supported by Python library modules. Read the "Outlook.com" documentation about how you can connect via those protocols. From Marco.Sulla.Python at gmail.com Thu Mar 31 13:33:55 2022 From: Marco.Sulla.Python at gmail.com (Marco Sulla) Date: Thu, 31 Mar 2022 19:33:55 +0200 Subject: Suggestion for Linux Distro (from PSA: Linux vulnerability) In-Reply-To: <875ynulg9c.fsf@munus.decebal.nl> References: <11041678-4819-a201-a465-b40c8d55441a@gmail.com> <877d8xnej9.fsf@hornfels.zedat.fu-berlin.de> <87pmm6kxo4.fsf@munus.decebal.nl> <20220328220302.n2pjqqbu7qf3aslf@hjp.at> <875ynulg9c.fsf@munus.decebal.nl> Message-ID: On Thu, 31 Mar 2022 at 18:38, Cecil Westerhof via Python-list wrote: > Most people think that > Ubuntu is that also, because it is based on Debian. But Ubuntu wants > also provide the newest versions of software and this will affect the > stability and security negatively. I think you're referring to the fact that Ubuntu releases a new stable version every 6 months, while Debian every 2 years. This is true, but Ubuntu also releases a LTS every 2 years. You can install a LTS and change the options so you'll update the system only where a new LTS is coming out. Furthermore you're not forced to upgrade, you can do it when the LTS comes to the end. On the other hand, you can live on the edge with Debian too. You can install an unstable branch. Furthermore, there's the company factor. According to Google, Debian has about 1k devs, while Ubuntu only about 250. But these devs work full time on Ubuntu and they are paid for. Not sure this is not an important point. For what I know, historically the distros with the reputation to be more stable are distros maintained by companies, Red Hat and Gentoo for example. About stability and security, I can't disagree. But I suppose the people that use the unstable version of some Linux distro are useful for testing and reporting bugs, also security one. So they contribute to the stable versions, and I think we have to be grateful to these "pioneers". From dciprus at cisco.com Thu Mar 31 14:23:21 2022 From: dciprus at cisco.com (Dan Ciprus (dciprus)) Date: Thu, 31 Mar 2022 18:23:21 +0000 Subject: Exchange OWA using Python? In-Reply-To: <25157.57578.186995.641060@ixdm.fritz.box> References: <6245bd93.1c69fb81.aa4d6.39b7@mx.google.com> <25157.57578.186995.641060@ixdm.fritz.box> Message-ID: <20220331182319.ebxlxarshzyrmimy@carbon-gen9> Our org does not have imap nor pop enabled so I guess it depends on your IT and what's enabled for your org. Just my $.02 .. On Thu, Mar 31, 2022 at 07:12:10PM +0200, Dieter Maurer wrote: >Grant Edwards wrote at 2022-3-31 07:41 -0700: >>Is anybody aware of any Python code for the Exchange OWA protocol/API? > >According to "https://en.wikipedia.org/wiki/Outlook.com#Mail_client_access" >Outlook.com (the modern name for OWA) supports "pop3" and "imap", >both supported by Python library modules. > >Read the "Outlook.com" documentation about how you can connect >via those protocols. >-- >https://mail.python.org/mailman/listinfo/python-list -- Daniel Ciprus .:|:.:|:. CONSULTING ENGINEER.CUSTOMER DELIVERY Cisco Systems Inc. dciprus at cisco.com tel: +1-703-484-0205 mob: +1-540-223-7098 [ curl -L http://git.io/unix ] -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 659 bytes Desc: not available URL: From rosuav at gmail.com Thu Mar 31 12:59:32 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 1 Apr 2022 03:59:32 +1100 Subject: =?UTF-8?B?UmU6ICfDpMOEw7bDlsO8w5wnIGluIFVuaWNvZGUgKHV0Zi04KQ==?= In-Reply-To: References: <30d6b0a8-731b-41a8-aa8e-1e77c21e0e9en@googlegroups.com> Message-ID: On Fri, 1 Apr 2022 at 03:45, Dennis Lee Bieber wrote: > > On Thu, 31 Mar 2022 00:36:10 -0700 (PDT), moi > declaimed the following: > > >>>> '??????'.encode('utf-8') > >b'\xc3\xa4\xc3\x84\xc3\xb6\xc3\x96\xc3\xbc\xc3\x9c' > >>>> len('??????'.encode('utf-8')) > >12 > >>>> > >>>> ? > > Is there a question in there somewhere? > > Crystal ball is hazy... > > However... Note that once you encode the Unicode literal, you have a > BYTE string. There are 12 bytes in that binary -- it is NOT considered > Unicode at that point (only when you decode it with the same CODEC will it > be Unicode). > That's jmf. Ignore him. He knows nothing about Unicode and is determined to make everyone aware of that fact. He got blocked from the mailing list ages ago, and I don't think anyone's regretted it. ChrisA From auriocus at gmx.de Thu Mar 31 15:22:34 2022 From: auriocus at gmx.de (Christian Gollwitzer) Date: Thu, 31 Mar 2022 21:22:34 +0200 Subject: Exchange OWA using Python? In-Reply-To: References: <6245bd93.1c69fb81.aa4d6.39b7@mx.google.com> Message-ID: Am 31.03.22 um 16:41 schrieb Grant Edwards: > Is anybody aware of any Python code for the Exchange OWA protocol/API? > > The OWA e-mail client I've been using has stopped working. It was a > commerical Python application named Hiri, wich has been abandoned by > the developer. > > So, for now, I'm stuck with the OWA web client. It's clumsy and > everything takes 3X as long to accomplish as it would with a real > e-mail client, but I can survive with it if I have to. I can recommend davmail as a workaround: http://davmail.sourceforge.net/ Davmail converts OWA into regular mail protocols like POP3, IMAP, SMTP. You can use any email/calender client (I've used thunderbird) and most of it worked well. Davmail is written in Java, not Python, but basically this should not matter if you only use it. Christian From dciprus at cisco.com Thu Mar 31 17:17:30 2022 From: dciprus at cisco.com (Dan Ciprus (dciprus)) Date: Thu, 31 Mar 2022 21:17:30 +0000 Subject: Exchange OWA using Python? In-Reply-To: References: <6245bd93.1c69fb81.aa4d6.39b7@mx.google.com> Message-ID: Yes, this ... I've been using this successfully for years and it obviously has its sad parts but it works pretty well overall. ________________________________________ From: Python-list on behalf of Christian Gollwitzer Sent: Thursday, March 31, 2022 3:22 PM To: python-list at python.org Subject: Re: Exchange OWA using Python? Am 31.03.22 um 16:41 schrieb Grant Edwards: > Is anybody aware of any Python code for the Exchange OWA protocol/API? > > The OWA e-mail client I've been using has stopped working. It was a > commerical Python application named Hiri, wich has been abandoned by > the developer. > > So, for now, I'm stuck with the OWA web client. It's clumsy and > everything takes 3X as long to accomplish as it would with a real > e-mail client, but I can survive with it if I have to. I can recommend davmail as a workaround: http://davmail.sourceforge.net/ Davmail converts OWA into regular mail protocols like POP3, IMAP, SMTP. You can use any email/calender client (I've used thunderbird) and most of it worked well. Davmail is written in Java, not Python, but basically this should not matter if you only use it. Christian -- https://mail.python.org/mailman/listinfo/python-list From grant.b.edwards at gmail.com Thu Mar 31 19:22:33 2022 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Thu, 31 Mar 2022 16:22:33 -0700 (PDT) Subject: Exchange OWA using Python? References: <6245bd93.1c69fb81.aa4d6.39b7@mx.google.com> <25157.57578.186995.641060@ixdm.fritz.box> Message-ID: <624637b9.1c69fb81.47628.2c76@mx.google.com> On 2022-03-31, Dieter Maurer wrote: > Grant Edwards wrote at 2022-3-31 07:41 -0700: >>Is anybody aware of any Python code for the Exchange OWA protocol/API? > > According to "https://en.wikipedia.org/wiki/Outlook.com#Mail_client_access" > Outlook.com (the modern name for OWA) supports "pop3" and "imap", > both supported by Python library modules. No, outlook.com is a Microsft web hosting service. OWA is an HTTP-based API suspport by MS Exchange server. > Read the "Outlook.com" documentation about how you can connect > via those protocols. I don't want to connect to Outlook.com. I want to connect to an Exchange server that doesn't not have IMAP, POP3, or SMTP enabled. From grant.b.edwards at gmail.com Thu Mar 31 19:26:22 2022 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Thu, 31 Mar 2022 16:26:22 -0700 (PDT) Subject: Exchange OWA using Python? References: <6245bd93.1c69fb81.aa4d6.39b7@mx.google.com> Message-ID: <6246389e.1c69fb81.a040.264a@mx.google.com> On 2022-03-31, Christian Gollwitzer wrote: > Am 31.03.22 um 16:41 schrieb Grant Edwards: >> Is anybody aware of any Python code for the Exchange OWA protocol/API? >> >> The OWA e-mail client I've been using has stopped working. It was a >> commerical Python application named Hiri, wich has been abandoned by >> the developer. >> >> So, for now, I'm stuck with the OWA web client. It's clumsy and >> everything takes 3X as long to accomplish as it would with a real >> e-mail client, but I can survive with it if I have to. > > I can recommend davmail as a workaround: > > http://davmail.sourceforge.net/ > > Davmail converts OWA into regular mail protocols like POP3, IMAP, SMTP. > You can use any email/calender client (I've used thunderbird) and most > of it worked well. I used Davmail in the past when the server supported EWS. AFAICT, it does not work with the current server. If you look at the Exchange protocols listed at http://davmail.sourceforge.net/gettingstarted.html, OWA is not one of them: * O365Modern Office 365 modern authentication (Oauth2) * O365Interactive Office 365 with interactive browser window, not available in headless mode (OpenJFX required) * O365Manual Office 365 with interactive dialog, not available in headless mode * O365 Office 365 EWS mode * EWS Exchange 2007 and later * WebDav Exchange 2007 and earliear WebDav mode * Auto WebDav mode with EWS failover > Davmail is written in Java, not Python, but basically this should > not matter if you only use it. Have you used it with OWA as the protocol? -- Grant From grant.b.edwards at gmail.com Thu Mar 31 19:27:32 2022 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Thu, 31 Mar 2022 16:27:32 -0700 (PDT) Subject: Exchange OWA using Python? References: <6245bd93.1c69fb81.aa4d6.39b7@mx.google.com> =?utf-8?q?=3CBL0PR11MB3441744AA55B045AD5784BF9?= =?utf-8?q?AAE19=40BL0PR11MB3441=2Enamprd11=2Eprod=2Eoutlook=2Ecom=3E?= Message-ID: <624638e4.1c69fb81.adc7b.2ff9@mx.google.com> On 2022-03-31, Dan Ciprus (dciprus) via Python-list wrote: > Yes, this ... I've been using this successfully for years and it > obviously has its sad parts but it works pretty well overall. Using the _OWA_ protocol? When our server switched from EWS to OWA, I could no longer get it to work. -- Grant From grant.b.edwards at gmail.com Thu Mar 31 19:34:17 2022 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Thu, 31 Mar 2022 16:34:17 -0700 (PDT) Subject: Exchange OWA using Python? References: <6245bd93.1c69fb81.aa4d6.39b7@mx.google.com> Message-ID: <62463a79.1c69fb81.78c1e.467c@mx.google.com> On 2022-03-31, Grant Edwards wrote: > Is anybody aware of any Python code for the Exchange OWA protocol/API? > > The OWA e-mail client I've been using has stopped working. It was a > commerical Python application named Hiri, wich has been abandoned by > the developer. I'm trying out Thunderbird with an OWA plugin https://www.beonex.com/owl/ and it's working very nicely. There are some quirks in Thunderbird that I'm still tring to figure out, but I have the same issues if it were using IMAP/SMTP. -- Grant From wlfraed at ix.netcom.com Thu Mar 31 19:29:23 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Thu, 31 Mar 2022 19:29:23 -0400 Subject: =?ISO-8859-1?Q?'=E4=C4=F6=D6=FC=DC'_in_Unicode?= =?ISO-8859-1?Q?_(utf-8)?= References: <30d6b0a8-731b-41a8-aa8e-1e77c21e0e9en@googlegroups.com> Message-ID: <78ec4hhoc5m17enimphfl002me8rnghtnh@4ax.com> On Fri, 1 Apr 2022 03:59:32 +1100, Chris Angelico declaimed the following: >That's jmf. Ignore him. He knows nothing about Unicode and is >determined to make everyone aware of that fact. > >He got blocked from the mailing list ages ago, and I don't think >anyone's regretted it. > Ah yes... Unfortunately, when gmane made the mirror read-only, I had to revert to comp.lang.python... and all the junk that gets in via that and Google Groups... -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From wlfraed at ix.netcom.com Thu Mar 31 19:34:49 2022 From: wlfraed at ix.netcom.com (Dennis Lee Bieber) Date: Thu, 31 Mar 2022 19:34:49 -0400 Subject: Exchange OWA using Python? References: <6245bd93.1c69fb81.aa4d6.39b7@mx.google.com> <25157.57578.186995.641060@ixdm.fritz.box> <624637b9.1c69fb81.47628.2c76@mx.google.com> Message-ID: On Thu, 31 Mar 2022 16:22:33 -0700 (PDT), Grant Edwards declaimed the following: > >OWA is an HTTP-based API suspport by MS Exchange server. > From what I found, M$ doesn't consider OWA to be an API... It expands to "Outlook Web App", period, (and more recently is called "Outlook for the Web"). That would explain why there is no documentation of an "API"... I get the impression that any so-called API programs have had to reverse engineer (maybe using things like WireShark) the HTTP communication. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/ From skip.montanaro at gmail.com Thu Mar 31 20:19:53 2022 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Thu, 31 Mar 2022 19:19:53 -0500 Subject: asyncio+tkinter In-Reply-To: References: Message-ID: > > > Given that both asyncio & tkinter are modules in the standard lib and > both > > have event loops, I would have expected to find some "best practice" > > solution to mixing the two. > > Agreed. For GTK, you can use a dedicated loop policy like this: > > import asyncio_glib > asyncio.set_event_loop_policy(asyncio_glib.GLibEventLoopPolicy()) > > It should be possible to make it this easy for asyncio + tkinter. > I haven't used GTK in ages and ages, but I do still have a tkinter program that I drag out every now and then when my wrists start acting up. After opining about this on the PyIdeas discuss forum, I spent a few minutes and came up with a simple AsyncTk subclass of tkinter.Tk. Here's its hello world: https://gist.github.com/smontanaro/5e12c557602a76c609e46ca59387ad1c I modified my activity watcher to use it: https://github.com/smontanaro/python-bits/blob/master/src/watch.py A few warnings: 1. copy/paste reuse of that AsyncTk class 2. the code is a mess (it's probably 20-25 years old, I make no apologies for the mess) 3. as of about an hour ago it now uses pynput to watch for mouse/kbd watching So, Tk+asyncio turns out to be fairly easy to do, at least for simple stuff. Skip From grant.b.edwards at gmail.com Thu Mar 31 21:17:44 2022 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Thu, 31 Mar 2022 18:17:44 -0700 (PDT) Subject: Exchange OWA using Python? References: <6245bd93.1c69fb81.aa4d6.39b7@mx.google.com> <25157.57578.186995.641060@ixdm.fritz.box> <624637b9.1c69fb81.47628.2c76@mx.google.com> Message-ID: <624652b8.1c69fb81.89f92.2c4e@mx.google.com> On 2022-03-31, Dennis Lee Bieber wrote: > On Thu, 31 Mar 2022 16:22:33 -0700 (PDT), Grant Edwards > declaimed the following: > >> >>OWA is an HTTP-based API suspport by MS Exchange server. > > From what I found, M$ doesn't consider OWA to be an API... It expands > to "Outlook Web App", period, (and more recently is called "Outlook for the > Web"). Right. There used to be an API called "EWS", but that's no longer available (at least on the servers I've dealt with). Instead, there's a REST/JSON api that's used by the outlook web client. There are a few email clients (that I'm aware of) that have figured out how to use that API: Hiri (now abandoned) which stopped working for me yesterday. Webmail for OWA https://play.google.com/store/apps/details?id=smtb.owa Android app. OWL Thunderbird plug-in This is working nicely for me. There is an older Thunderbird plugin (by the developers of OWL) that used the EWS protocol (but not OWA) called ExQuilla. I think that the protocol used by Outlook is MAPI/RPC or MAPI/HTTP. I believe that Evolution used to be able to use that protocol, but from what I've read it only works with very old versions of Exchange. For newer Exchange servers, Evolution supports EWS, but the servers I've dealt with over the past few years didn't seem to support EWS. [Though I haven't tried for a while.] From rosuav at gmail.com Thu Mar 31 20:31:12 2022 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 1 Apr 2022 11:31:12 +1100 Subject: =?UTF-8?B?UmU6ICfDpMOEw7bDlsO8w5wnIGluIFVuaWNvZGUgKHV0Zi04KQ==?= In-Reply-To: <78ec4hhoc5m17enimphfl002me8rnghtnh@4ax.com> References: <30d6b0a8-731b-41a8-aa8e-1e77c21e0e9en@googlegroups.com> <78ec4hhoc5m17enimphfl002me8rnghtnh@4ax.com> Message-ID: On Fri, 1 Apr 2022 at 11:16, Dennis Lee Bieber wrote: > > On Fri, 1 Apr 2022 03:59:32 +1100, Chris Angelico > declaimed the following: > > > >That's jmf. Ignore him. He knows nothing about Unicode and is > >determined to make everyone aware of that fact. > > > >He got blocked from the mailing list ages ago, and I don't think > >anyone's regretted it. > > > Ah yes... Unfortunately, when gmane made the mirror read-only, I had to > revert to comp.lang.python... and all the junk that gets in via that and > Google Groups... > Killfiles can help. ChrisA