From shishaozhong at gmail.com  Wed Dec  1 05:18:15 2021
From: shishaozhong at gmail.com (Shaozhong SHI)
Date: Wed, 1 Dec 2021 10:18:15 +0000
Subject: Installation of GeoPandas - failed at fiona
Message-ID: <CA+i5JwbUFQLqhMZuvXmqxdoHu3uiZAPEOpDZ3+43Bhoq1ZvZeg@mail.gmail.com>

I am trying to install geopandas.

I navigated to c:\programData|Anaconda3\Scripts>
and typed in 'pip install geopandas'.

It ran but failed at fiona.

I tried import geopandas as gp, but Error Message says: No module names
'geopandas'.

Can anyone help?

Regards,

David

From mats at wichmann.us  Wed Dec  1 09:52:41 2021
From: mats at wichmann.us (Mats Wichmann)
Date: Wed, 1 Dec 2021 07:52:41 -0700
Subject: Installation of GeoPandas - failed at fiona
In-Reply-To: <CA+i5JwbUFQLqhMZuvXmqxdoHu3uiZAPEOpDZ3+43Bhoq1ZvZeg@mail.gmail.com>
References: <CA+i5JwbUFQLqhMZuvXmqxdoHu3uiZAPEOpDZ3+43Bhoq1ZvZeg@mail.gmail.com>
Message-ID: <e9cd2245-2187-b749-e94d-7d240dba7e1e@wichmann.us>

On 12/1/21 03:18, Shaozhong SHI wrote:
> I am trying to install geopandas.
> 
> I navigated to c:\programData|Anaconda3\Scripts>
> and typed in 'pip install geopandas'.

Usually if you're using the Anaconda environment, you should use the 
"conda" installer, for consistency.

> 
> It ran but failed at fiona.

Failed how?
> 
> I tried import geopandas as gp, but Error Message says: No module names
> 'geopandas'.

That almost certianly means the install went to a different place than 
the environment where you're trying to use it.


From rajvenders9 at gmail.com  Wed Dec  1 21:26:24 2021
From: rajvenders9 at gmail.com (Ragavendar)
Date: Wed, 1 Dec 2021 18:26:24 -0800
Subject: NOT ABLE TO SEE PICTURES 
Message-ID: <5F566151-DDD9-4EAB-A5B5-9B39373CB836@hxcore.ol>

   When the python runs in file explorer when I open a folder it shows NO
   ITEAMS FOUND and if I open same folder without python ITEAMS ARE THERE so
   kindly help me

   ?

   ?

   Sent from [1]Mail for Windows

   ?

References

   Visible links
   1. https://go.microsoft.com/fwlink/?LinkId=550986

From inhahe at gmail.com  Wed Dec  1 10:47:45 2021
From: inhahe at gmail.com (inhahe)
Date: Wed, 1 Dec 2021 10:47:45 -0500
Subject: NOT ABLE TO SEE PICTURES
In-Reply-To: <5F566151-DDD9-4EAB-A5B5-9B39373CB836@hxcore.ol>
References: <5F566151-DDD9-4EAB-A5B5-9B39373CB836@hxcore.ol>
Message-ID: <CAPeSgWpVMxOPdS4vt=YjXkLxULGbiKJy1Fon69+ERKUjwAXagw@mail.gmail.com>

Maybe your file explorer is set to ignore system and/or hidden files, and
Python isn't? You can check that in folder settings..

On Wed, Dec 1, 2021 at 10:31 AM Ragavendar <rajvenders9 at gmail.com> wrote:

>    When the python runs in file explorer when I open a folder it shows NO
>    ITEAMS FOUND and if I open same folder without python ITEAMS ARE THERE
> so
>    kindly help me
>
>
>
>
>
>    Sent from [1]Mail for Windows
>
>
>
> References
>
>    Visible links
>    1. https://go.microsoft.com/fwlink/?LinkId=550986
> --
> https://mail.python.org/mailman/listinfo/python-list
>

From jenkris at tutanota.com  Wed Dec  1 11:01:36 2021
From: jenkris at tutanota.com (Jen Kris)
Date: Wed, 1 Dec 2021 17:01:36 +0100 (CET)
Subject: Python child process in while True loop blocks parent
In-Reply-To: <7A3CE5EB-C53A-404D-BF32-77CF0243BF3D@barrys-emacs.org>
References: <MphOu5i--3-2@tutanota.com>
 <B05343F2-30B5-470C-8CFC-EBFA7C5EFD6A@barrys-emacs.org>
 <MphoqOf--3-2@tutanota.com>
 <7A3CE5EB-C53A-404D-BF32-77CF0243BF3D@barrys-emacs.org>
Message-ID: <MpqihHm--3-2@tutanota.com>

Thanks for your comment re blocking.? 

I removed pipes from the Python and C programs to see if it blocks without them, and it does.? It looks now like the problem is not pipes.? I use fork() and execv() in C to run Python in a child process, but the Python process blocks because fork() does not create a new thread, so the Python global interpreter lock (GIL) prevents the C program from running once Python starts.? So the solution appears to be run Python in a separate thread, which I can do with pthread create.? See "Thread State and the Global Interpreter Lock" https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock and the sections below that "Non-Python created threads" and "Cautions about fork()."?

I'm working on that today and I hope all goes well :)?



Nov 30, 2021, 11:42 by barry at barrys-emacs.org:

>
>
>
>> On 29 Nov 2021, at 22:31, Jen Kris <>> jenkris at tutanota.com>> > wrote:
>>
>> Thanks to you and Cameron for your replies.? The C side has an epoll_ctl set, but no event loop to handle it yet.? I'm putting that in now with a pipe write in Python-- as Cameron pointed out that is the likely source of blocking on C.? The pipes are opened as rdwr in Python because that's nonblocking by default.? The child will become more complex, but not in a way that affects polling.? And thanks for the tip about the c-string termination.?
>>
>>
>
> flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK.
> You should not use O_RDWR when you only need O_RDONLY access or only O_WRONLY access.
>
> You may find
>
> man 2 open
>
> useful to understand in detail what is behind os.open().
>
> Barry
>
>
>
>
>>
>>
>> Nov 29, 2021, 14:12 by >> barry at barrys-emacs.org>> :
>>
>>>
>>>
>>>> On 29 Nov 2021, at 20:36, Jen Kris via Python-list <>>>> python-list at python.org>>>> > wrote:
>>>>
>>>> ?I have a C program that forks to create a child process and uses execv to call a Python program.  The Python program communicates with the parent process (in C) through a FIFO pipe monitored with epoll(). 
>>>>
>>>> The Python child process is in a while True loop, which is intended to keep it running while the parent process proceeds, and perform functions for the C program only at intervals when the parent sends data to the child -- similar to a daemon process. 
>>>>
>>>> The C process writes to its end of the pipe and the child process reads it, but then the child process continues to loop, thereby blocking the parent. 
>>>>
>>>> This is the Python code:
>>>>
>>>> #!/usr/bin/python3
>>>> import os
>>>> import select
>>>>
>>>> #Open the named pipes
>>>> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
>>>>
>>> Why open rdwr if you are only going to read the pipe?
>>>
>>>> pw = os.open('/tmp/Pipe_02', os.O_RDWR)
>>>>
>>> Only need to open for write.
>>>
>>>>
>>>> ep = select.epoll(-1)
>>>> ep.register(pr, select.EPOLLIN)
>>>>
>>>
>>> Is the only thing that the child does this:
>>> 1. Read message from pr
>>> 2. Process message
>>> 3. Write result to pw.
>>> 4. Loop from 1
>>>
>>> If so as Cameron said you do not need to worry about the poll.
>>> Do you plan for the child to become more complex?
>>>
>>>>
>>>> while True:
>>>>
>>>> events = ep.poll(timeout=2.5, maxevents=-1)
>>>> #events = ep.poll(timeout=None, maxevents=-1)
>>>>
>>>> print("child is looping")
>>>>
>>>> for fileno, event in events:
>>>> print("Python fileno")
>>>> print(fileno)
>>>> print("Python event")
>>>> print(event)
>>>> v = os.read(pr,64)
>>>> print("Pipe value")
>>>> print(v)
>>>>
>>>> The child process correctly receives the signal from ep.poll and correctly reads the data in the pipe, but then it continues looping.  For example, when I put in a timeout:
>>>>
>>>> child is looping
>>>> Python fileno
>>>> 4
>>>> Python event
>>>> 1
>>>> Pipe value
>>>> b'10\x00'
>>>>
>>> The C code does not need to write a 0 bytes at the end.
>>> I assume the 0 is from the end of a C string.
>>> UDS messages have a length.
>>> In the C just write 2 byes in the case.
>>>
>>> Barry
>>>
>>>> child is looping
>>>> child is looping
>>>>
>>>> That suggests that a while True loop is not the right thing to do in this case.  My question is, what type of process loop is best for this situation?  The multiprocessing, asyncio and subprocess libraries are very extensive, and it would help if someone could suggest the best alternative for what I am doing here. 
>>>>
>>>> Thanks very much for any ideas. 
>>>>
>>>>
>>>> -- 
>>>> https://mail.python.org/mailman/listinfo/python-list
>>>>
>>
>>


From cspealma at redhat.com  Wed Dec  1 12:01:59 2021
From: cspealma at redhat.com (Calvin Spealman)
Date: Wed, 1 Dec 2021 12:01:59 -0500
Subject: NOT ABLE TO SEE PICTURES
In-Reply-To: <5F566151-DDD9-4EAB-A5B5-9B39373CB836@hxcore.ol>
References: <5F566151-DDD9-4EAB-A5B5-9B39373CB836@hxcore.ol>
Message-ID: <CACo5Rz7FFXUV1YcfXajDW5XDnH6+tj6_ib1UdvjuT3v8Wqh1vg@mail.gmail.com>

What does "python runs in file explorer" mean? How do you know its the same
folder? How are you looking at the contents of the folder?

On Wed, Dec 1, 2021 at 10:33 AM Ragavendar <rajvenders9 at gmail.com> wrote:

>    When the python runs in file explorer when I open a folder it shows NO
>    ITEAMS FOUND and if I open same folder without python ITEAMS ARE THERE
> so
>    kindly help me
>
>
>
>
>
>    Sent from [1]Mail for Windows
>
>
>
> References
>
>    Visible links
>    1. https://go.microsoft.com/fwlink/?LinkId=550986
> --
> https://mail.python.org/mailman/listinfo/python-list
>


-- 

CALVIN SPEALMAN

SENIOR QUALITY ENGINEER

calvin.spealman at redhat.com  M: +1.336.210.5107
[image: https://red.ht/sig] <https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

From PythonList at DancesWithMice.info  Wed Dec  1 18:20:35 2021
From: PythonList at DancesWithMice.info (dn)
Date: Thu, 2 Dec 2021 12:20:35 +1300
Subject: PyCharm settings - per: print('\N{flag: Mauritius}') not supported in
 py3.9
In-Reply-To: <c4f62875-5c4e-055e-cc5d-7a555e25e88f@DancesWithMice.info>
References: <CADrxXXkQcFMJhAt5AGt=BL=QqHCgJ7qbx=ts0zR8H9s2L1ZMdg@mail.gmail.com>
 <CAPTjJmoxxRU=-uA0_RJnocZFm6WG0+OpZXFM3TDX2Tgcr1=WBg@mail.gmail.com>
 <c4f62875-5c4e-055e-cc5d-7a555e25e88f@DancesWithMice.info>
Message-ID: <02c13a24-6736-6eaf-7b2a-4c85e28caf8e@DancesWithMice.info>

On 29/11/2021 10.08, dn via Python-list wrote:
> On 29/11/2021 02.18, Chris Angelico wrote:
>> On Mon, Nov 29, 2021 at 12:10 AM Abdur-Rahmaan Janhangeer
>> <arj.python at gmail.com> wrote:
>>
>> Flags are actually constructed from multiple codepoints. What you want
>> is to insert each codepoint separately. You can see them listed in the
>> second column of the table you linked to.
>>
>>>>> "\U0001F1F2\U0001F1FA"
>> '??'
>>
>> To do this with names, you need the names of those two codepoints:
>>
>> '\U0001f1f2' REGIONAL INDICATOR SYMBOL LETTER M
>> '\U0001f1fa' REGIONAL INDICATOR SYMBOL LETTER U
>>
>>>>> "\N{REGIONAL INDICATOR SYMBOL LETTER M}\N{REGIONAL INDICATOR SYMBOL LETTER U}"
>> '??'
> 
> 
> Don't use Emojis that often. The colored circles (U+1F534 etc) display
> in full, glorious, technicolor.
> 
> However, when trying the above, with our local flag in (Fedora Linux,
> Gnome) Terminal or PyCharm's Run terminal; the two letters "N" and "Z"
> are shown with dotted-outlines. Similarly, the Mauritius' flag is shown
> as "M" and "U".


Investing a bit of time/waiting for a meeting, found a number of Issues
lodged with JetBrains relating to emoji-support/display:-

Among the most recent advice is to add a "Fallback font" which is known
to include (colored!) emojis.

Thus (this includes personal settings):

File > Settings > Editor > Font
	Font = IBM Plex Mono
	drop down > Typography Settings
	Fallback font = Twemoji

After which, patriotic-pride may be expressed!


However, the Issues (that is to say, those which I had time/energy to
read) indicate that there may still be differences between Linux, Mac,
Windows, and that 'franken-thing' which is Linux on top of Windows but
we don't label it "Linux" so that people still think it is "Windows".


Further wrinkles (drifting OT!):

Fedora 33's Gnome Font Viewer shows:
Emoji One
Noto Color Emoji
Twemoji

- don't ask me the why/when/where of Twemoji
- the other two were added via dnf (yum) today

Neither shows in PyCharm's drop-down list of choices - even after
stop/start (which doesn't appear strictly necessary, per above, but...).


Performed:

fc-cache -fv

and the listing verifies their presence, and ensures all is compos-mentis.

In Writer, all three appear (without prefix). Still only the one appears
in PyCharm. (sigh...)


PS thanks to the OP! Certainly opened my eyes and befuddled (what's left
of) my brain. Was planning to use the aforementioned 'colored circles'
in a PUG coding exercise, but have now amended to require coders to
'wave the flag'. Hopefully they will find such slightly more amusing...
Salute!
-- 
Regards,
=dn

From rob.cliffe at btinternet.com  Thu Dec  2 00:18:32 2021
From: rob.cliffe at btinternet.com (Rob Cliffe)
Date: Thu, 2 Dec 2021 05:18:32 +0000
Subject: A decade or so of Python programming, and I've never thought to
 "for-elif"
In-Reply-To: <CAPTjJmqtdjqmJ7jiOB4z1G2oMv9btiLcFqn4v9gnqbHtxGJwNA@mail.gmail.com>
References: <CAPTjJmqtdjqmJ7jiOB4z1G2oMv9btiLcFqn4v9gnqbHtxGJwNA@mail.gmail.com>
Message-ID: <df476b82-55df-444d-0d61-f8e72044f7ca@btinternet.com>

If for ... else was spelt more intelligibly, e.g. for ... nobreak, there 
would be no temptation to use anything like `elif'. `nobreakif' wouldn't 
be a keyword.
Rob Cliffe

On 30/11/2021 06:24, Chris Angelico wrote:
> for ns in namespaces:
>      if name in ns:
>          print("Found!")
>          break
> elif name.isupper():
>      print("All-caps name that wasn't found")
>
> This actually doesn't work. I have been programming in Python for well
> over a decade, and never before been in a situation where this would
> be useful.
>
> As YAGNIs go, this is right up there.
>
> (For the record, since this was the last thing in the function, I just
> made the break a return. Alternatively, an extra indentation level
> "else: if name.isupper():" wouldn't have been that terrible.)
>
> Your random piece of amusement for today.
>
> ChrisA


From petsiosn at gmail.com  Thu Dec  2 15:37:53 2021
From: petsiosn at gmail.com (nikos petsios)
Date: Thu, 2 Dec 2021 22:37:53 +0200
Subject: how I can get python idle?
Message-ID: <447B32AB-3A61-4B71-9687-4B5C7CD5A1A7@hxcore.ol>

   ?

   ?

   Sent from [1]Mail for Windows 10

   ?

   [2][IMG] Virus-free. [3]www.avast.com

References

   Visible links
   1. https://go.microsoft.com/fwlink/?LinkId=550986
   2. https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=icon
   3. https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=link

From mats at wichmann.us  Thu Dec  2 17:33:05 2021
From: mats at wichmann.us (Mats Wichmann)
Date: Thu, 2 Dec 2021 15:33:05 -0700
Subject: how I can get python idle?
In-Reply-To: <447B32AB-3A61-4B71-9687-4B5C7CD5A1A7@hxcore.ol>
References: <447B32AB-3A61-4B71-9687-4B5C7CD5A1A7@hxcore.ol>
Message-ID: <3795dcc7-8485-7531-d4bf-c3a23fb16c75@wichmann.us>


On 12/2/21 13:37, nikos petsios wrote:


See here: https://docs.python.org/3/using/

If you're on windows you likely have it if you installed Python.

From barry at barrys-emacs.org  Sat Dec  4 12:22:15 2021
From: barry at barrys-emacs.org (Barry Scott)
Date: Sat, 4 Dec 2021 17:22:15 +0000
Subject: Python child process in while True loop blocks parent
In-Reply-To: <MpqihHm--3-2@tutanota.com>
References: <MphOu5i--3-2@tutanota.com>
 <B05343F2-30B5-470C-8CFC-EBFA7C5EFD6A@barrys-emacs.org>
 <MphoqOf--3-2@tutanota.com>
 <7A3CE5EB-C53A-404D-BF32-77CF0243BF3D@barrys-emacs.org>
 <MpqihHm--3-2@tutanota.com>
Message-ID: <27742189-51F0-49A8-ACA7-344EFCD2B1BD@barrys-emacs.org>


> On 1 Dec 2021, at 16:01, Jen Kris <jenkris at tutanota.com> wrote:
> 
> Thanks for your comment re blocking.  
> 
> I removed pipes from the Python and C programs to see if it blocks without them, and it does.
> It looks now like the problem is not pipes.

Ok.

> I use fork() and execv() in C to run Python in a child process, but the Python process blocks

Use strace on the parent process to see what is happening.
You will need to use the option to follow subprocesses so that you can see what goes on in the python process.

See man strace and the --follow-forks and --output-separately options.
That will allow you to find the blocking system call that your code is making.

> because fork() does not create a new thread, so the Python global interpreter lock (GIL) prevents the C program from running once Python starts.

Not sure why you think this.

>   So the solution appears to be run Python in a separate thread, which I can do with pthread create.
>   See "Thread State and the Global Interpreter Lock" https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock <https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock> and the sections below that "Non-Python created threads" and "Cautions about fork()." 

I take it you mean that in the parent you think that using pthreads will affect python after the exec() call?
I does not. After exec() the process has one main thread create by the kernel and a new address space as defined by the /usr/bin/python.
The only state that in inherited from the parent are open file descriptors, the current working directory and security state like UID, GID.

> I'm working on that today and I hope all goes well :) 

You seem to be missing background information on how processes work.
Maybe "Advanced Programming in the UNIX Environment" would be helpful?

https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk <https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk> 

It's a great book and covers a wide range of Unix systems programming topics.

Have you created a small C program that just does the fork and exec of a python program to test out your assumptions?
If not I recommend that you do.

Barry


> 
> 
> 
> Nov 30, 2021, 11:42 by barry at barrys-emacs.org:
> 
> 
>> On 29 Nov 2021, at 22:31, Jen Kris <jenkris at tutanota.com <mailto:jenkris at tutanota.com>> wrote:
>> 
>> Thanks to you and Cameron for your replies.  The C side has an epoll_ctl set, but no event loop to handle it yet.  I'm putting that in now with a pipe write in Python-- as Cameron pointed out that is the likely source of blocking on C.  The pipes are opened as rdwr in Python because that's nonblocking by default.  The child will become more complex, but not in a way that affects polling.  And thanks for the tip about the c-string termination. 
>> 
> 
> flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK.
> You should not use O_RDWR when you only need O_RDONLY access or only O_WRONLY access.
> 
> You may find
> 
> man 2 open
> 
> useful to understand in detail what is behind os.open().
> 
> Barry
> 
> 
> 
>> 
>> 
>> Nov 29, 2021, 14:12 by barry at barrys-emacs.org <mailto:barry at barrys-emacs.org>:
>> 
>> On 29 Nov 2021, at 20:36, Jen Kris via Python-list <python-list at python.org <mailto:python-list at python.org>> wrote:
>> 
>> ?I have a C program that forks to create a child process and uses execv to call a Python program. The Python program communicates with the parent process (in C) through a FIFO pipe monitored with epoll(). 
>> 
>> The Python child process is in a while True loop, which is intended to keep it running while the parent process proceeds, and perform functions for the C program only at intervals when the parent sends data to the child -- similar to a daemon process. 
>> 
>> The C process writes to its end of the pipe and the child process reads it, but then the child process continues to loop, thereby blocking the parent. 
>> 
>> This is the Python code:
>> 
>> #!/usr/bin/python3
>> import os
>> import select
>> 
>> #Open the named pipes
>> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
>> Why open rdwr if you are only going to read the pipe?
>> pw = os.open('/tmp/Pipe_02', os.O_RDWR)
>> Only need to open for write.
>> 
>> ep = select.epoll(-1)
>> ep.register(pr, select.EPOLLIN)
>> 
>> Is the only thing that the child does this:
>> 1. Read message from pr
>> 2. Process message
>> 3. Write result to pw.
>> 4. Loop from 1
>> 
>> If so as Cameron said you do not need to worry about the poll.
>> Do you plan for the child to become more complex?
>> 
>> while True:
>> 
>> events = ep.poll(timeout=2.5, maxevents=-1)
>> #events = ep.poll(timeout=None, maxevents=-1)
>> 
>> print("child is looping")
>> 
>> for fileno, event in events:
>> print("Python fileno")
>> print(fileno)
>> print("Python event")
>> print(event)
>> v = os.read(pr,64)
>> print("Pipe value")
>> print(v)
>> 
>> The child process correctly receives the signal from ep.poll and correctly reads the data in the pipe, but then it continues looping. For example, when I put in a timeout:
>> 
>> child is looping
>> Python fileno
>> 4
>> Python event
>> 1
>> Pipe value
>> b'10\x00'
>> The C code does not need to write a 0 bytes at the end.
>> I assume the 0 is from the end of a C string.
>> UDS messages have a length.
>> In the C just write 2 byes in the case.
>> 
>> Barry
>> child is looping
>> child is looping
>> 
>> That suggests that a while True loop is not the right thing to do in this case. My question is, what type of process loop is best for this situation? The multiprocessing, asyncio and subprocess libraries are very extensive, and it would help if someone could suggest the best alternative for what I am doing here. 
>> 
>> Thanks very much for any ideas. 
>> 
>> 
>> -- 
>> https://mail.python.org/mailman/listinfo/python-list <https://mail.python.org/mailman/listinfo/python-list>
>> 
> 
> 


From wasia.maya at gmail.com  Sat Dec  4 15:40:50 2021
From: wasia.maya at gmail.com (Wasia Maya)
Date: Sat, 4 Dec 2021 12:40:50 -0800 (PST)
Subject: TypeError: 'bytes' object is not callable error while trying to
 converting to bytes.
In-Reply-To: <mailman.12687.1407303433.18130.python-list@python.org>
References: <02baad14-39b1-41a0-a47d-28772c97ea54@googlegroups.com>
 <CAPTjJmqZdHdx7VJf7HMEUKe8J7uCH2HuKN1PO2i4azHygFGuDA@mail.gmail.com>
 <77C4289C-FBCD-449F-8460-B33029A4A236@gmail.com>
 <mailman.12687.1407303433.18130.python-list@python.org>
Message-ID: <09de7280-734f-4498-ae70-4189a7374f8bn@googlegroups.com>

You have assigned a bytes value to the name bytes:

>>> bytes([10, 20, 30, 40])
b'\n\x14\x1e('
>>> bytes = bytes([10, 20, 30, 40])
>>> bytes([10, 20, 30, 40])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'bytes' object is not callable
bytes is now bound to the value b'\n\x14\x1e(', which is not callable. This global is shadowing the built-in. Delete it:

del bytes

From vinay_sajip at yahoo.co.uk  Sun Dec  5 08:17:19 2021
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sun, 5 Dec 2021 13:17:19 +0000 (UTC)
Subject: ANN: A new library for encryption and signing has been released.
References: <235606996.12979309.1638710239462.ref@mail.yahoo.com>
Message-ID: <235606996.12979309.1638710239462@mail.yahoo.com>

A new library called pagesign (Python-age-sign) has been released on PyPI [1].
It covers similar functionality to python-gnupg, but uses the modern encryption tool
age [2] and the modern signing tool minisign [3]. The initial release allows you to:

* Create and manage identities (which are containers for the keys used for
? encryption/decryption and signing/verification).
* Encrypt and decrypt files to multiple recipients.
* Sign files and verify signatures.

This release has been signed with my code signing key:

Vinay Sajip (CODE SIGNING KEY) <vinay_sajip at yahoo.co.uk>
Fingerprint: CA74 9061 914E AC13 8E66 EADB 9147 B477 339A 9B86

Recent changes to PyPI don't show the GPG signature with the download links.
An alternative download source where the signatures are available is at [4].
Documentation is available at [5].

As always, your feedback is most welcome (especially bug reports [6],
patches and suggestions for improvement, or any other points).

Enjoy!

Cheers

Vinay Sajip

[1] https://pypi.org/project/pagesign/0.1.0/
[2] https://age-encryption.org/
[3] https://jedisct1.github.io/minisign/
[4] https://bitbucket.org/vinay.sajip/pagesign/downloads/
[5] https://docs.red-dove.com/pagesign/
[6] https://github.com/vsajip/pagesign/issues/new/choose


From jenkris at tutanota.com  Sun Dec  5 12:54:27 2021
From: jenkris at tutanota.com (Jen Kris)
Date: Sun, 5 Dec 2021 18:54:27 +0100 (CET)
Subject: Python child process in while True loop blocks parent
In-Reply-To: <27742189-51F0-49A8-ACA7-344EFCD2B1BD@barrys-emacs.org>
References: <MphOu5i--3-2@tutanota.com>
 <B05343F2-30B5-470C-8CFC-EBFA7C5EFD6A@barrys-emacs.org>
 <MphoqOf--3-2@tutanota.com>
 <7A3CE5EB-C53A-404D-BF32-77CF0243BF3D@barrys-emacs.org>
 <MpqihHm--3-2@tutanota.com>
 <27742189-51F0-49A8-ACA7-344EFCD2B1BD@barrys-emacs.org>
Message-ID: <MqAitZV--3-2@tutanota.com>

Thanks for your comments.? 

I put the Python program on its own pthread, and call a small C program to fork-execv to call the Python program as a child process.? I revised the Python program to be a multiprocessing loop using the Python multiprocessing module.? That bypasses the GIL and allows Python to run concurrently with C.? So far so good.? 

Next I will use Linux pipes, not Python multiprocessing pipes, for IPC between Python and C.? Multiprocessing pipes are (as far as I can tell) only for commo between two Python processes.? I will have the parent thread send a signal through the pipe to the child process to exit when the parent thread is ready to exit, then call wait() to finalize the child process.? 

I will reply back when it's finished and post the code so you can see what I have done.? 

Thanks again.? 

Jen


Dec 4, 2021, 09:22 by barry at barrys-emacs.org:

>
>
>> On 1 Dec 2021, at 16:01, Jen Kris <>> jenkris at tutanota.com>> > wrote:
>>
>> Thanks for your comment re blocking.? 
>>
>> I removed pipes from the Python and C programs to see if it blocks without them, and it does.
>>
>> It looks now like the problem is not pipes.
>>
>
> Ok.
>
>
>> I use fork() and execv() in C to run Python in a child process, but the Python process blocks
>>
>
> Use strace on the parent process to see what is happening.
> You will need to use the option to follow subprocesses so that you can see what goes on in the python process.
>
> See man strace and the?--follow-forks and?--output-separately options.
> That will allow you to find the blocking system call that your code is making.
>
>
>> because fork() does not create a new thread, so the Python global interpreter lock (GIL) prevents the C program from running once Python starts.
>>
>
> Not sure why you think this.
>
>
>> ? So the solution appears to be run Python in a separate thread, which I can do with pthread create.
>>
>> ? See "Thread State and the Global Interpreter Lock" >> https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock>>  and the sections below that "Non-Python created threads" and "Cautions about fork()."?
>>
>
> I take it you mean that in the parent you think that using pthreads will affect python after the exec() call?
> I does not. After exec() the process has one main thread create by the kernel and a new address space as defined by the /usr/bin/python.
> The only state that in inherited from the parent are open file descriptors, the current working directory and security state like UID, GID.
>
>
>> I'm working on that today and I hope all goes well :)?
>>
>
> You seem to be missing background information on how processes work.
> Maybe "Advanced Programming in the UNIX Environment"?> would be helpful?
>
> https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk> ?
>
> It's a great book and covers a wide range of Unix systems programming topics.
>
> Have you created a small C program that just does the fork and exec of a python program to test out your assumptions?
> If not I recommend that you do.
>
> Barry
>
>
>
>>
>>
>>
>> Nov 30, 2021, 11:42 by >> barry at barrys-emacs.org>> :
>>
>>>
>>>
>>>
>>>> On 29 Nov 2021, at 22:31, Jen Kris <>>>> jenkris at tutanota.com>>>> > wrote:
>>>>
>>>> Thanks to you and Cameron for your replies.? The C side has an epoll_ctl set, but no event loop to handle it yet.? I'm putting that in now with a pipe write in Python-- as Cameron pointed out that is the likely source of blocking on C.? The pipes are opened as rdwr in Python because that's nonblocking by default.? The child will become more complex, but not in a way that affects polling.? And thanks for the tip about the c-string termination.?
>>>>
>>>>
>>>
>>> flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK.
>>> You should not use O_RDWR when you only need O_RDONLY access or only O_WRONLY access.
>>>
>>> You may find
>>>
>>> man 2 open
>>>
>>> useful to understand in detail what is behind os.open().
>>>
>>> Barry
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>> Nov 29, 2021, 14:12 by >>>> barry at barrys-emacs.org>>>> :
>>>>
>>>>>
>>>>>
>>>>>> On 29 Nov 2021, at 20:36, Jen Kris via Python-list <>>>>>> python-list at python.org>>>>>> > wrote:
>>>>>>
>>>>>> ?I have a C program that forks to create a child process and uses execv to call a Python program.  The Python program communicates with the parent process (in C) through a FIFO pipe monitored with epoll(). 
>>>>>>
>>>>>> The Python child process is in a while True loop, which is intended to keep it running while the parent process proceeds, and perform functions for the C program only at intervals when the parent sends data to the child -- similar to a daemon process. 
>>>>>>
>>>>>> The C process writes to its end of the pipe and the child process reads it, but then the child process continues to loop, thereby blocking the parent. 
>>>>>>
>>>>>> This is the Python code:
>>>>>>
>>>>>> #!/usr/bin/python3
>>>>>> import os
>>>>>> import select
>>>>>>
>>>>>> #Open the named pipes
>>>>>> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
>>>>>>
>>>>> Why open rdwr if you are only going to read the pipe?
>>>>>
>>>>>> pw = os.open('/tmp/Pipe_02', os.O_RDWR)
>>>>>>
>>>>> Only need to open for write.
>>>>>
>>>>>>
>>>>>> ep = select.epoll(-1)
>>>>>> ep.register(pr, select.EPOLLIN)
>>>>>>
>>>>>
>>>>> Is the only thing that the child does this:
>>>>> 1. Read message from pr
>>>>> 2. Process message
>>>>> 3. Write result to pw.
>>>>> 4. Loop from 1
>>>>>
>>>>> If so as Cameron said you do not need to worry about the poll.
>>>>> Do you plan for the child to become more complex?
>>>>>
>>>>>>
>>>>>> while True:
>>>>>>
>>>>>> events = ep.poll(timeout=2.5, maxevents=-1)
>>>>>> #events = ep.poll(timeout=None, maxevents=-1)
>>>>>>
>>>>>> print("child is looping")
>>>>>>
>>>>>> for fileno, event in events:
>>>>>> print("Python fileno")
>>>>>> print(fileno)
>>>>>> print("Python event")
>>>>>> print(event)
>>>>>> v = os.read(pr,64)
>>>>>> print("Pipe value")
>>>>>> print(v)
>>>>>>
>>>>>> The child process correctly receives the signal from ep.poll and correctly reads the data in the pipe, but then it continues looping.  For example, when I put in a timeout:
>>>>>>
>>>>>> child is looping
>>>>>> Python fileno
>>>>>> 4
>>>>>> Python event
>>>>>> 1
>>>>>> Pipe value
>>>>>> b'10\x00'
>>>>>>
>>>>> The C code does not need to write a 0 bytes at the end.
>>>>> I assume the 0 is from the end of a C string.
>>>>> UDS messages have a length.
>>>>> In the C just write 2 byes in the case.
>>>>>
>>>>> Barry
>>>>>
>>>>>> child is looping
>>>>>> child is looping
>>>>>>
>>>>>> That suggests that a while True loop is not the right thing to do in this case.  My question is, what type of process loop is best for this situation?  The multiprocessing, asyncio and subprocess libraries are very extensive, and it would help if someone could suggest the best alternative for what I am doing here. 
>>>>>>
>>>>>> Thanks very much for any ideas. 
>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> https://mail.python.org/mailman/listinfo/python-list
>>>>>>
>>>>
>>>>
>>
>>


From barry at barrys-emacs.org  Sun Dec  5 18:19:50 2021
From: barry at barrys-emacs.org (Barry)
Date: Sun, 5 Dec 2021 23:19:50 +0000
Subject: Python child process in while True loop blocks parent
In-Reply-To: <MqAitZV--3-2@tutanota.com>
References: <MqAitZV--3-2@tutanota.com>
Message-ID: <0633A53B-A722-4CEE-8F20-E2D0AEBB8D01@barrys-emacs.org>



> On 5 Dec 2021, at 17:54, Jen Kris <jenkris at tutanota.com> wrote:
> 
> ?
> Thanks for your comments.  
> 
> I put the Python program on its own pthread, and call a small C program to fork-execv to call the Python program as a child process. 

What do you mean by putting python in it?s own pthread?
Are you embedding python in an other program?

Barry


> I revised the Python program to be a multiprocessing loop using the Python multiprocessing module.  That bypasses the GIL and allows Python to run concurrently with C.  So far so good.  
> 
> Next I will use Linux pipes, not Python multiprocessing pipes, for IPC between Python and C.  Multiprocessing pipes are (as far as I can tell) only for commo between two Python processes.  I will have the parent thread send a signal through the pipe to the child process to exit when the parent thread is ready to exit, then call wait() to finalize the child process.  
> 
> I will reply back when it's finished and post the code so you can see what I have done. 
> 
> Thanks again.  
> 
> Jen
> 
> 
> Dec 4, 2021, 09:22 by barry at barrys-emacs.org:
> 
>>> On 1 Dec 2021, at 16:01, Jen Kris <jenkris at tutanota.com> wrote:
>>> 
>>> Thanks for your comment re blocking.  
>>> 
>>> I removed pipes from the Python and C programs to see if it blocks without them, and it does.
>>> It looks now like the problem is not pipes.
>> 
>> Ok.
>> 
>> I use fork() and execv() in C to run Python in a child process, but the Python process blocks
> 
> Use strace on the parent process to see what is happening.
> You will need to use the option to follow subprocesses so that you can see what goes on in the python process.
> 
> See man strace and the --follow-forks and --output-separately options.
> That will allow you to find the blocking system call that your code is making.
> 
>> because fork() does not create a new thread, so the Python global interpreter lock (GIL) prevents the C program from running once Python starts.
> 
> Not sure why you think this.
> 
>>   So the solution appears to be run Python in a separate thread, which I can do with pthread create.
>>   See "Thread State and the Global Interpreter Lock" https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock and the sections below that "Non-Python created threads" and "Cautions about fork()." 
> 
> I take it you mean that in the parent you think that using pthreads will affect python after the exec() call?
> I does not. After exec() the process has one main thread create by the kernel and a new address space as defined by the /usr/bin/python.
> The only state that in inherited from the parent are open file descriptors, the current working directory and security state like UID, GID.
> 
>> I'm working on that today and I hope all goes well :) 
> 
> You seem to be missing background information on how processes work.
> Maybe "Advanced Programming in the UNIX Environment" would be helpful?
> 
> https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk 
> 
> It's a great book and covers a wide range of Unix systems programming topics.
> 
> Have you created a small C program that just does the fork and exec of a python program to test out your assumptions?
> If not I recommend that you do.
> 
> Barry
> 
> 
>> 
>> 
>> 
>> Nov 30, 2021, 11:42 by barry at barrys-emacs.org:
>> 
>> 
>>> On 29 Nov 2021, at 22:31, Jen Kris <jenkris at tutanota.com> wrote:
>>> 
>>> Thanks to you and Cameron for your replies.  The C side has an epoll_ctl set, but no event loop to handle it yet.  I'm putting that in now with a pipe write in Python-- as Cameron pointed out that is the likely source of blocking on C.  The pipes are opened as rdwr in Python because that's nonblocking by default.  The child will become more complex, but not in a way that affects polling.  And thanks for the tip about the c-string termination. 
>>> 
>> 
>> flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK.
>> You should not use O_RDWR when you only need O_RDONLY access or only O_WRONLY access.
>> 
>> You may find
>> 
>> man 2 open
>> 
>> useful to understand in detail what is behind os.open().
>> 
>> Barry
>> 
>> 
>> 
>>> 
>>> 
>>> Nov 29, 2021, 14:12 by barry at barrys-emacs.org:
>>> 
>>> On 29 Nov 2021, at 20:36, Jen Kris via Python-list <python-list at python.org> wrote:
>>> 
>>> ?I have a C program that forks to create a child process and uses execv to call a Python program. The Python program communicates with the parent process (in C) through a FIFO pipe monitored with epoll(). 
>>> 
>>> The Python child process is in a while True loop, which is intended to keep it running while the parent process proceeds, and perform functions for the C program only at intervals when the parent sends data to the child -- similar to a daemon process. 
>>> 
>>> The C process writes to its end of the pipe and the child process reads it, but then the child process continues to loop, thereby blocking the parent. 
>>> 
>>> This is the Python code:
>>> 
>>> #!/usr/bin/python3
>>> import os
>>> import select
>>> 
>>> #Open the named pipes
>>> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
>>> Why open rdwr if you are only going to read the pipe?
>>> pw = os.open('/tmp/Pipe_02', os.O_RDWR)
>>> Only need to open for write.
>>> 
>>> ep = select.epoll(-1)
>>> ep.register(pr, select.EPOLLIN)
>>> 
>>> Is the only thing that the child does this:
>>> 1. Read message from pr
>>> 2. Process message
>>> 3. Write result to pw.
>>> 4. Loop from 1
>>> 
>>> If so as Cameron said you do not need to worry about the poll.
>>> Do you plan for the child to become more complex?
>>> 
>>> while True:
>>> 
>>> events = ep.poll(timeout=2.5, maxevents=-1)
>>> #events = ep.poll(timeout=None, maxevents=-1)
>>> 
>>> print("child is looping")
>>> 
>>> for fileno, event in events:
>>> print("Python fileno")
>>> print(fileno)
>>> print("Python event")
>>> print(event)
>>> v = os.read(pr,64)
>>> print("Pipe value")
>>> print(v)
>>> 
>>> The child process correctly receives the signal from ep.poll and correctly reads the data in the pipe, but then it continues looping. For example, when I put in a timeout:
>>> 
>>> child is looping
>>> Python fileno
>>> 4
>>> Python event
>>> 1
>>> Pipe value
>>> b'10\x00'
>>> The C code does not need to write a 0 bytes at the end.
>>> I assume the 0 is from the end of a C string.
>>> UDS messages have a length.
>>> In the C just write 2 byes in the case.
>>> 
>>> Barry
>>> child is looping
>>> child is looping
>>> 
>>> That suggests that a while True loop is not the right thing to do in this case. My question is, what type of process loop is best for this situation? The multiprocessing, asyncio and subprocess libraries are very extensive, and it would help if someone could suggest the best alternative for what I am doing here. 
>>> 
>>> Thanks very much for any ideas. 
>>> 
>>> 
>>> -- 
>>> https://mail.python.org/mailman/listinfo/python-list
>>> 
>> 
>> 
> 
> 

From jenkris at tutanota.com  Sun Dec  5 18:51:13 2021
From: jenkris at tutanota.com (Jen Kris)
Date: Mon, 6 Dec 2021 00:51:13 +0100 (CET)
Subject: Python child process in while True loop blocks parent
In-Reply-To: <0633A53B-A722-4CEE-8F20-E2D0AEBB8D01@barrys-emacs.org>
References: <MphOu5i--3-2@tutanota.com>
 <B05343F2-30B5-470C-8CFC-EBFA7C5EFD6A@barrys-emacs.org>
 <MphoqOf--3-2@tutanota.com>
 <7A3CE5EB-C53A-404D-BF32-77CF0243BF3D@barrys-emacs.org>
 <MpqihHm--3-2@tutanota.com>
 <27742189-51F0-49A8-ACA7-344EFCD2B1BD@barrys-emacs.org>
 <MqAitZV--3-2@tutanota.com>
 <0633A53B-A722-4CEE-8F20-E2D0AEBB8D01@barrys-emacs.org>
Message-ID: <MqC-YXO--B-2@tutanota.com>

By embedding, I think you may be referring to embedding Python in a C program with the Python C API.?? That's not what I'm doing here -- I'm not using the Python C API.? The C program creates two threads (using pthreads), one for itself and one for the child process.? On creation, the second pthread is pointed to a C program that calls fork-execv to run the Python program.? That way Python runs on a separate thread.? The multiprocessing library "effectively side-step[s] the Global Interpreter Lock by using subprocesses instead of threads."? https://docs.python.org/3/library/multiprocessing.html.? This way I can get the Python functionality I want on call from the C program through pipes and shared memory.? 

I don't want to use the C API because I will be making certain library calls from the C program, and the syntax is much easier with native Python code than with C API code.?

I hope that clarifies what I'm doing.?

Jen



Dec 5, 2021, 15:19 by barry at barrys-emacs.org:

>
>
>
>
>> On 5 Dec 2021, at 17:54, Jen Kris <jenkris at tutanota.com> wrote:
>>
>> ?  
>> Thanks for your comments.? 
>>
>> I put the Python program on its own pthread, and call a small C program to fork-execv to call the Python program as a child process.?
>>
>
> What do you mean by putting python in it?s own pthread?
> Are you embedding python in an other program?
>
> Barry
>
>
>
>> I revised the Python program to be a multiprocessing loop using the Python multiprocessing module.? That bypasses the GIL and allows Python to run concurrently with C.? So far so good.? 
>>
>> Next I will use Linux pipes, not Python multiprocessing pipes, for IPC between Python and C.? Multiprocessing pipes are (as far as I can tell) only for commo between two Python processes.? I will have the parent thread send a signal through the pipe to the child process to exit when the parent thread is ready to exit, then call wait() to finalize the child process.? 
>>
>> I will reply back when it's finished and post the code so you can see what I have done.? 
>>
>> Thanks again.? 
>>
>> Jen
>>
>>
>> Dec 4, 2021, 09:22 by barry at barrys-emacs.org:
>>
>>>
>>>
>>>> On 1 Dec 2021, at 16:01, Jen Kris <>>>> jenkris at tutanota.com>>>> > wrote:
>>>>
>>>> Thanks for your comment re blocking.? 
>>>>
>>>> I removed pipes from the Python and C programs to see if it blocks without them, and it does.
>>>>
>>>> It looks now like the problem is not pipes.
>>>>
>>>
>>> Ok.
>>>
>>>
>>>> I use fork() and execv() in C to run Python in a child process, but the Python process blocks
>>>>
>>>
>>> Use strace on the parent process to see what is happening.
>>> You will need to use the option to follow subprocesses so that you can see what goes on in the python process.
>>>
>>> See man strace and the?--follow-forks and?--output-separately options.
>>> That will allow you to find the blocking system call that your code is making.
>>>
>>>
>>>> because fork() does not create a new thread, so the Python global interpreter lock (GIL) prevents the C program from running once Python starts.
>>>>
>>>
>>> Not sure why you think this.
>>>
>>>
>>>> ? So the solution appears to be run Python in a separate thread, which I can do with pthread create.
>>>>
>>>> ? See "Thread State and the Global Interpreter Lock" >>>> https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock>>>>  and the sections below that "Non-Python created threads" and "Cautions about fork()."?
>>>>
>>>
>>> I take it you mean that in the parent you think that using pthreads will affect python after the exec() call?
>>> I does not. After exec() the process has one main thread create by the kernel and a new address space as defined by the /usr/bin/python.
>>> The only state that in inherited from the parent are open file descriptors, the current working directory and security state like UID, GID.
>>>
>>>
>>>> I'm working on that today and I hope all goes well :)?
>>>>
>>>
>>> You seem to be missing background information on how processes work.
>>> Maybe "Advanced Programming in the UNIX Environment"?>>> would be helpful?
>>>
>>> https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk>>> ?
>>>
>>> It's a great book and covers a wide range of Unix systems programming topics.
>>>
>>> Have you created a small C program that just does the fork and exec of a python program to test out your assumptions?
>>> If not I recommend that you do.
>>>
>>> Barry
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>> Nov 30, 2021, 11:42 by >>>> barry at barrys-emacs.org>>>> :
>>>>
>>>>>
>>>>>
>>>>>
>>>>>> On 29 Nov 2021, at 22:31, Jen Kris <>>>>>> jenkris at tutanota.com>>>>>> > wrote:
>>>>>>
>>>>>> Thanks to you and Cameron for your replies.? The C side has an epoll_ctl set, but no event loop to handle it yet.? I'm putting that in now with a pipe write in Python-- as Cameron pointed out that is the likely source of blocking on C.? The pipes are opened as rdwr in Python because that's nonblocking by default.? The child will become more complex, but not in a way that affects polling.? And thanks for the tip about the c-string termination.?
>>>>>>
>>>>>>
>>>>>
>>>>> flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK.
>>>>> You should not use O_RDWR when you only need O_RDONLY access or only O_WRONLY access.
>>>>>
>>>>> You may find
>>>>>
>>>>> man 2 open
>>>>>
>>>>> useful to understand in detail what is behind os.open().
>>>>>
>>>>> Barry
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> Nov 29, 2021, 14:12 by >>>>>> barry at barrys-emacs.org>>>>>> :
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On 29 Nov 2021, at 20:36, Jen Kris via Python-list <>>>>>>>> python-list at python.org>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>> ?I have a C program that forks to create a child process and uses execv to call a Python program.  The Python program communicates with the parent process (in C) through a FIFO pipe monitored with epoll(). 
>>>>>>>>
>>>>>>>> The Python child process is in a while True loop, which is intended to keep it running while the parent process proceeds, and perform functions for the C program only at intervals when the parent sends data to the child -- similar to a daemon process. 
>>>>>>>>
>>>>>>>> The C process writes to its end of the pipe and the child process reads it, but then the child process continues to loop, thereby blocking the parent. 
>>>>>>>>
>>>>>>>> This is the Python code:
>>>>>>>>
>>>>>>>> #!/usr/bin/python3
>>>>>>>> import os
>>>>>>>> import select
>>>>>>>>
>>>>>>>> #Open the named pipes
>>>>>>>> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
>>>>>>>>
>>>>>>> Why open rdwr if you are only going to read the pipe?
>>>>>>>
>>>>>>>> pw = os.open('/tmp/Pipe_02', os.O_RDWR)
>>>>>>>>
>>>>>>> Only need to open for write.
>>>>>>>
>>>>>>>>
>>>>>>>> ep = select.epoll(-1)
>>>>>>>> ep.register(pr, select.EPOLLIN)
>>>>>>>>
>>>>>>>
>>>>>>> Is the only thing that the child does this:
>>>>>>> 1. Read message from pr
>>>>>>> 2. Process message
>>>>>>> 3. Write result to pw.
>>>>>>> 4. Loop from 1
>>>>>>>
>>>>>>> If so as Cameron said you do not need to worry about the poll.
>>>>>>> Do you plan for the child to become more complex?
>>>>>>>
>>>>>>>>
>>>>>>>> while True:
>>>>>>>>
>>>>>>>> events = ep.poll(timeout=2.5, maxevents=-1)
>>>>>>>> #events = ep.poll(timeout=None, maxevents=-1)
>>>>>>>>
>>>>>>>> print("child is looping")
>>>>>>>>
>>>>>>>> for fileno, event in events:
>>>>>>>> print("Python fileno")
>>>>>>>> print(fileno)
>>>>>>>> print("Python event")
>>>>>>>> print(event)
>>>>>>>> v = os.read(pr,64)
>>>>>>>> print("Pipe value")
>>>>>>>> print(v)
>>>>>>>>
>>>>>>>> The child process correctly receives the signal from ep.poll and correctly reads the data in the pipe, but then it continues looping.  For example, when I put in a timeout:
>>>>>>>>
>>>>>>>> child is looping
>>>>>>>> Python fileno
>>>>>>>> 4
>>>>>>>> Python event
>>>>>>>> 1
>>>>>>>> Pipe value
>>>>>>>> b'10\x00'
>>>>>>>>
>>>>>>> The C code does not need to write a 0 bytes at the end.
>>>>>>> I assume the 0 is from the end of a C string.
>>>>>>> UDS messages have a length.
>>>>>>> In the C just write 2 byes in the case.
>>>>>>>
>>>>>>> Barry
>>>>>>>
>>>>>>>> child is looping
>>>>>>>> child is looping
>>>>>>>>
>>>>>>>> That suggests that a while True loop is not the right thing to do in this case.  My question is, what type of process loop is best for this situation?  The multiprocessing, asyncio and subprocess libraries are very extensive, and it would help if someone could suggest the best alternative for what I am doing here. 
>>>>>>>>
>>>>>>>> Thanks very much for any ideas. 
>>>>>>>>
>>>>>>>>
>>>>>>>> -- 
>>>>>>>> https://mail.python.org/mailman/listinfo/python-list
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>


From hjp-python at hjp.at  Sun Dec  5 21:08:31 2021
From: hjp-python at hjp.at (Peter J. Holzer)
Date: Mon, 6 Dec 2021 03:08:31 +0100
Subject: Python child process in while True loop blocks parent
In-Reply-To: <MqC-YXO--B-2@tutanota.com>
References: <MphOu5i--3-2@tutanota.com>
 <B05343F2-30B5-470C-8CFC-EBFA7C5EFD6A@barrys-emacs.org>
 <MphoqOf--3-2@tutanota.com>
 <7A3CE5EB-C53A-404D-BF32-77CF0243BF3D@barrys-emacs.org>
 <MpqihHm--3-2@tutanota.com>
 <27742189-51F0-49A8-ACA7-344EFCD2B1BD@barrys-emacs.org>
 <MqAitZV--3-2@tutanota.com>
 <0633A53B-A722-4CEE-8F20-E2D0AEBB8D01@barrys-emacs.org>
 <MqC-YXO--B-2@tutanota.com>
Message-ID: <20211206020831.ytkglawzaeoueyhh@hjp.at>

On 2021-12-06 00:51:13 +0100, Jen Kris via Python-list wrote:
> The C program creates two threads (using pthreads), one for itself and
> one for the child process.? On creation, the second pthread is pointed
> to a C program that calls fork-execv to run the Python program.? That
> way Python runs on a separate thread.?

I think you have the relationship between processes and threads
backwards. A process consists of one or more threads. Fork creates a new
process and therefore also a new thread.

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp at hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://mail.python.org/pipermail/python-list/attachments/20211206/47612757/attachment.sig>

From barpasc at yahoo.com  Sun Dec  5 10:46:21 2021
From: barpasc at yahoo.com (Pascal B)
Date: Sun, 5 Dec 2021 15:46:21 +0000 (UTC)
Subject: python list files and folder using tkinter
References: <1358207403.13026476.1638719181691.ref@mail.yahoo.com>
Message-ID: <1358207403.13026476.1638719181691@mail.yahoo.com>

Hello,
I have already posted a message some time ago for this app. Since then, I didn't code in python or made any changes. I think before getting further with functionnalities a few things or the whole thing need to be changed.
For exemple, it would need a button to pick folders and maybe ask if the csv resulting file can be saved in the same folder.
And more important, files are listing ok in windows but not in linux after running it a few times.
https://github.com/barpasc/listfiles
code extract without indentation, see source script on github

elif vvchkboxF == 1: 
|   # *** FOLDERS AND FILES ONLY *** |
|


|   for root, dirs, files in os.walk(Lpath, topdown=False): |
|


|   ### calcul taille dossier |
|


|   size = 0 |
|


|   for x, y, z in os.walk(root): |
|


|   for i in z: |
|


|   ftmp_che = x + os.sep + i |
|


|   f_size += os.path.getsize(ftmp_che) |
|


|   ### ecriture taille dossier |
|


|   counter = root.count(os.path.sep) - counterPath |
|


|   vfile_name = root |
|


|   vfile_name = vfile_name + os.path.sep |
|


|   vfile_name = os.path.split(os.path.dirname(vfile_name))[1] |
|


|   vfile_name += os.path.sep |
|


|   if counter <= f_vscale: |
|


|   csv_contents += "%s;%s;%.0f;%.2f;%.2f;%.2f;%s\n" % (root, vfile_name, f_size, f_size/1024, f_size/1048576,f_size/1073741824, "folder") |
|


|  
 |
|


|   ### calcul +ecriture taille fichier |
|


|   for f in os.listdir(Lpath): |
|


|   path = os.path.join(Lpath, f) |
|


|   if os.path.isfile(path): |
|


|   f_size = 0 |
|


|   f_size = os.path.getsize(path) |
|


|   csv_contents += "%s;%s;%.0f;%.2f;%.2f;%.2f;%s\n" % (path, f, f_size, f_size/1024, f_size/1048576,f_size/1073741824, "file") |
|


|  
 |
|


|   fx_writeCSV_str(csv_contents) |
|

  print("job adv listing files ok")

From dfnsonfsduifb at gmx.de  Sun Dec  5 18:50:11 2021
From: dfnsonfsduifb at gmx.de (Johannes Bauer)
Date: Mon, 6 Dec 2021 00:50:11 +0100
Subject: threading and multiprocessing deadlock
Message-ID: <sojj7k$1ek1$1@gioia.aioe.org>

Hi there,

I'm a bit confused. In my scenario I a mixing threading with
multiprocessing. Threading by itself would be nice, but for GIL reasons
I need both, unfortunately. I've encountered a weird situation in which
multiprocessing Process()es which are started in a new thread don't
actually start and so they deadlock on join.

I've created a minimal example that demonstrates the issue. I'm running
on x86_64 Linux using Python 3.9.5 (default, May 11 2021, 08:20:37)
([GCC 10.3.0] on linux).

Here's the code:


import time
import multiprocessing
import threading

def myfnc():
	print("myfnc")

def run(result_queue, callback):
	result = callback()
	result_queue.put(result)

def start(fnc):
	def background_thread():
		queue = multiprocessing.Queue()
		proc = multiprocessing.Process(target = run, args = (queue, fnc))
		proc.start()
		print("join?")
		proc.join()
		print("joined.")
		result = queue.get()
	threading.Thread(target = background_thread).start()

start(myfnc)
start(myfnc)
start(myfnc)
start(myfnc)
while True:
	time.sleep(1)


What you'll see is that "join?" and "joined." nondeterministically does
*not* appear in pairs. For example:

join?
join?
myfnc
myfnc
join?
join?
joined.
joined.

What's worse is that when this happens and I Ctrl-C out of Python, the
started Thread is still running in the background:

$ ps ax | grep minimal
 370167 pts/0    S      0:00 python3 minimal.py
 370175 pts/2    S+     0:00 grep minimal

Can someone figure out what is going on there?

Best,
Johannes

From barry at barrys-emacs.org  Mon Dec  6 03:22:17 2021
From: barry at barrys-emacs.org (Barry)
Date: Mon, 6 Dec 2021 08:22:17 +0000
Subject: Python child process in while True loop blocks parent
In-Reply-To: <MqC-YXO--B-2@tutanota.com>
References: <MqC-YXO--B-2@tutanota.com>
Message-ID: <ED1BAC36-DAB9-4C8C-986B-56E27D78206E@barrys-emacs.org>



> On 5 Dec 2021, at 23:51, Jen Kris <jenkris at tutanota.com> wrote:
> 
> ?
> By embedding, I think you may be referring to embedding Python in a C program with the Python C API.   That's not what I'm doing here -- I'm not using the Python C API.  The C program creates two threads (using pthreads), one for itself and one for the child process.  On creation, the second pthread is pointed to a C program that calls fork-execv to run the Python program.  That way Python runs on a separate thread.  The multiprocessing library "effectively side-step[s] the Global Interpreter Lock by using subprocesses instead of threads."  https://docs.python.org/3/library/multiprocessing.html.  This way I can get the Python functionality I want on call from the C program through pipes and shared memory.  
> 
> I don't want to use the C API because I will be making certain library calls from the C program, and the syntax is much easier with native Python code than with C API code. 
> 
> I hope that clarifies what I'm doing. 

You are confusing the threading needs of a python program and extension written in C
and what you need to do to start a child process.

I see no benifit from calling fork and exec from a new thread. Indeed you may
encounter more issues.

What ever the bug is that you and trying to fix threading has nothing to do with it.

What if you change you code to run bash script instead of python?
Does you C program block them?

What did strace tell you was the reason that your C parent program blocked?

Barry



> 
> Jen
> 
> 
> 
> Dec 5, 2021, 15:19 by barry at barrys-emacs.org:
> 
> 
> 
>>> On 5 Dec 2021, at 17:54, Jen Kris <jenkris at tutanota.com> wrote:
>> ? 
>> Thanks for your comments.  
>> 
>> I put the Python program on its own pthread, and call a small C program to fork-execv to call the Python program as a child process. 
> 
> What do you mean by putting python in it?s own pthread?
> Are you embedding python in an other program?
> 
> Barry
> 
> 
>> I revised the Python program to be a multiprocessing loop using the Python multiprocessing module.  That bypasses the GIL and allows Python to run concurrently with C.  So far so good.  
>> 
>> Next I will use Linux pipes, not Python multiprocessing pipes, for IPC between Python and C.  Multiprocessing pipes are (as far as I can tell) only for commo between two Python processes.  I will have the parent thread send a signal through the pipe to the child process to exit when the parent thread is ready to exit, then call wait() to finalize the child process.  
>> 
>> I will reply back when it's finished and post the code so you can see what I have done.  
>> 
>> Thanks again.  
>> 
>> Jen
>> 
>> 
>> Dec 4, 2021, 09:22 by barry at barrys-emacs.org:
>> 
>>> On 1 Dec 2021, at 16:01, Jen Kris <jenkris at tutanota.com> wrote:
>>> 
>>> Thanks for your comment re blocking.  
>>> 
>>> I removed pipes from the Python and C programs to see if it blocks without them, and it does.
>>> It looks now like the problem is not pipes.
>> 
>> Ok.
>> 
>>> I use fork() and execv() in C to run Python in a child process, but the Python process blocks
>> 
>> Use strace on the parent process to see what is happening.
>> You will need to use the option to follow subprocesses so that you can see what goes on in the python process.
>> 
>> See man strace and the --follow-forks and --output-separately options.
>> That will allow you to find the blocking system call that your code is making.
>> 
>>> because fork() does not create a new thread, so the Python global interpreter lock (GIL) prevents the C program from running once Python starts.
>> 
>> Not sure why you think this.
>> 
>>>   So the solution appears to be run Python in a separate thread, which I can do with pthread create.
>>>   See "Thread State and the Global Interpreter Lock" https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock and the sections below that "Non-Python created threads" and "Cautions about fork()." 
>> 
>> I take it you mean that in the parent you think that using pthreads will affect python after the exec() call?
>> I does not. After exec() the process has one main thread create by the kernel and a new address space as defined by the /usr/bin/python.
>> The only state that in inherited from the parent are open file descriptors, the current working directory and security state like UID, GID.
>> 
>>> I'm working on that today and I hope all goes well :) 
>> 
>> You seem to be missing background information on how processes work.
>> Maybe "Advanced Programming in the UNIX Environment" would be helpful?
>> 
>> https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk 
>> 
>> It's a great book and covers a wide range of Unix systems programming topics.
>> 
>> Have you created a small C program that just does the fork and exec of a python program to test out your assumptions?
>> If not I recommend that you do.
>> 
>> Barry
>> 
>> 
>>> 
>>> 
>>> 
>>> Nov 30, 2021, 11:42 by barry at barrys-emacs.org:
>>> 
>>> 
>>>> On 29 Nov 2021, at 22:31, Jen Kris <jenkris at tutanota.com> wrote:
>>>> 
>>>> Thanks to you and Cameron for your replies.  The C side has an epoll_ctl set, but no event loop to handle it yet.  I'm putting that in now with a pipe write in Python-- as Cameron pointed out that is the likely source of blocking on C.  The pipes are opened as rdwr in Python because that's nonblocking by default.  The child will become more complex, but not in a way that affects polling.  And thanks for the tip about the c-string termination. 
>>>> 
>>> 
>>> flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK.
>>> You should not use O_RDWR when you only need O_RDONLY access or only O_WRONLY access.
>>> 
>>> You may find
>>> 
>>> man 2 open
>>> 
>>> useful to understand in detail what is behind os.open().
>>> 
>>> Barry
>>> 
>>> 
>>> 
>>>> 
>>>> 
>>>> Nov 29, 2021, 14:12 by barry at barrys-emacs.org:
>>>> 
>>>> On 29 Nov 2021, at 20:36, Jen Kris via Python-list <python-list at python.org> wrote:
>>>> 
>>>> ?I have a C program that forks to create a child process and uses execv to call a Python program. The Python program communicates with the parent process (in C) through a FIFO pipe monitored with epoll(). 
>>>> 
>>>> The Python child process is in a while True loop, which is intended to keep it running while the parent process proceeds, and perform functions for the C program only at intervals when the parent sends data to the child -- similar to a daemon process. 
>>>> 
>>>> The C process writes to its end of the pipe and the child process reads it, but then the child process continues to loop, thereby blocking the parent. 
>>>> 
>>>> This is the Python code:
>>>> 
>>>> #!/usr/bin/python3
>>>> import os
>>>> import select
>>>> 
>>>> #Open the named pipes
>>>> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
>>>> Why open rdwr if you are only going to read the pipe?
>>>> pw = os.open('/tmp/Pipe_02', os.O_RDWR)
>>>> Only need to open for write.
>>>> 
>>>> ep = select.epoll(-1)
>>>> ep.register(pr, select.EPOLLIN)
>>>> 
>>>> Is the only thing that the child does this:
>>>> 1. Read message from pr
>>>> 2. Process message
>>>> 3. Write result to pw.
>>>> 4. Loop from 1
>>>> 
>>>> If so as Cameron said you do not need to worry about the poll.
>>>> Do you plan for the child to become more complex?
>>>> 
>>>> while True:
>>>> 
>>>> events = ep.poll(timeout=2.5, maxevents=-1)
>>>> #events = ep.poll(timeout=None, maxevents=-1)
>>>> 
>>>> print("child is looping")
>>>> 
>>>> for fileno, event in events:
>>>> print("Python fileno")
>>>> print(fileno)
>>>> print("Python event")
>>>> print(event)
>>>> v = os.read(pr,64)
>>>> print("Pipe value")
>>>> print(v)
>>>> 
>>>> The child process correctly receives the signal from ep.poll and correctly reads the data in the pipe, but then it continues looping. For example, when I put in a timeout:
>>>> 
>>>> child is looping
>>>> Python fileno
>>>> 4
>>>> Python event
>>>> 1
>>>> Pipe value
>>>> b'10\x00'
>>>> The C code does not need to write a 0 bytes at the end.
>>>> I assume the 0 is from the end of a C string.
>>>> UDS messages have a length.
>>>> In the C just write 2 byes in the case.
>>>> 
>>>> Barry
>>>> child is looping
>>>> child is looping
>>>> 
>>>> That suggests that a while True loop is not the right thing to do in this case. My question is, what type of process loop is best for this situation? The multiprocessing, asyncio and subprocess libraries are very extensive, and it would help if someone could suggest the best alternative for what I am doing here. 
>>>> 
>>>> Thanks very much for any ideas. 
>>>> 
>>>> 
>>>> -- 
>>>> https://mail.python.org/mailman/listinfo/python-list
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 

From martinp.dipaola at gmail.com  Mon Dec  6 07:56:43 2021
From: martinp.dipaola at gmail.com (Martin Di Paola)
Date: Mon, 6 Dec 2021 12:56:43 +0000
Subject: threading and multiprocessing deadlock
In-Reply-To: <sojj7k$1ek1$1@gioia.aioe.org>
References: <sojj7k$1ek1$1@gioia.aioe.org>
Message-ID: <20211206125643.7g55nf4a45naji7u@gmail.com>

Hi!, in short your code should work.

I think that the join-joined problem is just an interpretation problem.

In pseudo code the background_thread function does:

def background_thread()
   # bla
   print("join?")
   # bla
   print("joined")

When running this function in parallel using threads, you will probably
get a few "join?" first before receiving any "joined?". That is because
the functions are running in parallel.

The order "join?" then "joined" is preserved within a thread but not
preserved globally.

Now, I see another issue in the output (and perhaps you was asking about 
this one):

join?
join?
myfnc
myfnc
join?
join?
joined.
joined.

So you have 4 "join?" that correspond to the 4 background_thread 
function calls in threads but only 2 "myfnc" and 2 "joined".

Could be possible that the output is truncated by accident?

I ran the same program and I got a reasonable output (4 "join?", "myfnc" 
and "joined"):

join?
join?
myfnc
join?
myfnc
join?
joined.
myfnc
joined.
joined.
myfnc
joined.

Another issue that I see is that you are not joining the threads that 
you spawned (background_thread functions).

I hope that this can guide you to fix or at least narrow the issue.

Thanks,
Martin.


On Mon, Dec 06, 2021 at 12:50:11AM +0100, Johannes Bauer wrote:
>Hi there,
>
>I'm a bit confused. In my scenario I a mixing threading with
>multiprocessing. Threading by itself would be nice, but for GIL reasons
>I need both, unfortunately. I've encountered a weird situation in which
>multiprocessing Process()es which are started in a new thread don't
>actually start and so they deadlock on join.
>
>I've created a minimal example that demonstrates the issue. I'm running
>on x86_64 Linux using Python 3.9.5 (default, May 11 2021, 08:20:37)
>([GCC 10.3.0] on linux).
>
>Here's the code:
>
>
>import time
>import multiprocessing
>import threading
>
>def myfnc():
>	print("myfnc")
>
>def run(result_queue, callback):
>	result = callback()
>	result_queue.put(result)
>
>def start(fnc):
>	def background_thread():
>		queue = multiprocessing.Queue()
>		proc = multiprocessing.Process(target = run, args = (queue, fnc))
>		proc.start()
>		print("join?")
>		proc.join()
>		print("joined.")
>		result = queue.get()
>	threading.Thread(target = background_thread).start()
>
>start(myfnc)
>start(myfnc)
>start(myfnc)
>start(myfnc)
>while True:
>	time.sleep(1)
>
>
>What you'll see is that "join?" and "joined." nondeterministically does
>*not* appear in pairs. For example:
>
>join?
>join?
>myfnc
>myfnc
>join?
>join?
>joined.
>joined.
>
>What's worse is that when this happens and I Ctrl-C out of Python, the
>started Thread is still running in the background:
>
>$ ps ax | grep minimal
> 370167 pts/0    S      0:00 python3 minimal.py
> 370175 pts/2    S+     0:00 grep minimal
>
>Can someone figure out what is going on there?
>
>Best,
>Johannes
>-- 
>https://mail.python.org/mailman/listinfo/python-list

From dfnsonfsduifb at gmx.de  Mon Dec  6 10:05:17 2021
From: dfnsonfsduifb at gmx.de (Johannes Bauer)
Date: Mon, 6 Dec 2021 16:05:17 +0100
Subject: threading and multiprocessing deadlock
References: <sojj7k$1ek1$1@gioia.aioe.org>
 <20211206125643.7g55nf4a45naji7u@gmail.com>
 <mailman.19.1638795411.15287.python-list@python.org>
Message-ID: <sol8rd$arj$1@gioia.aioe.org>

Am 06.12.21 um 13:56 schrieb Martin Di Paola:
> Hi!, in short your code should work.
> 
> I think that the join-joined problem is just an interpretation problem.
> 
> In pseudo code the background_thread function does:
> 
> def background_thread()
> ? # bla
> ? print("join?")
> ? # bla
> ? print("joined")
> 
> When running this function in parallel using threads, you will probably
> get a few "join?" first before receiving any "joined?". That is because
> the functions are running in parallel.
> 
> The order "join?" then "joined" is preserved within a thread but not
> preserved globally.

Yes, completely understood and really not the issue. That these pairs
are not in sequence is fine.

> Now, I see another issue in the output (and perhaps you was asking about
> this one):
> 
> join?
> join?
> myfnc
> myfnc
> join?
> join?
> joined.
> joined.
> 
> So you have 4 "join?" that correspond to the 4 background_thread
> function calls in threads but only 2 "myfnc" and 2 "joined".

Exactly that is the issue. Then it hangs. Deadlocked.

> Could be possible that the output is truncated by accident?

No. This is it. The exact output varies, but when it hangs, it always
also does not execute the function (note the lack of "myfnc"). For example:

join?
join?
myfnc
join?
myfnc
join?
myfnc
joined.
joined.
joined.

(only three threads get started there)

join?
myfnc
join?
join?
join?
joined.

(this time only a single one made it)

join?
join?
join?
myfnc
join?
myfnc
joined.
myfnc
joined.
joined.

(three get started)

> I ran the same program and I got a reasonable output (4 "join?", "myfnc"
> and "joined"):
> 
> join?
> join?
> myfnc
> join?
> myfnc
> join?
> joined.
> myfnc
> joined.
> joined.
> myfnc
> joined.

This happens to me occasionally, but most of the time one of the
processes deadlocks. Did you consistently get four of each? What
OS/Python version were you using?

> Another issue that I see is that you are not joining the threads that
> you spawned (background_thread functions).

True, I kindof assumed those would be detached threads.

> I hope that this can guide you to fix or at least narrow the issue.

Depending on what OS/Python version you're using, that points in that
direction and kindof reinforces my belief that the code is correct.

Very curious.

Thanks & all the best,
Joe

From jenkris at tutanota.com  Mon Dec  6 12:09:34 2021
From: jenkris at tutanota.com (Jen Kris)
Date: Mon, 6 Dec 2021 18:09:34 +0100 (CET)
Subject: Python child process in while True loop blocks parent
In-Reply-To: <20211206020831.ytkglawzaeoueyhh@hjp.at>
References: <MphOu5i--3-2@tutanota.com>
 <B05343F2-30B5-470C-8CFC-EBFA7C5EFD6A@barrys-emacs.org>
 <MphoqOf--3-2@tutanota.com>
 <7A3CE5EB-C53A-404D-BF32-77CF0243BF3D@barrys-emacs.org>
 <MpqihHm--3-2@tutanota.com>
 <27742189-51F0-49A8-ACA7-344EFCD2B1BD@barrys-emacs.org>
 <MqAitZV--3-2@tutanota.com>
 <0633A53B-A722-4CEE-8F20-E2D0AEBB8D01@barrys-emacs.org>
 <MqC-YXO--B-2@tutanota.com> <20211206020831.ytkglawzaeoueyhh@hjp.at>
Message-ID: <MqFiCj---3-2@tutanota.com>

I can't find any support for your comment that "Fork creates a new
process and therefore also a new thread."? From the Linux man pages https://www.man7.org/linux/man-pages/man2/fork.2.html, "The child process is created with a single thread?the one that called fork()."?

I have a one-core one-thread instance at Digital Ocean available running Ubuntu 18.04.? I can fork and create a new process on it, but it doesn't create a new thread because it doesn't have one available.?

You may also want to see "Forking vs Threading" (https://www.geekride.com/fork-forking-vs-threading-thread-linux-kernel), "Fork vs Thread" (https://medium.com/obscure-system/fork-vs-thread-38e09ec099e2), and "Linux process and thread" (https://zliu.org/post/linux-process-and-thread) ("This means that to create a normal process fork() is used that further calls clone() with appropriate arguments while to create a thread or LWP, a function from pthread library calls clone() with relvant flags. So, the main difference is generated by using different flags that can be passed to clone() funciton(to be exact, it is a system call").?

You may be confused by the fact that threads are called light-weight processes.?

Or maybe I'm confused :)

If you have other information, please let me know.? Thanks.?

Jen


Dec 5, 2021, 18:08 by hjp-python at hjp.at:

> On 2021-12-06 00:51:13 +0100, Jen Kris via Python-list wrote:
>
>> The C program creates two threads (using pthreads), one for itself and
>> one for the child process.? On creation, the second pthread is pointed
>> to a C program that calls fork-execv to run the Python program.? That
>> way Python runs on a separate thread.?
>>
>
> I think you have the relationship between processes and threads
> backwards. A process consists of one or more threads. Fork creates a new
> process and therefore also a new thread.
>
>  hp
>
> -- 
>  _  | Peter J. Holzer    | Story must make more sense than reality.
> |_|_) |                    |
> | |   | hjp at hjp.at         |    -- Charles Stross, "Creative writing
> __/   | http://www.hjp.at/ |       challenge!"
>


From rosuav at gmail.com  Mon Dec  6 13:27:43 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 7 Dec 2021 05:27:43 +1100
Subject: Python child process in while True loop blocks parent
In-Reply-To: <MqFiCj---3-2@tutanota.com>
References: <MphOu5i--3-2@tutanota.com>
 <B05343F2-30B5-470C-8CFC-EBFA7C5EFD6A@barrys-emacs.org>
 <MphoqOf--3-2@tutanota.com>
 <7A3CE5EB-C53A-404D-BF32-77CF0243BF3D@barrys-emacs.org>
 <MpqihHm--3-2@tutanota.com>
 <27742189-51F0-49A8-ACA7-344EFCD2B1BD@barrys-emacs.org>
 <MqAitZV--3-2@tutanota.com>
 <0633A53B-A722-4CEE-8F20-E2D0AEBB8D01@barrys-emacs.org>
 <MqC-YXO--B-2@tutanota.com> <20211206020831.ytkglawzaeoueyhh@hjp.at>
 <MqFiCj---3-2@tutanota.com>
Message-ID: <CAPTjJmqS+UE_-PPaz=1eS-juAAM2DgxXWY6wvEp3WJqsb8ErxA@mail.gmail.com>

On Tue, Dec 7, 2021 at 4:10 AM Jen Kris via Python-list
<python-list at python.org> wrote:
>
> I can't find any support for your comment that "Fork creates a new
> process and therefore also a new thread."  From the Linux man pages https://www.man7.org/linux/man-pages/man2/fork.2.html, "The child process is created with a single thread?the one that called fork()."
>
> I have a one-core one-thread instance at Digital Ocean available running Ubuntu 18.04.  I can fork and create a new process on it, but it doesn't create a new thread because it doesn't have one available.
>

A CPU core is capable of running one or more threads *concurrently*,
and having multiple cores obviously multiplies that out. But for what
you're doing here, there's no material difference between threads
running concurrently and threads switching out between themselves
(other than performance, of course). From the operating system and
application perspectives, a "thread" is one runnable thread.

Every Unix process must contain a minimum of one thread. As a rough
rule of thumb, a process owns resources, a thread runs code. Without a
thread, you can't run any code.

Forking a process creates a child process and leaves the parent
running. The child process must by definition have a minimum of one
thread, and the man page is stating that one thread is all it gets.

Hope that helps explain things a bit. I don't know exactly what's
going on in your code, but maybe that'll clarify the word "thread".

ChrisA

From barry at barrys-emacs.org  Mon Dec  6 13:37:01 2021
From: barry at barrys-emacs.org (Barry Scott)
Date: Mon, 6 Dec 2021 18:37:01 +0000
Subject: Python child process in while True loop blocks parent
In-Reply-To: <MqFiCj---3-2@tutanota.com>
References: <MphOu5i--3-2@tutanota.com>
 <B05343F2-30B5-470C-8CFC-EBFA7C5EFD6A@barrys-emacs.org>
 <MphoqOf--3-2@tutanota.com>
 <7A3CE5EB-C53A-404D-BF32-77CF0243BF3D@barrys-emacs.org>
 <MpqihHm--3-2@tutanota.com>
 <27742189-51F0-49A8-ACA7-344EFCD2B1BD@barrys-emacs.org>
 <MqAitZV--3-2@tutanota.com>
 <0633A53B-A722-4CEE-8F20-E2D0AEBB8D01@barrys-emacs.org>
 <MqC-YXO--B-2@tutanota.com> <20211206020831.ytkglawzaeoueyhh@hjp.at>
 <MqFiCj---3-2@tutanota.com>
Message-ID: <3065988D-E2CF-4C36-B319-1DEC87B864E3@barrys-emacs.org>



> On 6 Dec 2021, at 17:09, Jen Kris via Python-list <python-list at python.org> wrote:
> 
> I can't find any support for your comment that "Fork creates a new
> process and therefore also a new thread."  From the Linux man pages https://www.man7.org/linux/man-pages/man2/fork.2.html, "The child process is created with a single thread?the one that called fork()." 

You just quoted the evidence!

All new processes on unix (may all OS) only ever have one thread when they start.
The thread-id of the first thread is the same as the process-id and referred to as the main thread.

> 
> I have a one-core one-thread instance at Digital Ocean available running Ubuntu 18.04.  I can fork and create a new process on it, but it doesn't create a new thread because it doesn't have one available. 


By that logic it can only run one process...

It has one hardware core that support one hardware thread.
Linux can create as many software threads as it likes.

> You may also want to see "Forking vs Threading" (https://www.geekride.com/fork-forking-vs-threading-thread-linux-kernel), "Fork vs Thread" (https://medium.com/obscure-system/fork-vs-thread-38e09ec099e2), and "Linux process and thread" (https://zliu.org/post/linux-process-and-thread) ("This means that to create a normal process fork() is used that further calls clone() with appropriate arguments while to create a thread or LWP, a function from pthread library calls clone() with relvant flags. So, the main difference is generated by using different flags that can be passed to clone() funciton(to be exact, it is a system call"). 
> 
> You may be confused by the fact that threads are called light-weight processes. 

No Peter and I are not confused.

> 
> Or maybe I'm confused :)

Yes you are confused.

> 
> If you have other information, please let me know.  Thanks. 

Please get the book I recommended, or another that covers systems programming on unix, and have a read.

Barry

> 
> Jen
> 
> 
> Dec 5, 2021, 18:08 by hjp-python at hjp.at:
> 
>> On 2021-12-06 00:51:13 +0100, Jen Kris via Python-list wrote:
>> 
>>> The C program creates two threads (using pthreads), one for itself and
>>> one for the child process.  On creation, the second pthread is pointed
>>> to a C program that calls fork-execv to run the Python program.  That
>>> way Python runs on a separate thread. 
>>> 
>> 
>> I think you have the relationship between processes and threads
>> backwards. A process consists of one or more threads. Fork creates a new
>> process and therefore also a new thread.
>> 
>> hp
>> 
>> -- 
>> _  | Peter J. Holzer    | Story must make more sense than reality.
>> |_|_) |                    |
>> | |   | hjp at hjp.at         |    -- Charles Stross, "Creative writing
>> __/   | http://www.hjp.at/ |       challenge!"
>> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list


From barry at barrys-emacs.org  Mon Dec  6 13:41:35 2021
From: barry at barrys-emacs.org (Barry Scott)
Date: Mon, 6 Dec 2021 18:41:35 +0000
Subject: threading and multiprocessing deadlock
In-Reply-To: <sojj7k$1ek1$1@gioia.aioe.org>
References: <sojj7k$1ek1$1@gioia.aioe.org>
Message-ID: <F929683A-61BE-4B5D-B475-422B474E34A6@barrys-emacs.org>



> On 5 Dec 2021, at 23:50, Johannes Bauer <dfnsonfsduifb at gmx.de> wrote:
> 
> Hi there,
> 
> I'm a bit confused. In my scenario I a mixing threading with
> multiprocessing. Threading by itself would be nice, but for GIL reasons
> I need both, unfortunately. I've encountered a weird situation in which
> multiprocessing Process()es which are started in a new thread don't
> actually start and so they deadlock on join.
> 
> I've created a minimal example that demonstrates the issue. I'm running
> on x86_64 Linux using Python 3.9.5 (default, May 11 2021, 08:20:37)
> ([GCC 10.3.0] on linux).
> 
> Here's the code:

I suggest that you include the threading.current_thread() in your messages.
Then you can see which thread is saying what.

Barry

> 
> 
> import time
> import multiprocessing
> import threading
> 
> def myfnc():
> 	print("myfnc")
> 
> def run(result_queue, callback):
> 	result = callback()
> 	result_queue.put(result)
> 
> def start(fnc):
> 	def background_thread():
> 		queue = multiprocessing.Queue()
> 		proc = multiprocessing.Process(target = run, args = (queue, fnc))
> 		proc.start()
> 		print("join?")
> 		proc.join()
> 		print("joined.")
> 		result = queue.get()
> 	threading.Thread(target = background_thread).start()
> 
> start(myfnc)
> start(myfnc)
> start(myfnc)
> start(myfnc)
> while True:
> 	time.sleep(1)
> 
> 
> What you'll see is that "join?" and "joined." nondeterministically does
> *not* appear in pairs. For example:
> 
> join?
> join?
> myfnc
> myfnc
> join?
> join?
> joined.
> joined.
> 
> What's worse is that when this happens and I Ctrl-C out of Python, the
> started Thread is still running in the background:
> 
> $ ps ax | grep minimal
> 370167 pts/0    S      0:00 python3 minimal.py
> 370175 pts/2    S+     0:00 grep minimal
> 
> Can someone figure out what is going on there?
> 
> Best,
> Johannes
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 


From dieter at handshake.de  Mon Dec  6 13:38:08 2021
From: dieter at handshake.de (Dieter Maurer)
Date: Mon, 6 Dec 2021 19:38:08 +0100
Subject: threading and multiprocessing deadlock
In-Reply-To: <sojj7k$1ek1$1@gioia.aioe.org>
References: <sojj7k$1ek1$1@gioia.aioe.org>
Message-ID: <25006.22672.283410.422526@ixdm.fritz.box>

Johannes Bauer wrote at 2021-12-6 00:50 +0100:
>I'm a bit confused. In my scenario I a mixing threading with
>multiprocessing. Threading by itself would be nice, but for GIL reasons
>I need both, unfortunately. I've encountered a weird situation in which
>multiprocessing Process()es which are started in a new thread don't
>actually start and so they deadlock on join.

The `multiprocessing` doc
(--> "https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing")
has the following warning:
"Note that safely forking a multithreaded process is problematic."

Thus, if you use the `fork` method to start processes, some
surprises are to be expected.

From jenkris at tutanota.com  Mon Dec  6 16:05:48 2021
From: jenkris at tutanota.com (Jen Kris)
Date: Mon, 6 Dec 2021 22:05:48 +0100 (CET)
Subject: Python child process in while True loop blocks parent
In-Reply-To: <3065988D-E2CF-4C36-B319-1DEC87B864E3@barrys-emacs.org>
References: <MphoqOf--3-2@tutanota.com>
 <7A3CE5EB-C53A-404D-BF32-77CF0243BF3D@barrys-emacs.org>
 <MpqihHm--3-2@tutanota.com>
 <27742189-51F0-49A8-ACA7-344EFCD2B1BD@barrys-emacs.org>
 <MqAitZV--3-2@tutanota.com>
 <0633A53B-A722-4CEE-8F20-E2D0AEBB8D01@barrys-emacs.org>
 <MqC-YXO--B-2@tutanota.com> <20211206020831.ytkglawzaeoueyhh@hjp.at>
 <MqFiCj---3-2@tutanota.com>
 <3065988D-E2CF-4C36-B319-1DEC87B864E3@barrys-emacs.org>
Message-ID: <MqGZH61--3-2@tutanota.com>

Here is what I don't understand from what you said.? "The child process is created with a single thread?the one that called fork()."? To me that implies that the thread that called fork() is the same thread as the child process.? I guess you're talking about the distinction between logical threads and physical threads.? 

But the main issue is your suggestion that I should call fork-execv from the thread that runs the main C program, not from a separate physical pthread.? That would certainly eliminate the overhead of creating a new pthread.?

I am working now to finish this, and I will try your suggestion of calling fork-execv from the "main" thread.? When I reply back next I can give you a complete picture of what I'm doing.?

Your comments, and those of Peter Holzer and Chris Angelico, are most appreciated.?




Dec 6, 2021, 10:37 by barry at barrys-emacs.org:

>
>
>> On 6 Dec 2021, at 17:09, Jen Kris via Python-list <python-list at python.org> wrote:
>>
>> I can't find any support for your comment that "Fork creates a new
>> process and therefore also a new thread."  From the Linux man pages https://www.man7.org/linux/man-pages/man2/fork.2.html, "The child process is created with a single thread?the one that called fork()."
>>
>
> You just quoted the evidence!
>
> All new processes on unix (may all OS) only ever have one thread when they start.
> The thread-id of the first thread is the same as the process-id and referred to as the main thread.
>
>>
>> I have a one-core one-thread instance at Digital Ocean available running Ubuntu 18.04.  I can fork and create a new process on it, but it doesn't create a new thread because it doesn't have one available.
>>
>
>
> By that logic it can only run one process...
>
> It has one hardware core that support one hardware thread.
> Linux can create as many software threads as it likes.
>
>> You may also want to see "Forking vs Threading" (https://www.geekride.com/fork-forking-vs-threading-thread-linux-kernel), "Fork vs Thread" (https://medium.com/obscure-system/fork-vs-thread-38e09ec099e2), and "Linux process and thread" (https://zliu.org/post/linux-process-and-thread) ("This means that to create a normal process fork() is used that further calls clone() with appropriate arguments while to create a thread or LWP, a function from pthread library calls clone() with relvant flags. So, the main difference is generated by using different flags that can be passed to clone() funciton(to be exact, it is a system call"). 
>>
>> You may be confused by the fact that threads are called light-weight processes.
>>
>
> No Peter and I are not confused.
>
>>
>> Or maybe I'm confused :)
>>
>
> Yes you are confused.
>
>>
>> If you have other information, please let me know.  Thanks.
>>
>
> Please get the book I recommended, or another that covers systems programming on unix, and have a read.
>
> Barry
>
>>
>> Jen
>>
>>
>> Dec 5, 2021, 18:08 by hjp-python at hjp.at:
>>
>>> On 2021-12-06 00:51:13 +0100, Jen Kris via Python-list wrote:
>>>
>>>> The C program creates two threads (using pthreads), one for itself and
>>>> one for the child process.  On creation, the second pthread is pointed
>>>> to a C program that calls fork-execv to run the Python program.  That
>>>> way Python runs on a separate thread. 
>>>>
>>>
>>> I think you have the relationship between processes and threads
>>> backwards. A process consists of one or more threads. Fork creates a new
>>> process and therefore also a new thread.
>>>
>>> hp
>>>
>>> -- 
>>> _  | Peter J. Holzer    | Story must make more sense than reality.
>>> |_|_) |                    |
>>> | |   | hjp at hjp.at         |    -- Charles Stross, "Creative writing
>>> __/   | http://www.hjp.at/ |       challenge!"
>>>
>>
>> -- 
>> https://mail.python.org/mailman/listinfo/python-list
>>


From barry at barrys-emacs.org  Mon Dec  6 16:33:09 2021
From: barry at barrys-emacs.org (Barry Scott)
Date: Mon, 6 Dec 2021 21:33:09 +0000
Subject: Python child process in while True loop blocks parent
In-Reply-To: <MqGZH61--3-2@tutanota.com>
References: <MphoqOf--3-2@tutanota.com>
 <7A3CE5EB-C53A-404D-BF32-77CF0243BF3D@barrys-emacs.org>
 <MpqihHm--3-2@tutanota.com>
 <27742189-51F0-49A8-ACA7-344EFCD2B1BD@barrys-emacs.org>
 <MqAitZV--3-2@tutanota.com>
 <0633A53B-A722-4CEE-8F20-E2D0AEBB8D01@barrys-emacs.org>
 <MqC-YXO--B-2@tutanota.com> <20211206020831.ytkglawzaeoueyhh@hjp.at>
 <MqFiCj---3-2@tutanota.com>
 <3065988D-E2CF-4C36-B319-1DEC87B864E3@barrys-emacs.org>
 <MqGZH61--3-2@tutanota.com>
Message-ID: <6E104B3E-1FD8-476F-A560-87CF47B183CC@barrys-emacs.org>



> On 6 Dec 2021, at 21:05, Jen Kris <jenkris at tutanota.com> wrote:
> 
> Here is what I don't understand from what you said.  "The child process is created with a single thread?the one that called fork()."  To me that implies that the thread that called fork() is the same thread as the child process.  I guess you're talking about the distinction between logical threads and physical threads.  

The thread that called fork is cloned into a new thread in the new process.
It has a clone of the processor registers of the thread, stack memory segment of that thread,
which means that it has all the stack variables and stack frames from the parent process's thread.


> But the main issue is your suggestion that I should call fork-execv from the thread that runs the main C program, not from a separate physical pthread.  That would certainly eliminate the overhead of creating a new pthread. 

Forget about physical threads, they only matter when you are performance tuning your code.

> I am working now to finish this, and I will try your suggestion of calling fork-execv from the "main" thread.  When I reply back next I can give you a complete picture of what I'm doing. 
> 
> Your comments, and those of Peter Holzer and Chris Angelico, are most appreciated. 

Barry


> 
> 
> Dec 6, 2021, 10:37 by barry at barrys-emacs.org:
> 
> On 6 Dec 2021, at 17:09, Jen Kris via Python-list <python-list at python.org> wrote:
> 
> I can't find any support for your comment that "Fork creates a new
> process and therefore also a new thread." From the Linux man pages https://www.man7.org/linux/man-pages/man2/fork.2.html, "The child process is created with a single thread?the one that called fork()."
> 
> You just quoted the evidence!
> 
> All new processes on unix (may all OS) only ever have one thread when they start.
> The thread-id of the first thread is the same as the process-id and referred to as the main thread.
> 
> I have a one-core one-thread instance at Digital Ocean available running Ubuntu 18.04. I can fork and create a new process on it, but it doesn't create a new thread because it doesn't have one available.
> 
> 
> By that logic it can only run one process...
> 
> It has one hardware core that support one hardware thread.
> Linux can create as many software threads as it likes.
> You may also want to see "Forking vs Threading" (https://www.geekride.com/fork-forking-vs-threading-thread-linux-kernel), "Fork vs Thread" (https://medium.com/obscure-system/fork-vs-thread-38e09ec099e2), and "Linux process and thread" (https://zliu.org/post/linux-process-and-thread) ("This means that to create a normal process fork() is used that further calls clone() with appropriate arguments while to create a thread or LWP, a function from pthread library calls clone() with relvant flags. So, the main difference is generated by using different flags that can be passed to clone() funciton(to be exact, it is a system call"). 
> 
> You may be confused by the fact that threads are called light-weight processes.
> 
> No Peter and I are not confused.
> 
> Or maybe I'm confused :)
> 
> Yes you are confused.
> 
> If you have other information, please let me know. Thanks.
> 
> Please get the book I recommended, or another that covers systems programming on unix, and have a read.
> 
> Barry
> 
> Jen
> 
> 
> Dec 5, 2021, 18:08 by hjp-python at hjp.at:
> On 2021-12-06 00:51:13 +0100, Jen Kris via Python-list wrote:
> The C program creates two threads (using pthreads), one for itself and
> one for the child process. On creation, the second pthread is pointed
> to a C program that calls fork-execv to run the Python program. That
> way Python runs on a separate thread. 
> 
> I think you have the relationship between processes and threads
> backwards. A process consists of one or more threads. Fork creates a new
> process and therefore also a new thread.
> 
> hp
> 
> -- 
> _ | Peter J. Holzer | Story must make more sense than reality.
> |_|_) | |
> | | | hjp at hjp.at | -- Charles Stross, "Creative writing
> __/ | http://www.hjp.at/ | challenge!"
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 


From pablogsal at gmail.com  Mon Dec  6 19:02:22 2021
From: pablogsal at gmail.com (Pablo Galindo Salgado)
Date: Tue, 7 Dec 2021 00:02:22 +0000
Subject: [RELEASE] Python 3.10.1 is available
Message-ID: <CAFjbc8FxQFrFVDwWzdQqifR+7pfg3muf6gx4HseOT057+OFyjA@mail.gmail.com>

I hope you like bug fixes, because we have a whole shipment of them! Python
3.10.1 is the first maintenance release of Python 3.10 as we have packed
more than 330 commits of fixes and general improvements. You can get it
here:

https://www.python.org/downloads/release/python-3101/

# This is the first maintenance release of Python 3.10

Python 3.10.1 is the newest major release of the Python programming
language, and it contains many new features and optimizations.

# Major new features of the 3.10 series, compared to 3.9

Among the new major new features and changes so far:

* [PEP 623](https://www.python.org/dev/peps/pep-0623/) -- Deprecate and
prepare for the removal of the wstr member in PyUnicodeObject.
* [PEP 604](https://www.python.org/dev/peps/pep-0604/) -- Allow writing
union types as X | Y
* [PEP 612](https://www.python.org/dev/peps/pep-0612/) -- Parameter
Specification Variables
* [PEP 626](https://www.python.org/dev/peps/pep-0626/) -- Precise line
numbers for debugging and other tools.
* [PEP 618 ](https://www.python.org/dev/peps/pep-0618/) -- Add Optional
Length-Checking To zip.
* [bpo-12782](https://bugs.python.org/issue12782): Parenthesized context
managers are now officially allowed.
* [PEP 632 ](https://www.python.org/dev/peps/pep-0632/) -- Deprecate
distutils module.
* [PEP 613 ](https://www.python.org/dev/peps/pep-0613/) -- Explicit Type
Aliases
* [PEP 634 ](https://www.python.org/dev/peps/pep-0634/) -- Structural
Pattern Matching: Specification
* [PEP 635 ](https://www.python.org/dev/peps/pep-0635/) -- Structural
Pattern Matching: Motivation and Rationale
* [PEP 636 ](https://www.python.org/dev/peps/pep-0636/) -- Structural
Pattern Matching: Tutorial
* [PEP 644 ](https://www.python.org/dev/peps/pep-0644/) -- Require OpenSSL
1.1.1 or newer
* [PEP 624 ](https://www.python.org/dev/peps/pep-0624/) -- Remove
Py_UNICODE encoder APIs
* [PEP 597 ](https://www.python.org/dev/peps/pep-0597/) -- Add optional
EncodingWarning

[bpo-38605](https://bugs.python.org/issue38605): `from __future__ import
annotations` ([PEP 563](https://www.python.org/dev/peps/pep-0563/)) used to
be on this list
in previous pre-releases but it has been postponed to Python 3.11 due to
some compatibility concerns. You can read the Steering Council
communication about it [here](
https://mail.python.org/archives/list/python-dev at python.org/thread/CLVXXPQ2T2LQ5MP2Y53VVQFCXYWQJHKZ/)
to learn more.

# More resources

* [Changelog](https://docs.python.org/3.10/whatsnew/changelog.html#changelog
)
* [Online Documentation](https://docs.python.org/3.10/)
* [PEP 619](https://www.python.org/dev/peps/pep-0619/), 3.10 Release
Schedule
* Report bugs at [https://bugs.python.org](https://bugs.python.org).
* [Help fund Python and its community](/psf/donations/).

# And now for something completely different

The Meissner effect (or Meissner?Ochsenfeld effect) is the expulsion of a
magnetic field from a superconductor during its transition to the
superconducting state when it is cooled below the critical temperature.
This expulsion will repel a nearby magnet. The German physicists Walther
Meissner and Robert Ochsenfeld discovered this phenomenon in 1933 by
measuring the magnetic field distribution outside superconducting tin and
lead samples. The experiment demonstrated for the first time that
superconductors were more than just perfect conductors and provided a
uniquely defining property of the superconductor state. The ability for the
expulsion effect is determined by the nature of equilibrium formed by the
neutralization within the unit cell of a superconductor.

You can do [ver cool things](https://www.youtube.com/watch?v=HRLvVkkq5GE)
with it!

# We hope you enjoy the new releases!

Thanks to all of the many volunteers who help make Python Development and
these releases possible! Please consider supporting our efforts by
volunteering yourself or through organization contributions to the Python
Software Foundation.

https://www.python.org/psf/

Your friendly release team,
Ned Deily @nad https://discuss.python.org/u/nad
Steve Dower @steve.dower https://discuss.python.org/u/steve.dower
Pablo Galindo Salgado @pablogsal https://discuss.python.org/u/pablogsal

From arulvani66 at gmail.com  Tue Dec  7 02:23:44 2021
From: arulvani66 at gmail.com (vani arul)
Date: Tue, 7 Dec 2021 12:53:44 +0530
Subject: Return
Message-ID: <CAHVaEHhKkpB49o5fUzwXH3s0r_zsTwDUsFy1jLk8b+NWGpXAGA@mail.gmail.com>

Hey There,
Can someone help to understand how a python function can return value with
using return in the code?
It is not not about explicit or implicit function call.


Thanks
Vani

From mohsen.owzar at gmail.com  Mon Dec  6 23:58:57 2021
From: mohsen.owzar at gmail.com (Mohsen Owzar)
Date: Mon, 6 Dec 2021 20:58:57 -0800 (PST)
Subject: For a hierarchical project, the EXE file generated by "pyinstaller"
 does not start.
Message-ID: <a1e1d8be-e2cc-4187-a600-01c2cbb3c8fbn@googlegroups.com>

Hi all, 

I have a problem with "pyinstaller".
When I compile a single Python file, an EXE file is created in the "dist" directory, with which I can start the program and the GUI appears after a few seconds.
But when I try to compile my project with "pyinstaller Relais_LastDauerTester.py" and generate an EXE file from it, an ".exe" file is also created in the "dist" directory, but nothing happens when I run both .exe files from Windows Explorer with a double click or from the DOS shell by invoking it.
The mouse pointer changes briefly to an hourglass and then back and nothing happens.
The only difference between these two programs is that the first is just a single file and my project is structured hierarchically. I tried to demonstrate the structure of the project below with two "DIR" DOS commands.
In PyCharm you only need to run the top file "Relais_LastDauerTester.py", so I used this file for "pyinstaller".
But it looks like I have to give (I believe) an extra argument for a hierarchical project.
I googled all weekend to find a workable solution. But all I found was for converting a ".py" file and not a project like mine.
With some suggestions, I should add a "pause" or "input ()" command to the code to prevent the DOS shell from disappearing quickly, which is out of the question for me.
Do you have any idea what it could be?
I thank you in advance for any useful advice that could help me.

Greeting
Mohsen

PS 
I work on a PC with 
1201 INFO: PyInstaller: 4.7 
1202 INFO: Python: 3.9.5 
1282 INFO: Platform: Windows-10-10.0.18363-SP0 

Project Structure from PyCharm
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&& 
...\SW\Relais_LastDauerTester_V0.5 
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&& 
<DIR> .idea 
<DIR> Logfiles 
<DIR> Relais_LastDauerTester 
276     Relais_LastDauerTester.py 
<DIR> Screenshotfiles 
405     settings.ini 

&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&& 
...\SW\Relais_LastDauerTester_V0.5\Relais_LastDauerTester 
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&& 
9?308      GPIOControl.py 
90?618   GUI_View.py 
998         main.py 
28?625  TestControl.py 
269      __init__.py 
<DIR>   __pycache__ 


Simplified project files with the "import" lines ******************************************************************* 
Relais_LastDauerTester.py.py 
******************************************************************* 
#!/usr/bin/env python3 
# -*- coding: utf-8 -*- 
from Relais_LastDauerTester.main import main 

if __name__ == "__main__": 
main() 

******************************************************************* 
main.py 
******************************************************************* 
import sys 

from PyQt5.QtCore import * 
from PyQt5.QtWidgets import QApplication 
from .GUI_View import MainWindow 

def main(): 
app = QApplication(sys.argv) 

window = MainWindow() 
window.show() 

sys.exit(app.exec_()) 

if __name__ == '__main__': 
main() 

******************************************************************* 
GUI_View.py 
******************************************************************* 
import sys 
import subprocess 

import PyQt5.QtGui as qtg 
from PyQt5.QtWidgets import (QLabel, QPushButton, QLineEdit, QCheckBox, QWidget, 
QVBoxLayout, QHBoxLayout, QGridLayout, QDialog, QFileDialog) 
from .TestControl import * 

class MainWindow(QWidget): 
def __init__(self): 
super().__init__() 

def createMainWindow(self): 
... 

def exitMainWindow(self): 
... 

def ChangeToPrefWindow(self): 
self.prefwindow.show() 
self.hide() 

class PrefWindow(QWidget): 
def __init__(self, parent=None): 
super().__init__() 
self.parent = parent 
... 
self.createPrefWindow() 

def ChangeToMainWindow(self): 
... 

def createPrefWindow(self): 
... 

class CustomLineEdit(QLineEdit): 
clicked = pyqtSignal() 

def mousePressEvent(self, QMouseEvent): 
self.clicked.emit() 

class Keypad_Window_New(QDialog): 
def __init__(self, num=0, parent=None): 
super().__init__(parent) 
self.parent = parent 
... 

TestContrl.py 
******************************************************************* 
from PyQt5.QtCore import * 
from .GPIOControl import GPIOControl 

class WorkerSignals(QObject): 
signal_Update_Label = pyqtSignal() 

class TestControl(QRunnable): 
signals = WorkerSignals() 

def __init__(self, parent=None): 
super().__init__() 
self.parent = parent 
... 

******************************************************************* 
GPIOContrl.py 
******************************************************************* 
class GPIOControl: 
def my_print(self, args): 
if print_allowed == 1: 
print(args) 

def __init__(self):

From juliushamilton100 at gmail.com  Tue Dec  7 06:35:06 2021
From: juliushamilton100 at gmail.com (Julius Hamilton)
Date: Tue, 7 Dec 2021 12:35:06 +0100
Subject: Urllib.request vs. Requests.get
Message-ID: <CAEsMKX3NiKyvrzYpFQcRSwsh4tf2=mpBEuSnwGaU0c9HavR=Yg@mail.gmail.com>

Hey,

I am currently working on a simple program which scrapes text from webpages
via a URL, then segments it (with Spacy).

I?m trying to refine my program to use just the right tools for the job,
for each of the steps.

Requests.get works great, but I?ve seen people use urllib.request.urlopen()
in some examples. It appealed to me because it seemed lower level than
requests.get, so it just makes the program feel leaner and purer and more
direct.

However, requests.get works fine on this url:

https://juno.sh/direct-connection-to-jupyter-server/

But urllib returns a ?403 forbidden?.

Could anyone please comment on what the fundamental differences are between
urllib vs. requests, why this would happen, and if urllib has any option to
prevent this and get the page source?

Thanks,
Julius

From juliushamilton100 at gmail.com  Tue Dec  7 06:53:42 2021
From: juliushamilton100 at gmail.com (Julius Hamilton)
Date: Tue, 7 Dec 2021 12:53:42 +0100
Subject: HTML extraction
Message-ID: <CAEsMKX3TkUK==fNcZVZXhDrEWFA8RW6PTY47quACz7LmJ-Xy_Q@mail.gmail.com>

Hey,

Could anyone please comment on the purest way simply to strip HTML tags
from the internal text they surround?

I know Beautiful Soup is a convenient tool, but I?m interested to know what
the most minimal way to do it would be.

People say you usually don?t use Regex for a second order language like
HTML, so I was thinking about using xpath or lxml, which seem like very
pure, universal tools for the job.

I did find an example for doing this with the re module, though.

Would it be fair to say that to just strip the tags, Regex is fine, but you
need to build a tree-like object if you want the ability to select which
nodes to keep and which to discard?

Can xpath / lxml do that?

What are the chief differences between xpath / lxml and Beautiful Soup?

Thanks,
Julius

From cl at isbd.net  Tue Dec  7 10:28:25 2021
From: cl at isbd.net (Chris Green)
Date: Tue, 7 Dec 2021 15:28:25 +0000
Subject: Odd locale error that has disappeared on reboot.
Message-ID: <p5b48i-o59f1.ln1@esprimo.zbmc.eu>

I have a very short Python program that runs on one of my Raspberry
Pis to collect temperatures from a 1-wire sensor and write them to a
database:-

    #!/usr/bin/python3
    #
    #
    # read temperature from 1-wire sensor and store in database with date and time
    #
    import sqlite3
    import time

    ftxt = str(open("/sys/bus/w1/devices/28-01204e1e64c3/w1_slave").read(100))
    temp = (float(ftxt[ftxt.find("t=") +2:]))/1000
    #
    #
    # insert date, time and temperature into the database
    #
    tdb = sqlite3.connect("/home/chris/.cfg/share/temperature/temperature.db")
    cr = tdb.cursor()
    dt = time.strftime("%Y-%m-%d %H:%M")
    cr.execute("Insert INTO temperatures (DateTime, Temperature) VALUES(?, round(?, 2))", (dt, temp)
    )
    tdb.commit()
    tdb.close()

It's run by cron every 10 minutes.


At 03:40 last night it suddenly started throwing the following error every
time it ran:-

    Fatal Python error: initfsencoding: Unable to get the locale encoding
    LookupError: unknown encoding: UTF-8

    Current thread 0xb6f8db40 (most recent call first):
    Aborted

Running the program from the command line produced the same error.
Restarting the Pi system has fixed the problem.


What could have caused this?  I certainly wasn't around at 03:40! :-)
There aren't any automatic updates enabled on the system, the only 
thing that might have been going on was a backup as that Pi is also
my 'NAS' with a big USB drive connected to it.  The backups have been
running without problems for more than a year.  Looking at the system
logs shows that a backup was started at 03:35 so I suppose that *could*
have provoked something but I fail to understand how.

-- 
Chris Green
?

From rosuav at gmail.com  Tue Dec  7 13:07:36 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 8 Dec 2021 05:07:36 +1100
Subject: HTML extraction
In-Reply-To: <CAEsMKX3TkUK==fNcZVZXhDrEWFA8RW6PTY47quACz7LmJ-Xy_Q@mail.gmail.com>
References: <CAEsMKX3TkUK==fNcZVZXhDrEWFA8RW6PTY47quACz7LmJ-Xy_Q@mail.gmail.com>
Message-ID: <CAPTjJmp7F5M-R+yG763x8uEDFoVD_rUomDDHv8hsXFqUun20uA@mail.gmail.com>

On Wed, Dec 8, 2021 at 4:55 AM Julius Hamilton
<juliushamilton100 at gmail.com> wrote:
>
> Hey,
>
> Could anyone please comment on the purest way simply to strip HTML tags
> from the internal text they surround?
>
> I know Beautiful Soup is a convenient tool, but I?m interested to know what
> the most minimal way to do it would be.

That's definitely the best and most general way, and would still be my
first thought most of the time.

> People say you usually don?t use Regex for a second order language like
> HTML, so I was thinking about using xpath or lxml, which seem like very
> pure, universal tools for the job.
>
> I did find an example for doing this with the re module, though.
>
> Would it be fair to say that to just strip the tags, Regex is fine, but you
> need to build a tree-like object if you want the ability to select which
> nodes to keep and which to discard?

Obligatory reference:

https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags

> Can xpath / lxml do that?
>
> What are the chief differences between xpath / lxml and Beautiful Soup?
>

I've never directly used lxml, mainly because bs4 offers all the same
advantages and more, with about the same costs. However, if you're
looking for a no-external-deps option, Python *does* include an HTML
parser in the standard library:

https://docs.python.org/3/library/html.parser.html

If your purpose is extremely simple (like "strip tags, search for
text"), then it should be easy enough to whip up something using that
module. No external deps, not a lot of code, pretty straight-forward.
On the other hand, if you're trying to do an "HTML to text"
conversion, you'd probably need to be aware of which tags are
block-level and which are inline content, so that (for instance)
"<div>Hello</div> <div>world</div>" would come out as two separate
paragraphs of text, whereas the same thing with <b> tags would become
just "Hello world". But for the most part, handle_data will probably
do everything you need.

ChrisA

From rosuav at gmail.com  Tue Dec  7 13:13:34 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 8 Dec 2021 05:13:34 +1100
Subject: Urllib.request vs. Requests.get
In-Reply-To: <CAEsMKX3NiKyvrzYpFQcRSwsh4tf2=mpBEuSnwGaU0c9HavR=Yg@mail.gmail.com>
References: <CAEsMKX3NiKyvrzYpFQcRSwsh4tf2=mpBEuSnwGaU0c9HavR=Yg@mail.gmail.com>
Message-ID: <CAPTjJmpOHcsMRkE5d-X+tHRDbcbqNKBtjuu-DV4T9ZRstbnXaQ@mail.gmail.com>

On Wed, Dec 8, 2021 at 4:51 AM Julius Hamilton
<juliushamilton100 at gmail.com> wrote:
>
> Hey,
>
> I am currently working on a simple program which scrapes text from webpages
> via a URL, then segments it (with Spacy).
>
> I?m trying to refine my program to use just the right tools for the job,
> for each of the steps.
>
> Requests.get works great, but I?ve seen people use urllib.request.urlopen()
> in some examples. It appealed to me because it seemed lower level than
> requests.get, so it just makes the program feel leaner and purer and more
> direct.
>
> However, requests.get works fine on this url:
>
> https://juno.sh/direct-connection-to-jupyter-server/
>
> But urllib returns a ?403 forbidden?.
>
> Could anyone please comment on what the fundamental differences are between
> urllib vs. requests, why this would happen, and if urllib has any option to
> prevent this and get the page source?
>

*Fundamental* differences? Not many. The requests module is designed
to be easy to use, whereas urllib is designed to be basic and simple.
Not really a fundamental difference, but perhaps indicative.

I'd recommend doing the query with requests, and seeing exactly what
headers are being sent. Most likely, there'll be something that you
need to add explicitly when using urllib that the server is looking
for (maybe a user agent or something). Requests uses Python's logging
module to configure everything, so it should be a simple matter of
setting log level to DEBUG and sending the request.

TBH though, I'd just recommend using requests, unless you specifically
need to avoid the dependency :)

ChrisA

From pbryan at anode.ca  Tue Dec  7 13:14:31 2021
From: pbryan at anode.ca (Paul Bryan)
Date: Tue, 07 Dec 2021 10:14:31 -0800
Subject: Urllib.request vs. Requests.get
In-Reply-To: <CAEsMKX3NiKyvrzYpFQcRSwsh4tf2=mpBEuSnwGaU0c9HavR=Yg@mail.gmail.com>
References: <CAEsMKX3NiKyvrzYpFQcRSwsh4tf2=mpBEuSnwGaU0c9HavR=Yg@mail.gmail.com>
Message-ID: <b8d49c0c17904851363c9531040436da9ce11bfe.camel@anode.ca>

Cloudflare, for whatever reason, appears to be rejecting the `User-
Agent` header that urllib is providing:`Python-urllib/3.9`. Using a
different `User-Agent` seems to get around the issue:

import urllib.request

req = urllib.request.Request(
    url="https://juno.sh/direct-connection-to-jupyter-server/",
    method="GET",
    headers={"User-Agent": "Workaround/1.0"},
)

res = urllib.request.urlopen(req)

Paul

On Tue, 2021-12-07 at 12:35 +0100, Julius Hamilton wrote:
> Hey,
> 
> I am currently working on a simple program which scrapes text from
> webpages
> via a URL, then segments it (with Spacy).
> 
> I?m trying to refine my program to use just the right tools for the
> job,
> for each of the steps.
> 
> Requests.get works great, but I?ve seen people use
> urllib.request.urlopen()
> in some examples. It appealed to me because it seemed lower level
> than
> requests.get, so it just makes the program feel leaner and purer and
> more
> direct.
> 
> However, requests.get works fine on this url:
> 
> https://juno.sh/direct-connection-to-jupyter-server/
> 
> But urllib returns a ?403 forbidden?.
> 
> Could anyone please comment on what the fundamental differences are
> between
> urllib vs. requests, why this would happen, and if urllib has any
> option to
> prevent this and get the page source?
> 
> Thanks,
> Julius


From rosuav at gmail.com  Tue Dec  7 13:16:21 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 8 Dec 2021 05:16:21 +1100
Subject: For a hierarchical project, the EXE file generated by
 "pyinstaller" does not start.
In-Reply-To: <a1e1d8be-e2cc-4187-a600-01c2cbb3c8fbn@googlegroups.com>
References: <a1e1d8be-e2cc-4187-a600-01c2cbb3c8fbn@googlegroups.com>
Message-ID: <CAPTjJmqcqSTRmfSJpyGagh=ZcM--XGN9zVW7RoD8XKLg6QQ+9A@mail.gmail.com>

On Wed, Dec 8, 2021 at 4:49 AM Mohsen Owzar <mohsen.owzar at gmail.com> wrote:
> *******************************************************************
> GPIOContrl.py
> *******************************************************************
> class GPIOControl:
> def my_print(self, args):
> if print_allowed == 1:
> print(args)
>
> def __init__(self):

Can't much help with your main question as I don't do Windows, but one
small side point: Instead of having a my_print that checks if printing
is allowed, you can conditionally replace the print function itself.

if not print_allowed:
    def print(*args, **kwargs): pass

ChrisA

From roland.em0001 at googlemail.com  Tue Dec  7 15:45:18 2021
From: roland.em0001 at googlemail.com (Roland Mueller)
Date: Tue, 7 Dec 2021 22:45:18 +0200
Subject: Return
In-Reply-To: <CAHVaEHhKkpB49o5fUzwXH3s0r_zsTwDUsFy1jLk8b+NWGpXAGA@mail.gmail.com>
References: <CAHVaEHhKkpB49o5fUzwXH3s0r_zsTwDUsFy1jLk8b+NWGpXAGA@mail.gmail.com>
Message-ID: <CALk2KRXdXypvMTJrAkPR0xL6BheLaS1u+Ot3KWWQ8feMqVh_jQ@mail.gmail.com>

Hello

ti 7. jouluk. 2021 klo 19.47 vani arul (arulvani66 at gmail.com) kirjoitti:

> Hey There,
> Can someone help to understand how a python function can return value with
> using return in the code?
> It is not not about explicit or implicit function call.
>
>
Not sure whether I understood your question: I have a simple example about
return.
  * f() and h() explicitely return something
  * g() and h() return None

BTW, also Python documentation tool pydoc has an article about return.

#!/usr/bin/python

def f():
    return 42
def g():
    pass
def h():
    return

if __name__ == "__main__":
    print(f"f(): {f()}")
    print(f"g(): {g()}")
    print(f"h(): {h()}")

Result:
f(): 42
g(): None
h(): None

Pydoc:
$ pydoc return

BR,
Roland


> Thanks
> Vani
> --
> https://mail.python.org/mailman/listinfo/python-list
>

From roland.em0001 at googlemail.com  Tue Dec  7 15:55:26 2021
From: roland.em0001 at googlemail.com (Roland Mueller)
Date: Tue, 7 Dec 2021 22:55:26 +0200
Subject: HTML extraction
In-Reply-To: <CAPTjJmp7F5M-R+yG763x8uEDFoVD_rUomDDHv8hsXFqUun20uA@mail.gmail.com>
References: <CAEsMKX3TkUK==fNcZVZXhDrEWFA8RW6PTY47quACz7LmJ-Xy_Q@mail.gmail.com>
 <CAPTjJmp7F5M-R+yG763x8uEDFoVD_rUomDDHv8hsXFqUun20uA@mail.gmail.com>
Message-ID: <CALk2KRVGa-0DEwivNJbnDsLuyg8rJjAB=Vo5ZAW-J0FdswKuFQ@mail.gmail.com>

Hello,

ti 7. jouluk. 2021 klo 20.08 Chris Angelico (rosuav at gmail.com) kirjoitti:

> On Wed, Dec 8, 2021 at 4:55 AM Julius Hamilton
> <juliushamilton100 at gmail.com> wrote:
> >
> > Hey,
> >
> > Could anyone please comment on the purest way simply to strip HTML tags
> > from the internal text they surround?
> >
> > I know Beautiful Soup is a convenient tool, but I?m interested to know
> what
> > the most minimal way to do it would be.
>
> That's definitely the best and most general way, and would still be my
> first thought most of the time.
>
> > People say you usually don?t use Regex for a second order language like
> > HTML, so I was thinking about using xpath or lxml, which seem like very
> > pure, universal tools for the job.
> >
> > I did find an example for doing this with the re module, though.
> >
> > Would it be fair to say that to just strip the tags, Regex is fine, but
> you
> > need to build a tree-like object if you want the ability to select which
> > nodes to keep and which to discard?
>
> Obligatory reference:
>
>
> https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags
>
> > Can xpath / lxml do that?
> >
> > What are the chief differences between xpath / lxml and Beautiful Soup?
> >
>
> I've never directly used lxml, mainly because bs4 offers all the same
> advantages and more, with about the same costs. However, if you're
> looking for a no-external-deps option, Python *does* include an HTML
> parser in the standard library:
>
>
But isn't bs4 only for SOAP content?
Can bs4 or lxml cope with HTML code that does not comply with XML as the
following fragment?

<p>A
<p>B
<hr>

BR,
Roland


> https://docs.python.org/3/library/html.parser.html
>
> If your purpose is extremely simple (like "strip tags, search for
> text"), then it should be easy enough to whip up something using that
> module. No external deps, not a lot of code, pretty straight-forward.
> On the other hand, if you're trying to do an "HTML to text"
> conversion, you'd probably need to be aware of which tags are
> block-level and which are inline content, so that (for instance)
> "<div>Hello</div> <div>world</div>" would come out as two separate
> paragraphs of text, whereas the same thing with <b> tags would become
> just "Hello world". But for the most part, handle_data will probably
> do everything you need.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>

From rosuav at gmail.com  Tue Dec  7 16:57:06 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 8 Dec 2021 08:57:06 +1100
Subject: HTML extraction
In-Reply-To: <CALk2KRVGa-0DEwivNJbnDsLuyg8rJjAB=Vo5ZAW-J0FdswKuFQ@mail.gmail.com>
References: <CAEsMKX3TkUK==fNcZVZXhDrEWFA8RW6PTY47quACz7LmJ-Xy_Q@mail.gmail.com>
 <CAPTjJmp7F5M-R+yG763x8uEDFoVD_rUomDDHv8hsXFqUun20uA@mail.gmail.com>
 <CALk2KRVGa-0DEwivNJbnDsLuyg8rJjAB=Vo5ZAW-J0FdswKuFQ@mail.gmail.com>
Message-ID: <CAPTjJmrXc-pLX+ttk=2J3582dvMizrMAmDBo_H7TbeK68ah8Zw@mail.gmail.com>

On Wed, Dec 8, 2021 at 7:55 AM Roland Mueller
<roland.em0001 at googlemail.com> wrote:
>
> Hello,
>
> ti 7. jouluk. 2021 klo 20.08 Chris Angelico (rosuav at gmail.com) kirjoitti:
>>
>> On Wed, Dec 8, 2021 at 4:55 AM Julius Hamilton
>> <juliushamilton100 at gmail.com> wrote:
>> >
>> > Hey,
>> >
>> > Could anyone please comment on the purest way simply to strip HTML tags
>> > from the internal text they surround?
>> >
>> > I know Beautiful Soup is a convenient tool, but I?m interested to know what
>> > the most minimal way to do it would be.
>>
>> That's definitely the best and most general way, and would still be my
>> first thought most of the time.
>>
>> > People say you usually don?t use Regex for a second order language like
>> > HTML, so I was thinking about using xpath or lxml, which seem like very
>> > pure, universal tools for the job.
>> >
>> > I did find an example for doing this with the re module, though.
>> >
>> > Would it be fair to say that to just strip the tags, Regex is fine, but you
>> > need to build a tree-like object if you want the ability to select which
>> > nodes to keep and which to discard?
>>
>> Obligatory reference:
>>
>> https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags
>>
>> > Can xpath / lxml do that?
>> >
>> > What are the chief differences between xpath / lxml and Beautiful Soup?
>> >
>>
>> I've never directly used lxml, mainly because bs4 offers all the same
>> advantages and more, with about the same costs. However, if you're
>> looking for a no-external-deps option, Python *does* include an HTML
>> parser in the standard library:
>>
>
> But isn't bs4 only for SOAP content?
> Can bs4 or lxml cope with HTML code that does not comply with XML as the following fragment?
>
> <p>A
> <p>B
> <hr>
>
> BR,
> Roland
>

Check out the bs4 docs for some of the things you can do with it :)

ChrisA

From PythonList at DancesWithMice.info  Tue Dec  7 17:02:57 2021
From: PythonList at DancesWithMice.info (dn)
Date: Wed, 8 Dec 2021 11:02:57 +1300
Subject: Return
In-Reply-To: <CALk2KRXdXypvMTJrAkPR0xL6BheLaS1u+Ot3KWWQ8feMqVh_jQ@mail.gmail.com>
References: <CAHVaEHhKkpB49o5fUzwXH3s0r_zsTwDUsFy1jLk8b+NWGpXAGA@mail.gmail.com>
 <CALk2KRXdXypvMTJrAkPR0xL6BheLaS1u+Ot3KWWQ8feMqVh_jQ@mail.gmail.com>
Message-ID: <43f4d05e-e994-f90a-b397-d8af2a6f2817@DancesWithMice.info>

On 08/12/2021 09.45, Roland Mueller via Python-list wrote:
> Hello
> 
> ti 7. jouluk. 2021 klo 19.47 vani arul (arulvani66 at gmail.com) kirjoitti:
> 
>> Hey There,
>> Can someone help to understand how a python function can return value with
>> using return in the code?
>> It is not not about explicit or implicit function call.
>>
>>
> Not sure whether I understood your question: I have a simple example about
> return.
>   * f() and h() explicitely return something
>   * g() and h() return None
> 
> BTW, also Python documentation tool pydoc has an article about return.
> 
> #!/usr/bin/python
> 
> def f():
>     return 42
> def g():
>     pass
> def h():
>     return
> 
> if __name__ == "__main__":
>     print(f"f(): {f()}")
>     print(f"g(): {g()}")
>     print(f"h(): {h()}")
> 
> Result:
> f(): 42
> g(): None
> h(): None
> 
> Pydoc:
> $ pydoc return
> 
> BR,
> Roland
> 
> 
>> Thanks
>> Vani


plus Python, unlike some other languages, allows us to return multiple
values, either as a collection or as an implied-tuple:

def function_list():
    a_list = [ i for i in range( 9 ) ]
    return a_list

def function_multiples():
    a = 1
    b = 2
    c = 3
    return a, b, c

thus:

x, y, z = function_multiples()
-- 
Regards,
=dn

From rosuav at gmail.com  Tue Dec  7 17:07:15 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 8 Dec 2021 09:07:15 +1100
Subject: Return
In-Reply-To: <43f4d05e-e994-f90a-b397-d8af2a6f2817@DancesWithMice.info>
References: <CAHVaEHhKkpB49o5fUzwXH3s0r_zsTwDUsFy1jLk8b+NWGpXAGA@mail.gmail.com>
 <CALk2KRXdXypvMTJrAkPR0xL6BheLaS1u+Ot3KWWQ8feMqVh_jQ@mail.gmail.com>
 <43f4d05e-e994-f90a-b397-d8af2a6f2817@DancesWithMice.info>
Message-ID: <CAPTjJmrBih8VRpASf1gjRwX83qHg5FGkUT9EWdqEMHtKpkPc=Q@mail.gmail.com>

On Wed, Dec 8, 2021 at 9:04 AM dn via Python-list
<python-list at python.org> wrote:
>
> plus Python, unlike some other languages, allows us to return multiple
> values, either as a collection or as an implied-tuple:
>
> def function_list():
>     a_list = [ i for i in range( 9 ) ]
>     return a_list
>
> def function_multiples():
>     a = 1
>     b = 2
>     c = 3
>     return a, b, c
>
> thus:
>
> x, y, z = function_multiples()

Not sure what you mean by "implied". You're returning a tuple formed
from three values, and then unpacking that into three destinations.
Since, at a technical level, a function can only return one value,
returning a tuple is the standard way to return more than one thing.

ChrisA

From PythonList at DancesWithMice.info  Tue Dec  7 17:49:14 2021
From: PythonList at DancesWithMice.info (dn)
Date: Wed, 8 Dec 2021 11:49:14 +1300
Subject: Return
In-Reply-To: <CAPTjJmrBih8VRpASf1gjRwX83qHg5FGkUT9EWdqEMHtKpkPc=Q@mail.gmail.com>
References: <CAHVaEHhKkpB49o5fUzwXH3s0r_zsTwDUsFy1jLk8b+NWGpXAGA@mail.gmail.com>
 <CALk2KRXdXypvMTJrAkPR0xL6BheLaS1u+Ot3KWWQ8feMqVh_jQ@mail.gmail.com>
 <43f4d05e-e994-f90a-b397-d8af2a6f2817@DancesWithMice.info>
 <CAPTjJmrBih8VRpASf1gjRwX83qHg5FGkUT9EWdqEMHtKpkPc=Q@mail.gmail.com>
Message-ID: <3c698a11-af69-328c-0cf8-1531bbfbf36f@DancesWithMice.info>

On 08/12/2021 11.07, Chris Angelico wrote:
> On Wed, Dec 8, 2021 at 9:04 AM dn via Python-list
> <python-list at python.org> wrote:
>>
>> plus Python, unlike some other languages, allows us to return multiple
>> values, either as a collection or as an implied-tuple:
>>
>> def function_list():
>>     a_list = [ i for i in range( 9 ) ]
>>     return a_list
>>
>> def function_multiples():
>>     a = 1
>>     b = 2
>>     c = 3
>>     return a, b, c
>>
>> thus:
>>
>> x, y, z = function_multiples()
> 
> Not sure what you mean by "implied". You're returning a tuple formed
> from three values, and then unpacking that into three destinations.
> Since, at a technical level, a function can only return one value,
> returning a tuple is the standard way to return more than one thing.


How's it going @Chris?
(we have another 'overseas-speaker' scheduled for next week's PUG
meeting. Rodrigo Gir?o Serr?o will 'beam-in' from Portugal. He presented
at EuroPython. His topic with us will be "Python's Objects" - firstly at
an intro-level for people who've not built a custom-class previously,
and thereafter to more-advanced folk - details upon request...)


Back to tuples: You are (strictly) correct.

As we both know, a lot of people think that the parentheses 'make' the
tuple, whereas in-fact it is the comma-separators.

I'd estimate the OP to be in a learning situation/converting-over from
another language, so allowance for lax terminology/definitions.
-- 
Regards,
=dn

From jenkris at tutanota.com  Wed Dec  8 12:11:48 2021
From: jenkris at tutanota.com (Jen Kris)
Date: Wed, 8 Dec 2021 18:11:48 +0100 (CET)
Subject: Python child process in while True loop blocks parent
In-Reply-To: <6E104B3E-1FD8-476F-A560-87CF47B183CC@barrys-emacs.org>
References: <MpqihHm--3-2@tutanota.com>
 <27742189-51F0-49A8-ACA7-344EFCD2B1BD@barrys-emacs.org>
 <MqAitZV--3-2@tutanota.com>
 <0633A53B-A722-4CEE-8F20-E2D0AEBB8D01@barrys-emacs.org>
 <MqC-YXO--B-2@tutanota.com> <20211206020831.ytkglawzaeoueyhh@hjp.at>
 <MqFiCj---3-2@tutanota.com>
 <3065988D-E2CF-4C36-B319-1DEC87B864E3@barrys-emacs.org>
 <MqGZH61--3-2@tutanota.com>
 <6E104B3E-1FD8-476F-A560-87CF47B183CC@barrys-emacs.org>
Message-ID: <MqQ0tig--3-2@tutanota.com>

I started this post on November 29, and there have been helpful comments since then from Barry Scott, Cameron Simpson, Peter Holzer and Chris Angelico.? Thanks to all of you.? 

I've found a solution that works for my purpose, and I said earlier that I would post the solution I found. If anyone has a better solution I would appreciate any feedback.?

To recap, I'm using a pair of named pipes for IPC between C and Python.? Python runs as a child process after fork-execv.? The Python program continues to run concurrently in a while True loop, and responds to requests from C at intervals, and continues to run until it receives a signal from C to exit.? C sends signals to Python, then waits to receive data back from Python.? My problem was that C was blocked when Python started.?

The solution was twofold:? (1) for Python to run concurrently it must be a multiprocessing loop (from the multiprocessing module), and (2) Python must terminate its write strings with \n, or read will block in C waiting for something that never comes.? The multiprocessing module sidesteps the GIL; without multiprocessing the GIL will block all other threads once Python starts.?

Originally I used epoll() on the pipes.? Cameron Smith and Barry Scott advised against epoll, and for this case they are right.? Blocking pipes work here, and epoll is too much overhead for watching on a single file descriptor.?

This is the Python code now:

#!/usr/bin/python3
from multiprocessing import Process
import os

print("Python is running")

child_pid = os.getpid()
print('child process id:', child_pid)

def f(a, b):

??? print("Python now in function f")

??? pr = os.open('/tmp/Pipe_01', os.O_RDONLY)
??? print("File Descriptor1 Opened " + str(pr))
??? pw = os.open('/tmp/Pipe_02', os.O_WRONLY)
??? print("File Descriptor2 Opened " + str(pw))

??? while True:

??????? v = os.read(pr,64)
??????? print("Python read from pipe pr")
??????? print(v)

??????? if v == b'99':
??????????? os.close(pr)
??????????? os.close(pw)
??????????? print("Python is terminating")
??????????? os._exit(os.EX_OK)

??????? if v != "Send child PID":
??????????? os.write(pw, b"OK message received\n")
??????????? print("Python wrote back")

if __name__ == '__main__':
??? a = 0
??? b = 0
??? p = Process(target=f, args=(a, b,))
??? p.start()
??? p.join()

The variables a and b are not currently used in the body, but they will be later.?

This is the part of the C code that communicates with Python:

??? fifo_fd1 = open(fifo_path1, O_WRONLY);
??? fifo_fd2 = open(fifo_path2, O_RDONLY);

??? status_write = write(fifo_fd1, py_msg_01, sizeof(py_msg_01));
??? if (status_write < 0) perror("write");

??? status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_01));
??? if (status_read < 0) perror("read");
??? printf("C received message 1 from Python\n");
??? printf("%.*s",(int)buf_len, fifo_readbuf);

??? status_write = write(fifo_fd1, py_msg_02, sizeof(py_msg_02));
??? if (status_write < 0) perror("write");

??? status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_02));
??? if (status_read < 0) perror("read");
??? printf("C received message 2 from Python\n");
??? printf("%.*s",(int)buf_len, fifo_readbuf);

??? // Terminate Python multiprocessing
??? printf("C is sending exit message to Python\n");
??? status_write = write(fifo_fd1, py_msg_03, 2);

??? printf("C is closing\n");
??? close(fifo_fd1);
??? close(fifo_fd2);

Screen output:

Python is running
child process id: 5353
Python now in function f
File Descriptor1 Opened 6
Thread created 0
File Descriptor2 Opened 7
Process ID: 5351
Parent Process ID: 5351
I am the parent
Core joined 0
I am the child
Python read from pipe pr
b'Hello to Python from C\x00\x00'
Python wrote back
C received message 1 from Python
OK message received
Python read from pipe pr
b'Message to Python 2\x00\x00'
Python wrote back
C received message 2 from Python
OK message received
C is sending exit message to Python
C is closing
Python read from pipe pr
b'99'
Python is terminating

Python runs on a separate thread (created with pthreads) because I want the flexibility of using this same basic code as a stand-alone .exe, or for a C extension from Python called with ctypes.? If I use it as a C extension then I want the Python code on a separate thread because I can't have two instances of the Python interpreter running on one thread, and one instance will already be running on the main thread, albeit "suspended" by the call from ctypes.?

So that's my solution:? (1) Python multiprocessing module; (2) Python strings written to the pipe must be terminated with \n.?

Thanks again to all who commented.?



Dec 6, 2021, 13:33 by barry at barrys-emacs.org:

>
>
>
>> On 6 Dec 2021, at 21:05, Jen Kris <>> jenkris at tutanota.com>> > wrote:
>>
>> Here is what I don't understand from what you said.? "The child process is created with a single thread?the one that called fork()."? To me that implies that the thread that called fork() is the same thread as the child process.? I guess you're talking about the distinction between logical threads and physical threads.? 
>>
>
> The thread that called fork is cloned into a new thread in the new process.
> It has a clone of the processor registers of the thread, stack memory segment of that thread,
> which means that it has all the stack variables and stack frames from the parent process's thread.
>
>
>
>> But the main issue is your suggestion that I should call fork-execv from the thread that runs the main C program, not from a separate physical pthread.? That would certainly eliminate the overhead of creating a new pthread.?
>>
>
> Forget about physical threads, they only matter when you are performance tuning your code.
>
>
>> I am working now to finish this, and I will try your suggestion of calling fork-execv from the "main" thread.? When I reply back next I can give you a complete picture of what I'm doing.?
>>
>> Your comments, and those of Peter Holzer and Chris Angelico, are most appreciated.?
>>
>
> Barry
>
>
>


From vinay_sajip at yahoo.co.uk  Wed Dec  8 12:43:21 2021
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Wed, 8 Dec 2021 17:43:21 +0000 (UTC)
Subject: ANN: distlib 0.3.4 released on PyPI
References: <809668465.15074333.1638985401332.ref@mail.yahoo.com>
Message-ID: <809668465.15074333.1638985401332@mail.yahoo.com>

I've recently released version 0.3.4 of distlib on PyPI [1]. For newcomers,
distlib is a library of packaging functionality which is intended to be
usable as the basis for third-party packaging tools.

The main changes in this release are as follows:

* Fixed #153: Raise warnings in get_distributions() if bad metadata seen, but keep
? going.

* Fixed #154: Determine Python versions correctly for Python >= 3.10.

* Updated launcher executables with changes to handle duplication logic.

Code relating to support for Python 2.6 was also removed (support for Python 2.6 was
dropped in an earlier release, but supporting code wasn't removed until now).

A more detailed change log is available at [2].

Please try it out, and if you find any problems or have any suggestions for improvements,
please give some feedback using the issue tracker! [3]

Regards,

Vinay Sajip

[1] https://pypi.org/project/distlib/0.3.4/
[2] https://distlib.readthedocs.io/en/0.3.4/
[3] https://bitbucket.org/pypa/distlib/issues/new


From dieter at handshake.de  Wed Dec  8 12:48:06 2021
From: dieter at handshake.de (Dieter Maurer)
Date: Wed, 8 Dec 2021 18:48:06 +0100
Subject: HTML extraction
In-Reply-To: <CALk2KRVGa-0DEwivNJbnDsLuyg8rJjAB=Vo5ZAW-J0FdswKuFQ@mail.gmail.com>
References: <CAEsMKX3TkUK==fNcZVZXhDrEWFA8RW6PTY47quACz7LmJ-Xy_Q@mail.gmail.com>
 <CAPTjJmp7F5M-R+yG763x8uEDFoVD_rUomDDHv8hsXFqUun20uA@mail.gmail.com>
 <CALk2KRVGa-0DEwivNJbnDsLuyg8rJjAB=Vo5ZAW-J0FdswKuFQ@mail.gmail.com>
Message-ID: <25008.61398.312692.292346@ixdm.fritz.box>

Roland Mueller wrote at 2021-12-7 22:55 +0200:
> ...
>Can bs4 or lxml cope with HTML code that does not comply with XML as the
>following fragment?

`lxml` comes with an HTML parser; that can be configured to check loosely.

From dieter at handshake.de  Wed Dec  8 13:02:01 2021
From: dieter at handshake.de (Dieter Maurer)
Date: Wed, 8 Dec 2021 19:02:01 +0100
Subject: Odd locale error that has disappeared on reboot.
In-Reply-To: <p5b48i-o59f1.ln1@esprimo.zbmc.eu>
References: <p5b48i-o59f1.ln1@esprimo.zbmc.eu>
Message-ID: <25008.62233.919289.573481@ixdm.fritz.box>

Chris Green wrote at 2021-12-7 15:28 +0000:
>I have a very short Python program that runs on one of my Raspberry
>Pis to collect temperatures from a 1-wire sensor and write them to a
>database:-
> ...
>At 03:40 last night it suddenly started throwing the following error every
>time it ran:-
>
>    Fatal Python error: initfsencoding: Unable to get the locale encoding
>    LookupError: unknown encoding: UTF-8
>
>    Current thread 0xb6f8db40 (most recent call first):
>    Aborted
>
>Running the program from the command line produced the same error.
>Restarting the Pi system has fixed the problem.

Python has not its own locale database but uses that of the operating
system. From my point of view, the operating system state seems
to have got corrupted. A restart apparently has ensured a clean state again.

From lcwarner11 at gmail.com  Wed Dec  8 13:18:11 2021
From: lcwarner11 at gmail.com (Larry Warner)
Date: Wed, 8 Dec 2021 13:18:11 -0500
Subject: python problem
Message-ID: <CAG2cnigNOpPLMC2VUANmrpKFCwxDUwss7WCJ3LaH5spDRKPttA@mail.gmail.com>

I am new at Python. I have installed Python 3.10.1 and the latest Pycharm.
When I attempt to execute anything via Pycharm or the command line, I
receive a message it can not find Python.

I do not know where Python was loaded or where to find and to update PATH
to the program.

Larry

From mohsen.owzar at gmail.com  Wed Dec  8 04:40:10 2021
From: mohsen.owzar at gmail.com (Mohsen Owzar)
Date: Wed, 8 Dec 2021 01:40:10 -0800 (PST)
Subject: For a hierarchical project, the EXE file generated by
 "pyinstaller" does not start.
In-Reply-To: <mailman.34.1638900994.15287.python-list@python.org>
References: <CAPTjJmqcqSTRmfSJpyGagh=ZcM--XGN9zVW7RoD8XKLg6QQ+9A@mail.gmail.com>
 <a1e1d8be-e2cc-4187-a600-01c2cbb3c8fbn@googlegroups.com>
 <mailman.34.1638900994.15287.python-list@python.org>
Message-ID: <2abb9404-aa1e-45fc-b887-d97a0a837bf3n@googlegroups.com>

Chris Angelico schrieb am Dienstag, 7. Dezember 2021 um 19:16:54 UTC+1:
> On Wed, Dec 8, 2021 at 4:49 AM Mohsen Owzar <mohsen... at gmail.com> wrote: 
> > ******************************************************************* 
> > GPIOContrl.py 
> > ******************************************************************* 
> > class GPIOControl: 
> > def my_print(self, args): 
> > if print_allowed == 1: 
> > print(args) 
> > 
> > def __init__(self):
> Can't much help with your main question as I don't do Windows, but one 
> small side point: Instead of having a my_print that checks if printing 
> is allowed, you can conditionally replace the print function itself. 
> 
> if not print_allowed: 
> def print(*args, **kwargs): pass 
> 
> ChrisA

Thanks Chris
Your answer didn't help me to solve my problem, but gave me another idea to write a conditional print statement.

Regards
Mohsen

From pieter-l at vanoostrum.org  Wed Dec  8 05:00:26 2021
From: pieter-l at vanoostrum.org (Pieter van Oostrum)
Date: Wed, 08 Dec 2021 11:00:26 +0100
Subject: HTML extraction
References: <CAEsMKX3TkUK==fNcZVZXhDrEWFA8RW6PTY47quACz7LmJ-Xy_Q@mail.gmail.com>
 <CAPTjJmp7F5M-R+yG763x8uEDFoVD_rUomDDHv8hsXFqUun20uA@mail.gmail.com>
 <CALk2KRVGa-0DEwivNJbnDsLuyg8rJjAB=Vo5ZAW-J0FdswKuFQ@mail.gmail.com>
 <mailman.36.1638910538.15287.python-list@python.org>
Message-ID: <m2bl1rwh51.fsf@vanoostrum.org>

Roland Mueller <roland.em0001 at googlemail.com> writes:

> But isn't bs4 only for SOAP content?
> Can bs4 or lxml cope with HTML code that does not comply with XML as the
> following fragment?
>
> <p>A
> <p>B
> <hr>
>

bs4 can do it, but lxml wants correct XML.

Jupyter console 6.4.0

Python 3.9.9 (main, Nov 16 2021, 07:21:43) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.29.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from bs4 import BeautifulSoup as bs

In [2]: soup = bs('<p>A<p>B<hr>')

In [3]: soup.p
Out[3]: <p>A</p>

In [4]: soup.find_all('p')
Out[4]: [<p>A</p>, <p>B</p>]

In [5]: from lxml import etree

In [6]: root = etree.fromstring('<p>A<p>B<hr>')
Traceback (most recent call last):

  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3444, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "/var/folders/2l/pdng2d2x18d00m41l6r2ccjr0000gn/T/ipykernel_96220/3376613260.py", line 1, in <module>
    root = etree.fromstring('<p>A<p>B<hr>')

  File "src/lxml/etree.pyx", line 3237, in lxml.etree.fromstring

  File "src/lxml/parser.pxi", line 1896, in lxml.etree._parseMemoryDocument

  File "src/lxml/parser.pxi", line 1777, in lxml.etree._parseDoc

  File "src/lxml/parser.pxi", line 1082, in lxml.etree._BaseParser._parseUnicodeDoc

  File "src/lxml/parser.pxi", line 615, in lxml.etree._ParserContext._handleParseResultDoc

  File "src/lxml/parser.pxi", line 725, in lxml.etree._handleParseResult

  File "src/lxml/parser.pxi", line 654, in lxml.etree._raiseParseError

  File "<string>", line 1
XMLSyntaxError: Premature end of data in tag hr line 1, line 1, column 13
-- 
Pieter van Oostrum <pieter at vanoostrum.org>
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

From cl at isbd.net  Wed Dec  8 04:50:32 2021
From: cl at isbd.net (Chris Green)
Date: Wed, 8 Dec 2021 09:50:32 +0000
Subject: Odd locale error that has disappeared on reboot.
References: <p5b48i-o59f1.ln1@esprimo.zbmc.eu> <soo3ip$u1v$1@dont-email.me>
Message-ID: <8ob68i-v6dj1.ln1@esprimo.zbmc.eu>

Julio Di Egidio <julio at diegidio.name> wrote:
> On 07/12/2021 16:28, Chris Green wrote:
> > I have a very short Python program that runs on one of my Raspberry
> > Pis to collect temperatures from a 1-wire sensor and write them to a
> > database:-
> > 
> >      #!/usr/bin/python3
> >      #
> >      #
> >      # read temperature from 1-wire sensor and store in database with date and time
> >      #
> >      import sqlite3
> >      import time
> > 
> >      ftxt = str(open("/sys/bus/w1/devices/28-01204e1e64c3/w1_slave").read(100))
> >      temp = (float(ftxt[ftxt.find("t=") +2:]))/1000
> >      #
> >      #
> >      # insert date, time and temperature into the database
> >      #
> >      tdb = sqlite3.connect("/home/chris/.cfg/share/temperature/temperature.db")
> >      cr = tdb.cursor()
> >      dt = time.strftime("%Y-%m-%d %H:%M")
> > cr.execute("Insert INTO temperatures (DateTime, Temperature) VALUES(?, 
> round(?, 2))", (dt, temp) 
> >      )
> >      tdb.commit()
> >      tdb.close()
> > 
> > It's run by cron every 10 minutes.
> > 
> > 
> > At 03:40 last night it suddenly started throwing the following error every
> > time it ran:-
> > 
> >      Fatal Python error: initfsencoding: Unable to get the locale encoding
> >      LookupError: unknown encoding: UTF-8
> > 
> >      Current thread 0xb6f8db40 (most recent call first):
> >      Aborted
> > 
> > Running the program from the command line produced the same error.
> > Restarting the Pi system has fixed the problem.
> > 
> > 
> > What could have caused this?  I certainly wasn't around at 03:40! :-)
> > There aren't any automatic updates enabled on the system, the only
> > thing that might have been going on was a backup as that Pi is also
> > my 'NAS' with a big USB drive connected to it.  The backups have been
> > running without problems for more than a year.  Looking at the system
> > logs shows that a backup was started at 03:35 so I suppose that *could*
> > have provoked something but I fail to understand how.
> 
> Since it's a one-off, doesn't sound like a system problem.  The easiest 
> might be that you try-catch that call and retry when needed, and I'd 
> also check that 'ftxt' is what it should be: "external devices" may 
> fail, including when they do produce output...
> 
Well it repeated every ten minutes through the night until I rebooted
the system, so it wasn't really a "one off".  I hasn't repeated since
though.

-- 
Chris Green
?

From ml_news at posteo.de  Wed Dec  8 06:31:36 2021
From: ml_news at posteo.de (Manfred Lotz)
Date: Wed, 8 Dec 2021 12:31:36 +0100
Subject: How to package a Python command line app?
Message-ID: <20211208123136.52690b4f.ml_news@posteo.de>

The are many possibilities to package a Python app, and I have to admit
I am pretty confused.

Here is what I have:

A Python command line app which requires some packages which are not in
the standard library.

I am on Linux and like to have an executable (perhaps a zip file with a
shebang; whatever) which runs on different Linux systems.

Different mean
- perhaps different glibc versions
- perhaps different Python versions

In my specific case this is: 
- RedHat 8.4 with Python 3.6.8
- Ubuntu 20.04 LTS with Python 3.8.10 
- and finally Fedora 33 with Python 3.9.9


Is this possible to do? If yes which tool would I use for this?


-- 
Manfred


From loris.bennett at fu-berlin.de  Wed Dec  8 09:38:48 2021
From: loris.bennett at fu-berlin.de (Loris Bennett)
Date: Wed, 08 Dec 2021 15:38:48 +0100
Subject: How to package a Python command line app?
References: <20211208123136.52690b4f.ml_news@posteo.de>
Message-ID: <87v8zzb1qf.fsf@hornfels.zedat.fu-berlin.de>

Hi Manfred,

Manfred Lotz <ml_news at posteo.de> writes:

> The are many possibilities to package a Python app, and I have to admit
> I am pretty confused.
>
> Here is what I have:
>
> A Python command line app which requires some packages which are not in
> the standard library.
>
> I am on Linux and like to have an executable (perhaps a zip file with a
> shebang; whatever) which runs on different Linux systems.
>
> Different mean
> - perhaps different glibc versions
> - perhaps different Python versions
>
> In my specific case this is: 
> - RedHat 8.4 with Python 3.6.8
> - Ubuntu 20.04 LTS with Python 3.8.10 
> - and finally Fedora 33 with Python 3.9.9
>
>
> Is this possible to do? If yes which tool would I use for this?

I use poetry[1] on CentOS 7 to handle all the dependencies and create a
wheel which I then install to a custom directory with pip3.

You would checkout the repository with your code on the target system,
start a poetry shell using the Python version required, and then build
the wheel.  From outside the poetry shell you can set PYTHONUSERBASE and
then install with pip3.  You then just need to set PYTHONPATH
appropriately where ever you want to use your software.

Different Python versions shouldn't be a problem.  If some module
depends on a specific glibc version, then you might end up in standard
dependency-hell territory, but you can pin module versions of
dependencies in poetry, and you could also possibly use different
branches within your repository to handle that.

HTH

Loris   

Footnotes: 
[1]  https://python-poetry.org

-- 
This signature is currently under construction.

From ml_news at posteo.de  Wed Dec  8 11:47:00 2021
From: ml_news at posteo.de (Manfred Lotz)
Date: Wed, 8 Dec 2021 17:47:00 +0100
Subject: How to package a Python command line app?
References: <20211208123136.52690b4f.ml_news@posteo.de>
 <87v8zzb1qf.fsf@hornfels.zedat.fu-berlin.de>
Message-ID: <20211208174700.038dd59b.ml_news@posteo.de>

Hi Loris,

On Wed, 08 Dec 2021 15:38:48 +0100
"Loris Bennett" <loris.bennett at fu-berlin.de> wrote:

> Hi Manfred,
> 
> Manfred Lotz <ml_news at posteo.de> writes:
> 
> > The are many possibilities to package a Python app, and I have to
> > admit I am pretty confused.
> >
> > Here is what I have:
> >
> > A Python command line app which requires some packages which are
> > not in the standard library.
> >
> > I am on Linux and like to have an executable (perhaps a zip file
> > with a shebang; whatever) which runs on different Linux systems.
> >
> > Different mean
> > - perhaps different glibc versions
> > - perhaps different Python versions
> >
> > In my specific case this is: 
> > - RedHat 8.4 with Python 3.6.8
> > - Ubuntu 20.04 LTS with Python 3.8.10 
> > - and finally Fedora 33 with Python 3.9.9
> >
> >
> > Is this possible to do? If yes which tool would I use for this?  
> 
> I use poetry[1] on CentOS 7 to handle all the dependencies and create
> a wheel which I then install to a custom directory with pip3.
> 
> You would checkout the repository with your code on the target system,
> start a poetry shell using the Python version required, and then build
> the wheel.  From outside the poetry shell you can set PYTHONUSERBASE
> and then install with pip3.  You then just need to set PYTHONPATH
> appropriately where ever you want to use your software.
> 

In my case it could happen that I do not have access to the target
system but wants to handover the Python app to somebody else. This
person wants just to run it.


> Different Python versions shouldn't be a problem.  If some module
> depends on a specific glibc version, then you might end up in standard
> dependency-hell territory, but you can pin module versions of
> dependencies in poetry, and you could also possibly use different
> branches within your repository to handle that.
> 

I try to avoid using modules which depeng on specific glibc. 

Although, it seems that it doesn't really help for my use case I will
play with poetry to get a better understanding of its capabilities.

-- 
Thanks a lot,
Manfred




From mats at wichmann.us  Wed Dec  8 13:43:50 2021
From: mats at wichmann.us (Mats Wichmann)
Date: Wed, 8 Dec 2021 11:43:50 -0700
Subject: python problem
In-Reply-To: <CAG2cnigNOpPLMC2VUANmrpKFCwxDUwss7WCJ3LaH5spDRKPttA@mail.gmail.com>
References: <CAG2cnigNOpPLMC2VUANmrpKFCwxDUwss7WCJ3LaH5spDRKPttA@mail.gmail.com>
Message-ID: <8ad9cfa1-575f-6992-f6b5-d3199d4a6b8f@wichmann.us>

On 12/8/21 11:18, Larry Warner wrote:
> I am new at Python. I have installed Python 3.10.1 and the latest Pycharm.
> When I attempt to execute anything via Pycharm or the command line, I
> receive a message it can not find Python.
> 
> I do not know where Python was loaded or where to find and to update PATH
> to the program.

Since you don't mention which operating system you are using or where 
you got your Python from, it's hard for anyone to help.

Do the general notes here help?

https://docs.python.org/3/using/

This problem usually strikes on Windows, where all the commands don't go 
into a small number of directories which are searched by default. 
Following the Windows link in the page above should describe some 
approaches to dealing with that.


From juliushamilton100 at gmail.com  Wed Dec  8 14:39:05 2021
From: juliushamilton100 at gmail.com (Julius Hamilton)
Date: Wed, 8 Dec 2021 20:39:05 +0100
Subject: Short, perfect program to read sentences of webpage
Message-ID: <CAEsMKX31XWuhSLumh4smFk2BuTEt1OAuUa+wYwqAwfL1MOXqqg@mail.gmail.com>

Hey,

This is something I have been working on for a very long time. It?s one of
the reasons I got into programming at all. I?d really appreciate if people
could input some advice on this.

This is a really simple program which extracts the text from webpages and
displays them one sentence at a time. It?s meant to help you study dense
material, especially documentation, with much more focus and comprehension.
I actually hope it can be of help to people who have difficulty reading. I
know it?s been of use to me at least.

This is a minimally acceptable way to pull it off currently:

deepreader.py:

import sys
import requests
import html2text
import nltk

url = sys.argv[1]

# Get the html, pull out the text, and sentence-segment it in one line of
code

sentences = nltk.sent_tokenize(html2text.html2text(requests.get(url).text))

# Activate an elementary reader interface for the text

for index, sentence in enumerate(sentences):

  # Print the sentence
  print(?\n? + str(index) + ?/? + str(len(sentences)) + ?: ? + sentence +
?\n?)

  # Wait for user key-press
  x = input(?\n> ?)


EOF



That?s it.

A lot of refining is possible, and I?d really like to see how some more
experienced people might handle it.

1. The HTML extraction is not perfect. It doesn?t produce as clean text as
I would like. Sometimes random links or tags get left in there. And the
sentences are sometimes randomly broken by newlines.

2. Neither is the segmentation perfect. I am currently researching
developing an optimal segmenter with tools from Spacy.

Brevity is greatly valued. I mean, anyone who can make the program more
perfect, that?s hugely appreciated. But if someone can do it in very few
lines of code, that?s also appreciated.

Thanks very much,
Julius

From python at mrabarnett.plus.com  Wed Dec  8 15:58:52 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 8 Dec 2021 20:58:52 +0000
Subject: Short, perfect program to read sentences of webpage
In-Reply-To: <CAEsMKX31XWuhSLumh4smFk2BuTEt1OAuUa+wYwqAwfL1MOXqqg@mail.gmail.com>
References: <CAEsMKX31XWuhSLumh4smFk2BuTEt1OAuUa+wYwqAwfL1MOXqqg@mail.gmail.com>
Message-ID: <be1f58f0-1e59-4f96-7e11-118f5faab9c8@mrabarnett.plus.com>

On 2021-12-08 19:39, Julius Hamilton wrote:
> Hey,
> 
> This is something I have been working on for a very long time. It?s one of
> the reasons I got into programming at all. I?d really appreciate if people
> could input some advice on this.
> 
> This is a really simple program which extracts the text from webpages and
> displays them one sentence at a time. It?s meant to help you study dense
> material, especially documentation, with much more focus and comprehension.
> I actually hope it can be of help to people who have difficulty reading. I
> know it?s been of use to me at least.
> 
> This is a minimally acceptable way to pull it off currently:
> 
> deepreader.py:
> 
> import sys
> import requests
> import html2text
> import nltk
> 
> url = sys.argv[1]
> 
> # Get the html, pull out the text, and sentence-segment it in one line of
> code
> 
> sentences = nltk.sent_tokenize(html2text.html2text(requests.get(url).text))
> 
> # Activate an elementary reader interface for the text
> 
> for index, sentence in enumerate(sentences):
> 
>    # Print the sentence
>    print(?\n? + str(index) + ?/? + str(len(sentences)) + ?: ? + sentence +
> ?\n?)
> 
You can shorten that with format strings:

     print("\n{}/{}: {}\n".format(index, len(sentences), sentence))

or even:

     print(f"\n{index}/{len(sentences)}: {sentence}\n")

>    # Wait for user key-press
>    x = input(?\n> ?)
> 
> 
> EOF
> 
> 
> 
> That?s it.
> 
> A lot of refining is possible, and I?d really like to see how some more
> experienced people might handle it.
> 
> 1. The HTML extraction is not perfect. It doesn?t produce as clean text as
> I would like. Sometimes random links or tags get left in there. And the
> sentences are sometimes randomly broken by newlines.
> 
> 2. Neither is the segmentation perfect. I am currently researching
> developing an optimal segmenter with tools from Spacy.
> 
> Brevity is greatly valued. I mean, anyone who can make the program more
> perfect, that?s hugely appreciated. But if someone can do it in very few
> lines of code, that?s also appreciated.
> 
> Thanks very much,
> Julius
> 


From cs at cskk.id.au  Wed Dec  8 16:12:17 2021
From: cs at cskk.id.au (Cameron Simpson)
Date: Thu, 9 Dec 2021 08:12:17 +1100
Subject: Short, perfect program to read sentences of webpage
In-Reply-To: <CAEsMKX31XWuhSLumh4smFk2BuTEt1OAuUa+wYwqAwfL1MOXqqg@mail.gmail.com>
References: <CAEsMKX31XWuhSLumh4smFk2BuTEt1OAuUa+wYwqAwfL1MOXqqg@mail.gmail.com>
Message-ID: <YbEfse0mIKyqzZnv@cskk.homeip.net>

Assorted remarks inline below:

On 08Dec2021 20:39, Julius Hamilton <juliushamilton100 at gmail.com> wrote:
>deepreader.py:
>
>import sys
>import requests
>import html2text
>import nltk
>
>url = sys.argv[1]

I might spell this:

    cmd, url = sys.argv

which enforces exactly one argument. And since you don't care about the 
command name, maybe:

    _, url = sys.argv

because "_" is a conventional name for "a value we do not care about".

>sentences = nltk.sent_tokenize(html2text.html2text(requests.get(url).text))

Neat!

># Activate an elementary reader interface for the text
>for index, sentence in enumerate(sentences):

I would be inclined to count from 1, so "enumerate(sentences, 1)".

>  # Print the sentence
>  print(?\n? + str(index) + ?/? + str(len(sentences)) + ?: ? + sentence +
>?\n?)

Personally, since print() adds a trailing newline, I would drop the 
final +"\n". If you want an additional blank line, I would put it in the 
input() call below:

>  # Wait for user key-press
>  x = input(?\n> ?)

You're not using "x". Just discard input()'s return value:

    input("\n> ")

>A lot of refining is possible, and I?d really like to see how some more
>experienced people might handle it.
>
>1. The HTML extraction is not perfect. It doesn?t produce as clean text as
>I would like. Sometimes random links or tags get left in there.

Maybe try beautifulsoup instead of html2text? The module name is "bs4".

>And the
>sentences are sometimes randomly broken by newlines.

I would flatten the newlines. Either the simple:

    sentence = sentence.strip().replace("\n", " ")

or maybe better:

    sentence = " ".join(sentence.split()

Cheers,
Cameron Simpson <cs at cskk.id.au>

From cs at cskk.id.au  Wed Dec  8 16:44:52 2021
From: cs at cskk.id.au (Cameron Simpson)
Date: Thu, 9 Dec 2021 08:44:52 +1100
Subject: Python child process in while True loop blocks parent
In-Reply-To: <MqQ0tig--3-2@tutanota.com>
References: <MqQ0tig--3-2@tutanota.com>
Message-ID: <YbEnVHyeqrNr3sBx@cskk.homeip.net>

On 08Dec2021 18:11, Jen Kris <jenkris at tutanota.com> wrote:
>Python must terminate its write strings with \n, or read will block in 
>C waiting for something that never comes.

There are two aspects to this:
- if upstream is rding "lines of text" then you need a newline to 
  terminate the lines
- you (probably) should flush the output pipe (Python to C) after the 
  newline

I see you're using file descriptors and os.write() to sent data. This is 
unbuffered, so there is nothing to flush, so you have not encountered 
the second point above.

But if you shift to using a Python file object for your output (eg 
f=os.fdopen(pw)), which would let you use print() or any number of other 
things which do things with Python files) your file object would have a 
buffer and normally that would not be sent to the pipe unless it was 
full.

So your deadlock issue has 2 components:
- you need to terminate your records for upstream (C) to see complete 
  records. Your records are lines, so you need a newline character.
- you need to ensure the whole record has been sent upstream (been 
  written to the pipe). If you use a buffered Python file object for 
  your output, you need to flush it at your synchronisation points or 
  upstream will not receive the buffer contents. That synchronisation 
  point for you is the end of the record.

Hopefully this makes the flow considerations more clear.

Cheers,
Cameron Simpson <cs at cskk.id.au>

From jon+usenet at unequivocal.eu  Wed Dec  8 17:19:59 2021
From: jon+usenet at unequivocal.eu (Jon Ribbens)
Date: Wed, 8 Dec 2021 22:19:59 -0000 (UTC)
Subject: Short, perfect program to read sentences of webpage
References: <CAEsMKX31XWuhSLumh4smFk2BuTEt1OAuUa+wYwqAwfL1MOXqqg@mail.gmail.com>
 <mailman.47.1638995449.15287.python-list@python.org>
Message-ID: <slrnsr2bsf.4j7.jon+usenet@raven.unequivocal.eu>

On 2021-12-08, Julius Hamilton <juliushamilton100 at gmail.com> wrote:
> 1. The HTML extraction is not perfect. It doesn?t produce as clean text as
> I would like. Sometimes random links or tags get left in there. And the
> sentences are sometimes randomly broken by newlines.

Oh. Leaving tags in suggests you are doing this very wrongly. Python
has plenty of open source libraries you can use that will parse the
HTML reliably into tags and text for you.

> 2. Neither is the segmentation perfect. I am currently researching
> developing an optimal segmenter with tools from Spacy.
>
> Brevity is greatly valued. I mean, anyone who can make the program more
> perfect, that?s hugely appreciated. But if someone can do it in very few
> lines of code, that?s also appreciated.

It isn't something that can be done in a few lines of code. There's the
spaces issue you mention for example. Nor is it something that can
necessarily be done just by inspecting the HTML alone. To take a trivial
example:

  powergen<div>italia</div>          = powergen <nl> italia

but:

  powergen<span>italia</span>        = powergenitalia

but the second with the addition of:

  <style>span { dispaly: block }</style>

is back to "powergen <nl> italia". So you need to parse and apply styles
(including external stylesheets) as well. Potentially you may also need
to execute JavaScript on the page, which means you also need a JavaScript
interpreter and a DOM implementation. Basically you need a complete
browser to do it on general web pages.

From hjp-python at hjp.at  Wed Dec  8 17:36:17 2021
From: hjp-python at hjp.at (Peter J. Holzer)
Date: Wed, 8 Dec 2021 23:36:17 +0100
Subject: Python child process in while True loop blocks parent
In-Reply-To: <MqQ0tig--3-2@tutanota.com>
References: <27742189-51F0-49A8-ACA7-344EFCD2B1BD@barrys-emacs.org>
 <MqAitZV--3-2@tutanota.com>
 <0633A53B-A722-4CEE-8F20-E2D0AEBB8D01@barrys-emacs.org>
 <MqC-YXO--B-2@tutanota.com>
 <20211206020831.ytkglawzaeoueyhh@hjp.at>
 <MqFiCj---3-2@tutanota.com>
 <3065988D-E2CF-4C36-B319-1DEC87B864E3@barrys-emacs.org>
 <MqGZH61--3-2@tutanota.com>
 <6E104B3E-1FD8-476F-A560-87CF47B183CC@barrys-emacs.org>
 <MqQ0tig--3-2@tutanota.com>
Message-ID: <20211208223617.ebcp3hjfcjtqr44k@hjp.at>

On 2021-12-08 18:11:48 +0100, Jen Kris via Python-list wrote:
> To recap, I'm using a pair of named pipes for IPC between C and
> Python.? Python runs as a child process after fork-execv.? The Python
> program continues to run concurrently in a while True loop, and
> responds to requests from C at intervals, and continues to run until
> it receives a signal from C to exit.? C sends signals to Python, then
> waits to receive data back from Python.? My problem was that C was
> blocked when Python started.?
> 
> The solution was twofold:? (1) for Python to run concurrently it must
> be a multiprocessing loop (from the multiprocessing module),

I don't see how this could achieve anything. It starts another (third)
process, but then it just does all the work in that process and just
waits for it. Doing the same work in the original Python process should
have exactly the same effect.

> and (2) Python must terminate its write strings with \n, or read will
> block in C waiting for something that never comes.

That's also strange. You are using os.write in Python and read in C,
both of which shoudn't care about newlines.

> The multiprocessing module sidesteps the GIL; without multiprocessing
> the GIL will block all other threads once Python starts.?

Your Python interpreter runs in a different process than your C code.
There is absolutely no way the GIL could block threads in your C
program. And your Python code doesn't need to use more than one thread
or process.

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp at hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://mail.python.org/pipermail/python-list/attachments/20211208/940ba61a/attachment.sig>

From cs at cskk.id.au  Wed Dec  8 17:42:07 2021
From: cs at cskk.id.au (Cameron Simpson)
Date: Thu, 9 Dec 2021 09:42:07 +1100
Subject: Short, perfect program to read sentences of webpage
In-Reply-To: <sentences-20211208223927@ram.dialup.fu-berlin.de>
References: <sentences-20211208223927@ram.dialup.fu-berlin.de>
Message-ID: <YbE0vxHz+vd7hKHo@cskk.homeip.net>

On 08Dec2021 21:41, Stefan Ram <ram at zedat.fu-berlin.de> wrote:
>Julius Hamilton <juliushamilton100 at gmail.com> writes:
>>This is a really simple program which extracts the text from webpages and
>>displays them one sentence at a time.
>
>  Our teacher said NLTK will not come up until next year, so
>  I tried to do with regexps. It still has bugs, for example
>  it can not tell the dot at the end of an abbreviation from
>  the dot at the end of a sentence!

This is almost a classic demo of why regexps are a poor tool as a first 
choice. You can do much with them, but they are cryptic and bug prone.

I am not seeking to mock you, but trying to make apparent why regexps 
are to be avoided a lot of the time. They have their place.

You've read the whole re module docs I hope:

    https://docs.python.org/3/library/re.html#module-re

>import re
>import urllib.request
>uri = r'''http://example.com/article''' # replace this with your URI!
>request = urllib.request.Request( uri )
>resource = urllib.request.urlopen( request )
>cs = resource.headers.get_content_charset()
>content = resource.read().decode( cs, errors="ignore" )
>content = re.sub( r'''[\r\n\t\s]+''', r''' ''', content )

You're not multiline, so I would recommend a plain raw string:

    content = re.sub( r'[\r\n\t\s]+', r' ', content )

No need for \r in the class, \s covers that. From the docs:

  \s
    For Unicode (str) patterns:

      Matches Unicode whitespace characters (which includes [ 
      \t\n\r\f\v], and also many other characters, for example the 
      non-breaking spaces mandated by typography rules in many 
      languages). If the ASCII flag is used, only [ \t\n\r\f\v] is 
      matched.

>upper = r"[A-Z?????????????????????????????]" # "[\\p{Lu}]"
>lower = r"[a-z????????????????????????????????]" # "[\\p{Ll}]"

This is very fragile - you have an arbitrary set of additional uppercase 
characters, almost certainly incomplete, and visually hard to inspect 
for completeness.

Instead, consider the \b (word boundary) and \w (word character) 
markers, which will let you break strings up, and then maybe test the 
results with str.isupper().

>digit = r"[0-9]" #"[\\p{Nd}]"

There's a \d character class for this, covers nondecimal digits too.

>firstwordstart = upper;
>firstwordnext = "(?:[a-z????????????????????????????????-])";

Again, an inline arbitrary list of characters. This is fragile.

>wordcharacter = "[A-Z?????????????????????????????a-z??????????????????\
>??????????????0-9-]"

Again inline. Why not construct it?

    wordcharacter = upper + lower + digit

but I recommend \w instead, or for this: [\w\d]

>addition = "(?:(?:[']" + wordcharacter + "+)*[']?)?"

As a matter of good practice with regexp strings, use raw quotes:

    addition = r"(?:(?:[']" + wordcharacter + r"+)*[']?)?"

even when there are no backslahes.

Seriously, doing this with regexps is difficult. A useful exercise for 
learning regexps, but in the general case not the first tool to reach 
for.

Cheers,
Cameron Simpson <cs at cskk.id.au>

From hjp-python at hjp.at  Wed Dec  8 18:09:47 2021
From: hjp-python at hjp.at (Peter J. Holzer)
Date: Thu, 9 Dec 2021 00:09:47 +0100
Subject: Short, perfect program to read sentences of webpage
In-Reply-To: <YbE0vxHz+vd7hKHo@cskk.homeip.net>
References: <sentences-20211208223927@ram.dialup.fu-berlin.de>
 <YbE0vxHz+vd7hKHo@cskk.homeip.net>
Message-ID: <20211208230947.o4wcrq4r26x3l752@hjp.at>

On 2021-12-09 09:42:07 +1100, Cameron Simpson wrote:
> On 08Dec2021 21:41, Stefan Ram <ram at zedat.fu-berlin.de> wrote:
> >Julius Hamilton <juliushamilton100 at gmail.com> writes:
> >>This is a really simple program which extracts the text from webpages and
> >>displays them one sentence at a time.
> >
> >  Our teacher said NLTK will not come up until next year, so
> >  I tried to do with regexps. It still has bugs, for example
> >  it can not tell the dot at the end of an abbreviation from
> >  the dot at the end of a sentence!
> 
> This is almost a classic demo of why regexps are a poor tool as a first 
> choice. You can do much with them, but they are cryptic and bug prone.

I don't think that's problem here. The problem is that natural languages
just aren't regular languages. In fact I'm not sure that they fit
anywhere within the Chomsky hierarchy (but if they aren't type-0, that
would be a strong argument against the possibility of human-level AI).

In English, if a sentence ends with an abbreviation you write only a
single dot. So if you look at these two fragments:

    For matching strings, numbers, etc. Python provides regular
    expressions.

    Let's say you want to match strings, numbers, etc. Python provides
    regular expresssions for these tasks.

In second case the dot ends a sentence in the first it doesn't. But to
distinguish those cases you need to at least parse the sentences at the
syntax level (which regular expressions can't do), maybe even understand
them semantically.

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp at hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://mail.python.org/pipermail/python-list/attachments/20211209/b3c841c4/attachment.sig>

From pablogsal at gmail.com  Wed Dec  8 18:51:48 2021
From: pablogsal at gmail.com (Pablo Galindo Salgado)
Date: Wed, 8 Dec 2021 23:51:48 +0000
Subject: [RELEASE] Python 3.11.0a3 is available
Message-ID: <CAFjbc8GDtQ867ap5GoL2zJCGByGxrQWRoCcRTM6hB8sWxng3kA@mail.gmail.com>

You can tell that we are slowly getting closer to the first beta as the
number of release blockers that we need to fix on every release starts to
increase [image: :sweat_smile:] But we did it! Thanks to Steve Dower, Ned
Deily, Christian Heimes, ?ukasz Langa and Mark Shannon that helped get
things ready for this release :)

Go get the new version here:

https://www.python.org/downloads/release/python-3110a3/

**This is an early developer preview of Python 3.11**

# Major new features of the 3.11 series, compared to 3.10

Python 3.11 is still in development.  This release, 3.11.0a3 is the third
of seven planned alpha releases.

Alpha releases are intended to make it easier to test the current state of
new features and bug fixes and to test the release process.

During the alpha phase, features may be added up until the start of the
beta phase (2022-05-06) and, if necessary, may be modified or deleted up
until the release candidate phase (2022-08-01).  Please keep in mind that
this is a preview release and its use is **not** recommended for production
environments.

Many new features for Python 3.11 are still being planned and written.
Among the new major new features and changes so far:

* [PEP 657](https://www.python.org/dev/peps/pep-0657/) -- Include
Fine-Grained Error Locations in Tracebacks
* [PEP 654](https://www.python.org/dev/peps/pep-0654/) -- Exception Groups
and except*
* The [Faster Cpython Project](https://github.com/faster-cpython) is
already yielding some exciting results: this version of CPython 3.11 is
~12% faster on the geometric mean of the [PyPerformance benchmarks](
speed.python.org), compared to 3.10.0.
 * Hey, **fellow core developer,** if a feature you find important is
missing from this list, let me know.

The next pre-release of Python 3.11 will be 3.11.0a4, currently scheduled
for Monday, 2022-01-03.

# More resources

* [Online Documentation](https://docs.python.org/3.11/)
* [PEP 664](https://www.python.org/dev/peps/pep-0664/), 3.11 Release
Schedule
* Report bugs at [https://bugs.python.org](https://bugs.python.org).
* [Help fund Python and its community](/psf/donations/).

# And now for something completely different

Rayleigh scattering, named after the nineteenth-century British physicist
Lord Rayleigh is the predominantly elastic scattering of light or other
electromagnetic radiation by particles much smaller than the wavelength of
the radiation. For light frequencies well below the resonance frequency of
the scattering particle, the amount of scattering is inversely proportional
to the fourth power of the wavelength. Rayleigh scattering results from the
electric polarizability of the particles. The oscillating electric field of
a light wave acts on the charges within a particle, causing them to move at
the same frequency. The particle, therefore, becomes a small radiating
dipole whose radiation we see as scattered light. The particles may be
individual atoms or molecules; it can occur when light travels through
transparent solids and liquids but is most prominently seen in gases.

The strong wavelength dependence of the scattering means that shorter
(blue) wavelengths are scattered more strongly than longer (red)
wavelengths. This results in the indirect blue light coming from all
regions of the sky.

# We hope you enjoy those new releases!

Thanks to all of the many volunteers who help make Python Development and
these releases possible! Please consider supporting our efforts by
volunteering yourself or through organization contributions to the Python
Software Foundation.

Your friendly release team,
Pablo Galindo @pablogsal
Ned Deily @nad
Steve Dower @steve.dower

From python at mrabarnett.plus.com  Wed Dec  8 20:31:13 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 9 Dec 2021 01:31:13 +0000
Subject: Short, perfect program to read sentences of webpage
In-Reply-To: <regexps-20211209001519@ram.dialup.fu-berlin.de>
References: <sentences-20211208223927@ram.dialup.fu-berlin.de>
 <YbE0vxHz+vd7hKHo@cskk.homeip.net>
 <mailman.52.1639003341.15287.python-list@python.org>
 <regexps-20211209001519@ram.dialup.fu-berlin.de>
Message-ID: <719155b1-5be1-88f6-de16-6c10a9516940@mrabarnett.plus.com>

On 2021-12-08 23:17, Stefan Ram wrote:
> Cameron Simpson <cs at cskk.id.au> writes:
>>Instead, consider the \b (word boundary) and \w (word character) 
>>markers, which will let you break strings up, and then maybe test the 
>>results with str.isupper().
> 
>    Thanks for your comments, most or all of them are
>    valid, and I will try to take them into account!
> 
>    Regexps might have their disadvantages, but when I use them,
>    it is clearer for me to do all the matching with regexps
>    instead of mixing them with Python calls like str.isupper.
>    Therefore, it is helpful for me to have a regexp to match
>    upper and lower case characters separately. Some regexp
>    dialects support "\p{Lu}" and "\p{Ll}" for this.
> 
If you want "\p{Lu}" and "\p{Ll}", have a look at the 'regex' module on 
PyPI:

     https://pypi.org/project/regex/

[snip]

From songofacandy at gmail.com  Wed Dec  8 21:56:04 2021
From: songofacandy at gmail.com (Inada Naoki)
Date: Thu, 9 Dec 2021 11:56:04 +0900
Subject: Odd locale error that has disappeared on reboot.
In-Reply-To: <p5b48i-o59f1.ln1@esprimo.zbmc.eu>
References: <p5b48i-o59f1.ln1@esprimo.zbmc.eu>
Message-ID: <CAEfz+TyNJmS4LZgNLuXsbcdwtAz739gXLqp-jFzT85Qg1Ey2yw@mail.gmail.com>

On Wed, Dec 8, 2021 at 2:52 AM Chris Green <cl at isbd.net> wrote:
>
>
> At 03:40 last night it suddenly started throwing the following error every
> time it ran:-
>
>     Fatal Python error: initfsencoding: Unable to get the locale encoding
>     LookupError: unknown encoding: UTF-8
>
>     Current thread 0xb6f8db40 (most recent call first):
>     Aborted
>
> Running the program from the command line produced the same error.
> Restarting the Pi system has fixed the problem.
>

This error means Python can not find its standard libraries. There are
some possibilities.

* You set the wrong PYTHONHOME
  PYTHONHOME is very rarely useful. It shouldn't be used if you can
not solve this kind of problem.

* Your Python installation is broken.
  Some files are deleted or overwritten. You need to *clean* install
Python again.

Bets,
-- 
Inada Naoki  <songofacandy at gmail.com>

From cs at cskk.id.au  Wed Dec  8 21:58:15 2021
From: cs at cskk.id.au (Cameron Simpson)
Date: Thu, 9 Dec 2021 13:58:15 +1100
Subject: Short, perfect program to read sentences of webpage
In-Reply-To: <regexps-20211209001519@ram.dialup.fu-berlin.de>
References: <regexps-20211209001519@ram.dialup.fu-berlin.de>
Message-ID: <YbFwxzZE8ObSFD0F@cskk.homeip.net>

On 08Dec2021 23:17, Stefan Ram <ram at zedat.fu-berlin.de> wrote:
>  Regexps might have their disadvantages, but when I use them,
>  it is clearer for me to do all the matching with regexps
>  instead of mixing them with Python calls like str.isupper.
>  Therefore, it is helpful for me to have a regexp to match
>  upper and lower case characters separately. Some regexp
>  dialects support "\p{Lu}" and "\p{Ll}" for this.

Aye. I went looking for that in the Python re module docs and could not 
find them. So the comprimise is match any word, then test the word with 
isupper() (or whatever is appropriate).

>  I have not yet incorporated (all) your advice into my code,
>  but I came to the conclusion myself that the repetition of
>  long sequences like r"A-Z?????????????????????????????" and
>  not using f strings to insert other strings was especially
>  ugly.

The tricky bit with f-strings and regexps is that \w{3,5} means from 3 
through 5 "word characters". So if you've got those in an f-string 
you're off to double-the-brackets land, a bit like double backslash land 
and non-raw-strings.

Otherwise, yes f-strings are a nice way to compose things.

Cheers,
Cameron Simpson <cs at cskk.id.au>

From barry at barrys-emacs.org  Thu Dec  9 06:57:11 2021
From: barry at barrys-emacs.org (Barry Scott)
Date: Thu, 9 Dec 2021 11:57:11 +0000
Subject: Python child process in while True loop blocks parent
In-Reply-To: <MqQ0tig--3-2@tutanota.com>
References: <MpqihHm--3-2@tutanota.com>
 <27742189-51F0-49A8-ACA7-344EFCD2B1BD@barrys-emacs.org>
 <MqAitZV--3-2@tutanota.com>
 <0633A53B-A722-4CEE-8F20-E2D0AEBB8D01@barrys-emacs.org>
 <MqC-YXO--B-2@tutanota.com> <20211206020831.ytkglawzaeoueyhh@hjp.at>
 <MqFiCj---3-2@tutanota.com>
 <3065988D-E2CF-4C36-B319-1DEC87B864E3@barrys-emacs.org>
 <MqGZH61--3-2@tutanota.com>
 <6E104B3E-1FD8-476F-A560-87CF47B183CC@barrys-emacs.org>
 <MqQ0tig--3-2@tutanota.com>
Message-ID: <793F9B19-935A-4CA1-8848-7AFCF48D8741@barrys-emacs.org>



> On 8 Dec 2021, at 17:11, Jen Kris <jenkris at tutanota.com> wrote:
> 
> I started this post on November 29, and there have been helpful comments since then from Barry Scott, Cameron Simpson, Peter Holzer and Chris Angelico.  Thanks to all of you.  
> 
> I've found a solution that works for my purpose, and I said earlier that I would post the solution I found. If anyone has a better solution I would appreciate any feedback. 
> 
> To recap, I'm using a pair of named pipes for IPC between C and Python.  Python runs as a child process after fork-execv.  The Python program continues to run concurrently in a while True loop, and responds to requests from C at intervals, and continues to run until it receives a signal from C to exit.  C sends signals to Python, then waits to receive data back from Python.  My problem was that C was blocked when Python started. 
> 
> The solution was twofold:  (1) for Python to run concurrently it must be a multiprocessing loop (from the multiprocessing module), and (2) Python must terminate its write strings with \n, or read will block in C waiting for something that never comes.  The multiprocessing module sidesteps the GIL; without multiprocessing the GIL will block all other threads once Python starts. 
> 
> Originally I used epoll() on the pipes.  Cameron Smith and Barry Scott advised against epoll, and for this case they are right.  Blocking pipes work here, and epoll is too much overhead for watching on a single file descriptor. 
> 
> This is the Python code now:
> 
> #!/usr/bin/python3
> from multiprocessing import Process

You already have feedback that multiprocessing is not required.

> import os
> 
> print("Python is running")
> 
> child_pid = os.getpid()
> print('child process id:', child_pid)
> 
> def f(a, b):
> 
>     print("Python now in function f")
> 
>     pr = os.open('/tmp/Pipe_01', os.O_RDONLY)
>     print("File Descriptor1 Opened " + str(pr))
>     pw = os.open('/tmp/Pipe_02', os.O_WRONLY)
>     print("File Descriptor2 Opened " + str(pw))
> 
>     while True:
> 
>         v = os.read(pr,64)
>         print("Python read from pipe pr")
>         print(v)
> 
>         if v == b'99':
>             os.close(pr)
>             os.close(pw)
>             print("Python is terminating")
>             os._exit(os.EX_OK)
> 
>         if v != "Send child PID":
>             os.write(pw, b"OK message received\n")

The \n should not be required as UDS (unix domain sockets) are message based not a stream of bytes.

>             print("Python wrote back")
> 
> if __name__ == '__main__':
>     a = 0
>     b = 0
>     p = Process(target=f, args=(a, b,))
>     p.start()
>     p.join()
> 
> The variables a and b are not currently used in the body, but they will be later. 
> 
> This is the part of the C code that communicates with Python:
> 
>     fifo_fd1 = open(fifo_path1, O_WRONLY);
>     fifo_fd2 = open(fifo_path2, O_RDONLY);
> 
>     status_write = write(fifo_fd1, py_msg_01, sizeof(py_msg_01));
>     if (status_write < 0) perror("write");
You continue on after the error, exit would be better.
> 
>     status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_01));
>     if (status_read < 0) perror("read");
>     printf("C received message 1 from Python\n");
>     printf("%.*s",(int)buf_len, fifo_readbuf);

The length of the data read is in status_read not buf_len.

> 
>     status_write = write(fifo_fd1, py_msg_02, sizeof(py_msg_02));

How much data did you put into py_msg_01 buffer?
Is it a C string? If so you want to write sizeof(py_msg_02)-1 to avoid sending the trailing NUL (0)
of the C string.

>     if (status_write < 0) perror("write");
If it failed exit until you have figured out the error handling.
> 
>     status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_02));
>     if (status_read < 0) perror("read");
>     printf("C received message 2 from Python\n");
>     printf("%.*s",(int)buf_len, fifo_readbuf);

The length of the data read is in status_read not buf_len.

At no point do I see in this C code the need for a \n in the data sent between python and C.

> 
>     // Terminate Python multiprocessing
>     printf("C is sending exit message to Python\n");
>     status_write = write(fifo_fd1, py_msg_03, 2);

Would be better to not be using "2" for the length to write here.
If py_msg_03 only has the exit msg in it the use sizeof(py_msg_03).
If its a C string the subtract 1 for the trailing NUL (0).

> 
>     printf("C is closing\n");
>     close(fifo_fd1);
>     close(fifo_fd2);
> 
> Screen output:
> 
> Python is running
> child process id: 5353
> Python now in function f
> File Descriptor1 Opened 6
> Thread created 0
> File Descriptor2 Opened 7
> Process ID: 5351
> Parent Process ID: 5351
> I am the parent
> Core joined 0
> I am the child
> Python read from pipe pr
> b'Hello to Python from C\x00\x00'
The \x00 is because the length you used is wrong.

> Python wrote back
> C received message 1 from Python
> OK message received
> Python read from pipe pr
> b'Message to Python 2\x00\x00'
The \x00 is because the length you used is wrong.

> Python wrote back
> C received message 2 from Python
> OK message received
> C is sending exit message to Python
> C is closing
> Python read from pipe pr
> b'99'
> Python is terminating
> 
> Python runs on a separate thread (created with pthreads) because I want the flexibility of using this same basic code as a stand-alone .exe, or for a C extension from Python called with ctypes.  If I use it as a C extension then I want the Python code on a separate thread because I can't have two instances of the Python interpreter running on one thread, and one instance will already be running on the main thread, albeit "suspended" by the call from ctypes. 
> 
> So that's my solution:  (1) Python multiprocessing module; (2) Python strings written to the pipe must be terminated with \n. 
> 
> Thanks again to all who commented. 
> 
> 
> 
> Dec 6, 2021, 13:33 by barry at barrys-emacs.org:
> 
> 
>> On 6 Dec 2021, at 21:05, Jen Kris <jenkris at tutanota.com <mailto:jenkris at tutanota.com>> wrote:
>> 
>> Here is what I don't understand from what you said.  "The child process is created with a single thread?the one that called fork()."  To me that implies that the thread that called fork() is the same thread as the child process.  I guess you're talking about the distinction between logical threads and physical threads.  
> 
> The thread that called fork is cloned into a new thread in the new process.
> It has a clone of the processor registers of the thread, stack memory segment of that thread,
> which means that it has all the stack variables and stack frames from the parent process's thread.
> 
> 
>> But the main issue is your suggestion that I should call fork-execv from the thread that runs the main C program, not from a separate physical pthread.  That would certainly eliminate the overhead of creating a new pthread. 
> 
> Forget about physical threads, they only matter when you are performance tuning your code.
> 
>> I am working now to finish this, and I will try your suggestion of calling fork-execv from the "main" thread.  When I reply back next I can give you a complete picture of what I'm doing. 
>> 
>> Your comments, and those of Peter Holzer and Chris Angelico, are most appreciated. 
> 
> Barry
> 
> 
> 


From smitapandey078 at gmail.com  Thu Dec  9 05:39:50 2021
From: smitapandey078 at gmail.com (smita)
Date: Thu, 9 Dec 2021 16:09:50 +0530
Subject: help 
Message-ID: <841FE0A6-A2D1-4F62-B289-B2155F9603A9@hxcore.ol>

   ?

   I am not able to open python on my laptop plzz help

   ?

   Sent from [1]Mail for Windows

   ?

References

   Visible links
   1. https://go.microsoft.com/fwlink/?LinkId=550986

From loris.bennett at fu-berlin.de  Thu Dec  9 03:16:50 2021
From: loris.bennett at fu-berlin.de (Loris Bennett)
Date: Thu, 09 Dec 2021 09:16:50 +0100
Subject: How to package a Python command line app?
References: <20211208123136.52690b4f.ml_news@posteo.de>
 <87v8zzb1qf.fsf@hornfels.zedat.fu-berlin.de>
 <20211208174700.038dd59b.ml_news@posteo.de>
Message-ID: <87tufi5h1p.fsf@hornfels.zedat.fu-berlin.de>

Hi Manfred,

Manfred Lotz <ml_news at posteo.de> writes:

> Hi Loris,
>
> On Wed, 08 Dec 2021 15:38:48 +0100
> "Loris Bennett" <loris.bennett at fu-berlin.de> wrote:
>
>> Hi Manfred,
>> 
>> Manfred Lotz <ml_news at posteo.de> writes:
>> 
>> > The are many possibilities to package a Python app, and I have to
>> > admit I am pretty confused.
>> >
>> > Here is what I have:
>> >
>> > A Python command line app which requires some packages which are
>> > not in the standard library.
>> >
>> > I am on Linux and like to have an executable (perhaps a zip file
>> > with a shebang; whatever) which runs on different Linux systems.
>> >
>> > Different mean
>> > - perhaps different glibc versions
>> > - perhaps different Python versions
>> >
>> > In my specific case this is: 
>> > - RedHat 8.4 with Python 3.6.8
>> > - Ubuntu 20.04 LTS with Python 3.8.10 
>> > - and finally Fedora 33 with Python 3.9.9
>> >
>> >
>> > Is this possible to do? If yes which tool would I use for this?  
>> 
>> I use poetry[1] on CentOS 7 to handle all the dependencies and create
>> a wheel which I then install to a custom directory with pip3.
>> 
>> You would checkout the repository with your code on the target system,
>> start a poetry shell using the Python version required, and then build
>> the wheel.  From outside the poetry shell you can set PYTHONUSERBASE
>> and then install with pip3.  You then just need to set PYTHONPATH
>> appropriately where ever you want to use your software.
>> 
>
> In my case it could happen that I do not have access to the target
> system but wants to handover the Python app to somebody else. This
> person wants just to run it.

For what ever reasons, there does not seem to be much focus on this kind
of deployment for Python.  Similar to the way things are with Perl, the
assumption seems to be that you have a working environment and install
any dependencies needed to get the program you have been given working.

I have never used it, but you might want to look at something like
pyinstaller 

  https://pyinstaller.readthedocs.io 

However, it looks as if it is aimed towards bundling a single script.  I
don't know how it would work if you have a more complex program
consisting of a number of modules.

>> Different Python versions shouldn't be a problem.  If some module
>> depends on a specific glibc version, then you might end up in standard
>> dependency-hell territory, but you can pin module versions of
>> dependencies in poetry, and you could also possibly use different
>> branches within your repository to handle that.
>> 
>
> I try to avoid using modules which depeng on specific glibc. 

So would I, but you mentioned it above in your definition of
'different'. 

> Although, it seems that it doesn't really help for my use case I will
> play with poetry to get a better understanding of its capabilities.

You're right, poetry doesn't seem to address your main problem.
Nevertheless, it might be useful for developing your program before you
get to the question of how to distribute it

Cheers,

Loris

-- 
This signature is currently under construction.

From cl at isbd.net  Thu Dec  9 05:38:58 2021
From: cl at isbd.net (Chris Green)
Date: Thu, 9 Dec 2021 10:38:58 +0000
Subject: Odd locale error that has disappeared on reboot.
References: <p5b48i-o59f1.ln1@esprimo.zbmc.eu> <soo3ip$u1v$1@dont-email.me>
 <8ob68i-v6dj1.ln1@esprimo.zbmc.eu> <soqrj3$57n$1@dont-email.me>
Message-ID: <2v298i-jtko1.ln1@esprimo.zbmc.eu>

Julio Di Egidio <julio at diegidio.name> wrote:
> On 08/12/2021 10:50, Chris Green wrote:
> > Julio Di Egidio <julio at diegidio.name> wrote:
> >> On 07/12/2021 16:28, Chris Green wrote:
> >>> What could have caused this?  I certainly wasn't around at 03:40! :-)
> >>> There aren't any automatic updates enabled on the system, the only
> >>> thing that might have been going on was a backup as that Pi is also
> >>> my 'NAS' with a big USB drive connected to it.  The backups have been
> >>> running without problems for more than a year.  Looking at the system
> >>> logs shows that a backup was started at 03:35 so I suppose that *could*
> >>> have provoked something but I fail to understand how.
> >>
> >> Since it's a one-off, doesn't sound like a system problem.  The easiest
> >> might be that you try-catch that call and retry when needed, and I'd
> >> also check that 'ftxt' is what it should be: "external devices" may
> >> fail, including when they do produce output...
> >>
> > Well it repeated every ten minutes through the night until I rebooted
> > the system, so it wasn't really a "one off".  I hasn't repeated since
> > though.
> 
> Still your code wouldn't pass review: you do need some exception 
> handling there, that it's not failing right now is no excuse.  Moreover, 
> next time it fails, you won't be any wiser since you keep not handling 
> it...  Unless this is all just for fun, in which case of course never mind.
> 
It is basically 'just for fun' we wanted to see how warm/cold the
cats' home above our garage was now that winter has arrived.

However catching and re-trying isn't going to help at all.  It happily
produced the same arror every 10 minutes throughout the night until I
rebooted the system.  I suppose I *could* reboot after (say) three
failures but it seems a bit drastic! :-)

-- 
Chris Green
?

From cl at isbd.net  Thu Dec  9 05:42:48 2021
From: cl at isbd.net (Chris Green)
Date: Thu, 9 Dec 2021 10:42:48 +0000
Subject: Odd locale error that has disappeared on reboot.
References: <p5b48i-o59f1.ln1@esprimo.zbmc.eu>
 <CAEfz+TyNJmS4LZgNLuXsbcdwtAz739gXLqp-jFzT85Qg1Ey2yw@mail.gmail.com>
 <mailman.56.1639018592.15287.python-list@python.org>
Message-ID: <86398i-jtko1.ln1@esprimo.zbmc.eu>

Inada Naoki <songofacandy at gmail.com> wrote:
> On Wed, Dec 8, 2021 at 2:52 AM Chris Green <cl at isbd.net> wrote:
> >
> >
> > At 03:40 last night it suddenly started throwing the following error every
> > time it ran:-
> >
> >     Fatal Python error: initfsencoding: Unable to get the locale encoding
> >     LookupError: unknown encoding: UTF-8
> >
> >     Current thread 0xb6f8db40 (most recent call first):
> >     Aborted
> >
> > Running the program from the command line produced the same error.
> > Restarting the Pi system has fixed the problem.
> >
> 
> This error means Python can not find its standard libraries. There are
> some possibilities.
> 
> * You set the wrong PYTHONHOME
>   PYTHONHOME is very rarely useful. It shouldn't be used if you can
> not solve this kind of problem.
> 
> * Your Python installation is broken.
>   Some files are deleted or overwritten. You need to *clean* install
> Python again.
> 
So how was it that rebooting the system fixed the error?  It has been
working perfectly OK now for a few days since the above failure and
I've changed nothing.

I think it was probably a bit of power supply noise or something that
corrupted a memory read or write, or something of that sort anyway. 


-- 
Chris Green
?

From mats at wichmann.us  Thu Dec  9 11:49:23 2021
From: mats at wichmann.us (Mats Wichmann)
Date: Thu, 9 Dec 2021 09:49:23 -0700
Subject: help
In-Reply-To: <841FE0A6-A2D1-4F62-B289-B2155F9603A9@hxcore.ol>
References: <841FE0A6-A2D1-4F62-B289-B2155F9603A9@hxcore.ol>
Message-ID: <e611115e-4061-035d-2e03-a762c89e4329@wichmann.us>

On 12/9/21 03:39, smita wrote:
>      
> 
>     I am not able to open python on my laptop plzz help

You're going to have to provide some details.



From ikorot01 at gmail.com  Thu Dec  9 11:52:40 2021
From: ikorot01 at gmail.com (Igor Korot)
Date: Thu, 9 Dec 2021 10:52:40 -0600
Subject: help
In-Reply-To: <841FE0A6-A2D1-4F62-B289-B2155F9603A9@hxcore.ol>
References: <841FE0A6-A2D1-4F62-B289-B2155F9603A9@hxcore.ol>
Message-ID: <CA+FnnTwzkahkNDFpHXiLuZacfwMBE3OdQ_T1rjH5d4oWu3eQYQ@mail.gmail.com>

Hi,

On Thu, Dec 9, 2021 at 10:31 AM smita <smitapandey078 at gmail.com> wrote:
>
>
>
>    I am not able to open python on my laptop plzz help

What do you mean by saying "open python"?

Thank you.

>
>
>
>    Sent from [1]Mail for Windows
>
>
>
> References
>
>    Visible links
>    1. https://go.microsoft.com/fwlink/?LinkId=550986
> --
> https://mail.python.org/mailman/listinfo/python-list

From dieter at handshake.de  Thu Dec  9 12:04:15 2021
From: dieter at handshake.de (Dieter Maurer)
Date: Thu, 9 Dec 2021 18:04:15 +0100
Subject: HTML extraction
In-Reply-To: <m2bl1rwh51.fsf@vanoostrum.org>
References: <CAEsMKX3TkUK==fNcZVZXhDrEWFA8RW6PTY47quACz7LmJ-Xy_Q@mail.gmail.com>
 <CAPTjJmp7F5M-R+yG763x8uEDFoVD_rUomDDHv8hsXFqUun20uA@mail.gmail.com>
 <CALk2KRVGa-0DEwivNJbnDsLuyg8rJjAB=Vo5ZAW-J0FdswKuFQ@mail.gmail.com>
 <mailman.36.1638910538.15287.python-list@python.org>
 <m2bl1rwh51.fsf@vanoostrum.org>
Message-ID: <25010.14095.995966.277030@ixdm.fritz.box>

Pieter van Oostrum wrote at 2021-12-8 11:00 +0100:
> ...
>bs4 can do it, but lxml wants correct XML.

Use `lxml's the `HTMLParser` to parse HTML
(--> "see https://lxml.de/parsing.html#parsing-html").

From dieter at handshake.de  Thu Dec  9 12:18:26 2021
From: dieter at handshake.de (Dieter Maurer)
Date: Thu, 9 Dec 2021 18:18:26 +0100
Subject: How to package a Python command line app?
In-Reply-To: <20211208123136.52690b4f.ml_news@posteo.de>
References: <20211208123136.52690b4f.ml_news@posteo.de>
Message-ID: <25010.14946.854543.991758@ixdm.fritz.box>

Manfred Lotz wrote at 2021-12-8 12:31 +0100:
>The are many possibilities to package a Python app, and I have to admit
>I am pretty confused.
>
>Here is what I have:
>
>A Python command line app which requires some packages which are not in
>the standard library.

Are they on PyPI or can they be downloaded to PyPI?
In this case, you could install it via `pip` and
a "requirements" file (describing what is necessary).

You could also build an archive (e.g. `tar`, `zip`)
and create small installer script which unpacks and installs as necessary.
You could put the archive at the end of the script (such
that you have a single file which gets executed to do everything).

From barry at barrys-emacs.org  Thu Dec  9 12:34:03 2021
From: barry at barrys-emacs.org (Barry)
Date: Thu, 9 Dec 2021 17:34:03 +0000
Subject: How to package a Python command line app?
In-Reply-To: <20211208123136.52690b4f.ml_news@posteo.de>
References: <20211208123136.52690b4f.ml_news@posteo.de>
Message-ID: <4E247BF6-1D23-414E-BCA2-63C1112D0004@barrys-emacs.org>



> On 8 Dec 2021, at 18:27, Manfred Lotz <ml_news at posteo.de> wrote:
> 
> ?The are many possibilities to package a Python app, and I have to admit
> I am pretty confused.
> 
> Here is what I have:
> 
> A Python command line app which requires some packages which are not in
> the standard library.
> 
> I am on Linux and like to have an executable (perhaps a zip file with a
> shebang; whatever) which runs on different Linux systems.
> 
> Different mean
> - perhaps different glibc versions
> - perhaps different Python versions
> 
> In my specific case this is: 
> - RedHat 8.4 with Python 3.6.8
> - Ubuntu 20.04 LTS with Python 3.8.10 
> - and finally Fedora 33 with Python 3.9.9
> 
> 
> Is this possible to do? If yes which tool would I use for this?

Have a look at pyinstaller it knows how to collect up all your dependancies and build a folder of files you can distribute. It?s also got a single file mode that you might find useful. The end user does not need to install python.

I am looking at using it to build gui apps a macOs.

You will have to experiment to find out if it solves you glib case concerns.

Barry
> 
> 
> -- 
> Manfred
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 


From ml_news at posteo.de  Thu Dec  9 13:35:37 2021
From: ml_news at posteo.de (Manfred Lotz)
Date: Thu, 9 Dec 2021 19:35:37 +0100
Subject: How to package a Python command line app?
References: <20211208123136.52690b4f.ml_news@posteo.de>
 <4E247BF6-1D23-414E-BCA2-63C1112D0004@barrys-emacs.org>
 <mailman.64.1639071248.15287.python-list@python.org>
Message-ID: <20211209193537.6e9920b1.ml_news@posteo.de>

On Thu, 9 Dec 2021 17:34:03 +0000
Barry <barry at barrys-emacs.org> wrote:

> > On 8 Dec 2021, at 18:27, Manfred Lotz <ml_news at posteo.de> wrote:
> > 
> > ?The are many possibilities to package a Python app, and I have to
> > admit I am pretty confused.
> > 
> > Here is what I have:
> > 
> > A Python command line app which requires some packages which are
> > not in the standard library.
> > 
> > I am on Linux and like to have an executable (perhaps a zip file
> > with a shebang; whatever) which runs on different Linux systems.
> > 
> > Different mean
> > - perhaps different glibc versions
> > - perhaps different Python versions
> > 
> > In my specific case this is: 
> > - RedHat 8.4 with Python 3.6.8
> > - Ubuntu 20.04 LTS with Python 3.8.10 
> > - and finally Fedora 33 with Python 3.9.9
> > 
> > 
> > Is this possible to do? If yes which tool would I use for this?  
> 
> Have a look at pyinstaller it knows how to collect up all your
> dependancies and build a folder of files you can distribute. It?s
> also got a single file mode that you might find useful. The end user
> does not need to install python.
> 
> I am looking at using it to build gui apps a macOs.
> 
> You will have to experiment to find out if it solves you glib case
> concerns.
> 

I played with pyinstaller which worked fine. However, it builds a
dynamic executable and thus it is glibc version dependent. Means, I
have to build different executables for differen glibc versions.

So, it seems I will have to check how executable zip archives are
supposed to work.

-- 
Manfred


From ml_news at posteo.de  Thu Dec  9 13:39:36 2021
From: ml_news at posteo.de (Manfred Lotz)
Date: Thu, 9 Dec 2021 19:39:36 +0100
Subject: How to package a Python command line app?
References: <20211208123136.52690b4f.ml_news@posteo.de>
 <25010.14946.854543.991758@ixdm.fritz.box>
 <mailman.63.1639070308.15287.python-list@python.org>
Message-ID: <20211209193936.2b7cb2c2.ml_news@posteo.de>

On Thu, 9 Dec 2021 18:18:26 +0100
"Dieter Maurer" <dieter at handshake.de> wrote:

> Manfred Lotz wrote at 2021-12-8 12:31 +0100:
> >The are many possibilities to package a Python app, and I have to
> >admit I am pretty confused.
> >
> >Here is what I have:
> >
> >A Python command line app which requires some packages which are not
> >in the standard library.  
> 
> Are they on PyPI or can they be downloaded to PyPI?
> In this case, you could install it via `pip` and
> a "requirements" file (describing what is necessary).
> 

Acutally, those packages are on pypi.

I created a minimal example where I have two files

hello.py
========

#!/usr/bin/python3

from message import hello
import typer

def main():
    hello()

if __name__ == "__main__":
    typer.run(main)

message.py 
==========

def hello():
    print("hello world")

pyinstaller worked fine taking care of message.py and typer module. But
as said in my other reply it is glibc version dependent.


> You could also build an archive (e.g. `tar`, `zip`)
> and create small installer script which unpacks and installs as
> necessary. You could put the archive at the end of the script (such
> that you have a single file which gets executed to do everything).

This I will try next.


-- 
Manfred





From hjp-python at hjp.at  Thu Dec  9 15:07:17 2021
From: hjp-python at hjp.at (Peter J. Holzer)
Date: Thu, 9 Dec 2021 21:07:17 +0100
Subject: Odd locale error that has disappeared on reboot.
In-Reply-To: <2v298i-jtko1.ln1@esprimo.zbmc.eu>
References: <p5b48i-o59f1.ln1@esprimo.zbmc.eu> <soo3ip$u1v$1@dont-email.me>
 <8ob68i-v6dj1.ln1@esprimo.zbmc.eu> <soqrj3$57n$1@dont-email.me>
 <2v298i-jtko1.ln1@esprimo.zbmc.eu>
Message-ID: <20211209200717.eapmcfznvnjybthz@hjp.at>

On 2021-12-09 10:38:58 +0000, Chris Green wrote:
> Julio Di Egidio <julio at diegidio.name> wrote:
> > Still your code wouldn't pass review: you do need some exception 
> > handling there
[...]
> However catching and re-trying isn't going to help at all.  It happily
> produced the same arror every 10 minutes throughout the night until I
> rebooted the system.

I agree. Only catch exceptions which you can reasonably handle. For
those which you can't handle (and "some part of my Python installation
has just gone AWOL" is one of them) crashing with a stack trace is
almost certainly your best option.

> I suppose I *could* reboot after (say) three failures

Maybe, but I wouldn't do that in the python program itself, but in a
separate monitor process.

> but it seems a bit drastic! :-)

Yup. I'd do that only if I've run out of other options (which has
happened in the past and probably will happen in the future).

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp at hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://mail.python.org/pipermail/python-list/attachments/20211209/ac576da4/attachment.sig>

From hjp-python at hjp.at  Thu Dec  9 15:23:58 2021
From: hjp-python at hjp.at (Peter J. Holzer)
Date: Thu, 9 Dec 2021 21:23:58 +0100
Subject: How to package a Python command line app?
In-Reply-To: <20211209193537.6e9920b1.ml_news@posteo.de>
References: <20211208123136.52690b4f.ml_news@posteo.de>
 <4E247BF6-1D23-414E-BCA2-63C1112D0004@barrys-emacs.org>
 <mailman.64.1639071248.15287.python-list@python.org>
 <20211209193537.6e9920b1.ml_news@posteo.de>
Message-ID: <20211209202358.vr4ae244tacuhigk@hjp.at>

On 2021-12-09 19:35:37 +0100, Manfred Lotz wrote:
> I played with pyinstaller which worked fine. However, it builds a
> dynamic executable and thus it is glibc version dependent. Means, I
> have to build different executables for differen glibc versions.

Just build it for the oldest version. Even if you don't, it may not
matter. I just compiled a hello world program on Ubuntu 20 and ran it on
Debian 6 (which is 10 years old). Obviusly a real program has more
opportunities to run into compatibility problems, but glibc doesn't
change all that much.

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp at hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://mail.python.org/pipermail/python-list/attachments/20211209/c0eeb1c3/attachment.sig>

From roland.em0001 at googlemail.com  Thu Dec  9 15:46:28 2021
From: roland.em0001 at googlemail.com (=?UTF-8?Q?Roland_M=c3=bcller?=)
Date: Thu, 9 Dec 2021 22:46:28 +0200
Subject: Return
In-Reply-To: <CAHVaEHiirVBrs3h4yNkqqqChU2cZsoeN-4wJ_7q71jYJLjr_1Q@mail.gmail.com>
References: <CAHVaEHhKkpB49o5fUzwXH3s0r_zsTwDUsFy1jLk8b+NWGpXAGA@mail.gmail.com>
 <CALk2KRXdXypvMTJrAkPR0xL6BheLaS1u+Ot3KWWQ8feMqVh_jQ@mail.gmail.com>
 <CAHVaEHiirVBrs3h4yNkqqqChU2cZsoeN-4wJ_7q71jYJLjr_1Q@mail.gmail.com>
Message-ID: <707730d8-52e0-265c-1e85-d05feac77bb9@googlemail.com>

Hello,


On 12/8/21 11:29, vani arul wrote:
> Thanks for your help.
> I am not good at programming.My code works perfectly.But I am bit 
> confused about the return values.
>
> Binary Search Program
>
> /def Binarysearch(a,key):
> ??? l=0
> ??? r=len(a)-1
> ??? while l<=r:
> ??????? med = (l+r)//2
> ??????? if key==a[med]:
> ??????????? return med
> ??????? elif key>a[med]:
> ??????????? l=med+1/
>
> /? ? ? elif key<a[med]:
> ??????????? r=med-1
> ??? return -1
> a=[6,8,23,33,44,55,66]
> found=Binarysearch(a,66)
> print("result:",found)"""/
>
> ?if key==a[med] condition is satisfied,the the block returns med.
>
> But in the following block,how does it return values if there is no 
> return specified?
>
First of all please reply always to the list address!

If Python does not encounter a return statement it will continue to run 
the function.

In your example the while loop will continue until the condition 'l<=r' 
is true. If Python exits the while loop then there is a return -1.

BR,

Roland



>
>
> Regards
> Vani
>
> On Wed, Dec 8, 2021 at 2:15 AM Roland Mueller 
> <roland.em0001 at googlemail.com> wrote:
>
>     Hello
>
>     ti 7. jouluk. 2021 klo 19.47 vani arul (arulvani66 at gmail.com)
>     kirjoitti:
>
>         Hey There,
>         Can someone help to understand how a python function can
>         return value with
>         using return in the code?
>         It is not not about explicit or implicit function call.
>
>
>     Not sure whether I understood your question: I have a simple
>     example about return.
>     ? * f() and h() explicitely return something
>     ? * g() and h() return None
>
>     BTW, also Python documentation tool pydoc has an article about return.
>
>     #!/usr/bin/python
>
>     def f():
>     ? ? return 42
>     def g():
>     ? ? pass
>     def h():
>     ? ? return
>
>     if __name__ == "__main__":
>     ? ? print(f"f(): {f()}")
>     ? ? print(f"g(): {g()}")
>     ? ? print(f"h(): {h()}")
>
>     Result:
>     f(): 42
>     g(): None
>     h(): None
>
>     Pydoc:
>     $ pydoc return
>
>     BR,
>     Roland
>
>
>         Thanks
>         Vani
>         -- 
>         https://mail.python.org/mailman/listinfo/python-list
>

From mats at wichmann.us  Thu Dec  9 15:51:00 2021
From: mats at wichmann.us (Mats Wichmann)
Date: Thu, 9 Dec 2021 13:51:00 -0700
Subject: How to package a Python command line app?
In-Reply-To: <20211209193537.6e9920b1.ml_news@posteo.de>
References: <20211208123136.52690b4f.ml_news@posteo.de>
 <4E247BF6-1D23-414E-BCA2-63C1112D0004@barrys-emacs.org>
 <mailman.64.1639071248.15287.python-list@python.org>
 <20211209193537.6e9920b1.ml_news@posteo.de>
Message-ID: <ab47c868-62bf-6194-7825-9134fb6089b8@wichmann.us>

On 12/9/21 11:35, Manfred Lotz wrote:

> I played with pyinstaller which worked fine. However, it builds a
> dynamic executable and thus it is glibc version dependent. Means, I
> have to build different executables for differen glibc versions.
> 
> So, it seems I will have to check how executable zip archives are
> supposed to work.
> 

For me at least, I'm still not sure what you are trying to accomplish.

"A Python command line app which requires some packages which are not in
the standard library."

If your code and its dependencies are "pure python" then you only need 
to build a wheel package and you should be good to go.  If you think you 
have Python version issues, most of the good checkers will take a 
version argument, so if you think you need to be compatible with 
3.6-3.10 you give it a version of 3.6 to check against, and it should 
spot things that don't work all the way back to that base version, and 
then you can find ways to code around it.

If the things you call on are binary, it gets a little more complicated. 
Wheels that contain compiled bits need to match the version of Python 
that's going to run them (unless they use the stable ABI, but that's not 
terribly common yet).

You shouldn't run into glibc versioning problems.  Most of glibc has 
been extraordinarily stable for nearly two decades, and within the range 
of distributions you've mentioned there should not be problems unless 
something is reaching into very esoteric areas.  Other system libraries 
- maybe not so much.  If you really think you're going to have this 
level of binary-compatibility problem, the flavor-of-the-month technique 
is to build a self-contained bundle, such as a Snap or Flatpak, or...

Quote:

 > New packaging formats like Snap, Flatpak and AppImage are providing 
distribution agnostic packages that work on most Linux distributions.

But that's no longer a Python-specific question at that point...

As I said, I can't really deduce the details of what you're trying to 
accomplish, just hoping you don't buy yourself more trouble than you 
actually need to - this might be easier than you think.


From ml_news at posteo.de  Fri Dec 10 06:32:24 2021
From: ml_news at posteo.de (Manfred Lotz)
Date: Fri, 10 Dec 2021 12:32:24 +0100
Subject: How to package a Python command line app?
References: <20211208123136.52690b4f.ml_news@posteo.de>
 <4E247BF6-1D23-414E-BCA2-63C1112D0004@barrys-emacs.org>
 <mailman.64.1639071248.15287.python-list@python.org>
 <20211209193537.6e9920b1.ml_news@posteo.de>
 <20211209202358.vr4ae244tacuhigk@hjp.at>
 <mailman.66.1639081440.15287.python-list@python.org>
Message-ID: <20211210123224.6a4e71dd.ml_news@posteo.de>

On Thu, 9 Dec 2021 21:23:58 +0100
"Peter J. Holzer" <hjp-python at hjp.at> wrote:

> On 2021-12-09 19:35:37 +0100, Manfred Lotz wrote:
> > I played with pyinstaller which worked fine. However, it builds a
> > dynamic executable and thus it is glibc version dependent. Means, I
> > have to build different executables for differen glibc versions.  
> 
> Just build it for the oldest version.

Wasn't aware of that.

>  Even if you don't, it may not
> matter. I just compiled a hello world program on Ubuntu 20 and ran it
> on Debian 6 (which is 10 years old). Obviusly a real program has more
> opportunities to run into compatibility problems, but glibc doesn't
> change all that much.
> 
>         hp
> 

I have build an executable with pyinstaller on Ubuntu 20.04 and it
didn't run on Redhat 8.4.

Doing it the other way round helps indeed, i.e. building on Redhat 8.4
and then it runs on Ubuntu 20.04. 

Thank you.


-- 
Manfred


From ml_news at posteo.de  Fri Dec 10 09:37:14 2021
From: ml_news at posteo.de (Manfred Lotz)
Date: Fri, 10 Dec 2021 15:37:14 +0100
Subject: How to package a Python command line app?
References: <20211208123136.52690b4f.ml_news@posteo.de>
 <4E247BF6-1D23-414E-BCA2-63C1112D0004@barrys-emacs.org>
 <mailman.64.1639071248.15287.python-list@python.org>
 <20211209193537.6e9920b1.ml_news@posteo.de>
 <ab47c868-62bf-6194-7825-9134fb6089b8@wichmann.us>
 <mailman.68.1639083070.15287.python-list@python.org>
Message-ID: <20211210153714.46de7174.ml_news@posteo.de>

On Thu, 9 Dec 2021 13:51:00 -0700
Mats Wichmann <mats at wichmann.us> wrote:

> On 12/9/21 11:35, Manfred Lotz wrote:
> 
> > I played with pyinstaller which worked fine. However, it builds a
> > dynamic executable and thus it is glibc version dependent. Means, I
> > have to build different executables for differen glibc versions.
> > 
> > So, it seems I will have to check how executable zip archives are
> > supposed to work.
> >   
> 
> For me at least, I'm still not sure what you are trying to accomplish.
> 

I like to offer my command line app to some people who are not really
command line geeks. Means, I don't want to have to tell them to install
packages via pip and stuff like that. Simply take a file, make it
executable and run it.


-- 
Manfred






From as at sci.fi  Fri Dec 10 12:38:34 2021
From: as at sci.fi (Anssi Saari)
Date: Fri, 10 Dec 2021 19:38:34 +0200
Subject: How to package a Python command line app?
In-Reply-To: <20211209193936.2b7cb2c2.ml_news@posteo.de> (Manfred Lotz's
 message of "Thu, 9 Dec 2021 19:39:36 +0100")
References: <20211208123136.52690b4f.ml_news@posteo.de>
 <25010.14946.854543.991758@ixdm.fritz.box>
 <mailman.63.1639070308.15287.python-list@python.org>
 <20211209193936.2b7cb2c2.ml_news@posteo.de>
Message-ID: <sm0wnkc1ht1.fsf@lakka.kapsi.fi>

Manfred Lotz <ml_news at posteo.de> writes:

> pyinstaller worked fine taking care of message.py and typer module. But
> as said in my other reply it is glibc version dependent.

Perhaps the included freeze.py script (included in the CPython source
that is, in Tools/freeze) is worth considering as well. Although it also
seems to create a dynamic executable by default (I tried with your hello
example), it seems to me it's possible to edit the generated makefile
and replace -shared with -static in a few places. Didn't try that
though.

From roel at roelschroeven.net  Fri Dec 10 20:36:41 2021
From: roel at roelschroeven.net (Roel Schroeven)
Date: Sat, 11 Dec 2021 02:36:41 +0100
Subject: Sad news: Fredrik Lundh ("Effbot") has passed away
Message-ID: <a30a299e-5a46-62d5-7bc6-ec8ebdd6de3e@roelschroeven.net>

Message from Guido van Rossum 
(https://mail.python.org/archives/list/python-dev at python.org/thread/36Q5QBILL3QIFIA3KHNGFBNJQKXKN7SD/):

> A former core dev who works at Google just passed the news that 
> Fredrik Lundh (also known as Effbot) has died.
>
> Fredrik was an early Python contributor (e.g. Elementtree and the 're' 
> module) and his enthusiasm for the language and community were 
> inspiring for all who encountered him or his work. He spent countless 
> hours on comp.lang.python answering questions from newbies and 
> advanced users alike.
>
> He also co-founded an early Python startup, Secret Labs AB, which 
> among other software released an IDE named PythonWorks. Fredrik also 
> created the Python Imaging Library (PIL) which is still THE way to 
> interact with images in Python, now most often through its Pillow 
> fork. His effbot.org site was a valuable resource for generations of 
> Python users, especially its Tkinter documentation.
>
> Fredrik's private Facebook page contains the following message from 
> November 25 by Ylva Larensson (translated from Swedish):
>
> """
>
> It is with such sorrow and pain that I write this. Fredrik has left us 
> suddenly.
>
> """
>
> A core dev wrote: "I felt privileged to be able to study Fredrik's 
> code and to read his writing. He was a huge asset to our community in 
> the early days. I enjoyed his sense of humor as well. I'm sad that he 
> passed away."
>
> We will miss him.
>

-- 
"Your scientists were so preoccupied with whether they could, they didn't
stop to think if they should"
         -- Dr. Ian Malcolm


From ninjagaming107gamer at gmail.com  Sat Dec 11 02:35:25 2021
From: ninjagaming107gamer at gmail.com (Sidharth S Nair)
Date: Sat, 11 Dec 2021 13:05:25 +0530
Subject: FW: pip Error
In-Reply-To: <10A22AEF-17F0-470A-BC62-16CF25896373@hxcore.ol>
References: <10A22AEF-17F0-470A-BC62-16CF25896373@hxcore.ol>
Message-ID: <213798E4-61DD-4B06-BCCB-F31533415B48@hxcore.ol>

   ?

   ?

   Sent from [1]Mail for Windows

   ?

   From: [2]Sidharth S Nair
   Sent: 11 December 2021 01:00 PM
   To: [3]python-list at python.org
   Subject: pip Error

   ?

   ?

   ?

   Sent from [4]Mail for Windows

   ?

   Hi, I am NINJAGAMING107 a active python user and lately I have been trying
   to make my own personal AI assistant but I am not able to make because I
   am not able to import Speech Recognition and some other more. The error
   says module not found even though in my folder where I installed python
   and the packages there is a folder of speech recognition and the rest of
   the things which I installed. I used Pycharm and VS code for this still
   not working. If there is any way to fix this please tell me or if I am
   doing the installation wrong please tell me. Hope that you will reply to
   ASAP.

   ?

   Thank you in advance.

   NINJAGAMING107

   ?

                 This email has been checked for viruses by Avast antivirus
   [5]Avast logo software.
                 [6]www.avast.com

   ?

References

   Visible links
   1. https://go.microsoft.com/fwlink/?LinkId=550986
   2. mailto:ninjagaming107gamer at gmail.com
   3. mailto:python-list at python.org
   4. https://go.microsoft.com/fwlink/?LinkId=550986
   5. https://www.avast.com/antivirus
   6. https://www.avast.com/antivirus

From souvikghosh872 at gmail.com  Sat Dec 11 07:09:12 2021
From: souvikghosh872 at gmail.com (Souvik Ghosh)
Date: Sat, 11 Dec 2021 17:39:12 +0530
Subject: Fwd: I/O bound threads got to no chance to run with small CPU bound
 threads with new GIL
In-Reply-To: <CACaaUvH7OrFSzj3DUcGOO+swtLDAG9ijCDTLOJduHxFs+ZBm7g@mail.gmail.com>
References: <CACaaUvH7OrFSzj3DUcGOO+swtLDAG9ijCDTLOJduHxFs+ZBm7g@mail.gmail.com>
Message-ID: <CACaaUvH0ZOUx8xo6Ms_-8ekhKTfiw-YKZzkZVUcM4Whcj=PX=g@mail.gmail.com>

*Resending this message after subscribing in python-mail-list*

---------- Forwarded message ---------
From: Souvik Ghosh <souvikghosh872 at gmail.com>
Date: Sat, Dec 11, 2021 at 5:10 PM
Subject: I/O bound threads got to no chance to run with small CPU bound
threads with new GIL
To: <python-list at python.org>


Hello PSF,

I'm Souvik Ghosh from India. I've been coding for Python for almost 5
years now. And, I love Python and care about it so much.

The issue is stated below,

According to David Beazley' talk in PyCon'2010 in Atlanta Georgia, he
demonstrated about a new GIL with running CPU bound and I/O bound
threads together.

He said the talk that the threads which are forced to timeout of 5ms,
will have the lower priority(which is CPU bound) and the thread which
suspends the GIL within 5ms will have higher priority (which is I/O
bound).

What happens in the following code is if I set args=(10000000,) (seven
zero after 1) then only I/O bound runs and returns when CPU bound
takes much time to execute. But if I decrease that args to
args=(1000,) then I/O bound got no chance to reaquire the GIL in the
meantime even though the sys.getswitchinterval() is equal to 5ms(By
default). If I/O bound doesn't reacquire GIL with args=(10000,) then
the time to execute to run
only the CPU bound takes 0.42777760000035414 seconds. Thats means
almost ticks 0.42777760000035414/0.005=85 (approx) times to set the
priority in between the two threads. In that case if the I/O got more
priority within that time, it should have returned the value within
that ticks. But I didn't happen.
import threading
from queue import Queue
from timeit import default_timer as timer
import urllib.request


q = Queue()  # Queue technique to pass returns among threads while running


def decrement(numbers):  # CPU bound
    while numbers > 0:
        numbers -= 1
        if not q.empty():
            """I added this method because this thread will run most of the time
            because it's mostly cpu bound"""
            print(numbers)
            print(q.get(block=False))
            print(timer() - start)  # It tell after when exactly I/O
bound returns value after both the threads started to run


def get_data():  # I/O bound

    with urllib.request.urlopen("https://www.google.com") as dt:
        q.put(dt.read(), block=False)


if __name__ == "__main__":
    start = timer()
    t1 = threading.Thread(target=get_data)
    #t2 = threading.Thread(target=decrement, args=(10000000,)) #For
this I/O responds and returns
    t2 = threading.Thread(target=decrement, args=(100000,)) # I/O
doesn't responds at all
    t1.start()
    t2.start()

    t1.join()
    t2.join()
    print(timer() - start)

Look at the second code...

import threading
from queue import Queue
from timeit import default_timer as timer
import urllib.request
import sys


q = Queue()  # Queue technique to pass returns among threads while running


def decrement(numbers):  # CPU bound
    while numbers > 0:
        numbers -= 1
        if not q.empty():
            """I added this method because this thread will run most of the time
            because it's mostly cpu bound"""
            print(numbers)
            print(q.get(block=False))
            print(timer() - start)  # It tell after when exactly I/O
bound returns value after both the threads started to run


def get_data():  # I/O bound

    with urllib.request.urlopen("https://www.google.com") as dt:
        q.put(dt.read(), block=False)


if __name__ == "__main__":
    sys.setswitchinterval(0.0000000000000000000000000001)
    start = timer()
    t1 = threading.Thread(target=get_data)
    #t2 = threading.Thread(target=decrement, args=(1000000,)) #I/O
responds with this
    t2 = threading.Thread(target=decrement, args=(10000,))    # I/O
doesn't responds at all even with this 0.0000000000000000000000000001
seconds of threads switching interval
    t1.start()
    t2.start()

    t1.join()
    t2.join()
    print(timer() - start)

Can't we have a more better version of GIL to set I/O threads(overall)
priorities even more better and not to degrade the CPU bound and
better callbacks in response? Or, try to remove the GIL?

The issue I submitted in here:- https://bugs.python.org/issue46046

Thank you so much, great future of Python!

From jkn_gg at nicorp.f9.co.uk  Sat Dec 11 04:02:45 2021
From: jkn_gg at nicorp.f9.co.uk (jkn)
Date: Sat, 11 Dec 2021 01:02:45 -0800 (PST)
Subject: Sad news: Fredrik Lundh ("Effbot") has passed away
In-Reply-To: <mailman.70.1639186608.15287.python-list@python.org>
References: <a30a299e-5a46-62d5-7bc6-ec8ebdd6de3e@roelschroeven.net>
 <mailman.70.1639186608.15287.python-list@python.org>
Message-ID: <82f45b1e-8d1e-4ed8-8140-7151e3d80cc9n@googlegroups.com>

On Saturday, December 11, 2021 at 1:37:07 AM UTC, Roel Schroeven wrote:
> Message from Guido van Rossum 
> (https://mail.python.org/archives/list/pytho... at python.org/thread/36Q5QBILL3QIFIA3KHNGFBNJQKXKN7SD/): 
> 
> > A former core dev who works at Google just passed the news that 
> > Fredrik Lundh (also known as Effbot) has died. 
> > 
> > Fredrik was an early Python contributor (e.g. Elementtree and the 're' 
> > module) and his enthusiasm for the language and community were 
> > inspiring for all who encountered him or his work. He spent countless 
> > hours on comp.lang.python answering questions from newbies and 
> > advanced users alike. 
> > 
> > He also co-founded an early Python startup, Secret Labs AB, which 
> > among other software released an IDE named PythonWorks. Fredrik also 
> > created the Python Imaging Library (PIL) which is still THE way to 
> > interact with images in Python, now most often through its Pillow 
> > fork. His effbot.org site was a valuable resource for generations of 
> > Python users, especially its Tkinter documentation. 
> > 
> > Fredrik's private Facebook page contains the following message from 
> > November 25 by Ylva Larensson (translated from Swedish): 
> > 
> > """ 
> > 
> > It is with such sorrow and pain that I write this. Fredrik has left us 
> > suddenly. 
> > 
> > """ 
> > 
> > A core dev wrote: "I felt privileged to be able to study Fredrik's 
> > code and to read his writing. He was a huge asset to our community in 
> > the early days. I enjoyed his sense of humor as well. I'm sad that he 
> > passed away." 
> > 
> > We will miss him. 
> > 
> 
> -- 
> "Your scientists were so preoccupied with whether they could, they didn't 
> stop to think if they should" 
> -- Dr. Ian Malcolm

Thanks for passing on this sad news. Effbot was one of the luminaries when
I started writing Python (v1.4/v1.52 times). I still have his 'standard library'
book, as well as being a user of PIL/Pillow.

Jon N

From mats at wichmann.us  Sat Dec 11 12:51:15 2021
From: mats at wichmann.us (Mats Wichmann)
Date: Sat, 11 Dec 2021 10:51:15 -0700
Subject: FW: pip Error
In-Reply-To: <213798E4-61DD-4B06-BCCB-F31533415B48@hxcore.ol>
References: <10A22AEF-17F0-470A-BC62-16CF25896373@hxcore.ol>
 <213798E4-61DD-4B06-BCCB-F31533415B48@hxcore.ol>
Message-ID: <ce4523f2-85b9-dd95-fbbe-9ea8728b05b1@wichmann.us>

On 12/11/21 00:35, Sidharth S Nair wrote:

>     Hi, I am NINJAGAMING107 a active python user and lately I have been trying
>     to make my own personal AI assistant but I am not able to make because I
>     am not able to import Speech Recognition and some other more. The error
>     says module not found even though in my folder where I installed python
>     and the packages there is a folder of speech recognition and the rest of
>     the things which I installed. I used Pycharm and VS code for this still
>     not working. If there is any way to fix this please tell me or if I am
>     doing the installation wrong please tell me. Hope that you will reply to
>     ASAP.

Rule #1 on module not found errors: it's always a path problem.

Unfortunately, on today's systems, there are often many places things 
can go as you may have several Pythons, several virtualenvs, etc.

The basic rule is to install with the Python you're intending to use, 
that way the paths it's going to look in are the same paths your 
packages got installed in.

In the case of VS Code, you can open a terminal (inside the IDE) and run 
your pip commands there, and that will be happening in the environment 
that is defined for your project.  In the case of PyCharm, you can click 
on the Python version in the lower right and click Interpreter Settings, 
through there you can search for and install packages from PyPi. Either 
one will give you the option to setup a fresh virtualenv for the 
project, which you may want to do since you're unsure of the current state.




From tomkcpr at mdevsys.com  Sat Dec 11 21:28:32 2021
From: tomkcpr at mdevsys.com (TomK)
Date: Sat, 11 Dec 2021 21:28:32 -0500
Subject: Set git config credential.helper cache and timeout via python
Message-ID: <abe4d626-fa28-9d80-f599-d285bb9ab5da@mdevsys.com>

Hey All,

I have a set of bash and python scripts that all interact with a remote 
git repository.? While running the scripts, when ever a repo is cloned 
using HTTP and a user/pass combination, that information is stored in 
the .git/config file on disk including the password.? I'm able to issue 
the following commands in git to effectively enable the caching of 
credentials in memory for a predefined amount of time off the linux shell:

git config credential.helper 'cache
git config credential.helper 'cache --timeout=300'

However, what is the best way to do so via Python?? I've read through 
the following to try and make sense of how I could do so:

https://gitpython.readthedocs.io/en/stable/reference.html?highlight=cache#git.index.fun.read_cache
https://pypi.org/project/git-credential-helpers/

But neither means appears to have or be able to do what I'm interested 
in.? I would like to prevent the user and password from being stored on 
disk in any shape or form, including eliminating it from .gitconfig / 
.git/config files.?? I keep trying to search in google for possible 
solutions using "python git credential helper" and "git prevent storing 
passwords in git config files"? but google keeps returning results on 
how to store passwords IN .git/config files, which is the exact opposite 
of what I want.? :)

An alternative to the above is to use ssh keys, however I would like to 
know an alternate way that avoids using ssh keys.?? Wondering if folks 
here did something similar using python and could provide a bit of 
insight how I could go about doing so?

-- 
Thx,
TK.

From jsf80238 at gmail.com  Sat Dec 11 22:28:06 2021
From: jsf80238 at gmail.com (Jason Friedman)
Date: Sat, 11 Dec 2021 20:28:06 -0700
Subject: Set git config credential.helper cache and timeout via python
In-Reply-To: <abe4d626-fa28-9d80-f599-d285bb9ab5da@mdevsys.com>
References: <abe4d626-fa28-9d80-f599-d285bb9ab5da@mdevsys.com>
Message-ID: <CANy1k1gBJq9RRrLtS-dLGujtr7gchJskj74uqO4THWKngoz0Cw@mail.gmail.com>

> Hey All,
>
> I have a set of bash and python scripts that all interact with a remote
> git repository.
>

This does not exactly answer your question, but whenever I have wanted to
interact with (popular) software via Python I have checked to see if
someone has already written that code for me.

https://pypi.org/search/?q=git

https://gitpython.readthedocs.io/en/stable/tutorial.html

From jsf80238 at gmail.com  Sat Dec 11 22:30:04 2021
From: jsf80238 at gmail.com (Jason Friedman)
Date: Sat, 11 Dec 2021 20:30:04 -0700
Subject: Set git config credential.helper cache and timeout via python
In-Reply-To: <abe4d626-fa28-9d80-f599-d285bb9ab5da@mdevsys.com>
References: <abe4d626-fa28-9d80-f599-d285bb9ab5da@mdevsys.com>
Message-ID: <CANy1k1hfLK5mS2nYfj2+z_h3komYG5dxPmtrY54BXKHPFC-3vA@mail.gmail.com>

>
>
> Hey All,
>
> I have a set of bash and python scripts that all interact with a remote
> git repository.
>
>
> https://gitpython.readthedocs.io/en/stable/reference.html?highlight=cache#git.index.fun.read_cache
> https://pypi.org/project/git-credential-helpers/
>
> But neither means appears to have or be able to do what I'm interested
> in.
>

I guess I should have read more carefully :(

Ignore my second link. Maybe the first (https://pypi.org/search/?q=git)
will be helpful?

From skip.montanaro at gmail.com  Sun Dec 12 09:00:32 2021
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Sun, 12 Dec 2021 08:00:32 -0600
Subject: Sad news: Fredrik Lundh ("Effbot") has passed away
In-Reply-To: <Lundh-20211211140549@ram.dialup.fu-berlin.de>
References: <a30a299e-5a46-62d5-7bc6-ec8ebdd6de3e@roelschroeven.net>
 <mailman.70.1639186608.15287.python-list@python.org>
 <Lundh-20211211140549@ram.dialup.fu-berlin.de>
Message-ID: <CANc-5UwTAj+BECS4-bapkJb0oQewvB=hfOa+_DPTrxwcY50a6g@mail.gmail.com>

Like many others, I'm saddened to hear of Fredrik Lundh's passing. I
vaguely recall meeting him just once, probably at a Python workshop,
before they grew big enough to be called conferences. Effbot.org was
my Tkinter, ElemenTree, and PIL reference and cheat sheet.

My attention to Python development has waxed and waned over the years.
Most of the time, the trip through the Python folder in my mail
program was generally pretty quick, hitting the 'd' key far more often
than I'd stop to read a message. There are only a few people whose
messages I'd always read. Effbot was one. In my opinion, Fredrik ranks
up there with Guido, Tim Peters and Barry Warsaw.

I went to effbot.org and saw the "on hiatus" message. Searching
through The Wayback Machine, it seems it went on hiatus in late
November, 2020. The 11 November 2020 snapshot appears to be the last
usable version:

https://web.archive.org/web/20201111145627/http://effbot.org/

Probably worth a bookmark in your browser.

Rest easy /F ...

Skip

From miked at dewhirst.com.au  Sun Dec 12 20:22:28 2021
From: miked at dewhirst.com.au (Mike Dewhirst)
Date: Mon, 13 Dec 2021 12:22:28 +1100
Subject: Isn't TypeError built in?
Message-ID: <b4597886-e7f6-22f3-5605-67ed34e60c5f@dewhirst.com.au>

Obviously something is wrong elsewhere but I'm not sure where to look. 
Ubuntu 20.04 with plenty of RAM.

 ??? def __del__(self):
 ??????? try:
 ??????????? for context_obj in self._context_refs:
 ??????????????? try:
 ??????????????????? delattr(context_obj, self._attr_name)
 ??????????????? except AttributeError:
 ??????????????????? pass

 ??????? except TypeError:??? ??? # THIS IS LINE 96 IN THE APACHE2 ERROR 
LOG BELOW

 ??????????? # WeakSet.__iter__ can crash when interpreter is shutting 
down due
 ??????????? # to _IterationGuard being None.
 ??????????? pass

[Mon Dec 13 01:15:49.875993 2021] [mpm_event:notice] [pid 1033:tid 
140446449658944] AH00493: SIGUSR1 received.? Doing graceful restart
[Mon Dec 13 01:15:49.878443 2021] [so:warn] [pid 1033] AH01574: module 
dav_module is already loaded, skipping
[Mon Dec 13 01:15:49.885659 2021] [mpm_event:notice] [pid 1033:tid 
140446449658944] AH00489: Apache/2.4.41 (Ubuntu) SVN/1.13.0 
OpenSSL/1.1.1f mod_wsgi/4.6.8 Python/3.8 configured -- resuming normal 
operations
[Mon Dec 13 01:15:49.885664 2021] [core:notice] [pid 1033:tid 
140446449658944] AH00094: Command line: '/usr/sbin/apache2'
Exception ignored in: <function Local.__del__ at 0x7fbc380964c0>
Traceback (most recent call last):
 ? File "/usr/local/lib/python3.8/dist-packages/asgiref/local.py", line 
96, in __del__
NameError: name 'TypeError' is not defined
Exception ignored in: <function Local.__del__ at 0x7fbc380964c0>
Traceback (most recent call last):
 ? File "/usr/local/lib/python3.8/dist-packages/asgiref/local.py", line 
96, in __del__
NameError: name 'TypeError' is not defined

Any hints welcome ...

Thanks

Mike

-- 
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can
automatically decrypt my signature and be sure it came from me. Just
ask and I'll send it to you. Your email software can handle signing.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <https://mail.python.org/pipermail/python-list/attachments/20211213/11bd02d8/attachment.sig>

From pbryan at anode.ca  Sun Dec 12 20:39:25 2021
From: pbryan at anode.ca (Paul Bryan)
Date: Sun, 12 Dec 2021 17:39:25 -0800
Subject: Isn't TypeError built in?
In-Reply-To: <b4597886-e7f6-22f3-5605-67ed34e60c5f@dewhirst.com.au>
References: <b4597886-e7f6-22f3-5605-67ed34e60c5f@dewhirst.com.au>
Message-ID: <7c99b07046a3f5eb33a80a04a25a53634abb169b.camel@anode.ca>

Yes, TypeError is built in. The only thing I can think of is that
something has deleted `TypeError` from `__builtins__`? It would be
interesting to see what's in `__builtins__` when `__del__` is called.

On Mon, 2021-12-13 at 12:22 +1100, Mike Dewhirst via Python-list wrote:
> Obviously something is wrong elsewhere but I'm not sure where to
> look. 
> Ubuntu 20.04 with plenty of RAM.
> 
> ???? def __del__(self):
> ???????? try:
> ???????????? for context_obj in self._context_refs:
> ???????????????? try:
> ???????????????????? delattr(context_obj, self._attr_name)
> ???????????????? except AttributeError:
> ???????????????????? pass
> 
> ???????? except TypeError:??? ??? # THIS IS LINE 96 IN THE APACHE2
> ERROR 
> LOG BELOW
> 
> ???????????? # WeakSet.__iter__ can crash when interpreter is
> shutting 
> down due
> ???????????? # to _IterationGuard being None.
> ???????????? pass
> 
> [Mon Dec 13 01:15:49.875993 2021] [mpm_event:notice] [pid 1033:tid 
> 140446449658944] AH00493: SIGUSR1 received.? Doing graceful restart
> [Mon Dec 13 01:15:49.878443 2021] [so:warn] [pid 1033] AH01574:
> module 
> dav_module is already loaded, skipping
> [Mon Dec 13 01:15:49.885659 2021] [mpm_event:notice] [pid 1033:tid 
> 140446449658944] AH00489: Apache/2.4.41 (Ubuntu) SVN/1.13.0 
> OpenSSL/1.1.1f mod_wsgi/4.6.8 Python/3.8 configured -- resuming
> normal 
> operations
> [Mon Dec 13 01:15:49.885664 2021] [core:notice] [pid 1033:tid 
> 140446449658944] AH00094: Command line: '/usr/sbin/apache2'
> Exception ignored in: <function Local.__del__ at 0x7fbc380964c0>
> Traceback (most recent call last):
> ?? File "/usr/local/lib/python3.8/dist-packages/asgiref/local.py",
> line 
> 96, in __del__
> NameError: name 'TypeError' is not defined
> Exception ignored in: <function Local.__del__ at 0x7fbc380964c0>
> Traceback (most recent call last):
> ?? File "/usr/local/lib/python3.8/dist-packages/asgiref/local.py",
> line 
> 96, in __del__
> NameError: name 'TypeError' is not defined
> 
> Any hints welcome ...
> 
> Thanks
> 
> Mike
> 
> -- 
> Signed email is an absolute defence against phishing. This email has
> been signed with my private key. If you import my public key you can
> automatically decrypt my signature and be sure it came from me. Just
> ask and I'll send it to you. Your email software can handle signing.
> 


From rosuav at gmail.com  Sun Dec 12 20:42:24 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 13 Dec 2021 12:42:24 +1100
Subject: Isn't TypeError built in?
In-Reply-To: <b4597886-e7f6-22f3-5605-67ed34e60c5f@dewhirst.com.au>
References: <b4597886-e7f6-22f3-5605-67ed34e60c5f@dewhirst.com.au>
Message-ID: <CAPTjJmrFLamyATZFpACpekBtdme9zp85LDm15kD5c-ydZ5WHuA@mail.gmail.com>

On Mon, Dec 13, 2021 at 12:31 PM Mike Dewhirst via Python-list
<python-list at python.org> wrote:
>
> Obviously something is wrong elsewhere but I'm not sure where to look.
> Ubuntu 20.04 with plenty of RAM.
>
>      def __del__(self):
>          try:
>              for context_obj in self._context_refs:
>                  try:
>                      delattr(context_obj, self._attr_name)
>                  except AttributeError:
>                      pass
>
>          except TypeError:        # THIS IS LINE 96 IN THE APACHE2 ERROR
> LOG BELOW
>
>              # WeakSet.__iter__ can crash when interpreter is shutting
> down due
>              # to _IterationGuard being None.
>              pass
>
> [Mon Dec 13 01:15:49.875993 2021] [mpm_event:notice] [pid 1033:tid
> 140446449658944] AH00493: SIGUSR1 received.  Doing graceful restart
> [Mon Dec 13 01:15:49.878443 2021] [so:warn] [pid 1033] AH01574: module
> dav_module is already loaded, skipping
> [Mon Dec 13 01:15:49.885659 2021] [mpm_event:notice] [pid 1033:tid
> 140446449658944] AH00489: Apache/2.4.41 (Ubuntu) SVN/1.13.0
> OpenSSL/1.1.1f mod_wsgi/4.6.8 Python/3.8 configured -- resuming normal
> operations
> [Mon Dec 13 01:15:49.885664 2021] [core:notice] [pid 1033:tid
> 140446449658944] AH00094: Command line: '/usr/sbin/apache2'
> Exception ignored in: <function Local.__del__ at 0x7fbc380964c0>
> Traceback (most recent call last):
>    File "/usr/local/lib/python3.8/dist-packages/asgiref/local.py", line
> 96, in __del__
> NameError: name 'TypeError' is not defined
> Exception ignored in: <function Local.__del__ at 0x7fbc380964c0>
> Traceback (most recent call last):
>    File "/usr/local/lib/python3.8/dist-packages/asgiref/local.py", line
> 96, in __del__
> NameError: name 'TypeError' is not defined
>
> Any hints welcome ...

During interpreter shutdown, all kinds of things can go weird. The
order that things get destroyed isn't always clearly defined.

Is it possible to do the cleanup work before interpreter shutdown?
Alternatively: maybe take a local reference to the exception types you
need to refer to (just "AttributeError = AttributeError" at top level
should do it), which should keep them alive longer (I think). Worst
case, capture *all* exceptions, then look at e.__class__.__name__ and
match on that.

ChrisA

From qccpantalunan at tip.edu.ph  Sat Dec 11 23:44:54 2021
From: qccpantalunan at tip.edu.ph (CINDY PANTALUNAN)
Date: Sun, 12 Dec 2021 12:44:54 +0800
Subject: ERROR: No matching distribution found for PyQt5==5.15.4
Message-ID: <CAKmHOAjZMmPmndP70+RHWAVi083-0d3Ghw=impGNMV160zYz7A@mail.gmail.com>



-- 
*The contents of this email message and any attachments?**thereto**?are 
intended solely for the addressee(s) and may contain confidential and/or 
privileged?information and may be legally protected from disclosure. If you 
are not the intended recipient of this message or their agent, or if this 
message has been addressed to you in error, please immediately?**notify**?
the sender by reply email and?delete this message and?**its?**attachments.?
Any unauthorized use, dissemination, copying, or storage of this message or 
its attachments?**is subject to criminal and civil liability**?under the 
Data Privacy Act of 2012 (RA 10173)**?and the Intellectual Property Code of 
the Philippines (RA 8293), as may be applicable**.*

From lcwarner11 at gmail.com  Sun Dec 12 18:37:15 2021
From: lcwarner11 at gmail.com (Larry Warner)
Date: Sun, 12 Dec 2021 18:37:15 -0500
Subject: problem reading a CSV file
Message-ID: <CAG2cnigEeMkYXPNWFVvhzyk6QH6E7nUGciQO+R1suBb6Ypp4eQ@mail.gmail.com>

Win 10, Chrome, Python 3.10.1
New at python
error on open statement

Probably simple error but I do not see it.

The program is a python example with the file name being changed.  I want
to experiment with changing the literal file name in the open statement to
a variable name later.

Larry

From cloro at rogertechnology.it  Mon Dec 13 02:25:44 2021
From: cloro at rogertechnology.it (Cristiano Loro)
Date: Mon, 13 Dec 2021 07:25:44 +0000
Subject: Installation problems
Message-ID: <AM0P193MB04654887336CB0DBE3E885D9DF749@AM0P193MB0465.EURP193.PROD.OUTLOOK.COM>

Good morning,
i am using a python application to program ESP8266. in my office everything works fine, but in production, where users don't have the same access privileges, I can't get it to go. In production I performed a custom installation defining "for all users", but the modules are placed elsewhere and this is perhaps the problem. How can I put the modules in the correct folder? in general how can I get around the obstacle
many thanks in advance for your help


[Firma-Mail-Roger_Cristiano-Loro]

<Le informazioni contenute in questa comunicazione sono riservate e destinate esclusivamente alla/e persona/e o all'ente/i sopra indicati.
? vietato ai soggetti diversi dai destinatari qualsiasi uso, copia, diffusione di quanto in essa contenuto sia ai sensi dell'art. 616 c..p., sia ai sensi del Regolamento UE 679/2016.
Se questa comunicazione vi ? pervenuta per errore, vi preghiamo di rispondere a questa e-mail comunicando l'errore e successivamente cancellarla, compresi gli eventuali allegati, dal vostro sistema.
La informiamo che il trattamento dei suoi dati personali ? effettuato dalla presente struttura con l'adozione delle misure previste dalla normativa vigente a tutela della protezione dei dati personali (Regolamento UE 679/2016).>


From jkn_gg at nicorp.f9.co.uk  Sun Dec 12 15:20:16 2021
From: jkn_gg at nicorp.f9.co.uk (jkn)
Date: Sun, 12 Dec 2021 12:20:16 -0800 (PST)
Subject: Sad news: Fredrik Lundh ("Effbot") has passed away
In-Reply-To: <mailman.77.1639317660.15287.python-list@python.org>
References: <a30a299e-5a46-62d5-7bc6-ec8ebdd6de3e@roelschroeven.net>
 <mailman.70.1639186608.15287.python-list@python.org>
 <CANc-5UwTAj+BECS4-bapkJb0oQewvB=hfOa+_DPTrxwcY50a6g@mail.gmail.com>
 <Lundh-20211211140549@ram.dialup.fu-berlin.de>
 <mailman.77.1639317660.15287.python-list@python.org>
Message-ID: <9b005526-3a04-453f-82bb-1ee3a549c59an@googlegroups.com>

On Sunday, December 12, 2021 at 2:01:19 PM UTC, Skip Montanaro wrote:
> Like many others, I'm saddened to hear of Fredrik Lundh's passing. I 
> vaguely recall meeting him just once, probably at a Python workshop, 
> before they grew big enough to be called conferences. Effbot.org was 
> my Tkinter, ElemenTree, and PIL reference and cheat sheet. 
> 
> My attention to Python development has waxed and waned over the years. 
> Most of the time, the trip through the Python folder in my mail 
> program was generally pretty quick, hitting the 'd' key far more often 
> than I'd stop to read a message. There are only a few people whose 
> messages I'd always read. Effbot was one. In my opinion, Fredrik ranks 
> up there with Guido, Tim Peters and Barry Warsaw. 
> 
> I went to effbot.org and saw the "on hiatus" message. Searching 
> through The Wayback Machine, it seems it went on hiatus in late 
> November, 2020. The 11 November 2020 snapshot appears to be the last 
> usable version: 
> 
> https://web.archive.org/web/20201111145627/http://effbot.org/ 
> 
> Probably worth a bookmark in your browser. 
> 
> Rest easy /F ... 
> 
> Skip

I hadn't realised effbot.org was 'on hiatus'. Yes, definitely worth
bookmarking The Wayback Machine backup - or even (someone?)
making a separate archive?

From rosuav at gmail.com  Mon Dec 13 09:22:39 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 14 Dec 2021 01:22:39 +1100
Subject: problem reading a CSV file
In-Reply-To: <CAG2cnigEeMkYXPNWFVvhzyk6QH6E7nUGciQO+R1suBb6Ypp4eQ@mail.gmail.com>
References: <CAG2cnigEeMkYXPNWFVvhzyk6QH6E7nUGciQO+R1suBb6Ypp4eQ@mail.gmail.com>
Message-ID: <CAPTjJmrZe57qtVA2baGvm9X839fqLe9V1oKk9rOG66UNSP3pwA@mail.gmail.com>

On Tue, Dec 14, 2021 at 12:05 AM Larry Warner <lcwarner11 at gmail.com> wrote:
>
> Win 10, Chrome, Python 3.10.1
> New at python
> error on open statement
>
> Probably simple error but I do not see it.
>
> The program is a python example with the file name being changed.  I want
> to experiment with changing the literal file name in the open statement to
> a variable name later.
>

Use forward slashes "/" instead of backslashes "\".

ChrisA

From python at mrabarnett.plus.com  Mon Dec 13 13:07:16 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Mon, 13 Dec 2021 18:07:16 +0000
Subject: problem reading a CSV file
In-Reply-To: <CAG2cnigEeMkYXPNWFVvhzyk6QH6E7nUGciQO+R1suBb6Ypp4eQ@mail.gmail.com>
References: <CAG2cnigEeMkYXPNWFVvhzyk6QH6E7nUGciQO+R1suBb6Ypp4eQ@mail.gmail.com>
Message-ID: <8ec4940a-caeb-e285-b5a1-2090f472f16e@mrabarnett.plus.com>

On 2021-12-12 23:37, Larry Warner wrote:
> Win 10, Chrome, Python 3.10.1
> New at python
> error on open statement
> 
> Probably simple error but I do not see it.
> 
> The program is a python example with the file name being changed.  I want
> to experiment with changing the literal file name in the open statement to
> a variable name later.
> 
It's difficult to say what the problem is when you haven't given us the 
code!

From greg.ewing at canterbury.ac.nz  Mon Dec 13 17:43:07 2021
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 14 Dec 2021 11:43:07 +1300
Subject: Installation problems
In-Reply-To: <mailman.83.1639400668.15287.python-list@python.org>
References: <AM0P193MB04654887336CB0DBE3E885D9DF749@AM0P193MB0465.EURP193.PROD.OUTLOOK.COM>
 <mailman.83.1639400668.15287.python-list@python.org>
Message-ID: <j1q0jtFsr64U1@mid.individual.net>

On 13/12/21 8:25 pm, Cristiano Loro wrote:
> In production I performed a custom installation defining "for all users", but the modules are placed elsewhere and this is perhaps the problem. How can I put the modules in the correct folder? 

How are you installing the modules?

Are you doing it with sufficient privileges to put them in the
shared area?

Are you sure that your users are running the same Python that
you installed?

-- 
Greg


From greg.ewing at canterbury.ac.nz  Mon Dec 13 17:56:42 2021
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 14 Dec 2021 11:56:42 +1300
Subject: problem reading a CSV file
In-Reply-To: <mailman.85.1639419028.15287.python-list@python.org>
References: <CAG2cnigEeMkYXPNWFVvhzyk6QH6E7nUGciQO+R1suBb6Ypp4eQ@mail.gmail.com>
 <8ec4940a-caeb-e285-b5a1-2090f472f16e@mrabarnett.plus.com>
 <mailman.85.1639419028.15287.python-list@python.org>
Message-ID: <j1q1daFsuqkU1@mid.individual.net>

On 14/12/21 7:07 am, MRAB wrote:
> It's difficult to say what the problem is when you haven't given us the 
> code!

Note: Attachments do not make it to this list.
You will need to insert the code into the message as text.

-- 
Greg

From miked at dewhirst.com.au  Mon Dec 13 19:37:22 2021
From: miked at dewhirst.com.au (Mike Dewhirst)
Date: Tue, 14 Dec 2021 11:37:22 +1100
Subject: Isn't TypeError built in?
In-Reply-To: <CAPTjJmrFLamyATZFpACpekBtdme9zp85LDm15kD5c-ydZ5WHuA@mail.gmail.com>
References: <b4597886-e7f6-22f3-5605-67ed34e60c5f@dewhirst.com.au>
 <CAPTjJmrFLamyATZFpACpekBtdme9zp85LDm15kD5c-ydZ5WHuA@mail.gmail.com>
Message-ID: <3c3f4a1f-7950-d89b-1788-dcc719ede514@dewhirst.com.au>

On 13/12/2021 12:42 pm, Chris Angelico wrote:
> On Mon, Dec 13, 2021 at 12:31 PM Mike Dewhirst via Python-list
> <python-list at python.org>  wrote:
>> Obviously something is wrong elsewhere but I'm not sure where to look.
>> Ubuntu 20.04 with plenty of RAM.
>>
>>       def __del__(self):
>>           try:
>>               for context_obj in self._context_refs:
>>                   try:
>>                       delattr(context_obj, self._attr_name)
>>                   except AttributeError:
>>                       pass
>>
>>           except TypeError:        # THIS IS LINE 96 IN THE APACHE2 ERROR
>> LOG BELOW
>>
>>               # WeakSet.__iter__ can crash when interpreter is shutting
>> down due
>>               # to _IterationGuard being None.
>>               pass
>>
>> [Mon Dec 13 01:15:49.875993 2021] [mpm_event:notice] [pid 1033:tid
>> 140446449658944] AH00493: SIGUSR1 received.  Doing graceful restart
>> [Mon Dec 13 01:15:49.878443 2021] [so:warn] [pid 1033] AH01574: module
>> dav_module is already loaded, skipping
>> [Mon Dec 13 01:15:49.885659 2021] [mpm_event:notice] [pid 1033:tid
>> 140446449658944] AH00489: Apache/2.4.41 (Ubuntu) SVN/1.13.0
>> OpenSSL/1.1.1f mod_wsgi/4.6.8 Python/3.8 configured -- resuming normal
>> operations
>> [Mon Dec 13 01:15:49.885664 2021] [core:notice] [pid 1033:tid
>> 140446449658944] AH00094: Command line: '/usr/sbin/apache2'
>> Exception ignored in: <function Local.__del__ at 0x7fbc380964c0>
>> Traceback (most recent call last):
>>     File "/usr/local/lib/python3.8/dist-packages/asgiref/local.py", line
>> 96, in __del__
>> NameError: name 'TypeError' is not defined
>> Exception ignored in: <function Local.__del__ at 0x7fbc380964c0>
>> Traceback (most recent call last):
>>     File "/usr/local/lib/python3.8/dist-packages/asgiref/local.py", line
>> 96, in __del__
>> NameError: name 'TypeError' is not defined
>>
>> Any hints welcome ...
> During interpreter shutdown, all kinds of things can go weird. The
> order that things get destroyed isn't always clearly defined.
>
> Is it possible to do the cleanup work before interpreter shutdown?
> Alternatively: maybe take a local reference to the exception types you
> need to refer to (just "AttributeError = AttributeError" at top level
> should do it), which should keep them alive longer (I think). Worst
> case, capture *all* exceptions, then look at e.__class__.__name__ and
> match on that.
>
> ChrisA

OK - not solved but resolved anyway.

The Django project was failing on the new server and working fine in dev.

The problem seems to be a mis-export from an old subversion repo.The 
project was being reconstituted on a new staging server from a reloaded 
svn dump from the old (ancient) svn server to the new svn server on the 
same machine.

Resolution "happened" when I updated the (c) notice in all source files 
and recommitted to the new server and re-exported from that. The repo 
looks fine. The log is all there back to the beginning. As a test I have 
deployed from an earlier revision and that all works. It's either a 
miracle or I did something else (which I didn't notice) which fixed it.

Thanks to everyone for thinking about this.

Cheers

Mike

-- 
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can
automatically decrypt my signature and be sure it came from me. Just
ask and I'll send it to you. Your email software can handle signing.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <https://mail.python.org/pipermail/python-list/attachments/20211214/36a7a11d/attachment.sig>

From shishaozhong at gmail.com  Tue Dec 14 14:44:11 2021
From: shishaozhong at gmail.com (Shaozhong SHI)
Date: Tue, 14 Dec 2021 19:44:11 +0000
Subject: ogr2ogr can not open gfs file when loading GML
Message-ID: <CA+i5JwY-TrderkMAibiyKDqMFtHSPLaXCJMYUvDNF63X0Y7Svg@mail.gmail.com>

My command line kept telling me that it ogr2ogr can not open gfs file.  It
does find it.

I was trying to load GML onto PostGIS.

Alternatively, how to specify XSD file to go along with reading GML files?

Regards,

David

From electron at emypeople.net  Wed Dec 15 22:38:48 2021
From: electron at emypeople.net (Jason)
Date: Wed, 15 Dec 2021 21:38:48 -0600
Subject: Getting the exit code of a subprocess
Message-ID: <20211216033848.5dwulawke3s6434p@net1.local.lan>

Hello,

How can I find out the exit code of a process when using the subprocess module? I am passing an email message to a shell script and I need to know whether the shell script threw an error.

Here is my code:
p = Popen(cmd, stdout=None, stdin=PIPE, stderr=None)
p.communicate(input=msg)

I tried something like this:
e_stat = p.communicate(input=msg)

but the value of e_stat is always '(None, None)'


-- 
Thanks & best regards,
Jason Rissler

From samuelmarks at gmail.com  Wed Dec 15 22:37:29 2021
From: samuelmarks at gmail.com (samue...@gmail.com)
Date: Wed, 15 Dec 2021 19:37:29 -0800 (PST)
Subject: ast.parse, ast.dump, but with comment preservation?
Message-ID: <f36486c8-2380-4eaa-becf-504a0be8c882n@googlegroups.com>

I wrote a little open-source tool to expose internal constructs in OpenAPI. Along the way, I added related functionality to:
- Generate/update a function prototype to/from a class
- JSON schema
- Automatically add type annotations to all function arguments, class attributes, declarations, and assignments

alongside a bunch of other features. All implemented using just the builtin modules (plus astor on Python < 3.9; and optionally black).

Now I'm almost at the point where I can run it?without issue?against, e.g., the entire TensorFlow codebase. Unfortunately this is causing huge `diff`s because the comments aren't preserved (and there are some whitespace issues? but I should be able to resolve the latter).

Is the only viable solution available to rewrite around redbaron | libcst? - I don't need to parse the comments just dump them out unedited whence they're found?

Thanks for any suggestions

PS: Library is https://github.com/SamuelMarks/cdd-python (might relicense with CC0? anyway too early for others to use; wait for the 0.1.0 release ;])

From kushal at locationd.net  Wed Dec 15 23:19:16 2021
From: kushal at locationd.net (Kushal Kumaran)
Date: Wed, 15 Dec 2021 20:19:16 -0800
Subject: Getting the exit code of a subprocess
In-Reply-To: <20211216033848.5dwulawke3s6434p@net1.local.lan> (Jason's message
 of "Wed, 15 Dec 2021 21:38:48 -0600")
References: <20211216033848.5dwulawke3s6434p@net1.local.lan>
Message-ID: <87lf0lb2rf.fsf@locationd.net>

On Wed, Dec 15 2021 at 09:38:48 PM, Jason <electron at emypeople.net> wrote:
> Hello,
>
> How can I find out the exit code of a process when using the
> subprocess module? I am passing an email message to a shell script and
> I need to know whether the shell script threw an error.
>
> Here is my code:
> p = Popen(cmd, stdout=None, stdin=PIPE, stderr=None)
> p.communicate(input=msg)
>
> I tried something like this:
> e_stat = p.communicate(input=msg)
>
> but the value of e_stat is always '(None, None)'
>

You want to look at p.returncode after communicate returns:
https://docs.python.org/3/library/subprocess.html#subprocess.Popen.returncode

https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate
returns a tuple with the stdout and stderr contents.

-- 
regards,
kushal

From electron at emypeople.net  Wed Dec 15 23:58:23 2021
From: electron at emypeople.net (Jason)
Date: Wed, 15 Dec 2021 22:58:23 -0600
Subject: Getting the exit code of a subprocess
In-Reply-To: <87lf0lb2rf.fsf@locationd.net>
References: <20211216033848.5dwulawke3s6434p@net1.local.lan>
 <87lf0lb2rf.fsf@locationd.net>
Message-ID: <20211216045823.qjpy65637lw3pigm@net1.local.lan>

On Wed, Dec 15, 2021 at 08:19:16PM -0800, Kushal Kumaran wrote:
> On Wed, Dec 15 2021 at 09:38:48 PM, Jason <electron at emypeople.net> wrote:
> > Hello,
> >
> > How can I find out the exit code of a process when using the
> > subprocess module? I am passing an email message to a shell script and
> > I need to know whether the shell script threw an error.
> >
> > Here is my code:
> > p = Popen(cmd, stdout=None, stdin=PIPE, stderr=None)
> > p.communicate(input=msg)
> >
> > I tried something like this:
> > e_stat = p.communicate(input=msg)
> >
> > but the value of e_stat is always '(None, None)'
> >
> 
> You want to look at p.returncode after communicate returns:
> https://docs.python.org/3/library/subprocess.html#subprocess.Popen.returncode
> 
> https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate
> returns a tuple with the stdout and stderr contents.

Thank you, that is the info I needed!

-- 
Jason Rissler


From rosuav at gmail.com  Thu Dec 16 01:44:19 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 16 Dec 2021 17:44:19 +1100
Subject: ast.parse, ast.dump, but with comment preservation?
In-Reply-To: <f36486c8-2380-4eaa-becf-504a0be8c882n@googlegroups.com>
References: <f36486c8-2380-4eaa-becf-504a0be8c882n@googlegroups.com>
Message-ID: <CAPTjJmoCymrEzX+13q87MhHZ9WTe8Ps1LkM3wU3OBkVxTj4now@mail.gmail.com>

On Thu, Dec 16, 2021 at 2:47 PM samue... at gmail.com
<samuelmarks at gmail.com> wrote:
>
> I wrote a little open-source tool to expose internal constructs in OpenAPI. Along the way, I added related functionality to:
> - Generate/update a function prototype to/from a class
> - JSON schema
> - Automatically add type annotations to all function arguments, class attributes, declarations, and assignments
>
> alongside a bunch of other features. All implemented using just the builtin modules (plus astor on Python < 3.9; and optionally black).
>
> Now I'm almost at the point where I can run it?without issue?against, e.g., the entire TensorFlow codebase. Unfortunately this is causing huge `diff`s because the comments aren't preserved (and there are some whitespace issues? but I should be able to resolve the latter).
>
> Is the only viable solution available to rewrite around redbaron | libcst? - I don't need to parse the comments just dump them out unedited whence they're found?
>
> Thanks for any suggestions
>
> PS: Library is https://github.com/SamuelMarks/cdd-python (might relicense with CC0? anyway too early for others to use; wait for the 0.1.0 release ;])

I haven't actually used it, but what you may want to try is lib2to3.
It's capable of full text reconstruction like you're trying to do.

Otherwise: Every AST node contains line and column information, so you
could possibly work the other way: keep the source code as well as the
AST, and make changes line by line as you have need.

ChrisA

From barry at barrys-emacs.org  Thu Dec 16 03:30:31 2021
From: barry at barrys-emacs.org (Barry)
Date: Thu, 16 Dec 2021 08:30:31 +0000
Subject: ast.parse, ast.dump, but with comment preservation?
In-Reply-To: <f36486c8-2380-4eaa-becf-504a0be8c882n@googlegroups.com>
References: <f36486c8-2380-4eaa-becf-504a0be8c882n@googlegroups.com>
Message-ID: <469ECCAD-6782-434D-B26E-37182E047369@barrys-emacs.org>



> On 16 Dec 2021, at 03:49, samue... at gmail.com <samuelmarks at gmail.com> wrote:
> 
> ?I wrote a little open-source tool to expose internal constructs in OpenAPI. Along the way, I added related functionality to:
> - Generate/update a function prototype to/from a class
> - JSON schema
> - Automatically add type annotations to all function arguments, class attributes, declarations, and assignments
> 
> alongside a bunch of other features. All implemented using just the builtin modules (plus astor on Python < 3.9; and optionally black).
> 
> Now I'm almost at the point where I can run it?without issue?against, e.g., the entire TensorFlow codebase. Unfortunately this is causing huge `diff`s because the comments aren't preserved (and there are some whitespace issues? but I should be able to resolve the latter).
> 
> Is the only viable solution available to rewrite around redbaron | libcst? - I don't need to parse the comments just dump them out unedited whence they're found?
> 
> Thanks for any suggestions

Have a look at the code that is used by https://github.com/asottile/pyupgrade
There are a couple of libraries that it uses that does what I think you want to do.

Barry

> 
> PS: Library is https://github.com/SamuelMarks/cdd-python (might relicense with CC0? anyway too early for others to use; wait for the 0.1.0 release ;])
> -- 
> https://mail.python.org/mailman/listinfo/python-list

From lucas at bourneuf.net  Thu Dec 16 05:53:22 2021
From: lucas at bourneuf.net (lucas)
Date: Thu, 16 Dec 2021 11:53:22 +0100
Subject: ast.parse, ast.dump, but with comment preservation?
In-Reply-To: <f36486c8-2380-4eaa-becf-504a0be8c882n@googlegroups.com>
References: <f36486c8-2380-4eaa-becf-504a0be8c882n@googlegroups.com>
Message-ID: <07e8e6c6-1dcb-f25e-1a7c-30b10db3d818@bourneuf.net>

Hi !

Maybe RedBaron may help you ?

https://github.com/PyCQA/redbaron

IIRC, it aims to conserve the exact same representation of the source 
code, including comments and empty lines.

--lucas


On 16/12/2021 04:37, samue... at gmail.com wrote:
> I wrote a little open-source tool to expose internal constructs in OpenAPI. Along the way, I added related functionality to:
> - Generate/update a function prototype to/from a class
> - JSON schema
> - Automatically add type annotations to all function arguments, class attributes, declarations, and assignments
> 
> alongside a bunch of other features. All implemented using just the builtin modules (plus astor on Python < 3.9; and optionally black).
> 
> Now I'm almost at the point where I can run it?without issue?against, e.g., the entire TensorFlow codebase. Unfortunately this is causing huge `diff`s because the comments aren't preserved (and there are some whitespace issues? but I should be able to resolve the latter).
> 
> Is the only viable solution available to rewrite around redbaron | libcst? - I don't need to parse the comments just dump them out unedited whence they're found?
> 
> Thanks for any suggestions
> 
> PS: Library is https://github.com/SamuelMarks/cdd-python (might relicense with CC0? anyway too early for others to use; wait for the 0.1.0 release ;])

From hanan.lamaazi at gmail.com  Thu Dec 16 09:00:57 2021
From: hanan.lamaazi at gmail.com (hanan lamaazi)
Date: Thu, 16 Dec 2021 18:00:57 +0400
Subject: Update a specific element in all a list of N lists
Message-ID: <CADFspfotj+9h42yAb4+Yu5Gxw_jYJ7KNSff7UP0UGpabGs-P_Q@mail.gmail.com>

Dear All,

I really need your assistance,

I have a dataset with 1005000 rows and 25 columns,

The main column that I repeatedly use are Time, ID, and Reputation

First I sliced the data based on the time, and I append the sliced data in
a list called "df_list". So I get 201 lists with 25 columns

The main code is starting for here:

for elem in df_list:

{do something.....}

{Here I'm trying to calculate the outliers}

Out.append(outliers)

Now my problem is that I need to locate those outliers in the df_list and
then update another column with is the "Reputation"

Note that the there is a duplicated IDs but at different time slot

example is ID = 1 is outliers, I need to select all ID = 1 in the list and
update their reputation column

I tried those solutions:
1)

grp = data11.groupby(['ID'])
        for i in GlobalNotOutliers.ID:
            data11.loc[grp.get_group(i).index, 'Reput'] += 1

        for j in GlobalOutliers.ID:
            data11.loc[grp.get_group(j).index, 'Reput'] -= 1


It works for a dataframe but not for a list

2)

for elem in df_list:

elem.loc[elem['ID'].isin(Outlier['ID'])]


It doesn't select the right IDs, it gives the whole values in elem

3) Here I set the index using IDs:

for i in Outlier.index:
    for elem in df_list:
        print(elem.Reput)
        if i in elem.index:
#             elem.loc[elem[i] , 'Reput'] += 1
            m = elem.iloc[i, :]
            print(m)


It gives this error:

IndexError: single positional indexer is out-of-bounds


I'm greatly thankful to anyone who can help me,

From sjeik_appie at hotmail.com  Fri Dec 17 10:01:58 2021
From: sjeik_appie at hotmail.com (Albert-Jan Roskam)
Date: Fri, 17 Dec 2021 15:01:58 +0000
Subject: Call julia from Python: which package?
Message-ID: <DB6PR01MB3895EBB76034AF96B9BD5D8883789@DB6PR01MB3895.eurprd01.prod.exchangelabs.com>

Hi,

I have a Python program that uses Tkinter for its GUI. It's rather slow so I hope to replace many or all of the non-GUI parts by Julia code. Has anybody experience with this? Any packages you can recommend? I found three alternatives:

* https://pyjulia.readthedocs.io/en/latest/usage.html#
* https://pypi.org/project/juliacall/
* https://github.com/JuliaPy/PyCall.jl

Thanks in advance!

Albert-Jan

From drsalists at gmail.com  Fri Dec 17 10:12:22 2021
From: drsalists at gmail.com (Dan Stromberg)
Date: Fri, 17 Dec 2021 07:12:22 -0800
Subject: Call julia from Python: which package?
In-Reply-To: <DB6PR01MB3895EBB76034AF96B9BD5D8883789@DB6PR01MB3895.eurprd01.prod.exchangelabs.com>
References: <DB6PR01MB3895EBB76034AF96B9BD5D8883789@DB6PR01MB3895.eurprd01.prod.exchangelabs.com>
Message-ID: <CAGGBd_oH=4c1FBiko1gtPCoPx_ekkWHddNG6PY3coo0ipetEvg@mail.gmail.com>

On Fri, Dec 17, 2021 at 7:02 AM Albert-Jan Roskam <sjeik_appie at hotmail.com>
wrote:

> Hi,
>
> I have a Python program that uses Tkinter for its GUI. It's rather slow so
> I hope to replace many or all of the non-GUI parts by Julia code. Has
> anybody experience with this? Any packages you can recommend? I found three
> alternatives:
>
> * https://pyjulia.readthedocs.io/en/latest/usage.html#
> * https://pypi.org/project/juliacall/
> * https://github.com/JuliaPy/PyCall.jl
>
> Thanks in advance!
>

I have zero Julia experience.

I thought I would share this though:
https://stromberg.dnsalias.org/~strombrg/speeding-python/

Even if you go the Julia route, it's probably still best to profile your
Python code to identify the slow ("hot") spots, and rewrite only them.

From s.molnar at sbcglobal.net  Fri Dec 17 10:01:02 2021
From: s.molnar at sbcglobal.net (Stephen P. Molnar)
Date: Fri, 17 Dec 2021 10:01:02 -0500
Subject: Problem Installing Pipenv
References: <000601d7f356$e8c105a0$ba4310e0$.ref@sbcglobal.net>
Message-ID: <000601d7f356$e8c105a0$ba4310e0$@sbcglobal.net>

I have Python 3.9 installed in Windows 10 on my Laptop and have installed
pipenv v-2021.11.5. When I invoke the python shell command, it fails with
thee following errors:

 

 

C:\Users\SPM\Apps\Spyder-5.2.1>pipenv shell

Traceback (most recent call last):

  File "C:\Python39\lib\runpy.py", line 197, in _run_module_as_main

    return _run_code(code, main_globals, None,

  File "C:\Python39\lib\runpy.py", line 87, in _run_code

    exec(code, run_globals)

  File "C:\Python39\Scripts\pipenv.exe\__main__.py", line 7, in <module>

  File "C:\Python39\lib\site-packages\pipenv\vendor\click\core.py", line
1137, in __call__

    return self.main(*args, **kwargs)

  File "C:\Python39\lib\site-packages\pipenv\vendor\click\core.py", line
1062, in main

    rv = self.invoke(ctx)

  File "C:\Python39\lib\site-packages\pipenv\vendor\click\core.py", line
1668, in invoke

    return _process_result(sub_ctx.command.invoke(sub_ctx))

  File "C:\Python39\lib\site-packages\pipenv\vendor\click\core.py", line
1404, in invoke

    return ctx.invoke(self.callback, **ctx.params)

  File "C:\Python39\lib\site-packages\pipenv\vendor\click\core.py", line
763, in invoke

    return __callback(*args, **kwargs)

  File "C:\Python39\lib\site-packages\pipenv\vendor\click\decorators.py",
line 84, in new_func

    return ctx.invoke(f, obj, *args, **kwargs)

  File "C:\Python39\lib\site-packages\pipenv\vendor\click\core.py", line
763, in invoke

    return __callback(*args, **kwargs)

 File "C:\Python39\lib\site-packages\pipenv\cli\command.py", line 419, in
shell

    do_shell(

  File "C:\Python39\lib\site-packages\pipenv\core.py", line 2309, in
do_shell

    shell = choose_shell()

TypeError: choose_shell() missing 1 required positional argument: 'project'

 

Stephen P. Molnar, Ph.D.

(614)312-7528

Skype: smolnar1

 


From ast at invalid  Fri Dec 17 03:25:03 2021
From: ast at invalid (ast)
Date: Fri, 17 Dec 2021 09:25:03 +0100
Subject: A problem with itertools.groupby
Message-ID: <61bc495f$0$29477$426a74cc@news.free.fr>

Python 3.9.9

Hello

I have some troubles with groupby from itertools

from itertools import groupby

for k, grp in groupby("aahfffddssssnnb"):
     print(k, list(grp))
     print(k, list(grp))

a ['a', 'a']
a []
h ['h']
h []
f ['f', 'f', 'f']
f []
d ['d', 'd']
d []
s ['s', 's', 's', 's']
s []
n ['n', 'n']
n []
b ['b']
b []

It works as expected.
itertools._grouper objects are probably iterators
so they provide their datas only once. OK

but:

li = [grp for k, grp in groupby("aahfffddssssnnb")]
list(li[0])

[]

list(li[1])

[]

It seems empty ... I don't understand why, this is
the first read of an iterator, it should provide its
data.

This one works:

["".join(grp) for k, grp in groupby("aahfffddssssnnb")]

['aa', 'h', 'fff', 'dd', 'ssss', 'nn', 'b']

regards

From liedtke at punkt.de  Fri Dec 17 10:21:32 2021
From: liedtke at punkt.de (Lars Liedtke)
Date: Fri, 17 Dec 2021 16:21:32 +0100
Subject: Call julia from Python: which package?
In-Reply-To: <CAGGBd_oH=4c1FBiko1gtPCoPx_ekkWHddNG6PY3coo0ipetEvg@mail.gmail.com>
References: <DB6PR01MB3895EBB76034AF96B9BD5D8883789@DB6PR01MB3895.eurprd01.prod.exchangelabs.com>
 <CAGGBd_oH=4c1FBiko1gtPCoPx_ekkWHddNG6PY3coo0ipetEvg@mail.gmail.com>
Message-ID: <511aa804-14d1-0e24-e8c5-f1c06282ef2e@punkt.de>

Additionally I'd like to ask, if you e.g. used pandas and numpy before
you make the effort of rewriting everything.

Yes python is slower than compiled languages, but especially with numpy
you can make use of compiled code underneath.

Of course this could mean rewriting some existing code but staying in
python.

If you have done all of that or there are other impediments, then of
course no one will hold you back.

Am 17.12.21 um 16:12 schrieb Dan Stromberg:
> On Fri, Dec 17, 2021 at 7:02 AM Albert-Jan Roskam <sjeik_appie at hotmail.com>
> wrote:
>
>> Hi,
>>
>> I have a Python program that uses Tkinter for its GUI. It's rather slow so
>> I hope to replace many or all of the non-GUI parts by Julia code. Has
>> anybody experience with this? Any packages you can recommend? I found three
>> alternatives:
>>
>> * https://pyjulia.readthedocs.io/en/latest/usage.html#
>> * https://pypi.org/project/juliacall/
>> * https://github.com/JuliaPy/PyCall.jl
>>
>> Thanks in advance!
>>
> I have zero Julia experience.
>
> I thought I would share this though:
> https://stromberg.dnsalias.org/~strombrg/speeding-python/
>
> Even if you go the Julia route, it's probably still best to profile your
> Python code to identify the slow ("hot") spots, and rewrite only them.

-- 
punkt.de GmbH
Lars Liedtke
.infrastructure

Kaiserallee 13a	
76133 Karlsruhe

Tel. +49 721 9109 500
https://infrastructure.punkt.de
info at punkt.de

AG Mannheim 108285
Gesch?ftsf?hrer: J?rgen Egeling, Daniel Lienert, Fabian Stein


From antoon.pardon at vub.be  Fri Dec 17 10:41:57 2021
From: antoon.pardon at vub.be (Antoon Pardon)
Date: Fri, 17 Dec 2021 16:41:57 +0100
Subject: A problem with itertools.groupby
In-Reply-To: <61bc495f$0$29477$426a74cc@news.free.fr>
References: <61bc495f$0$29477$426a74cc@news.free.fr>
Message-ID: <f8165b55-7e70-8f91-ee80-597c71dd6599@vub.be>


> but:
>
> li = [grp for k, grp in groupby("aahfffddssssnnb")]
> list(li[0])
>
> []
>
> list(li[1])
>
> []
>
> It seems empty ... I don't understand why, this is
> the first read of an iterator, it should provide its
> data.

The group-iterators are connected. Each group-iterator is a wrapper 
around the original iterator
with an extra termination condition. So in order to start the next 
group-iterator the previous
group-iterator is exhausted, because the original iterator has to be 
ready to produce values
for the next group-iterator.

-- 
Antoon Pardon.

From rosuav at gmail.com  Fri Dec 17 10:44:04 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 18 Dec 2021 02:44:04 +1100
Subject: A problem with itertools.groupby
In-Reply-To: <61bc495f$0$29477$426a74cc@news.free.fr>
References: <61bc495f$0$29477$426a74cc@news.free.fr>
Message-ID: <CAPTjJmqUQ6PUAbkvJVotzZF9882dkB4ZGiQWQFBdaw97Y0WUPg@mail.gmail.com>

On Sat, Dec 18, 2021 at 2:32 AM ast <ast at invalid> wrote:
>
> Python 3.9.9
>
> Hello
>
> I have some troubles with groupby from itertools
>
> from itertools import groupby
>
> li = [grp for k, grp in groupby("aahfffddssssnnb")]
> list(li[0])
>
> []
>
> list(li[1])
>
> []
>
> It seems empty ... I don't understand why, this is
> the first read of an iterator, it should provide its
> data.
>

https://docs.python.org/3/library/itertools.html#itertools.groupby

Check the explanatory third paragraph :)

ChrisA

From pkpearson at nowhere.invalid  Fri Dec 17 11:06:10 2021
From: pkpearson at nowhere.invalid (Peter Pearson)
Date: 17 Dec 2021 16:06:10 GMT
Subject: A problem with itertools.groupby
References: <61bc495f$0$29477$426a74cc@news.free.fr>
Message-ID: <j23qriFosbrU1@mid.individual.net>

On Fri, 17 Dec 2021 09:25:03 +0100, ast <ast at invalid> wrote:
[snip]
>
> but:
>
> li = [grp for k, grp in groupby("aahfffddssssnnb")]
> list(li[0])
>
> []
>
> list(li[1])
>
> []
>
> It seems empty ... I don't understand why, this is
> the first read of an iterator, it should provide its
> data.

Baffling.  Here's a shorter and less readable illustration:


>>> list(groupby("aabbb"))
[('a', <itertools._grouper object at 0xb7018c0c>), 
 ('b', <itertools._grouper object at 0xb7018a0c>)]
>>> list(groupby("aabbb"))[0]
('a', <itertools._grouper object at 0xb71d3e4c>)
>>> list(list(groupby("aabbb"))[0][1])
[]


-- 
To email me, substitute nowhere->runbox, invalid->com.

From oscar.j.benjamin at gmail.com  Fri Dec 17 17:23:00 2021
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Fri, 17 Dec 2021 22:23:00 +0000
Subject: Call julia from Python: which package?
In-Reply-To: <DB6PR01MB3895EBB76034AF96B9BD5D8883789@DB6PR01MB3895.eurprd01.prod.exchangelabs.com>
References: <DB6PR01MB3895EBB76034AF96B9BD5D8883789@DB6PR01MB3895.eurprd01.prod.exchangelabs.com>
Message-ID: <CAHVvXxScU6ONFGq66Uy37Bk0k+HOnKiKF51y_9ZUbT0S+-N6yA@mail.gmail.com>

On Fri, 17 Dec 2021 at 15:04, Albert-Jan Roskam <sjeik_appie at hotmail.com> wrote:
>
> Hi,
>
> I have a Python program that uses Tkinter for its GUI. It's rather slow so I hope to replace many or all of the non-GUI parts by Julia code. Has anybody experience with this? Any packages you can recommend? I found three alternatives:
>
> * https://pyjulia.readthedocs.io/en/latest/usage.html#
> * https://pypi.org/project/juliacall/
> * https://github.com/JuliaPy/PyCall.jl

Like others here I don't have any particular advice to offer for the
actual question you have asked. Probably you would get better answers
on a Julia mailing list.

Before you consider rewriting any of your Python code in Julia I
suggest trying it out to see if you can actually demonstrate a proof
of concept that it can even be faster for your particular problem.
Also like others here I suggest that you explore the options for
speeding things up within Python. Putting together a Frankenstein
application that's half in Julia and half in Python might be worth it
but that approach has significant costs so make sure you are clear
that there are going to be benefits before you go down that road.

I do have a very small amount of experience using Julia. As a
maintainer of SymPy I was very interested by Symbolics.jl which was
created in an attempt to be faster than SymPy. I spent some time going
through tutorials for Julia and making toy programs and then I tried
out Symbolics.jl. I also had various discussions with those involved
in Symbolics.jl on Julia mailing lists and GitHub.

Based on my experience with Julia the immediate conclusion I have that
is relevant for you is this:

Do *not* presume that writing your code in Julia rather than Python
will make it faster.

There is a lot of hype about Julia and its proselytisers will claim
that it "runs at the speed of C". That might be sort of true for
certain specific kinds of code or particular kinds of problems but it
is certainly not true in general that Julia runs at the speed of C. In
fact I have seen plenty of basic cases where Julia is slower than
Python.

The other thing that I have seen plenty of in discussions about
performance in both Python and Julia is poor use of benchmark results.
The big culprit is timeit and its equivalent btime in Julia. When I
tried to discuss the speed of operations in Julia it was suggested
that I should use btime but there are so many different levels of
caching in Julia (at least in Symbolics.jl) that the btime results are
meaningless. It seems to be common practice in the Julia community to
refer to btime results just as it is in Python to refer to timeit
results.

I think that there is a specific class of problems where Julia is
significantly faster than (CPython) which in practice is mostly around
intensive non-vectorisable floating point calculations. Note that
these are the same kind of things that can be made a lot faster if you
use PyPy or Numba or any of the related options that can be done
without rewriting your existing Python code in another language. In
Julia just as in PyPy/Numba etc it's important for the speed gain that
you can actually reduce your operations down to low-level machine
types like float64, int32 etc.

I tried writing a matrix multiplication routine in Julia to see how it
would compare with Python. I used a straight-forward dumb
implementation in both languages. I wanted to compare with something
like this which is pure Python:
https://github.com/sympy/sympy/blob/88ed7abb488da615b007dd2ed5404312caef473c/sympy/polys/matrices/dense.py#L85-L91
Note that in that particular application it isn't possible to use
something like int32 because it's part of a symbolic library that
performs exact calculations that can easily overflow any standard
machine types e.g. the determinant of a 20x20 matrix of integers
between 0 and 100 easily overflows the range of a 64 bit integer:

    >>> from sympy import *
    >>> det(randMatrix(20, 20))
    1389363438512397826059139369892638115688

When I timed the result in Julia and in Python I found that the Julia
code was slower than the Python code. Of course I don't know how to
optimise Julia code so I asked one of my colleagues who does (and who
likes to proselytise about Julia). He pointed me to here where the
creator of Julia says "BigInts are currently pretty slow in Julia":
https://stackoverflow.com/questions/37193586/bigints-seem-slow-in-julia#:~:text=BigInts%20are%20currently%20pretty%20slow,that%20basic%20operations%20are%20fast.
I should make clear here that I used the gmpy2 library in Python for
the basic integer operations which is a wrapper around the same gmp
library that is used by Julia. That means that the basic integer
operations were being done by the same gmp library (a C library) in
each case. The timing differences between Python and Julia are purely
about overhead around usage of the same underlying C library.

The Julia code was 2x slower than Python in this task. By comparison
flint is a library that actually runs at the speed of C (because it's
written in C) is about 100x faster than Python for this particular
operation:
https://www.flintlib.org/
Regardless of the explanation by Stefan Karpinski about Python's
optimisations for small integers it is very clear to me that the
"speed of C" claim needs significant qualification.

Here is another demonstration: hello world. Let's compare this between
C, Python and Julia. First C:

$ cat hello.c
#include <stdio.h>

int main(int argc, char *argv[])
{
  printf("hello\n");
  return 0;
}
$ gcc hello.c -o hello
$ time ./hello
hello
real 0m0.085s
user 0m0.001s
sys 0m0.003s
$ time ./hello
hello
real 0m0.012s
user 0m0.002s
sys 0m0.004s

So with warm file system cache (the second run) we see that it takes
around 10ms to run a hello world on this particular MacBook. If we try
the same for Python it's:

$ cat hello.py
print('hello')
$ time python hello.py
hello
real 0m0.050s
user 0m0.031s
sys 0m0.014s

Now in Python it takes 50ms for hello world and subsequent runs are
similar. Now in Julia it's:

$ cat hello.jl
print("hello")
$ time julia hello.jl
hello
real 0m4.264s
user 0m0.327s
sys 0m1.346s
$ time julia hello.jl
hello
real 0m0.303s
user 0m0.164s
sys 0m0.098s

So the first time running Julia it took 4 seconds. The second time 0.3
seconds. That's a lot slower than C and Python. These timings are no
doubt specific to OSX and my hardware to some extent. On my Linux
desktop I can get significantly shorter times for C and Python but not
for Julia (last time I tried, I don't currently have a recent version
of Julia there for testing).

For many applications this start up cost is already a dealbreaker. It
doesn't stop there though as you get the same every time you try to
import something e.g. from the repl:

julia> @time using Symbolics
  4.997553 seconds (7.87 M allocations: 563.148 MiB, 2.19% gc time,
16.74% compilation time)

Note that this time doesn't get any faster with a warm filesystem. So
if I quit Julia and restart it then "using Symbolics" will still take
5 seconds. I think that the idea is that Julia takes some time to warm
up but then does things very fast so this startup time is amortised in
some way. Clearly though it needs to be a long running process for
this 5 second startup time to become insignificant.

Even then when I try to do things with Symbolics I get more start up
costs. The first time you try to call anything it's really slow. Let's
use solve_for to solve a system of 2 linear equations in 2 unknowns:

julia> using Symbolics

julia> @variables x, y
2-element Vector{Num}:
 x
 y

julia> @time Symbolics.solve_for([x + y ~ 1, x - y ~ 0], [x, y])
  4.944882 seconds (8.13 M allocations: 466.870 MiB, 5.71% gc time,
99.90% compilation time)
2-element Vector{Float64}:
 0.5
 0.5

That's definitely not the speed of C but if we rerun that:

julia> @time Symbolics.solve_for([x + y ~ 1, x - y ~ 0], [x, y])
  0.000301 seconds (539 allocations: 22.828 KiB)
2-element Vector{Float64}:
 0.5
 0.5

Okay so the second time we call the function it returns in 0.3
milliseconds. Now all of a sudden it starts to look possibly faster
than Python but wait a minute... What happens if I change the
equations? In practice I don't want to solve the exact same equations
over and over again because once I have the answer I don't need to
recompute it. Let's change the number on the right hand side of one of
the equations:

julia> @time Symbolics.solve_for([x + y ~ 1, x - y ~ 3], [x, y])
  0.150477 seconds (348.17 k allocations: 17.864 MiB, 99.60% compilation time)
2-element Vector{Float64}:
  2.0
 -1.0

Okay so this is a lot slower than when repeatedly solving the exact
same equations. We see now that it takes about 150 milliseconds to
solve a system of 2 equations for 2 unknowns (after around 10 seconds
of startup overhead just to get to this point). That's slow compared
to SymPy which is a pure Python library (note that Symbolics.jl was
created because SymPy was considered too slow by the Julia folks). In
fact, given that it's actually just computing the result in machine
precision floating point, that's slow compared to any language that I
know.

When I asked about this slowness on the Julia mailing lists I was told
that I should be using btime instead of time to measure the time taken
by these functions:

julia> using BenchmarkTools

julia> @btime Symbolics.solve_for([x + 3*y ~ 1, x - y ~ 3], [x, y])
  94.245 ?s (611 allocations: 25.02 KiB)
2-element Vector{Float64}:
  2.5
 -0.5

So according to btime this operation takes 90 microseconds. I can only
presume that the b is short for bullshit because that timing is out by
a factor of 1000 compared to a reasonable assessment of how long this
actually takes (150 milliseconds). The reason btime reports a much
lower time is because it's repeatedly solving the exact same
equations: we've already seen that a trivial change in the equations
makes it take much longer. Clearly this is reporting the time taken to
do something that is fully cached in some way.

It is standard practice in the Julia community to report btime timings
because they eliminate (ignore) all of the startup overhead that I've
discussed above. I don't know whether btime is more accurate for other
situations but at least with Symbolics it's wildly out which makes me
question any timings that I see reported by proselytisers of Julia.

I don't want to knock Julia as a language or the Symbolics.jl library.
I think it's a nice language and I'm sure it's well suited to certain
applications. I'm also sure that many of the things I've mentioned
above are being worked on and will improve over time. However the idea
that Julia is faster than Python in general should not go
unquestioned.

--
Oscar

From rosuav at gmail.com  Fri Dec 17 17:38:21 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 18 Dec 2021 09:38:21 +1100
Subject: Call julia from Python: which package?
In-Reply-To: <CAHVvXxScU6ONFGq66Uy37Bk0k+HOnKiKF51y_9ZUbT0S+-N6yA@mail.gmail.com>
References: <DB6PR01MB3895EBB76034AF96B9BD5D8883789@DB6PR01MB3895.eurprd01.prod.exchangelabs.com>
 <CAHVvXxScU6ONFGq66Uy37Bk0k+HOnKiKF51y_9ZUbT0S+-N6yA@mail.gmail.com>
Message-ID: <CAPTjJmormmRVfRj-HWYcT1yLvGifA=MV7KjO_c2HG5NauufA0g@mail.gmail.com>

On Sat, Dec 18, 2021 at 9:24 AM Oscar Benjamin
<oscar.j.benjamin at gmail.com> wrote:
> When I timed the result in Julia and in Python I found that the Julia
> code was slower than the Python code. Of course I don't know how to
> optimise Julia code so I asked one of my colleagues who does (and who
> likes to proselytise about Julia). He pointed me to here where the
> creator of Julia says "BigInts are currently pretty slow in Julia":
> https://stackoverflow.com/questions/37193586/bigints-seem-slow-in-julia#:~:text=BigInts%20are%20currently%20pretty%20slow,that%20basic%20operations%20are%20fast.
> I should make clear here that I used the gmpy2 library in Python for
> the basic integer operations which is a wrapper around the same gmp
> library that is used by Julia. That means that the basic integer
> operations were being done by the same gmp library (a C library) in
> each case. The timing differences between Python and Julia are purely
> about overhead around usage of the same underlying C library.

Point of note: "Python actually uses a hybrid approach where small
integer values are represented inline and only when values get too
large are they represented as BigInts" might be sorta-kinda true for
Python 2, but it's not true for Python 3. In all versions of CPython
to date, all integers are objects, they're not "represented inline"
(there *are* some languages that do this intrinsically, and I think
that PyPy can sometimes unbox integers, but CPython never will); and
the performance advantage of Py2's machine-sized integers clearly
wasn't worth recreating in Py3. (It would be possible to implement it
in Py3 as an optimization, while still having the same 'int' type for
both, but nobody's done it.)

So if Python's integers are faster than Julia's, it's not because
there's any sort of hybrid approach, it's simply because CPython is
more heavily optimized for that sort of work.

(That said: I have no idea whether a StackOverflow answer from 2016 is
still applicable.)

ChrisA

From oscar.j.benjamin at gmail.com  Fri Dec 17 18:01:21 2021
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Fri, 17 Dec 2021 23:01:21 +0000
Subject: Call julia from Python: which package?
In-Reply-To: <CAPTjJmormmRVfRj-HWYcT1yLvGifA=MV7KjO_c2HG5NauufA0g@mail.gmail.com>
References: <DB6PR01MB3895EBB76034AF96B9BD5D8883789@DB6PR01MB3895.eurprd01.prod.exchangelabs.com>
 <CAHVvXxScU6ONFGq66Uy37Bk0k+HOnKiKF51y_9ZUbT0S+-N6yA@mail.gmail.com>
 <CAPTjJmormmRVfRj-HWYcT1yLvGifA=MV7KjO_c2HG5NauufA0g@mail.gmail.com>
Message-ID: <CAHVvXxRQmwaPdNgXPBb31jn48k7gHP3LT-c-eyYsV42WYxfybg@mail.gmail.com>

On Fri, 17 Dec 2021 at 22:40, Chris Angelico <rosuav at gmail.com> wrote:
>
> On Sat, Dec 18, 2021 at 9:24 AM Oscar Benjamin
> <oscar.j.benjamin at gmail.com> wrote:
> > When I timed the result in Julia and in Python I found that the Julia
> > code was slower than the Python code. Of course I don't know how to
> > optimise Julia code so I asked one of my colleagues who does (and who
> > likes to proselytise about Julia). He pointed me to here where the
> > creator of Julia says "BigInts are currently pretty slow in Julia":
> > https://stackoverflow.com/questions/37193586/bigints-seem-slow-in-julia#:~:text=BigInts%20are%20currently%20pretty%20slow,that%20basic%20operations%20are%20fast.
> > I should make clear here that I used the gmpy2 library in Python for
> > the basic integer operations which is a wrapper around the same gmp
> > library that is used by Julia. That means that the basic integer
> > operations were being done by the same gmp library (a C library) in
> > each case. The timing differences between Python and Julia are purely
> > about overhead around usage of the same underlying C library.
>
> Point of note: "Python actually uses a hybrid approach where small
> integer values are represented inline and only when values get too
> large are they represented as BigInts" might be sorta-kinda true for
> Python 2, but it's not true for Python 3. In all versions of CPython
> to date, all integers are objects, they're not "represented inline"
> (there *are* some languages that do this intrinsically, and I think
> that PyPy can sometimes unbox integers, but CPython never will); and
> the performance advantage of Py2's machine-sized integers clearly
> wasn't worth recreating in Py3. (It would be possible to implement it
> in Py3 as an optimization, while still having the same 'int' type for
> both, but nobody's done it.)
>
> So if Python's integers are faster than Julia's, it's not because
> there's any sort of hybrid approach, it's simply because CPython is
> more heavily optimized for that sort of work.
>
> (That said: I have no idea whether a StackOverflow answer from 2016 is
> still applicable.)

To be clear: I wasn't using Python's int type. I used the gmpy2.mpz
type which is precisely the same mpz from the same gmp library that
Julia uses. I'm told by my colleague that Julia has a lot of overhead
when using "heap types" which is possibly the cause of the problem.

The other library I referenced (flint) does have a super-optimised
approach for handling small integers natively as machine types at the
same time as supporting large integers and even includes hand-written
assembly for the most basic operations. That library is significantly
faster than Python's int or gmpy2/gmp.

In any case what it shows is that Julia does not have reduced overhead
*in general* (i.e. for all situations) as compared to Python. I
expected that but I also thought that this particular example would be
within the range of things where Julia out-performed Python and that
turned out not to be the case.

--
Oscar

From rosuav at gmail.com  Fri Dec 17 18:08:49 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 18 Dec 2021 10:08:49 +1100
Subject: Call julia from Python: which package?
In-Reply-To: <CAHVvXxRQmwaPdNgXPBb31jn48k7gHP3LT-c-eyYsV42WYxfybg@mail.gmail.com>
References: <DB6PR01MB3895EBB76034AF96B9BD5D8883789@DB6PR01MB3895.eurprd01.prod.exchangelabs.com>
 <CAHVvXxScU6ONFGq66Uy37Bk0k+HOnKiKF51y_9ZUbT0S+-N6yA@mail.gmail.com>
 <CAPTjJmormmRVfRj-HWYcT1yLvGifA=MV7KjO_c2HG5NauufA0g@mail.gmail.com>
 <CAHVvXxRQmwaPdNgXPBb31jn48k7gHP3LT-c-eyYsV42WYxfybg@mail.gmail.com>
Message-ID: <CAPTjJmp4Fw+oO-RMf8yJr4u1kh82M6+5Cb7oH2TfGXZ4ktTBmQ@mail.gmail.com>

On Sat, Dec 18, 2021 at 10:01 AM Oscar Benjamin
<oscar.j.benjamin at gmail.com> wrote:
>
> On Fri, 17 Dec 2021 at 22:40, Chris Angelico <rosuav at gmail.com> wrote:
> >
> > On Sat, Dec 18, 2021 at 9:24 AM Oscar Benjamin
> > <oscar.j.benjamin at gmail.com> wrote:
> > > When I timed the result in Julia and in Python I found that the Julia
> > > code was slower than the Python code. Of course I don't know how to
> > > optimise Julia code so I asked one of my colleagues who does (and who
> > > likes to proselytise about Julia). He pointed me to here where the
> > > creator of Julia says "BigInts are currently pretty slow in Julia":
> > > https://stackoverflow.com/questions/37193586/bigints-seem-slow-in-julia#:~:text=BigInts%20are%20currently%20pretty%20slow,that%20basic%20operations%20are%20fast.
> > > I should make clear here that I used the gmpy2 library in Python for
> > > the basic integer operations which is a wrapper around the same gmp
> > > library that is used by Julia. That means that the basic integer
> > > operations were being done by the same gmp library (a C library) in
> > > each case. The timing differences between Python and Julia are purely
> > > about overhead around usage of the same underlying C library.
> >
> > Point of note: "Python actually uses a hybrid approach where small
> > integer values are represented inline and only when values get too
> > large are they represented as BigInts" might be sorta-kinda true for
> > Python 2, but it's not true for Python 3. In all versions of CPython
> > to date, all integers are objects, they're not "represented inline"
> > (there *are* some languages that do this intrinsically, and I think
> > that PyPy can sometimes unbox integers, but CPython never will); and
> > the performance advantage of Py2's machine-sized integers clearly
> > wasn't worth recreating in Py3. (It would be possible to implement it
> > in Py3 as an optimization, while still having the same 'int' type for
> > both, but nobody's done it.)
> >
> > So if Python's integers are faster than Julia's, it's not because
> > there's any sort of hybrid approach, it's simply because CPython is
> > more heavily optimized for that sort of work.
> >
> > (That said: I have no idea whether a StackOverflow answer from 2016 is
> > still applicable.)
>
> To be clear: I wasn't using Python's int type. I used the gmpy2.mpz
> type which is precisely the same mpz from the same gmp library that
> Julia uses. I'm told by my colleague that Julia has a lot of overhead
> when using "heap types" which is possibly the cause of the problem.

Ah, interesting. What's the advantage of using mpz instead of Python's
builtin int?

ChrisA

From oscar.j.benjamin at gmail.com  Fri Dec 17 18:30:12 2021
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Fri, 17 Dec 2021 23:30:12 +0000
Subject: Call julia from Python: which package?
In-Reply-To: <CAPTjJmp4Fw+oO-RMf8yJr4u1kh82M6+5Cb7oH2TfGXZ4ktTBmQ@mail.gmail.com>
References: <DB6PR01MB3895EBB76034AF96B9BD5D8883789@DB6PR01MB3895.eurprd01.prod.exchangelabs.com>
 <CAHVvXxScU6ONFGq66Uy37Bk0k+HOnKiKF51y_9ZUbT0S+-N6yA@mail.gmail.com>
 <CAPTjJmormmRVfRj-HWYcT1yLvGifA=MV7KjO_c2HG5NauufA0g@mail.gmail.com>
 <CAHVvXxRQmwaPdNgXPBb31jn48k7gHP3LT-c-eyYsV42WYxfybg@mail.gmail.com>
 <CAPTjJmp4Fw+oO-RMf8yJr4u1kh82M6+5Cb7oH2TfGXZ4ktTBmQ@mail.gmail.com>
Message-ID: <CAHVvXxSZ-M_+SsnpNYk35gBNrFkGKfzhmjU-tyin2xD8KvfQNA@mail.gmail.com>

On Fri, 17 Dec 2021 at 23:11, Chris Angelico <rosuav at gmail.com> wrote:
>
> On Sat, Dec 18, 2021 at 10:01 AM Oscar Benjamin
> <oscar.j.benjamin at gmail.com> wrote:
> >
> > On Fri, 17 Dec 2021 at 22:40, Chris Angelico <rosuav at gmail.com> wrote:
> > >
> > > On Sat, Dec 18, 2021 at 9:24 AM Oscar Benjamin
> > > <oscar.j.benjamin at gmail.com> wrote:
> > > > When I timed the result in Julia and in Python I found that the Julia
> > > > code was slower than the Python code. Of course I don't know how to
> > > > optimise Julia code so I asked one of my colleagues who does (and who
> > > > likes to proselytise about Julia). He pointed me to here where the
> > > > creator of Julia says "BigInts are currently pretty slow in Julia":
> > > > https://stackoverflow.com/questions/37193586/bigints-seem-slow-in-julia#:~:text=BigInts%20are%20currently%20pretty%20slow,that%20basic%20operations%20are%20fast.
> > > > I should make clear here that I used the gmpy2 library in Python for
> > > > the basic integer operations which is a wrapper around the same gmp
> > > > library that is used by Julia. That means that the basic integer
> > > > operations were being done by the same gmp library (a C library) in
> > > > each case. The timing differences between Python and Julia are purely
> > > > about overhead around usage of the same underlying C library.
> > >
> > > Point of note: "Python actually uses a hybrid approach where small
> > > integer values are represented inline and only when values get too
> > > large are they represented as BigInts" might be sorta-kinda true for
> > > Python 2, but it's not true for Python 3. In all versions of CPython
> > > to date, all integers are objects, they're not "represented inline"
> > > (there *are* some languages that do this intrinsically, and I think
> > > that PyPy can sometimes unbox integers, but CPython never will); and
> > > the performance advantage of Py2's machine-sized integers clearly
> > > wasn't worth recreating in Py3. (It would be possible to implement it
> > > in Py3 as an optimization, while still having the same 'int' type for
> > > both, but nobody's done it.)
> > >
> > > So if Python's integers are faster than Julia's, it's not because
> > > there's any sort of hybrid approach, it's simply because CPython is
> > > more heavily optimized for that sort of work.
> > >
> > > (That said: I have no idea whether a StackOverflow answer from 2016 is
> > > still applicable.)
> >
> > To be clear: I wasn't using Python's int type. I used the gmpy2.mpz
> > type which is precisely the same mpz from the same gmp library that
> > Julia uses. I'm told by my colleague that Julia has a lot of overhead
> > when using "heap types" which is possibly the cause of the problem.
>
> Ah, interesting. What's the advantage of using mpz instead of Python's
> builtin int?

In this particular context the advantage was to give parity to the two
languages I was profiling but in general gmpy2/flint have faster large
integer arithmetic than Python's int type:

In [13]: from gmpy2 import mpz

In [14]: nums_int = [3**i for i in range(1000)]

In [15]: nums_mpz = [mpz(3)**i for i in range(1000)]

In [16]: def prod(nums):
    ...:     result = nums[0]
    ...:     for num in nums[1:]:
    ...:         result *= num
    ...:     return result
    ...:

In [17]: %time ok = prod(nums_int)
CPU times: user 384 ms, sys: 12 ms, total: 396 ms
Wall time: 398 ms

In [18]: %time ok = prod(nums_mpz)
CPU times: user 124 ms, sys: 0 ns, total: 124 ms
Wall time: 125 ms

That's somewhat significant but the big difference for SymPy in using
gmpy2 (as an optional dependency) is the mpq rational type which is
significantly faster for small rational numbers:

In [19]: from gmpy2 import mpq

In [20]: from fractions import Fraction

In [21]: nums_mpq = [mpq(i, 3) for i in range(1000)]

In [22]: nums_frac = [Fraction(i, 3) for i in range(1000)]

In [23]: %time ok = sum(nums_mpq)
CPU times: user 0 ns, sys: 0 ns, total: 0 ns
Wall time: 633 ?s

In [24]: %time ok = sum(nums_frac)
CPU times: user 8 ms, sys: 0 ns, total: 8 ms
Wall time: 10.2 ms

For some slow operations SymPy is about 30x faster when gmpy2 is installed.

--
Oscar

From Marco.Sulla.Python at gmail.com  Sat Dec 18 08:10:53 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Sat, 18 Dec 2021 14:10:53 +0100
Subject: pytest segfault, not with -v
In-Reply-To: <CABbU2U8z=r3vgukN==fT9dFJN0d1fTLboB8ek+jukG7ASYKzqg@mail.gmail.com>
References: <CABbU2U_cTjL031iQ3S7BtAQM4Yc=rhRF6NzK=_C1kOoe+JwjPA@mail.gmail.com>
 <255c2cb8-b5ea-6e32-d6eb-1765af3967d5@mrabarnett.plus.com>
 <CABbU2U-cWQbJBog-k6vr9pMPpmoY+viHdc9s1DEaihSwaNcB5A@mail.gmail.com>
 <e5d32ffa-eedd-715e-7306-e6c099258193@mrabarnett.plus.com>
 <CABbU2U8z=r3vgukN==fT9dFJN0d1fTLboB8ek+jukG7ASYKzqg@mail.gmail.com>
Message-ID: <CABbU2U97AM9BzgtJOB=Pa2WUtKVfgnx-b5T0E08nMoQ0JXai9w@mail.gmail.com>

Ok, I created the script:

https://github.com/Marco-Sulla/python-frozendict/blob/master/test/debug.py

The problem is it does _not_ crash, while a get a segfault using
pytest with python 3.9 on MacOS 10.15

Maybe it's because I'm using eval / exec in the script?

On Sat, 20 Nov 2021 at 18:40, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
>
> Indeed I have introduced a command line parameter in my bench.py
> script that simply specifies the number of times the benchmarks are
> performed. This way I have a sort of segfault checker.
>
> But I don't bench any part of the library. I suppose I have to create
> a separate script that does a simple loop for all the cases, and
> remove the optional parameter from bench. How boring.
> PS: is there a way to monitor the Python consumed memory inside Python
> itself? In this way I could also trap memory leaks.
>
> On Sat, 20 Nov 2021 at 01:46, MRAB <python at mrabarnett.plus.com> wrote:
> >
> > On 2021-11-19 23:44, Marco Sulla wrote:
> > > On Fri, 19 Nov 2021 at 20:38, MRAB <python at mrabarnett.plus.com> wrote:
> > >>
> > >> On 2021-11-19 17:48, Marco Sulla wrote:
> > >> > I have a battery of tests done with pytest. My tests break with a
> > >> > segfault if I run them normally. If I run them using pytest -v, the
> > >> > segfault does not happen.
> > >> >
> > >> > What could cause this quantical phenomenon?
> > >> >
> > >> Are you testing an extension that you're compiling? That kind of problem
> > >> can occur if there's an uninitialised variable or incorrect reference
> > >> counting (Py_INCREF/Py_DECREF).
> > >
> > > Ok, I know. But why can't it be reproduced if I do pytest -v? This way
> > > I don't know which test fails.
> > > Furthermore I noticed that if I remove the __pycache__ dir of tests,
> > > pytest does not crash, until I re-ran it with the __pycache__ dir
> > > present.
> > > This way is very hard for me to understand what caused the segfault.
> > > I'm starting to think pytest is not good for testing C extensions.
> > >
> > If there are too few Py_INCREF or too many Py_DECREF, it'll free the
> > object too soon, and whether or when that will cause a segfault will
> > depend on whatever other code is running. That's the nature of the
> > beast: it's unpredictable!
> >
> > You could try running each of the tests in a loop to see which one
> > causes a segfault. (Trying several in a loop will let you narrow it down
> > more quickly.)
> >
> > pytest et al. are good for testing behaviour, but not for narrowing down
> > segfaults.
> > --
> > https://mail.python.org/mailman/listinfo/python-list

From dieter at handshake.de  Sat Dec 18 12:33:25 2021
From: dieter at handshake.de (Dieter Maurer)
Date: Sat, 18 Dec 2021 18:33:25 +0100
Subject: pytest segfault, not with -v
In-Reply-To: <CABbU2U97AM9BzgtJOB=Pa2WUtKVfgnx-b5T0E08nMoQ0JXai9w@mail.gmail.com>
References: <CABbU2U_cTjL031iQ3S7BtAQM4Yc=rhRF6NzK=_C1kOoe+JwjPA@mail.gmail.com>
 <255c2cb8-b5ea-6e32-d6eb-1765af3967d5@mrabarnett.plus.com>
 <CABbU2U-cWQbJBog-k6vr9pMPpmoY+viHdc9s1DEaihSwaNcB5A@mail.gmail.com>
 <e5d32ffa-eedd-715e-7306-e6c099258193@mrabarnett.plus.com>
 <CABbU2U8z=r3vgukN==fT9dFJN0d1fTLboB8ek+jukG7ASYKzqg@mail.gmail.com>
 <CABbU2U97AM9BzgtJOB=Pa2WUtKVfgnx-b5T0E08nMoQ0JXai9w@mail.gmail.com>
Message-ID: <25022.7013.857728.364425@ixdm.fritz.box>

Marco Sulla wrote at 2021-12-18 14:10 +0100:
>Ok, I created the script:
>
>https://github.com/Marco-Sulla/python-frozendict/blob/master/test/debug.py
>
>The problem is it does _not_ crash, while a get a segfault using
>pytest with python 3.9 on MacOS 10.15
>
>Maybe it's because I'm using eval / exec in the script?

Segfaults can result from C stack overflow which in turn can
be caused in special cases by too deeply nested function calls
(usually, Python's "maximal recursion depth exceeded" prevents
this before a C stack overflow).

Otherwise, whatever you do in Python (this includes "eval/exec")
should not cause a segfault. The cause for it likely comes from
a memory management bug in some C implemented part of your
application.

Note that memory management bugs may not show deterministic
behavior. Minor changes (such as "with/without -v")
can significantly change the outcome.

From janburse at fastmail.fm  Sat Dec 18 07:11:02 2021
From: janburse at fastmail.fm (Mostowski Collapse)
Date: Sat, 18 Dec 2021 13:11:02 +0100
Subject: ANN: Dogelog, Invitation to the Moon!
In-Reply-To: <1b7856f2-071b-4f73-956c-6197a61ba126n@googlegroups.com>
References: <1b7856f2-071b-4f73-956c-6197a61ba126n@googlegroups.com>
Message-ID: <spkj4k$b75$2@solani.org>

Dear All,

We are happy to announce a new edition
of the Dogelog Player:

- Binary Release:
   The new version 0.9.7 of the Dogelog player
now masters DCG. We have decided to upload the
transpiled and compacted editions for the
JavaScript and Python platforms to www.xlog.ch
under the heading "Products".

- Quelltexte:
   As an alternative to GitHub, we have now
activated pages.xlog.ch. In addition to the
source texts of the transpiler and the built-ins
of the Dogelog player, this website also
contains the manuals and the source texts of
the tutorials. Both JavaScript and Python source
code can be found.

- Blogging:
   As an alternative to Gist, we are now reporting
on medium.com/@janburse_2989 about the developments
in the Dogelog player. We also use
twitter.com/dogelogch and
www.facebook.com/groups/dogelog for short messages.

Have Fun! #StaySafe
Jan Burse, 18.12.2021

Mostowski Collapse schrieb:
> Dear All,
> 
> The website http://www.xlog.ch/ is now open for
> newsletter registration. What are the plans for
> the next 10 years (sic!):
> 
> - Past: 100% Java Prolog
> We had the Jekejeke Suite consisting of Runtime,
> Minlog and Debugger. The Runtime was initially
> 100% Java, including things like consult and top-level.
> 
> - Present: 100% Prolog Prolog
> This year in 2021 we managed to deliver a new breed
> of Prolog, with the Dogelog Player we demonstrated
> a Prolog system which had most of it written in Prolog itself.
> The Dogelog Player is available for JavaScript and Python.
> 
> - Future:
> The goal is to produce a Dogelog Suite consisting
> of Runtime, Minlog and Debugger, all based on the new
> 100% Prolog approach of the Dogelog Player. It is planned
> that Dogelog Suite will again cover Java, but we could
> also try novel targets such as Steel Bank Common Lisp (SBCL), etc..
> 
> Disclaimer: It might take some time till the new website
> http://www.xlog.ch/ shows some binaries, since we have
> removed us from GitHub. We are working on it.
> 
> Have Fun! #StaySafe
> Jan Burse, 11.12.2021
> 


From Marco.Sulla.Python at gmail.com  Sat Dec 18 15:01:34 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Sat, 18 Dec 2021 21:01:34 +0100
Subject: pytest segfault, not with -v
In-Reply-To: <25022.7013.857728.364425@ixdm.fritz.box>
References: <CABbU2U_cTjL031iQ3S7BtAQM4Yc=rhRF6NzK=_C1kOoe+JwjPA@mail.gmail.com>
 <255c2cb8-b5ea-6e32-d6eb-1765af3967d5@mrabarnett.plus.com>
 <CABbU2U-cWQbJBog-k6vr9pMPpmoY+viHdc9s1DEaihSwaNcB5A@mail.gmail.com>
 <e5d32ffa-eedd-715e-7306-e6c099258193@mrabarnett.plus.com>
 <CABbU2U8z=r3vgukN==fT9dFJN0d1fTLboB8ek+jukG7ASYKzqg@mail.gmail.com>
 <CABbU2U97AM9BzgtJOB=Pa2WUtKVfgnx-b5T0E08nMoQ0JXai9w@mail.gmail.com>
 <25022.7013.857728.364425@ixdm.fritz.box>
Message-ID: <CABbU2U8edPrtN8wWEOcjr_2nSm65U46iGji-d1HOUsJEVR__Gw@mail.gmail.com>

Emh, maybe I was not clear. I created a C extension and it segfaults.
So I created that script to see where it segfaults. But the script
does not segfault. My doubt is: is that because I'm using eval and
exec in the script?

On Sat, 18 Dec 2021 at 18:33, Dieter Maurer <dieter at handshake.de> wrote:
>
> Marco Sulla wrote at 2021-12-18 14:10 +0100:
> >Ok, I created the script:
> >
> >https://github.com/Marco-Sulla/python-frozendict/blob/master/test/debug.py
> >
> >The problem is it does _not_ crash, while a get a segfault using
> >pytest with python 3.9 on MacOS 10.15
> >
> >Maybe it's because I'm using eval / exec in the script?
>
> Segfaults can result from C stack overflow which in turn can
> be caused in special cases by too deeply nested function calls
> (usually, Python's "maximal recursion depth exceeded" prevents
> this before a C stack overflow).
>
> Otherwise, whatever you do in Python (this includes "eval/exec")
> should not cause a segfault. The cause for it likely comes from
> a memory management bug in some C implemented part of your
> application.
>
> Note that memory management bugs may not show deterministic
> behavior. Minor changes (such as "with/without -v")
> can significantly change the outcome.

From barry at barrys-emacs.org  Sat Dec 18 16:51:17 2021
From: barry at barrys-emacs.org (Barry)
Date: Sat, 18 Dec 2021 21:51:17 +0000
Subject: pytest segfault, not with -v
In-Reply-To: <CABbU2U8edPrtN8wWEOcjr_2nSm65U46iGji-d1HOUsJEVR__Gw@mail.gmail.com>
References: <CABbU2U8edPrtN8wWEOcjr_2nSm65U46iGji-d1HOUsJEVR__Gw@mail.gmail.com>
Message-ID: <B675B859-7BA3-4131-A618-E179D6B190C0@barrys-emacs.org>



> On 18 Dec 2021, at 20:05, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
> 
> ?Emh, maybe I was not clear. I created a C extension and it segfaults.
> So I created that script to see where it segfaults. But the script
> does not segfault. My doubt is: is that because I'm using eval and
> exec in the script?

You may struggle to duplicate the pattern of memory allocations and accesses that lead to the segv.

Assuming you are on a unix, turn on coredumps and run the code until it segfaults.
Then use gdb on the core file.

Barry

> 
>> On Sat, 18 Dec 2021 at 18:33, Dieter Maurer <dieter at handshake.de> wrote:
>> 
>> Marco Sulla wrote at 2021-12-18 14:10 +0100:
>>> Ok, I created the script:
>>> 
>>> https://github.com/Marco-Sulla/python-frozendict/blob/master/test/debug.py
>>> 
>>> The problem is it does _not_ crash, while a get a segfault using
>>> pytest with python 3.9 on MacOS 10.15
>>> 
>>> Maybe it's because I'm using eval / exec in the script?
>> 
>> Segfaults can result from C stack overflow which in turn can
>> be caused in special cases by too deeply nested function calls
>> (usually, Python's "maximal recursion depth exceeded" prevents
>> this before a C stack overflow).
>> 
>> Otherwise, whatever you do in Python (this includes "eval/exec")
>> should not cause a segfault. The cause for it likely comes from
>> a memory management bug in some C implemented part of your
>> application.
>> 
>> Note that memory management bugs may not show deterministic
>> behavior. Minor changes (such as "with/without -v")
>> can significantly change the outcome.
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 


From martinp.dipaola at gmail.com  Sat Dec 18 18:09:44 2021
From: martinp.dipaola at gmail.com (Martin Di Paola)
Date: Sat, 18 Dec 2021 23:09:44 +0000
Subject: Call julia from Python: which package?
In-Reply-To: <CAGGBd_oH=4c1FBiko1gtPCoPx_ekkWHddNG6PY3coo0ipetEvg@mail.gmail.com>
References: <DB6PR01MB3895EBB76034AF96B9BD5D8883789@DB6PR01MB3895.eurprd01.prod.exchangelabs.com>
 <CAGGBd_oH=4c1FBiko1gtPCoPx_ekkWHddNG6PY3coo0ipetEvg@mail.gmail.com>
Message-ID: <20211218230944.c5pvfcnoverqfhot@gmail.com>

I played with Julia a few months ago. I was doing some data-science 
stuff with Pandas and the performance was terrible so I decided to give 
Julia a try.

My plan was to do the slow part in Julia and call it from Python.  
I tried juliacall (if I don't remember wrong) but I couldn't set up it.

It wasn't as smooth as it was advertised (but hey! you may have a better 
luck than me).

The other thing that I couldn't figure out is *how the data is shared 
between Python and Julia*. Basically numpy/pandas structures cannot be 
used by Julia own libraries as they are, they need to be copied at least 
and this can be a real performance hit.

I desisted the idea but you may still consider this as an real option.  
Just validate how much data you need to share (in my cases where quite 
large datasets, hence the issue).

Having said that, is Julia a real option? May be.

In my case the processing that I needed was quite basic and Julia did 
a really good job.

But I felt that the library is too fragmented. In Python you can relay 
on Pandas/numpy for processing and matplotlib/seaborn for plotting and 
you will 99% covered.

In Julia I need DataFrames.jl, Parquet.jl, CategoricalArrays.jl, 
StatsBase.jl, Statistics.jl and Peaks.jl

And I'm not including any plotting stuff.

This fragmentation is not just "inconvenient" because you need to 
install more packages but it is more difficult to code.

The API is not consistent so you need to be careful and double check 
that what you are calling is really doing what you think.

About the performance, Julia is not magic. It depends on how well it was 
coded each package.

In my case I had a good experience except with Parquet.jl which it 
didn't understand how to handle categories in the dataset and ended up 
loading a lot of duplicated strings and blew up the memory a few times.

I suggest you to write down what you need to speed up and see if it is 
implemented in Julia (do a proof of concept). Only then consider to do 
the switch.

Good luck and share your results!
Martin.


On Fri, Dec 17, 2021 at 07:12:22AM -0800, Dan Stromberg wrote:
>On Fri, Dec 17, 2021 at 7:02 AM Albert-Jan Roskam <sjeik_appie at hotmail.com>
>wrote:
>
>> Hi,
>>
>> I have a Python program that uses Tkinter for its GUI. It's rather slow so
>> I hope to replace many or all of the non-GUI parts by Julia code. Has
>> anybody experience with this? Any packages you can recommend? I found three
>> alternatives:
>>
>> * https://pyjulia.readthedocs.io/en/latest/usage.html#
>> * https://pypi.org/project/juliacall/
>> * https://github.com/JuliaPy/PyCall.jl
>>
>> Thanks in advance!
>>
>
>I have zero Julia experience.
>
>I thought I would share this though:
>https://stromberg.dnsalias.org/~strombrg/speeding-python/
>
>Even if you go the Julia route, it's probably still best to profile your
>Python code to identify the slow ("hot") spots, and rewrite only them.
>-- 
>https://mail.python.org/mailman/listinfo/python-list

From Gronicus at SGA.Ninja  Sat Dec 18 18:36:34 2021
From: Gronicus at SGA.Ninja (Steve)
Date: Sat, 18 Dec 2021 18:36:34 -0500
Subject: Custom designed alarm clock
Message-ID: <025b01d7f468$182da660$4888f320$@SGA.Ninja>


I have designed a simple alarm using Python. It has about 10 limes.
How do I convert it to an app that I can on my android Moto E power 2021
phone?

Steve


From pbryan at anode.ca  Sat Dec 18 19:49:45 2021
From: pbryan at anode.ca (Paul Bryan)
Date: Sat, 18 Dec 2021 16:49:45 -0800
Subject: Custom designed alarm clock
In-Reply-To: <025b01d7f468$182da660$4888f320$@SGA.Ninja>
References: <025b01d7f468$182da660$4888f320$@SGA.Ninja>
Message-ID: <cf01a45d63fae600a61f43bfa50a749a10579886.camel@anode.ca>

Suggested reading:

https://pypi.org/project/python-for-android/
https://play.google.com/store/apps/details?id=org.qpython.qpy3
https://www.androidauthority.com/an-introduction-to-python-on-android-759685/
https://data-flair.training/blogs/android-app-using-python/

On Sat, 2021-12-18 at 18:36 -0500, Steve wrote:
> 
> I have designed a simple alarm using Python. It has about 10 limes.
> How do I convert it to an app that I can on my android Moto E power
> 2021
> phone?
> 
> Steve
> 


From Gronicus at SGA.Ninja  Sat Dec 18 20:25:34 2021
From: Gronicus at SGA.Ninja (Steve)
Date: Sat, 18 Dec 2021 20:25:34 -0500
Subject: Custom designed alarm clock
In-Reply-To: <cf01a45d63fae600a61f43bfa50a749a10579886.camel@anode.ca>
References: <025b01d7f468$182da660$4888f320$@SGA.Ninja>
 <cf01a45d63fae600a61f43bfa50a749a10579886.camel@anode.ca>
Message-ID: <026401d7f477$52ae28f0$f80a7ad0$@SGA.Ninja>

Most of what is in that tutorial is old-hat for me. I have been coding Python for 10 years now and have a program with over 3000 lines of code.

 

However, I am reminded that Kivy has to be reinstalled on my system and now I see that using Linux is the way to go.  Buildozer? I also have to get the s14a library?

Oh well, nothing is as simple as it first seems.  It may take a while but I am willing to give it a go.

 

Steve

 

 

 George Melly remarked to Mike Jagger on how lined his face was for one so young. Jagger replied ?They?re laughter lines George? to which Melly countered: ?Mick, nothing?s that f**king funny!?.

 

From: Paul Bryan <pbryan at anode.ca> 
Sent: Saturday, December 18, 2021 7:50 PM
To: Steve <Gronicus at SGA.Ninja>; 'Python Main' <python-list at python.org>
Subject: Re: Custom designed alarm clock

 

Suggested reading:

 

https://pypi.org/project/python-for-android/

https://play.google.com/store/apps/details?id=org.qpython.qpy3

https://www.androidauthority.com/an-introduction-to-python-on-android-759685/

https://data-flair.training/blogs/android-app-using-python/

 

On Sat, 2021-12-18 at 18:36 -0500, Steve wrote:

 

I have designed a simple alarm using Python. It has about 10 lines.

How do I convert it to an app that I can on my android Moto E power 2021

phone?

 

Steve

 

 


From anthra.norell at bluewin.ch  Sun Dec 19 07:41:34 2021
From: anthra.norell at bluewin.ch (Friedrich Rentsch)
Date: Sun, 19 Dec 2021 13:41:34 +0100
Subject: Update a specific element in all a list of N lists
In-Reply-To: <CADFspfotj+9h42yAb4+Yu5Gxw_jYJ7KNSff7UP0UGpabGs-P_Q@mail.gmail.com>
References: <CADFspfotj+9h42yAb4+Yu5Gxw_jYJ7KNSff7UP0UGpabGs-P_Q@mail.gmail.com>
Message-ID: <d77abddc-ab35-4854-4b7c-7ca14ca5bdb6@bluewin.ch>



On 12/16/21 3:00 PM, hanan lamaazi wrote:
> Dear All,
>
> I really need your assistance,
>
> I have a dataset with 1005000 rows and 25 columns,
>
> The main column that I repeatedly use are Time, ID, and Reputation
>
> First I sliced the data based on the time, and I append the sliced data in
> a list called "df_list". So I get 201 lists with 25 columns
>
> The main code is starting for here:
>
> for elem in df_list:
>
> {do something.....}
>
> {Here I'm trying to calculate the outliers}
>
> Out.append(outliers)
>
> Now my problem is that I need to locate those outliers in the df_list and
> then update another column with is the "Reputation"
>
> Note that the there is a duplicated IDs but at different time slot
>
> example is ID = 1 is outliers, I need to select all ID = 1 in the list and
> update their reputation column
>
> I tried those solutions:
> 1)
>
> grp = data11.groupby(['ID'])
>          for i in GlobalNotOutliers.ID:
>              data11.loc[grp.get_group(i).index, 'Reput'] += 1
>
>          for j in GlobalOutliers.ID:
>              data11.loc[grp.get_group(j).index, 'Reput'] -= 1
>
>
> It works for a dataframe but not for a list
>
> 2)
>
> for elem in df_list:
>
> elem.loc[elem['ID'].isin(Outlier['ID'])]
>
>
> It doesn't select the right IDs, it gives the whole values in elem
>
> 3) Here I set the index using IDs:
>
> for i in Outlier.index:
>      for elem in df_list:
>          print(elem.Reput)
>          if i in elem.index:
> #             elem.loc[elem[i] , 'Reput'] += 1
>              m = elem.iloc[i, :]
>              print(m)
>
>
> It gives this error:
>
> IndexError: single positional indexer is out-of-bounds
>
>
> I'm greatly thankful to anyone who can help me,

I'd suggest you group your records by date and put each group into a 
dict whose key is date. Collecting each record into its group, append to 
it the index of the respective record in the original list. Then go 
through all your groups, record by record, finding outliers. The last 
item in the record is the index of the record in the original list 
identifying the record you want to update. Something like this:

 ??? dictionary = {}
 ??? for i, record in enumerate (original_list):
 ??? ??? date = record [DATE_INDEX]
 ??? ??? if date in dictionary:
 ??? ??? ??? dictionary [date].append ((record, i))
 ??? ??? else:
 ??? ??? ??? dictionary[date] = [(record, i)]

 ??? reputation_indexes = set ()
 ??? for date, records in dictionary.items ():
 ??? ??? for record, i in records:
 ??? ??? ??? if has_outlier (record):
 ??? ??? ??? ??? reputation_indexes.add (i)

 ??? for i in reputation_idexes:
 ??? ??? update_reputation (original_list [i])

Frederic




From sssumve at gmail.com  Sun Dec 19 09:05:00 2021
From: sssumve at gmail.com (SS Sumve)
Date: Sun, 19 Dec 2021 20:05:00 +0600
Subject: Having an issue with the command "pip install"
Message-ID: <CAGwqNEsZGGtKELxoBQMCV3-25fj8KKjopdaQwF2ORQjqP=e7YA@mail.gmail.com>

Whenever I try to install any packages with ?pip install?, I am getting an
error.



Traceback (most recent call last):

  File
"C:\Users\Lenovo\AppData\Local\Programs\Python\Python310\lib\runpy.py",
line 196, in _run_module_as_main

    return _run_code(code, main_globals, None,

  File
"C:\Users\Lenovo\AppData\Local\Programs\Python\Python310\lib\runpy.py",
line 86, in _run_code

    exec(code, run_globals)

  File
"C:\Users\Lenovo\AppData\Local\Programs\Python\Python310\Scripts\pip.exe\__main__.py",
line 7, in <module>

  File
"C:\Users\Lenovo\AppData\Local\Programs\Python\Python310\lib\site-packages\pip\_internal\cli\main.py",
line 68, in main

    command = create_command(cmd_name, isolated=("--isolated" in cmd_args))

  File
"C:\Users\Lenovo\AppData\Local\Programs\Python\Python310\lib\site-packages\pip\_internal\commands\__init__.py",
line 109, in create_command

    module = importlib.import_module(module_path)

  File
"C:\Users\Lenovo\AppData\Local\Programs\Python\Python310\lib\importlib\__init__.py",
line 126, in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in
_find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 883, in exec_module

  File "<frozen importlib._bootstrap>", line 241, in
_call_with_frames_removed

  File
"C:\Users\Lenovo\AppData\Local\Programs\Python\Python310\lib\site-packages\pip\_internal\commands\install.py",
line 14, in <module>

    from pip._internal.cli.req_command import (

  File
"C:\Users\Lenovo\AppData\Local\Programs\Python\Python310\lib\site-packages\pip\_internal\cli\req_command.py",
line 20, in <module>

    from pip._internal.index.collector import LinkCollector

  File
"C:\Users\Lenovo\AppData\Local\Programs\Python\Python310\lib\site-packages\pip\_internal\index\collector.py",
line 34, in <module>

    from pip._internal.network.session import PipSession

  File
"C:\Users\Lenovo\AppData\Local\Programs\Python\Python310\lib\site-packages\pip\_internal\network\session.py",
line 31, in <module>

    from pip._internal.network.auth import MultiDomainBasicAuth

  File
"C:\Users\Lenovo\AppData\Local\Programs\Python\Python310\lib\site-packages\pip\_internal\network\auth.py",
line 22, in <module>

    from pip._internal.vcs.versioncontrol import AuthInfo

  File
"C:\Users\Lenovo\AppData\Local\Programs\Python\Python310\lib\site-packages\pip\_internal\vcs\__init__.py",
line 6, in <module>

    import pip._internal.vcs.git

ModuleNotFoundError: No module named 'pip._internal.vcs.git'


I got this error when I tried to install numpy.
Please help me to solve this problem.

From pieter-l at vanoostrum.org  Sun Dec 19 12:22:19 2021
From: pieter-l at vanoostrum.org (Pieter van Oostrum)
Date: Sun, 19 Dec 2021 18:22:19 +0100
Subject: jupyter console vs. ipython in Emacs
Message-ID: <m2lf0g8q7o.fsf@vanoostrum.org>

My Python development environment is Emacs. I used to use 'jupyter console --simple-prompt' (driven by Emacs comint) to do interactive work, but it has the disadvantage that it doesn't work properly with multiline input, including pastes.

However, I discovered that 'ipython --simple-prompt' in Emacs does work with multiline statements. No idea why jupyter console doesn't.

Is there any advantage in using jupyter console over ipython?
-- 
Pieter van Oostrum <pieter at vanoostrum.org>
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

From dieter at handshake.de  Sun Dec 19 13:12:54 2021
From: dieter at handshake.de (Dieter Maurer)
Date: Sun, 19 Dec 2021 19:12:54 +0100
Subject: pytest segfault, not with -v
In-Reply-To: <CABbU2U8edPrtN8wWEOcjr_2nSm65U46iGji-d1HOUsJEVR__Gw@mail.gmail.com>
References: <CABbU2U_cTjL031iQ3S7BtAQM4Yc=rhRF6NzK=_C1kOoe+JwjPA@mail.gmail.com>
 <255c2cb8-b5ea-6e32-d6eb-1765af3967d5@mrabarnett.plus.com>
 <CABbU2U-cWQbJBog-k6vr9pMPpmoY+viHdc9s1DEaihSwaNcB5A@mail.gmail.com>
 <e5d32ffa-eedd-715e-7306-e6c099258193@mrabarnett.plus.com>
 <CABbU2U8z=r3vgukN==fT9dFJN0d1fTLboB8ek+jukG7ASYKzqg@mail.gmail.com>
 <CABbU2U97AM9BzgtJOB=Pa2WUtKVfgnx-b5T0E08nMoQ0JXai9w@mail.gmail.com>
 <25022.7013.857728.364425@ixdm.fritz.box>
 <CABbU2U8edPrtN8wWEOcjr_2nSm65U46iGji-d1HOUsJEVR__Gw@mail.gmail.com>
Message-ID: <25023.30246.44692.185039@ixdm.fritz.box>

Marco Sulla wrote at 2021-12-18 21:01 +0100:
>Emh, maybe I was not clear. I created a C extension and it segfaults.
>So I created that script to see where it segfaults. But the script
>does not segfault. My doubt is: is that because I'm using eval and
>exec in the script?

The segfault in your C extension is likely caused by a memory management
error. The effects of such errors are typically non local and
apparently non deterministic: small things can decide whether
you see or do not see such an effect.

Use other tools (than a script) to hunt memory management errors.

Python has a compile time option which can help (I forgot its name,
but when you search for Python compile time options, you should
find it): it puts marks before and after the memory areas used for
objects allocated via its API and checks that those marks remain intact.
There is some chance that effects of Memory management errors are
detected earlier by those checks and therefore easier to analyse.

There are specialized tools for the analysis of memory management
errors, e.g. `valgrind/memcheck`. Use one of those for complex problems.


Python's memory management rules (to be observed by C extensions)
are complex. It is quite easy to violate them.

For this reason, I try not to directly write C extensions for Python
but let them generate by `cython`.
The source of a `cython` program resembles Python source code
with extensions. The extensions control the compilation to C,
mostly for optimization purposes. For example, there is a
declaration to mark a variable as a C (rather than Python) variable.
When `cython` compiles the program to "C", it observes
all the complex rules of the Python C API.

From pieter-l at vanoostrum.org  Mon Dec 20 07:38:26 2021
From: pieter-l at vanoostrum.org (Pieter van Oostrum)
Date: Mon, 20 Dec 2021 13:38:26 +0100
Subject: jupyter console vs. ipython in Emacs
References: <m2lf0g8q7o.fsf@vanoostrum.org>
Message-ID: <m2h7b38n99.fsf@vanoostrum.org>

Pieter van Oostrum <pieter-l at vanoostrum.org> writes:

> My Python development environment is Emacs. I used to use 'jupyter
> console --simple-prompt' (driven by Emacs comint) to do interactive
> work, but it has the disadvantage that it doesn't work properly with
> multiline input, including pastes.
>
> However, I discovered that 'ipython --simple-prompt' in Emacs does work
> with multiline statements. No idea why jupyter console doesn't.
>
> Is there any advantage in using jupyter console over ipython?

Actually, it also happens when run in Terminal, so it is not an Emacs issue.
By the way, my OS is MacOS Catalina. I have no idea how it works on other systems.
-- 
Pieter van Oostrum <pieter at vanoostrum.org>
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

From electriclightheads at gmail.com  Tue Dec 21 05:55:25 2021
From: electriclightheads at gmail.com ('2+)
Date: Tue, 21 Dec 2021 19:55:25 +0900
Subject: 'array.array' object has no attribute 'tostring' which 3 got it fixed?
Message-ID: <CACnmWETHYyZfNo+d+pdVa+-hCszPBJxLF5if36wNyFjRKgsV+A@mail.gmail.com>

hi ;)

got popos installed on my raspberry pi4 and it is currently running python
3.9.7

i get this error when running my script:

'array.array' object has no attribute 'tostring'

this bug seems to be pretty old .. how long should i be waiting to get it
fixed with apt upgrade?

or should i use other attribute instead?

From rosuav at gmail.com  Tue Dec 21 06:30:58 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 21 Dec 2021 22:30:58 +1100
Subject: 'array.array' object has no attribute 'tostring' which 3 got it
 fixed?
In-Reply-To: <CACnmWETHYyZfNo+d+pdVa+-hCszPBJxLF5if36wNyFjRKgsV+A@mail.gmail.com>
References: <CACnmWETHYyZfNo+d+pdVa+-hCszPBJxLF5if36wNyFjRKgsV+A@mail.gmail.com>
Message-ID: <CAPTjJmqBn1V+Zkd_y=Nh1B7=q9uJws+DDgikZy6r678425bOUg@mail.gmail.com>

On Tue, Dec 21, 2021 at 9:56 PM '2+ <electriclightheads at gmail.com> wrote:
>
> hi ;)
>
> got popos installed on my raspberry pi4 and it is currently running python
> 3.9.7
>
> i get this error when running my script:
>
> 'array.array' object has no attribute 'tostring'
>
> this bug seems to be pretty old .. how long should i be waiting to get it
> fixed with apt upgrade?
>
> or should i use other attribute instead?

I'd recommend reading the documentation :)

https://docs.python.org/3/library/array.html#array.array.tobytes

ChrisA

From sjeik_appie at hotmail.com  Tue Dec 21 12:14:52 2021
From: sjeik_appie at hotmail.com (Albert-Jan Roskam)
Date: Tue, 21 Dec 2021 18:14:52 +0100
Subject: Call julia from Python: which package?
In-Reply-To: <20211218230944.c5pvfcnoverqfhot@gmail.com>
Message-ID: <DB6PR01MB38957B543896DA9C1DD09020837C9@DB6PR01MB3895.eurprd01.prod.exchangelabs.com>

   Hi all,
   Thank you very much for your valuable replies! I will definitely do some
   tracing to see where the bottlenecks really are. It's good to know that
   pypy is still alive and kicking, I thought it was stuck in py2.7. I will
   also write a mini program during the holiday to see how this Julia/Python
   interaction might work. The little bit of experience with Julia more or
   less coincides with what Oscar mentioned: a lot of "warm up" time. This is
   actually a py2.7 project that I inherited. I was asked to convert it to
   py3.8.
   Thanks and merry xmas!
   Albert-Jan

From Jose-Marcio.Martins at minesparis.psl.eu  Mon Dec 20 13:46:32 2021
From: Jose-Marcio.Martins at minesparis.psl.eu (Jose-Marcio Martins da Cruz)
Date: Mon, 20 Dec 2021 19:46:32 +0100
Subject: [Python-announce] Benchmark Smil vs Scikit-image (morphological
 features)
Message-ID: <5e765062-2965-4bd4-5152-60d75776e9b0@minesparis.psl.eu>


Hello,

We've done a comparative benchmark (speed and memory usage) with Smil and Scikit-Image.

Smil is a mathematical morphology dedicated library of functions. So comparisons are done only on this area.

We've been working on Mathematical Morphology for more than 50 years now the discipline was created here at our research 
department in the sixties. Smil inherits the experience of previous libraries and software we've been writing since the 
70's.

In just some few words, Smil can be orders of magnitude faster than Scikit-image (hundreds or even thousands) on some 
operations thanks to parallelization and vectorization (SIMD), depending on the computer architecture.

Smil doesn't replace scikit-image but may be a good complement to scikit-image when speed is important.

Benchmark results are available at :
	https://smil.cmm.minesparis.psl.eu/smil-vs-skimage

Comments are welcome,

Best regards

Jos?-Marcio



-- 

  ---------------------------------------------------------------
  Jose Marcio MARTINS DA CRUZ, Ph.D.
  Ecole des Mines de Paris
  Centre de Morphologie Math?matique
  https://orcid.org/0000-0002-2981-7028
  ---------------------------------------------------------------
   Mon livre sur le spam : http://amzn.to/LEscRu
  ---------------------------------------------------------------


From arj.python at gmail.com  Tue Dec 21 16:42:11 2021
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Wed, 22 Dec 2021 01:42:11 +0400
Subject: PyCharm settings - per: print('\N{flag: Mauritius}') not
 supported in py3.9
In-Reply-To: <02c13a24-6736-6eaf-7b2a-4c85e28caf8e@DancesWithMice.info>
References: <CADrxXXkQcFMJhAt5AGt=BL=QqHCgJ7qbx=ts0zR8H9s2L1ZMdg@mail.gmail.com>
 <CAPTjJmoxxRU=-uA0_RJnocZFm6WG0+OpZXFM3TDX2Tgcr1=WBg@mail.gmail.com>
 <c4f62875-5c4e-055e-cc5d-7a555e25e88f@DancesWithMice.info>
 <02c13a24-6736-6eaf-7b2a-4c85e28caf8e@DancesWithMice.info>
Message-ID: <CADrxXX=UvK4_Kmz5=k7t9JAT9-ANJaYj61rX4f7bGXOp29D34g@mail.gmail.com>

Yet another unicode issue XD

Kind Regards,

Abdur-Rahmaan Janhangeer
about <https://compileralchemy.github.io/> | blog
<https://www.pythonkitchen.com>
github <https://github.com/Abdur-RahmaanJ>
Mauritius

From Marco.Sulla.Python at gmail.com  Tue Dec 21 17:05:02 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Tue, 21 Dec 2021 23:05:02 +0100
Subject: Py_TRASHCAN_SAFE_BEGIN/END in C extension?
Message-ID: <CABbU2U-UCsxYV+gnsPy3qG06mqC0k0WBarCkAC6GRTD66CTodQ@mail.gmail.com>

In Python 3.7, must Py_TRASHCAN_SAFE_BEGIN - Py_TRASHCAN_SAFE_END be
used in a C extension?

I'm asking because in my C extension I use them in the deallocator
without problems, but users signalled me that they segfault in Python
3.7 on Debian 10. I checked and this is true.

From barry at barrys-emacs.org  Tue Dec 21 17:22:23 2021
From: barry at barrys-emacs.org (Barry)
Date: Tue, 21 Dec 2021 22:22:23 +0000
Subject: Py_TRASHCAN_SAFE_BEGIN/END in C extension?
In-Reply-To: <CABbU2U-UCsxYV+gnsPy3qG06mqC0k0WBarCkAC6GRTD66CTodQ@mail.gmail.com>
References: <CABbU2U-UCsxYV+gnsPy3qG06mqC0k0WBarCkAC6GRTD66CTodQ@mail.gmail.com>
Message-ID: <1E3FF104-33CF-46F3-B20C-2E6E2729D8A3@barrys-emacs.org>



> On 21 Dec 2021, at 22:08, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
> 
> ?In Python 3.7, must Py_TRASHCAN_SAFE_BEGIN - Py_TRASHCAN_SAFE_END be
> used in a C extension?
> 
> I'm asking because in my C extension I use them in the deallocator
> without problems, but users signalled me that they segfault in Python
> 3.7 on Debian 10. I checked and this is true.

I searched the web for Py_TRASHCAN_SAFE_BEGIN
And that quickly lead me to this bug.

https://bugs.python.org/issue40608.

That gives lots of clues for what might be the problem.
It seems that is a deprecated api.

Barry

> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

From PythonList at DancesWithMice.info  Tue Dec 21 18:14:04 2021
From: PythonList at DancesWithMice.info (dn)
Date: Wed, 22 Dec 2021 12:14:04 +1300
Subject: print('\N{flag: Mauritius}') not supported in py3.9
In-Reply-To: <YaVizKu2y3/ml75h@cskk.homeip.net>
References: <a511b1cf-c634-9302-fa9b-8fbe5d4ae050@DancesWithMice.info>
 <YaVizKu2y3/ml75h@cskk.homeip.net>
Message-ID: <1c797cc1-c851-2ef5-31f1-9889a4192adb@DancesWithMice.info>

On 30/11/2021 12.31, Cameron Simpson wrote:
> On 30Nov2021 10:59, DL Neil <PythonList at DancesWithMice.info> wrote:
...

>> I've nominated Kitty as
>> Fedora's default terminal. We'll see how it goes with work-loads beyond
>> raising the flag...
> 
> I'd like to hear how that goes down the track. If I find myself on a 
> Linux desktop again a good terminal emulator would be very welcome.


(given that @A-R has brought this thread back-to-life)


I have been surprised/shocked/horrified/annoyed to discover that the
(Linux) clipboard is not accessible from "Kitty".

Go on, I dare you to remind me that good-old 'dumb-terminals' didn't
have 'clipboards'...
(neither did they have to cope with Unicode, emoticons, or national flags!)


Accordingly, having developed an idea in the REPL (running within
Kitty), could not later copy-paste into a Python module or tutorial text
(nor, next time the situation arises, to be able to illustrate an answer
to a question posted 'here').

Am somewhat in disbelief, and my fingers feel slightly singed. Grump!
(or its seasonal variation: "Grinch")


Am open to further non-terminal, terminal suggestions...

(during my post-op recovery period, am hoping to experiment with another
Linux distro (and Window Manager), which may alter the playing-field...)


Meantime, casting-off the terminal-Grinch, compliments of the season to
you...
-- 
Regards,
=dn

From robertvstepp at gmail.com  Tue Dec 21 22:22:49 2021
From: robertvstepp at gmail.com (boB Stepp)
Date: Tue, 21 Dec 2021 21:22:49 -0600
Subject: print('\N{flag: Mauritius}') not supported in py3.9
In-Reply-To: <1c797cc1-c851-2ef5-31f1-9889a4192adb@DancesWithMice.info>
References: <a511b1cf-c634-9302-fa9b-8fbe5d4ae050@DancesWithMice.info>
 <YaVizKu2y3/ml75h@cskk.homeip.net>
 <1c797cc1-c851-2ef5-31f1-9889a4192adb@DancesWithMice.info>
Message-ID: <YcKaCXAWnIRKB/XC@Dream-Machine1>

On 21/12/22 12:14PM, Python wrote:
>On 30/11/2021 12.31, Cameron Simpson wrote:
>> On 30Nov2021 10:59, DL Neil <PythonList at DancesWithMice.info> wrote:
>...
>
>>> I've nominated Kitty as
>>> Fedora's default terminal. We'll see how it goes with work-loads beyond
>>> raising the flag...
>>
>> I'd like to hear how that goes down the track. If I find myself on a
>> Linux desktop again a good terminal emulator would be very welcome.
>
>
>(given that @A-R has brought this thread back-to-life)
>
>
>I have been surprised/shocked/horrified/annoyed to discover that the
>(Linux) clipboard is not accessible from "Kitty".

Huh?  I use kitty and copy to and from the clipboard all the time.
     1) ctrl+shift+c Copy to clipboard
     2) ctrl+shift+v Paste from the clipboard
     3) Using mouse to select text automatically copies it to the primary clipboard.
     4) Middle-click of mouse to paste from the primary clipboard.


-- 
Wishing you only the best,
boB Stepp

Speeches are like steer horns -- a point here, a point there and a lot of bull
in between.
     -- E. Anderson

From drsalists at gmail.com  Tue Dec 21 22:40:54 2021
From: drsalists at gmail.com (Dan Stromberg)
Date: Tue, 21 Dec 2021 19:40:54 -0800
Subject: Call julia from Python: which package?
In-Reply-To: <DB6PR01MB38957B543896DA9C1DD09020837C9@DB6PR01MB3895.eurprd01.prod.exchangelabs.com>
References: <20211218230944.c5pvfcnoverqfhot@gmail.com>
 <DB6PR01MB38957B543896DA9C1DD09020837C9@DB6PR01MB3895.eurprd01.prod.exchangelabs.com>
Message-ID: <CAGGBd_pGEy48KzKzu0-en4oidrYfDatm2+sucnd30ERLYCTGEw@mail.gmail.com>

Last I heard, Pypy itself will remain written in a statically typed dialect
of Python that is closest to Python 2.7.  However, what's written in that
language includes JIT-compiled interpreters for both Python 2.x and Python
3.x.

On Tue, Dec 21, 2021 at 9:15 AM Albert-Jan Roskam <sjeik_appie at hotmail.com>
wrote:

>    Hi all,
>    Thank you very much for your valuable replies! I will definitely do some
>    tracing to see where the bottlenecks really are. It's good to know that
>    pypy is still alive and kicking, I thought it was stuck in py2.7. I will
>    also write a mini program during the holiday to see how this
> Julia/Python
>    interaction might work. The little bit of experience with Julia more or
>    less coincides with what Oscar mentioned: a lot of "warm up" time. This
> is
>    actually a py2.7 project that I inherited. I was asked to convert it to
>    py3.8.
>    Thanks and merry xmas!
>    Albert-Jan
> --
> https://mail.python.org/mailman/listinfo/python-list
>

From Marco.Sulla.Python at gmail.com  Wed Dec 22 03:14:01 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Wed, 22 Dec 2021 09:14:01 +0100
Subject: Py_TRASHCAN_SAFE_BEGIN/END in C extension?
In-Reply-To: <1E3FF104-33CF-46F3-B20C-2E6E2729D8A3@barrys-emacs.org>
References: <CABbU2U-UCsxYV+gnsPy3qG06mqC0k0WBarCkAC6GRTD66CTodQ@mail.gmail.com>
 <1E3FF104-33CF-46F3-B20C-2E6E2729D8A3@barrys-emacs.org>
Message-ID: <CABbU2U9FBYP9ueXuoYwGLBTCajK-t+QvNjuUfz8__+ncAVMuQQ@mail.gmail.com>

Yes, it's deprecated, but I need it for Python 3.7, since there was
yet no Py_TRASHCAN_BEGIN / END

On Tue, 21 Dec 2021 at 23:22, Barry <barry at barrys-emacs.org> wrote:
>
>
>
> On 21 Dec 2021, at 22:08, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
>
> ?In Python 3.7, must Py_TRASHCAN_SAFE_BEGIN - Py_TRASHCAN_SAFE_END be
> used in a C extension?
>
> I'm asking because in my C extension I use them in the deallocator
> without problems, but users signalled me that they segfault in Python
> 3.7 on Debian 10. I checked and this is true.
>
>
> I searched the web for Py_TRASHCAN_SAFE_BEGIN
> And that quickly lead me to this bug.
>
> https://bugs.python.org/issue40608.
>
> That gives lots of clues for what might be the problem.
> It seems that is a deprecated api.
>
> Barry
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

From barry at barrys-emacs.org  Wed Dec 22 06:09:21 2021
From: barry at barrys-emacs.org (Barry Scott)
Date: Wed, 22 Dec 2021 11:09:21 +0000
Subject: Py_TRASHCAN_SAFE_BEGIN/END in C extension?
In-Reply-To: <CABbU2U9FBYP9ueXuoYwGLBTCajK-t+QvNjuUfz8__+ncAVMuQQ@mail.gmail.com>
References: <CABbU2U-UCsxYV+gnsPy3qG06mqC0k0WBarCkAC6GRTD66CTodQ@mail.gmail.com>
 <1E3FF104-33CF-46F3-B20C-2E6E2729D8A3@barrys-emacs.org>
 <CABbU2U9FBYP9ueXuoYwGLBTCajK-t+QvNjuUfz8__+ncAVMuQQ@mail.gmail.com>
Message-ID: <6B367D53-1718-4B7F-8D41-4100BA9E649F@barrys-emacs.org>



> On 22 Dec 2021, at 08:14, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
> 
> Yes, it's deprecated, but I need it for Python 3.7, since there was
> yet no Py_TRASHCAN_BEGIN / END

Hopefully the bug report will makes clear what you have to do in your code to get things working.

Barry

> 
> On Tue, 21 Dec 2021 at 23:22, Barry <barry at barrys-emacs.org> wrote:
>> 
>> 
>> 
>> On 21 Dec 2021, at 22:08, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
>> 
>> ?In Python 3.7, must Py_TRASHCAN_SAFE_BEGIN - Py_TRASHCAN_SAFE_END be
>> used in a C extension?
>> 
>> I'm asking because in my C extension I use them in the deallocator
>> without problems, but users signalled me that they segfault in Python
>> 3.7 on Debian 10. I checked and this is true.
>> 
>> 
>> I searched the web for Py_TRASHCAN_SAFE_BEGIN
>> And that quickly lead me to this bug.
>> 
>> https://bugs.python.org/issue40608.
>> 
>> That gives lots of clues for what might be the problem.
>> It seems that is a deprecated api.
>> 
>> Barry
>> 
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>> 
> 


From juliushamilton100 at gmail.com  Thu Dec 23 06:01:45 2021
From: juliushamilton100 at gmail.com (Julius Hamilton)
Date: Thu, 23 Dec 2021 12:01:45 +0100
Subject: Advanced ways to get object information from within python
Message-ID: <CAEsMKX0eOPONpQSjsRG_-eRFRUD+Md5=wrBEGAD2c7f8=Z590w@mail.gmail.com>

Hello,

I would like to significantly increase my abilities to find the information
I am seeking about any Python object I am using from within Python. I find
this to be a really essential skill set. After reading documentation, it
really helps to get under the hood at the command line and start testing
your own competence by examining all the methods and classes, and their
arguments and return types and so on.

I was hoping someone could help me fill in more details about what I
currently know.

I'd like to use Scrapy as an example, since it's a library I'm currently
learning.

import scrapy

I assume I'll start with "dir", as it's the most convenient.

dir(scrapy) shows this:

['Field', 'FormRequest', 'Item', 'Request', 'Selector', 'Spider',
'__all__', '__builtins__', '__cached__', '__doc__', '__file__',
'__loader__', '__name__', '__package__', '__path__', '__spec__',
'__version__', '_txv', 'exceptions', 'http', 'item', 'link',
'linkextractors', 'selector', 'signals', 'spiders', 'twisted_version',
'utils', 'version_info']

I wish there was a convenient way for me to know what all of these are. I
understand "dir" shows everything from the namespace - so that includes
methods which are present only because they are imported from other
modules, by this module.

Let's assume at minimum I know that I should be able to call all these
"attributes" (I believe this is what Python calls them - an attribute can
be anything, a method, a variable, etc. But then, how to distinguish
between this general notion of an "attribute" vs a specific attribute of a
class? Or is that called a "property" or something?)

I can confirm that every single name in the above list works when I call it
from scrapy, like this:

>>> scrapy.Field
<class 'scrapy.item.Field'>

>>> scrapy.utils
<module 'scrapy.utils' from
'/usr/local/lib/python3.9/dist-packages/scrapy/utils/__init__.py'>

But I can't conveniently iterate over all of these to see all their types,
because dir() returns a list of strings. How can I iterate over all
attributes?

I can't use "getattr" because that requires you to enter the name of what
you're looking for. I would like to spit out all attributes with their
types, so I can know what my options are in more detail than dir() provides.

This is basically a dead-end for me until someone can illuminate this
strategy I'm pursuing, so now I'd like to focus on inspect and help.

inspect.getmembers is useful in principle, but I find the results to give
information overload.

This is just an excerpt of what it returns:

pprint.pprint(inspect.getmembers(scrapy))
[('Field', <class 'scrapy.item.Field'>),
 ('Selector', <class 'scrapy.selector.unified.Selector'>),
 ('Spider', <class 'scrapy.spiders.Spider'>),
 ('__all__',
  ['__version__',
   'version_info',
   'twisted_version',
   'Spider',

Why does it just list the name and type for some classes, but for others
goes on to a sublist? __all__ does not list any type in adjacent angular
brackets, it just goes on to list some attributes without any information
about what they are. Can I suppress sublists from being printed with
inspect.getmethods? Or can I recursively require sublists also display
their type?

Lastly, the "help" function.

I find "help" to similarly be a situation of information overload. Again,
it starts with a list of "package contents". I'm not sure I see the use of
this long list of names, without much description of what they are. Next,
it lists "classes", but I don't understand:

    builtins.dict(builtins.object)
        scrapy.item.Field
    parsel.selector.Selector(builtins.object)
        scrapy.selector.unified.Selector(parsel.selector.Selector,
scrapy.utils.trackref.object_ref)

What determines the order of these classes - the order in which they appear
in the source code? What about the indentation? builtins.dict() is a Python
builtin. Then why is it listed inside of Scrapy's "help" - are all builtins
necessarily listed inside a class or just the builtins it specifically
imported or inherited?

My best guess is the most indented lines are what is actually written in
the class, the lines above are just listing the inheritance? So
scrapy.item.Field inherits the Python dictionary class, and it does this
because that way you can treat the class like a dictionary sometimes, using
dictionary methods and so on?

    class Field(builtins.dict)
     |  Container of field metadata
     |
     |  Method resolution order:
     |      Field
     |      builtins.dict
     |      builtins.object
     |
     |  Data descriptors defined here:

What are data descriptors?

I understand IDE's tend to just print the docstring of a method as a sort
of overlay while you are writing with it, but I'm not able to use the
__docstring__ variable well - scrapy.__docstring__,
scrapy.Spider.__docstring__, and so on, return "object has no attribute
__docstring__".

I'm really fond of inspect.getsource(), all else failing, though - that's
very clear and insightful.

There's more to learn but that's enough questions for now. I'd really
appreciate anybody helping me find effective ways of investigating modules,
classes and methods from the command line.

Thanks very much,
Julius

From shishaozhong at gmail.com  Thu Dec 23 11:38:25 2021
From: shishaozhong at gmail.com (Shaozhong SHI)
Date: Thu, 23 Dec 2021 16:38:25 +0000
Subject: Can Python call and use FME modules and functions such as
 StreamOrderCalculator?
Message-ID: <CA+i5JwYUjxJT_iResxvcC+9dUkdzKq+zKkA3ijBTKOm=75iXcA@mail.gmail.com>

Can we do something like import an fme.something and make use of FME
modules and functions?

Regards,

David

From dieter at handshake.de  Thu Dec 23 12:17:33 2021
From: dieter at handshake.de (Dieter Maurer)
Date: Thu, 23 Dec 2021 18:17:33 +0100
Subject: Advanced ways to get object information from within python
In-Reply-To: <CAEsMKX0eOPONpQSjsRG_-eRFRUD+Md5=wrBEGAD2c7f8=Z590w@mail.gmail.com>
References: <CAEsMKX0eOPONpQSjsRG_-eRFRUD+Md5=wrBEGAD2c7f8=Z590w@mail.gmail.com>
Message-ID: <25028.44845.136679.110834@ixdm.fritz.box>

Julius Hamilton wrote at 2021-12-23 12:01 +0100:
>I would like to significantly increase my abilities to find the information
>I am seeking about any Python object I am using from within Python. I find
>this to be a really essential skill set. After reading documentation, it
>really helps to get under the hood at the command line and start testing
>your own competence by examining all the methods and classes, and their
>arguments and return types and so on.
>
>I was hoping someone could help me fill in more details about what I
>currently know.

Look at the `inspect` module.

From imtiazahmadhk822 at gmail.com  Thu Dec 23 12:16:45 2021
From: imtiazahmadhk822 at gmail.com (IMTIAZ  AHMAD)
Date: Thu, 23 Dec 2021 09:16:45 -0800
Subject: Sir jupyter note book is not working on my laptop? 
Message-ID: <1C5551A2-FCAD-4B0E-BFCE-7C743920735B@hxcore.ol>





   Sent from [1]Mail for Windows



References

   Visible links
   1. https://go.microsoft.com/fwlink/?LinkId=550986

From boblatest at yahoo.com  Thu Dec 23 12:12:33 2021
From: boblatest at yahoo.com (Robert Latest)
Date: 23 Dec 2021 17:12:33 GMT
Subject: Advanced ways to get object information from within python
References: <CAEsMKX0eOPONpQSjsRG_-eRFRUD+Md5=wrBEGAD2c7f8=Z590w@mail.gmail.com>
 <mailman.12.1640257319.3079.python-list@python.org>
Message-ID: <j2jp01FqvquU2@mid.individual.net>

Julius Hamilton wrote:
> dir(scrapy) shows this:
>
> ['Field', 'FormRequest', 'Item', 'Request', 'Selector', 'Spider',
> '__all__', '__builtins__', '__cached__', '__doc__', '__file__',
> '__loader__', '__name__', '__package__', '__path__', '__spec__',
> '__version__', '_txv', 'exceptions', 'http', 'item', 'link',
> 'linkextractors', 'selector', 'signals', 'spiders', 'twisted_version',
> 'utils', 'version_info']
>
> I wish there was a convenient way for me to know what all of these are.

['%s: %s' % (x, type(getattr(scrapy, x))) for x in dir(scrapy)]


From ikorot01 at gmail.com  Thu Dec 23 14:46:24 2021
From: ikorot01 at gmail.com (Igor Korot)
Date: Thu, 23 Dec 2021 13:46:24 -0600
Subject: Sir jupyter note book is not working on my laptop?
In-Reply-To: <1C5551A2-FCAD-4B0E-BFCE-7C743920735B@hxcore.ol>
References: <1C5551A2-FCAD-4B0E-BFCE-7C743920735B@hxcore.ol>
Message-ID: <CA+FnnTy8FW9=sC5JwmAL2KC5uEDmmPcPmaAX3ciHmW5uUDAW5g@mail.gmail.com>

Hi,

On Thu, Dec 23, 2021 at 1:42 PM IMTIAZ AHMAD <imtiazahmadhk822 at gmail.com> wrote:
>
>

Well, my program just crashed.

Can you help?

Thank you.

>
>
>
>    Sent from [1]Mail for Windows
>
>
>
> References
>
>    Visible links
>    1. https://go.microsoft.com/fwlink/?LinkId=550986
> --
> https://mail.python.org/mailman/listinfo/python-list

From skip.montanaro at gmail.com  Thu Dec 23 15:50:36 2021
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Thu, 23 Dec 2021 14:50:36 -0600
Subject: email.message_from_file & quoted printable
Message-ID: <CANc-5UyPuqb3vN5czu5RE=wb1jm=voah6yg89U1qkWfGOHHJVQ@mail.gmail.com>

I have a bunch of old email archives I'm messing around with, for example,
rendering them on-demand in HTML. Some of the files use quoted printable
content transfer encoding. Here's one (with a number of headers elided):

>From classicrendezvous-admin  Mon Dec  4 15:29:22 2000
Message-ID: <027801bfcdaa$fa175230$268cbcd4 at stewart>
Reply-To: "Stewart Barrie" <sbarrie at bellefield.worldonline.co.uk>
From: "Stewart Barrie" <sbarrie at bellefield.worldonline.co.uk>
To: <Classicrendezvous at bikelist.org>
Date: Sat, 3 Jun 2000 23:28:00 +0100
MIME-Version: 1.0
X-StripMime: Non-text section removed by stripmime
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;charset="iso-8859-1"
Subject: {ClassicRend]Frame dent


Just inherited a nice 531 Claud Butler from the 60's I think. No =
eylelets and pencil stays. Unfortunately there is nice dent in the =
middle of the seat tube just above front mech clip height. Otherwise it = seems
fine.=20


Any thoughts on restoration? Are there any techniques for fixing dents =
in situ (I can hear the screams already) or are we definitely looking at =
a new seat tube? In which case, is it worth it?


Cheers



Stewart


--- StripMime Report -- processed MIME parts ---
multipart/alternative
  text/plain (text body -- kept)
  text/html
---


This message is stored in a file. I read it using email.message_from_file.
In the example above, the payload still retains the quoted printable bits.
Maybe it's not correctly encoded (no "=20" at the end of the second
paragraph, for example), but I would have thought the email package would
do what it could to decode things, but nope:

>>> print(msg.get_payload())
Just inherited a nice 531 Claud Butler from the 60's I think. No =
eylelets and pencil stays. Unfortunately there is nice dent in the =
middle of the seat tube just above front mech clip height. Otherwise it =
seems fine.=20


Any thoughts on restoration? Are there any techniques for fixing dents =
in situ (I can hear the screams already) or are we definitely looking at =
a new seat tube? In which case, is it worth it?


Cheers



Stewart


--- StripMime Report -- processed MIME parts ---
multipart/alternative

  text/plain (text body -- kept)
  text/html
---


Am I expecting too much from the email package when munching on crufty
20+yo archived email messages?

Thx,

Skip

From mats at wichmann.us  Thu Dec 23 17:41:05 2021
From: mats at wichmann.us (Mats Wichmann)
Date: Thu, 23 Dec 2021 15:41:05 -0700
Subject: Can Python call and use FME modules and functions such as
 StreamOrderCalculator?
In-Reply-To: <CA+i5JwYUjxJT_iResxvcC+9dUkdzKq+zKkA3ijBTKOm=75iXcA@mail.gmail.com>
References: <CA+i5JwYUjxJT_iResxvcC+9dUkdzKq+zKkA3ijBTKOm=75iXcA@mail.gmail.com>
Message-ID: <2dec65dc-15c8-ab4c-987d-9aabab0fe7e6@wichmann.us>

On 12/23/21 09:38, Shaozhong SHI wrote:
> Can we do something like import an fme.something and make use of FME
> modules and functions?

And what, pray tell, is FME?



From cs at cskk.id.au  Thu Dec 23 17:55:14 2021
From: cs at cskk.id.au (Cameron Simpson)
Date: Fri, 24 Dec 2021 09:55:14 +1100
Subject: email.message_from_file & quoted printable
In-Reply-To: <CANc-5UyPuqb3vN5czu5RE=wb1jm=voah6yg89U1qkWfGOHHJVQ@mail.gmail.com>
References: <CANc-5UyPuqb3vN5czu5RE=wb1jm=voah6yg89U1qkWfGOHHJVQ@mail.gmail.com>
Message-ID: <YcT+Unu8ytexrRiN@cskk.homeip.net>

On 23Dec2021 14:50, Skip Montanaro <skip.montanaro at gmail.com> wrote:
>I have a bunch of old email archives I'm messing around with, for example,
>rendering them on-demand in HTML. Some of the files use quoted printable
>content transfer encoding. Here's one (with a number of headers elided):
[...]
>This message is stored in a file. I read it using 
>email.message_from_file.
>In the example above, the payload still retains the quoted printable bits.
>Maybe it's not correctly encoded (no "=20" at the end of the second
>paragraph, for example)

That seems legit, as it is a space in the message text.

>, but I would have thought the email package would
>do what it could to decode things, but nope:
>
>>>> print(msg.get_payload())

From the docs:

 get_payload(i=None, decode=False)

    Return the current payload, which will be a list of Message
    objects when is_multipart() is True, or a string when is_multipart()
    is False. [...]
    Optional decode is a flag indicating whether the payload should
    be decoded or not, according to the Content-Transfer-Encoding
    header. When True and the message is not a multipart, the payload
    will be decoded if this header?s value is quoted-printable or
    base64.

Try decode=True.

Cheers,
Cameron Simpson <cs at cskk.id.au>

From learn2program at gmail.com  Thu Dec 23 16:24:59 2021
From: learn2program at gmail.com (Alan Gauld)
Date: Thu, 23 Dec 2021 21:24:59 +0000
Subject: Advanced ways to get object information from within python
In-Reply-To: <CAEsMKX0eOPONpQSjsRG_-eRFRUD+Md5=wrBEGAD2c7f8=Z590w@mail.gmail.com>
References: <CAEsMKX0eOPONpQSjsRG_-eRFRUD+Md5=wrBEGAD2c7f8=Z590w@mail.gmail.com>
Message-ID: <4a64272c-5e80-4c52-e1c1-c146b6f86bb7@yahoo.co.uk>

On 23/12/2021 11:01, Julius Hamilton wrote:
> Lastly, the "help" function.
> 
> I find "help" to similarly be a situation of information overload. 

I assume you know that you can target help() to the specific
attribute or function you need not just the top level classes?

So combined with dir() you can call help on each of the names
that dir() reveals.

That usually produces a much more focused form of documentation.

So in your example:

> dir(scrapy) shows this:

>['Field', 'FormRequest', 'Item', 'Request', 'Selector', 'Spider',
'__all__', '__builtins__', '__cached__', '__doc__', '__file__',
'__loader__', '__name__', '__package__', '__path__', '__spec__',
'__version__', '_txv', 'exceptions', 'http', 'item', 'link',
'linkextractors', 'selector', 'signals', 'spiders', 'twisted_version',
'utils', 'version_info']

> I wish there was a convenient way for me to know what
> all of these are.

help(scrapy.http)
help(scrapy.spiders)
etc...

And if it turns out they are not functions or classes
you can use [p]print to get the values. You can also
use type() to clarify what kind of thing an attribute is.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


From skip.montanaro at gmail.com  Thu Dec 23 20:25:32 2021
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Thu, 23 Dec 2021 19:25:32 -0600
Subject: email.message_from_file & quoted printable
In-Reply-To: <YcT+Unu8ytexrRiN@cskk.homeip.net>
References: <CANc-5UyPuqb3vN5czu5RE=wb1jm=voah6yg89U1qkWfGOHHJVQ@mail.gmail.com>
 <YcT+Unu8ytexrRiN@cskk.homeip.net>
Message-ID: <CANc-5Uz13jppAn2gb8q7miuH2bLvJ5nEYj79qvyNWnACqTBRrg@mail.gmail.com>

>
> From the docs:
>
>  get_payload(i=None, decode=False)


...

Try decode=True.


:dopeslap: Thanks. Never been all that consistent reading documentation.

Skip

From arulvani66 at gmail.com  Fri Dec 24 09:22:58 2021
From: arulvani66 at gmail.com (vani arul)
Date: Fri, 24 Dec 2021 19:52:58 +0530
Subject: Selection sort
Message-ID: <CAHVaEHhaaQ+b7xES9Ktv3f3DdJcaM2Nkb5c4k45QGR-DdPi_hA@mail.gmail.com>

Hello,
I am trying write a code.Can some help me find the error in my code.
Thanks!


def selectionsort(arr):
   # le=len(arr)
    for b in range(0,len(arr)-1):
        pos=b
        for a in range(b+1,len(arr)-1):
            if arr[b]>arr[a+1]:
                arr[b],arr[a+1]=arr[a+1],arr[b]
    return arr

arr=[3,5,9,8,2,6]
print(selectionsort(arr))

From barry at barrys-emacs.org  Fri Dec 24 09:39:42 2021
From: barry at barrys-emacs.org (Barry Scott)
Date: Fri, 24 Dec 2021 14:39:42 +0000
Subject: Selection sort
In-Reply-To: <CAHVaEHhaaQ+b7xES9Ktv3f3DdJcaM2Nkb5c4k45QGR-DdPi_hA@mail.gmail.com>
References: <CAHVaEHhaaQ+b7xES9Ktv3f3DdJcaM2Nkb5c4k45QGR-DdPi_hA@mail.gmail.com>
Message-ID: <A14220C6-124B-437C-B730-D9919411862F@barrys-emacs.org>



> On 24 Dec 2021, at 14:22, vani arul <arulvani66 at gmail.com> wrote:
> 
> Hello,
> I am trying write a code.Can some help me find the error in my code.
> Thanks!
> 
> 
> def selectionsort(arr):
>   # le=len(arr)
>    for b in range(0,len(arr)-1):
>        pos=b
>        for a in range(b+1,len(arr)-1):
>            if arr[b]>arr[a+1]:
>                arr[b],arr[a+1]=arr[a+1],arr[b]
>    return arr
> 
> arr=[3,5,9,8,2,6]
> print(selectionsort(arr))

What are you expecting the code to do?

Barry



> --

> 
> https://mail.python.org/mailman/listinfo/python-list
> 


From mats at wichmann.us  Fri Dec 24 09:49:45 2021
From: mats at wichmann.us (Mats Wichmann)
Date: Fri, 24 Dec 2021 07:49:45 -0700
Subject: Selection sort
In-Reply-To: <CAHVaEHhaaQ+b7xES9Ktv3f3DdJcaM2Nkb5c4k45QGR-DdPi_hA@mail.gmail.com>
References: <CAHVaEHhaaQ+b7xES9Ktv3f3DdJcaM2Nkb5c4k45QGR-DdPi_hA@mail.gmail.com>
Message-ID: <3aae0460-3f27-2672-8551-a80df8528568@wichmann.us>

On 12/24/21 07:22, vani arul wrote:
> Hello,
> I am trying write a code.Can some help me find the error in my code.
> Thanks!
> 
> 
> def selectionsort(arr):
>    # le=len(arr)
>     for b in range(0,len(arr)-1):
>         pos=b
>         for a in range(b+1,len(arr)-1):
>             if arr[b]>arr[a+1]:
>                 arr[b],arr[a+1]=arr[a+1],arr[b]
>     return arr
> 
> arr=[3,5,9,8,2,6]
> print(selectionsort(arr))
> 

Hint: what are you using 'pos' for?  A placeholder for (something) has
an actual purpose in the typical selection-sort, but you only initialize
it and never use or update it.


From skip.montanaro at gmail.com  Fri Dec 24 10:31:58 2021
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Fri, 24 Dec 2021 09:31:58 -0600
Subject: email.message_from_file & quoted printable
In-Reply-To: <CANc-5Uz13jppAn2gb8q7miuH2bLvJ5nEYj79qvyNWnACqTBRrg@mail.gmail.com>
References: <CANc-5UyPuqb3vN5czu5RE=wb1jm=voah6yg89U1qkWfGOHHJVQ@mail.gmail.com>
 <YcT+Unu8ytexrRiN@cskk.homeip.net>
 <CANc-5Uz13jppAn2gb8q7miuH2bLvJ5nEYj79qvyNWnACqTBRrg@mail.gmail.com>
Message-ID: <CANc-5Ux=GODZmx-MyyO9a6UyfqjWv9FnSXWf5qiabZW8enxj5A@mail.gmail.com>

Cameron> Try decode=True.

Skip> :dopeslap: Thanks. Never been all that consistent reading documentation.

The more I think about it, the more I think maybe my lack of
documentation reading wasn't all that unreasonable. The content
transfer encoding and charset are properties of the message as a
whole. Iexpected the internal form of the message body to be already
decoded. Am I missing some key bit of perspective here?

Skip

From cl at isbd.net  Fri Dec 24 11:03:17 2021
From: cl at isbd.net (Chris Green)
Date: Fri, 24 Dec 2021 16:03:17 +0000
Subject: Is there any way to make sense of these E-Mail subjects?
Message-ID: <5j7h9i-uvpu1.ln1@esprimo.zbmc.eu>

I have a Python 3 script which processes E-Mail caught in my hosting
provider's 'catchall' mailbox.  It looks for things that *might* be
useful E-Mails, forwards them, and throws the rest away.

I have a function which, given a header name, extracts the header and
returns it as a string:-

    #
    #
    # Get a message header as a string
    #
    def getHdr(msg, header):
       return str("\n  " + header + ": " + str(msg.get(header, "empty"))) 

msg is a mailbox.mboxMessage object.


This is mostly working as expected, returning the header contents as
strings so I can output them to my log files as necessary.  However
some Subject: lines are being returned like the following:-

      Subject: [SPAM] =?UTF-8?B?8J+TtyBKb2huIEJheHRlci1C?=
     =?UTF-8?B?cm93biByZWNlbnRseSBw?=
     =?UTF-8?B?b3N0ZWQgYSBuZXcgcGhv?=
     =?UTF-8?B?dG8=?=

It looks like some sort of mis-encoding of UTF-8 strings, can anyone
suggest what might be going on and/or a way to get some sense out of
this?

FWIW the above example is from "Facebook" <friendupdates at facebookmail.com>
so while it is probably (as indicated) [SPAM] it shouldn't be so illegible.

At the moment I can't see an easy way to actually inspect the message
as it's disappeared off somewhere else.  I guess I could add some code
to the script to send it to myself as well but if there's something
obvious in the above it would avoid having to do this.


-- 
Chris Green
?

From barry at barrys-emacs.org  Fri Dec 24 13:02:16 2021
From: barry at barrys-emacs.org (Barry)
Date: Fri, 24 Dec 2021 18:02:16 +0000
Subject: Is there any way to make sense of these E-Mail subjects?
In-Reply-To: <5j7h9i-uvpu1.ln1@esprimo.zbmc.eu>
References: <5j7h9i-uvpu1.ln1@esprimo.zbmc.eu>
Message-ID: <7A5D6104-0E9E-4A94-95CE-EFE7A65C6527@barrys-emacs.org>



> On 24 Dec 2021, at 16:40, Chris Green <cl at isbd.net> wrote:
> 
> ?I have a Python 3 script which processes E-Mail caught in my hosting
> provider's 'catchall' mailbox.  It looks for things that *might* be
> useful E-Mails, forwards them, and throws the rest away.
> 
> I have a function which, given a header name, extracts the header and
> returns it as a string:-
> 
>    #
>    #
>    # Get a message header as a string
>    #
>    def getHdr(msg, header):
>       return str("\n  " + header + ": " + str(msg.get(header, "empty"))) 
> 
> msg is a mailbox.mboxMessage object.
> 
> 
> This is mostly working as expected, returning the header contents as
> strings so I can output them to my log files as necessary.  However
> some Subject: lines are being returned like the following:-
> 
>      Subject: [SPAM] =?UTF-8?B?8J+TtyBKb2huIEJheHRlci1C?=
>     =?UTF-8?B?cm93biByZWNlbnRseSBw?=
>     =?UTF-8?B?b3N0ZWQgYSBuZXcgcGhv?=
>     =?UTF-8?B?dG8=?=
> 
> It looks like some sort of mis-encoding of UTF-8 strings, can anyone
> suggest what might be going on and/or a way to get some sense out of
> this?

I think this is correctly encoded.
See https://datatracker.ietf.org/doc/html/rfc1342

Barry
> 
> FWIW the above example is from "Facebook" <friendupdates at facebookmail.com>
> so while it is probably (as indicated) [SPAM] it shouldn't be so illegible.
> 
> At the moment I can't see an easy way to actually inspect the message
> as it's disappeared off somewhere else.  I guess I could add some code
> to the script to send it to myself as well but if there's something
> obvious in the above it would avoid having to do this.
> 
> 
> -- 
> Chris Green
> ?
> -- 
> https://mail.python.org/mailman/listinfo/python-list

From dieter at handshake.de  Fri Dec 24 16:55:20 2021
From: dieter at handshake.de (Dieter Maurer)
Date: Fri, 24 Dec 2021 22:55:20 +0100
Subject: Is there any way to make sense of these E-Mail subjects?
In-Reply-To: <5j7h9i-uvpu1.ln1@esprimo.zbmc.eu>
References: <5j7h9i-uvpu1.ln1@esprimo.zbmc.eu>
Message-ID: <25030.16840.690955.479138@ixdm.fritz.box>

Chris Green wrote at 2021-12-24 16:03 +0000:
>I have a Python 3 script which processes E-Mail caught in my hosting
>provider's 'catchall' mailbox.  It looks for things that *might* be
>useful E-Mails, forwards them, and throws the rest away.
> ...
>I have a function which, given a header name, extracts the header and
>returns it as a string:-
>
>    #
>    #
>    # Get a message header as a string
>    #
>    def getHdr(msg, header):
>       return str("\n  " + header + ": " + str(msg.get(header, "empty")))
>
>msg is a mailbox.mboxMessage object.
>
>
>This is mostly working as expected, returning the header contents as
>strings so I can output them to my log files as necessary.  However
>some Subject: lines are being returned like the following:-
>
>      Subject: [SPAM] =?UTF-8?B?8J+TtyBKb2huIEJheHRlci1C?=
...

Email headers follow the MIME standard and it requires that
they use ASCII only characters.
Thus, if the header content should contain non ASCII characters
some form of encoding becomees necessary.

The `=?UTF-8?B?8J+TtyBKb2huIEJheHRlci1C?=` is an encoded word.

An encoded word has the form `=?<charset>?<code type>?<encoding>?=`
<code type> is either `B` (base 64 encoding) or
`Q` (quoted printable encoding).
You decode an encoded word by decoding <encoding> with <code type>
and then interpret the resulting byte sequence in <charset>.

The `email` package contains functions to do this encoding.


I would likely serialize your `mboxMessage` as a string and
then use an `email` function to turn this string into an
`email` `Message` object. Those messages can return headers
in a decoded form.

From cs at cskk.id.au  Fri Dec 24 17:41:24 2021
From: cs at cskk.id.au (Cameron Simpson)
Date: Sat, 25 Dec 2021 09:41:24 +1100
Subject: email.message_from_file & quoted printable
In-Reply-To: <CANc-5Ux=GODZmx-MyyO9a6UyfqjWv9FnSXWf5qiabZW8enxj5A@mail.gmail.com>
References: <CANc-5Ux=GODZmx-MyyO9a6UyfqjWv9FnSXWf5qiabZW8enxj5A@mail.gmail.com>
Message-ID: <YcZMlFqMU6xqFzvh@cskk.homeip.net>

On 24Dec2021 09:31, Skip Montanaro <skip.montanaro at gmail.com> wrote:
>Cameron> Try decode=True.
>
>Skip> :dopeslap: Thanks. Never been all that consistent reading documentation.
>
>The more I think about it, the more I think maybe my lack of
>documentation reading wasn't all that unreasonable. The content
>transfer encoding and charset are properties of the message as a
>whole. Iexpected the internal form of the message body to be already
>decoded. Am I missing some key bit of perspective here?

That sometimes you want the raw content as it was in the file? That if 
you have that choice (decoded or raw) the default should be False, as it 
does less?

Aside: a personal design criterion of mine is that default values should 
be False or zero or empty - leaving off an optional parameter _should_ 
be like setting it to "falseish". I'm prepared to reach for a thesaurus 
to achieve that goal.

Returning to the issue: you had a method that gives you the main 
payload. It seemed to be raw. You wanted the decoded form, and thought 
that'd be a pretty common thing to want. So it is well worth a look at 
the function docs to see if that entirely reasonable thing was 
convenient.

That we my _first_ instinct - I've never used get_payload(), myself, and 
I've written my own complete mailfiler :-)

Really, if a function isn't doing what you want you should always read 
the docs for that function specificly, even if you don't want to read 
the entire module docs (which is a big ask).

Cheers,
Cameron Simpson <cs at cskk.id.au>

From PythonList at DancesWithMice.info  Fri Dec 24 18:43:32 2021
From: PythonList at DancesWithMice.info (dn)
Date: Sat, 25 Dec 2021 12:43:32 +1300
Subject: Selection sort
In-Reply-To: <CAHVaEHhaaQ+b7xES9Ktv3f3DdJcaM2Nkb5c4k45QGR-DdPi_hA@mail.gmail.com>
References: <CAHVaEHhaaQ+b7xES9Ktv3f3DdJcaM2Nkb5c4k45QGR-DdPi_hA@mail.gmail.com>
Message-ID: <5aeaea49-80b9-740c-3efc-e7640719039a@DancesWithMice.info>

On 25/12/2021 03.22, vani arul wrote:
> Hello,
> I am trying write a code.Can some help me find the error in my code.
> Thanks!
> 
> 
> def selectionsort(arr):
>    # le=len(arr)
>     for b in range(0,len(arr)-1):
>         pos=b
>         for a in range(b+1,len(arr)-1):
>             if arr[b]>arr[a+1]:
>                 arr[b],arr[a+1]=arr[a+1],arr[b]
>     return arr
> 
> arr=[3,5,9,8,2,6]
> print(selectionsort(arr))


This looks like a typical 'homework question'. Accordingly, we are happy
to help, but that means helping you to learn Python, NOT to help by
writing a homework-solution!


Evidently you are expected to learn some Python, but additionally an
algorithm (or method) for sorting. Also apparent, is that the current
attempt has been taken from another programming language - rather than
using a more 'pythonic' idiom. All fine, but with a "but...".

One assumes, the Python sort() built-in function may not be used. Even
so, you can ask the computer to test the success (or otherwise) of your
efforts, by adding to the end of the existing code, something like:

print( "Was the algorithm successful?",
    selection_sort( arr ) == sorted( arr )
    )

(I've recently 'enjoyed' an eye operation, so actively seek ways of
having the computer 'see' or check things that are difficult for me -
pending already-excellent recovery progress...)


The question has already been asked: "What are you expecting the code to
do?". This is vital. What is your target? (and when you ask us a
question, how do we know exactly what your target might be?).

Regardless, it is *always* the first question to be asked in attempting
to find a solution to any problem - thus, has much wider application
than computing then! How does one know if the aim is true, without first
defining "the target"?


The key to coding any algorithm is to (temporarily) forget Python, and
to first understand the method - how it works.

What I have often done in the past, is to ask trainees to bring a
pack/deck of Playing Cards - however a few scraps of paper with the
numbers: 3,5,9,8,2, and 6 written on them, will do just as well.

What we want is three sets of the same numbers. One set to move around
(the above), and the other two static - so could write them (in a row)
across a sheet of paper. The first 'static' set, should be the numbers
in the sequence given by the question, ie 'the start position'. Put this
at the 'top' of a desk or flat surface. The second 'static' set, is the
"target", ie the same digits, but in sorted-order. Artie them across a
second piece of paper, and place that at the 'bottom' of the desk. So,
now we can see a starting-position, and a final sequence.

The 'magic' is the bit that happens "in the middle" ("in-between")!

Now, using the space on your desk, between the 'start' and 'end',
arrange the 'numbers' in the given sequence, across a 'row'.

With your brain (and hands) as the 'computer', work the algorithm you've
already started using:

1	commence with the first item in the list:
	ie take the "3" in your left hand
	(or perhaps put a suitable finger on it)
2	then compare that with the "5" (right hand/finger)
3	given that three is less than five, what should happen?
4	next compare the "3" with the "9"
	(left finger doesn't move, right hand/finger does!)
5	what should happen at this time?
6	keep going, until...
	you compare the "3" with the "2"
	what happens now?

That's the basic algorithm done!

Continuing the above, after a while there are no more items left to
compare with the one under your left hand/finger. What is the state of
the list?

Not much has changed, but what can we say about the situation
(placement) of the item which is in the first-position/at the left side
of the scraps of paper/playing cards? How does it relate to the
first-position in the target-row?

For the next stage, start with the second item in the list (move your
left hand/pointing-finger!):
1	run through, similar to the above
2	until you (your right hand) reach the end
	(yes, it is starting to become a bit boring, but...)

When you have once-again 'run out' of items to compare, what can now be
said about the state of the list? About the values in the first two
positions (and their equivalents in the 'target')?

So, change of thinking: could it be considered that we have two
'sub-lists' (amongst our paper-scraps/playing-cards)? One with two items
in it, the other containing 'all the others'. What can we say about the
'left-sub-list' (and specifically its sequencing)?


So, at this time, please decide if you need to continue with the above
cycles in order to fully-understand how the method arrives at the
desired conclusion - or keep going to prove 'everything' to your
satisfaction, by running right-through. Has the algorithm managed to
arrange the slips of paper/playing cards into the correct sequence, as
portrayed by the bottom-row of numbers?


Once you have confidence in the algorithm, it can be implemented in
code. (but not before! Don't be in too much of a hurry!) So, let's think
about how to represent the algorithm *and* the data, as program-code.

When sorting, items each can be described in two ways. The first item
(in the sample data provided) has the *value* "3". That same item, is in
the first *position* of the list.

Alternately, we can say/think, that the first item in the sample-data
occupies the first position in the list, and has the value "3".

NB I've written the value "3" as inside quotation-marks for
emphasis/identification - it may or may not be a Python-string because
(the sample list) will sort correctly either way, but it appears to be
of integer type, so the quotation-marks are just a way of writing and
emphasis, OK?)


Consider this concept of the two ways to look at each item in the list,
from the perspective of the algorithm/method:-

The *value* of each item will never change (during the sort). We are not
going to increase each value by 10%, for example - we're not giving
employees a salary-raise, are we?

However, the *position* of items may change - because the objective of
sorting is to arrange the items into some definition of sequence (and we
kind-of assume that 'change' is required, because the (sample) list does
not start-out as being in-order).


Accordingly, one of the challenges is to keep separate in our minds, an
item's value, and its position in the list - even though the two
describe the same item!

NB the English word "position", is usually called an "index" or even an
"offset" in Computer Science - ie the item's relative-position in
relation to the beginning of the list. In Python, lists, or rather their
indexes/indices, are zero-based - which means that the first item is
"addressed" as having an index of "0" - ie in Python:

	list_name[ 0 ]


Thinking back to our 'playing with paper/cards' above, what is the
purpose of the outer loop? Considering "targets" again: by the end of
each cycle through the loop, what will have been achieved?

Similarly, that of the inner loop? During a single cycle of the
outer-loop, what is the objective of the complete run through the
inner-loop?


When you answer these two questions, how are you expressing the problem
in terms of each item's "position" - and its "value"?


It turns-out, we only need to look any item's value or items' values
within the if-condition (and no-where else in the code). Whereas, we use
indexing to control both of the for-loops (and later, if an exchange of
position is necessary).


So, now (finally!) we can build a 'skeleton' of code, taking it (an
algorithmic) step-by-step to add, or 'flesh-out', more detail gradually:

1
code an outer-loop to traverse the list. With nothing else 'inside' it,
print the index once for each execution of the loop, ie execution should
show you "0, 1, 2, etc" (except each on its own line, not
comma-separated - unless you know how to do that in Python). Ensure the
indexing starts and stops correctly - not too early, nor too late, in
terms of the length of the list - watch-out for that "zero-based"
consideration mentioned earlier, and ensure that last item is not
'visited'! (why not?)

2
add code for the inner-loop. Remembering the observation (above) that
this should only traverse the right-hand sub-list (but should this one
go all the way to 'the end'?). Insert another print statement and
observe the behavior of this (second) index. Ensure that it starts, and
stops, correctly (wrt to the length of the *sub*-list)?

3
Now the program(me) features two indexes/indices. Thinking back to the
algorithm, to what does the first (outer-loop) index 'point'? Similarly,
to what does the second (inner-loop) index point? With those two
thoughts in-mind, does this make it easier to decide which two values to
compare when it comes to writing the if-condition?

NB you could pause 'development' at this point, and use this step in the
process to check that the if-condition is correct (employ one or two
judiciously-placed debug-print statements).

Thereafter, do the (aforementioned) two indexes/indices also make it
easy to decide on the positions of the two items that (may) need to be
'swapped'?

At this point, code the entire if-suite, and (if you haven't already) it
may be helpful to add another debug-print (or two), to confirm which
indexes and values the code is comparing, and which values and indexes
are being inter-changed (but only should such action be necessary).


Hopefully, you have now figured-out both the algorithm, and the code to
implement same. Does it work? Is the list properly sorted by the end?

If not, come back to us with a more detailed question (plus code and
sample output).

If you're successful, also revert (with pride), and we'll try showing
you another way to implement the same algorithm - but in a manner
many?most other programming languages cannot. In other words, a
'pythonic' solution...


Apologies - I've taken several sessions to write this msg, with breaks
to rest my eyes in-between. Now I've clicked the wrong button on the
Spelling-Checker, but can't figure-out what I did/didn't correct, or
correct correctly. Confused? So am I. Please be understanding for this
and any other typos or proof-reading failings...
-- 
Regards,
=dn

From luca.anzilli at gmail.com  Fri Dec 24 12:01:41 2021
From: luca.anzilli at gmail.com (Luca Anzilli)
Date: Fri, 24 Dec 2021 18:01:41 +0100
Subject: Selection sort
In-Reply-To: <3aae0460-3f27-2672-8551-a80df8528568@wichmann.us>
References: <CAHVaEHhaaQ+b7xES9Ktv3f3DdJcaM2Nkb5c4k45QGR-DdPi_hA@mail.gmail.com>
 <3aae0460-3f27-2672-8551-a80df8528568@wichmann.us>
Message-ID: <CAMgw8HwaG0OH05GUsV2ZoX99cHjxjEfKXzZcATdujeLg343r0Q@mail.gmail.com>

Hello

try this code

def selectionsort(arr):
#le=len(arr)
    for b in range(0,len(arr)-1):
       # pos=b
        for a in range(b+1,len(arr)):
            if arr[b]>arr[a]:
                arr[b],arr[a]=arr[a],arr[b]
    return arr

arr=[3,5,9,8,2,6]
print(selectionsort(arr))

Il giorno ven 24 dic 2021 alle ore 15:50 Mats Wichmann <mats at wichmann.us>
ha scritto:

> On 12/24/21 07:22, vani arul wrote:
> > Hello,
> > I am trying write a code.Can some help me find the error in my code.
> > Thanks!
> >
> >
> > def selectionsort(arr):
> >    # le=len(arr)
> >     for b in range(0,len(arr)-1):
> >         pos=b
> >         for a in range(b+1,len(arr)-1):
> >             if arr[b]>arr[a+1]:
> >                 arr[b],arr[a+1]=arr[a+1],arr[b]
> >     return arr
> >
> > arr=[3,5,9,8,2,6]
> > print(selectionsort(arr))
> >
>
> Hint: what are you using 'pos' for?  A placeholder for (something) has
> an actual purpose in the typical selection-sort, but you only initialize
> it and never use or update it.
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

From python at example.invalid  Fri Dec 24 11:41:00 2021
From: python at example.invalid (Python)
Date: Fri, 24 Dec 2021 17:41:00 +0100
Subject: Is there any way to make sense of these E-Mail subjects?
References: <5j7h9i-uvpu1.ln1@esprimo.zbmc.eu>
Message-ID: <sq4t6p$sd8$1@gioia.aioe.org>

Chris Green wrote:
>        Subject: [SPAM] =?UTF-8?B?8J+TtyBKb2huIEJheHRlci1C?=
>       =?UTF-8?B?cm93biByZWNlbnRseSBw?=
>       =?UTF-8?B?b3N0ZWQgYSBuZXcgcGhv?=
>       =?UTF-8?B?dG8=?=
> 
> It looks like some sort of mis-encoding of UTF-8 strings, can anyone
> suggest what might be going on and/or a way to get some sense out of
> this?

It's not mis-encoding, it the standard way to embed non-ascii
characters in the header. ?UTF-8?B? means base64 encoded utf-8,
then the content is on multiple lines:

$ base64 -d < t
? John Baxter-Brown recently posted a new photo

email.header.decode_header should be able to handle it.


From cl at isbd.net  Fri Dec 24 12:18:06 2021
From: cl at isbd.net (Chris Green)
Date: Fri, 24 Dec 2021 17:18:06 +0000
Subject: Is there any way to make sense of these E-Mail subjects?
References: <5j7h9i-uvpu1.ln1@esprimo.zbmc.eu> <sq4t6p$sd8$1@gioia.aioe.org>
Message-ID: <evbh9i-ul3v1.ln1@esprimo.zbmc.eu>

Python <python at example.invalid> wrote:
> Chris Green wrote:
> >        Subject: [SPAM] =?UTF-8?B?8J+TtyBKb2huIEJheHRlci1C?=
> >       =?UTF-8?B?cm93biByZWNlbnRseSBw?=
> >       =?UTF-8?B?b3N0ZWQgYSBuZXcgcGhv?=
> >       =?UTF-8?B?dG8=?=
> > 
> > It looks like some sort of mis-encoding of UTF-8 strings, can anyone
> > suggest what might be going on and/or a way to get some sense out of
> > this?
> 
> It's not mis-encoding, it the standard way to embed non-ascii
> characters in the header. ?UTF-8?B? means base64 encoded utf-8,
> then the content is on multiple lines:
> 
> $ base64 -d < t
> ? John Baxter-Brown recently posted a new photo
> 
> email.header.decode_header should be able to handle it.
> 
Brilliant, thank you, just what I needed to know.

-- 
Chris Green
?

From hjp-python at hjp.at  Sat Dec 25 14:09:34 2021
From: hjp-python at hjp.at (Peter J. Holzer)
Date: Sat, 25 Dec 2021 20:09:34 +0100
Subject: Isn't TypeError built in?
In-Reply-To: <b4597886-e7f6-22f3-5605-67ed34e60c5f@dewhirst.com.au>
References: <b4597886-e7f6-22f3-5605-67ed34e60c5f@dewhirst.com.au>
Message-ID: <20211225190934.cjq7zrbe52nkflms@hjp.at>

On 2021-12-13 12:22:28 +1100, Mike Dewhirst via Python-list wrote:
> Obviously something is wrong elsewhere but I'm not sure where to look.
> Ubuntu 20.04 with plenty of RAM.
[...]
> [Mon Dec 13 01:15:49.885659 2021] [mpm_event:notice] [pid 1033:tid
> 140446449658944] AH00489: Apache/2.4.41 (Ubuntu) SVN/1.13.0 OpenSSL/1.1.1f
> mod_wsgi/4.6.8 Python/3.8 configured -- resuming normal operations
> [Mon Dec 13 01:15:49.885664 2021] [core:notice] [pid 1033:tid
> 140446449658944] AH00094: Command line: '/usr/sbin/apache2'
> Exception ignored in: <function Local.__del__ at 0x7fbc380964c0>
> Traceback (most recent call last):
> ? File "/usr/local/lib/python3.8/dist-packages/asgiref/local.py", line 96,
> in __del__
> NameError: name 'TypeError' is not defined

Did you upgrade the OS recently and do you use virtual environents with
mod_wsgi?

"Impossible" errors like the above are common when the virtual
environment was built with a different (older) version of the Python
interpreter than the one trying to use it. Nuking and rebuilding the
virtual environment is usually the quickest way to fix it.

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp at hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://mail.python.org/pipermail/python-list/attachments/20211225/2b2eaae8/attachment.sig>

From skip.montanaro at gmail.com  Sat Dec 25 17:23:42 2021
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Sat, 25 Dec 2021 16:23:42 -0600
Subject: email.message_from_file & quoted printable
In-Reply-To: <YcZMlFqMU6xqFzvh@cskk.homeip.net>
References: <CANc-5Ux=GODZmx-MyyO9a6UyfqjWv9FnSXWf5qiabZW8enxj5A@mail.gmail.com>
 <YcZMlFqMU6xqFzvh@cskk.homeip.net>
Message-ID: <CANc-5UyKPRqQFP9FVsqe7kFi1_X4hx2ZG8AprFmjpCLx7wX8DQ@mail.gmail.com>

> That sometimes you want the raw content as it was in the file? That if
> you have that choice (decoded or raw) the default should be False, as it
> does less?

Yeah, thinking back later, the round-trip possibility seemed plausible.
I'll stop overthinking this now...

> That we my _first_ instinct - I've never used get_payload(), myself, and
> I've written my own complete mailfiler :-)

I'm only trying to display email through a web browser, possibly with a
link to the raw message.


> Really, if a function isn't doing what you want you should always read
> the docs for that function specificly, even if you don't want to read
> the entire module docs (which is a big ask).

Yeah, my mental model was simply off. I frequently pull up docs in my
browser. I was thinking the message contents would be normalized when
reading. It didn't occur to me the transformation would occur on the way
out.

I think another way I might have saved myself was if I was using a modern
IDE where I might have gotten some hints hovering over the method call.
But, I'm an End user. I'm sure there is some add-on package I could
install, but I haven't looked.

Skip

From miked at dewhirst.com.au  Sat Dec 25 18:18:29 2021
From: miked at dewhirst.com.au (Mike Dewhirst)
Date: Sun, 26 Dec 2021 10:18:29 +1100
Subject: Isn't TypeError built in?
In-Reply-To: <20211225190934.cjq7zrbe52nkflms@hjp.at>
Message-ID: <4JM0QH4ysFznVDX@mail.python.org>

Thanks Peter ... the problem was all mine.Can't quite remember what it was exactly but I introduced a bug in my own code and it manifested down there in the bowels of the snake.I suppose that indicates a possible kink to be ironed out and if that interests you I am happy to retrace my steps and describe it.Fyi, there was nothing out of kilter in the machine. It was in production and is stable.CheersMike--(Unsigned mail from my phone)
-------- Original message --------From: "Peter J. Holzer" <hjp-python at hjp.at> Date: 26/12/21  06:19  (GMT+10:00) To: python-list at python.org Subject: Re: Isn't TypeError built in? On 2021-12-13 12:22:28 +1100, Mike Dewhirst via Python-list wrote:> Obviously something is wrong elsewhere but I'm not sure where to look.> Ubuntu 20.04 with plenty of RAM.[...]> [Mon Dec 13 01:15:49.885659 2021] [mpm_event:notice] [pid 1033:tid> 140446449658944] AH00489: Apache/2.4.41 (Ubuntu) SVN/1.13.0 OpenSSL/1.1.1f> mod_wsgi/4.6.8 Python/3.8 configured -- resuming normal operations> [Mon Dec 13 01:15:49.885664 2021] [core:notice] [pid 1033:tid> 140446449658944] AH00094: Command line: '/usr/sbin/apache2'> Exception ignored in: <function Local.__del__ at 0x7fbc380964c0>> Traceback (most recent call last):> ? File "/usr/local/lib/python3.8/dist-packages/asgiref/local.py", line 96,> in __del__> NameError: name 'TypeError' is not definedDid you upgrade the OS recently and do you use virtual environents withmod_wsgi?"Impossible" errors like the above are common when the virtualenvironment was built with a different (older) version of the Pythoninterpreter than the one trying to use it. Nuking and rebuilding thevirtual environment is usually the quickest way to fix it.??????? hp-- ?? _? | Peter J. Holzer??? | Story must make more sense than reality.|_|_) |??????????????????? || |?? | hjp at hjp.at???????? |??? -- Charles Stross, "Creative writing__/?? | http://www.hjp.at/ |?????? challenge!"-- https://mail.python.org/mailman/listinfo/python-list

From cowsmile1911 at gmail.com  Sat Dec 25 01:13:10 2021
From: cowsmile1911 at gmail.com (=?utf-8?Q?C=C3=B4ng_Huy?=)
Date: Sat, 25 Dec 2021 13:13:10 +0700
Subject: Uninstall old python version(Python 3.9.7) 
Message-ID: <50F0D6C7-86F4-4C48-902A-618B1BE98B5E@hxcore.ol>

   I had uninstalled Python 3.9.7 before but it wasn't uninstalled
   completely. I found it still in my computer and when I click "uninstall",
   it sent me a board "No Python 3.9 installation was detected". Of course,
   it won't be an issue if it defaulted my main Python version on my computer
   and I can't use some libraries I installed like numpy, matplotlib,
   networkx,... To solve my python homework. I found some help on the
   internet but it wasn't improved. I don't know how to fix it. Please help
   me.



   Thanks and Regards,

   Cong Huy



   Sent from [1]Mail for Windows



References

   Visible links
   1. https://go.microsoft.com/fwlink/?LinkId=550986

From cowsmile1911 at gmail.com  Sat Dec 25 01:19:07 2021
From: cowsmile1911 at gmail.com (=?utf-8?Q?C=C3=B4ng_Huy?=)
Date: Sat, 25 Dec 2021 13:19:07 +0700
Subject: Uninstall old python version(Python 3.9.7)
Message-ID: <9B3C4D12-1F46-4192-9E57-3CE14CCA54DF@hxcore.ol>

   I uninstalled Python 3.9.7 but I can't delete completely. And I received a
   notification "no Python 3.9 was detected". Of course, it won't be an issue
   if it defaulted my python version on my computer and it makes me can't use
   some libraries I installed before like numpy, matplotlib and network to
   solve my homework. I don't know how to fix it. Help me, please.



   Sent from [1]Mail for Windows



References

   Visible links
   1. https://go.microsoft.com/fwlink/?LinkId=550986

From redstone-cold at 163.com  Sat Dec 25 07:01:07 2021
From: redstone-cold at 163.com (iMath)
Date: Sat, 25 Dec 2021 04:01:07 -0800 (PST)
Subject: recover pickled data: pickle data was truncated
Message-ID: <62d5de12-5e74-46f1-8d31-ea6b879dcb68n@googlegroups.com>

Normally, the shelve data should be read and write by only one process at a time, but unfortunately it was simultaneously read and write by two processes, thus corrupted it. Is there any way to recover all data in it ? Currently I just get "pickle data was truncated" exception after reading a portion of the data?

Data and code here :https://drive.google.com/file/d/137nJFc1TvOge88EjzhnFX9bXg6vd0RYQ/view?usp=sharing

From Marco.Sulla.Python at gmail.com  Sun Dec 26 08:44:26 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Sun, 26 Dec 2021 14:44:26 +0100
Subject: recover pickled data: pickle data was truncated
In-Reply-To: <62d5de12-5e74-46f1-8d31-ea6b879dcb68n@googlegroups.com>
References: <62d5de12-5e74-46f1-8d31-ea6b879dcb68n@googlegroups.com>
Message-ID: <CABbU2U993rM06sVHSqnU31wZDfBc5d9C0g7dotBb2vht8KNqdA@mail.gmail.com>

Use a semaphore.

On Sun, 26 Dec 2021 at 03:30, iMath <redstone-cold at 163.com> wrote:
>
> Normally, the shelve data should be read and write by only one process at a time, but unfortunately it was simultaneously read and write by two processes, thus corrupted it. Is there any way to recover all data in it ? Currently I just get "pickle data was truncated" exception after reading a portion of the data?
>
> Data and code here :https://drive.google.com/file/d/137nJFc1TvOge88EjzhnFX9bXg6vd0RYQ/view?usp=sharing
> --
> https://mail.python.org/mailman/listinfo/python-list

From Marco.Sulla.Python at gmail.com  Sun Dec 26 08:48:21 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Sun, 26 Dec 2021 14:48:21 +0100
Subject: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
Message-ID: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>

I have to use _PyObject_GC_IS_TRACKED(). It can't be used unless you
define Py_BUILD_CORE. I want to avoid this. What macro or function can
substitute it?

From barry at barrys-emacs.org  Sun Dec 26 11:28:55 2021
From: barry at barrys-emacs.org (Barry Scott)
Date: Sun, 26 Dec 2021 16:28:55 +0000
Subject: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
In-Reply-To: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>
References: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>
Message-ID: <CC507B7F-0A6C-415A-952E-2C7CDC75C9EC@barrys-emacs.org>



> On 26 Dec 2021, at 13:48, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
> 
> I have to use _PyObject_GC_IS_TRACKED(). It can't be used unless you
> define Py_BUILD_CORE. I want to avoid this. What macro or function can
> substitute it?

Why is this needed by your code? Surely the GC does its thing as an implementation detail of python.

Barry


> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 


From barry at barrys-emacs.org  Sun Dec 26 11:33:33 2021
From: barry at barrys-emacs.org (Barry Scott)
Date: Sun, 26 Dec 2021 16:33:33 +0000
Subject: Uninstall old python version(Python 3.9.7)
In-Reply-To: <50F0D6C7-86F4-4C48-902A-618B1BE98B5E@hxcore.ol>
References: <50F0D6C7-86F4-4C48-902A-618B1BE98B5E@hxcore.ol>
Message-ID: <421145A4-F414-4888-81C1-F7C6096F8770@barrys-emacs.org>



> On 25 Dec 2021, at 06:13, C?ng Huy <cowsmile1911 at gmail.com> wrote:
> 
>   I had uninstalled Python 3.9.7 before but it wasn't uninstalled
>   completely. I found it still in my computer and when I click "uninstall",
>   it sent me a board "No Python 3.9 installation was detected". Of course,
>   it won't be an issue if it defaulted my main Python version on my computer
>   and I can't use some libraries I installed like numpy, matplotlib,
>   networkx,... To solve my python homework. I found some help on the
>   internet but it wasn't improved. I don't know how to fix it. Please help
>   me.

You seem to saying that you need 3.9, but you have removed it.
I'm guessing there is more going on then you have explained.
Otherwise simply reinstall python 3.9.

Barry

p.s. You signature contains an advert for outlook.
You might want to get rid of that.


From arj.python at gmail.com  Sun Dec 26 11:40:03 2021
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Sun, 26 Dec 2021 20:40:03 +0400
Subject: A Newspaper for Python Mailing Lists
Message-ID: <CADrxXXkyz3RQcvfqCqV5U=o7r8N2O+z6Y5SX3Yc-6YMhHcn9SA@mail.gmail.com>

Greetings lists,

I have started a newspaper (not newsletter) focused
on interesting reads on Python mailing lists. Don't tag
on the fact that holiday seasons are the worst times for
launch according to marketing folks, I started this to note
down interesting mails. This might also be a great way to
bring mailing list gems to a wider readership. So, here's
the url https://pyherald.com/

Kind Regards,

Abdur-Rahmaan Janhangeer
about <https://compileralchemy.github.io/> | blog
<https://www.pythonkitchen.com>
github <https://github.com/Abdur-RahmaanJ>
Mauritius

From barry at barrys-emacs.org  Sun Dec 26 11:43:53 2021
From: barry at barrys-emacs.org (Barry Scott)
Date: Sun, 26 Dec 2021 16:43:53 +0000
Subject: recover pickled data: pickle data was truncated
In-Reply-To: <CABbU2U993rM06sVHSqnU31wZDfBc5d9C0g7dotBb2vht8KNqdA@mail.gmail.com>
References: <62d5de12-5e74-46f1-8d31-ea6b879dcb68n@googlegroups.com>
 <CABbU2U993rM06sVHSqnU31wZDfBc5d9C0g7dotBb2vht8KNqdA@mail.gmail.com>
Message-ID: <8D16D1A2-F374-44B1-A062-E60D5962F1FF@barrys-emacs.org>



> On 26 Dec 2021, at 13:44, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
> 
> Use a semaphore.
> 
> On Sun, 26 Dec 2021 at 03:30, iMath <redstone-cold at 163.com> wrote:
>> 
>> Normally, the shelve data should be read and write by only one process at a time, but unfortunately it was simultaneously read and write by two processes, thus corrupted it. Is there any way to recover all data in it ? Currently I just get "pickle data was truncated" exception after reading a portion of the data?

You have lost the data in that case.

You will need to do what Marco suggests and lock access to the file.
How you do that depends your OS. If is unix OS then its likely you
will want to use fcntl.flock().

Barry


From Marco.Sulla.Python at gmail.com  Mon Dec 27 13:30:12 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Mon, 27 Dec 2021 19:30:12 +0100
Subject: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
In-Reply-To: <CC507B7F-0A6C-415A-952E-2C7CDC75C9EC@barrys-emacs.org>
References: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>
 <CC507B7F-0A6C-415A-952E-2C7CDC75C9EC@barrys-emacs.org>
Message-ID: <CABbU2U9JLyt=7Up81sry7HV+4OyyZ23hZ=i1fxpGk-Y+_zhNaA@mail.gmail.com>

I need it since I'm developing an immutable dict. And in dict that
function is used.

I do not understand why there's no public API for that function. It
seems very useful.

On Sun, 26 Dec 2021 at 17:28, Barry Scott <barry at barrys-emacs.org> wrote:
>
>
>
> > On 26 Dec 2021, at 13:48, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
> >
> > I have to use _PyObject_GC_IS_TRACKED(). It can't be used unless you
> > define Py_BUILD_CORE. I want to avoid this. What macro or function can
> > substitute it?
>
> Why is this needed by your code? Surely the GC does its thing as an implementation detail of python.
>
> Barry
>
>
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> >
>

From aniket.howrah at gmail.com  Sun Dec 26 00:30:59 2021
From: aniket.howrah at gmail.com (Aniket)
Date: Sun, 26 Dec 2021 11:00:59 +0530
Subject: PIP Certificate problem
Message-ID: <B552FC18-1C88-4342-B430-62DD1ACA1AB8@hxcore.ol>

   Hi there,

   System Specs:

   Edition Windows 11 Home Single Language

   Version              21H2

   OS build            22000.376

   Experience       Windows Feature Experience Pack 1000.22000.376.0



   Processor         Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz   1.80 GHz

   Installed RAM  8.00 GB (7.89 GB usable)

   System type     64-bit operating system, x64-based processor





   I am repeatedly facing this issue while installing ANY pip module:



   PS D:\Documents\GitHub\mysite> pip install flask

   WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None,
   status=None)) after connection broken by
   'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
   certificate verify failed: self signed certificate in certificate chain
   (_ssl.c:997)'))': /simple/flask/

   WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None,
   status=None)) after connection broken by
   'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
   certificate verify failed: self signed certificate in certificate chain
   (_ssl.c:997)'))': /simple/flask/

   WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None,
   status=None)) after connection broken by
   'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
   certificate verify failed: self signed certificate in certificate chain
   (_ssl.c:997)'))': /simple/flask/

   WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None,
   status=None)) after connection broken by
   'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
   certificate verify failed: self signed certificate in certificate chain
   (_ssl.c:997)'))': /simple/flask/

   WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None,
   status=None)) after connection broken by
   'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
   certificate verify failed: self signed certificate in certificate chain
   (_ssl.c:997)'))': /simple/flask/

   Could not fetch URL https://pypi.org/simple/flask/: There was a problem
   confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org',
   port=443): Max retries exceeded with url: /simple/flask/ (Caused by
   SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
   certificate verify failed: self signed certificate in certificate chain
   (_ssl.c:997)'))) - skipping

   ERROR: Could not find a version that satisfies the requirement flask (from
   versions: none)

   ERROR: No matching distribution found for flask



   I have reinstalled python, and repaired but no help at all. My version of
   Python is 3.10.1



   Thank You

   Aniket Patra





From rosuav at gmail.com  Mon Dec 27 16:02:19 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 28 Dec 2021 08:02:19 +1100
Subject: PIP Certificate problem
In-Reply-To: <B552FC18-1C88-4342-B430-62DD1ACA1AB8@hxcore.ol>
References: <B552FC18-1C88-4342-B430-62DD1ACA1AB8@hxcore.ol>
Message-ID: <CAPTjJmqPTVq5sQD8qD7aNLGPU6ACe3nKCJHn7gq8E6thapNcVw@mail.gmail.com>

On Tue, Dec 28, 2021 at 8:00 AM Aniket <aniket.howrah at gmail.com> wrote:
>
>    I have reinstalled python, and repaired but no help at all. My version of
>    Python is 3.10.1
>

What does "pip --version" tell you?

ChrisA

From songofacandy at gmail.com  Tue Dec 28 01:58:09 2021
From: songofacandy at gmail.com (Inada Naoki)
Date: Tue, 28 Dec 2021 15:58:09 +0900
Subject: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
In-Reply-To: <CABbU2U9JLyt=7Up81sry7HV+4OyyZ23hZ=i1fxpGk-Y+_zhNaA@mail.gmail.com>
References: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>
 <CC507B7F-0A6C-415A-952E-2C7CDC75C9EC@barrys-emacs.org>
 <CABbU2U9JLyt=7Up81sry7HV+4OyyZ23hZ=i1fxpGk-Y+_zhNaA@mail.gmail.com>
Message-ID: <CAEfz+TwYhn71gEoZ5Xd0=0ni-YCnwjpjuqHokWJCr8CXeGbq4g@mail.gmail.com>

On Tue, Dec 28, 2021 at 3:31 AM Marco Sulla
<Marco.Sulla.Python at gmail.com> wrote:
>
> I need it since I'm developing an immutable dict. And in dict that
> function is used.
>
> I do not understand why there's no public API for that function. It
> seems very useful.
>

I think it is useful only for optimization based on *current* Python internals.
That's why it is not a public API. If we expose it as public API, it
makes harder to change Python's GC internals.


-- 
Inada Naoki  <songofacandy at gmail.com>

From Marco.Sulla.Python at gmail.com  Tue Dec 28 02:40:37 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Tue, 28 Dec 2021 08:40:37 +0100
Subject: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
In-Reply-To: <CAEfz+TwYhn71gEoZ5Xd0=0ni-YCnwjpjuqHokWJCr8CXeGbq4g@mail.gmail.com>
References: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>
 <CC507B7F-0A6C-415A-952E-2C7CDC75C9EC@barrys-emacs.org>
 <CABbU2U9JLyt=7Up81sry7HV+4OyyZ23hZ=i1fxpGk-Y+_zhNaA@mail.gmail.com>
 <CAEfz+TwYhn71gEoZ5Xd0=0ni-YCnwjpjuqHokWJCr8CXeGbq4g@mail.gmail.com>
Message-ID: <CABbU2U8YEjjQgd029YkYeDFBNbz5da8E+Nfyr-q0GYgLf2WRuA@mail.gmail.com>

Hi, Inada Senpai. So I do not need PyObject_GC_Track on cloning or
merging, or MAINTAIN_TRACKING on insert?

On Tue, 28 Dec 2021 at 07:58, Inada Naoki <songofacandy at gmail.com> wrote:
>
> On Tue, Dec 28, 2021 at 3:31 AM Marco Sulla
> <Marco.Sulla.Python at gmail.com> wrote:
> >
> > I need it since I'm developing an immutable dict. And in dict that
> > function is used.
> >
> > I do not understand why there's no public API for that function. It
> > seems very useful.
> >
>
> I think it is useful only for optimization based on *current* Python internals.
> That's why it is not a public API. If we expose it as public API, it
> makes harder to change Python's GC internals.
>
>
> --
> Inada Naoki  <songofacandy at gmail.com>

From songofacandy at gmail.com  Tue Dec 28 06:38:30 2021
From: songofacandy at gmail.com (Inada Naoki)
Date: Tue, 28 Dec 2021 20:38:30 +0900
Subject: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
In-Reply-To: <CABbU2U8YEjjQgd029YkYeDFBNbz5da8E+Nfyr-q0GYgLf2WRuA@mail.gmail.com>
References: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>
 <CC507B7F-0A6C-415A-952E-2C7CDC75C9EC@barrys-emacs.org>
 <CABbU2U9JLyt=7Up81sry7HV+4OyyZ23hZ=i1fxpGk-Y+_zhNaA@mail.gmail.com>
 <CAEfz+TwYhn71gEoZ5Xd0=0ni-YCnwjpjuqHokWJCr8CXeGbq4g@mail.gmail.com>
 <CABbU2U8YEjjQgd029YkYeDFBNbz5da8E+Nfyr-q0GYgLf2WRuA@mail.gmail.com>
Message-ID: <CAEfz+TzcPkC6k8xB1NBmEN4+eGzmPONYeGR-W6u_qrKkBStPkw@mail.gmail.com>

On Tue, Dec 28, 2021 at 4:41 PM Marco Sulla
<Marco.Sulla.Python at gmail.com> wrote:
>
> Hi, Inada Senpai. So I do not need PyObject_GC_Track on cloning or
> merging, or MAINTAIN_TRACKING on insert?
>

Your case is special.
You want to create a frozendict which performance is same to builtin dict.
Builtin dict has special optimization which tightly coupled with
current CPython implementation.
So you need to use private APIs for MAINTAIN_TRACKING.

But PyObject_GC_Track() is a public API.

Regards,

-- 
Inada Naoki  <songofacandy at gmail.com>

From Marco.Sulla.Python at gmail.com  Tue Dec 28 14:53:39 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Tue, 28 Dec 2021 20:53:39 +0100
Subject: Option for venv to upgrade pip automatically?
Message-ID: <CABbU2U8Zc+PdWOEn32jp-7L70PB+V7ZcCwdBgnw_n98exd0qew@mail.gmail.com>

I think it's very boring that, after creating a venv, you have
immediately to do every time:

pip install -U pip

Can't venv have an option for doing this automatically or, better, a
config file where you can put commands that will be launched every
time after you create a venv?

From Marco.Sulla.Python at gmail.com  Tue Dec 28 15:17:43 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Tue, 28 Dec 2021 21:17:43 +0100
Subject: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
In-Reply-To: <CAEfz+TzcPkC6k8xB1NBmEN4+eGzmPONYeGR-W6u_qrKkBStPkw@mail.gmail.com>
References: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>
 <CC507B7F-0A6C-415A-952E-2C7CDC75C9EC@barrys-emacs.org>
 <CABbU2U9JLyt=7Up81sry7HV+4OyyZ23hZ=i1fxpGk-Y+_zhNaA@mail.gmail.com>
 <CAEfz+TwYhn71gEoZ5Xd0=0ni-YCnwjpjuqHokWJCr8CXeGbq4g@mail.gmail.com>
 <CABbU2U8YEjjQgd029YkYeDFBNbz5da8E+Nfyr-q0GYgLf2WRuA@mail.gmail.com>
 <CAEfz+TzcPkC6k8xB1NBmEN4+eGzmPONYeGR-W6u_qrKkBStPkw@mail.gmail.com>
Message-ID: <CABbU2U-W=7RX9HkhGE6C8_MNTUEsJvZ-XpVdj5feH-+zHHhQzg@mail.gmail.com>

On Tue, 28 Dec 2021 at 12:38, Inada Naoki <songofacandy at gmail.com> wrote:
> Your case is special.
> You want to create a frozendict which performance is same to builtin dict.
> Builtin dict has special optimization which tightly coupled with
> current CPython implementation.
> So you need to use private APIs for MAINTAIN_TRACKING.

I solved this problem with a hacky trick: I included a reduced and
slightly modified version of dictobject.c. Furthermore I copy / pasted
stringlib\eq.h and _Py_bit_length. I'm currently doing this in a
refactor branch.
(Yes, I know that including a .c is very bad... but I need to do this
to separate the code of dict from the code of frozendict. Putting all
in the same files mess my head)

> But PyObject_GC_Track() is a public API.

The problem is I can't invoke PyObject_GC_Track() on an already
tracked object. I tried it and Python segfaulted. That's why CPython
uses _PyObject_GC_IS_TRACKED() before.

I'll try to copy/paste it too... :D but I do not understand why
there's not a public version of it.

From dieter at handshake.de  Tue Dec 28 18:03:53 2021
From: dieter at handshake.de (Dieter Maurer)
Date: Wed, 29 Dec 2021 00:03:53 +0100
Subject: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
In-Reply-To: <CABbU2U9JLyt=7Up81sry7HV+4OyyZ23hZ=i1fxpGk-Y+_zhNaA@mail.gmail.com>
References: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>
 <CC507B7F-0A6C-415A-952E-2C7CDC75C9EC@barrys-emacs.org>
 <CABbU2U9JLyt=7Up81sry7HV+4OyyZ23hZ=i1fxpGk-Y+_zhNaA@mail.gmail.com>
Message-ID: <25035.38873.999288.900094@ixdm.fritz.box>

Marco Sulla wrote at 2021-12-27 19:30 +0100:
>I need it since I'm developing an immutable dict. And in dict that
>function is used.

Why do you not derive from `dict` and override its mutating methods
(to raise a type error after initialization is complete)?

From songofacandy at gmail.com  Wed Dec 29 01:10:05 2021
From: songofacandy at gmail.com (Inada Naoki)
Date: Wed, 29 Dec 2021 15:10:05 +0900
Subject: Option for venv to upgrade pip automatically?
In-Reply-To: <CABbU2U8Zc+PdWOEn32jp-7L70PB+V7ZcCwdBgnw_n98exd0qew@mail.gmail.com>
References: <CABbU2U8Zc+PdWOEn32jp-7L70PB+V7ZcCwdBgnw_n98exd0qew@mail.gmail.com>
Message-ID: <CAEfz+Twr3zFEX0y9GS1xYWD7Zw-kMPzO+NCp1iP-Kt5Gb0zFCg@mail.gmail.com>

You can use --upgrade-deps option. My alias is:

  alias mkvenv='python3 -m venv --upgrade-deps --prompt . venv'

On Wed, Dec 29, 2021 at 4:55 AM Marco Sulla
<Marco.Sulla.Python at gmail.com> wrote:
>
> I think it's very boring that, after creating a venv, you have
> immediately to do every time:
>
> pip install -U pip
>
> Can't venv have an option for doing this automatically or, better, a
> config file where you can put commands that will be launched every
> time after you create a venv?
> --
> https://mail.python.org/mailman/listinfo/python-list



-- 
Inada Naoki  <songofacandy at gmail.com>

From songofacandy at gmail.com  Wed Dec 29 01:46:17 2021
From: songofacandy at gmail.com (Inada Naoki)
Date: Wed, 29 Dec 2021 15:46:17 +0900
Subject: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
In-Reply-To: <CABbU2U-W=7RX9HkhGE6C8_MNTUEsJvZ-XpVdj5feH-+zHHhQzg@mail.gmail.com>
References: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>
 <CC507B7F-0A6C-415A-952E-2C7CDC75C9EC@barrys-emacs.org>
 <CABbU2U9JLyt=7Up81sry7HV+4OyyZ23hZ=i1fxpGk-Y+_zhNaA@mail.gmail.com>
 <CAEfz+TwYhn71gEoZ5Xd0=0ni-YCnwjpjuqHokWJCr8CXeGbq4g@mail.gmail.com>
 <CABbU2U8YEjjQgd029YkYeDFBNbz5da8E+Nfyr-q0GYgLf2WRuA@mail.gmail.com>
 <CAEfz+TzcPkC6k8xB1NBmEN4+eGzmPONYeGR-W6u_qrKkBStPkw@mail.gmail.com>
 <CABbU2U-W=7RX9HkhGE6C8_MNTUEsJvZ-XpVdj5feH-+zHHhQzg@mail.gmail.com>
Message-ID: <CAEfz+Tz3Yn=du5e8J-XSemjt+_mPL3HxqTfntCagwyYpG08-jw@mail.gmail.com>

On Wed, Dec 29, 2021 at 5:18 AM Marco Sulla
<Marco.Sulla.Python at gmail.com> wrote:
>
>
> > But PyObject_GC_Track() is a public API.
>
> The problem is I can't invoke PyObject_GC_Track() on an already
> tracked object. I tried it and Python segfaulted. That's why CPython
> uses _PyObject_GC_IS_TRACKED() before.
>

You are right. I thought PyObject_GC_Track() can be used to tracked
objects because PyObject_GC_Untrack() can be used untracked object.
I think there is no enough reason for this asymmetry.

Additionally, adding PyObject_GC_IsTracked() to public API will not
bother future Python improvements.
If Python changed its GC to mark-and-sweep, PyObject_GC_IsTracked()
can return true always.

Regards,

-- 
Inada Naoki  <songofacandy at gmail.com>

From Marco.Sulla.Python at gmail.com  Wed Dec 29 02:08:49 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Wed, 29 Dec 2021 08:08:49 +0100
Subject: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
In-Reply-To: <25035.38873.999288.900094@ixdm.fritz.box>
References: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>
 <CC507B7F-0A6C-415A-952E-2C7CDC75C9EC@barrys-emacs.org>
 <CABbU2U9JLyt=7Up81sry7HV+4OyyZ23hZ=i1fxpGk-Y+_zhNaA@mail.gmail.com>
 <25035.38873.999288.900094@ixdm.fritz.box>
Message-ID: <CABbU2U-pqXeu6Ezoz09TDzANrSs+VfHT9cCFzP2xy5zLOcM6qQ@mail.gmail.com>

On Wed, 29 Dec 2021 at 00:03, Dieter Maurer <dieter at handshake.de> wrote:
> Why do you not derive from `dict` and override its mutating methods
> (to raise a type error after initialization is complete)?

I've done this for the pure py version, for speed. But in this way,
frozendict results to be a subclass of MutableMapping.

From Marco.Sulla.Python at gmail.com  Wed Dec 29 02:10:20 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Wed, 29 Dec 2021 08:10:20 +0100
Subject: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
In-Reply-To: <CAEfz+Tz3Yn=du5e8J-XSemjt+_mPL3HxqTfntCagwyYpG08-jw@mail.gmail.com>
References: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>
 <CC507B7F-0A6C-415A-952E-2C7CDC75C9EC@barrys-emacs.org>
 <CABbU2U9JLyt=7Up81sry7HV+4OyyZ23hZ=i1fxpGk-Y+_zhNaA@mail.gmail.com>
 <CAEfz+TwYhn71gEoZ5Xd0=0ni-YCnwjpjuqHokWJCr8CXeGbq4g@mail.gmail.com>
 <CABbU2U8YEjjQgd029YkYeDFBNbz5da8E+Nfyr-q0GYgLf2WRuA@mail.gmail.com>
 <CAEfz+TzcPkC6k8xB1NBmEN4+eGzmPONYeGR-W6u_qrKkBStPkw@mail.gmail.com>
 <CABbU2U-W=7RX9HkhGE6C8_MNTUEsJvZ-XpVdj5feH-+zHHhQzg@mail.gmail.com>
 <CAEfz+Tz3Yn=du5e8J-XSemjt+_mPL3HxqTfntCagwyYpG08-jw@mail.gmail.com>
Message-ID: <CABbU2U8NeBmmsvRtXCCa5KrR1tTC6wRnNb001kwNT_Xqe06BRQ@mail.gmail.com>

On Wed, 29 Dec 2021 at 07:46, Inada Naoki <songofacandy at gmail.com> wrote:
> You are right. I thought PyObject_GC_Track() can be used to tracked
> objects because PyObject_GC_Untrack() can be used untracked object.
> I think there is no enough reason for this asymmetry.
>
> Additionally, adding PyObject_GC_IsTracked() to public API will not
> bother future Python improvements.
> If Python changed its GC to mark-and-sweep, PyObject_GC_IsTracked()
> can return true always.

I think you are right :)

From dieter at handshake.de  Wed Dec 29 03:12:36 2021
From: dieter at handshake.de (Dieter Maurer)
Date: Wed, 29 Dec 2021 09:12:36 +0100
Subject: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
In-Reply-To: <CABbU2U-pqXeu6Ezoz09TDzANrSs+VfHT9cCFzP2xy5zLOcM6qQ@mail.gmail.com>
References: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>
 <CC507B7F-0A6C-415A-952E-2C7CDC75C9EC@barrys-emacs.org>
 <CABbU2U9JLyt=7Up81sry7HV+4OyyZ23hZ=i1fxpGk-Y+_zhNaA@mail.gmail.com>
 <25035.38873.999288.900094@ixdm.fritz.box>
 <CABbU2U-pqXeu6Ezoz09TDzANrSs+VfHT9cCFzP2xy5zLOcM6qQ@mail.gmail.com>
Message-ID: <25036.6260.111047.628470@ixdm.fritz.box>

Marco Sulla wrote at 2021-12-29 08:08 +0100:
>On Wed, 29 Dec 2021 at 00:03, Dieter Maurer <dieter at handshake.de> wrote:
>> Why do you not derive from `dict` and override its mutating methods
>> (to raise a type error after initialization is complete)?
>
>I've done this for the pure py version, for speed. But in this way,
>frozendict results to be a subclass of MutableMapping.

`MutableMapping` is a so called abstract base class (--> `abc`).

It uses the `__subclass_check__` (and `__instance_check__`) of
`abc.ABCMeta` to ensure `issubclass(dict, MutableMapping)`.
Those can be customized by overriding `MutableMapping.__subclasshook__`
to ensure that your `frozendict` class (and their subclasses)
are not considered subclasses of `MutableMapping`.

There is a PEP (I forgot its number, but searching for any of
the special method names above should locate it) describing
the `issubclass/isinstance` extension logik.

From Marco.Sulla.Python at gmail.com  Wed Dec 29 03:18:30 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Wed, 29 Dec 2021 09:18:30 +0100
Subject: Option for venv to upgrade pip automatically?
In-Reply-To: <CAEfz+Twr3zFEX0y9GS1xYWD7Zw-kMPzO+NCp1iP-Kt5Gb0zFCg@mail.gmail.com>
References: <CABbU2U8Zc+PdWOEn32jp-7L70PB+V7ZcCwdBgnw_n98exd0qew@mail.gmail.com>
 <CAEfz+Twr3zFEX0y9GS1xYWD7Zw-kMPzO+NCp1iP-Kt5Gb0zFCg@mail.gmail.com>
Message-ID: <CABbU2U-5sUD03WrVtrDYbNq1-FbtW84LXMT4_-UhUVDwK6GjGA@mail.gmail.com>

Cool, thanks!

On Wed, 29 Dec 2021 at 07:10, Inada Naoki <songofacandy at gmail.com> wrote:
>
> You can use --upgrade-deps option. My alias is:
>
>   alias mkvenv='python3 -m venv --upgrade-deps --prompt . venv'
>
> On Wed, Dec 29, 2021 at 4:55 AM Marco Sulla
> <Marco.Sulla.Python at gmail.com> wrote:
> >
> > I think it's very boring that, after creating a venv, you have
> > immediately to do every time:
> >
> > pip install -U pip
> >
> > Can't venv have an option for doing this automatically or, better, a
> > config file where you can put commands that will be launched every
> > time after you create a venv?
> > --
> > https://mail.python.org/mailman/listinfo/python-list
>
>
>
> --
> Inada Naoki  <songofacandy at gmail.com>

From Marco.Sulla.Python at gmail.com  Wed Dec 29 03:24:35 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Wed, 29 Dec 2021 09:24:35 +0100
Subject: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
In-Reply-To: <25036.6260.111047.628470@ixdm.fritz.box>
References: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>
 <CC507B7F-0A6C-415A-952E-2C7CDC75C9EC@barrys-emacs.org>
 <CABbU2U9JLyt=7Up81sry7HV+4OyyZ23hZ=i1fxpGk-Y+_zhNaA@mail.gmail.com>
 <25035.38873.999288.900094@ixdm.fritz.box>
 <CABbU2U-pqXeu6Ezoz09TDzANrSs+VfHT9cCFzP2xy5zLOcM6qQ@mail.gmail.com>
 <25036.6260.111047.628470@ixdm.fritz.box>
Message-ID: <CABbU2U8AwOi7wTDH4yPO=NW61hoTF2DuU203d4Ss7rg1i7VJwg@mail.gmail.com>

On Wed, 29 Dec 2021 at 09:12, Dieter Maurer <dieter at handshake.de> wrote:
>
> Marco Sulla wrote at 2021-12-29 08:08 +0100:
> >On Wed, 29 Dec 2021 at 00:03, Dieter Maurer <dieter at handshake.de> wrote:
> >> Why do you not derive from `dict` and override its mutating methods
> >> (to raise a type error after initialization is complete)?
> >
> >I've done this for the pure py version, for speed. But in this way,
> >frozendict results to be a subclass of MutableMapping.
>
> `MutableMapping` is a so called abstract base class (--> `abc`).
>
> It uses the `__subclass_check__` (and `__instance_check__`) of
> `abc.ABCMeta` to ensure `issubclass(dict, MutableMapping)`.
> Those can be customized by overriding `MutableMapping.__subclasshook__`
> to ensure that your `frozendict` class (and their subclasses)
> are not considered subclasses of `MutableMapping`.

Emh. Too hacky for me too, sorry :D

From Marco.Sulla.Python at gmail.com  Wed Dec 29 03:29:06 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Wed, 29 Dec 2021 09:29:06 +0100
Subject: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
In-Reply-To: <CABbU2U8AwOi7wTDH4yPO=NW61hoTF2DuU203d4Ss7rg1i7VJwg@mail.gmail.com>
References: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>
 <CC507B7F-0A6C-415A-952E-2C7CDC75C9EC@barrys-emacs.org>
 <CABbU2U9JLyt=7Up81sry7HV+4OyyZ23hZ=i1fxpGk-Y+_zhNaA@mail.gmail.com>
 <25035.38873.999288.900094@ixdm.fritz.box>
 <CABbU2U-pqXeu6Ezoz09TDzANrSs+VfHT9cCFzP2xy5zLOcM6qQ@mail.gmail.com>
 <25036.6260.111047.628470@ixdm.fritz.box>
 <CABbU2U8AwOi7wTDH4yPO=NW61hoTF2DuU203d4Ss7rg1i7VJwg@mail.gmail.com>
Message-ID: <CABbU2U-Dv+=4eMP8mHxhyRTOWC21Q2bJiEaC8PG12fDSVVg5zw@mail.gmail.com>

On second thought, I think I'll do this for the pure py version. But I
will definitely not do this for the C extension, since it's anyway
strange that an immutable mapping inherits from a mutable one! I've
done it in the pure py version only for a matter of speed.

On Wed, 29 Dec 2021 at 09:24, Marco Sulla <Marco.Sulla.Python at gmail.com> wrote:
>
> On Wed, 29 Dec 2021 at 09:12, Dieter Maurer <dieter at handshake.de> wrote:
> >
> > Marco Sulla wrote at 2021-12-29 08:08 +0100:
> > >On Wed, 29 Dec 2021 at 00:03, Dieter Maurer <dieter at handshake.de> wrote:
> > >> Why do you not derive from `dict` and override its mutating methods
> > >> (to raise a type error after initialization is complete)?
> > >
> > >I've done this for the pure py version, for speed. But in this way,
> > >frozendict results to be a subclass of MutableMapping.
> >
> > `MutableMapping` is a so called abstract base class (--> `abc`).
> >
> > It uses the `__subclass_check__` (and `__instance_check__`) of
> > `abc.ABCMeta` to ensure `issubclass(dict, MutableMapping)`.
> > Those can be customized by overriding `MutableMapping.__subclasshook__`
> > to ensure that your `frozendict` class (and their subclasses)
> > are not considered subclasses of `MutableMapping`.
>
> Emh. Too hacky for me too, sorry :D

From dieter at handshake.de  Wed Dec 29 04:06:23 2021
From: dieter at handshake.de (Dieter Maurer)
Date: Wed, 29 Dec 2021 10:06:23 +0100
Subject: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
In-Reply-To: <CABbU2U-Dv+=4eMP8mHxhyRTOWC21Q2bJiEaC8PG12fDSVVg5zw@mail.gmail.com>
References: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>
 <CC507B7F-0A6C-415A-952E-2C7CDC75C9EC@barrys-emacs.org>
 <CABbU2U9JLyt=7Up81sry7HV+4OyyZ23hZ=i1fxpGk-Y+_zhNaA@mail.gmail.com>
 <25035.38873.999288.900094@ixdm.fritz.box>
 <CABbU2U-pqXeu6Ezoz09TDzANrSs+VfHT9cCFzP2xy5zLOcM6qQ@mail.gmail.com>
 <25036.6260.111047.628470@ixdm.fritz.box>
 <CABbU2U8AwOi7wTDH4yPO=NW61hoTF2DuU203d4Ss7rg1i7VJwg@mail.gmail.com>
 <CABbU2U-Dv+=4eMP8mHxhyRTOWC21Q2bJiEaC8PG12fDSVVg5zw@mail.gmail.com>
Message-ID: <25036.9487.982997.883611@ixdm.fritz.box>

Marco Sulla wrote at 2021-12-29 09:29 +0100:
>On second thought, I think I'll do this for the pure py version. But I
>will definitely not do this for the C extension

Are you sure you need to implement your type in C at all?

I made a small `timeit` test:
```
>>> class cd(dict): pass
...
>>> timeit("d[1]", "d={1:1}", globals=globals())
0.02474160000019765
>>> timeit("d[1]", "d=cd({1:1})", globals=globals())
0.08281239100051607
```
This means that for the above trivial case, access is 3.5 times slower
(the difference is smaller for more complex cases when hashing
becomes more expensive) but it is still only 83 ns per access.
Thus, if your application is not highly dominated by dict accesses,
the overall difference will not be large.

From rosuav at gmail.com  Wed Dec 29 04:27:53 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 29 Dec 2021 20:27:53 +1100
Subject: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
In-Reply-To: <25036.9487.982997.883611@ixdm.fritz.box>
References: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>
 <CC507B7F-0A6C-415A-952E-2C7CDC75C9EC@barrys-emacs.org>
 <CABbU2U9JLyt=7Up81sry7HV+4OyyZ23hZ=i1fxpGk-Y+_zhNaA@mail.gmail.com>
 <25035.38873.999288.900094@ixdm.fritz.box>
 <CABbU2U-pqXeu6Ezoz09TDzANrSs+VfHT9cCFzP2xy5zLOcM6qQ@mail.gmail.com>
 <25036.6260.111047.628470@ixdm.fritz.box>
 <CABbU2U8AwOi7wTDH4yPO=NW61hoTF2DuU203d4Ss7rg1i7VJwg@mail.gmail.com>
 <CABbU2U-Dv+=4eMP8mHxhyRTOWC21Q2bJiEaC8PG12fDSVVg5zw@mail.gmail.com>
 <25036.9487.982997.883611@ixdm.fritz.box>
Message-ID: <CAPTjJmrmBTL01=tw=dMJHEgK51-p+oWh7XAKXr=qzqaac91SxA@mail.gmail.com>

On Wed, Dec 29, 2021 at 8:07 PM Dieter Maurer <dieter at handshake.de> wrote:
>
> Marco Sulla wrote at 2021-12-29 09:29 +0100:
> >On second thought, I think I'll do this for the pure py version. But I
> >will definitely not do this for the C extension
>
> Are you sure you need to implement your type in C at all?
>
> I made a small `timeit` test:
> ```
> >>> class cd(dict): pass
> ...
> >>> timeit("d[1]", "d={1:1}", globals=globals())
> 0.02474160000019765
> >>> timeit("d[1]", "d=cd({1:1})", globals=globals())
> 0.08281239100051607
> ```
> This means that for the above trivial case, access is 3.5 times slower
> (the difference is smaller for more complex cases when hashing
> becomes more expensive) but it is still only 83 ns per access.
> Thus, if your application is not highly dominated by dict accesses,
> the overall difference will not be large.

You forgot slots. The difference can be even smaller.

>>> class cd(dict): __slots__ = ()
...
>>> timeit("d[1]", "d={1:1}", globals=globals())
0.02511977031826973
>>> timeit("d[1]", "d=cd({1:1})", globals=globals())
0.0333079993724823

ChrisA

From Marco.Sulla.Python at gmail.com  Wed Dec 29 05:14:32 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Wed, 29 Dec 2021 11:14:32 +0100
Subject: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
In-Reply-To: <25036.9487.982997.883611@ixdm.fritz.box>
References: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>
 <CC507B7F-0A6C-415A-952E-2C7CDC75C9EC@barrys-emacs.org>
 <CABbU2U9JLyt=7Up81sry7HV+4OyyZ23hZ=i1fxpGk-Y+_zhNaA@mail.gmail.com>
 <25035.38873.999288.900094@ixdm.fritz.box>
 <CABbU2U-pqXeu6Ezoz09TDzANrSs+VfHT9cCFzP2xy5zLOcM6qQ@mail.gmail.com>
 <25036.6260.111047.628470@ixdm.fritz.box>
 <CABbU2U8AwOi7wTDH4yPO=NW61hoTF2DuU203d4Ss7rg1i7VJwg@mail.gmail.com>
 <CABbU2U-Dv+=4eMP8mHxhyRTOWC21Q2bJiEaC8PG12fDSVVg5zw@mail.gmail.com>
 <25036.9487.982997.883611@ixdm.fritz.box>
Message-ID: <CABbU2U8tfHTvaMbRA-haJ=Xp_xYt+26EtKXnuDBwhYwWCPa0CQ@mail.gmail.com>

On Wed, 29 Dec 2021 at 10:06, Dieter Maurer <dieter at handshake.de> wrote:
>
> Are you sure you need to implement your type in C at all?

It's already implemented, and, in some cases, is faster than dict:

https://github.com/Marco-Sulla/python-frozendict#benchmarks

PS: I'm doing a refactoring that speeds up creation even further,
making it almost as fast as dict.

From Marco.Sulla.Python at gmail.com  Wed Dec 29 05:24:16 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Wed, 29 Dec 2021 11:24:16 +0100
Subject: How to implement freelists in dict 3.10 for previous versions?
Message-ID: <CABbU2U8nqnFTjF9FUp_YcWgmQSPK4c52pu+Gahw9GO2mPGhhBw@mail.gmail.com>

I noticed that now freelists in dict use _Py_dict_state. I suppose
this is done for thread safety.

I would implement it also for a C extension that uses CPython < 3.10.
How can I achieve this?

From Marco.Sulla.Python at gmail.com  Wed Dec 29 05:59:21 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Wed, 29 Dec 2021 11:59:21 +0100
Subject: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
In-Reply-To: <25036.6260.111047.628470@ixdm.fritz.box>
References: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>
 <CC507B7F-0A6C-415A-952E-2C7CDC75C9EC@barrys-emacs.org>
 <CABbU2U9JLyt=7Up81sry7HV+4OyyZ23hZ=i1fxpGk-Y+_zhNaA@mail.gmail.com>
 <25035.38873.999288.900094@ixdm.fritz.box>
 <CABbU2U-pqXeu6Ezoz09TDzANrSs+VfHT9cCFzP2xy5zLOcM6qQ@mail.gmail.com>
 <25036.6260.111047.628470@ixdm.fritz.box>
Message-ID: <CABbU2U8Phh0Jk+UkbDxojScDct9wAeOqdgU5pr4+8DyrOZhuXA@mail.gmail.com>

On Wed, 29 Dec 2021 at 09:12, Dieter Maurer <dieter at handshake.de> wrote:
> `MutableMapping` is a so called abstract base class (--> `abc`).
>
> It uses the `__subclass_check__` (and `__instance_check__`) of
> `abc.ABCMeta` to ensure `issubclass(dict, MutableMapping)`.
> Those can be customized by overriding `MutableMapping.__subclasshook__`
> to ensure that your `frozendict` class (and their subclasses)
> are not considered subclasses of `MutableMapping`.

It does not work:

$ python
Python 3.10.0 (heads/3.10-dirty:f6e8b80d20, Nov 18 2021, 19:16:18)
[GCC 10.1.1 20200718] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import frozendict
>>> frozendict.c_ext
False
>>> from frozendict import frozendict as fd
>>> from collections.abc import MutableMapping as Mm
>>> issubclass(fd, Mm)
True
>>> @classmethod
... def _my_subclasshook(klass, subclass):
...     if subclass == fd:
...             return False
...     return NotImplemented
...
>>> @classmethod
... def _my_subclasshook(klass, subclass):
...     print(subclass)
...     if subclass == fd:
...             return False
...     return NotImplemented
...
>>> Mm.__subclasshook__ = _my_subclasshook
>>> issubclass(fd, Mm)
True
>>> issubclass(tuple, Mm)
<class 'tuple'>
<class 'tuple'>
<class 'tuple'>
<class 'tuple'>
<class 'tuple'>
<class 'tuple'>
False
>>>

From dieter at handshake.de  Wed Dec 29 06:10:57 2021
From: dieter at handshake.de (Dieter Maurer)
Date: Wed, 29 Dec 2021 12:10:57 +0100
Subject: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
In-Reply-To: <CABbU2U8Phh0Jk+UkbDxojScDct9wAeOqdgU5pr4+8DyrOZhuXA@mail.gmail.com>
References: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>
 <CC507B7F-0A6C-415A-952E-2C7CDC75C9EC@barrys-emacs.org>
 <CABbU2U9JLyt=7Up81sry7HV+4OyyZ23hZ=i1fxpGk-Y+_zhNaA@mail.gmail.com>
 <25035.38873.999288.900094@ixdm.fritz.box>
 <CABbU2U-pqXeu6Ezoz09TDzANrSs+VfHT9cCFzP2xy5zLOcM6qQ@mail.gmail.com>
 <25036.6260.111047.628470@ixdm.fritz.box>
 <CABbU2U8Phh0Jk+UkbDxojScDct9wAeOqdgU5pr4+8DyrOZhuXA@mail.gmail.com>
Message-ID: <25036.16961.430330.965010@ixdm.fritz.box>

Marco Sulla wrote at 2021-12-29 11:59 +0100:
>On Wed, 29 Dec 2021 at 09:12, Dieter Maurer <dieter at handshake.de> wrote:
>> `MutableMapping` is a so called abstract base class (--> `abc`).
>>
>> It uses the `__subclass_check__` (and `__instance_check__`) of
>> `abc.ABCMeta` to ensure `issubclass(dict, MutableMapping)`.
>> Those can be customized by overriding `MutableMapping.__subclasshook__`
>> to ensure that your `frozendict` class (and their subclasses)
>> are not considered subclasses of `MutableMapping`.
>
>It does not work:
> ...
>>>> issubclass(fd, Mm)
>True

There is a cache involved. The `issubclass` above,
brings your `fd` in the `Mn`'s subclass cache.

> ...
>>>> Mm.__subclasshook__ = _my_subclasshook
>>>> issubclass(fd, Mm)
>True

See above.

From Marco.Sulla.Python at gmail.com  Wed Dec 29 07:45:12 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Wed, 29 Dec 2021 13:45:12 +0100
Subject: What's the public API alternative to _PyObject_GC_IS_TRACKED()?
In-Reply-To: <25036.16961.430330.965010@ixdm.fritz.box>
References: <CABbU2U89U=itsPwsU8+POREYMCf9F3U=_CciwVO0ynCAJ0aFcQ@mail.gmail.com>
 <CC507B7F-0A6C-415A-952E-2C7CDC75C9EC@barrys-emacs.org>
 <CABbU2U9JLyt=7Up81sry7HV+4OyyZ23hZ=i1fxpGk-Y+_zhNaA@mail.gmail.com>
 <25035.38873.999288.900094@ixdm.fritz.box>
 <CABbU2U-pqXeu6Ezoz09TDzANrSs+VfHT9cCFzP2xy5zLOcM6qQ@mail.gmail.com>
 <25036.6260.111047.628470@ixdm.fritz.box>
 <CABbU2U8Phh0Jk+UkbDxojScDct9wAeOqdgU5pr4+8DyrOZhuXA@mail.gmail.com>
 <25036.16961.430330.965010@ixdm.fritz.box>
Message-ID: <CABbU2U8uXN9Be24OFV1COhBNhfy9MR+UZg2Z5EM3fL90um9bSQ@mail.gmail.com>

On Wed, 29 Dec 2021 at 12:11, Dieter Maurer <dieter at handshake.de> wrote:
>
> Marco Sulla wrote at 2021-12-29 11:59 +0100:
> >On Wed, 29 Dec 2021 at 09:12, Dieter Maurer <dieter at handshake.de> wrote:
> >> `MutableMapping` is a so called abstract base class (--> `abc`).
> >>
> >> It uses the `__subclass_check__` (and `__instance_check__`) of
> >> `abc.ABCMeta` to ensure `issubclass(dict, MutableMapping)`.
> >> Those can be customized by overriding `MutableMapping.__subclasshook__`
> >> to ensure that your `frozendict` class (and their subclasses)
> >> are not considered subclasses of `MutableMapping`.
> >
> >It does not work:
> > ...
> >>>> issubclass(fd, Mm)
> >True
>
> There is a cache involved. The `issubclass` above,
> brings your `fd` in the `Mn`'s subclass cache.

It works, thank you! I had to put it before

Mapping.register(frozendict)

From redstone-cold at 163.com  Wed Dec 29 10:50:43 2021
From: redstone-cold at 163.com (iMath)
Date: Wed, 29 Dec 2021 07:50:43 -0800 (PST)
Subject: recover pickled data: pickle data was truncated
In-Reply-To: <mailman.41.1640537343.3079.python-list@python.org>
References: <62d5de12-5e74-46f1-8d31-ea6b879dcb68n@googlegroups.com>
 <8D16D1A2-F374-44B1-A062-E60D5962F1FF@barrys-emacs.org>
 <CABbU2U993rM06sVHSqnU31wZDfBc5d9C0g7dotBb2vht8KNqdA@mail.gmail.com>
 <mailman.41.1640537343.3079.python-list@python.org>
Message-ID: <3e32c4ad-f736-405f-bef2-ffa5ca340421n@googlegroups.com>

> You have lost the data in that case. 

But I found the size of the file of the shelve data didn't change much, so I guess the data are still in it , I just wonder any way to recover my data.

From rosuav at gmail.com  Wed Dec 29 13:24:14 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 30 Dec 2021 05:24:14 +1100
Subject: recover pickled data: pickle data was truncated
In-Reply-To: <3e32c4ad-f736-405f-bef2-ffa5ca340421n@googlegroups.com>
References: <62d5de12-5e74-46f1-8d31-ea6b879dcb68n@googlegroups.com>
 <8D16D1A2-F374-44B1-A062-E60D5962F1FF@barrys-emacs.org>
 <CABbU2U993rM06sVHSqnU31wZDfBc5d9C0g7dotBb2vht8KNqdA@mail.gmail.com>
 <mailman.41.1640537343.3079.python-list@python.org>
 <3e32c4ad-f736-405f-bef2-ffa5ca340421n@googlegroups.com>
Message-ID: <CAPTjJmrCoD0tUMxAGsB8fagBLe-wx8eC7RoKBV7CU2E18XUfQQ@mail.gmail.com>

On Thu, Dec 30, 2021 at 4:32 AM iMath <redstone-cold at 163.com> wrote:
>
> > You have lost the data in that case.
>
> But I found the size of the file of the shelve data didn't change much, so I guess the data are still in it , I just wonder any way to recover my data.

Unless two conflicting versions got interleaved, in which case I
strongly advise you NOT to try unpickling it.

If you really feel like delving into it, try manually decoding the
pickle stream, but be very careful.

ChrisA

From avigross at verizon.net  Wed Dec 29 13:54:14 2021
From: avigross at verizon.net (Avi Gross)
Date: Wed, 29 Dec 2021 13:54:14 -0500
Subject: recover pickled data: pickle data was truncated
In-Reply-To: <3e32c4ad-f736-405f-bef2-ffa5ca340421n@googlegroups.com>
References: <62d5de12-5e74-46f1-8d31-ea6b879dcb68n@googlegroups.com>
 <8D16D1A2-F374-44B1-A062-E60D5962F1FF@barrys-emacs.org>
 <CABbU2U993rM06sVHSqnU31wZDfBc5d9C0g7dotBb2vht8KNqdA@mail.gmail.com>
 <mailman.41.1640537343.3079.python-list@python.org>
 <3e32c4ad-f736-405f-bef2-ffa5ca340421n@googlegroups.com>
Message-ID: <014f01d7fce5$79a16690$6ce433b0$@verizon.net>

I am not an expert on the topic but my first reaction is it depends on how
the data is corrupted and we do not know that. So I am addressing a more
general concept here.

Some algorithms break if a single byte or even bit changes and nothing
beyond that point makes sense. Many encryption techniques are like that and
adding or deleting a byte might throw things off completely. 

But if your problem is that two processes or threads wrote interleaved and
yet resulted in an output of a similar size, then, yes, in some cases some
of the data could be retrieved, albeit be fragmentary and unreliable. If
they both included say a data structure with names and phone numbers, it is
possible you get two partial or complete copies and maybe retrieve a phone
number you can try and see if it works. But the tax authorities might not
react favorably to your recovery of a business expense if it is possible the
currency amount was corrupted and perhaps a few zeroes were appended at the
end.

For some mission-critical purposes, I am sure people have come up with many
ideas including perhaps making multiple copies before an exit spread across
multiple disks and sites or reading the file back in and checking it. But
corruption can happen for many reasons including at the level of the disk it
is written to.

-----Original Message-----
From: Python-list <python-list-bounces+avigross=verizon.net at python.org> On
Behalf Of iMath
Sent: Wednesday, December 29, 2021 10:51 AM
To: python-list at python.org
Subject: Re: recover pickled data: pickle data was truncated

> You have lost the data in that case. 

But I found the size of the file of the shelve data didn't change much, so I
guess the data are still in it , I just wonder any way to recover my data.
-- 
https://mail.python.org/mailman/listinfo/python-list


From Marco.Sulla.Python at gmail.com  Wed Dec 29 14:12:24 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Wed, 29 Dec 2021 20:12:24 +0100
Subject: recover pickled data: pickle data was truncated
In-Reply-To: <3e32c4ad-f736-405f-bef2-ffa5ca340421n@googlegroups.com>
References: <62d5de12-5e74-46f1-8d31-ea6b879dcb68n@googlegroups.com>
 <8D16D1A2-F374-44B1-A062-E60D5962F1FF@barrys-emacs.org>
 <CABbU2U993rM06sVHSqnU31wZDfBc5d9C0g7dotBb2vht8KNqdA@mail.gmail.com>
 <mailman.41.1640537343.3079.python-list@python.org>
 <3e32c4ad-f736-405f-bef2-ffa5ca340421n@googlegroups.com>
Message-ID: <CABbU2U_JuDog4MKcPgAjVPHj6MrNa-22Y5Xo7QKy8hptH6MbZA@mail.gmail.com>

On Wed, 29 Dec 2021 at 18:33, iMath <redstone-cold at 163.com> wrote:
> But I found the size of the file of the shelve data didn't change much, so I guess the data are still in it , I just wonder any way to recover my data.

I agree with Barry, Chris and Avi. IMHO your data is lost. Unpickling
it by hand is a harsh work and maybe unreliable.

Is there any reason you can't simply add a semaphore to avoid writing
at the same time and re-run the code and regenerate the data?

From songofacandy at gmail.com  Thu Dec 30 01:59:29 2021
From: songofacandy at gmail.com (Inada Naoki)
Date: Thu, 30 Dec 2021 15:59:29 +0900
Subject: How to implement freelists in dict 3.10 for previous versions?
In-Reply-To: <CABbU2U8nqnFTjF9FUp_YcWgmQSPK4c52pu+Gahw9GO2mPGhhBw@mail.gmail.com>
References: <CABbU2U8nqnFTjF9FUp_YcWgmQSPK4c52pu+Gahw9GO2mPGhhBw@mail.gmail.com>
Message-ID: <CAEfz+Tz-0VJBD4EX3FqiAbRuLpiRAU4avS3DmkS7Jcib9uH1qA@mail.gmail.com>

On Wed, Dec 29, 2021 at 7:25 PM Marco Sulla
<Marco.Sulla.Python at gmail.com> wrote:
>
> I noticed that now freelists in dict use _Py_dict_state. I suppose
> this is done for thread safety.
>

Some core-dev are working on per-interpreter GIL. But it is not done yet.
So you don't need to follow it soon. Your extension module will work
well in Python 3.11.

> I would implement it also for a C extension that uses CPython < 3.10.
> How can I achieve this?

See PyModule_GetState() to have per-interpreter module state instead
of static variables.
https://docs.python.org/3/c-api/module.html#c.PyModule_GetState


-- 
Inada Naoki  <songofacandy at gmail.com>

From hongyi.zhao at gmail.com  Thu Dec 30 08:04:24 2021
From: hongyi.zhao at gmail.com (hongy...@gmail.com)
Date: Thu, 30 Dec 2021 05:04:24 -0800 (PST)
Subject: builtins.TypeError: catching classes that do not inherit from
 BaseException is not allowed
Message-ID: <d7cd019a-9a61-43e5-a3aa-8256a528580bn@googlegroups.com>

I try to compute the factorial of a large number with tail-recursion optimization decorator in Python3. The following code snippet is converted from the code snippet given here [1] by the following steps:

$ pyenv shell datasci
$ python --version
Python 3.9.1
$ pip install 2to3
$ 2to3 -w this-script.py

```
# This program shows off a python decorator(
# which implements tail call optimization. It
# does this by throwing an exception if it is 
# its own grandparent, and catching such 
# exceptions to recall the stack.

import sys

class TailRecurseException:
  def __init__(self, args, kwargs):
    self.args = args
    self.kwargs = kwargs

def tail_call_optimized(g):
  """
  This function decorates a function with tail call
  optimization. It does this by throwing an exception
  if it is its own grandparent, and catching such
  exceptions to fake the tail call optimization.

  This function fails if the decorated
  function recurses in a non-tail context.
  """
  def func(*args, **kwargs):
    f = sys._getframe()
    if f.f_back and f.f_back.f_back \
        and f.f_back.f_back.f_code == f.f_code:
      raise TailRecurseException(args, kwargs)
    else:
      while 1:
        try:
          return g(*args, **kwargs)
        except TailRecurseException as e:
          args = e.args
          kwargs = e.kwargs
  func.__doc__ = g.__doc__
  return func

@tail_call_optimized
def factorial(n, acc=1):
  "calculate a factorial"
  if n == 0:
    return acc
  return factorial(n-1, n*acc)

print(factorial(10000))
# prints a big, big number,
# but doesn't hit the recursion limit.

@tail_call_optimized
def fib(i, current = 0, next = 1):
  if i == 0:
    return current
  else:
    return fib(i - 1, next, current + next)

print(fib(10000))
# also prints a big number,
# but doesn't hit the recursion limit.
```
However, when I try to test the above script, the following error will be triggered:
```
$ python this-script.py 
Traceback (most recent call last):
  File "/home/werner/this-script.py", line 32, in func
    return g(*args, **kwargs)
  File "/home/werner/this-script.py", line 44, in factorial
    return factorial(n-1, n*acc)
  File "/home/werner/this-script.py", line 28, in func
    raise TailRecurseException(args, kwargs)
TypeError: exceptions must derive from BaseException

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/werner/this-script.py", line 46, in <module>
    print(factorial(10000))
  File "/home/werner/this-script.py", line 33, in func
    except TailRecurseException as e:
TypeError: catching classes that do not inherit from BaseException is not allowed
```

Any hints for fixing this problem will be highly appreciated.

[1]  https://stackoverflow.com/q/27417874

Regards,
HZ

From hongyi.zhao at gmail.com  Thu Dec 30 08:14:17 2021
From: hongyi.zhao at gmail.com (hongy...@gmail.com)
Date: Thu, 30 Dec 2021 05:14:17 -0800 (PST)
Subject: builtins.TypeError: catching classes that do not inherit from
 BaseException is not allowed
In-Reply-To: <d7cd019a-9a61-43e5-a3aa-8256a528580bn@googlegroups.com>
References: <d7cd019a-9a61-43e5-a3aa-8256a528580bn@googlegroups.com>
Message-ID: <c1f8e807-d1a5-4947-be56-7cdb20514f1dn@googlegroups.com>

On Thursday, December 30, 2021 at 9:04:36 PM UTC+8, hongy... at gmail.com wrote:
> I try to compute the factorial of a large number with tail-recursion optimization decorator in Python3. The following code snippet is converted from the code snippet given here [1] by the following steps: 
> 
> $ pyenv shell datasci 
> $ python --version 
> Python 3.9.1 
> $ pip install 2to3 
> $ 2to3 -w this-script.py 
> 
> ``` 
> # This program shows off a python decorator( 
> # which implements tail call optimization. It 
> # does this by throwing an exception if it is 
> # its own grandparent, and catching such 
> # exceptions to recall the stack. 
> 
> import sys 
> 
> class TailRecurseException: 
> def __init__(self, args, kwargs): 
> self.args = args 
> self.kwargs = kwargs 
> 
> def tail_call_optimized(g): 
> """ 
> This function decorates a function with tail call 
> optimization. It does this by throwing an exception 
> if it is its own grandparent, and catching such 
> exceptions to fake the tail call optimization. 
> 
> This function fails if the decorated 
> function recurses in a non-tail context. 
> """ 
> def func(*args, **kwargs): 
> f = sys._getframe() 
> if f.f_back and f.f_back.f_back \ 
> and f.f_back.f_back.f_code == f.f_code: 
> raise TailRecurseException(args, kwargs) 
> else: 
> while 1: 
> try: 
> return g(*args, **kwargs) 
> except TailRecurseException as e: 
> args = e.args 
> kwargs = e.kwargs 
> func.__doc__ = g.__doc__ 
> return func 
> 
> @tail_call_optimized 
> def factorial(n, acc=1): 
> "calculate a factorial" 
> if n == 0: 
> return acc 
> return factorial(n-1, n*acc) 
> 
> print(factorial(10000)) 
> # prints a big, big number, 
> # but doesn't hit the recursion limit. 
> 
> @tail_call_optimized 
> def fib(i, current = 0, next = 1): 
> if i == 0: 
> return current 
> else: 
> return fib(i - 1, next, current + next) 
> 
> print(fib(10000)) 
> # also prints a big number, 
> # but doesn't hit the recursion limit. 
> ``` 
> However, when I try to test the above script, the following error will be triggered: 
> ``` 
> $ python this-script.py 
> Traceback (most recent call last): 
> File "/home/werner/this-script.py", line 32, in func 
> return g(*args, **kwargs) 
> File "/home/werner/this-script.py", line 44, in factorial 
> return factorial(n-1, n*acc) 
> File "/home/werner/this-script.py", line 28, in func 
> raise TailRecurseException(args, kwargs) 
> TypeError: exceptions must derive from BaseException 
> 
> During handling of the above exception, another exception occurred: 
> 
> Traceback (most recent call last): 
> File "/home/werner/this-script.py", line 46, in <module> 
> print(factorial(10000)) 
> File "/home/werner/this-script.py", line 33, in func 
> except TailRecurseException as e: 
> TypeError: catching classes that do not inherit from BaseException is not allowed 
> ``` 
> 
> Any hints for fixing this problem will be highly appreciated. 

See here [1] for the related discussion. 

[1] https://discuss.python.org/t/typeerror-catching-classes-that-do-not-inherit-from-baseexception-is-not-allowed/12800

HZ

From rosuav at gmail.com  Thu Dec 30 10:23:05 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 31 Dec 2021 02:23:05 +1100
Subject: builtins.TypeError: catching classes that do not inherit from
 BaseException is not allowed
In-Reply-To: <d7cd019a-9a61-43e5-a3aa-8256a528580bn@googlegroups.com>
References: <d7cd019a-9a61-43e5-a3aa-8256a528580bn@googlegroups.com>
Message-ID: <CAPTjJmq22Xez9SuNys971C_xmX+xE70HfxOSX_AVqVWeuoNOkA@mail.gmail.com>

On Fri, Dec 31, 2021 at 2:00 AM hongy... at gmail.com
<hongyi.zhao at gmail.com> wrote:
>
> I try to compute the factorial of a large number with tail-recursion optimization decorator in Python3. The following code snippet is converted from the code snippet given here [1] by the following steps:
>
> $ pyenv shell datasci
> $ python --version
> Python 3.9.1
> $ pip install 2to3
> $ 2to3 -w this-script.py
>
> ```
> # This program shows off a python decorator(
> # which implements tail call optimization. It
> # does this by throwing an exception if it is
> # its own grandparent, and catching such
> # exceptions to recall the stack.
>
> import sys
>
> class TailRecurseException:
>   def __init__(self, args, kwargs):
>     self.args = args
>     self.kwargs = kwargs
>

If it's an exception, it needs to subclass Exception or BaseException.

(Also, is this REALLY an optimization? Exception handling isn't the
fastest. Yes, it avoids some measure of recursion depth, but it looks
like a pretty inefficient way to do things. Python is not Lisp, and
there are very very few algorithms that actually benefit from tail
call optimization that wouldn't benefit far more from other ways of
doing the same thing.)

ChrisA

From rosuav at gmail.com  Thu Dec 30 10:23:49 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 31 Dec 2021 02:23:49 +1100
Subject: builtins.TypeError: catching classes that do not inherit from
 BaseException is not allowed
In-Reply-To: <c1f8e807-d1a5-4947-be56-7cdb20514f1dn@googlegroups.com>
References: <d7cd019a-9a61-43e5-a3aa-8256a528580bn@googlegroups.com>
 <c1f8e807-d1a5-4947-be56-7cdb20514f1dn@googlegroups.com>
Message-ID: <CAPTjJmpXZ9FbymOHUvXU0JaxETjCq0TXZCLHffcy9iJpzpOtaA@mail.gmail.com>

On Fri, Dec 31, 2021 at 2:03 AM hongy... at gmail.com
<hongyi.zhao at gmail.com> wrote:
> See here [1] for the related discussion.
>
> [1] https://discuss.python.org/t/typeerror-catching-classes-that-do-not-inherit-from-baseexception-is-not-allowed/12800

Why did you post in two places at once? Did you need more people to
tell you the same thing as the error message?

ChrisA

From hongyi.zhao at gmail.com  Thu Dec 30 17:27:28 2021
From: hongyi.zhao at gmail.com (hongy...@gmail.com)
Date: Thu, 30 Dec 2021 14:27:28 -0800 (PST)
Subject: builtins.TypeError: catching classes that do not inherit from
 BaseException is not allowed
In-Reply-To: <mailman.71.1640877841.3079.python-list@python.org>
References: <d7cd019a-9a61-43e5-a3aa-8256a528580bn@googlegroups.com>
 <CAPTjJmpXZ9FbymOHUvXU0JaxETjCq0TXZCLHffcy9iJpzpOtaA@mail.gmail.com>
 <c1f8e807-d1a5-4947-be56-7cdb20514f1dn@googlegroups.com>
 <mailman.71.1640877841.3079.python-list@python.org>
Message-ID: <2a276e53-b385-4323-8631-f92bb8a5df8dn@googlegroups.com>

On Thursday, December 30, 2021 at 11:24:20 PM UTC+8, Chris Angelico wrote:
> On Fri, Dec 31, 2021 at 2:03 AM hongy... at gmail.com 
> <hongy... at gmail.com> wrote: 
> > See here [1] for the related discussion. 
> > 
> > [1] https://discuss.python.org/t/typeerror-catching-classes-that-do-not-inherit-from-baseexception-is-not-allowed/12800
> Why did you post in two places at once? Did you need more people to 
> tell you the same thing as the error message? 

Doing so may attract the attention of developers, such as increasing the content of traceback information to make troubleshooting easier, just as Andr? Roberge says here [1].

[1] https://discuss.python.org/t/typeerror-catching-classes-that-do-not-inherit-from-baseexception-is-not-allowed/12800/6?u=hongyi-zhao

 > ChrisA

HZ

From hongyi.zhao at gmail.com  Thu Dec 30 17:32:51 2021
From: hongyi.zhao at gmail.com (hongy...@gmail.com)
Date: Thu, 30 Dec 2021 14:32:51 -0800 (PST)
Subject: builtins.TypeError: catching classes that do not inherit from
 BaseException is not allowed
In-Reply-To: <mailman.70.1640877797.3079.python-list@python.org>
References: <CAPTjJmq22Xez9SuNys971C_xmX+xE70HfxOSX_AVqVWeuoNOkA@mail.gmail.com>
 <d7cd019a-9a61-43e5-a3aa-8256a528580bn@googlegroups.com>
 <mailman.70.1640877797.3079.python-list@python.org>
Message-ID: <170b346e-2923-4519-b337-1db7b1978684n@googlegroups.com>

On Thursday, December 30, 2021 at 11:23:35 PM UTC+8, Chris Angelico wrote:
> If it's an exception, it needs to subclass Exception or BaseException. 

I see. That is, the following:

class TailRecurseException(Exception):
  def __init__(self, args, kwargs):
    self.args = args
    self.kwargs = kwargs

 
> (Also, is this REALLY an optimization? Exception handling isn't the 
> fastest. Yes, it avoids some measure of recursion depth, but it looks 
> like a pretty inefficient way to do things. Python is not Lisp, and 
> there are very very few algorithms that actually benefit from tail 
> call optimization that wouldn't benefit far more from other ways of 
> doing the same thing.) 

Could you give some examples of the other methods you mentioned above?
 
> ChrisA

HZ

From rosuav at gmail.com  Thu Dec 30 17:58:47 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 31 Dec 2021 09:58:47 +1100
Subject: builtins.TypeError: catching classes that do not inherit from
 BaseException is not allowed
In-Reply-To: <2a276e53-b385-4323-8631-f92bb8a5df8dn@googlegroups.com>
References: <d7cd019a-9a61-43e5-a3aa-8256a528580bn@googlegroups.com>
 <CAPTjJmpXZ9FbymOHUvXU0JaxETjCq0TXZCLHffcy9iJpzpOtaA@mail.gmail.com>
 <c1f8e807-d1a5-4947-be56-7cdb20514f1dn@googlegroups.com>
 <mailman.71.1640877841.3079.python-list@python.org>
 <2a276e53-b385-4323-8631-f92bb8a5df8dn@googlegroups.com>
Message-ID: <CAPTjJmoyAcKZm=Nn-Y0r=6cm3ewTRzk6AuekHgn_1+wNFcN5cg@mail.gmail.com>

On Fri, Dec 31, 2021 at 9:42 AM hongy... at gmail.com
<hongyi.zhao at gmail.com> wrote:
>
> On Thursday, December 30, 2021 at 11:24:20 PM UTC+8, Chris Angelico wrote:
> > On Fri, Dec 31, 2021 at 2:03 AM hongy... at gmail.com
> > <hongy... at gmail.com> wrote:
> > > See here [1] for the related discussion.
> > >
> > > [1] https://discuss.python.org/t/typeerror-catching-classes-that-do-not-inherit-from-baseexception-is-not-allowed/12800
> > Why did you post in two places at once? Did you need more people to
> > tell you the same thing as the error message?
>
> Doing so may attract the attention of developers, such as increasing the content of traceback information to make troubleshooting easier, just as Andr? Roberge says here [1].
>

It's a simple question with a simple answer. You don't need to
cross-post. All you do is split the discussion so people don't know
what's already been said.

ChrisA

From rosuav at gmail.com  Thu Dec 30 18:03:54 2021
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 31 Dec 2021 10:03:54 +1100
Subject: builtins.TypeError: catching classes that do not inherit from
 BaseException is not allowed
In-Reply-To: <170b346e-2923-4519-b337-1db7b1978684n@googlegroups.com>
References: <CAPTjJmq22Xez9SuNys971C_xmX+xE70HfxOSX_AVqVWeuoNOkA@mail.gmail.com>
 <d7cd019a-9a61-43e5-a3aa-8256a528580bn@googlegroups.com>
 <mailman.70.1640877797.3079.python-list@python.org>
 <170b346e-2923-4519-b337-1db7b1978684n@googlegroups.com>
Message-ID: <CAPTjJmodUF0_pZjpSVPa6RtuYuCUVOcJu69nY-yqU8etqT2v2g@mail.gmail.com>

On Fri, Dec 31, 2021 at 9:42 AM hongy... at gmail.com
<hongyi.zhao at gmail.com> wrote:
> > (Also, is this REALLY an optimization? Exception handling isn't the
> > fastest. Yes, it avoids some measure of recursion depth, but it looks
> > like a pretty inefficient way to do things. Python is not Lisp, and
> > there are very very few algorithms that actually benefit from tail
> > call optimization that wouldn't benefit far more from other ways of
> > doing the same thing.)
>
> Could you give some examples of the other methods you mentioned above?
>

If you have a function that has just a single tail call site, there's
usually no point writing it recursively.

def factorial(n):
    ret = 1
    for i in range(1, n + 1): ret *= i
    return ret

def fibonacci(n):
    a, b = 0, 1
    for _ in range(n):
        a, b = b, a + b
    return a

Neither of these wants to be recursive, and writing them recursively
pollutes the function signature with parameters that really exist just
to be local variables. Passing an accumulator down is a terrible way
to demonstrate the beauty of recursion - it instead shows off how you
can shoehorn anything into recursion and make it worse in the process.

Please, everyone, stop trying to optimize the wrong things.

Write good code, don't try to make bad code stop crashing.

ChrisA

From virginia.a.bliss at gmail.com  Thu Dec 30 18:23:57 2021
From: virginia.a.bliss at gmail.com (vrg bls)
Date: Thu, 30 Dec 2021 15:23:57 -0800
Subject: using OpenGL on windows 11
Message-ID: <CADNO+12UZv+YPqb+_CFzyVh=jVjaFT-_EH7XS6Y6Qcw9LaXR9Q@mail.gmail.com>

  Hi! I am very new to Python admittedly, but did try several options
regarding troubleshooting to get OpenGL to run with Python. I am using
PyCharm as my interface, did try installing setup tools and have
reinstalled using pip install function a few times. I did notice my machine
seems to be having trouble remembering that I did install python and keeps
prompting me to install from MS app store. I did try to do this but again
did not register the installation of Python. I am able to open a window in
PyCharm and get other functions to work but then get the Module not
Installed error for OpenGL. I even tried the setuptools install and
easy_install never popped up as an option. is this a windows 11 issue? does
anything about this sound familiar?
Thank you, and sorry if this has already been answered in a forum
somewhere, I did look and could not find it


-- 
Virginia Bliss
virginia.a.bliss at gmail.com
678 640 1902
Los Angeles. CA

From hongyi.zhao at gmail.com  Thu Dec 30 18:27:18 2021
From: hongyi.zhao at gmail.com (hongy...@gmail.com)
Date: Thu, 30 Dec 2021 15:27:18 -0800 (PST)
Subject: builtins.TypeError: catching classes that do not inherit from
 BaseException is not allowed
In-Reply-To: <mailman.73.1640905447.3079.python-list@python.org>
References: <CAPTjJmq22Xez9SuNys971C_xmX+xE70HfxOSX_AVqVWeuoNOkA@mail.gmail.com>
 <d7cd019a-9a61-43e5-a3aa-8256a528580bn@googlegroups.com>
 <mailman.70.1640877797.3079.python-list@python.org>
 <CAPTjJmodUF0_pZjpSVPa6RtuYuCUVOcJu69nY-yqU8etqT2v2g@mail.gmail.com>
 <170b346e-2923-4519-b337-1db7b1978684n@googlegroups.com>
 <mailman.73.1640905447.3079.python-list@python.org>
Message-ID: <df8487f1-2a28-48a1-9c9d-dafae37abf26n@googlegroups.com>

On Friday, December 31, 2021 at 7:04:24 AM UTC+8, Chris Angelico wrote:
> Neither of these wants to be recursive, and writing them recursively 
> pollutes the function signature with parameters that really exist just 
> to be local variables. Passing an accumulator down is a terrible way 
> to demonstrate the beauty of recursion - it instead shows off how you 
> can shoehorn anything into recursion and make it worse in the process. 

Then what cases/scenarios can demonstrate the beauty of recursion?

HZ

From python at mrabarnett.plus.com  Thu Dec 30 18:50:20 2021
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 30 Dec 2021 23:50:20 +0000
Subject: builtins.TypeError: catching classes that do not inherit from
 BaseException is not allowed
In-Reply-To: <df8487f1-2a28-48a1-9c9d-dafae37abf26n@googlegroups.com>
References: <CAPTjJmq22Xez9SuNys971C_xmX+xE70HfxOSX_AVqVWeuoNOkA@mail.gmail.com>
 <d7cd019a-9a61-43e5-a3aa-8256a528580bn@googlegroups.com>
 <mailman.70.1640877797.3079.python-list@python.org>
 <CAPTjJmodUF0_pZjpSVPa6RtuYuCUVOcJu69nY-yqU8etqT2v2g@mail.gmail.com>
 <170b346e-2923-4519-b337-1db7b1978684n@googlegroups.com>
 <mailman.73.1640905447.3079.python-list@python.org>
 <df8487f1-2a28-48a1-9c9d-dafae37abf26n@googlegroups.com>
Message-ID: <2218d5fc-8862-9525-4c75-c2225e6247ac@mrabarnett.plus.com>

On 2021-12-30 23:27, hongy... at gmail.com wrote:
> On Friday, December 31, 2021 at 7:04:24 AM UTC+8, Chris Angelico wrote:
>> Neither of these wants to be recursive, and writing them recursively 
>> pollutes the function signature with parameters that really exist just 
>> to be local variables. Passing an accumulator down is a terrible way 
>> to demonstrate the beauty of recursion - it instead shows off how you 
>> can shoehorn anything into recursion and make it worse in the process. 
> 
> Then what cases/scenarios can demonstrate the beauty of recursion?
> 
Walking a tree.

From hongyi.zhao at gmail.com  Thu Dec 30 18:57:25 2021
From: hongyi.zhao at gmail.com (hongy...@gmail.com)
Date: Thu, 30 Dec 2021 15:57:25 -0800 (PST)
Subject: builtins.TypeError: catching classes that do not inherit from
 BaseException is not allowed
In-Reply-To: <mailman.75.1640908231.3079.python-list@python.org>
References: <CAPTjJmq22Xez9SuNys971C_xmX+xE70HfxOSX_AVqVWeuoNOkA@mail.gmail.com>
 <d7cd019a-9a61-43e5-a3aa-8256a528580bn@googlegroups.com>
 <mailman.70.1640877797.3079.python-list@python.org>
 <CAPTjJmodUF0_pZjpSVPa6RtuYuCUVOcJu69nY-yqU8etqT2v2g@mail.gmail.com>
 <170b346e-2923-4519-b337-1db7b1978684n@googlegroups.com>
 <mailman.73.1640905447.3079.python-list@python.org>
 <2218d5fc-8862-9525-4c75-c2225e6247ac@mrabarnett.plus.com>
 <df8487f1-2a28-48a1-9c9d-dafae37abf26n@googlegroups.com>
 <mailman.75.1640908231.3079.python-list@python.org>
Message-ID: <661ece76-fe30-49a9-85be-efc70d7dc395n@googlegroups.com>

On Friday, December 31, 2021 at 7:50:48 AM UTC+8, MRAB wrote:
> On 2021-12-30 23:27, hongy... at gmail.com wrote: 
> > On Friday, December 31, 2021 at 7:04:24 AM UTC+8, Chris Angelico wrote: 
> >> Neither of these wants to be recursive, and writing them recursively 
> >> pollutes the function signature with parameters that really exist just 
> >> to be local variables. Passing an accumulator down is a terrible way 
> >> to demonstrate the beauty of recursion - it instead shows off how you 
> >> can shoehorn anything into recursion and make it worse in the process. 
> > 
> > Then what cases/scenarios can demonstrate the beauty of recursion? 
> >
> Walking a tree.

There are many type of trees. Do you mean all of them?

From avigross at verizon.net  Thu Dec 30 19:10:57 2021
From: avigross at verizon.net (Avi Gross)
Date: Thu, 30 Dec 2021 19:10:57 -0500
Subject: builtins.TypeError: catching classes that do not inherit from
 BaseException is not allowed
In-Reply-To: <df8487f1-2a28-48a1-9c9d-dafae37abf26n@googlegroups.com>
References: <CAPTjJmq22Xez9SuNys971C_xmX+xE70HfxOSX_AVqVWeuoNOkA@mail.gmail.com>
 <d7cd019a-9a61-43e5-a3aa-8256a528580bn@googlegroups.com>
 <mailman.70.1640877797.3079.python-list@python.org>
 <CAPTjJmodUF0_pZjpSVPa6RtuYuCUVOcJu69nY-yqU8etqT2v2g@mail.gmail.com>
 <170b346e-2923-4519-b337-1db7b1978684n@googlegroups.com>
 <mailman.73.1640905447.3079.python-list@python.org>
 <df8487f1-2a28-48a1-9c9d-dafae37abf26n@googlegroups.com>
Message-ID: <014601d7fdda$e2dcad90$a89608b0$@verizon.net>

Beauty of Recursion?

Well, there is what I call Mathematical Beauty, and then there is reality.

It is fantastic to prove neat theorems that something is possible by methods
like mathematical induction that in some sense use recursion as in if
something is true for some base value and it can be shown that if it is true
for N then it also is true for N+1, then by a sort of recursion, it is true
for every larger value!

And, yes, you can write some seemingly neat and compact routines using
recursion, but there are many run-time costs as Chris and others can
suggest.

Where is recursion most useful? I can think of many places but will briefly
mention a few.

There is a recursive sort algorithm that keeps dividing the data it is
handled in two and calls itself recursively to process the first half and
then again on the second half and simply merges the sorted results and
returns that. Clearly this kind of algorithm calls itself to a depth of
about log to the base two times. And a possible advantage here is that this
can take some advantage of parallel architectures and some simultaneity. 

Another general set of recursive applications  is anything that wanders
tree-structures and looks for items or places new items in an appropriate
place. If it is a binary tree, there is a slight similarity with my first
example as there is a do_left and a do_right type of recursion but not
really. And of course trees can have more than two branches and you can use
recursion on other complex structures. But in some cases, you can come up
with a non-recursive way to do things by keeping track of where you are and
where you have been.

Recursion is a good thing to teach but also a horrible thing when misused. A
book I read decades ago, called The Little Lisper, showed algorithms such as
asking for greater(A, B) to be done sort of like by recursively calling
greater(subtract1(A), subtract1(B)) as long as neither A nor B becomes zero.
Neat algorithm and tolerable for greater(3,5) returning 5 after a few
recursions when 3 has been lowered to zero. Not so good with negative
numbers. But greater(10000000000, 10000000001) might take a while and your
machine may run out of memory and since you want to return the original
value, tail recursion may not be enough unless you modify things a bit such
as using a helper function that also includes the original numbers so it
returns the right one or ...

Many machines simply restrict you to compare numbers only up to the point
where an int or float of some kind support it and then the operation of
comparison is mostly just done in a few steps in the hardware no matter what
size the two things are. Python with support for unlimited size integers
though, ...


-----Original Message-----
From: Python-list <python-list-bounces+avigross=verizon.net at python.org> On
Behalf Of hongy... at gmail.com
Sent: Thursday, December 30, 2021 6:27 PM
To: python-list at python.org
Subject: Re: builtins.TypeError: catching classes that do not inherit from
BaseException is not allowed

On Friday, December 31, 2021 at 7:04:24 AM UTC+8, Chris Angelico wrote:
> Neither of these wants to be recursive, and writing them recursively 
> pollutes the function signature with parameters that really exist just 
> to be local variables. Passing an accumulator down is a terrible way 
> to demonstrate the beauty of recursion - it instead shows off how you 
> can shoehorn anything into recursion and make it worse in the process.

Then what cases/scenarios can demonstrate the beauty of recursion?

HZ
--
https://mail.python.org/mailman/listinfo/python-list


From Marco.Sulla.Python at gmail.com  Fri Dec 31 03:17:29 2021
From: Marco.Sulla.Python at gmail.com (Marco Sulla)
Date: Fri, 31 Dec 2021 09:17:29 +0100
Subject: builtins.TypeError: catching classes that do not inherit from
 BaseException is not allowed
In-Reply-To: <d7cd019a-9a61-43e5-a3aa-8256a528580bn@googlegroups.com>
References: <d7cd019a-9a61-43e5-a3aa-8256a528580bn@googlegroups.com>
Message-ID: <CABbU2U_Rg5XZk6hfQJMbpJDVsqLBzAhGLJSzEb=_e=UBSObc0Q@mail.gmail.com>

It was already done: https://pypi.org/project/tail-recursive/

On Thu, 30 Dec 2021 at 16:00, hongy... at gmail.com <hongyi.zhao at gmail.com> wrote:
>
> I try to compute the factorial of a large number with tail-recursion optimization decorator in Python3. The following code snippet is converted from the code snippet given here [1] by the following steps:
>
> $ pyenv shell datasci
> $ python --version
> Python 3.9.1
> $ pip install 2to3
> $ 2to3 -w this-script.py
>
> ```
> # This program shows off a python decorator(
> # which implements tail call optimization. It
> # does this by throwing an exception if it is
> # its own grandparent, and catching such
> # exceptions to recall the stack.
>
> import sys
>
> class TailRecurseException:
>   def __init__(self, args, kwargs):
>     self.args = args
>     self.kwargs = kwargs
>
> def tail_call_optimized(g):
>   """
>   This function decorates a function with tail call
>   optimization. It does this by throwing an exception
>   if it is its own grandparent, and catching such
>   exceptions to fake the tail call optimization.
>
>   This function fails if the decorated
>   function recurses in a non-tail context.
>   """
>   def func(*args, **kwargs):
>     f = sys._getframe()
>     if f.f_back and f.f_back.f_back \
>         and f.f_back.f_back.f_code == f.f_code:
>       raise TailRecurseException(args, kwargs)
>     else:
>       while 1:
>         try:
>           return g(*args, **kwargs)
>         except TailRecurseException as e:
>           args = e.args
>           kwargs = e.kwargs
>   func.__doc__ = g.__doc__
>   return func
>
> @tail_call_optimized
> def factorial(n, acc=1):
>   "calculate a factorial"
>   if n == 0:
>     return acc
>   return factorial(n-1, n*acc)
>
> print(factorial(10000))
> # prints a big, big number,
> # but doesn't hit the recursion limit.
>
> @tail_call_optimized
> def fib(i, current = 0, next = 1):
>   if i == 0:
>     return current
>   else:
>     return fib(i - 1, next, current + next)
>
> print(fib(10000))
> # also prints a big number,
> # but doesn't hit the recursion limit.
> ```
> However, when I try to test the above script, the following error will be triggered:
> ```
> $ python this-script.py
> Traceback (most recent call last):
>   File "/home/werner/this-script.py", line 32, in func
>     return g(*args, **kwargs)
>   File "/home/werner/this-script.py", line 44, in factorial
>     return factorial(n-1, n*acc)
>   File "/home/werner/this-script.py", line 28, in func
>     raise TailRecurseException(args, kwargs)
> TypeError: exceptions must derive from BaseException
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
>   File "/home/werner/this-script.py", line 46, in <module>
>     print(factorial(10000))
>   File "/home/werner/this-script.py", line 33, in func
>     except TailRecurseException as e:
> TypeError: catching classes that do not inherit from BaseException is not allowed
> ```
>
> Any hints for fixing this problem will be highly appreciated.
>
> [1]  https://stackoverflow.com/q/27417874
>
> Regards,
> HZ
> --
> https://mail.python.org/mailman/listinfo/python-list

From Karsten.Hilbert at gmx.net  Fri Dec 31 07:08:58 2021
From: Karsten.Hilbert at gmx.net (Karsten Hilbert)
Date: Fri, 31 Dec 2021 13:08:58 +0100
Subject: builtins.TypeError: catching classes that do not inherit from
 BaseException is not allowed
In-Reply-To: <661ece76-fe30-49a9-85be-efc70d7dc395n@googlegroups.com>
References: <CAPTjJmq22Xez9SuNys971C_xmX+xE70HfxOSX_AVqVWeuoNOkA@mail.gmail.com>
 <d7cd019a-9a61-43e5-a3aa-8256a528580bn@googlegroups.com>
 <mailman.70.1640877797.3079.python-list@python.org>
 <CAPTjJmodUF0_pZjpSVPa6RtuYuCUVOcJu69nY-yqU8etqT2v2g@mail.gmail.com>
 <170b346e-2923-4519-b337-1db7b1978684n@googlegroups.com>
 <mailman.73.1640905447.3079.python-list@python.org>
 <2218d5fc-8862-9525-4c75-c2225e6247ac@mrabarnett.plus.com>
 <df8487f1-2a28-48a1-9c9d-dafae37abf26n@googlegroups.com>
 <mailman.75.1640908231.3079.python-list@python.org>
 <661ece76-fe30-49a9-85be-efc70d7dc395n@googlegroups.com>
Message-ID: <Yc7y2uhvqI9BTxhv@hermes.hilbert.loc>

Am Thu, Dec 30, 2021 at 03:57:25PM -0800 schrieb hongy... at gmail.com:

> > > Then what cases/scenarios can demonstrate the beauty of recursion?
> > >
> > Walking a tree.
>
> There are many type of trees. Do you mean all of them?

Palm trees don't lend themselves to recursion all that much.

Karsten
--
GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B

From redstone-cold at 163.com  Fri Dec 31 05:55:05 2021
From: redstone-cold at 163.com (iMath)
Date: Fri, 31 Dec 2021 02:55:05 -0800 (PST)
Subject: recover pickled data: pickle data was truncated
In-Reply-To: <mailman.68.1640805182.3079.python-list@python.org>
References: <62d5de12-5e74-46f1-8d31-ea6b879dcb68n@googlegroups.com>
 <8D16D1A2-F374-44B1-A062-E60D5962F1FF@barrys-emacs.org>
 <CABbU2U993rM06sVHSqnU31wZDfBc5d9C0g7dotBb2vht8KNqdA@mail.gmail.com>
 <mailman.41.1640537343.3079.python-list@python.org>
 <CABbU2U_JuDog4MKcPgAjVPHj6MrNa-22Y5Xo7QKy8hptH6MbZA@mail.gmail.com>
 <3e32c4ad-f736-405f-bef2-ffa5ca340421n@googlegroups.com>
 <mailman.68.1640805182.3079.python-list@python.org>
Message-ID: <1549b721-f6b6-4754-a5ff-ead72eb78d3an@googlegroups.com>

? 2021?12?30???? UTC+8 03:13:21?<Marco Sulla> ???
> On Wed, 29 Dec 2021 at 18:33, iMath <redsto... at 163.com> wrote: 
> > But I found the size of the file of the shelve data didn't change much, so I guess the data are still in it , I just wonder any way to recover my data.
> I agree with Barry, Chris and Avi. IMHO your data is lost. Unpickling 
> it by hand is a harsh work and maybe unreliable. 
> 
> Is there any reason you can't simply add a semaphore to avoid writing 
> at the same time and re-run the code and regenerate the data?

Thanks for your replies! I didn't have a sense of adding a semaphore on writing to pickle data before, so  corrupted the data.
Since my data was colleted in the daily usage, so cannot re-run the code and regenerate the data.
In order to avoid corrupting my data again and the complicity of using  a semaphore, now I am using json text to store my data.

From hongyi.zhao at gmail.com  Fri Dec 31 06:17:08 2021
From: hongyi.zhao at gmail.com (hongy...@gmail.com)
Date: Fri, 31 Dec 2021 03:17:08 -0800 (PST)
Subject: builtins.TypeError: catching classes that do not inherit from
 BaseException is not allowed
In-Reply-To: <mailman.77.1640938688.3079.python-list@python.org>
References: <CABbU2U_Rg5XZk6hfQJMbpJDVsqLBzAhGLJSzEb=_e=UBSObc0Q@mail.gmail.com>
 <d7cd019a-9a61-43e5-a3aa-8256a528580bn@googlegroups.com>
 <mailman.77.1640938688.3079.python-list@python.org>
Message-ID: <5ae08d0a-a447-4877-9a44-e9bc970f9cc5n@googlegroups.com>

On Friday, December 31, 2021 at 4:18:28 PM UTC+8, Marco Sulla wrote:
> It was already done: https://pypi.org/project/tail-recursive/

A few days ago, I also noticed another similar project:
https://github.com/baruchel/tco

It seems that they are very similar, even identical. But I'm not sure, so I filed an issue here [1].

[1] https://github.com/0scarB/tail-recursive/issues/1

From hongyi.zhao at gmail.com  Fri Dec 31 06:24:12 2021
From: hongyi.zhao at gmail.com (hongy...@gmail.com)
Date: Fri, 31 Dec 2021 03:24:12 -0800 (PST)
Subject: builtins.TypeError: catching classes that do not inherit from
 BaseException is not allowed
In-Reply-To: <5ae08d0a-a447-4877-9a44-e9bc970f9cc5n@googlegroups.com>
References: <CABbU2U_Rg5XZk6hfQJMbpJDVsqLBzAhGLJSzEb=_e=UBSObc0Q@mail.gmail.com>
 <d7cd019a-9a61-43e5-a3aa-8256a528580bn@googlegroups.com>
 <mailman.77.1640938688.3079.python-list@python.org>
 <5ae08d0a-a447-4877-9a44-e9bc970f9cc5n@googlegroups.com>
Message-ID: <083a6dfc-535f-4099-84ef-61cd8e6bdc69n@googlegroups.com>

On Friday, December 31, 2021 at 7:17:18 PM UTC+8, hongy... at gmail.com wrote:
> On Friday, December 31, 2021 at 4:18:28 PM UTC+8, Marco Sulla wrote: 
> > It was already done: https://pypi.org/project/tail-recursive/
> A few days ago, I also noticed another similar project: 
> https://github.com/baruchel/tco 

And also see the following one:

https://pypi.org/project/tail-recursion/

 
> It seems that they are very similar, even identical. But I'm not sure, so I filed an issue here [1]. 
> 
> [1] https://github.com/0scarB/tail-recursive/issues/1